[Oracle Cloud] Data Flow をクイックスタートしてみた その1 (Java)

# はじめに

Oracle Cloud Infrastructure(以下OCI)では、Data Flow と呼ばれる、フルマネージドな分散処理基盤が提供されています。たくさんのノードで並列処理することで、大量なデータを短い時間で処理が出来ます。Apache Spark ベースで出来ているため、いままで Spark を使っていた方は、違和感なく Data Flow を使えます。

Data Flow でフルマネージドされる Spark クラスタの基盤の運用は、ユーザー側が意識する必要がありません。データ処理業務を行いたいときに Spark クラスタを使って分散データ処理を行います。ユーザー側で定義したデータ処理が終わったときには、自動的に Spark クラスタがシャットダウンされます。

今回の記事では、Data Flow のドキュメントに書かれているクイックスタートをやってみます。Data Flow がどんな感じで使うのかを確認します。

# Qiita 記事 Index

その1 Java
https://qiita.com/sugimount/items/d67f8aa02f0af89299de

その2 SQL
https://qiita.com/sugimount/items/1d8fe15ddbd58f0cdd76

その3 Python
https://qiita.com/sugimount/items/9a4c87493b569f174ba0

# Data Flow でサポートされているもの

https://docs.cloud.oracle.com/en-us/iaas/data-flow/using/dfs_getting_started.htm#before_you_begin

- Spark Version : 2.4.4
- Application で使える言語
  - Java
  - Scala
  - SparkSQL
  - PySpark (Python 3) のみ



なお、2020年4月現在では、Spark Streaming はサポートされていないようです。



# クイックスタート手順

以下のドキュメントに書かれている手順を参考にすすめていきます

https://docs.cloud.oracle.com/en-us/iaas/data-flow/data-flow-tutorial/tutorial/dfs_tut_etl_java.htm

- OCI で事前設定
- Object Storage に Spark Application をアップロード
- Data Flow でアプリケーションを定義
- Data Flow でアプリケーション実行



# IAM Policy

最低限必要な設定をしていきます。環境によって内容を変更する必要があります。Data Flow 側が Object Storage のバケットを参照するための IAM Policy が必要です。

```
ALLOW SERVICE dataflow TO READ objects IN tenancy WHERE target.bucket.name='dataflow-logs'
```



# Object Storage Bucketを作成

Data Flow の実行ログやアプリケーションを格納、Spark SQL での data warehouse のために Object Storage の Bucket を作成します。

- name : `dataflow-logs `
- name : `dataflow-warehouse`
- name : `dataflow-application`



次の画面が、OCI Console で Object Storage の Bucket を作成した例です。

![1587220769448.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/3ee1031b-2b6a-c8c2-c6a3-e2860f8d9c0d.png)



dataflow-logs の詳細画面を表示して、ネームスペースを確認します。以下の画面の例だと `nryjxkqe0mhq`  です。Data Flow で使うので、メモっておきます。

![1587222686097.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/a8a0b8a1-3dfc-a511-0d6a-6c7f15c27490.png)



# アプリケーションの定義

それでは、Data Flow で Application を作成します。OCI のメニューから、データ・フローの下にあるアプリケーションを選びます。その後、Create Application ボタンを押します。

![1587221359617.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/8e436be1-316e-c440-0ac1-ac98dc6e21f5.png)



以下のパラメータを入れます。

- 言語 : Java
- 名前 : Tutorial Example 1
- 説明 : Tutorial Example 1
- SPARK バージョン : 2.4.4
- ドライバ・タイプ : VM.Standard2.1
  - Spark アプリケーションの実行管理や制御を行うサービスに使用する Compute Instance を指定
- エグゼキュータ・タイプ : VM.Standard2.1
  - Spark アプリケーションのコードを実行する Compute Instance を指定
- エグゼキュータの数 : 1
- ファイルURL : `oci://oow_2019_dataflow_lab@bigdatadatasciencelarge/usercontent/oow-lab-2019-java-etl-1.0-SNAPSHOT.jar`
  - Object Storage にアップロードされている Data Flow で動かす Spack 用アプリケーションを指定します。今回はチュートリアルとして Oracle 側で準備されているものを使います
- メイン・クラス名：convert.Convert
  - Oracle 側で準備されている Java アプリケーションの Class 名を指定します。Java アプリケーションのソースコードを見ればわかるため、後で説明します
- 引数 : `${input} ${output}`
  - Java アプリケーションに渡す引数を定義します。Java ソースコードの中身を理解したうえで引数を指定します。
- input : `oci://oow_2019_dataflow_lab@bigdatadatasciencelarge/usercontent/kaggle_berlin_airbnb_listings_summary.csv`
- output : `oci://<yourbucket>@<namespace>/optimized_listings`
  - 事前に作成した Object Storage の Bucket と Namespace を指定します。今回の手順の例だと次の指定です。
  - `oci://dataflow-logs@nryjxkqe0mhq/optimized_listings`



パラメータを入力して、作成ボタンを押します。

![screencapture-console-ap-tokyo-1-oraclecloud-data-flow-apps-2020-04-19-00_13_56.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/933622bd-30a5-5c11-e96f-def614c0b18c.png)



アプリケーションが定義されます。

![1587222929166.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/d00603d3-0979-3e55-daa3-2c883fa63c2d.png)



詳細画面はこんなかんじです。

![1587222954755.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/6736368f-0aaf-f274-cbd6-8dca2e9845fe.png)





# アプリケーションの中身確認

チュートリアルで準備されているアプリケーションの中身を確認していきます。Java で出来ている JAR ファイルは、以下のURLで公開されています。

Jar ファイル : Object Storage として GET する URL
https://objectstorage.ap-tokyo-1.oraclecloud.com/n/bigdatadatasciencelarge/b/oow_2019_dataflow_lab/o/usercontent/oow-lab-2019-java-etl-1.0-SNAPSHOT.jar

ソースコードは逆コンパイルで確認しました。もしかしたらどこかに公開されているかもしれません。

```java

package convert;

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.types.StructType;

public class Convert {
  public static void convert(SparkSession spark, String input, String output) {
    StructType schema = (new StructType()).add("id", "long").add("listing_url", "string").add("scrape_id", "string").add("last_scraped", "string").add("name", "string").add("summary", "string").add("space", "string").add("description", "string").add("experiences_offered", "string").add("neighborhood_overview", "string").add("notes", "string").add("transit", "string").add("access", "string").add("interaction", "string").add("house_rules", "string").add("thumbnail_url", "string").add("medium_url", "string").add("picture_url", "string").add("xl_picture_url", "string").add("host_id", "string").add("host_url", "string").add("host_name", "string").add("host_since", "string").add("host_location", "string").add("host_about", "string").add("host_response_time", "string").add("host_response_rate", "string").add("host_acceptance_rate", "string").add("host_is_superhost", "string").add("host_thumbnail_url", "string").add("host_picture_url", "string").add("host_neighborhood", "string").add("host_listings_count", "string").add("host_total_listings_count", "string").add("host_verifications", "string").add("host_has_profile_pic", "string").add("host_identity_verified", "string").add("street", "string").add("neighborhood", "string").add("neighborhood_cleansed", "string").add("neighborhood_group_cleansed", "string").add("city", "string").add("state", "string").add("zipcode", "string").add("market", "string").add("smart_location", "string").add("country_code", "string").add("country", "string").add("latitude", "string").add("longitude", "string").add("is_location_exact", "string").add("property_type", "string").add("room_type", "string").add("accommostrings", "string").add("bathrooms", "string").add("bedrooms", "string").add("beds", "string").add("bed_type", "string").add("amenities", "string").add("square_feet", "float").add("price", "float").add("weekly_price", "float").add("monthly_price", "float").add("security_deposit", "float").add("cleaning_fee", "float").add("guests_included", "string").add("extra_people", "string").add("minimum_nights", "string").add("maximum_nights", "string").add("calendar_upstringd", "string").add("has_availability", "string").add("availability_30", "string").add("availability_60", "string").add("availability_90", "string").add("availability_365", "string").add("calendar_last_scraped", "string").add("number_of_reviews", "string").add("first_review", "string").add("last_review", "string").add("review_scores_rating", "int").add("review_scores_accuracy", "int").add("review_scores_cleanliness", "int").add("review_scores_checkin", "int").add("review_scores_communication", "int").add("review_scores_location", "int").add("review_scores_value", "int").add("requires_license", "string").add("license", "string").add("jurisdiction_names", "string").add("instant_bookable", "string").add("is_business_travel_ready", "string").add("cancellation_policy", "string").add("require_guest_profile_picture", "string").add("require_guest_phone_verification", "string").add("calculated_host_listings_count", "string").add("reviews_per_month", "string");
    Dataset<Row> df = spark.read().option("header", "true").option("multiLine", "true").option("escape", "\"").schema(schema).csv(input);
    df.toDF().write().parquet(output);
  }
  
  public static void main(String[] args) {
    SparkSession spark = SparkSession.builder().appName("oow-lab").getOrCreate();
    convert(spark, args[0], args[1]);
    System.out.println("Conversion was successful");
    spark.stop();
  }
}
```



main メソッドには、String[] の引数が定義されています

```java

public static void main(String[] args) {
```



この引数に、Data Flow Application で定義した引数に対応しています。

- args[0] : input の `oci://oow_2019_dataflow_lab@bigdatadatasciencelarge/usercontent/kaggle_berlin_airbnb_listings_summary.csv` が格納
- args[1] : outputの `oci://dataflow-logs@nryjxkqe0mhq/optimized_listings` が格納されています

![1587225276523.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/7af73cd6-7fad-273c-9c38-1d45315c031b.png)



この、`oci://...`という指定は、OCI の Object Storage へ接続するための指定方法です。おそらく、[HDFS Connector for Object Storage](https://docs.cloud.oracle.com/en-us/iaas/Content/API/SDKDocs/hdfsconnector.htm) が使われていると思います。Object Storage 上のオブジェクトをダウンロードしたり、保存したりするのを Spark 上のコードから扱いやすくするためのものです。



main メソッドから、次のように引数を指定して convert メソッドを読んでいます

```java

convert(spark, args[0], args[1]);
```



convert メソッドの中では、csv ファイルを Object Storage からダウンロードして、parquet形式に変換しています。

```java

  public static void convert(SparkSession spark, String input, String output) {
    StructType schema = (new StructType()).add("id", "long").add("listing_url", "string").add("scrape_id", "string").add("last_scraped", "string").add("name", "string").add("summary", "string").add("space", "string").add("description", "string").add("experiences_offered", "string").add("neighborhood_overview", "string").add("notes", "string").add("transit", "string").add("access", "string").add("interaction", "string").add("house_rules", "string").add("thumbnail_url", "string").add("medium_url", "string").add("picture_url", "string").add("xl_picture_url", "string").add("host_id", "string").add("host_url", "string").add("host_name", "string").add("host_since", "string").add("host_location", "string").add("host_about", "string").add("host_response_time", "string").add("host_response_rate", "string").add("host_acceptance_rate", "string").add("host_is_superhost", "string").add("host_thumbnail_url", "string").add("host_picture_url", "string").add("host_neighborhood", "string").add("host_listings_count", "string").add("host_total_listings_count", "string").add("host_verifications", "string").add("host_has_profile_pic", "string").add("host_identity_verified", "string").add("street", "string").add("neighborhood", "string").add("neighborhood_cleansed", "string").add("neighborhood_group_cleansed", "string").add("city", "string").add("state", "string").add("zipcode", "string").add("market", "string").add("smart_location", "string").add("country_code", "string").add("country", "string").add("latitude", "string").add("longitude", "string").add("is_location_exact", "string").add("property_type", "string").add("room_type", "string").add("accommostrings", "string").add("bathrooms", "string").add("bedrooms", "string").add("beds", "string").add("bed_type", "string").add("amenities", "string").add("square_feet", "float").add("price", "float").add("weekly_price", "float").add("monthly_price", "float").add("security_deposit", "float").add("cleaning_fee", "float").add("guests_included", "string").add("extra_people", "string").add("minimum_nights", "string").add("maximum_nights", "string").add("calendar_upstringd", "string").add("has_availability", "string").add("availability_30", "string").add("availability_60", "string").add("availability_90", "string").add("availability_365", "string").add("calendar_last_scraped", "string").add("number_of_reviews", "string").add("first_review", "string").add("last_review", "string").add("review_scores_rating", "int").add("review_scores_accuracy", "int").add("review_scores_cleanliness", "int").add("review_scores_checkin", "int").add("review_scores_communication", "int").add("review_scores_location", "int").add("review_scores_value", "int").add("requires_license", "string").add("license", "string").add("jurisdiction_names", "string").add("instant_bookable", "string").add("is_business_travel_ready", "string").add("cancellation_policy", "string").add("require_guest_profile_picture", "string").add("require_guest_phone_verification", "string").add("calculated_host_listings_count", "string").add("reviews_per_month", "string");
    Dataset<Row> df = spark.read().option("header", "true").option("multiLine", "true").option("escape", "\"").schema(schema).csv(input);
    df.toDF().write().parquet(output);
  }
```



input には、`oci://oow_2019_dataflow_lab@bigdatadatasciencelarge/usercontent/kaggle_berlin_airbnb_listings_summary.csv` という文字列が入っているので、Object Storage 上のオブジェクトを読み込んでいます。簡単に指定できて便利ですね。

```java

csv(input);
```



csvをparquet形式に変換して保存するのがここです。outputには `oci://dataflow-logs@nryjxkqe0mhq/optimized_listings` という文字列が入っています。これも簡単に指定できて便利ですね。

```java

df.toDF().write().parquet(output);
```



使われている csv ファイルの中身を確認しましょう。以下のURLで公開されています。これは、KaggleのWebサイトで公開されているものを使用しているようです。

csvファイル (67.9MB)
https://objectstorage.ap-tokyo-1.oraclecloud.com/n/bigdatadatasciencelarge/b/oow_2019_dataflow_lab/o/usercontent/kaggle_berlin_airbnb_listings_summary.csv



中身 (ファイルの先頭から4行分抜粋)
1行目には各列の名前が書かれていて、2行目以降に実際のデータが書かれています。

```
id,listing_url,scrape_id,last_scraped,name,summary,space,description,experiences_offered,neighborhood_overview,notes,transit,access,interaction,house_rules,thumbnail_url,medium_url,picture_url,xl_picture_url,host_id,host_url,host_name,host_since,host_location,host_about,host_response_time,host_response_rate,host_acceptance_rate,host_is_superhost,host_thumbnail_url,host_picture_url,host_neighbourhood,host_listings_count,host_total_listings_count,host_verifications,host_has_profile_pic,host_identity_verified,street,neighbourhood,neighbourhood_cleansed,neighbourhood_group_cleansed,city,state,zipcode,market,smart_location,country_code,country,latitude,longitude,is_location_exact,property_type,room_type,accommodates,bathrooms,bedrooms,beds,bed_type,amenities,square_feet,price,weekly_price,monthly_price,security_deposit,cleaning_fee,guests_included,extra_people,minimum_nights,maximum_nights,calendar_updated,has_availability,availability_30,availability_60,availability_90,availability_365,calendar_last_scraped,number_of_reviews,first_review,last_review,review_scores_rating,review_scores_accuracy,review_scores_cleanliness,review_scores_checkin,review_scores_communication,review_scores_location,review_scores_value,requires_license,license,jurisdiction_names,instant_bookable,is_business_travel_ready,cancellation_policy,require_guest_profile_picture,require_guest_phone_verification,calculated_host_listings_count,reviews_per_month
2015,https://www.airbnb.com/rooms/2015,20181107122246,2018-11-07,Berlin-Mitte Value! Quiet courtyard/very central,Great location!  30 of 75 sq meters. This wood floored/high ceiling typical Berlin 'Altbau' section of an apartment consists of 1 simple large room a small kitchen and a bathroom + shower. The apartment is in Mitte close to Prenzlauer Berg/Mauerpark. Perfect for short visits singles or couples. Your section is closed from the rest of the bigger flat wich is not noticeable. You will not be sharing your space.,A+++ location! This Einliegerwohnung' is an extention of a larger apartment with a separate entrance bathroom and kitchen. The door to the rest of the apartment is soundproof hidden locked and barely noticable (behind mirror in pictures). Your 30 sq meters are facing a quiet courtyard. This wood floored/high ceiling typical Berlin 'Altbau' apartment consists of 1 large room with a large double bed optionally with an extra matress for a 3rd guest) a small kitchen suitable for preparing simple meals a bathroom + shower + bathtub and of course your own entrance (all linens and towels are provided.) it is located facing a quiet courtyard it is clean cozy and an authentic 'Berlin' place to live. The place is in Mitte close to Prenzlauer Berg one block from Kastanienallee Zionskirchplatz Rosenthaler Platz and Mauerpark. Generally this area is lively with galleries museums resturants cafes flea markets bars clubs shops good take away food a park U-Bahn trams supermarket,Great location!  30 of 75 sq meters. This wood floored/high ceiling typical Berlin 'Altbau' section of an apartment consists of 1 simple large room a small kitchen and a bathroom + shower. The apartment is in Mitte close to Prenzlauer Berg/Mauerpark. Perfect for short visits singles or couples. Your section is closed from the rest of the bigger flat wich is not noticeable. You will not be sharing your space. A+++ location! This Einliegerwohnung' is an extention of a larger apartment with a separate entrance bathroom and kitchen. The door to the rest of the apartment is soundproof hidden locked and barely noticable (behind mirror in pictures). Your 30 sq meters are facing a quiet courtyard. This wood floored/high ceiling typical Berlin 'Altbau' apartment consists of 1 large room with a large double bed optionally with an extra matress for a 3rd guest) a small kitchen suitable for preparing simple meals a bathroom + shower + bathtub and of course your own entrance (all linens a,none,It is located in the former East Berlin area of Kastanienallee and Zionskirchplatz (Berlins only natural elevation) very charming and original turn of the last century architecture; this area was barely been destroyed during the war. Close to galleries resturants museums flea markets cafes bars clubs shops good take away food a park great public transportation: U-Bahn S-Bahn Trams Busses supermarkets etc.,This is my home not a hotel. I rent out occasionally when I stay with my friend.,Close to U-Bahn U8 and U2 (metro) Trams M12 M10 and M8 Busses. Rosenthaler Platz and Hackescher Markt are close by. S-Bahn. Alexander Platz is 6 minutes away on the U8 line.,Simple kitchen/cooking refrigerator microwave oven stove dishwasher espresso/ coffee mashine cable TV deskspace shower bath tub little courtyard garden 1st floor. There is a large double bed and optional additional airbeds for a 3rd or 4th person/kids. (Please note that the apartment is 30 square meters if you stay a few nights and are more than 2 adults it will be tight),Always available,No parties No events No pets No smoking not even in the courtyard/garden Do not pull the door shut with the keys in the lock from the inside; should that happen the guest pays for locksmith,,,https://a0.muscache.com/im/pictures/260fd609-710a-4304-82ab-abd7498efd62.jpg?aki_policy=large,,2217,https://www.airbnb.com/users/show/2217,Ian,2008-08-18,Key Biscayne Florida United States,Believe in sharing economy.,within an hour,96%,N/A,t,https://a0.muscache.com/im/pictures/21428a22-4f38-444d-8c72-5892be9da109.jpg?aki_policy=profile_small,https://a0.muscache.com/im/pictures/21428a22-4f38-444d-8c72-5892be9da109.jpg?aki_policy=profile_x_medium,Mitte,4,4,['email' 'phone' 'reviews' 'jumio' 'offline_government_id' 'government_id'],t,t,Berlin Berlin Germany,Mitte,Brunnenstr. Sud,Mitte,Berlin,Berlin,10119,Berlin,Berlin Germany,DE,Germany,52.53453732241747,13.402556926822387,f,Guesthouse,Entire home/apt,3,1.0,1,2,Real Bed,{TV'Cable TV'WifiKitchenGymHeating'Family/kid friendly''Smoke detector'EssentialsShampoo'Lock on bedroom door'Hangers'Hair dryer'Iron'Laptop friendly workspace''Private living room'Bathtub'Hot water''Bed linens''Extra pillows and blankets'Microwave'Coffee maker'RefrigeratorDishwasher'Dishes and silverware''Cooking basics'Stove'Luggage dropoff allowed''Long term stays allowed'},,60.00,,,200.00,30.00,1,28.00,4,1125,3 months ago,t,0,21,51,141,2018-11-07,118,2016-04-11,2018-10-28,93,10,9,10,10,10,9,t,,,f,f,strict_14_with_grace_period,f,f,4,3.76
2695,https://www.airbnb.com/rooms/2695,20181107122246,2018-11-07,Prenzlauer Berg close to Mauerpark,,In the summertime we are spending most of our weekends in our little house in our garden and want to use the opportunity to rent a room in our cosy 2 bedroom apartment. But we might also stay in the other room. We will be very happy to give you tips on things to do and see in Berlin and make sure your stay runs smoothly. The room is about 25 square meters and is located on the 2nd floor. The bed measures 180 cm x 200 cm and is fitted with a kapok futon which is a hard-ish mattress but which we find very pleasant to sleep on.  It's a typical Berlin-style apartment located in a friendly and calm neighborhood plus your room is facing away from the street and into the interior area (so generally very quiet).  All the cafes bars shops and other action of the Prenzlauerberg district is easily accessible with a pleasant 10-15 minute stroll across Mauerpark. The main S-Bahn and U-Bahn public transport connections are also just a 10-15 minute walk away. There are all sorts of options for ea,In the summertime we are spending most of our weekends in our little house in our garden and want to use the opportunity to rent a room in our cosy 2 bedroom apartment. But we might also stay in the other room. We will be very happy to give you tips on things to do and see in Berlin and make sure your stay runs smoothly. The room is about 25 square meters and is located on the 2nd floor. The bed measures 180 cm x 200 cm and is fitted with a kapok futon which is a hard-ish mattress but which we find very pleasant to sleep on.  It's a typical Berlin-style apartment located in a friendly and calm neighborhood plus your room is facing away from the street and into the interior area (so generally very quiet).  All the cafes bars shops and other action of the Prenzlauerberg district is easily accessible with a pleasant 10-15 minute stroll across Mauerpark. The main S-Bahn and U-Bahn public transport connections are also just a 10-15 minute walk away. There are all sorts of options for ea,none,,,Within walking distance you'll find the S-Bahn two metro lines  tram lines and busses.  But the best way to discover Berlin is by bike. Please ask if you need a guest bike.,Ausser deinem Zimmer kannst du noch die Kuche und das Badezimmer benutzen. Es gibt auch einen Balkon den man tagsuber benutzen kann. Der Zugang fuhrt allerdings durch unser anderes Zimmer wo wir dann abends eventuel schlafen.,,,,,https://a0.muscache.com/im/pictures/97d4f6e8-1c77-47fc-958e-26ba5618083c.jpg?aki_policy=large,,2986,https://www.airbnb.com/users/show/2986,Michael,2008-09-16,Berlin Berlin Germany,Living with my wife in Berlin.  Like travelling hiking canoing beach volleyball yoga cinema....,N/A,N/A,N/A,f,https://a0.muscache.com/im/pictures/50434955-1888-4049-bbff-a78464dbfa5a.jpg?aki_policy=profile_small,https://a0.muscache.com/im/pictures/50434955-1888-4049-bbff-a78464dbfa5a.jpg?aki_policy=profile_x_medium,Prenzlauer Berg,1,1,['email' 'phone' 'reviews' 'jumio' 'government_id'],t,t,Berlin Berlin Germany,,Prenzlauer Berg Nordwest,Pankow,Berlin,Berlin,10437,Berlin,Berlin Germany,DE,Germany,52.54851279221664,13.404552826587466,t,Apartment,Private room,2,1.0,1,1,Real Bed,{WifiKitchenElevatorHeatingWasherEssentialsShampoo'Hair dryer''Laptop friendly workspace''Private living room'},,17.00,,,0.00,0.00,1,0.00,2,40,7 weeks ago,t,0,0,0,0,2018-11-07,6,2018-07-04,2018-10-01,100,10,10,10,10,10,10,t,,,f,f,flexible,f,f,1,1.42
3176,https://www.airbnb.com/rooms/3176,20181107122246,2018-11-07,Fabulous Flat in great Location,This beautiful first floor apartment  is situated at  Kollwitzplatz. Its ideal for 2 but can comfortably accommodate 4 with the extra double bed in the living room. PLEASE INQUIRE FIRST BEFORE MAKING A BOOKING. THANKS,1st floor (68m2) apartment on Kollwitzplatz/ Prenzlauer Berg. It's ideal for 2 but can sleep comfortably an extra 2 in the living room. (Please let us know in advance if you are 2 people sleeping separately that we can bring extra sheet for you.) Kids are welcome and for smaller visitors we can arrange  baby cot and high chair. The front room is spacious and open plain with a large dining table parquet flooring and a simple yet stylish decor. You can relax in an Indian easy chair or simply lounge on the daybed. The kitchen area has everything from frying pan to coffee maker an electric oven and stove refrigerator. To the back of the apartment is the bedroom with a large wardrobe and a comfy brand new double bed. (Decemb(PHONE NUMBER HIDDEN) cm wide) The bathroom has a bathtub/shower and a washing machine. Ironing Board/ Iron/ Towels and linen provided.  There is a sound system with a CD player record player a cable connection for an mp3 player as well as a wifi internet connectio,This beautiful first floor apartment  is situated at  Kollwitzplatz. Its ideal for 2 but can comfortably accommodate 4 with the extra double bed in the living room. PLEASE INQUIRE FIRST BEFORE MAKING A BOOKING. THANKS 1st floor (68m2) apartment on Kollwitzplatz/ Prenzlauer Berg. It's ideal for 2 but can sleep comfortably an extra 2 in the living room. (Please let us know in advance if you are 2 people sleeping separately that we can bring extra sheet for you.) Kids are welcome and for smaller visitors we can arrange  baby cot and high chair. The front room is spacious and open plain with a large dining table parquet flooring and a simple yet stylish decor. You can relax in an Indian easy chair or simply lounge on the daybed. The kitchen area has everything from frying pan to coffee maker an electric oven and stove refrigerator. To the back of the apartment is the bedroom with a large wardrobe and a comfy brand new double bed. (Decemb(PHONE NUMBER HIDDEN) cm wide) The bathroom has a,none,The neighbourhood is famous for its variety of international eateries pubs restaurants cafes galleries and little shops. The bakery next door is open everyday from up 7 am which makes warm croissants for breakfast very tempting. On Sundays don't miss the traditional flea markets on Arkonaplatz and in the nearby Mauerpark for special treats and souvenirs.  check out our  travel guide and please let us know what your highlights were so we can add them for future guests,We welcome FAMILIES and cater especially for young kids. You can make use from our highchair baby cot and baby bouncer. We also have some trusted BABYSITTERS at hand so you can enjoy an evening out in Berlin. Please inquire asap.   Check-in and out are flexible depending on your schedule. We guarantee check in from up 14:00 and check out before 11:00. A Late night check in fee will apply after 9pm.  The 5 % Berlin Tourist tax is included in the total price. We do require all guests to sign a short- rental agreement upon arrival.   THE MENTIONED BUILDING WORKS IN ONE OF THE REVIEWS HAVE BEEN CONCLUDED.,We are 5 min walk  away from the tram M2 which will take you to Alexander Platz- here you can connect to buses trains and trams that take you anywhere in the city and to both airports as well as the central train station.  You can also catch the underground U2 directly at Senefelderplatz which is approximately 7 min walk away.,The apartment will be entirely yours. We are conveniently located on the first floor with an extra wide doorway. A member of the management company we work with will meet you at the apartment.,Feel free to ask any questions prior to booking and during your stay. Once you have booked you will receive via airbnb all relevant information about the check- in and the apartment. PLEASE FOLLOW US ON (website hidden) 'a home away from home in berlin' for upcoming activities in Berlin such as concerts and exihibitions,It's a non smoking flat which likes to be treated with love and respect ;) we don't tolerate parties or pets. Family and kids are more than welcome. We don't rent the apartment out as a film or photo location either.,,,https://a0.muscache.com/im/pictures/243355/84afcff6_original.jpg?aki_policy=large,,3718,https://www.airbnb.com/users/show/3718,Britta,2008-10-19,Coledale New South Wales Australia,"We love to travel ourselves a lot and prefer to stay in apartments. Especially since we had a baby .
```





# アプリケーションの実行

定義したアプリケーションを実行します。詳細画面にある実行ボタンを押します

![1587223128551.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/92575db6-77f9-26ea-d246-53a9281c40f6.png)



定義した内容がそのまま表示されています。データが多い時にはエグゼキュータの数を増やしたり、いろいろ調整が出来ます。今回はこのまま実行をおします

![1587223187328.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/da31063d-aa00-db63-2195-e8f4bbdf0083.png)



画面が切り替わり、実行の一覧画面で、受入済みのステータスのものがみえます。

![1587223229018.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/69e49195-c6af-75f2-a356-6e5425f433a5.png)



自分の環境では、実行開始から実行完了まで10分ほどの時間がかかりました。VMの準備などのオーバーヘッド時間などがあると思うので、即時実行とはならないのでしょう。

- 実行開始 : 19:46
- 受入済み : 19:46~19:53
- 進行中 : 19:53~19:56



実行完了になったあとに、詳細画面を開きます。画面下にログが gz 形式で出力されています。Application, Driver, Executor のそれぞれの stdout(標準出力), stderr(標準エラー出力) が出力されているようです。

![1587208287878.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/e6bf2386-c6cd-edcc-98b9-4819400ba9b3.png)



Application のstdout(標準出力)を確認してみます

![1587226408105.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/33c6e44d-8f2b-d8e6-5a18-f57d37a11c51.png)



以下の画面が表示されます。

![1587226427915.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/506ae7e4-c5f9-b724-68d9-25f4127f0da4.png)



これがどこから出ているか確認すると、main メソッドの、`System.out.println("Conversion was successful");` の部分で出力されているものが、File として見えます。

```java

  public static void main(String[] args) {
    SparkSession spark = SparkSession.builder().appName("oow-lab").getOrCreate();
    convert(spark, args[0], args[1]);
    System.out.println("Conversion was successful");
    spark.stop();
  }
```



# Spark UI

実行の詳細画面で、Spark UI というボタンがあります。Spark Application の実行の結果を確認、トラブルシューティングが出来るものです。実際に開いてみましょう

![1587207359774.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/290c0b10-ee11-241b-93fe-21f10163b38b.png)



それぞれの画面を載せていきます。Java のコードがどれくらいの時間実行されているのか、といった情報が確認できます。

Jobs

![1587207413666.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/b6cc979a-ac8c-26ae-150f-2dad4017a84a.png)



Stage

![1587207521097.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/43faa236-58c8-971f-2de7-550c39125b03.png)



Storage

何も表示されない

![1587207544264.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/b99d2390-cbe7-9402-e419-bb57507ff54f.png)



Environment

Spark 環境の詳細が表示されています。いくつか気になった部分を次に抜粋します。

- Java Version : Java 8 (1.8.0_162)
- HDFS Connector for Object Storage を使ってるようなパラメータが見える (`spark.hadoop.fs.oci.client.XXX` など)

![screencapture-dataflow-ap-tokyo-1-oci-oraclecloud-sparkui-ocid1-dataflowrun-oc1-ap-tokyo-1-anxhiljrycetm7yau2jwnieub6uhwhpnvdw5pxi6amk2siyy4g5qtr4f4d5a-environment-2020-04-18-19_59_09.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/72523786-cecb-d787-ae21-69a49e496403.png)



Executors

Executor が動作した時間、実行完了したタスクの数、インプット容量、CPUコアのデータなどが表示されています。

![1587208142752.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/976eefda-4d7a-76aa-e1fe-b80c2b738428.png)



SQL

![1587208158456.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/166038bf-9b64-c4f8-4871-8032f94405a1.png)



# 結果を確認

csv ファイルを parquet形式へ変換した結果が、Object Storage のバケットに出力されています。それ以外にも色々ファイルが出力されていて、Data Flow Application のログなどが Object として出力されています。

![1587227126711.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/6efed020-8796-7a71-dca2-49178a252aee.png)

# Data Flow のコスト分析

Data Flow でアプリケーションをいくつか実行したあとに、コスト分析のページを表示しました。
コスト分析ページには、Block Storage と Compute と Object Storage が表示されています。Data Flow という名前では集計されず、各々のインフラサービスのコストとして課金されるようです。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/104247/440f7b97-8616-8bda-306c-c9f8481c0f4f.png)



# 参考URL

https://docs.cloud.oracle.com/en-us/iaas/data-flow/data-flow-tutorial/tutorial/dfs_tut_front.htm