第127回スクエアFreeセミナー, Feb 24 2022 18:00-18:30

明日から使える「Apache Hop」
データ統合プラットフォームを使えばデータ統合が簡単で楽しくなります。

Koji Annoura
案浦浩二
koji@annoura.com

Source: Apache Hop, https://hop.apache.org

Who am I❖ グラフ理論LOVE❖ Neo4jグラフデータベースにハマってます❖ Graph Community MVP❖ Neo4j Ambassador❖ Neo4j Speaker❖ Neo4j Ninja❖ Neo4j Developer❖ User Group❖ Neo4jユーザーグループ❖ 日本Apache Hop ユーザーグループ

Apache® Hop™ as a Top-Level Project (TLP)❖ Apache Hop❖ 2021-12-17 TLP

Source: Apache, https://www.apache.org/index.html#projects-list

Apache Hop 1.1❖ Apache Hop 1.1.0❖ Hop Server❖ SSL(https)対応で起動❖ 非同期型Webサービス❖ New❖ Apache Tika
❖ Metadata Input❖ Neo4j Graph Output transform (Add)❖ multiple relationships❖ multiple node labels❖ self-relationships❖ Hop GUI❖ SCROLL-drag)❖ Log4j removal

Source: Apache Hop, https://twitter.com/ApacheHop

Goal❖ 今日のゴールは❖ Apache Hopを知ってもらう❖ ちょっと使ってみようと思ってもらう❖ ユーザーグループへのお誘い

Copyright © いらすとやどのようにデータを集めていますか?
❖ COVID-19❖ covid19japan❖ https://covid19japan.com❖ Problem❖ 厚労省のサイトにまとまったデータがない❖ 都道府県・市町村❖ オープンデータ、Web❖ フォーマットが異なる❖ ヘッダーの名前が微妙に異なる❖ 発表の間違い❖ 患者の番号をリナンバーするところがあった❖ オープンデータやWebの間違いはフィードバック

Source: covid-19 Japan, https://covid19japan.com

ETL
❖ ETL
❖ Extract❖ Transform❖ Load❖ Data Transformer❖ データサイエンティスには必需品

Source: Apache Hop hhttps://hop.apache.org

Apache Hop?
❖ Hop?
❖ データオーケストレーション、データエンジニア

リングのプラットフォーム❖ オープンソース❖ Apache v2 license❖ Kettle 8.2.0.7からFork❖ Pentaho Data Integration ( ETL ) a.k.aKettle❖ 敵対的なフォークではない❖ weʼve invited Hitachi Vantara to contributeto Apache Hop
Source: github, https://github.com/pentaho/pentaho-kettle

Apache Hop❖ Apache Hop❖ The Hop Orchestration Platform❖ 2020年12月にApache のTLPになった❖ 日本Apache Hopユーザーグループ❖ Apache Hop User Group Japan

Source: Apache Hop hhttps://hop.apache.org

Apache Hop Incubating User Group Japan❖ Meetup❖ https://www.meetup.com/apache-hop-user-group-japan/❖ Committers (18 people)❖ Hiromu Hota
❖ Matt Casters (Neo4j)❖ Bart Maertens❖ Facebook❖ https://www.facebook.com/groups/

apachehopusergroupjapan
Source: Meetup https://www.meetup.com/apache-hop-user-group-japan/

Apache Hop - YouTube❖ Apache Hop❖ https://www.youtube.com/c/

ApacheHop❖ チャンネル登録をお願いします

Source: YouTube, https://www.youtube.com/c/ApacheHop

Past Events❖ Recording❖ https://www.youtube.com/c/

ApacheHop❖ Language❖ English❖ Spanish, Español❖ 日本語

Source: Apache Hop, https://hop.apache.org/community/events/

User Groups❖ User Groups❖ 3Hx - Apache Hop Hot Hop Hangouts❖ Grupo de usuarios de Apache Hop en

Español❖ Grupo de usuários do Apache Hop em

Português
❖ Apache Hop em Português noTelegram❖ 日本Apache Hop ユーザーグループ(Apache Hop User Group Japan)
Source: Apache Hop, https://hop.apache.org/community/events/

Java❖ Java❖ Java 8で開発・テストされています❖ Java 11正式対応は、2.0.0の予定

Source: docker hub https://hub.docker.com/r/apache/incubator-hop

Download❖ Version❖ 2.0.0 SNAPSHOT❖ 1.2.0 SNAPSHOT❖ 1.1.0 (2022-01-24)❖ 1.0.0 (2021-10-05)❖ 0.99 (2021-08-05)❖ docker hub❖ Hop❖ https://hub.docker.com/r/apache/hop❖ Hop Web❖ https://hub.docker.com/r/apache/hop-web
Source: Apache Hop, https://hop.apache.org/download/Apache Hop❖ Metadata Driven

Source: Apache Hop, https://hop.apache.org

Apache Hop❖ Design Once, Run Anywhere❖ Native Hop runtime❖ Apache Spark❖ Apache Flink❖ Google Dataow through theApache Beam runtimes.

flSource: Apache Hop, https://hop.apache.org

Apache Hop❖ Visual Development❖ intuitive drag and drop interface❖ workows❖ pipelines

flSource: Apache Hop, https://hop.apache.org

Apache Hop❖ Built-in Life Cycle Management❖ Hop Gui
❖ projects❖ environments❖ runtime congurations❖ manage git versions❖ etc.

fiSource: Apache Hop, https://hop.apache.org

Hop Tools❖ URL❖ https://hop.apache.org/manual/latest/getting-started/hop-tools.html❖ Tools❖ Hop Gui
❖ Hop Conf
❖ Hop Encrypt❖ Hop Run
❖ Hop Search❖ Hop Server❖ Hop Translator (日本語も)Source: Apache Hop, https://hop.apache.org/manual/latest/getting-started/hop-tools.html

Hop vs. Kettle (Concepts)
FeatureKettle

Hop

Transformation
Step

Pipeline
Transform

Job

Work ow

An action in a work ow

Job EntryAction

Shared metadata container

Metastore

Hop Metadata

The graphical user interfaceSpoon

Hop Gui

Script to run data pipelines

PanHop RunScript to run work ows

Kitchen

Hop RunServer for remote execution

Carte

Hop Server-Hop Con gParallel scalable data pipeline
An operation in a parallel pipeline
Sequential series of actions

Script for con guration

flflfififl

Source: Apache Hop, https://hop.apache.org/tech-manual/latest/hop-vs-kettle/hop-vs-kettle.html

Hop vs. Kettle (Engines)
EngineKettle

HopPlugin

Yes

Apache Spark Support

No (PDI EE)

Yes (Beam)

Apache Flink Support

No

Yes (Beam)

Google Cloud DataFlow Support

NoYes (Beam)

Unit Testing

Source: Apache Hop, https://hop.apache.org/tech-manual/latest/hop-vs-kettle/hop-vs-kettle.html

Hop vs. Kettle (Features)
FeatureProjects and Lifecycle Con guration
Search Information in projects and con gurations
Con guration management through UI andStandardized shared metadata
Pluggable runtime engines
Advanced GUI features: memory, native zooom, …
Metadata Injection
Mapping (sub-transformation/pipeline
Web Interface
APL 2.0 license compliance
Pluggable metadata objects
GUI plugin architecture

Kettle
NoNo
NoNo
NoNo
Yes
Yes
WebSpoon
LGPLNo
XUL based

HopYes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Hop Web
Yes
YesJavafififi

Source: Apache Hop, https://hop.apache.org/tech-manual/latest/hop-vs-kettle/hop-vs-kettle.html

Hop Server❖ Hop Server❖ lightweight server❖ http://localhost:8081❖ run workows and pipelines❖ REST services

flSource: Apache Hop, https://hop.apache.org/manual/latest/hop-server/index.html

Pipeline❖ Pipeline❖ 1つまたは複数のソースからデータを読み込む❖ いくつかの操作（結合、ルックアップ、フィルタなど）を実行する❖ 最終的に1つまたは複数のターゲットプラットフォームにデータを書き込む

Source: Apache Hop, http://hop.apache.org/manual/latest/pipeline/create-pipeline.html

Work ows
❖ Workows❖ Workowsは、Apache Hopの中核となる構成要素の一つ❖ Pipelineが重いデータ処理を行うのに対し、Work owsは環境の準備、リモートファイルの取得、エラー処理、childwork owsやPipelineの実行などのオーケストレーション作業を行う。

flfl

fl

fl

fl

fl

fl

Source: Apache Hop, http://hop.apache.org/manual/latest/work ow/create-work ow.html

Work ows (Action Plugins)❖ Action Pluginsの一部❖ Check Db connections❖ Get ale with SFTP

❖ Get Mail from POP❖ Mail Validator❖ MS SQL Server Bulk Loader❖ MySQL Bulk File
❖ MySQL Bulk Loader❖ Neo4j Check Connections❖ Neo4j Constraint❖ Neo4j Cypher Script❖ Neo4j Index❖ Pingflflflfi

Source: Apache Hop, http://hop.apache.org/manual/latest/work ow/create-work ow.html

Relational Database Connections❖ Driver❖ JDBC, Native❖ RDB❖ Apache Derby, Apache Doris, AS400, Cache,

ClickHouse, DB2, Exasol, rebird, Google
BigQuery, Greenplum, Gupta SQLBase, H2,
Hypersonic, IBM Universe, In niDB, InfobrightDB, informix, ingres, interbase, Kingbase ES,Mariadb, MaxDB (SAP DB), monetdb, MS
SqlServer, MS SqlServer (Native), MySql,Netezza, Oracle, Oracle RDB, PostgreSQL,Redshift, Snow ake, SqLite, Sybase, Sybase IQ,Teradata, Vectorwise, Vertica

fififlSource: Apache Hop, https://hop.apache.org/manual/latest/database/databases.html

Transform❖ Transform Plugins❖ 200近くの変換が可能❖ List❖ Abort❖ Add a checksum❖ Add Constants❖ Add Sequence❖ （中略）❖ XML Join
❖ XML Output❖ XSD Validator❖ XSL Transformation❖ Yaml Input❖ Ziplefi

Source: Apache Hop, https://hop.apache.org/manual/latest/pipeline/create-pipeline.html

External Plugins❖ TRANSFORMS❖ Dropbox Input❖ Dropbox Output❖ Excel Output❖ Google Analytics❖ Google Sheets Input❖ Google Sheets Output❖ LDIF (Lightweight Directory Interchange Format), LDAP❖ LDIF Input❖ MQTT (Message Queue Telemetry Transport)❖ MQTT Input❖ MQTT Output❖ Send message to syslog

Source: GitHub, https://github.com/project-hop/hop-plugins/ありがとうございました❖

ご質問があれば、koji@annoura.com まで