Mac上にEmbulkを試しに触ってみるための環境を構築する手順

* WEB+DB PRESS Vol.94の連載「Emerging Web Technology研究室」[第20回] 『OSSによるデータ分析基盤の構築 - Embulk,Re:dush,Digdag』 に刺激されて、Embulkを試してみる。



## Embulkインストール

* インストール実行

``` console
$ brew install embulk
```

* 確認

```
$ which embulk
/usr/local/bin/embulk

$ brew info embulk
embulk: stable 0.8.9
Data transfer between various databases, file formats and services
http://www.embulk.org/
/usr/local/Cellar/embulk/0.8.9 (3 files, 39.0M) *
  Built from source on 2016-08-25 at 16:53:43
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/embulk.rb
```

### サーバへインストールする場合

* jarの入手 + 実行権限付与

```
curl --create-dirs -o ~/.embulk/bin/embulk -L "http://dl.embulk.org/embulk-latest.jar"
chmod +x ~/.embulk/bin/embulk
```

* パスを通す

```
echo 'export PATH="$HOME/.embulk/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc
```


### プラグインのインストール方法

* 自分の環境や業務的に必要になりそうなプラグインを入れておく

```bash:terminal
embulk gem install embulk-filter-column
embulk gem install embulk-input-jdbc
embulk gem install embulk-input-mysql
embulk gem install embulk-output-mysql
```


## 使ってみる

### チュートリアルを実行してみる。

* サンプルデータ用意

```
embulk example tutorial && tree

.
└── tutorial
    ├── csv
    │   └── sample_01.csv.gz
    └── seed.yml
```

* サンプルデータを読み込んで、画面に出力するconfig.ymlを生成
	* 「embulk guess」はきっと、引数に渡したseed.ymlを元に、設定ファイルを生成してくれる機能なんだと思う。

```
embulk guess tutorial/seed.yml -o config.yml

2016-08-25 17:05:28.111 +0900: Embulk v0.8.9
2016-08-25 17:05:29.987 +0900 [INFO] (0001:guess): Listing local files at directory '/Users/Shared/Workspace.d/Embulk/try1/csv' filtering filename by prefix 'sample_'
2016-08-25 17:05:30.003 +0900 [INFO] (0001:guess): Loading files [/Users/Shared/Workspace.d/Embulk/try1/csv/sample_01.csv.gz]
2016-08-25 17:05:30.087 +0900 [INFO] (0001:guess): Loaded plugin embulk/guess/gzip from a load path
2016-08-25 17:05:30.103 +0900 [INFO] (0001:guess): Loaded plugin embulk/guess/bzip2 from a load path
2016-08-25 17:05:30.124 +0900 [INFO] (0001:guess): Loaded plugin embulk/guess/json from a load path
2016-08-25 17:05:30.128 +0900 [INFO] (0001:guess): Loaded plugin embulk/guess/csv from a load path
in:
  type: file
  path_prefix: /Users/Shared/Workspace.d/Embulk/try1/csv/sample_
  decoders:
  - {type: gzip}
  parser:
    charset: UTF-8
    newline: CRLF
    type: csv
    delimiter: ','
    quote: '"'
    escape: '"'
    null_string: 'NULL'
    trim_if_not_quoted: false
    skip_header_lines: 1
    allow_extra_columns: false
    allow_optional_columns: false
    columns:
    - {name: id, type: long}
    - {name: account, type: long}
    - {name: time, type: timestamp, format: '%Y-%m-%d %H:%M:%S'}
    - {name: purchase, type: timestamp, format: '%Y%m%d'}
    - {name: comment, type: string}
out: {type: stdout}
Created 'config.yml' file.
```

* プレビュー

```
embulk preview config.yml

2016-08-25 17:06:31.507 +0900: Embulk v0.8.9
2016-08-25 17:06:33.506 +0900 [INFO] (0001:preview): Listing local files at directory '/Users/Shared/Workspace.d/Embulk/try1/csv' filtering filename by prefix 'sample_'
2016-08-25 17:06:33.516 +0900 [INFO] (0001:preview): Loading files [/Users/Shared/Workspace.d/Embulk/try1/csv/sample_01.csv.gz]
+---------+--------------+-------------------------+-------------------------+----------------------------+
| id:long | account:long |          time:timestamp |      purchase:timestamp |             comment:string |
+---------+--------------+-------------------------+-------------------------+----------------------------+
|       1 |       32,864 | 2015-01-27 19:23:49 UTC | 2015-01-27 00:00:00 UTC |                     embulk |
|       2 |       14,824 | 2015-01-27 19:01:23 UTC | 2015-01-27 00:00:00 UTC |               embulk jruby |
|       3 |       27,559 | 2015-01-28 02:20:02 UTC | 2015-01-28 00:00:00 UTC | Embulk "csv" parser plugin |
|       4 |       11,270 | 2015-01-29 11:54:36 UTC | 2015-01-29 00:00:00 UTC |                            |
+---------+--------------+-------------------------+-------------------------+----------------------------+

```


* 実行

```
$ embulk run config.yml
```
* できた。