Tweet + elasticsearch + kibana

# 環境構築

+ [環境構築手順](http://qiita.com/takaheraw@github/items/a5a36916ed2e87d54d1a)

# データを準備

## tweet取得

+ https://github.com/Tatzyr/alltweets
+ 他人のtweetは、3000件まで取れるみたい
+ 話題の[becky_bekiko](https://twitter.com/becky_bekiko)を取得してみる

```
$ git clone https://github.com/Tatzyr/alltweets.git
$ cd alltweets
$ ./bin/setup
$ ./exe/alltweets becky_bekiko --retweets --json
$ ls alltweets_becky_bekiko.json
alltweets_becky_bekiko.json
```

## データ整形

整形せずにそのまま突っ込みたかったが、geo_pointが配列を受け入れてくれなかった。[ドキュメント](https://www.elastic.co/guide/en/elasticsearch/reference/2.x/geo-point.html#geo-point)には```"location": [ -71.34, 41.12 ]```はOKそうなのだが。
どうせ整形するなら扱いやすいデータ構造にjqコマンドで整形した。

```
$ cat alltweets_becky_bekiko.json | jq ' .[] |
{
  id:                     .id_str,
  created_at:             .created_at,
  text:                   .text,
  analyze_text:           .text,
  source:                 .source,
  in_reply: {
    status:               .in_reply_to_status_id_str,
    user_id:              .in_reply_to_user_id_str,
    screen_name:          .in_reply_to_screen_name
  },
  user: {
    id:                   .user.id_str,
    name:                 .user.name,
    screen_name:          .user.screen_name,
    location:             .user.location,
    description:          .user.description,
    url:                  .user.url,
    followers_count:      .user.followers_count,
    friends_count:        .user.friends_count,
    listed_count:         .user.listed_count,
    created_at:           .user.created_at,
    favourites_count:     .user.favourites_count,
    statuses_count:       .user.statuses_count,
    lang:                 .user.lang,
    profile_image_url:    .user.profile_image_url_https,
    profile_banner_url:   .user.profile_banner_url
  },
  geo: {
    lat:                  (.geo.coordinates[0] // 0),
    lon:                  (.geo.coordinates[1] // 0)
  },
  place: {
    id:                   .place.id,
    url:                  .place.url,
    place_type:           .place.place_type,
    name:                 .place.name,
    full_name:            .place.full_name,
    country_code:         .place.country_code,
    country:              .place.country
  },
  retweet_count:          .retweet_count,
  favorite_count:         .favorite_count,
  lang:                   .lang,
  hashtags:               .entities.hashtags,
  user_mentions:          .entities.user_mentions,
  media:                  .entities.media,
  retweeted_status: {
    created_at:           .retweeted_status.created_at,
    id:                   .retweeted_status.id_str,
    text:                 .retweeted_status.text,
    source:               .retweeted_status.source,
    in_reply: {
      status:             .retweeted_status.in_reply_to_status_id_str,
      user_id:            .retweeted_status.in_reply_to_user_id_str,
      screen_name:        .retweeted_status.in_reply_to_screen_name
    },
    user: {
      id:                 .retweeted_status.user.id_str,
      name:               .retweeted_status.user.name,
      screen_name:        .retweeted_status.user.screen_name,
      location:           .retweeted_status.user.location,
      description:        .retweeted_status.user.description,
      url:                .retweeted_status.user.url,
      followers_count:    .retweeted_status.user.followers_count,
      friends_count:      .retweeted_status.user.friends_count,
      listed_count:       .retweeted_status.user.listed_count,
      created_at:         .retweeted_status.user.created_at,
      favourites_count:   .retweeted_status.user.favourites_count,
      statuses_count:     .retweeted_status.user.statuses_count,
      lang:               .retweeted_status.user.lang,
      profile_image_url:  .retweeted_status.user.profile_image_url_https,
      profile_banner_url: .retweeted_status.user.profile_banner_url
    }
  }
}
' > becky_bekiko.json 
```

## bulkに整形

+ [bulk用のフォーマット](https://www.elastic.co/guide/en/elasticsearch/reference/2.x/docs-bulk.html#docs-bulk)に整形する。
+ 雑な整形プログラム

```rb
require 'json'

File.open('./becky_bekiko.json', 'w') do |f|
  open('./bulk_becky_bekiko.json').each do |line|
    json=JSON.load(line)
    index = { "index" => { "_index" => "twitter", "_type" => "tweet", "_id" => json["id"]} }
    f.puts index.to_json
    f.puts line
  end
end
```
```
$ ruby to_bulk.rb
$ ls bulk_becky_bekiko.json
bulk_becky_bekiko.json
```

+ [stream2es](http://blog.johtani.info/blog/2014/04/24/usage-stream2es/)を使えば整形せずにいけるとあったが、_idのmappingsで```"reason": "Failed to parse mapping [tweet]: _id is not configurable"```と怒られたのであきらめた。

# IndexとMappingの作成

+ [リファレンス](https://www.elastic.co/guide/en/elasticsearch/reference/2.x/mapping.html)、[このサイト](https://medium.com/hello-elasticsearch/elasticsearch-kibana-6ad80550939f)を参考にした

```
$ curl -XPOST 'http://192.168.33.10:9200/twitter' -d '
{
  "settings": {
    "analysis": {
      "analyzer": {
        "ngram_analyzer": {
          "type": "custom",
          "char_filter": ["html_strip"],
          "tokenizer": "ngram_tokenizer",
          "filter": ["cjk_width", "lowercase"]
        },
        "facet_analyzer": {
          "type": "custom",
          "char_filter": ["html_strip"],
          "tokenizer": "keyword",
          "filter": ["cjk_width", "lowercase"]
        }
      },
      "tokenizer": {
        "ngram_tokenizer": {
          "type": "nGram",
          "min_gram": 2,
          "max_gram": 3,
          "token_chars": ["letter", "digit"]
        }
      }
    }
  },
  "mappings": {
    "tweet": {
      "properties": {
        "created_at": {
          "type": "date",
          "format" : "EE MMM d HH:mm:ss Z yyyy"
        },
        "text": {
          "type": "string",
          "index": "analyzed",
          "analyzer": "ngram_analyzer"
        },
        "analyze_text": {
          "type": "string",
          "index": "not_analyzed"
        },
        "source": {
          "type": "string",
          "index": "analyzed",
          "analyzer": "facet_analyzer"
        },
        "in_reply": {
          "type": "object",
          "properties": {
            "status": {
              "type": "long"
            },
            "user_id": {
              "type": "long"
            },
            "screen_name": {
              "type": "string",
              "index": "not_analyzed"
            }
          }
        },
        "user": {
          "type": "object",
          "properties": {
            "id": {
              "type": "long"
            },
            "name": {
              "type": "string",
              "index": "not_analyzed"
            },
            "screen_name": {
              "type": "string",
              "index": "not_analyzed"
            },
            "location": {
              "type": "string"
            },
            "description": {
              "type": "string",
              "index": "analyzed",
              "analyzer": "ngram_analyzer"
            },
            "url": {
              "type": "string"
            },
            "followers_count": {
              "type": "long"
            },
            "friends_count": {
              "type": "long"
            },
            "listed_count": {
              "type": "long"
            },
            "created_at": {
              "type": "date",
              "format" : "EE MMM d HH:mm:ss Z yyyy"
            },
            "favourites_count": {
              "type": "long"
            },
            "statuses_count": {
              "type": "long"
            },
            "lang": {
              "type": "string"
            },
            "profile_image_url": {
              "type": "string"
            },
            "profile_banner_url": {
              "type": "string"
            }
          }
        },
        "geo": {
          "type": "geo_point",
          "ignore_malformed": true
        },
        "place": {
          "type": "object",
          "properties": {
            "id": {
              "type": "string"
            },
            "url": {
              "type": "string"
            },
            "place_type": {
              "type": "string"
            },
            "name": {
              "type": "string"
            },
            "full_name": {
              "type": "string",
              "index": "analyzed",
              "analyzer": "ngram_analyzer"
            },
            "country_code": {
              "type": "string"
            },
            "country": {
              "type": "string"
            }
          }
        },
        "retweet_count": {
          "type": "long"
        },
        "favorite_count": {
          "type": "long"
        },
        "lang": {
          "type": "string"
        },
        "hashtags": {
          "properties": {
            "text": {
              "type": "string",
              "index": "analyzed",
              "analyzer": "facet_analyzer"
            }
          }
        },
        "user_mentions": {
          "properties": {
            "screen_name": {
              "type": "string",
              "index": "not_analyzed"
            },
            "name": {
              "type": "string",
              "index": "not_analyzed"
            },
            "id": {
              "type": "long"
            }
          }
        },
        "media": {
          "properties": {
            "id": {
              "type": "long"
            },
            "media_url_https": {
              "type": "string"
            },
            "expanded_url": {
              "type": "string"
            },
            "type": {
              "type": "string"
            }
          }
        },
        "retweeted_status": {
          "type": "object",
          "properties": {
            "created_at": {
              "type": "date",
              "format" : "EE MMM d HH:mm:ss Z yyyy"
            },
            "id": {
              "type": "long"
            },
            "text": {
              "type": "string",
              "index": "analyzed",
              "analyzer": "ngram_analyzer"
            },
            "source": {
              "type": "string",
              "index": "analyzed",
              "analyzer": "facet_analyzer"
            },
            "in_reply": {
              "type": "object",
              "properties": {
                "status": {
                  "type": "long"
                },
                "user_id": {
                  "type": "long"
                },
                "screen_name": {
                  "type": "string",
                  "index": "not_analyzed"
                }
              }
            },
            "user": {
              "type": "object",
              "properties": {
                "id": {
                  "type": "long"
                },
                "name": {
                  "type": "string",
                  "index": "not_analyzed"
                },
                "screen_name": {
                  "type": "string",
                  "index": "not_analyzed"
                },
                "location": {
                  "type": "string"
                },
                "description": {
                  "type": "string",
                  "index": "analyzed",
                  "analyzer": "ngram_analyzer"
                },
                "url": {
                  "type": "string"
                },
                "followers_count": {
                  "type": "long"
                },
                "friends_count": {
                  "type": "long"
                },
                "listed_count": {
                  "type": "long"
                },
                "created_at": {
                  "type": "date",
                  "format" : "EE MMM d HH:mm:ss Z yyyy"
                },
                "favourites_count": {
                  "type": "long"
                },
                "statuses_count": {
                  "type": "long"
                },
                "lang": {
                  "type": "string"
                },
                "profile_image_url": {
                  "type": "string"
                },
                "profile_banner_url": {
                  "type": "string"
                }
              }
            }
          }
        }
      }
    }
  }
}
'
```

# データの登録

```
$ curl -s -XPOST 192.168.33.10:9200/_bulk --data-binary "@bulk_becky_bekiko.json"; echo
```

# kibana

## 設定

+ 「Index name or patter」に「twitter」
+ 「Time-field name」に「created_at」

![1.png](https://qiita-image-store.s3.amazonaws.com/0/7086/4d1fd0de-f3aa-295a-977f-deb79d6ee387.png)

## Discoverページ

+ 「Discover」を選択

![2.png](https://qiita-image-store.s3.amazonaws.com/0/7086/da364b98-685d-17ce-a939-df9d3c607b20.png)


## Geo

+ 「Visualize」⇒「Tile map」
+ geo_pointにnull(```"ignore_malformed": true```)は受け入れてもらえなかったので、適当な初期値のgeoです

![3.png](https://qiita-image-store.s3.amazonaws.com/0/7086/dc11e33e-1be4-51c4-4a6f-f69578f68e42.png)


## 円グラフ

+ 「Visualize」⇒「Pie chart」

### ハッシュタグ数

+ Field「hashtags.text」

![4.png](https://qiita-image-store.s3.amazonaws.com/0/7086/398427d3-80ae-1a47-e80a-d99f9e69bad3.png)


### メンションしたユーザの多い数

+ Field「in_reply.screen_name」

![5.png](https://qiita-image-store.s3.amazonaws.com/0/7086/5903c907-fe19-0b92-3dfc-902d0628ebf4.png)


### RTしたユーザの多い数

+ Field「retweeted_status.user.name」

![6.png](https://qiita-image-store.s3.amazonaws.com/0/7086/f06c7feb-e3b4-028a-2c7a-ddc070aa0e6a.png)

## 縦棒グラフ

+ 「Visualize」⇒「Vertical bar chart」

### Fav数の多いtweet

![7.png](https://qiita-image-store.s3.amazonaws.com/0/7086/4a6e1ed4-7df9-e1c3-63e2-ca6533f24183.png)

<blockquote class="twitter-tweet" lang="en"><p lang="ja" dir="ltr">ゲスの極み乙女。さんの新譜を聴く時のあのドキドキ感はなんていうんだろう。&#10;&#10;もう、恋のドキドキと同じなんですよね。&#10;&#10;胸がきゅーんって。&#10;&#10;そのあとに&#10;&#10;すきすきすきぃー！って。&#10;&#10;今年、ライブを拝見するのが目標です。&#10;&#10;はぁ。。。幸せなためいき。。。</p>&mdash; ベッキー♪♯ (@Becky_bekiko) <a href="https://twitter.com/Becky_bekiko/status/614390603116232704">June 26, 2015</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>


# まとめ

+ データ整形とmappingsを整理するのが大変だった
+ 日本語検索の精度が適当なので、[analysis](https://www.elastic.co/guide/en/elasticsearch/reference/2.x/analysis.html#analysis)の理解深めたい