Upgrade Assistantで簡単アップグレード

# はじめに

とりとめなく書いてます。Elastic Stack新機能シリーズです。
書いたは良いけど、未リリースものばかりなので、その後の修正で変わっちゃったりすると悲しいですね・・・

ま、それはともかく。
今回はUpgrade Assistantを使った6.x系から7.x系へのアップグレードがお題です。

（2019.03.14 [追記](#20190314分)あり）
(2019.03.27 [さらに追記](#20190327分)あり）

# Upgrade Assistantとは

## 5.6系での実装
Upgrade Assistant自体は、古くは5.6系で導入されてます。この時は、設定の確認と指摘、システムが使うIndex（.kibanaとか）の確認とReindexをしてくれてました。

## 6.6での実装
時は流れて6.6でも、7.0系に向けたUpgrade Assistant機能が実装されました。

![UpgradeAssistant_6.6.png](https://qiita-image-store.s3.amazonaws.com/0/221612/f1052ea8-2d42-0cf4-13b4-e2018fc611ef.png)

ここでも、5.6系と同じように設定に関する部分と、Indexのチェックが行われます。
上の画像で分かる通り、5.6系までに作られたIndexだよ、という指摘はしてくれますが、これらは手動でReindexが必要となります。
5.6以前のElasticsearchって結構導入実績ありますよね。それを全部Reindexか・・・と思うと気が重くなります。

## 6.7での実装
で、6.7です。
Reindexしてくれる機能が実装されました。

![UpgradeAssistant_6.7.png](https://qiita-image-store.s3.amazonaws.com/0/221612/59a4b639-085d-9076-660f-64d3d96548b3.png)

6.6の場合と違って、左側に[Reindex]というボタンが表示されてるのがわかります。
じゃ、これをポチポチ押していけば良いわけですね。早速試してみましょう。

# Upgrade Assistantを試す
## パターン1
ごく普通のIndexを試してみます。
こんなんを、

```json
PUT /my_index3
{
   "mappings": {
     "user": {
       "_all": {
         "enabled": true
       }
     }
   }
}
```

5.6系で作ってから、6.7系にもっていって、Upgrade Assistantにかけてみると、

![my_index3_1.png](https://qiita-image-store.s3.amazonaws.com/0/221612/5ec906a9-7c6f-ca48-42c3-e75c60af8552.png)

「`_all`は削除されるよ」と。こんな感じに、7.0以降で使えなくなる機能は指摘してくれます。画面のようにチェックを入れて実行すると、

![my_index3_2.png](https://qiita-image-store.s3.amazonaws.com/0/221612/749b8a20-827a-4b1f-9352-30de9cec9912.png)

無事に移行が終わります（画面右下に「Done」って出てますよね）。
この程度は問題なさそうです。

## パターン2
もちょっと違うものもやってみます。

```json
PUT my_index5
{
   "mappings": {
     "my_type": {
       "properties": {
         "yes_or_no": {
           "type": "boolean"
         }
       }
     }
   }
}
```

`boolean`のフィールドがあります。同じように5.6系で作成⇒6.7系でUpgrade Assistantにかけてみます。

![my_index5_1.png](https://qiita-image-store.s3.amazonaws.com/0/221612/a053aead-6b85-75ca-f7d0-e3b0da367a12.png)

`_all`だけでなく、`_source`にはいる`boolean`の値が変わるよ！という指摘を受けます。
これもチェックして進めると、

![my_index5_2.png](https://qiita-image-store.s3.amazonaws.com/0/221612/13368355-159f-0789-4181-e4f508aa286e.png)

無事に移行が終わります。
なんか、大丈夫そうですね。

## パターン3
さらにやってみます。

```json
PUT my_index2
{
   "mappings": {
     "my_type": {
       "properties": {
         "title": {
           "type": "text"
         },
         "content": {
           "type": "text"
         },
         "date": {
           "type": "date",
           "include_in_all": false
         }
       }
     }
   }
}
```

今度のは`include_in_all`が入っています。これをかけてみると・・・

![my_index2_1.png](https://qiita-image-store.s3.amazonaws.com/0/221612/5296f78f-8395-843b-abfb-eb025306ddca.png)

とりあえず、`_all`は削除されるよという指摘を受けます。チェックを入れて進めると・・・

![my_index2_2.png](https://qiita-image-store.s3.amazonaws.com/0/221612/314ffc88-81c4-30bc-c4a7-0df290e5d2d3.png)

おっと、新しいIndexを作成してる時にエラーが出ました。

```
[mapper_parsing_exception] [include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field.
```

6.0.0以降は`include_in_all`を付けたIndexは作れませんよ、と。Upgrade Assistantでは、これ以上進めることはできません。
これを解消するには、元のmy_index2のMappingを修正して`include_in_all`を削除するか、自分でReindexするしかありません。

## パターン4
さらにやってみます。

```json
PUT my_index
{
   "mappings": {
     "my_parent": {},
     "my_child": {
       "_parent": {
         "type": "my_parent"
       }
     }
   }
}
```

これは明らかにダメなパターンで、そもそもReindexをしてくれません。

![UpgradeAssistant_6.7_2 .png](https://qiita-image-store.s3.amazonaws.com/0/221612/aec60202-d95d-5766-d05d-19a8d16e7b62.png)

青で囲んだところに書いてあります。

```
This index has more than one mapping type, which is not supported in 7.0. This index must be reindexed into one or more single-type indices. Mapping types in use: [my_child, my_parent]
```

このパターンのIndexを持っている人は、自分で何とかするしかありません（Elastic 2.xで作ってた人は、結構このパターンあるかな？）。

## パターン5
ちょっと変なパターンです。

もととなるIndexはこんな感じ。

```json
PUT my_index4
{
   "settings": {
     "number_of_shards": 5
   }
}
```
なんの変哲もないですね。
これを、6.7以降でShrinkして、それをUpgradeしようとすると・・・

![my_index4_shrinked_1.png](https://qiita-image-store.s3.amazonaws.com/0/221612/c25b7d67-39d5-1048-921c-983d4005c458.png)

できそうなんですが・・・

![my_index4_shrinked_2.png](https://qiita-image-store.s3.amazonaws.com/0/221612/202b4df9-e30f-eb79-13ba-a6e1b2d1ba79.png)

エラーが出ます。

```
[illegal_argument_exception] unknown setting [index.resize.source.name] please check that any required plugins are installed, or check the breaking changes documentation for removed settings :: {"path":"/reindexed-v6-my_index4_shrinked","query":{}
```

`index.resize.souce.name`なんて知らんぞ！

```json
GET my_index4_shrinked/_settings
{
   "my_index4_shrinked" : {
     "settings" : {
       "index" : {
         "routing" : {
           "allocation" : {
             "initial_recovery" : {
               "_id" : "C8onVg6uS5Cy9Zd-bvunwA"
             }
           }
         },
         "allocation" : {
           "max_retries" : "1"
         },
         "number_of_shards" : "1",
         "routing_partition_size" : "1",
         "shrink" : {
           "source" : {
             "name" : "my_index4",
             "uuid" : "H5MKwbBUQtCyWd2Kl6mVZA"
           }
         },
         "provided_name" : "my_index4_shrinked",
         "resize" : {
           "source" : {
             "name" : "my_index4",
             "uuid" : "H5MKwbBUQtCyWd2Kl6mVZA"
           }
         },
         "creation_date" : "1550458731700",
         "number_of_replicas" : "1",
         "uuid" : "pbI5MsGnReCGboODcwJ9aQ",
         "version" : {
           "created" : "5061499",
           "upgraded" : "6070099"
         }
       }
     }
   }
}
```
なんか見慣れない設定がいくつかあります。前出の`index.resize`だけでなく`index.shrink`も。
どうも、6.7以降はShrinkするとこいつらがつけられるようです。おまけに消せない。
こうなると、もうUpgrade Assistantは使えないので、自分でReindexするしかありません。

# おまけ
## その1
Upgrade Assistantで実行した結果は、`.kibana`の中で`type`フィールドの値が`upgrade-assistant-reindex-operation`のものの中に入ってます。

## その2
6.0～6.6と違って、7.x系（7.0ブランチ、7.xブランチ）、8.0系（masterブランチ）も変わらずUpgrade Assistantが提供されています。
このまま提供され続けて、機能もアップしてくれると良いですね。

# さいごに
これあると、だいぶ移行がしやすいのかともいますが、注意が必要な部分もあります。
移行に当たっては、7.0系のBreaking Changesをよく見て進めていきましょう！

# 追記

## 2019.03.14分

こんなんが出るようになりました。

![Screenshot_2019-03-14 Kibana.png](https://qiita-image-store.s3.amazonaws.com/0/221612/e5f97ca2-a147-f27c-346f-382463aa6d6b.png)

```:filebeat-6.6.0-2019.02.07-000001
This index has [1127] fields, which exceeds the automatic field 
expansion limit of 1024 and does not have [index.query.default_field] 
set, which may cause queries which use automatic field expansion, such 
as query_string, simple_query_string, and multi_match to fail if fields 
are not explicitly specified in the query.
```

Beats系のIndexは総じてフィールド数が多いんですが、あんまり多いと特定のクエリで失敗する可能性があるよ、と。

## 2019.03.27分

6.7.0がリリースされたので、さっそく6.7.1をビルド。

![Screenshot_2019-03-27 Kibana.png](https://qiita-image-store.s3.amazonaws.com/0/221612/d1fe1f6b-d62a-cc91-fda4-289eb06ea073.png)

あれ？ボタンが追加されちょる。
ポチッとな。

![Screenshot_2019-03-27 Kibana(1).png](https://qiita-image-store.s3.amazonaws.com/0/221612/a9e0b113-4c0b-e112-72bf-c4c708306bfe.png)

ログ見たら、なんか設定変更してくれてるし。

```

[2019-03-27T11:54:07,147][INFO ][o.e.c.s.IndexScopedSettings] [iNU-sMH] updating [index.query.default_field] from [["*"]] to [["apache2.access.agent","apache2.access.geoip.city_name","apache2.access.geoip.continent_name","apache2.access.geoip.country_iso_code","apache2.access.geoip.region_iso_code","apache2.access.geoip.region_name","apache2.access.http_version","apache2.access.method","apache2.access.referrer","apache2.access.remote_ip","apache2.access.url","apache2.access.user_agent.device","apache2.access.user_agent.name","apache2.access.user_agent.original","apache2.access.user_agent.os","apache2.access.user_agent.os_name","apache2.access.user_agent.patch","apache2.access.user_name","apache2.error.client","apache2.error.level","apache2.error.message","apache2.error.module","auditd.log.a0","auditd.log.acct","auditd.log.geoip.city_name","auditd.log.geoip.continent_name","auditd.log.geoip.country_iso_code","auditd.log.geoip.region_iso_code","auditd.log.geoip.region_name","auditd.log.item","auditd.log.items","auditd.log.new_auid","auditd.log.new_ses","auditd.log.old_auid","auditd.log.old_ses","auditd.log.pid","auditd.log.ppid","auditd.log.record_type","auditd.log.res","beat.hostname","beat.name","beat.timezone","beat.version","destination.geo.city_name","destination.geo.continent_name","destination.geo.country_iso_code","destination.geo.region_iso_code","destination.geo.region_name","destination.ip","docker.container.id","docker.container.image","docker.container.name","elasticsearch.audit.action","elasticsearch.audit.event_type","elasticsearch.audit.layer","elasticsearch.audit.origin_address","elasticsearch.audit.origin_type","elasticsearch.audit.principal","elasticsearch.audit.request","elasticsearch.audit.request_body","elasticsearch.audit.uri","elasticsearch.gc.phase.name","elasticsearch.gc.tags","elasticsearch.index.id","elasticsearch.index.name","elasticsearch.node.name","elasticsearch.server.component","elasticsearch.shard.id","elasticsearch.slowlog.extra_source","elasticsearch.slowlog.id","elasticsearch.slowlog.logger","elasticsearch.slowlog.routing","elasticsearch.slowlog.search_type","elasticsearch.slowlog.source_query","elasticsearch.slowlog.stats","elasticsearch.slowlog.took","elasticsearch.slowlog.took_millis","elasticsearch.slowlog.total_hits","elasticsearch.slowlog.total_shards","elasticsearch.slowlog.type","elasticsearch.slowlog.types","error.message","error.type","event.dataset","event.type","file.path","fileset.module","fileset.name","haproxy.backend_name","haproxy.bind_name","haproxy.client.ip","haproxy.destination.ip","haproxy.error_message","haproxy.frontend_name","haproxy.geoip.city_name","haproxy.geoip.continent_name","haproxy.geoip.country_iso_code","haproxy.geoip.region_iso_code","haproxy.geoip.region_name","haproxy.http.request.captured_cookie","haproxy.http.request.captured_headers","haproxy.http.request.raw_request_line","haproxy.http.response.captured_cookie","haproxy.http.response.captured_headers","haproxy.mode","haproxy.process_name","haproxy.server_name","haproxy.source","haproxy.termination_state","host.architecture","host.id","host.ip","host.mac","host.name","host.os.family","host.os.platform","host.os.version","http.request.method","icinga.debug.facility","icinga.debug.message","icinga.debug.severity","icinga.main.facility","icinga.main.message","icinga.main.severity","icinga.startup.facility","icinga.startup.message","icinga.startup.severity","iis.access.agent","iis.access.cookie","iis.access.geoip.city_name","iis.access.geoip.continent_name","iis.access.geoip.country_iso_code","iis.access.geoip.region_iso_code","iis.access.geoip.region_name","iis.access.hostname","iis.access.http_version","iis.access.method","iis.access.query_string","iis.access.referrer","iis.access.remote_ip","iis.access.server_ip","iis.access.server_name","iis.access.site_name","iis.access.url","iis.access.user_agent.device","iis.access.user_agent.name","iis.access.user_agent.original","iis.access.user_agent.os","iis.access.user_agent.os_name","iis.access.user_agent.patch","iis.access.user_name","iis.error.geoip.city_name","iis.error.geoip.continent_name","iis.error.geoip.country_iso_code","iis.error.geoip.region_iso_code","iis.error.geoip.region_name","iis.error.http_version","iis.error.method","iis.error.queue_name","iis.error.reason_phrase","iis.error.remote_ip","iis.error.server_ip","iis.error.url","input.type","kafka.log.class","kafka.log.component","kafka.log.level","kafka.log.message","kafka.log.timestamp","kafka.log.trace.class","kafka.log.trace.full","kafka.log.trace.message","kibana.log.state","kibana.log.tags","kubernetes.container.image","kubernetes.container.name","kubernetes.namespace","kubernetes.node.name","kubernetes.pod.name","kubernetes.pod.uid","log.file.path","log.flags","log.level","log.source.address","logstash.log.level","logstash.log.message","logstash.log.module","logstash.log.thread","logstash.slowlog.event","logstash.slowlog.level","logstash.slowlog.message","logstash.slowlog.module","logstash.slowlog.plugin_name","logstash.slowlog.plugin_params","logstash.slowlog.plugin_type","logstash.slowlog.thread","message","meta.cloud.availability_zone","meta.cloud.instance_id","meta.cloud.instance_name","meta.cloud.machine_type","meta.cloud.project_id","meta.cloud.provider","meta.cloud.region","mongodb.log.component","mongodb.log.context","mongodb.log.message","mongodb.log.severity","mysql.error.level","mysql.error.message","mysql.error.timestamp","mysql.slowlog.host","mysql.slowlog.ip","mysql.slowlog.query","mysql.slowlog.user","netflow.application_category_name","netflow.application_description","netflow.application_group_name","netflow.application_name","netflow.application_sub_category_name","netflow.bgp_next_hop_ipv4_address","netflow.bgp_next_hop_ipv6_address","netflow.class_name","netflow.collector_ipv4_address","netflow.collector_ipv6_address","netflow.destination_ipv4_address","netflow.destination_ipv4_prefix","netflow.destination_ipv6_address","netflow.destination_ipv6_prefix","netflow.destination_mac_address","netflow.dot1q_customer_destination_mac_address","netflow.dot1q_customer_source_mac_address","netflow.encrypted_technology","netflow.exporter.address","netflow.exporter_ipv4_address","netflow.exporter_ipv6_address","netflow.http_content_type","netflow.http_message_version","netflow.http_reason_phrase","netflow.http_request_host","netflow.http_request_method","netflow.http_request_target","netflow.http_user_agent","netflow.information_element_description","netflow.information_element_name","netflow.interface_description","netflow.interface_name","netflow.ip_next_hop_ipv4_address","netflow.ip_next_hop_ipv6_address","netflow.ipv4_router_sc","netflow.metro_evc_id","netflow.mib_context_name","netflow.mib_module_name","netflow.mib_object_description","netflow.mib_object_name","netflow.mib_object_syntax","netflow.mib_object_valuei_pa_ddress","netflow.mobile_imsi","netflow.mobile_msisdn","netflow.mpls_top_label_ipv4_address","netflow.mpls_top_label_ipv6_address","netflow.nat_pool_name","netflow.observation_domain_name","netflow.original_exporter_ipv4_address","netflow.original_exporter_ipv6_address","netflow.p2p_technology","netflow.post_destination_mac_address","netflow.post_nadt_estination_ipv4_address","netflow.post_nadt_estination_ipv6_address","netflow.post_nast_ource_ipv4_address","netflow.post_nast_ource_ipv6_address","netflow.post_source_mac_address","netflow.pseudo_wire_destination_ipv4_address","netflow.sampler_name","netflow.selector_name","netflow.source_ipv4_address","netflow.source_ipv4_prefix","netflow.source_ipv6_address","netflow.source_ipv6_prefix","netflow.source_mac_address","netflow.sta_ipv4_address","netflow.sta_mac_address","netflow.tunnel_technology","netflow.type","netflow.user_name","netflow.virtual_station_interface_name","netflow.virtual_station_name","netflow.vr_fname","netflow.wlan_ssid","netflow.wtp_mac_address","nginx.access.agent","nginx.access.geoip.city_name","nginx.access.geoip.continent_name","nginx.access.geoip.country_iso_code","nginx.access.geoip.region_iso_code","nginx.access.geoip.region_name","nginx.access.http_version","nginx.access.method","nginx.access.referrer","nginx.access.remote_ip","nginx.access.url","nginx.access.user_agent.device","nginx.access.user_agent.name","nginx.access.user_agent.original","nginx.access.user_agent.os","nginx.access.user_agent.os_name","nginx.access.user_agent.patch","nginx.access.user_name","nginx.error.level","nginx.error.message","osquery.result.action","osquery.result.calendar_time","osquery.result.host_identifier","osquery.result.name","postgresql.log.database","postgresql.log.level","postgresql.log.message","postgresql.log.query","postgresql.log.timestamp","postgresql.log.timezone","postgresql.log.user","process.program","prospector.type","read_timestamp","redis.log.level","redis.log.message","redis.log.role","redis.slowlog.args","redis.slowlog.cmd","redis.slowlog.key","service.name","source","source_ecs.geo.city_name","source_ecs.geo.continent_name","source_ecs.geo.country_iso_code","source_ecs.geo.region_iso_code","source_ecs.geo.region_name","source_ecs.ip","stream","suricata.eve.alert.action","suricata.eve.alert.category","suricata.eve.alert.signature","suricata.eve.app_proto","suricata.eve.app_proto_expected","suricata.eve.app_proto_orig","suricata.eve.app_proto_tc","suricata.eve.app_proto_ts","suricata.eve.dest_ip","suricata.eve.dns.rcode","suricata.eve.dns.rdata","suricata.eve.dns.rrname","suricata.eve.dns.rrtype","suricata.eve.dns.type","suricata.eve.email.status","suricata.eve.event_type","suricata.eve.fileinfo.filename","suricata.eve.fileinfo.md5","suricata.eve.fileinfo.sha1","suricata.eve.fileinfo.sha256","suricata.eve.fileinfo.state","suricata.eve.flow.reason","suricata.eve.flow.state","suricata.eve.flow_id","suricata.eve.http.hostname","suricata.eve.http.http_content_type","suricata.eve.http.http_method","suricata.eve.http.http_refer","suricata.eve.http.http_user_agent","suricata.eve.http.protocol","suricata.eve.http.redirect","suricata.eve.http.url","sur
```

長すぎて、ログ切れてるしw