ElasticsearchのAPI監査ログ

## はじめに
Elasticsearchに対する操作をどこまで追えるのか検証する機会があり
有償機能であるSecurityの**Audit Logging**を試したので、簡単ですが記事にまとめてみました。

## 利用環境
| product | version |
|---------|---------|
| kibana | 6.6.0 |
| Elasticsearch | 6.6.0 |
| X-Pack | 6.6.0 |
| OS(EC2) | Amazon Linux2 |
| AMI ID | ami-04677bdaa3c2b6e24 |

## 有償機能 Securityとは
現行最新版6.6では、以下のセキュリティ機能を有償機能として提供しています。
※今回の投稿は★の**監査ログ(Audit Logging)**に関する内容になります。

**Elasticsearch**
・暗号化通信
・役割ベースのアクセス制御
・監査ログ　★
・IPフィルタリング
・ネイティブ、LDAP、PKI、Active Directory認証
・シングルサインオン認証（SAML、Kerberos)
・属性ベースのアクセス制御
・フィールドレベルおよびドキュメントレベルのセキュリティ
・サードパーティの統合(カスタムレルム)
・認可レルム
・暗号化による保存のサポート
・FIPS 140-2モード

**Kibana**
・暗号化通信
・役割ベースのアクセス制御(Spacesを含む)
・ネイティブ、LDAP、Active Directory認証
・シングルサインオン認証(SAML)

**【参考】**
・[セキュリティ機能のはじめ方][01]


## 前提事項
Securityを有効化するとデフォルトで以下2つの機能を設定する必要があります。
① Elasticsearchのパスワード認証設定
② Elasticsearchのノード間TLS暗号化通信設定

6.3からX-Packという文言がFeatureという表現に変わったため、ディレクトリ構造も変わっています。
※本投稿では割愛していますが、以下のサイトを参考に設定してみてください！

**6.2以前)** /usr/share/elasticsearch/bin/x-pack/setup-passwords
**6.3以降)** /usr/share/elasticsearch/bin/elasticsearch-setup-passwords

**6.2以前)** /usr/share/elasticsearch/bin/x-pack/certutil
**6.3以降)** /usr/share/elasticsearch/bin/elasticsearch-certutil

**【参考】**
・[X-Pack Securityの設定][02]

## Audit Logging設定方法
**Audit Logging**の設定は以下の通りになります。
※今回は全操作の監査ログをElasticsearchにIndexするように設定

```ruby:elasticsearch.yml
xpack.security.audit.enabled: true
xpack.security.audit.outputs: index
xpack.security.audit.index.events.emit_request_body: true
xpack.security.audit.logfile.events.include: access_denied, access_granted, anonymous_access_denied, authentication_failed, connection_denied, tampered_request, run_as_denied, run_as_granted,authentication_success,realm_authentication_failed,connection_granted
```

```ruby:kibana.yml
xpack.security.audit.enabled: true
```

**【参考】**
・[Auditing security events][03]
・[Audit event types][04]

## 実施したこと
セキュリティの監査ログ機能ですので、以下の情報がちゃんと記録されているかを確認してみました。

**【検証シナリオ】**
① Kibanaへのログイン操作
② Discover画面での検索操作
③ Dashboard画面でのダッシュボード参照操作
④ DevTools画面からGETクエリ操作
⑤ Monitoring画面からクラスタ状態確認の操作
⑥ Management画面から新規ユーザ作成操作

**【確認ポイント】**
・ 誰が(操作ユーザ)
・ いつ(timestamp)
・ どこから(送信元IP)
・ 何をしたのか(操作内容、実行クエリなど)
・ 実行結果(成功したのか失敗したのか)


## 結果
Elasticsearchに生成される`.security_audit_log-yyyy.MM.dd`にElasticsearchのAPI操作ログは記録されます。
`172.31.6.143`はKibana、`172.31.8.191`はElasticsearchのIPアドレスになります。

#### ① Kibanaへのログイン操作
`elastic`というユーザでのログイン成功になります。

```ruby:Elasticsearch_log_success
{
  "_index": ".security_audit_log-2019.02.19",
  "_type": "doc",
  "_id": "Uot_BGkBVzdfFIt86rgk",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-19T06:44:44.843Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "rest",
    "event_type": "authentication_success",
    "principal": "elastic",
    "realm": "reserved",
    "request_body" : "",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "uri": "/_xpack/security/_authenticate"
  },
  "fields": {
    "@timestamp": [
      "2019-02-19T06:44:44.843Z"
    ]
  },
  "sort": [
    1550558684843
  ]
}
```

`elastic1`というユーザでのログイン失敗になります。

```ruby:Elasticsearch_log_failed
{
  "_index": ".security_audit_log-2019.02.19",
  "_type": "doc",
  "_id": "1Yt_BGkBVzdfFIt81reZ",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-19T06:44:40.036Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "rest",
    "event_type": "realm_authentication_failed",
    "principal": "elastic1",
    "realm": "reserved",
    "request_body" : "",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "uri": "/_xpack/security/_authenticate"
  },
  "fields": {
    "@timestamp": [
      "2019-02-19T06:44:40.036Z"
    ]
  },
  "sort": [
    1550558680036
  ]
}
{
  "_index": ".security_audit_log-2019.02.19",
  "_type": "doc",
  "_id": "1ot_BGkBVzdfFIt81reZ",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-19T06:44:40.036Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "rest",
    "event_type": "realm_authentication_failed",
    "principal": "elastic1",
    "realm": "default_file",
    "request_body" : "",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "uri": "/_xpack/security/_authenticate"
  },
  "fields": {
    "@timestamp": [
      "2019-02-19T06:44:40.036Z"
    ]
  },
  "sort": [
    1550558680036
  ]
}
{
  "_index": ".security_audit_log-2019.02.19",
  "_type": "doc",
  "_id": "2Yt_BGkBVzdfFIt81reZ",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-19T06:44:40.040Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "rest",
    "event_type": "authentication_failed",
    "principal": "elastic1",
    "request_body" : "",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "uri": "/_xpack/security/_authenticate"
  },
  "fields": {
    "@timestamp": [
      "2019-02-19T06:44:40.040Z"
    ]
  },
  "sort": [
    1550558680040
  ]
}
{
  "_index": ".security_audit_log-2019.02.19",
  "_type": "doc",
  "_id": "2It_BGkBVzdfFIt81reZ",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-19T06:44:40.040Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "rest",
    "event_type": "realm_authentication_failed",
    "principal": "elastic1",
    "realm": "default_native",
    "request_body" : "",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "uri": "/_xpack/security/_authenticate"
  },
  "fields": {
    "@timestamp": [
      "2019-02-19T06:44:40.040Z"
    ]
  },
  "sort": [
    1550558680040
  ]
}
```

Kibana.ymlで指定したログ出力先にもKibanaのログインログは出力されます。

```ruby:Kibana_log_success
{"type":"log","@timestamp":"2019-02-15T08:54:11Z","tags":["info","audit","spaces","spaces_authorization_success"],"pid":6358,"eventType":"spaces_authorization_success","username":"elastic","action":"getAll","spaceIds":["default"],"message":"elastic authorized to getAll default spaces"}
```
```ruby:Kibana_log_failed
{"type":"log","@timestamp":"2019-02-15T08:57:00Z","tags":["debug","security","basic"],"pid":6358,"message":"Failed to authenticate request via login attempt: [security_exception] unable to authenticate user [elastic1] for REST request [/_xpack/security/_authenticate], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } }"}
```

#### ② Discover画面での検索操作

```ruby:search_discover
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "8l8-6mgBKr8WQ3DYrtrV",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:23:22.649Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "rest",
    "event_type": "authentication_success",
    "principal": "elastic",
    "realm": "reserved",
　　"request_body": "{\"index\":\".security_audit_log-*\",\"ignore_unavailable\":true,\"preference\":1551861247219}\n{\"version\":true,\"size\":500,\"sort\":[{\"@timestamp\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}}],\"_source\":{\"excludes\":[]},\"aggs\":{\"2\":{\"date_histogram\":{\"field\":\"@timestamp\",\"interval\":\"5m\",\"time_zone\":\"Asia/Tokyo\",\"min_doc_count\":1}}},\"stored_fields\":[\"*\"],\"script_fields\":{},\"docvalue_fields\":[{\"field\":\"@timestamp\",\"format\":\"date_time\"}],\"query\":{\"bool\":{\"must\":[{\"query_string\":{\"query\":\"elastic\",\"analyze_wildcard\":true,\"default_field\":\"*\"}},{\"range\":{\"@timestamp\":{\"gte\":1551861764755,\"lte\":1551884399999,\"format\":\"epoch_millis\"}}}],\"filter\":[],\"should\":[],\"must_not\":[]}},\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"post_tags\":[\"@/kibana-highlighted-field@\"],\"fields\":{\"*\":{}},\"fragment_size\":2147483647},\"timeout\":\"30000ms\"}\n",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "uri": "/_msearch?rest_total_hits_as_int=true&ignore_throttled=true"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:23:22.649Z"
    ]
  },
  "sort": [
    1550118202649
  ]
}
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "9F8-6mgBKr8WQ3DYrtrV",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:23:22.649Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "transport",
    "event_type": "access_granted",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "action": "indices:data/read/search",
    "principal": "elastic",
    "realm": "reserved",
    "roles": [
      "superuser"
    ],
    "indices": [
      ".security_audit_log-2019.02.14",
      ".security_audit_log-2019.02.13",
      ".security_audit_log-2019.02.12",
      ".security_audit_log-2019.02.11"
    ],
    "request": "SearchRequest"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:23:22.649Z"
    ]
  },
  "sort": [
    1550118202649
  ]
}
```

#### ③ Dashboard画面でのダッシュボード参照操作

```ruby:view_dashboard
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "mF9E6mgBKr8WQ3DY1fj6",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:30:05.459Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "rest",
    "event_type": "authentication_success",
    "principal": "elastic",
    "realm": "reserved",
    "request_body": "{\"index\":\".security_audit_log-*\",\"ignore_unavailable\":true,\"preference\":1551862232912}\n{\"aggs\":{},\"size\":0,\"_source\":{\"excludes\":[]},\"stored_fields\":[\"*\"],\"script_fields\":{},\"docvalue_fields\":{\"field\":\"timestamp\",\"format\":\"date_time\"},\"query\":{\"bool\":{\"must\":[{\"match_all\":{}},{\"range\":{\"timestamp\":{\"gte\":1551858325840,\"lte\":1551884399999,\"format\":\"epoch_millis\"}}}],\"filter\":[],\"should\":[],\"must_not\":[]}},\"timeout\":\"30000ms\"}\n",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "uri": "/_msearch?rest_total_hits_as_int=true&ignore_throttled=true"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:30:05.459Z"
    ]
  },
  "sort": [
    1550118605459
  ]
}
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "ml9E6mgBKr8WQ3DY1fj6",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:30:05.460Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "transport",
    "event_type": "access_granted",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "action": "indices:data/read/search",
    "principal": "elastic",
    "realm": "reserved",
    "roles": [
      "superuser"
    ],
    "indices": [
      ".security_audit_log-2019.02.14",
      ".security_audit_log-2019.02.13",
      ".security_audit_log-2019.02.12",
      ".security_audit_log-2019.02.11"
    ],
    "request": "SearchRequest"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:30:05.460Z"
    ]
  },
  "sort": [
    1550118605460
  ]
}
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "mV9E6mgBKr8WQ3DY1fj6",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:30:05.460Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "transport",
    "event_type": "access_granted",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "action": "indices:data/read/msearch",
    "principal": "elastic",
    "realm": "reserved",
    "roles": [
      "superuser"
    ],
    "request": "MultiSearchRequest"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:30:05.460Z"
    ]
  },
  "sort": [
    1550118605460
  ]
}
```

#### ④ DevTools画面からGETクエリ操作
[DevTools]から`GET _cat/indcies`を実行した操作ログになります。 

```ruby:getapi_devtools
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "L19F6mgBKr8WQ3DYXvvD",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:30:40.606Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "transport",
    "event_type": "access_granted",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "action": "indices:admin/get",
    "principal": "elastic",
    "realm": "reserved",
    "roles": [
      "superuser"
    ],
    "indices": [
      ".triggered_watches",
      ".monitoring-es-6-2019.02.08",
      ".monitoring-es-6-2019.02.07",
      ".monitoring-es-6-2019.02.09",
      ".security_audit_log-2019.02.14",
      ".security_audit_log-2019.02.13",
      ".kibana",
      ".security_audit_log-2019.02.12",
      ".security_audit_log-2019.02.11",
      ".monitoring-alerts-6",
      ".monitoring-kibana-6-2019.02.10",
      ".monitoring-kibana-6-2019.02.11",
      ".monitoring-kibana-6-2019.02.12",
      ".monitoring-kibana-6-2019.02.13",
      ".monitoring-kibana-6-2019.02.14",
      ".watches",
      ".monitoring-es-6-2019.02.11",
      ".monitoring-es-6-2019.02.10",
      ".monitoring-es-6-2019.02.14",
      ".monitoring-es-6-2019.02.13",
      ".monitoring-es-6-2019.02.12",
      ".kibana_1",
      ".monitoring-kibana-6-2019.02.07",
      ".security",
      ".monitoring-kibana-6-2019.02.08",
      ".monitoring-kibana-6-2019.02.09",
      "logstash-2019.02.03",
      ".security-6"
    ],
    "request": "GetIndexRequest"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:30:40.606Z"
    ]
  },
  "sort": [
    1550118640606
  ]
}
```

#### ⑤ Monitoring画面からクラスタ状態確認の操作
[Monitoring]でElasticsearch Clusterの性能を確認した操作ログになります。

```ruby:view_monitoring
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "mmBH6mgBKr8WQ3DYVwUZ",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:32:50.349Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "rest",
    "event_type": "authentication_success",
    "principal": "elastic",
    "realm": "reserved",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "uri": "/.monitoring-es-2-*%2C.monitoring-es-6-*/_search?size=10000&ignore_unavailable=true&filter_path=hits.hits._index%2Chits.hits._source.cluster_uuid%2Chits.hits._source.cluster_name%2Chits.hits._source.version%2Chits.hits._source.license.status%2Chits.hits._source.license.type%2Chits.hits._source.license.issue_date%2Chits.hits._source.license.expiry_date%2Chits.hits._source.license.expiry_date_in_millis%2Chits.hits._source.cluster_stats%2Chits.hits._source.cluster_state%2Chits.hits._source.cluster_settings.cluster.metadata.display_name"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:32:50.349Z"
    ]
  },
  "sort": [
    1550118770349
  ]
}
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "m2BH6mgBKr8WQ3DYVwUZ",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:32:50.349Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "transport",
    "event_type": "access_granted",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "action": "indices:data/read/search",
    "principal": "elastic",
    "realm": "reserved",
    "roles": [
      "superuser"
    ],
    "indices": [
      ".monitoring-es-6-2019.02.11",
      ".monitoring-es-6-2019.02.10",
      ".monitoring-es-6-2019.02.14",
      ".monitoring-es-6-2019.02.13",
      ".monitoring-es-6-2019.02.12",
      ".monitoring-es-6-2019.02.08",
      ".monitoring-es-6-2019.02.07",
      ".monitoring-es-6-2019.02.09"
    ],
    "request": "SearchRequest"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:32:50.349Z"
    ]
  },
  "sort": [
    1550118770349
  ]
}
```

#### ⑥ Management画面から新規ユーザ作成操作
`test_user`という新規ユーザを作成した操作ログになります。

```ruby:create_user
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "FmBG6mgBKr8WQ3DYxgJ_",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:32:12.465Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "rest",
    "event_type": "authentication_success",
    "principal": "elastic",
    "realm": "reserved",
    "request_body" : """{"full_name":"test_user","roles":["kibana_dashboard_only_user"],"email":""}""",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "uri": "/_xpack/security/user/test_user"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:32:12.465Z"
    ]
  },
  "sort": [
    1550118732465
  ]
}
{
  "_index": ".security_audit_log-2019.02.14",
  "_type": "doc",
  "_id": "H2BG6mgBKr8WQ3DYxgJ_",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T04:32:12.547Z",
    "node_name": "ip-172-31-8-191.ap-southeast-1.compute.internal",
    "node_host_name": "172.31.8.191",
    "node_host_address": "172.31.8.191",
    "layer": "transport",
    "event_type": "access_granted",
    "origin_type": "rest",
    "origin_address": "172.31.6.143",
    "action": "cluster:admin/xpack/security/user/put",
    "principal": "elastic",
    "realm": "reserved",
    "roles": [
      "superuser"
    ],
    "request": "PutUserRequest"
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T04:32:12.547Z"
    ]
  },
  "sort": [
    1550118732547
  ]
}
```


## まとめ
**・誰が(操作ユーザ)**
`"principal": "elastic"`でelasticというユーザの操作であることが判断出来ました。

**・いつ(timestamp)**
`@timestamp`でログの記録時間が追えました。

**・どこから(送信元IP)**
Kibana操作は全て`"origin_address": "172.31.6.143"`となり、操作者端末の特定は出来ませんでした。
今のところ、X-forwarded-forをElasticsearchのAudit Logに記録する方法は無いようです。

**・何をしたのか(操作内容、実行クエリなど)**
`xpack.security.audit.index.events.emit_request_body: true`を設定していることで
`request_body`がログに表示されます。こちらを見ることで実行した検索クエリを追うことが出来ました。

**・実行結果(成功したのか失敗したのか)**
以下の`event_type`で実行結果が判断出来ました。
・ログイン成功: authentication_success
・ログイン失敗: authentication_failed
・実行成功: access_granted
・実行失敗: access_denied


[01]:https://www.elastic.co/jp/blog/getting-started-with-security "セキュリティ機能のはじめ方"
[02]:https://www.server-world.info/query?os=CentOS_7&p=elasticstack6&f=12 "X-Pack Securityの設定"
[03]:https://www.elastic.co/guide/en/elastic-stack-overview/current/auditing.html "Auditing security events"
[04]:https://www.elastic.co/guide/en/elastic-stack-overview/current/audit-event-types.html "Audit event types"