Cloudflare の Audit Log を Google Cloud Storage に Logpush する

# Audit Log Retention

デフォルトで 18 ヶ月保持されますが、それ以上遡って保管したい場合は、[Logpush](https://developers.cloudflare.com/logs/get-started/) で随時外部にエクスポートすることができます。

> [Understanding Cloudflare Audit Logs – Cloudflare Help Center](https://support.cloudflare.com/hc/en-us/articles/115002833612-Understanding-Cloudflare-Audit-Logs)
>
> To maintain Audit Logs query performance, the public Audit Logs API was  modified on Jun 30 2019 to return records with a maximum age of 18  months.

こちらのドキュメントを参考に進めます。

> [Manage Logpush with cURL · Cloudflare Logs docs](https://developers.cloudflare.com/logs/reference/logpush-api-configuration/examples/example-logpush-curl)

# アカウントを特定する変数をセット

API 経由で Logpush Job を設定するために、変数をセットします。

> [Cloudflare API v4 Documentation](https://api.cloudflare.com/#logpush-jobs-properties)

```shell
export EMAIL='YOUR_EMAIL'
export APIKEY='YOUR_APIKEY'
export ACCOUNT_ID='YOUR_ACCOUNT_ID'
export DATASET='audit_logs' # can be gateway_dns, gateway_http
export BUCKET='kyouhei'
export FOLDER='audit_log'
export JOB_NAME='audit-log' # cannot be changed after job creation. don't use "_"
```

# Ownership challenge を取得

Google Cloud Storage を Logpush のターゲットにするための Ownership challenge を行います。

- 事前に Google Cloud Storage でバケット `kyouhei` とディレクトリ `audit_log` を作成します。
- `{DATE}` を指定することでディレクトリ内でログを日付ごとのフォルダに分けて保管できます。

```shell
export CHALLENGE_PATH=$(curl -s -X POST "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/logpush/ownership" \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
-H "Content-Type: application/json" \
--data '{"destination_conf":"gs://'$BUCKET'/'$FOLDER'/{DATE}"}' | jq -r .result.filename)
# Ownership challenge を取得
export CHALLENGE=$(gsutil cat gs://$BUCKET/$CHALLENGE_PATH)
echo $CHALLENGE
```

```txt:result.txt
eyJhbGciOiJkaXIiLCJlbmMiOxxxxxxxxxx
```

# Field を変数にセット

以下のコマンドでそれぞれセットします。

```shell
# 全フィールドを指定
export FIELDS=$(curl -s \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
"https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/logpush/datasets/$DATASET/fields" | jq -r '.result | keys | join(",")')
echo $FIELDS
```

# Logpush Job を作成

`dataset` を `audit_logs` として、`account` レベルの Logpush Job を作成します。

```shell
curl -s -X POST \
https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/logpush/jobs \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
-H "Content-Type: application/json" \
-d'{"name": "'$JOB_NAME'", "destination_conf":"gs://'$BUCKET'/'$FOLDER'/{DATE}", "dataset": "'$DATASET'", "logpull_options":"fields='$FIELDS'&timestamps=rfc3339", "ownership_challenge":"'$CHALLENGE'"}' | jq .
```

```json:result.json
{
  "errors": [],
  "messages": [],
  "result": {
    "id": 42883,
    "dataset": "audit_logs",
    "frequency": "high",
    "kind": "",
    "enabled": false,
    "name": audit-log,
    "logpull_options": "fields=ActionResult,ActionType,ActorEmail,ActorID,ActorIP,ActorType,ID,Interface,Metadata,NewValue,OldValue,OwnerID,ResourceID,ResourceType,When&timestamps=rfc3339",
    "destination_conf": "gs://kyouhei/audit_log/{DATE}",
    "last_complete": null,
    "last_error": null,
    "error_message": null
  },
  "success": true
}
```

# 作成した Logpush Job を有効化

JobID を指定して、`enabled` が `false` になっているのを `true` にセットして有効化します。

```shell
curl -s -X PUT \
-d'{"enabled":true}' \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
-H "Content-Type: application/json" \
https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/logpush/jobs/42883 | jq .
```

```json:result.json
{
  "errors": [],
  "messages": [],
  "result": {
    "id": 42883,
    "dataset": "audit_logs",
    "frequency": "high",
    "kind": "",
    "enabled": true,
    "name": null,
    "logpull_options": "fields=ActionResult,ActionType,ActorEmail,ActorID,ActorIP,ActorType,ID,Interface,Metadata,NewValue,OldValue,OwnerID,ResourceID,ResourceType,When&timestamps=rfc3339",
    "destination_conf": "gs://kyouhei/audit_log/{DATE}",
    "last_complete": null,
    "last_error": null,
    "error_message": null
  },
  "success": true
}
```

# Logpush Job のリスト（アカウントレベル）

アカウントレベルでの Logpush Job リストを確認できます。

`last_complete` が最後にエクスポートが実行された時間です。

```shell
curl -X GET "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/logpush/jobs" \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
-H "Content-Type: application/json" | jq .
```

```json:result.json
{
  "errors": [],
  "messages": [],
  "result": [
    {
      "id": 42883,
      "dataset": "audit_logs",
      "frequency": "high",
      "kind": "",
      "enabled": true,
      "name": null,
      "logpull_options": "fields=ActionResult,ActionType,ActorEmail,ActorID,ActorIP,ActorType,ID,Interface,Metadata,NewValue,OldValue,OwnerID,ResourceID,ResourceType,When&timestamps=rfc3339",
      "destination_conf": "gs://kyouhei/audit_log/{DATE}",
      "last_complete": "2021-10-28T02:24:15Z",
      "last_error": null,
      "error_message": null
    }
  ],
  "success": true
}
```

# Logpush の確認

## UI

Account Home > Audit Log から確認できます。

![image-20211028151833667](https://cdn.jsdelivr.net/gh/kyouheicf/image@master/uPic/image-20211028151833667.png)
![image-20211028151934446](https://cdn.jsdelivr.net/gh/kyouheicf/image@master/uPic/image-20211028151934446.png)

![image-20211028152035419](https://cdn.jsdelivr.net/gh/kyouheicf/image@master/uPic/image-20211028152035419.png)

## Google Cloud Storage

該当するログファイルを確認できました。

```shell
gsutil cat gs://kyouhei/audit_log/20211027/20211027T051004Z_20211027T051034Z_32068b24.log.gz | gunzip | jq .
```

```json:result.json
{
  "ActionResult": true,
  "ActionType": "waiting_rooms_edit",
  "ActorEmail": "xxx",
  "ActorID": "xx",
  "ActorIP": "240d:xxx",
  "ActorType": "user",
  "ID": "0a2fc5f6-4900-46ed-9364-c31a7cd9a110",
  "Interface": "UI",
  "Metadata": {
    "actor_email": "xxx",
    "zone_name": "xxx",
    "zone_tag": "xxx"
  },
  "NewValue": {
    "id": "ea05cb62a47a8e66e387002eae48ab1d",
    "name": "waiting_room",
    "description": "",
    "host": "wait.xxx",
    "path": "/",
    "suspended": false,
    "queue_all": false,
    "new_users_per_minute": 200,
    "total_active_users": 200,
    "session_duration": 5,
    "custom_page_html": "",
    "disable_session_renewal": false,
    "json_response_enabled": false,
    "queueing_method": "fifo",
    "cookie_attributes": {
      "samesite": "auto",
      "secure": "auto"
    },
    "created_on": "2021-10-20T05:14:59.449217Z",
    "modified_on": "2021-10-27T05:08:09.273506Z"
  },
  "OldValue": {
    "id": "ea05cb62a47a8e66e387002eae48ab1d",
    "name": "wait",
    "description": "",
    "host": "wait.xxx",
    "path": "/",
    "suspended": false,
    "queue_all": false,
    "new_users_per_minute": 200,
    "total_active_users": 200,
    "session_duration": 5,
    "custom_page_html": "",
    "disable_session_renewal": false,
    "json_response_enabled": false,
    "queueing_method": "fifo",
    "cookie_attributes": {
      "samesite": "auto",
      "secure": "auto"
    },
    "created_on": "2021-10-20T05:14:59.449217Z",
    "modified_on": "2021-10-22T05:45:19.625349Z"
  },
  "OwnerID": "xxx",
  "ResourceID": "xxx",
  "ResourceType": "Waiting Room",
  "When": "2021-10-27T05:08:09Z"
}
```

# 参考：Logpush Job を削除

JobID を指定して Logpush Job を削除できます。

```shell
curl -s -X DELETE \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
-H "Content-Type: application/json" \
https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/logpush/jobs/42080 | jq .
```

```json:result.json
{
  "errors": [],
  "messages": [],
  "result": null,
  "success": true
}
```

# 参考：Google Cloud Storage にある最新10個のログファイルの中身を表示

```shell
export BUCKET='kyouhei'
export FOLDER='audit_log'

gsutil list gs://$BUCKET/$FOLDER/$(date +"%Y%m%d") | grep .log.gz | tail -n 10 | while read line
do
  gsutil cat $line | gunzip | jq .
done
```

# 参考；ZONE ベースのスクリプト
```shell
export ZONE_ID='YOUR_ZONE_ID'
---
# Ownership challenge を取得
export CHALLENGE_PATH=$(curl -s -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logpush/ownership" \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
-H "Content-Type: application/json" \
--data '{"destination_conf":"gs://'$BUCKET'/'$FOLDER'/{DATE}"}' | jq -r .result.filename)

export CHALLENGE=$(gsutil cat gs://$BUCKET/$CHALLENGE_PATH)
echo $CHALLENGE
---
# 全フィールドを指定
export FIELDS=$(curl -s \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
"https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logpush/datasets/$DATASET/fields" | jq -r '.result | keys | join(",")')
echo $FIELDS
---
# ジョブ作成
curl -s -X POST \
https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logpush/jobs \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
-H "Content-Type: application/json" \
-d'{"name": "'$JOB_NAME'", "destination_conf":"gs://'$BUCKET'/'$FOLDER'/{DATE}", "dataset": "'$DATASET'", "logpull_options":"fields='$FIELDS'&timestamps=rfc3339", "ownership_challenge":"'$CHALLENGE'"}' | jq .
---
# 有効化
curl -s -X PUT \
-d'{"enabled":true}' \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
-H "Content-Type: application/json" \
https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logpush/jobs/42883 | jq .
---
# ジョブのリスト
curl -X GET "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logpush/jobs" \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
-H "Content-Type: application/json" | jq .
```