Cloudflare の HTTP リクエストログを Apache Combined Log Format 形式に変換する

# Apache LogFormat

> バージョン確認
>
> ```shell
> $ httpd -v
> Server version: Apache/2.4.37 (centos)
> Server built:   May 20 2021 04:33:06
> ```

デフォルトでは `/etc/httpd/conf/httpd.conf ` にて以下のようなフォーマットが使えます。

```shell
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
```

具体的には `/etc/httpd/logs/access_log` で、以下のような例を確認できます。

```shell
172.68.254.64 - - [13/Aug/2021:01:50:45 +0000] "GET / HTTP/1.1" 200 960 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4240.193 Safari/537.36"
```

# Cloudflare の HTTP リクエストログとのマッピング

`%l`, `%u` はほとんど使われていないフィールドかと思いますが、Cloudflare の HTTP リクエストログには該当する既定フィールドは見つけられていません。

> - [mod_log_config - Apache HTTP Server Version 2.4](https://httpd.apache.org/docs/2.4/mod/mod_log_config.html)
> - [HTTP requests · Cloudflare Logs docs](https://developers.cloudflare.com/logs/reference/log-fields/zone/http_requests)

|                              | Apache         | Example                                                      | Cloudflare                                                   | Example                                                      |
| ---------------------------- | -------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |
| アクセス元のホスト名         | %h             | 172.68.254.64                                                | ClientIP                                                     | 172.68.254.64                                                |
| クライアントの識別子         | %l             | -                                                            | 該当する既定フィールドなし                                   | 該当する既定フィールドなし                                   |
| 認証ユーザー名               | %u             | -                                                            | 該当する既定フィールドなし                                   | 該当する既定フィールドなし                                   |
| リクエストを受け付けた時刻   | %t             | [13/Aug/2021:01:50:45 +0900]                                 | EdgeStartTimestamp                                           | 1628819445609000000 (unixnano)<br/>1629438503 (unix)<br/>2021-08-20T05:29:38Z (rfc3339)                                          |
| リクエストの最初の行の値     | %r             | GET / HTTP/1.1                                               | ClientRequestMethod<br/>ClientRequestURI<br/>ClientRequestProtocol | GET<br/>/<br/>HTTP/1.1                                       |
| 最後のレスポンスのステータス | %>s            | 200                                                          | EdgeResponseStatus                                           | 200                                                          |
| 送信されたバイト数           | %b             | 960                                                          | EdgeResponseBytes                                            | 1336                                                         |
| リファラー                   | %{Referer}i    | https://example.com                                          | ClientRequestReferer                                         | https://example.com                                          |
| User Agent                   | %{User-Agent}i | Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0 | ClientRequestUserAgent                                       | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4240.193 Safari/537.36 |

# Cloudflare Log を取得し、Apache Combined Log Format 形式に変換

上記のマッピングを参考に、ログを取得・変換するコマンドです。

- `ClientRequestReferer` で値がない場合は `-` を設定
- `EdgeStartTimestamp` で `sort`
- `EdgeStartTimestamp` を `+32400` で JST に変換
- などの処理をおこなって、それぞれのフィールド値を半角スペース ` ` で結合

```shell
export EMAIL='YOUR_EMAIL'
export APIKEY='YOUR_APIKEY'
export ZONE_ID='YOUR_ZONE_ID'
export FIELDS=ClientIP,EdgeStartTimestamp,ClientRequestMethod,ClientRequestURI,ClientRequestProtocol,EdgeResponseStatus,EdgeResponseBytes,ClientRequestReferer,ClientRequestUserAgent
export APACHE_FIELDS=.${FIELDS//,/,.}

START='2021-08-13T14:30:00%2B09:00'
END='2021-08-13T15:30:00%2B09:00'

curl -s \
-H "X-Auth-Email: $EMAIL" \
-H "X-Auth-Key: $APIKEY" \
"https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logs/received?timestamps=unix&start=$START&end=$END&fields=$FIELDS" | jq -s -r '.|=map(select(.ClientRequestReferer=="").ClientRequestReferer|="-" // .) |sort_by(.EdgeStartTimestamp)|.[]|.EdgeStartTimestamp="- - [" + (.EdgeStartTimestamp|.+32400|strftime("%d/%b/%Y:%H:%M:%S +0900")) + "]"|.ClientRequestMethod="\"" + .ClientRequestMethod | .ClientRequestProtocol = .ClientRequestProtocol + "\"" |.ClientRequestReferer="\""+.ClientRequestReferer+"\""|.ClientRequestUserAgent="\""+.ClientRequestUserAgent+"\""|['"$APACHE_FIELDS"']|join(" ")'
```

# 変換後
このように変換しておくと、Apache ログ形式に対応した分析ツール等でそのまま解析できるので、既存のツールを活用できる可能性があります。

```shell:result.log
x.x.x.x - - [13/Aug/2021:14:57:48 +0900] "GET / HTTP/1.1" 200 1224 "-" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:14:58:25 +0900] "GET /favicon.ico HTTP/1.1" 404 659 "http://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:14:58:25 +0900] "GET / HTTP/1.1" 200 1185 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:14:58:33 +0900] "GET / HTTP/2" 200 1218 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:14:58:33 +0900] "GET /favicon.ico HTTP/2" 404 775 "https://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:14:58:53 +0900] "GET / HTTP/2" 200 1237 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:14:59:51 +0900] "GET / HTTP/1.1" 200 1224 "-" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:14:59:07 +0900] "GET / HTTP/2" 200 1238 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:14:59:07 +0900] "GET /favicon.ico HTTP/2" 404 783 "https://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
x.x.x.x - - [13/Aug/2021:15:06:43 +0900] "GET / HTTP/1.1" 200 1256 "-" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:07:11 +0900] "GET / HTTP/1.1" 200 1262 "-" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:07:14 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:22:13 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:22:15 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:22:16 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:22:17 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:22:18 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
x.x.x.x - - [13/Aug/2021:15:22:21 +0900] "GET / HTTP/1.1" 200 1264 "https://example.com" "curl/7.64.1"
```

# 参考

- [Cloudflare の HTTP リクエストログ - Qiita](https://qiita.com/khayama/items/85c87423261442a35d0b)
- [Apache | LogFormatディレクティブ：ログファイルのフォーマットを定義する](https://www.javadrive.jp/apache/log/index2.html)
- [jq で エポックミリ秒を変換したい - Qiita](https://qiita.com/sutoh/items/350bb96a06cb237f54af)
- [jq で特定条件にマッチする要素を置換する - なにもわからない](https://tamakiii.hatenablog.com/entry/2019/11/21/001343)
- [Date & time formats cheatsheet](https://devhints.io/datetime)

# 参考：Apache アクセスログ

```shell
$ sudo tail /etc/httpd/logs/access_log
- - - [13/Aug/2021:06:22:15 +0000] "GET / HTTP/1.1" 200 626 "https://example.com" "curl/7.64.1"
- - - [13/Aug/2021:06:22:16 +0000] "GET / HTTP/1.1" 200 626 "https://example.com" "curl/7.64.1"
- - - [13/Aug/2021:06:22:17 +0000] "GET / HTTP/1.1" 200 626 "https://example.com" "curl/7.64.1"
- - - [13/Aug/2021:06:22:18 +0000] "GET / HTTP/1.1" 200 626 "https://example.com" "curl/7.64.1"
- - - [13/Aug/2021:06:22:21 +0000] "GET / HTTP/1.1" 200 626 "https://example.com" "curl/7.64.1"
- - - [13/Aug/2021:06:49:29 +0000] "GET / HTTP/1.1" 200 832 "-" "Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers&#39; presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com"
- - - [13/Aug/2021:09:06:41 +0000] "GET /favicon.ico HTTP/1.1" 404 196 "http://example.com/.htaccess" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
- - - [13/Aug/2021:09:08:36 +0000] "GET /favicon.ico HTTP/1.1" 404 196 "https://example.com/.htaccess?param1=1234" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
- - - [13/Aug/2021:09:17:06 +0000] "GET /favicon.ico HTTP/1.1" 404 196 "https://example.com/.htaccess?param1=1234" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
- - - [13/Aug/2021:09:34:30 +0000] "GET /favicon.ico HTTP/1.1" 404 196 "http://example.com/.htaccess?param1=1234" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0"
```
```
sudo tail -f /etc/httpd/logs/access_log
sudo tail -f /etc/httpd/logs/ssl_access_log
```

# 参考：GoAccess で可視化

- [GoAccess - Visual Web Log Analyzer](https://goaccess.io/)

```
brew install goaccess
cat access.log | goaccess -a -o html --log-format COMBINED - > report.html
```

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/101361/1ff98ec8-d4f4-2963-2886-54b946bb51b3.png)