Kubernetes Auditingで誰がなにをしたのかをみてみる

## はじめに

本番環境やそれに相当するようなセンシティブな環境では、監査要件として

『**誰**が**どのようなコマンド**を打ったか』  

を収集することがよくある。  
自分の場合、大体は（内部向け説明がしやすい）従来の運用方法を踏襲するような形で上記要件をクリアする（してしまう）が、KubernetesのAuditingを使うことで似たような情報を収集することができる。  
（「どのようなコマンドか」ではなく「どのような**リクエスト**か」を収集）

[https://kubernetes.io/docs/tasks/debug-application-cluster/audit/:embed:cite]
どのような出力がされるのか、試しにみてみる。

## 環境
- minikube v1.9.2

## 疎通
[https://minikube.sigs.k8s.io/docs/tutorials/audit-policy/:title]

### AuditPolicyの作成
```bash
minikube stop
mkdir -p ~/.minikube/files/etc/ssl/certs
cat <<EOF > ~/.minikube/files/etc/ssl/certs/audit-policy.yaml
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
EOF
```

### minikubeの起動
```
minikube start \
  --extra-config=apiserver.audit-policy-file=/etc/ssl/certs/audit-policy.yaml \
  --extra-config=apiserver.audit-log-path=-
```

### Audit logの確認
```bash
kubectl logs kube-apiserver-minikube -n kube-system | grep audit.k8s.io/v1
```
出力の例

```json
{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "e8b44217-4018-4cac-b25a-978349d428e0",
  "stage": "RequestReceived",
  "requestURI": "/apis/batch/v1beta1/cronjobs?limit=500",
  "verb": "list",
  "user": {
    "username": "system:serviceaccount:kube-system:cronjob-controller",
    "uid": "5eb6b4b2-a6e3-4551-800f-1d1aeb638e0f",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:kube-system",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "192.168.99.100"
  ],
  "userAgent": "kube-controller-manager/v1.18.0 (linux/amd64) kubernetes/9e99141/system:serviceaccount:kube-system:cronjob-controller",
  "objectRef": {
    "resource": "cronjobs",
    "apiGroup": "batch",
    "apiVersion": "v1beta1"
  },
  "requestReceivedTimestamp": "2020-04-22T01:56:52.416692Z",
  "stageTimestamp": "2020-04-22T01:56:52.416692Z"
}
```

grepの条件をところを`minikube-user`に変えてみる。

```json
{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "105f9973-cbca-49ed-9d6b-41e232225738",
  "stage": "ResponseStarted",
  "requestURI": "/api/v1/namespaces/kube-system/pods/kube-apiserver-minikube/log?follow=true",
  "verb": "get",
  "user": {
    "username": "minikube-user",
    "groups": [
      "system:masters",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "192.168.99.1"
  ],
  "userAgent": "kubectl/v1.15.3 (linux/amd64) kubernetes/2d3c76f",
  "objectRef": {
    "resource": "pods",
    "namespace": "kube-system",
    "name": "kube-apiserver-minikube",
    "apiVersion": "v1",
    "subresource": "log"
  },
  "responseStatus": {
    "metadata": {},
    "code": 200
  },
  "requestReceivedTimestamp": "2020-04-22T02:00:11.412922Z",
  "stageTimestamp": "2020-04-22T02:00:11.424775Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": ""
  }
}
```

`kubectl logs`を打っているので、その情報が出力されていることがわかる。
出力する情報を増やしたいので、`minikube-user`のみ`Request`レベルのAudit Logを出力するようaudit-policy.yamlを修正し、Minikubeを再起動。

```yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Request
  users: ["minikube-user"]
```

試しに`kubectl create`してみる

```
kubectl create deployment --image=nginx nginx-app
```

上記コマンドを実行すると、Audit logは下記のようになり、

```json
{
    "kind": "Event",
    "apiVersion": "audit.k8s.io/v1",
    "level": "Request",
    "auditID": "2e935f75-6162-479e-a4b6-1bd311a4ff86",
    "stage": "ResponseComplete",
    "requestURI": "/apis/apps/v1/namespaces/default/deployments",
    "verb": "create",
    "user": {
        "username": "minikube-user",
        "groups": [
            "system:masters",
            "system:authenticated"
        ]
    },
    "sourceIPs": [
        "192.168.99.1"
    ],
    "userAgent": "kubectl/v1.15.3 (linux/amd64) kubernetes/2d3c76f",
    "objectRef": {
        "resource": "deployments",
        "namespace": "default",
        "name": "nginx-app",
        "apiGroup": "apps",
        "apiVersion": "v1"
    },
    "responseStatus": {
        "metadata": {},
        "code": 201
    },
    "requestObject": {
        "kind": "Deployment",
        "apiVersion": "apps/v1",
        "metadata": {
            "name": "nginx-app",
            "creationTimestamp": null,
            "labels": {
                "app": "nginx-app"
            }
        },
        "spec": {
            "replicas": 1,
            "selector": {
                "matchLabels": {
                    "app": "nginx-app"
                }
            },
            "template": {
                "metadata": {
                    "creationTimestamp": null,
                    "labels": {
                        "app": "nginx-app"
                    }
                },
                "spec": {
                    "containers": [
                        {
                            "name": "nginx",
                            "image": "nginx",
                            "resources": {},
                            "terminationMessagePath": "/dev/termination-log",
                            "terminationMessagePolicy": "File",
                            "imagePullPolicy": "Always"
                        }
                    ],
                    "restartPolicy": "Always",
                    "terminationGracePeriodSeconds": 30,
                    "dnsPolicy": "ClusterFirst",
                    "securityContext": {},
                    "schedulerName": "default-scheduler"
                }
            },
            "strategy": {
                "type": "RollingUpdate",
                "rollingUpdate": {
                    "maxUnavailable": "25%",
                    "maxSurge": "25%"
                }
            },
            "revisionHistoryLimit": 10,
            "progressDeadlineSeconds": 600
        },
        "status": {}
    },
    "requestReceivedTimestamp": "2020-04-22T02:05:52.184886Z",
    "stageTimestamp": "2020-04-22T02:05:52.203978Z",
    "annotations": {
        "authorization.k8s.io/decision": "allow",
        "authorization.k8s.io/reason": ""
    }
}
```

- `.user.username`（`minikube-user`）が
- `.requestObject`（`Deployment`)を`.verb`（`create`）した

ことがわかる。  

上記ではAudit Levelを`Request`にしたが、他にもあるのでいろいろ試してみると良いと思う。

| Level | 内容 |
| --- | --- |
| None | don’t log events that match this rule. |
| Metadata | log request metadata (requesting user, timestamp, resource, verb, etc.) but not request or response body. |
| Request | log event metadata and request body but not response body. This does not apply for non-resource requests. |
| RequestResponse | log event metadata, request and response bodies. This does not apply for non-resource requests. |