KubernetesのHPA with custom metricsをRabbitMQとPrometheusで試す

# やりたいこと

Queueにメッセージが溜まったときに、ConsumerのPodの数を増やして処理速度をあげ、Queueにメッセージがなくなったときに、Podの数を減らしてリソースを開放したい。

# 考慮したこと

- HPAでCPUやメモリの使用量でスケール方法もあるが、直接的に待ってるQueueの数を元にスケールしてみたい
- HPA with custom metricsの練習
- VPAは、PodがRestartしてしまうので、長い処理をするアプリケーションの場合、Restartするときに途中までの処理が無駄になってしまう

# 材料

- [Kubernetes Horizontal Pod Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale)
- [Kubernetes custom-metrics-api](https://github.com/kubernetes/metrics/blob/master/IMPLEMENTATIONS.md#custom-metrics-api)
- [prometheus-adapter](https://github.com/kubernetes-sigs/prometheus-adapter)
- [RabbitMQ monitoring](https://www.rabbitmq.com/monitoring.html)

# 方針

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/44ec751e-e6ca-9679-f52b-6dcd5263a4e7.png)


1. RabbitMQの `messages_ready` メトリクスをPrometheusで取得する
1. prometheus-adapter で custom-metricsでRabbitMQのMetricsを取得できるようにする
1. HPAを設定する


# Code

[https://github.com/nakamasato/kubernetes-training/tree/master/autoscaler/hpa/custom-metrics](https://github.com/nakamasato/kubernetes-training/tree/master/autoscaler/hpa/custom-metrics)

# ステップ

## 準備

### 1. Prometheusをデプロイ

1. prometheus-operatorをインストール

    ```
    kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml
    ```

1. `monitoring` namespaceを作成

    ```
    kubectl create ns monitoring
    ```

1. Prometheusをデプロイ

    ```
    kubectl apply -k ../../../prometheus-operator -n monitoring
    ```

1. http://localhost:30900 でUIをチェック



### 2. RabbitMQをデプロイ

1. RabbitMQ operatorをインストール

    ```
    kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml
    ```

1. RabbitMQをデプロイ


    ```yaml:rabbitmq/rabbitmq-cluster.yaml
    apiVersion: rabbitmq.com/v1beta1
    kind: RabbitmqCluster
    metadata:
      name: rabbitmq
    ```

    ```
    kubectl apply -f rabbitmq/rabbitmq-cluster.yaml
    ```

1. PodMonitorをデプロイ (RabbitMQをPrometheusがScrapeできるようにする)

    ```yaml:rabbitmq/pod-monitor-rabbitmq.yaml
    apiVersion: monitoring.coreos.com/v1
    kind: PodMonitor
    metadata:
      namespace: monitoring
      name: rabbitmq
    spec:
      podMetricsEndpoints:
      - interval: 15s
        port: prometheus
        path: /metrics
      selector:
        matchLabels:
          app.kubernetes.io/component: rabbitmq
      namespaceSelector:
        any: true
    ```

    ```
    kubectl apply -f rabbitmq/pod-monitor-rabbitmq.yaml
    ```

### 3. RabbitMQ producerをデプロイ

5分に一回20個RabbitMQのメッセージを送る`CronJob`

[rabbitmq-producer](https://github.com/nakamasato/rabbitmq-producer)

<details><summary>rabbitmq-producer-cronjob.yaml</summary><div>

```yaml:rabbitmq-producer-cronjob.yaml

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: rabbitmq-producer
spec:
  schedule: '*/5 * * * *'
  successfulJobsHistoryLimit: 1
  jobTemplate:
    metadata:
      name: rabbitmq-producer
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - image: nakamasato/rabbitmq-producer
              name: rabbitmq-producer
              env:
                - name: RABBITMQ_USERNAME
                  valueFrom:
                    secretKeyRef:
                      name: rabbitmq-default-user
                      key: username
                - name: RABBITMQ_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: rabbitmq-default-user
                      key: password
                - name: RABBITMQ_HOST
                  value: rabbitmq
                - name: NUM_OF_MESSAGES
                  value: "20"
```

</div></details>

```
kubectl apply -f rabbitmq-producer-cronjob.yaml
```

### 4. RabbitMQ consumerをデプロイ

RabbitMQのメッセージを一つずつConsumeし、1つのメッセージにつき10秒で処理する。

[rabbitmq-consumer](https://github.com/nakamasato/rabbitmq-consumer)

<details><summary>rabbitmq-consumer-deployment.yaml</summary><div>


```yaml:rabbitmq-consumer-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: rabbitmq-consumer
  name: rabbitmq-consumer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rabbitmq-consumer
  template:
    metadata:
      labels:
        app: rabbitmq-consumer
    spec:
      containers:
        - image: nakamasato/rabbitmq-consumer
          name: rabbitmq-consumer
          env:
            - name: RABBITMQ_USERNAME
              valueFrom:
                secretKeyRef:
                  name: rabbitmq-default-user
                  key: username
            - name: RABBITMQ_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: rabbitmq-default-user
                  key: password
            - name: RABBITMQ_HOST
              value: rabbitmq
            - name: PROCESS_SECONDS
              value: "10"

```

</div></details>

```
kubectl apply -f rabbitmq-consumer-deployment.yaml
```

### 5. Grafana

(GrafanaはDashboardでMetricsを確認するためだけなので、必須ではない。)

<details><summary>grafana-deployment.yaml</summary><div>

```yaml:grafana-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      name: grafana
      labels:
        app: grafana
    spec:
      containers:
        - name: grafana
          image: grafana/grafana:latest
          ports:
            - name: grafana
              containerPort: 3000
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafana-storage
      volumes:
        - name: grafana-storage
          emptyDir: {}
```

</div></details>

<details><summary>grafana-service.yaml</summary><div>

```yaml:grafana-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: grafana
spec:
  selector:
    app: grafana
  type: NodePort
  ports:
    - port: 3000
      targetPort: 3000
      nodePort: 32111
```

</div></details>

1. Grafanaをデプロイ

    ```
    kubectl apply -f grafana-deployment.yaml,grafana-service.yaml
    ```
1. http://localhost:32111 に usernameとpasswordをともに`admin`でログインする
1. [RabbitMQ-Overview](https://grafana.com/grafana/dashboards/10991) Dashboard (10991)をimportする

## 本題

### 1. prometheus-adapterをデプロイ

1. prometheus-adapterをclone

    ```
    git clone git@github.com:stefanprodan/k8s-prom-hpa.git && cd k8s-prom-hpa
    ```

1. certificateを準備

    ```
    touch metrics-ca.key metrics-ca.crt metrics-ca-config.json
    make certs
    ```

1. デプロイ

    ```
    kubectl create -f ./custom-metrics-api
    ```

1. RabbitMQのMetricsがcustom metricsで取得できることを確認


    ```
    kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/rabbitmq_queue_messages_ready"| jq .
    {
      "kind": "MetricValueList",
      "apiVersion": "custom.metrics.k8s.io/v1beta1",
      "metadata": {
        "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/rabbitmq_queue_messages_ready"
      },
      "items": [
        {
          "describedObject": {
            "kind": "Pod",
            "namespace": "default",
            "name": "rabbitmq-server-0",
            "apiVersion": "/v1"
          },
          "metricName": "rabbitmq_queue_messages_ready",
          "timestamp": "2021-03-27T12:01:15Z",
          "value": "1274"
        }
      ]
    }
    ```

### 3. HorizontalPodAutoscalerをデプロイ

<details><summary>rabbitmq-consumer-hpa.yaml</summary><div>

```yaml:rabbitmq-consumer-hpa.yaml
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: rabbitmq-consumer
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: rabbitmq-consumer
  minReplicas: 1
  maxReplicas: 20
  metrics:
    - type: Object
      object:
        metric:
          name: rabbitmq_queue_messages_ready
        describedObject:
          kind: Pod
          name: rabbitmq-server-0
          apiVersion: v1
        target:
          type: Value
          averageValue: 1
```

</div></details>


```
kubectl apply -f rabbitmq-consumer-hpa.yaml
```

### 4. Podの数がRabbitMQのQueueの数によって変化しているのをGrafanaで確認

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/7ff0ab20-87fc-aee8-b4cb-ac394d648923.png)

# Todo

- `HPAScaleToZero`: 1.16で追加されたHPAでReplicaを0までできるAlphaのFeature (https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates)