Kubernetes HPAの Alpha Feature HPAScaleToZeroをローカルで試す

# 背景

HPAではゼロにScaleIn出来ない。

```bash
kubectl explain HorizontalPodAutoscaler.spec.minReplicas
KIND:     HorizontalPodAutoscaler
VERSION:  autoscaling/v1

FIELD:    minReplicas <integer>

DESCRIPTION:
     minReplicas is the lower limit for the number of replicas to which the
     autoscaler can scale down. It defaults to 1 pod. minReplicas is allowed to
     be 0 if the alpha feature gate HPAScaleToZero is enabled and at least one
     Object or External metric is configured. Scaling is active as long as at
     least one metric value is available.
```

> minReplicas is allowed to
be 0 if the alpha feature gate HPAScaleToZero is enabled and at least one
Object or External metric is configured.

- alpha feature gateを enableする必要がある
- Object or External metric少なくとも1つは必要

[KubernetesのHPA with custom metricsをRabbitMQとPrometheusで試す](https://qiita.com/gymnstcs/items/ae375899d254b7b36c80) で紹介したHPA with custom metricsのようなケースだと、Queueのメッセージがゼロのときは完全にスケールインしたい場合もある。今回は、HPAで0にまでスケールインを試す。

# Feature Gateとは

[feature-gates](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/): 各Kubernetes FeatureのStageとDefaultでEnableされているか、どのバージョンから(どのバージョンまで)使用可能かがテーブルになっている。

Enableするには、 `-feature-gates="...,DynamicKubeletConfig=true"` 起動時のコマンドにEnableするFeatureペア (Feature名とtrue/false) をリストする必要がある

今回使いたい`HPAScaleToZero`を見てみると、 `HPAScaleToZero	false	Alpha	1.16` となっており、使用するためには明示的にenableする必要がある。


# ローカルで`HPAScaleToZero` をEnableにしたKubernetesクラスタを作成

今回はローカル挙動の確認をしたいのでローカルで指定が簡単にできそうな [kind](https://kind.sigs.k8s.io/) を使用

## Prerequisite

- go (1.11+)
- Docker

## kindのインストール

Macの場合は、brewを使ってインストールできる (詳細は [https://kind.sigs.k8s.io/docs/user/quick-start/#installation](https://kind.sigs.k8s.io/docs/user/quick-start/#installation))

```yaml
brew install kind
```

## kindのクラスタ用のYamlを準備

```yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  # any feature gate can be enabled here with "Name": true
  # or disabled here with "Name": false
  # not all feature gates are tested, however
  "HPAScaleToZero": true
```

## kindでKubernertes Clusterを作成

```yaml
kind create cluster --config cluster-with-alpha-feature.yaml
```

```yaml
kind create cluster --config cluster-with-alpha-feature.yaml
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.20.2) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Thanks for using kind! 😊
```

# HPAのための準備

[前回](https://qiita.com/gymnstcs/items/ae375899d254b7b36c80)と同じものをDeploy

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/44ec751e-e6ca-9679-f52b-6dcd5263a4e7.png)

（今回はGrafanaなどは省略）

1. Code をclone

    ```bash
    git clone https://github.com/nakamasato/kubernetes-training && cd kubernetes-training/autoscaler/hpa/custom-metrics
    ```

2. Prometheus with operator

    ```bash
    kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml
    kubectl create ns monitoring
    kubectl apply -k ../../../prometheus-operator -n monitoring
    ```

3. RabbitMQ with operator

    ```bash
    kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml
    kubectl apply -f rabbitmq/rabbitmq-cluster.yaml
    kubectl apply -f rabbitmq/pod-monitor-rabbitmq.yaml
    ```

4. RabbitMQConsumer & Producer

    ```bash
    kubectl apply -f rabbitmq-producer-cronjob.yaml
    kubectl apply -f rabbitmq-consumer-deployment.yaml
    ```

5. Prometheus Adapter をDeploy([https://github.com/stefanprodan/k8s-prom-hpa/](https://github.com/stefanprodan/k8s-prom-hpa/))

    ```bash
    git clone git@github.com:stefanprodan/k8s-prom-hpa.git && cd k8s-prom-hpa
    touch metrics-ca.key metrics-ca.crt metrics-ca-config.json
    make certs
    kubectl create -f ./custom-metrics-api
    ```

6. `rabbitmq_queue_messages_ready` が `[custom.metrics.k8s.io](http://custom.metrics.k8s.io)` から取得できるのを確認

    ```bash
    kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/rabbitmq_queue_messages_ready"| jq .
    {
      "kind": "MetricValueList",
      "apiVersion": "custom.metrics.k8s.io/v1beta1",
      "metadata": {
        "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/rabbitmq_queue_messages_ready"
      },
      "items": [
        {
          "describedObject": {
            "kind": "Pod",
            "namespace": "default",
            "name": "rabbitmq-server-0",
            "apiVersion": "/v1"
          },
          "metricName": "rabbitmq_queue_messages_ready",
          "timestamp": "2021-05-19T01:55:02Z",
          "value": "0"
        }
      ]
    }
    ```

# HPAをデプロイ (`minReplicas: 0`)

```yaml:rabbitmq-consumer-hpa-with-min-zero.yaml
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: rabbitmq-consumer
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: rabbitmq-consumer
  minReplicas: 0
  maxReplicas: 20
  metrics:
    - type: Object
      object:
        metric:
          name: rabbitmq_queue_messages_ready
        describedObject:
          kind: Pod
          name: rabbitmq-server-0
          apiVersion: v1
        target:
          type: Value
          averageValue: 1
```

```bash
kubectl apply -f rabbitmq-consumer-hpa-with-min-zero.yaml
```

確認 `New size: 0` となってることを確認!

```bash
Events:
  Type    Reason             Age                  From                       Message
  ----    ------             ----                 ----                       -------
  Normal  SuccessfulRescale  35m                  horizontal-pod-autoscaler  New size: 7; reason: All metrics below target
  Normal  SuccessfulRescale  30m (x2 over 40m)    horizontal-pod-autoscaler  New size: 16; reason: All metrics below target
  Normal  SuccessfulRescale  30m (x2 over 39m)    horizontal-pod-autoscaler  New size: 18; reason: All metrics below target
  Normal  SuccessfulRescale  25m (x2 over 35m)    horizontal-pod-autoscaler  New size: 14; reason: All metrics below target
  Normal  SuccessfulRescale  25m                  horizontal-pod-autoscaler  New size: 4; reason: All metrics below target
  Normal  SuccessfulRescale  20m                  horizontal-pod-autoscaler  New size: 1; reason: All metrics below target
  Normal  SuccessfulRescale  10m (x4 over 40m)    horizontal-pod-autoscaler  New size: 4; reason: external metric rabbitmq_queue_messages_ready(nil) above target
  Normal  SuccessfulRescale  5m13s (x4 over 35m)  horizontal-pod-autoscaler  New size: 2; reason: All metrics below target
  Normal  SuccessfulRescale  4m58s                horizontal-pod-autoscaler  New size: 0; reason: All metrics below target
  Normal  SuccessfulRescale  37s (x9 over 20m)    horizontal-pod-autoscaler  (combined from similar events): New size: 4; reason: external metric rabbitmq_queue_messages_ready(nil) above target
  Normal  SuccessfulRescale  22s (x4 over 40m)    horizontal-pod-autoscaler  New size: 8; reason: external metric rabbitmq_queue_messages_ready(nil) above target
  Normal  SuccessfulRescale  6s                   horizontal-pod-autoscaler  New size: 16; reason: external metric rabbitmq_queue_messages_ready(nil) above targe
```

# 片付け

1. Kubernetesリソースの片付け

    ```
    kubectl delete -f rabbitmq-consumer-hpa-with-min-zero.yaml
    kubectl delete -f rabbitmq-producer-cronjob.yaml
    kubectl delete -f rabbitmq-consumer-deployment.yaml
    kubectl delete -f https://k8s.io/examples/application/php-apache.yaml
    kubectl delete -f rabbitmq/rabbitmq-cluster.yaml
    kubectl delete -k ../../../prometheus-operator -n monitoring
    kubectl delete -f ./k8s-prom-hpa/custom-metrics-api                                                           
    kubectl delete -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml
    kubectl delete -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml
    ```

1. kindクラスタの削除

    ```
    kind delete cluster --name kind
    ```

# まとめ

Alpha Featureの `HPAScaleToZero` をローカルKubernetesクラスタkindでEnableして、Custom MetricsによるHPAで `minReplicas: 0` の動作を確認できた。

# 参考リンク

- https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/
- https://github.com/nakamasato/kubernetes-training/tree/master/autoscaler/hpa/custom-metrics
- https://qiita.com/gymnstcs/items/ae375899d254b7b36c80
- https://kind.sigs.k8s.io/docs/user/quick-start/#installation