【AKS Build編】 Confluent Platform を Azure Kubernetes Service (AKS) で動かしてみました


## 概要

[この記事](https://qiita.com/turupon/items/e637ded0a42806b4e901) から始まる３つのステップで稼働させた一連のコンテナアプリを定義した docker-compose.yml を kcompose convert することにより、Kuberbetesのマニフェストを作成し、それを元にAKSで同様に稼働できることを確認します。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/544022/f8734468-cf61-9f10-6859-74c202d1c558.png)

以下の２つのステップで上記内容を順次説明します。
[STEP-1．コンテナイメージを ACR にプッシュする](https://qiita.com/turupon/items/a41f0e8fc63e4152f1e5)
[STEP-2．AKSでコンテナアプリを稼働させる](https://qiita.com/turupon/items/ea380280c1f848b08733)

今回の STEP-2 では、対象の docker-compose.yml を kcompose convert で作成される Kuberbetesのマニフェストファイルを編集し、Azure Container Registry（ACR）に登録されたイメージを使用して、AKS with Azure CNI 上でコンテの稼働までの手順を説明します。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/544022/f6aedd96-cb17-a986-5ceb-80f947ee8286.png)

## ローカル環境

macOS Big Sur 11.3 
python 3.8.3

## 事前準備

[この記事](https://qiita.com/turupon/items/4220f0845ccd3549d355) が完了し、AKS with Azure CNI で AKS Cluster 環境があること。

---

## Kuberbetesのマニフェストファイルへのコンバート

### Kompose のインストール

```
$ curl -L https://github.com/kubernetes/kompose/releases/download/v1.22.0/kompose-darwin-amd64 -o kompose
$ chmod +x kompose
$ sudo mv ./kompose /usr/local/bin/kompose
```

### docker-compose.yml の編集

以下の docker-compose.yml を対象とします。

- 編集箇所は以下：
  - コンバート時のエラー回避のため 'depends_on' のところはコメントアウト
  - image 定義は acr0ituru.azurecr.io を追加して定義（ただし、control-center / schema-registry を除く）
  - ksqldb-cli はサービスでないのでコメントアウトしておく
  - rabbitmq / grafana は外部からの接続があるので LoadBalancer の定義を追加

```docker-compose.yml
---
version: '2'
services:
  zookeeper:
    image: acr0ituru.azurecr.io/cp-zookeeper:6.0.0　　　　　<-- 変更
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: acr0ituru.azurecr.io/cp-server:6.0.0　　　　　<-- 変更
    hostname: broker
    container_name: broker
    # depends_on:
    #   - zookeeper
    ports:
      - "29092:29092"
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
      KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'

  schema-registry:
    image: confluentinc/cp-schema-registry:6.0.0
    # image: acr0ituru.azurecr.io/cp-schema-registry:6.0.0
    hostname: schema-registry
    container_name: schema-registry
    # depends_on:
    #   - broker
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'broker:29092'
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

  connect:
    image: acr0ituru.azurecr.io/cp-connect-base:6.0.0　　　　　<-- 変更
    hostname: connect
    container_name: connect
    # depends_on:
    #   - broker
    #   - schema-registry
    ports:
      - "8083:8083"
    environment:
      CONNECT_BOOTSTRAP_SERVERS: 'broker:29092'
      CONNECT_REST_ADVERTISED_HOST_NAME: connect
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: compose-connect-group
      CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
      CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CLASSPATH: /usr/share/java/monitoring-interceptors/monitoring-interceptors-6.0.0.jar
      CONNECT_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor"
      CONNECT_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"
      CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components"
      CONNECT_LOG4J_LOGGERS: org.apache.zookeeper=ERROR,org.I0Itec.zkclient=ERROR,org.org.apache.kafka.connect.runtime.rest=WARN,reflections=ERROR
      
  control-center:
    image: confluentinc/cp-enterprise-control-center:6.0.0
    hostname: control-center
    container_name: control-center
    # depends_on:
    #   - broker
    #   - schema-registry
    #   - connect
    #   - ksqldb-server
    ports:
      - "9021:9021"
    environment:
      CONTROL_CENTER_BOOTSTRAP_SERVERS: 'broker:29092'
      CONTROL_CENTER_CONNECT_CLUSTER: 'connect:8083'
      CONTROL_CENTER_KSQL_KSQLDB1_URL: "http://ksqldb-server:8088"
      CONTROL_CENTER_KSQL_KSQLDB1_ADVERTISED_URL: "http://localhost:8088"
      CONTROL_CENTER_REPLICATION_FACTOR: 1
      CONTROL_CENTER_INTERNAL_TOPICS_PARTITIONS: 1
      CONTROL_CENTER_MONITORING_INTERCEPTOR_TOPIC_PARTITIONS: 1
      CONFLUENT_METRICS_TOPIC_REPLICATION: 1
      PORT: 9021

  ksqldb-server:
    image: acr0ituru.azurecr.io/cp-ksqldb-server:6.0.0　　　　　 <-- 変更
    hostname: ksqldb-server
    container_name: ksqldb-server
    # depends_on:
    #   - broker
    #   - connect
    ports:
      - "8088:8088"
    environment:
      KSQL_CONFIG_DIR: "/etc/ksql"
      KSQL_BOOTSTRAP_SERVERS: "broker:29092"
      KSQL_HOST_NAME: ksqldb-server
      KSQL_LISTENERS: "http://0.0.0.0:8088"
      KSQL_CACHE_MAX_BYTES_BUFFERING: 0
      KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      KSQL_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor"
      KSQL_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"
      KSQL_KSQL_CONNECT_URL: "http://connect:8083"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_REPLICATION_FACTOR: 1
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: 'true'
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: 'true'

  # ksqldb-cli:
  #   image: acr0ituru.azurecr.io/cp-ksqldb-cli:6.0.0
  #   container_name: ksqldb-cli
  #   # depends_on:
  #   #   - broker
  #   #   - connect
  #   #   - ksqldb-server
  #   entrypoint: /bin/sh
  #   tty: true

  rabbitmq:
    image: acr0ituru.azurecr.io/rabbitmq:3.8.17　　　　　<-- 変更
    restart: always
    ports:
      - '5672:5672'
      - '15672:15672'
    hostname: rabbitmq
    container_name: rabbitmq
    environment:
      RABBITMQ_DEFAULT_USER: guest
      RABBITMQ_DEFAULT_PASS: guest
    labels:                             　　　　　      <-- 追加
      kompose.service.type: LoadBalancer　　　　　      <-- 追加

  influxdb:
    image: acr0ituru.azurecr.io/influxdb:1.8.6　　　　　<-- 変更
    ports:
      - 8086:8086
    hostname: influxdb
    container_name: influxdb
  
  grafana:
    image: acr0ituru.azurecr.io/grafana:8.0.6　　　　　 <-- 変更
    ports:
      - 3000:3000
    hostname: grafana
    container_name: grafana
    environment:
      - GF_SERVER_ROOT_URL=http://grafana:3000
      - GF_INSTALL_PLUGINS=grafana-polystat-panel,bessler-pictureit-panel,marcuscalidus-svg-panel
      - GF_SECURITY_ADMIN_PASSWORD=admin
    # depends_on:
    #   - influxdb
    labels:                             　　　　　      <-- 追加
      kompose.service.type: LoadBalancer　　　　　      <-- 追加
```

### kubernetesマニフェストの作成（コンバートの実行）

```
$ kompose convert -f docker-compose.yml   

INFO Kubernetes file "broker-service.yaml" created 
INFO Kubernetes file "connect-service.yaml" created 
INFO Kubernetes file "control-center-service.yaml" created 
INFO Kubernetes file "grafana-service.yaml" created 
INFO Kubernetes file "influxdb-service.yaml" created 
INFO Kubernetes file "ksqldb-server-service.yaml" created 
INFO Kubernetes file "rabbitmq-service.yaml" created 
INFO Kubernetes file "schema-registry-service.yaml" created 
INFO Kubernetes file "zookeeper-service.yaml" created 
INFO Kubernetes file "broker-deployment.yaml" created 
INFO Kubernetes file "connect-deployment.yaml" created 
INFO Kubernetes file "control-center-deployment.yaml" created 
INFO Kubernetes file "grafana-deployment.yaml" created 
INFO Kubernetes file "influxdb-deployment.yaml" created 
INFO Kubernetes file "ksqldb-server-deployment.yaml" created 
INFO Kubernetes file "rabbitmq-deployment.yaml" created 
INFO Kubernetes file "schema-registry-deployment.yaml" created 
INFO Kubernetes file "zookeeper-deployment.yaml" created 
```

## マニフェスト編集

### schema-registry のマニフェストの編集

既知問題の回避方法を追加する（command: の部分）

```schema-registry-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
     ：
    中略
     ：
spec:
  replicas: 1
     ：
    中略
     ：
    spec:
      containers:
        - command:                                                    <-- 追加
            - bash                                                    <-- 追加
            - -c                                                      <-- 追加
            - unset SCHEMA_REGISTRY_PORT; /etc/confluent/docker/run   <-- 追加
          env:
             ：
            省略
             ：
```

### rabbitmq / grafana のマニフェストの編集

インターナルロードバランサの定義を追加する

```rabbitmq-service.yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
      ：
      中略
      ：
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"    <-- 追加
  creationTimestamp: null
  labels:
    io.kompose.service: rabbitmq
  name: rabbitmq
       ：
      省略
       ：
```

```grafana-service.yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
      ：
      中略
      ：
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"    <-- 追加
  creationTimestamp: null
  labels:
    io.kompose.service: grafana
  name: grafana
       ：
      省略
       ：
```

---

## AKS 上での Pod の起動

AKS with Azure CNI 構成上に必要なPodを起動させます。 
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/544022/2c861cb7-e27b-1179-8ed2-a645f70f8ec1.png)

### クラスター上で動く Node の確認

```
$ kubectl get node -o wide

aks-nodepool1-83327242-vmss000000   Ready    agent   29h   v1.20.9   10.0.1.4      <none>        Ubuntu 18.04.5 LTS   5.4.0-1055-azure   containerd://1.4.8+azure
aks-nodepool1-83327242-vmss000001   Ready    agent   29h   v1.20.9   10.0.1.35     <none>        Ubuntu 18.04.5 LTS   5.4.0-1055-azure   containerd://1.4.8+azure
aks-nodepool1-83327242-vmss000002   Ready    agent   29h   v1.20.9   10.0.1.66     <none>        Ubuntu 18.04.5 LTS   5.4.0-1055-azure   containerd://1.4.8+azure
```

### namespace の定義

```
$ kubectl create namespace akscp02
```

### Pod の作成

```
kubectl apply -f rabbitmq-service.yaml,rabbitmq-deployment.yaml -n akscp02
kubectl apply -f influxdb-service.yaml,influxdb-deployment.yaml -n akscp02
kubectl apply -f grafana-service.yaml,grafana-deployment.yaml -n akscp02
kubectl apply -f zookeeper-service.yaml,zookeeper-deployment.yaml -n akscp02
kubectl apply -f broker-service.yaml,broker-deployment.yaml -n akscp02
kubectl apply -f schema-registry-service.yaml,schema-registry-deployment.yaml -n akscp02
kubectl apply -f connect-service.yaml,connect-deployment.yaml -n akscp02
kubectl apply -f ksqldb-server-service.yaml,ksqldb-server-deployment.yaml -n akscp02
kubectl apply -f control-center-service.yaml,control-center-deployment.yaml -n akscp02
```

### Pod の確認

```
$ kubectl get pods -o wide

NAME                               READY   STATUS    RESTARTS   AGE     IP          NODE                                NOMINATED NODE   READINESS GATES
broker-84b88df749-4gnwc            1/1     Running   0          3m28s   10.0.1.45   aks-nodepool1-83327242-vmss000001   <none>           <none>
connect-6457f5587c-9qj5w           1/1     Running   0          3m27s   10.0.1.11   aks-nodepool1-83327242-vmss000000   <none>           <none>
control-center-7f6fdd6957-kvk5p    1/1     Running   0          3m27s   10.0.1.74   aks-nodepool1-83327242-vmss000002   <none>           <none>
grafana-54d5cf546c-kmx9k           1/1     Running   0          3m29s   10.0.1.12   aks-nodepool1-83327242-vmss000000   <none>           <none>
influxdb-79cb78575-vhhmn           1/1     Running   0          3m29s   10.0.1.77   aks-nodepool1-83327242-vmss000002   <none>           <none>
ksqldb-server-5df9df4f54-dtv27     1/1     Running   0          3m27s   10.0.1.55   aks-nodepool1-83327242-vmss000001   <none>           <none>
rabbitmq-85f7fddfb-z5sxj           1/1     Running   0          3m30s   10.0.1.37   aks-nodepool1-83327242-vmss000001   <none>           <none>
schema-registry-65c9fbbc4b-rmmks   1/1     Running   0          3m28s   10.0.1.93   aks-nodepool1-83327242-vmss000002   <none>           <none>
zookeeper-69655fbfcf-p5hgw         1/1     Running   0          3m29s   10.0.1.38   aks-nodepool1-83327242-vmss000001   <none>           <none>
```

### Service の確認

```
$ kubectl get svc -n akscp02 -o wide

broker            ClusterIP      10.1.0.60    <none>        29092/TCP,9092/TCP,9101/TCP      3m34s   io.kompose.service=broker
connect           ClusterIP      10.1.0.106   <none>        8083/TCP                         3m33s   io.kompose.service=connect
control-center    ClusterIP      10.1.0.123   <none>        9021/TCP                         3m33s   io.kompose.service=control-center
grafana           LoadBalancer   10.1.0.119   10.0.1.98     3000:32300/TCP                   3m35s   io.kompose.service=grafana
influxdb          ClusterIP      10.1.0.61    <none>        8086/TCP                         3m35s   io.kompose.service=influxdb
ksqldb-server     ClusterIP      10.1.0.83    <none>        8088/TCP                         3m33s   io.kompose.service=ksqldb-server
rabbitmq          LoadBalancer   10.1.0.157   10.0.1.97     5672:31096/TCP,15672:31606/TCP   3m36s   io.kompose.service=rabbitmq
schema-registry   ClusterIP      10.1.0.175   <none>        8081/TCP                         3m34s   io.kompose.service=schema-registry
zookeeper         ClusterIP      10.1.0.112   <none>        2181/TCP                         3m35s   io.kompose.service=zookeeper
```


---

## コンテナの詳細確認

```
## Service の詳細確認（例：grafana）
$ kubectl describe svc grafana -n akscp02

## Pod の詳細確認（例：grafana）
$ kubectl describe pod grafana -n akscp02 
```

---

## Podへの接続確認

AKS と同一サブネット上にデータ生成用兼可視化用の仮想マシンを稼働させます。 
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/544022/5e7d7063-2df5-3581-eef2-d897dc418de2.png)

### 同一サブネット上にある仮想マシンへの接続

[この記事](https://qiita.com/turupon/items/208acb8fdbe9dadb5a70) のように Windows10 pro の仮想マシンを同一サブネット上に作成し、ローカル端末から RDP接続します。

### 仮想マシンからの確認

コンソールを立ち上げ、ネットワーク情報確認後、curl で grafana(IP:10.0.1.98  port:3000)への接続を確認します。

```
## ネットワーク情報
C:\Users\nmcadmin> ipconfig
Windows IP Configuration
Ethernet adapter Ethernet 2:
   Connection-specific DNS Suffix  . : hogehogehogehogehoge.lx.internal.cloudapp.net
   Link-local IPv6 Address . . . . . : ff88::a55a:e4e4:9363:4a4a%7
   IPv4 Address. . . . . . . . . . . : 10.0.1.99
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 10.0.1.1

## grafanaへの接続確認
C:\Users\nmcadmin> curl http://10.0.1.98:3000
<a href="/login">Found</a>.
```

---

## まとめ

docker-compose.yml を kompose convert することにより deployment と Service のマニフェストが作成され、それらを AKS with Azure CNI 環境 に kubectl apply することにより問題なくコンテナが稼働することを確認しました。

---

## 付録

### Podのログの確認

```
kubectl logs schema-registry-65c9fbbc4b-rmmks -n akscp02
```

### Podの削除

```
kubectl delete -f control-center-service.yaml,control-center-deployment.yaml -n akscp02
kubectl delete -f ksqldb-server-service.yaml,ksqldb-server-deployment.yaml -n akscp02
kubectl delete -f connect-service.yaml,connect-deployment.yaml -n akscp02
kubectl delete -f schema-registry-service.yaml,schema-registry-deployment.yaml -n akscp02
kubectl delete -f broker-service.yaml,broker-deployment.yaml -n akscp02
kubectl delete -f zookeeper-service.yaml,zookeeper-deployment.yaml -n akscp02
kubectl delete -f grafana-service.yaml,grafana-deployment.yaml -n akscp02
kubectl delete -f influxdb-service.yaml,influxdb-deployment.yaml -n akscp02
kubectl delete -f rabbitmq-service.yaml,rabbitmq-deployment.yaml -n akscp02
```

### AKS Cluster の停止・開始・削除

```
az aks stop --name $AKS_CLUSTER_NAME -g $AKS_RES_GROUP
az aks start --name $AKS_CLUSTER_NAME -g $AKS_RES_GROUP
az aks delete --name $AKS_CLUSTER_NAME -g $AKS_RES_GROUP
```

### AKS リソースグループ の削除

```
az group delete --name $AKS_RES_GROUP --yes --no-wait
```