Project: APISIX (Orchestration & Management / API Gateway)
File: kubernetes.md

title: Kubernetes keywords: Kubernetes Apache APISIX Service discovery Cluster API Gateway description: This article introduce how to perform service discovery based on Kubernetes in Apache APISIX and summarize related issues. <!-- --> The service discovery real-time changes of resources, then store theirs value into `ngx.shared.DICT`. Discovery also provides a node query interface in accordance with the . Kubernetes service discovery both support single-cluster and multi-cluster modes, applicable to the case where the service is distributed in single or multiple Kubernetes clusters. A detailed configuration for single-cluster mode Kubernetes service discovery is as follows: ```yaml discovery: kubernetes: service: schema: https #default https host: ${KUBERNETESSERVICEHOST} #default ${KUBERNETESSERVICEHOST} port: ${KUBERNETESSERVICEPORT} #default ${KUBERNETESSERVICEPORT} client: token_file: /var/run/secrets/kubernetes.io/serviceaccount/token default_weight: 50 # weight assigned to each discovered endpoint. default 50, minimum 0 namespace_selector: equal: default label_selector: |- first="a",second="b" shared_size: 1m #default 1m watchendpointslices: false #default false ``` If the Kubernetes service discovery runs inside a pod, you can use minimal configuration: ```yaml discovery: kubernetes: { } ``` If the Kubernetes service discovery runs outside a pod, you need to create or select a specified , then get its token value, and use following configuration: ```yaml discovery: kubernetes: service: schema: https host: # enter apiserver host value here port: # enter apiserver port value here client: token: # enter serviceaccount token value here ``` The Kubernetes service discovery provides a query interface in accordance with the . function: nodes(service_name) description: nodes() function attempts to look up the ngx.shared.DICT for nodes corresponding to service_name, \ servicename should match pattern: [namespace]/[name]:[portName]_ namespace: The namespace where the Kubernetes endpoints is located name: The name of the Kubernetes endpoints portName: The `ports.name` value in the Kubernetes endpoints, if there is no `ports.name`, use `targetPort`, `port` instead. If `ports.name` exists, then port number cannot be used. return value: if the Kubernetes endpoints value is as follows: ```yaml apiVersion: v1 kind: Endpoints metadata: name: plat-dev namespace: default subsets: addresses: ip: "10.5.10.109" ip: "10.5.10.110" ports: port: 3306 name: port ``` a nodes("default/plat-dev:port") call will get follow result: ``` { { host="10.5.10.109", port= 3306, weight= 50, }, { host="10.5.10.110", port= 3306, weight= 50, }, } ``` A detailed configuration for multi-cluster mode Kubernetes service discovery is as follows: ```yaml discovery: kubernetes: id: release # a custom name refer to the cluster, pattern ^[a-z0-9]{1,8} service: schema: https #default https host: "1.cluster.com" port: "6443" client: token_file: /var/run/secrets/kubernetes.io/serviceaccount/token default_weight: 50 # weight assigned to each discovered endpoint. default 50, minimum 0 namespace_selector: equal: default label_selector: |- first="a",second="b" shared_size: 1m #default 1m watchendpointslices: false #default false ``` Multi-Kubernetes service discovery does not fill default values for service and client fields, you need to fill them according to the cluster configuration. The Kubernetes service discovery provides a query interface in accordance with the . function: nodes(service_name) description: nodes() function attempts to look up the

for nodes corresponding to service_name, \ servicename should match pattern: [id]/[namespace]/[name]:[portName]_ id: value defined in service discovery configuration namespace: The namespace where the Kubernetes endpoints is located name: The name of the Kubernetes endpoints portName: The `ports.name` value in the Kubernetes endpoints, if there is no `ports.name`, use `targetPort`, `port` instead. If `ports.name` exists, then port number cannot be used. return value: if the Kubernetes endpoints value is as follows: ```yaml apiVersion: v1 kind: Endpoints metadata: name: plat-dev namespace: default subsets: addresses: ip: "10.5.10.109" ip: "10.5.10.110" ports: port: 3306 name: port ``` a nodes("release/default/plat-dev:port") call will get follow result: ``` { { host="10.5.10.109", port= 3306, weight= 50, }, { host="10.5.10.110", port= 3306, weight= 50, }, } ``` Q: Why only support configuration token to access _Kubernetes APIServer_? A: Usually, we will use three ways to complete the authentication of Kubernetes APIServer: mTLS Token Basic authentication Because lua-resty-http does not currently support mTLS, and basic authentication is not recommended, so currently only the token authentication method is implemented. Q: APISIX inherits Nginx's multiple process model, does it mean that each nginx worker process will kubernetes endpoints resources? A: The Kubernetes service discovery only uses privileged processes to Kubernetes endpoints resources, then store theirs value into `ngx.shared.DICT`, worker processes get results by querying `ngx.shared.DICT`. Q: What permissions do require? A: ServiceAccount requires the permissions of cluster-level [ get, list, watch ] endpoints resources, the declarative definition is as follows: ```yaml kind: ServiceAccount apiVersion: v1 metadata: name: apisix-test namespace: default kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: apisix-test rules: apiGroups: [ "" ] resources: [ endpoints,endpointslices ] verbs: [ get,list,watch ] apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: apisix-test roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: apisix-test subjects: kind: ServiceAccount name: apisix-test namespace: default ``` Q: How to get token value? A: Assume your located in namespace apisix and name is Kubernetes-discovery, you can use the following steps to get token value. Get secret name. You can execute the following command, the output of the first column is the secret name we want: ```shell kubectl -n apisix get secrets | grep kubernetes-discovery ``` Get token value. Assume secret resources name is kubernetes-discovery-token-c64cv, you can execute the following command, the output is the service account token value we want: ```shell kubectl -n apisix get secret kubernetes-discovery-token-c64cv -o jsonpath={.data.token} | base64 -d ``` It also offers control api for debugging. To query/list the nodes discoverd by kubernetes discovery, you can query the /v1/discovery/kubernetes/dump control API endpoint like so: ```shell GET /v1/discovery/kubernetes/dump ``` Which will yield the following response: ``` { "endpoints": [ { "endpoints": [ { "value": "{\"https\":[{\"host\":\"172.18.164.170\",\"port\":6443,\"weight\":50},{\"host\":\"172.18.164.171\",\"port\":6443,\"weight\":50},{\"host\":\"172.18.164.172\",\"port\":6443,\"weight\":50}]}", "name": "default/kubernetes" }, { "value": "{\"metrics\":[{\"host\":\"172.18.164.170\",\"port\":2379,\"weight\":50},{\"host\":\"172.18.164.171\",\"port\":2379,\"weight\":50},{\"host\":\"172.18.164.172\",\"port\":2379,\"weight\":50}]}", "name": "kube-system/etcd" }, { "value": "{\"http-85\":[{\"host\":\"172.64.89.2\",\"port\":85,\"weight\":50}]}", "name": "test-ws/testing" } ], "id": "first" } ], "config": [ { "default_weight": 50, "id": "first", "client": { "token": "xxx" }, "service": { "host": "172.18.164.170", "port": "6443", "schema": "https" }, "shared_size": "1m" } ] } ```