Project: APISIX (Orchestration & Management / API Gateway)
File: consul_kv.md

title: consul_kv <!-- --> For users that are using and Consul KV as a service discovery, like the Weibo Mobile Team, this may be needed. Thanks to @fatman-x guy, who developed this module, called `consul_kv`, and its worker process data flow is below: Add following configuration in `conf/config.yaml` : ```yaml discovery: consul_kv: servers: "http://127.0.0.1:8500" "http://127.0.0.1:8600" token: "..." # if your consul cluster has enabled acl access control, you need to specify the token prefix: "upstreams" skip_keys: # if you need to skip special keys "upstreams/unused_api/" timeout: connect: 1000 # default 2000 ms read: 1000 # default 2000 ms wait: 60 # default 60 sec weight: 1 # default 1 fetch_interval: 5 # default 3 sec, only take effect for keepalive: false way keepalive: true # default true, use the long pull way to query consul servers default_server: # you can define default server when missing hit host: "127.0.0.1" port: 20999 metadata: fail_timeout: 1 # default 1 ms weight: 1 # default 1 max_fails: 1 # default 1 dump: # if you need, when registered nodes updated can dump into file path: "logs/consul_kv.dump" expire: 2592000 # unit sec, here is 30 day ``` And you can config it in short by default value: ```yaml discovery: consul_kv: servers: "http://127.0.0.1:8500" ``` The `keepalive` has two optional values: `true`, default and recommend value, use the long pull way to query consul servers `false`, not recommend, it would use the short pull way to query consul servers, then you can set the `fetch_interval` for fetch interval When we need reload `apisix` online, as the `consul_kv` module maybe loads data from CONSUL slower than load routes from ETCD, and would get the log at the moment before load successfully from consul: ``` httpaccessphase(): failed to set upstream: no valid upstream node ``` So, we import the `dump` function for `consul_kv` module. When reload, would load the dump file before from consul; when the registered nodes in consul been updated, would dump the upstream nodes into file automatically. The `dump` has three optional values now: `path`, the dump file save path support relative path, eg: `logs/consul_kv.dump` support absolute path, eg: `/tmp/consul_kv.bin` make sure the dump file's parent path exist make sure the `apisix` has the dump file's read-write access permission,eg: `chown www:root

`loadoninit`, default value is `true` if `true`, just try to load the data from the dump file before loading data from consul when starting, does not care the dump file exists or not if `false`, ignore loading data from the dump file Whether `true` or `false`, we don't need to prepare a dump file for apisix at anytime `expire`, unit sec, avoiding load expired dump data when load default `0`, it is unexpired forever recommend 2592000, which is 30 days(equals 3600 \ 24 \ 30) Service register Key&Value template: ``` Key: {Prefix}/{Service_Name}/{IP}:{Port} Value: {"weight": <Num>, "maxfails": <Num>, "failtimeout": <Num>} ``` The register consul key use `upstreams` as prefix by default. The http api service name called `webpages` for example, and you can also use `webpages/oneteam/hello` as service name. The api instance of node's ip and port make up new key: `<IP>:<Port>`. Now, register nodes into consul: ```shell curl \ -X PUT \ -d ' {"weight": 1, "maxfails": 2, "failtimeout": 1}' \ http://127.0.0.1:8500/v1/kv/upstreams/webpages/172.19.5.12:8000 curl \ -X PUT \ -d ' {"weight": 1, "maxfails": 2, "failtimeout": 1}' \ http://127.0.0.1:8500/v1/kv/upstreams/webpages/172.19.5.13:8000 ``` In some case, same keys exist in different consul servers. To avoid confusion, use the full consul key url path as service name in practice. Here is an example of routing a request with a URL of "/*" to a service which named "http://127.0.0.1:8500/v1/kv/upstreams/webpages/" and use consul_kv discovery client in the registry : :::note You can fetch the `admin_key` from `config.yaml` and save to an environment variable with the following command: ```bash adminkey=$(yq '.deployment.admin.adminkey[0].key' conf/config.yaml | sed 's/"//g') ``` ::: ```shell $ curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $admin_key" -X PUT -i -d ' { "uri": "/*", "upstream": { "service_name": "http://127.0.0.1:8500/v1/kv/upstreams/webpages/", "type": "roundrobin", "discoverytype": "consulkv" } }' ``` The format response as below: ```json { "node": { "value": { "priority": 0, "update_time": 1612755230, "upstream": { "discoverytype": "consulkv", "service_name": "http://127.0.0.1:8500/v1/kv/upstreams/webpages/", "hash_on": "vars", "type": "roundrobin", "pass_host": "pass" }, "id": "1", "uri": "/*", "create_time": 1612755230, "status": 1 }, "key": "/apisix/routes/1" } } ``` You could find more usage in the `apisix/t/discovery/consul_kv.t` file. Consul_kv service discovery also supports use in L4, the configuration method is similar to L7. ```shell $ curl http://127.0.0.1:9180/apisix/admin/streamroutes/1 -H "X-API-KEY: $adminkey" -X PUT -i -d ' { "remote_addr": "127.0.0.1", "upstream": { "scheme": "tcp", "service_name": "http://127.0.0.1:8500/v1/kv/upstreams/webpages/", "type": "roundrobin", "discoverytype": "consulkv" } }' ``` You could find more usage in the `apisix/t/discovery/stream/consul_kv.t` file. It also offers control api for debugging. ```shell GET /v1/discovery/consul_kv/dump ``` For example: ```shell { "config": { "fetch_interval": 3, "timeout": { "wait": 60, "connect": 6000, "read": 6000 }, "prefix": "upstreams", "weight": 1, "servers": [ "http://172.19.5.30:8500", "http://172.19.5.31:8500" ], "keepalive": true, "default_service": { "host": "172.19.5.11", "port": 8899, "metadata": { "fail_timeout": 1, "weight": 1, "max_fails": 1 } }, "skip_keys": [ "upstreams/myapi/gateway/apisix/" ] }, "services": { "http://172.19.5.31:8500/v1/kv/upstreams/webpages/": [ { "host": "127.0.0.1", "port": 30513, "weight": 1 }, { "host": "127.0.0.1", "port": 30514, "weight": 1 } ], "http://172.19.5.30:8500/v1/kv/upstreams/1614480/grpc/": [ { "host": "172.19.5.51", "port": 50051, "weight": 1 } ], "http://172.19.5.30:8500/v1/kv/upstreams/webpages/": [ { "host": "127.0.0.1", "port": 30511, "weight": 1 }, { "host": "127.0.0.1", "port": 30512, "weight": 1 } ] } } ``` It offers another control api for dump file view now. Maybe would add more api for debugging in future. ```shell GET /v1/discovery/consulkv/showdump_file ``` For example: ```shell curl http://127.0.0.1:9090/v1/discovery/consulkv/showdump_file | jq { "services": { "http://172.19.5.31:8500/v1/kv/upstreams/1614480/webpages/": [ { "host": "172.19.5.12", "port": 8000, "weight": 120 }, { "host": "172.19.5.13", "port": 8000, "weight": 120 } ] }, "expire": 0, "last_update": 1615877468 } ```