Project: APISIX (Orchestration & Management / API Gateway)
File: prometheus.md

title: prometheus keywords: Apache APISIX API Gateway Plugin Prometheus description: This document contains information about the Apache APISIX prometheus Plugin. <!-- --> The `prometheus` Plugin exports metrics in . | Name | Type | Required | Default | Description | | -- | - | -- | - | | | prefer_name | boolean | False | false | When set to `true`, prints Route/Service name instead of ID in Prometheus metric. | You can change the default export URI by configuring the `exporturi` attribute under `pluginattr` in your configuration file (`conf/config.yaml`). | Name | Type | Default | Description | | - | | - | - | | export_uri | string | "/apisix/prometheus/metrics" | URI to export the Prometheus metrics. | Here is a configuration example: ```yaml title="conf/config.yaml" plugin_attr: prometheus: export_uri: /apisix/metrics ``` For http request related metrics, you could specify extra labels, which match the APISIX variables. If you specify label for nonexist APISIX variable, the label value would be "". Currently, only below metrics are supported: http_status http_latency bandwidth Here is a configuration example: ```yaml title="conf/config.yaml" plugin_attr: prometheus: metrics: http_status: extra_labels: upstreamaddr: $upstreamaddr upstreamstatus: $upstreamstatus ``` `DEFAULTBUCKETS` is the default value for bucket array in `httplatency` metrics. You can change the `DEFAULTBUCKETS` by configuring `defaultbuckets` attribute in you configuration file. Here is a configuration example: ```yaml title="conf/config.yaml" plugin_attr: prometheus: default_buckets: 15 55 105 205 505 ``` This Plugin will add the API endpoint `/apisix/prometheus/metrics` or your custom export URI for exposing the metrics. These metrics are exposed by a separate Prometheus server address. By default, the address is `127.0.0.1:9091`. You can change it in your configuration file (`conf/config.yaml`): ```yaml title="conf/config.yaml" plugin_attr: prometheus: export_addr: ip: ${{INTRANET_IP}} port: 9092 ``` Now, if the environment variable `INTRANET_IP` is `172.1.1.1`, APISIX will export the metrics via `172.1.1.1:9092`. If you still want to expose the metrics via the data plane port (default: `9080`), you can configure it as shown below: ```yaml title="conf/config.yaml" plugin_attr: prometheus: enableexportserver: false ``` You can then expose it by using the Plugin. :::info IMPORTANT If the Prometheus plugin collects too many metrics, it will take CPU resources to calculate the metric data when getting the metrics via URI, which may affect APISIX to process normal requests. To solve this problem, APISIX exposes the URI and calculates the metrics in the . If the URI is exposed using the public-api plugin, then APISIX will calculate the metric data in a normal worker process, which may still affect APISIX processing of normal requests. This feature requires APISIX to run on . ::: The `prometheus` Plugin can be enabled with an empty table. The example below shows how you can configure the Plugin on a specific Route: :::note You can fetch the `admin_key` from `config.yaml` and save to an environment variable with the following command: ```bash adminkey=$(yq '.deployment.admin.adminkey[0].key' conf/config.yaml | sed 's/"//g') ``` ::: ```shell curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $admin_key" -X PUT -d ' { "uri": "/hello", "plugins": { "prometheus":{} }, "upstream": { "type": "roundrobin", "nodes": {

1 } } }' ``` :::note When `prefer_name` is set to `true` make sure to not duplicate names for multiple Routes/Services or it could be misleading. ::: <!-- You can use to complete the above operations through the web console. First, add a Route: Then add prometheus plugin: --> You can fetch the metrics from the specified export URI (default: `/apisix/prometheus/metrics`): ```shell curl -i http://127.0.0.1:9091/apisix/prometheus/metrics ``` You can add this address to Prometheus to fetch the data: ```yaml scrape_configs: job_name: "apisix" scrape_interval: 15s # This value will be related to the time range of the rate function in Prometheus QL. The time range in the rate function should be at least twice this value. metrics_path: "/apisix/prometheus/metrics" static_configs: targets: ["127.0.0.1:9091"] ``` Now, you will be able to check the status in your Prometheus console: Metrics exported by the `prometheus` Plugin can be graphed in Grafana using a drop in dashboard. To set it up, download and import it in Grafana. Or, you can go to for Grafana metadata. The following metrics are exported by the `prometheus` Plugin: Status code: HTTP status code returned from Upstream services. They are available for a single service and across all services. The available attributes are: | Name | Description | |--|-| | code | HTTP status code returned by the upstream service. | | route | `route_id` of the matched Route with request. Defaults to an empty string if the Routes don't match. | | matched_uri | `uri` of the Route matching the request. Defaults to an empty string if the Routes don't match. | | matched_host | `host` of the Route matching the request. Defaults to an empty string if the Routes don't match. | | service | `serviceid` of the Route matching the request. If the Route does not have a `serviceid` configured, it defaults to `$host`. | | consumer | `consumer_name` of the Consumer matching the request. Defaults to an empty string if it does not match. | | node | IP address of the Upstream node. | Bandwidth: Total amount of traffic (ingress and egress) flowing through APISIX. Total bandwidth of a service can also be obtained. The available attributes are: | Name | Description | |-|-| | type | Type of traffic (egress/ingress). | | route | `route_id` of the matched Route with request. Defaults to an empty string if the Routes don't match. | | service | `serviceid` of the Route matching the request. If the Route does not have a `serviceid` configured, it defaults to `$host`. | | consumer | `consumer_name` of the Consumer matching the request. Defaults to an empty string if it does not match. | | node | IP address of the Upstream node. | etcd reachability: A gauge type representing whether etcd can be reached by APISIX. A value of `1` represents reachable and `0` represents unreachable. Connections: Nginx connection metrics like active, reading, writing, and number of accepted connections. Batch process entries: A gauge type useful when Plugins like , , , , and use batch process to send data. Entries that hasn't been sent in batch process will be counted in the metrics. Latency: Histogram of the request time per service in different

The available attributes are: | Name | Description | |-|-| | type | Value can be one of `apisix`, `upstream`, or `request`. This translates to latency caused by APISIX, Upstream, or both (their sum). | | service | `serviceid` of the Route matching the request. If the Route does not have a `serviceid` configured, it defaults to `$host`. | | consumer | `consumer_name` of the Consumer matching the request. Defaults to an empty string if it does not match. | | node | IP address of the Upstream node. | Info: Information about the APISIX node. Shared dict: The capacity and free space of all nginx.shared.DICT in APISIX. `apisixupstreamstatus`: Health check result status of upstream nodes. A value of `1` represents healthy and `0` represents unhealthy. The available attributes are: | Name | Description | |--|-| | name | resource id where the upstream node is attached to, e.g. `/apisix/routes/1`, `/apisix/upstreams/1`. | | ip | ip address of the node. | | port | port number of the node. | Here are the original metrics from APISIX: ```shell curl http://127.0.0.1:9091/apisix/prometheus/metrics ``` ```shell apisix_bandwidth{type="egress",route="",service="",consumer="",node=""} 8417 apisix_bandwidth{type="egress",route="1",service="",consumer="",node="127.0.0.1"} 1420 apisix_bandwidth{type="egress",route="2",service="",consumer="",node="127.0.0.1"} 1420 apisix_bandwidth{type="ingress",route="",service="",consumer="",node=""} 189 apisix_bandwidth{type="ingress",route="1",service="",consumer="",node="127.0.0.1"} 332 apisix_bandwidth{type="ingress",route="2",service="",consumer="",node="127.0.0.1"} 332 apisixetcdmodify_indexes{key="consumers"} 0 apisixetcdmodifyindexes{key="globalrules"} 0 apisixetcdmodifyindexes{key="maxmodify_index"} 222 apisixetcdmodifyindexes{key="previndex"} 35 apisixetcdmodify_indexes{key="protos"} 0 apisixetcdmodify_indexes{key="routes"} 222 apisixetcdmodify_indexes{key="services"} 0 apisixetcdmodify_indexes{key="ssls"} 0 apisixetcdmodifyindexes{key="streamroutes"} 0 apisixetcdmodify_indexes{key="upstreams"} 0 apisixetcdmodifyindexes{key="xetcd_index"} 223 apisixbatchprocessentries{name="http-logger",routeid="9",server_addr="127.0.0.1"} 1 apisixbatchprocessentries{name="sls-logger",routeid="9",server_addr="127.0.0.1"} 1 apisixbatchprocessentries{name="tcp-logger",routeid="9",server_addr="127.0.0.1"} 1 apisixbatchprocessentries{name="udp-logger",routeid="9",server_addr="127.0.0.1"} 1 apisixbatchprocessentries{name="sys-logger",routeid="9",server_addr="127.0.0.1"} 1 apisixbatchprocessentries{name="zipkinreport",routeid="9",serveraddr="127.0.0.1"} 1 apisixetcdreachable 1 apisixhttpstatus{code="200",route="1",matcheduri="/hello",matchedhost="",service="",consumer="",node="127.0.0.1"} 4 apisixhttpstatus{code="200",route="2",matcheduri="/world",matchedhost="",service="",consumer="",node="127.0.0.1"} 4 apisixhttpstatus{code="404",route="",matcheduri="",matchedhost="",service="",consumer="",node=""} 1 apisixhttprequests_total 1191780 apisixnginxhttpcurrentconnections{state="accepted"} 11994 apisixnginxhttpcurrentconnections{state="active"} 2 apisixnginxhttpcurrentconnections{state="handled"} 11994 apisixnginxhttpcurrentconnections{state="reading"} 0 apisixnginxhttpcurrentconnections{state="waiting"} 1 apisixnginxhttpcurrentconnections{state="writing"} 1 apisixnginxmetricerrorstotal 0 apisixhttplatency_bucket{type="apisix",route="1",service="",consumer="",node="127.0.0.1",le="1"} 1 apisixhttplatency_bucket{type="apisix",route="1",service="",consumer="",node="127.0.0.1",le="2"} 1 apisixhttplatency_bucket{type="request",route="1",service="",consumer="",node="127.0.0.1",le="1"} 1 apisixhttplatency_bucket{type="request",route="1",service="",consumer="",node="127.0.0.1",le="2"} 1 apisixhttplatency_bucket{type="upstream",route="1",service="",consumer="",node="127.0.0.1",le="1"} 1 apisixhttplatency_bucket{type="upstream",route="1",service="",consumer="",node="127.0.0.1",le="2"} 1 ... apisixnodeinfo{hostname="desktop-2022q8f-wsl"} 1 apisixshareddictcapacitybytes{name="access-tokens"} 1048576 apisixshareddictcapacitybytes{name="balancer-ewma"} 10485760 apisixshareddictcapacitybytes{name="balancer-ewma-last-touched-at"} 10485760 apisixshareddictcapacitybytes{name="balancer-ewma-locks"} 10485760 apisixshareddictcapacitybytes{name="discovery"} 1048576 apisixshareddictcapacitybytes{name="etcd-cluster-health-check"} 10485760 ... apisixshareddictfreespace_bytes{name="access-tokens"} 1032192 apisixshareddictfreespace_bytes{name="balancer-ewma"} 10412032 apisixshareddictfreespace_bytes{name="balancer-ewma-last-touched-at"} 10412032 apisixshareddictfreespace_bytes{name="balancer-ewma-locks"} 10412032 apisixshareddictfreespace_bytes{name="discovery"} 1032192 apisixshareddictfreespace_bytes{name="etcd-cluster-health-check"} 10412032 ... apisixupstreamstatus{name="/apisix/routes/1",ip="100.24.156.8",port="80"} 0 apisixupstreamstatus{name="/apisix/routes/1",ip="52.86.68.46",port="80"} 1 ``` To remove the `prometheus` Plugin, you can delete the corresponding JSON configuration from the Plugin configuration. APISIX will automatically reload and you do not have to restart for this to take effect. ```shell curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $admin_key" -X PUT -d ' { "uri": "/hello", "plugins": {}, "upstream": { "type": "roundrobin", "nodes": { "127.0.0.1:80": 1 } } }' ``` :::info IMPORTANT This feature requires APISIX to run on . ::: We can also enable `prometheus` to collect metrics for TCP/UDP. First of all, ensure `prometheus` plugin is in your configuration file (`conf/config.yaml`): ```yaml title="conf/config.yaml" stream_plugins: ... prometheus ``` Then you need to configure the `prometheus` plugin on the stream route: ```shell curl http://127.0.0.1:9180/apisix/admin/streamroutes/1 -H "X-API-KEY: $adminkey" -X PUT -d ' { "plugins": { "prometheus":{} }, "upstream": { "type": "roundrobin", "nodes": { "127.0.0.1:80": 1 } } }' ``` The following metrics are available when using APISIX as an L4 proxy. `Stream Connections`: The number of processed connections at the route level. Attributes: | Name | Description | | - | -- | | route | matched stream route ID | `Connections`: Various Nginx connection metrics like active, reading, writing, and number of accepted connections. `Info`: Information about the current APISIX node. Here are examples of APISIX metrics: ```shell $ curl http://127.0.0.1:9091/apisix/prometheus/metrics ``` ``` ... apisixnodeinfo{hostname="desktop-2022q8f-wsl"} 1 apisixstreamconnection_total{route="1"} 1 ```