Project: APISIX (Orchestration & Management / API Gateway)
File: error-log-logger.md

title: error-log-logger keywords: Apache APISIX API Gateway Plugin Error log logger description: This document contains information about the Apache APISIX error-log-logger Plugin. <!-- --> The `error-log-logger` Plugin is used to push APISIX's error logs (`error.log`) to TCP, , Apache Kafka or ClickHouse servers. You can also set the error log level to send the logs to server. It might take some time to receive the log data. It will be automatically sent after the timer function in the expires. | Name | Type | Required | Default | Valid values | Description | |-||-|--|--|--| | tcp.host | string | True | | | IP address or the hostname of the TCP server. | | tcp.port | integer | True | | [0,...] | Target upstream port. | | tcp.tls | boolean | False | false | | When set to `true` performs SSL verification. | | tcp.tlsservername | string | False | | | Server name for the new TLS extension SNI. | | skywalking.endpoint_addr | string | False | http://127.0.0.1:12900/v3/logs | | Apache SkyWalking HTTP endpoint. | | skywalking.service_name | string | False | APISIX | | Service name for the SkyWalking reporter. | | skywalking.serviceinstancename | String | False | APISIX Instance Name | | Service instance name for the SkyWalking reporter. Set it to `$hostname` to directly get the local hostname. | | clickhouse.endpoint_addr | String | False | http://127.0.0.1:8213 | | ClickHouse endpoint. | | clickhouse.user | String | False | default | | ClickHouse username. | | clickhouse.password | String | False | | | ClickHouse password. | | clickhouse.database | String | False | | | Name of the database to store the logs. | | clickhouse.logtable | String | False | | | Table name to store the logs. | | kafka.brokers | array | True | | | List of Kafka brokers (nodes). | | kafka.brokers.host | string | True | | | The host of Kafka broker, e.g, `192.168.1.1`. | | kafka.brokers.port | integer | True | | [0, 65535] | The port of Kafka broker | | kafka.brokers.sasl_config | object | False | | | The sasl config of Kafka broker | | kafka.brokers.sasl_config.mechanism | string | False | "PLAIN" | ["PLAIN"] | The mechaism of sasl config | | kafka.brokers.saslconfig.user | string | True | | | The user of saslconfig. If sasl_config exists, it's required. | | kafka.brokers.saslconfig.password | string | True | | | The password of saslconfig. If sasl_config exists, it's required. | | kafka.kafka_topic | string | True | | | Target topic to push the logs for organisation. | | kafka.producer_type | string | False | async | ["async", "sync"] | Message sending mode of the producer. | | kafka.required_acks | integer | False | 1 | for more. | |

| string | False | | | Key used for allocating partitions for messages. | | kafka.clustername | integer | False | 1 | [0,...] | Name of the cluster. Used when there are two or more Kafka clusters. Only works if the `producertype` attribute is set to `async`. | | kafka.metarefreshinterval | integer | False | 30 | specifies the time to auto refresh the metadata, in seconds.| | timeout | integer | False | 3 | [1,...] | Timeout (in seconds) for the upstream to connect and send data. | | keepalive | integer | False | 30 | [1,...] | Time in seconds to keep the connection alive after sending data. | | level | string | False | WARN | ["STDERR", "EMERG", "ALERT", "CRIT", "ERR", "ERROR", "WARN", "NOTICE", "INFO", "DEBUG"] | Log level to filter the error logs. `ERR` is same as `ERROR`. | NOTE: `encrypt_fields = {"clickhouse.password"}` is also defined in the schema, which means that the field will be stored encrypted in etcd. See . This Plugin supports using batch processors to aggregate and process entries (logs/data) in a batch. This avoids the need for frequently submitting the data. The batch processor submits data every `5` seconds or when the data in the queue reaches `1000`. See for more information or setting your custom configuration. ```text ["2024/01/06 16:04:30 [warn] 11786#9692271: 1 [lua] plugin.lua:205: load(): new plugins: {"error-log-logger":true}, context: init_worker_by_lua","\n","2024/01/06 16:04:30 [warn] 11786#9692271: 1 [lua] plugin.lua:255: load_stream(): new plugins: {"limit-conn":true,"ip-restriction":true,"syslog":true,"mqtt-proxy":true}, context: init_worker_by_lua","\n"] ``` To enable the Plugin, you can add it in your configuration file (`conf/config.yaml`): ```yaml title="conf/config.yaml" plugins: request-id hmac-auth api-breaker error-log-logger ``` Once you have enabled the Plugin, you can configure it through the Plugin metadata. You can set the TCP server address by configuring the Plugin metadata as shown below: :::note You can fetch the `admin_key` from `config.yaml` and save to an environment variable with the following command: ```bash adminkey=$(yq '.deployment.admin.adminkey[0].key' conf/config.yaml | sed 's/"//g') ``` ::: ```shell curl http://127.0.0.1:9180/apisix/admin/pluginmetadata/error-log-logger -H "X-API-KEY: $adminkey" -X PUT -d ' { "tcp": { "host": "127.0.0.1", "port": 1999 }, "inactive_timeout": 1 }' ``` You can configure the SkyWalking OAP server address as shown below: ```shell curl http://127.0.0.1:9180/apisix/admin/pluginmetadata/error-log-logger -H "X-API-KEY: $adminkey" -X PUT -d ' { "skywalking": { "endpoint_addr":"http://127.0.0.1:12800/v3/logs" }, "inactive_timeout": 1 }' ``` The Plugin sends the error log as a string to the `data` field of a table in your ClickHouse server. You can configure it as shown below: ```shell curl http://127.0.0.1:9180/apisix/admin/pluginmetadata/error-log-logger -H "X-API-KEY: $adminkey" -X PUT -d ' { "clickhouse": { "user": "default", "password": "a", "database": "error_log", "logtable": "t", "endpoint_addr": "http://127.0.0.1:8123" } }' ``` The Plugin sends the error log to Kafka, you can configure it as shown below: ```shell curl http://127.0.0.1:9180/apisix/admin/plugin_metadata/error-log-logger \ -H "X-API-KEY: $admin_key" -X PUT -d ' { "kafka":{ "brokers":[ { "host":"127.0.0.1", "port":9092 } ], "kafka_topic":"test2" }, "level":"ERROR", "inactive_timeout":1 }' ``` To remove the Plugin, you can remove it from your configuration file (`conf/config.yaml`): ```yaml title="conf/config.yaml" plugins: request-id hmac-auth api-breaker ```