Chapter 10. Configuring logging for Kafka components Configure the logging levels of Kafka components directly in the configuration properties. You can also change the broker levels dynamically for Kafka brokers, Kafka Connect, and MirrorMaker 2. Increasing the log level detail, such as from INFO to DEBUG, can aid in troubleshooting a Kafka cluster. However, more verbose logs may also negatively impact performance and make it more difficult to diagnose issues. 10.1. Configuring Kafka logging properties Kafka components use the Log4j framework for error logging. By default, logging configuration is read from the classpath or config directory using the following properties files: log4j.properties for Kafka and ZooKeeper connect-log4j.properties for Kafka Connect and MirrorMaker 2 If they are not set explicitly, loggers inherit the log4j.rootLogger logging level configuration in each file. You can change the logging level in these files. You can also add and set logging levels for other loggers. You can change the location and name of logging properties file using the KAFKA_LOG4J_OPTS environment variable, which is used by the start script for the component. Passing the name and location of the logging properties file used by Kafka brokers su - kafka export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:/my/path/to/log4j.properties"; \ /opt/kafka/bin/kafka-server-start.sh \ /opt/kafka/config/server.properties Passing the name and location of the logging properties file used by ZooKeeper su - kafka export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:/my/path/to/log4j.properties"; \ /opt/kafka/bin/zookeeper-server-start.sh -daemon \ /opt/kafka/config/zookeeper.properties Passing the name and location of the logging properties file used by Kafka Connect su - kafka export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:/my/path/to/connect-log4j.properties"; \ /opt/kafka/bin/connect-distributed.sh \ /opt/kafka/config/connect-distributed.properties Passing the name and location of the logging properties file used by MirrorMaker 2 su - kafka export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:/my/path/to/connect-log4j.properties"; \ /opt/kafka/bin/connect-mirror-maker.sh \ /opt/kafka/config/connect-mirror-maker.properties 10.2. Dynamically change logging levels for Kafka broker loggers Kafka broker logging is provided by broker loggers in each broker. Dynamically change the logging level for broker loggers at runtime without having to restart the broker. You can also reset broker loggers dynamically to their default logging levels. Prerequisites AMQ Streams is installed on the host ZooKeeper and Kafka are running Procedure Switch to the kafka user: su - kafka List all the broker loggers for a broker by using the kafka-configs.sh tool: /opt/kafka/bin/kafka-configs.sh --bootstrap-server <broker_address> --describe --entity-type broker-loggers --entity-name BROKER-ID For example, for broker 0 : /opt/kafka/bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type broker-loggers --entity-name 0 This returns the logging level for each logger: TRACE , DEBUG , INFO , WARN , ERROR , or FATAL . For example: #... kafka.controller.ControllerChannelManager=INFO sensitive=false synonyms={} kafka.log.TimeIndex=INFO sensitive=false synonyms={} Change the logging level for one or more broker loggers. Use the --alter and --add-config options and specify each logger and its level as a comma-separated list in double quotes. /opt/kafka/bin/kafka-configs.sh --bootstrap-server <broker_address> --alter --add-config " LOGGER-ONE=NEW-LEVEL , LOGGER-TWO=NEW-LEVEL " --entity-type broker-loggers --entity-name BROKER-ID For example, for broker 0 : /opt/kafka/bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config "kafka.controller.ControllerChannelManager=WARN,kafka.log.TimeIndex=WARN" --entity-type broker-loggers --entity-name 0 If successful this returns: Completed updating config for broker: 0. Resetting a broker logger You can reset one or more broker loggers to their default logging levels by using the kafka-configs.sh tool. Use the --alter and --delete-config options and specify each broker logger as a comma-separated list in double quotes: /opt/kafka/bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --delete-config " LOGGER-ONE , LOGGER-TWO " --entity-type broker-loggers --entity-name BROKER-ID Additional resources Updating Broker Configs in the Apache Kafka documentation 10.3. Dynamically change logging levels for Kafka Connect and MirrorMaker 2 Dynamically change logging levels for Kafka Connect workers or MirrorMaker 2 connectors at runtime without having to restart. Use the Kafka Connect API to change the log level temporarily for a worker or connector logger. The Kafka Connect API provides an admin/loggers endpoint to get or modify logging levels. When you change the log level using the API, the logger configuration in the connect-log4j.properties configuration file does not change. If required, you can permanently change the logging levels in the configuration file. Note You can only change the logging level of MirrorMaker 2 at runtime when in distributed or standalone mode. Dedicated MirrorMaker 2 clusters have no Kafka Connect REST API, so changing the logging level is not possible. The default listener for the Kafka Connect API is on port 8083, which is used in this procedure. You can change or add more listeners, and also enable TLS authentication, using admin.listeners configuration. Example listener configuration for the admin endpoint admin.listeners=https://localhost:8083 admin.listeners.https.ssl.truststore.location=/path/to/truststore.jks admin.listeners.https.ssl.truststore.password=123456 admin.listeners.https.ssl.keystore.location=/path/to/keystore.jks admin.listeners.https.ssl.keystore.password=123456 If you do not want the admin endpoint to be available, you can disable it in the configuration by specifying an empty string. Example listener configuration to disable the admin endpoint admin.listeners= Prerequisites AMQ Streams is installed on the host ZooKeeper and Kafka are running Kafka Connect or MirrorMaker 2 is running Procedure Switch to the kafka user: su - kafka Check the current logging level for the loggers configured in the connect-log4j.properties file: USD cat /opt/kafka/config/connect-log4j.properties # ... log4j.rootLogger=INFO, stdout, connectAppender # ... log4j.logger.org.apache.zookeeper=ERROR log4j.logger.org.reflections=ERROR Use a curl command to check the logging levels from the admin/loggers endpoint of the Kafka Connect API: curl -s http://localhost:8083/admin/loggers/ | jq { "org.apache.zookeeper": { "level": "ERROR" }, "org.reflections": { "level": "ERROR" }, "root": { "level": "INFO" } } jq prints the output in JSON format. The list shows standard org and root level loggers, plus any specific loggers with modified logging levels. If you configure TLS (Transport Layer Security) authentication for the admin.listeners configuration in Kafka Connect, then the address of the loggers endpoint is the value specified for admin.listeners with the protocol as https, such as https://localhost:8083 . You can also get the log level of a specific logger: curl -s http://localhost:8083/admin/loggers/org.apache.kafka.connect.mirror.MirrorCheckpointConnector | jq { "level": "INFO" } Use a PUT method to change the log level for a logger: curl -Ss -X PUT -H 'Content-Type: application/json' -d '{"level": "TRACE"}' http://localhost:8083/admin/loggers/root { # ... "org.reflections": { "level": "TRACE" }, "org.reflections.Reflections": { "level": "TRACE" }, "root": { "level": "TRACE" } } If you change the root logger, the logging level for loggers that used the root logging level by default are also changed.

Commands:
su - kafka export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:/my/path/to/log4j.properties"; /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
su - kafka export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:/my/path/to/log4j.properties"; /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties
su - kafka export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:/my/path/to/connect-log4j.properties"; /opt/kafka/bin/connect-distributed.sh /opt/kafka/config/connect-distributed.properties
su - kafka export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:/my/path/to/connect-log4j.properties"; /opt/kafka/bin/connect-mirror-maker.sh /opt/kafka/config/connect-mirror-maker.properties
su - kafka
/opt/kafka/bin/kafka-configs.sh --bootstrap-server <broker_address> --describe --entity-type broker-loggers --entity-name BROKER-ID
/opt/kafka/bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type broker-loggers --entity-name 0
# kafka.controller.ControllerChannelManager=INFO sensitive=false synonyms={} kafka.log.TimeIndex=INFO sensitive=false synonyms={}
/opt/kafka/bin/kafka-configs.sh --bootstrap-server <broker_address> --alter --add-config " LOGGER-ONE=NEW-LEVEL , LOGGER-TWO=NEW-LEVEL " --entity-type broker-loggers --entity-name BROKER-ID
/opt/kafka/bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config "kafka.controller.ControllerChannelManager=WARN,kafka.log.TimeIndex=WARN" --entity-type broker-loggers --entity-name 0
Completed updating config for broker: 0.
/opt/kafka/bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --delete-config " LOGGER-ONE , LOGGER-TWO " --entity-type broker-loggers --entity-name BROKER-ID
admin.listeners=https://localhost:8083 admin.listeners.https.ssl.truststore.location=/path/to/truststore.jks admin.listeners.https.ssl.truststore.password=123456 admin.listeners.https.ssl.keystore.location=/path/to/keystore.jks admin.listeners.https.ssl.keystore.password=123456
admin.listeners=
su - kafka
cat /opt/kafka/config/connect-log4j.properties log4j.rootLogger=INFO, stdout, connectAppender log4j.logger.org.apache.zookeeper=ERROR log4j.logger.org.reflections=ERROR
curl -s http://localhost:8083/admin/loggers/ | jq { "org.apache.zookeeper": { "level": "ERROR" }, "org.reflections": { "level": "ERROR" }, "root": { "level": "INFO" } }
curl -s http://localhost:8083/admin/loggers/org.apache.kafka.connect.mirror.MirrorCheckpointConnector | jq { "level": "INFO" }
curl -Ss -X PUT -H 'Content-Type: application/json' -d '{"level": "TRACE"}' http://localhost:8083/admin/loggers/root { # "org.reflections": { "level": "TRACE" }, "org.reflections.Reflections": { "level": "TRACE" }, "root": { "level": "TRACE" } }