Chapter 15. Configuring Data Grid Server logging Data Grid Server uses Apache Log4j 2 to provide configurable logging mechanisms that capture details about the environment and record cache operations for troubleshooting purposes and root cause analysis. 15.1. Data Grid Server log files Data Grid writes server logs to the following files in the USDRHDG_HOME/server/log directory: server.log Messages in human readable format, including boot logs that relate to the server startup. Data Grid creates this file when you start the server. server.log.json Messages in JSON format that let you parse and analyze Data Grid logs. Data Grid creates this file when you enable the JSON-FILE appender. 15.1.1. Configuring Data Grid Server logs Data Grid uses Apache Log4j technology to write server log messages. You can configure server logs in the log4j2.xml file. Procedure Open USDRHDG_HOME/server/conf/log4j2.xml with any text editor. Change server logging as appropriate. Save and close log4j2.xml . Additional resources Apache Log4j manual 15.1.2. Log levels Log levels indicate the nature and severity of messages. Log level Description TRACE Fine-grained debug messages, capturing the flow of individual requests through the application. DEBUG Messages for general debugging, not related to an individual request. INFO Messages about the overall progress of applications, including lifecycle events. WARN Events that can lead to error or degrade performance. ERROR Error conditions that might prevent operations or activities from being successful but do not prevent applications from running. FATAL Events that could cause critical service failure and application shutdown. In addition to the levels of individual messages presented above, the configuration allows two more values: ALL to include all messages, and OFF to exclude all messages. 15.1.3. Data Grid logging categories Data Grid provides categories for INFO , WARN , ERROR , FATAL level messages that organize logs by functional area. org.infinispan.CLUSTER Messages specific to Data Grid clustering that include state transfer operations, rebalancing events, partitioning, and so on. org.infinispan.CONFIG Messages specific to Data Grid configuration. org.infinispan.CONTAINER Messages specific to the data container that include expiration and eviction operations, cache listener notifications, transactions, and so on. org.infinispan.PERSISTENCE Messages specific to cache loaders and stores. org.infinispan.SECURITY Messages specific to Data Grid security. org.infinispan.SERVER Messages specific to Data Grid servers. org.infinispan.XSITE Messages specific to cross-site replication operations. 15.1.4. Log appenders Log appenders define how Data Grid Server records log messages. CONSOLE Write log messages to the host standard out ( stdout ) or standard error ( stderr ) stream. Uses the org.apache.logging.log4j.core.appender.ConsoleAppender class by default. FILE Write log messages to a file. Uses the org.apache.logging.log4j.core.appender.RollingFileAppender class by default. JSON-FILE Write log messages to a file in JSON format. Uses the org.apache.logging.log4j.core.appender.RollingFileAppender class by default. 15.1.5. Log pattern formatters The CONSOLE and FILE appenders use a PatternLayout to format the log messages according to a pattern . An example is the default pattern in the FILE appender: %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p (%t) [%c{1}] %m%throwable%n %d{yyyy-MM-dd HH:mm:ss,SSS} adds the current time and date. %-5p specifies the log level, aligned to the right. %t adds the name of the current thread. %c{1} adds the short name of the logging category. %m adds the log message. %throwable adds the exception stack trace. %n adds a new line. Patterns are fully described in the PatternLayout documentation . 15.1.6. Enabling the JSON log handler Data Grid Server provides a log handler to write messages in JSON format. Prerequisites Stop Data Grid Server if it is running. You cannot dynamically enable log handlers. Procedure Open USDRHDG_HOME/server/conf/log4j2.xml with any text editor. Uncomment the JSON-FILE appender and comment out the FILE appender: <!--<AppenderRef ref="FILE"/>--> <AppenderRef ref="JSON-FILE"/> Optionally configure the JSON appender and JSON layout as required. Save and close log4j2.xml . When you start Data Grid, it writes each log message as a JSON map in the following file: USDRHDG_HOME/server/log/server.log.json Additional resources RollingFileAppender JSONLayout 15.2. Access logs Access logs record all inbound client requests for Hot Rod and REST endpoints to files in the USDRHDG_HOME/server/log directory. org.infinispan.HOTROD_ACCESS_LOG Logging category that writes Hot Rod access messages to a hotrod-access.log file. org.infinispan.REST_ACCESS_LOG Logging category that writes REST access messages to a rest-access.log file. 15.2.1. Enabling access logs To record Hot Rod, REST and Memcached endpoint access messages, you need to enable the logging categories in log4j2.xml . Procedure Open USDRHDG_HOME/server/conf/log4j2.xml with any text editor. Change the level for the org.infinispan.HOTROD_ACCESS_LOG , org.infinispan.REST_ACCESS_LOG and org.infinispan.MEMCACHED_ACCESS_LOG logging categories to TRACE . Save and close log4j2.xml . <Logger name="org.infinispan.HOTROD_ACCESS_LOG" additivity="false" level="TRACE"> <AppenderRef ref="HR-ACCESS-FILE"/> </Logger> 15.2.2. Access log properties The default format for access logs is as follows: The preceding format creates log entries such as the following: 127.0.0.1 - [DD/MM/YYYY:HH:MM:SS +0000] "PUT /rest/v2/caches/default/key HTTP/1.1" 404 5 77 10 Logging properties use the %X{name} notation and let you modify the format of access logs. The following are the default logging properties: Property Description address Either the X-Forwarded-For header or the client IP address. user Principal name, if using authentication. method The protocol-specific method used. PUT , GET , and so on. protocol Protocol used. HTTP/1.1 , HTTP/2 , HOTROD/2.9 , MCTXT , MCBIN and so on. status An HTTP status code for the REST endpoint. OK or an exception for the Hot Rod endpoint. requestSize Size, in bytes, of the request. responseSize Size, in bytes, of the response. duration Number of milliseconds that the server took to handle the request. Tip Use the header name prefixed with h: to log headers that were included in requests; for example, %X{h:User-Agent} . 15.3. Audit logs Audit logs let you track changes to your Data Grid Server deployment so you know when changes occur and which users make them. Enable and configure audit logging to record server configuration events and administrative operations. org.infinispan.AUDIT Logging category that writes security audit messages to an audit.log file in the USDRHDG_HOME/server/log directory. 15.3.1. Enabling audit logging To record security audit messages, you need to enable the logging category in log4j2.xml . Procedure Open USDRHDG_HOME/server/conf/log4j2.xml with any text editor. Change the level for the org.infinispan.AUDIT logging category to INFO . Save and close log4j2.xml . <!-- Set to INFO to enable audit logging --> <Logger name="org.infinispan.AUDIT" additivity="false" level="INFO"> <AppenderRef ref="AUDIT-FILE"/> </Logger> 15.3.2. Configuring audit logging appenders Apache Log4j provides different appenders that you can use to send audit messages to a destination other than the default log file. For instance, if you want to send audit logs to a syslog daemon, JDBC database, or Apache Kafka server, you can configure an appender in log4j2.xml . Procedure Open USDRHDG_HOME/server/conf/log4j2.xml with any text editor. Comment or remove the default AUDIT-FILE rolling file appender. <!--RollingFile name="AUDIT-FILE" ... </RollingFile--> Add the desired logging appender for audit messages. For example, you could add a logging appender for a Kafka server as follows: <Kafka name="AUDIT-KAFKA" topic="audit"> <PatternLayout pattern="%date %message"/> <Property name="bootstrap.servers">localhost:9092</Property> </Kafka> Save and close log4j2.xml . Additional resources Log4j Appenders 15.3.3. Using custom audit logging implementations You can create custom implementations of the org.infinispan.security.AuditLogger API if configuring Log4j appenders does not meet your needs. Prerequisites Implement org.infinispan.security.AuditLogger as required and package it in a JAR file. Procedure Add your JAR to the server/lib directory in your Data Grid Server installation. Specify the fully qualified class name of your custom audit logger as the value for the audit-logger attribute on the authorization element in your cache container security configuration. For example, the following configuration defines my.package.CustomAuditLogger as the class for logging audit messages: <infinispan> <cache-container> <security> <authorization audit-logger="my.package.CustomAuditLogger"/> </security> </cache-container> </infinispan> Additional resources org.infinispan.security.AuditLogger

Commands:
<!--<AppenderRef ref="FILE"/>--> <AppenderRef ref="JSON-FILE"/>
<Logger name="org.infinispan.HOTROD_ACCESS_LOG" additivity="false" level="TRACE"> <AppenderRef ref="HR-ACCESS-FILE"/> </Logger>
%X{address} %X{user} [%d{dd/MMM/yyyy:HH:mm:ss Z}] &quot;%X{method} %m %X{protocol}&quot; %X{status} %X{requestSize} %X{responseSize} %X{duration}%n
<!-- Set to INFO to enable audit logging --> <Logger name="org.infinispan.AUDIT" additivity="false" level="INFO"> <AppenderRef ref="AUDIT-FILE"/> </Logger>
<!--RollingFile name="AUDIT-FILE" </RollingFile-->
<Kafka name="AUDIT-KAFKA" topic="audit"> <PatternLayout pattern="%date %message"/> <Property name="bootstrap.servers">localhost:9092</Property> </Kafka>
<infinispan> <cache-container> <security> <authorization audit-logger="my.package.CustomAuditLogger"/> </security> </cache-container> </infinispan>