Chapter 10. Enabling and configuring Data Grid OpenTelemetry tracing Data Grid generates tracing spans compatible with the OpenTelemetry standard, allowing you to export, visualize, and analyze tracing data related to the most important cache operations. 10.1. Configuring Data Grid tracing Configure OpenTelemetry tracing, to enable monitoring and tracing of cache operations. Procedure Open your Data Grid configuration for editing. Add the tracing element or object to the cache container. Define the endpoint URL of the OpenTelemetry collector with the collector_endpoint attribute or field. It is mandatory to enable tracing. 4318 is typically the http/protobuf OTLP standard port. Enable or disable tracing globally with the enable attribute or field. Enable or disable security event tracing with the security attribute or field. Optionally change the tracing exporter protocol changing the exporter_protocol attribute or field. By default, it is otlp (OpenTelemetry protocol). Optionally change the tracing service name associated with the generated tracing span changing the service_name attribute or field. By default, it is infinispan-server . Save and close your client configuration. steps To apply any global tracing configuration changes, stop the server and repeat the procedure. Tracing configuration Data Grid applies the tracing configuration globally to all caches. XML <infinispan> <cache-container statistics="true"> <tracing collector-endpoint="http://localhost:4318" enabled="true" exporter-protocol="OTLP" service-name="infinispan-server" security="false" /> </cache-container> </infinispan> JSON { "infinispan" : { "cache-container" : { "statistics" : true, "tracing" : { "collector-endpoint" : "http://localhost:4318", "enabled" : true, "exporter-protocol" : "OTLP", "service-name" : "infinispan-server", "security" : false } } } } YAML infinispan: cacheContainer: statistics: true tracing: collector-endpoint: "http://localhost:4318" enabled: true exporter-protocol: "OTLP" service-name: "infinispan-server" security: false 10.1.1. Further Tracing Options To configure further tracing options it is possible to pass system properties or to set environment variables to the Data Grid server at startup to configure directly the OpenTelemetry SDK Autoconfigure that is used by the Data Grid server to configure the OpenTelemetry tracing. Procedure Pass the system properties to Data Grid Server at startup. Use -D<property-name>=<property-value> arguments like in the following example: Additional resources OpenTelemetry SDK Autoconfigure Tracing data format The Data Grid Server, by default, exports tracing data using the OTLP http/protobuf protocol. tracing.properties To use a different protocol, you must copy JAR files or dependencies to the USDISPN_HOME/server/lib directory of your Data Grid Server installation. 10.2. Configure tracing at cache level Once the tracing is configured at server level , it will be automatically enabled by default for all caches. A cache configuration level of tracing allows on the other hand to enable or disable it at cache level and at runtime. Tracing categories Several categories are potentially traced: Container. That are all the main cache operations, such as replace, put, clear, getForReplace, remove operations and size. With the exception of all the getting operation. Cluster. Operations that are replicated to another node in the same cluster. X-Site. Operations that are replicated to another external site. Persistence. All the operations involving persistence via a cache store and/or cache loader. Each of category can be enabled/disabled at start time or runtime listing them in the categories list attribute. By default only the container category is enabled. There is also the Security category, to trace security audit events. This category is configured globally, not only at cache level, since their events can have different scopes (cache, container or server), not only cache scope. Enable/disable tracing for a given cache XML <replicated-cache> <tracing enabled="true" categories="container cluster x-site persistence" /> </replicated-cache> JSON { "distributed-cache": { "tracing": { "enabled" : true, "categories": [ "container", "cluster", "x-site", "persistence" ] } } } YAML distributedCache: tracing: enabled: true categories: - "container" - "cluster" - "x-site" - "persistence" Enable/disable tracing at runtime The cache-level tracing attribute enable is a mutable attribute, it means it can be changed at runtime without the need to restart the Infinispan cluster. To change a mutable attribute both HotRod and REST APIs can be used. HotRod remoteCacheManager.administration() .updateConfigurationAttribute(CACHE_A, "tracing.enabled", "false"); REST restClient.cache(CACHE_A) .updateConfigurationAttribute("tracing.enabled", "false"); Additional resources Updating cache configuration attributes

Commands:
<infinispan> <cache-container statistics="true"> <tracing collector-endpoint="http://localhost:4318" enabled="true" exporter-protocol="OTLP" service-name="infinispan-server" security="false" /> </cache-container> </infinispan>
{ "infinispan" : { "cache-container" : { "statistics" : true, "tracing" : { "collector-endpoint" : "http://localhost:4318", "enabled" : true, "exporter-protocol" : "OTLP", "service-name" : "infinispan-server", "security" : false } } } }
infinispan: cacheContainer: statistics: true tracing: collector-endpoint: "http://localhost:4318" enabled: true exporter-protocol: "OTLP" service-name: "infinispan-server" security: false
bin/server.sh -Dotel.exporter.otlp.timeout=10000
otel.exporter.otlp.protocol = http/protobuf
<replicated-cache> <tracing enabled="true" categories="container cluster x-site persistence" /> </replicated-cache>
{ "distributed-cache": { "tracing": { "enabled" : true, "categories": [ "container", "cluster", "x-site", "persistence" ] } } }
distributedCache: tracing: enabled: true categories: - "container" - "cluster" - "x-site" - "persistence"
remoteCacheManager.administration() .updateConfigurationAttribute(CACHE_A, "tracing.enabled", "false");
restClient.cache(CACHE_A) .updateConfigurationAttribute("tracing.enabled", "false");