Chapter 10. Observability in JBoss EAP If you're a developer or system administrator, observability is a set of practices and technologies you can use to determine, based on certain signals from your application, the location and source of a problem in your application. The most common signals are metrics, events, and tracing. JBoss EAP uses OpenTelemetry for observability . 10.1. OpenTelemetry in JBoss EAP OpenTelemetry is a set of tools, application programming interfaces (APIs), and software development kits (SDKs) you can use to instrument, generate, collect, and export telemetry data for your applications. Telemetry data includes metrics, logs, and traces. Analyzing an application's telemetry data helps you to improve your application's performance. JBoss EAP provides OpenTelemetry capability through the opentelemetry subsystem. Note Red Hat JBoss Enterprise Application Platform 7.4 provides only OpenTelemetry tracing capabilities. Important OpenTelemetry is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview . Additional resources OpenTelemetry Documentation 10.2. OpenTelemetry configuration in JBoss EAP You configure a number of aspects of OpenTelemetry in JBoss EAP using the opentelemetry subsystem. These include exporter, span processor, and sampler. exporter To analyze and visualize traces and metrics, you export them to a collector such as Jaeger. You can configure JBoss EAP to use either Jaeger or any collector that supports the OpenTelemetry protocol (OTLP). span processor You can configure the span processor to export spans either as they are produced or in batches. You can also configure the number of traces to export. sampler You can configure the number of traces to record by configuring the sampler. Example configuration The following XML is an example of the full OpenTelemetry configuration, including default values. JBoss EAP does not persist the default values when you make changes, so your configuration might look different. <subsystem xmlns="urn:wildfly:opentelemetry:1.0" service-name="example"> <exporter type="jaeger" endpoint="http://localhost:14250"/> <span-processor type="batch" batch-delay="4500" max-queue-size="128" max-export-batch-size="512" export-timeout="45"/> <sampler type="on"/> </subsystem> Note You cannot use an OpenShift route object to connect with a Jaeger endpoint. Instead, use http:// <ip_address> : <port> or http:// <service_name> : <port> . Additional resources OpenTelemetry subsystem attributes 10.3. OpenTelemetry tracing in JBoss EAP JBoss EAP provides OpenTelemetry tracing to help you track the progress of user requests as they pass through different parts of your application. By analyzing traces, you can improve your application's performance and debug availability issues. OpenTelemetry tracing consists of the following components: Trace A collection of operations that a request goes through in an application. Span A single operation within a trace. It provides request, error, and duration (RED) metrics and contains a span context. Span context A set of unique identifiers that represents a request that the containing span is a part of. JBoss EAP automatically traces REST calls to your Jakarta RESTful Web Services applications and container-managed Jakarta RESTful Web Services client invocations. JBoss EAP traces REST calls implicitly as follows: For each incoming request: JBoss EAP extracts the span context from the request. JBoss EAP starts a new span, then closes it when the request is completed. For each outgoing request: JBoss EAP injects span context into the request. JBoss EAP starts a new span, then closes it when the request is completed. In addition to implicit tracing, you can create custom spans by injecting a Tracer instance into your application for granular tracing. Important If you see duplicate traces exported for REST calls, disable the microprofile-opentracing-smallrye subsystem. For information about disabling the microprofile-opentracing-smallrye , see Removing the microprofile-opentracing-smallrye subsystem . Additional resources Using Jaeger to observe the OpenTelemetry traces for an application OpenTelemetry application development in JBoss EAP 10.4. Enabling OpenTelemetry tracing in JBoss EAP To use OpenTelemetry tracing in JBoss EAP you must first enable the opentelemetry subsystem. Prerequisites You have installed JBoss EAP XP. Procedure Add the OpenTelemetry extension using the management CLI. Enable the opentelemetry subsystem using the management CLI. Reload JBoss EAP. Additional resources Configuring the opentelemetry subsystem 10.5. Configuring the opentelemetry subsystem You can configure the opentelemetry subsystem to set different aspects of tracing. Configure these based on the collector you use for observing the traces. Prerequisites You have enabled the opentelemetry subsystem. For more information, see Enabling OpenTelemetry tracing in JBoss EAP . Procedure Set the exporter type for the traces. Syntax Example Set the endpoint at which to export the traces. Syntax Example Set the service name under which the traces are exported. Syntax Example Additional resources Using Jaeger to observe the OpenTelemetry traces for an application 10.6. Using Jaeger to observe the OpenTelemetry traces for an application JBoss EAP automatically and implicitly traces REST calls to Jakarta RESTful Web Services applications. You do not need to add any configuration to your Jakarta RESTful Web Services application or configure the opentelemetry subsystem. The following procedure demonstrates how to observe traces for the helloworld-rs quickstart in the Jaeger console. Prerequisites You have installed Docker. For more information, see Get Docker . You have downloaded the helloworld-rs quickstart. The quickstart is available at helloworld-rs . You have configured the the opentelemetry subsystem. For more information, see Configuring the opentelemetry subsystem . Procedure Start the Jaeger console using its Docker image. Use Maven to deploy the helloworld-rs quickstart from its root directory. In a web browser, access the quickstart at http://localhost:8080/helloworld-rs/ , then click any link. In a web browser, open the Jaeger console at http://localhost:16686/search . hello-world.rs is listed under Service . Select hello-world.rs and click Find Traces . The details of the trace for hello-world.rs are listed. Additional resources OpenTelemetry application development in JBoss EAP 10.7. OpenTelemetry tracing application development Although JBoss EAP automatically and implicitly traces REST calls to Jakarta RESTful Web Services applications, you can create custom spans from your application for granular tracing. A span is a single operation within a trace. You can create a span when, for example, a resource is defined, a method is called, and so on, in your application. You create custom traces in your application by injecting a Tracer instance. 10.7.1. Configuring a Maven project for OpenTelemetry tracing For creating an OpenTelemetry tracing application, create a Maven project with the required dependencies and directory structure. Prerequisites You have installed Maven. For more information, see Downloading Apache Maven . You have configured your Maven repository for the latest release. For information about installing the latest Maven repository patch, see Maven and the JBoss EAP microprofile maven repository . Procedure In the CLI, use the mvn command to set up a Maven project. This command creates the directory structure for the project and the pom.xml configuration file. Syntax Example Navigate to the application root directory. Syntax Example Update the generated pom.xml file. Set the following properties: <properties> <maven.compiler.source>1.8</maven.compiler.source> <maven.compiler.target>1.8</maven.compiler.target> <failOnMissingWebXml>false</failOnMissingWebXml> <version.server.bom>4.0.0.GA</version.server.bom> <version.wildfly-jar.maven.plugin>6.1.1.Final</version.wildfly-jar.maven.plugin> </properties> Set the following dependencies: <dependencies> <dependency> <groupId>jakarta.enterprise</groupId> <artifactId>jakarta.enterprise.cdi-api</artifactId> <version>2.0.2</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.jboss.spec.javax.ws.rs</groupId> <artifactId>jboss-jaxrs-api_2.1_spec</artifactId> <version>2.0.2.Final</version> <scope>provided</scope> </dependency> <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-api</artifactId> <version>1.5.0</version> <scope>provided</scope> </dependency> </dependencies> Set the following build configuration to use mvn widlfy:deploy to deploy the application: <build> <!-- Set the name of the archive --> <finalName>USD{project.artifactId}</finalName> <plugins> <!-- Allows to use mvn wildfly:deploy --> <plugin> <groupId>org.wildfly.plugins</groupId> <artifactId>wildfly-maven-plugin</artifactId> </plugin> </plugins> </build> Verification In the application root directory, enter the following command: You get an output similar to the following: You can now create an OpenTelemetry tracing application. Additional resources Creating applications that create custom spans 10.7.2. Creating applications that create custom spans The following procedure demonstrates how to create an application that can create two custom spans like these: prepare-hello - When the method getHello() in the application is called. process-hello - When the value hello is assigned to a new String object hello . This procedure also demonstrates how to view these spans in a Jaeger console. <application_root> in the procedure denotes the directory that contains the pom.xml file, which contains the Maven configuration for your application. Prerequisites You have installed Docker. For more information, see Get Docker . You have created a Maven project. For more information, see Configuring Maven project for OpenTelemetry tracing . You have configured the the opentelemetry subsystem. For more information, see Configuring the opentelemetry subsystem . Procedure In the <application_root> , create a directory to store the Java files. Syntax Example Navigate to the new directory. Syntax Example Create a JakartaRestApplication.java file with the following content. This JakartaRestApplication class declares the application as a Jakarta RESTful Web Services application. package com.example.opentelemetry; import javax.ws.rs.ApplicationPath; import javax.ws.rs.core.Application; @ApplicationPath("/") public class JakartaRestApplication extends Application { } Create an ExplicitlyTracedBean.java file with the following content for the class ExplicitlyTracedBean . This class creates custom spans by injecting a Tracer class. package com.example.opentelemetry; import javax.enterprise.context.RequestScoped; import javax.inject.Inject; import io.opentelemetry.api.trace.Span; import io.opentelemetry.api.trace.Tracer; @RequestScoped public class ExplicitlyTracedBean { @Inject private Tracer tracer; 1 public String getHello() { Span prepareHelloSpan = tracer.spanBuilder("prepare-hello").startSpan(); 2 prepareHelloSpan.makeCurrent(); String hello = "hello"; Span processHelloSpan = tracer.spanBuilder("process-hello").startSpan(); 3 processHelloSpan.makeCurrent(); hello = hello.toUpperCase(); processHelloSpan.end(); prepareHelloSpan.end(); return hello; } } 1 Inject a Tracer class to create custom spans. 2 Create a span called prepare-hello to indicate that the method getHello() was called. 3 Create a span called process-hello to indicate that the value hello was assigned to a new String object called hello . Create a TracedResource.java file with the following content for TracedResource class. This file injects the ExplicitlyTracedBean class and declares two endpoints: traced and cdi-trace . package com.example.opentelemetry; import javax.enterprise.context.RequestScoped; import javax.inject.Inject; import javax.ws.rs.GET; import javax.ws.rs.Path; import javax.ws.rs.Produces; import javax.ws.rs.core.MediaType; @Path("/hello") @RequestScoped public class TracedResource { @Inject private ExplicitlyTracedBean tracedBean; @GET @Path("/traced") @Produces(MediaType.TEXT_PLAIN) public String hello() { return "hello"; } @GET @Path("/cdi-trace") @Produces(MediaType.TEXT_PLAIN) public String cdiHello() { return tracedBean.getHello(); } } Navigate to the application root directory. Syntax Example Compile and deploy the application with the following command: Start the Jaeger console. In a browser, navigate to \localhost:8080/simple-tracing-example/hello/cdi-trace . In a browser, open the Jaeger console at http://localhost:16686/search . In the Jaeger console, select JBoss EAP XP and click Find Traces . Click 3 Spans . The Jaeger console displays the following traces: 1 This is the span for the automatic implicit trace. 2 The custom span prepare-hello indicates that the method getHello() was called. It is the child of span for the automatic implicit trace. 3 The custom span process-hello indicates that the value hello was assigned to a new String object hello . It is the child of the prepare-hello span. Whenever you access the application endpoint at http://localhost:16686/search , a new trace is created with all the child spans. Additional resources OpenTelemetry tracing in JBoss EAP

Commands:
<subsystem xmlns="urn:wildfly:opentelemetry:1.0" service-name="example"> <exporter type="jaeger" endpoint="http://localhost:14250"/> <span-processor type="batch" batch-delay="4500" max-queue-size="128" max-export-batch-size="512" export-timeout="45"/> <sampler type="on"/> </subsystem>
/extension=org.wildfly.extension.opentelemetry:add
/subsystem=opentelemetry:add
reload
/subsystem=opentelemetry:write-attribute(name=exporter-type, value= <exporter_type> )
/subsystem=opentelemetry:write-attribute(name=exporter-type, value=jaeger)
/subsystem=opentelemetry:write-attribute(name=endpoint, value= <URL:port> )
/subsystem=opentelemetry:write-attribute(name=endpoint, value=http:localhost:14250)
/subsystem=opentelemetry:write-attribute(name=service-name, value= <service_name> )
/subsystem=opentelemetry:write-attribute(name=service-name, value=exampleOpenTelemetryService)
docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14268:14268 -p 14250:14250 -p 9411:9411 jaegertracing/all-in-one:1.29
mvn clean install wildfly:deploy
mvn archetype:generate -DgroupId= <group-to-which-your-application-belongs> -DartifactId= <name-of-your-application> -DarchetypeGroupId=org.apache.maven.archetypes -DarchetypeArtifactId=maven-archetype-webapp -DinteractiveMode=false
mvn archetype:generate -DgroupId=com.example.opentelemetry -DartifactId=simple-tracing-example -DarchetypeGroupId=org.apache.maven.archetypes -DarchetypeArtifactId=maven-archetype-webapp -DinteractiveMode=false
cd <name-of-your-application>
cd simple-tracing-example
<properties> <maven.compiler.source>1.8</maven.compiler.source> <maven.compiler.target>1.8</maven.compiler.target> <failOnMissingWebXml>false</failOnMissingWebXml> <version.server.bom>4.0.0.GA</version.server.bom> <version.wildfly-jar.maven.plugin>6.1.1.Final</version.wildfly-jar.maven.plugin> </properties>
<dependencies> <dependency> <groupId>jakarta.enterprise</groupId> <artifactId>jakarta.enterprise.cdi-api</artifactId> <version>2.0.2</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.jboss.spec.javax.ws.rs</groupId> <artifactId>jboss-jaxrs-api_2.1_spec</artifactId> <version>2.0.2.Final</version> <scope>provided</scope> </dependency> <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-api</artifactId> <version>1.5.0</version> <scope>provided</scope> </dependency> </dependencies>
<build> <!-- Set the name of the archive --> <finalName>USD{project.artifactId}</finalName> <plugins> <!-- Allows to use mvn wildfly:deploy --> <plugin> <groupId>org.wildfly.plugins</groupId> <artifactId>wildfly-maven-plugin</artifactId> </plugin> </plugins> </build>
mvn install
[INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1.440 s [INFO] Finished at: 2021-12-27T14:45:12+05:30 [INFO] ------------------------------------------------------------------------
mkdir -p src/main/java/com/example/opentelemetry
mkdir -p src/main/java/com/example/opentelemetry
cd src/main/java/com/example/opentelemetry
cd src/main/java/com/example/opentelemetry
package com.example.opentelemetry; import javax.ws.rs.ApplicationPath; import javax.ws.rs.core.Application; @ApplicationPath("/") public class JakartaRestApplication extends Application { }
package com.example.opentelemetry; import javax.enterprise.context.RequestScoped; import javax.inject.Inject; import io.opentelemetry.api.trace.Span; import io.opentelemetry.api.trace.Tracer; @RequestScoped public class ExplicitlyTracedBean { @Inject private Tracer tracer; 1 public String getHello() { Span prepareHelloSpan = tracer.spanBuilder("prepare-hello").startSpan(); 2 prepareHelloSpan.makeCurrent(); String hello = "hello"; Span processHelloSpan = tracer.spanBuilder("process-hello").startSpan(); 3 processHelloSpan.makeCurrent(); hello = hello.toUpperCase(); processHelloSpan.end(); prepareHelloSpan.end(); return hello; } }
package com.example.opentelemetry; import javax.enterprise.context.RequestScoped; import javax.inject.Inject; import javax.ws.rs.GET; import javax.ws.rs.Path; import javax.ws.rs.Produces; import javax.ws.rs.core.MediaType; @Path("/hello") @RequestScoped public class TracedResource { @Inject private ExplicitlyTracedBean tracedBean; @GET @Path("/traced") @Produces(MediaType.TEXT_PLAIN) public String hello() { return "hello"; } @GET @Path("/cdi-trace") @Produces(MediaType.TEXT_PLAIN) public String cdiHello() { return tracedBean.getHello(); } }
cd <path_to_application_root> / <application_root>
cd ~/applications/simple-tracing-example
mvn clean package wildfly:deploy
docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14268:14268 -p 14250:14250 -p 9411:9411 jaegertracing/all-in-one:1.29
|GET /hello/cdi-trace 1 - | prepare-hello 2 - | process-hello 3