Chapter 7. Tracing CodeReady Workspaces Tracing helps gather timing data to troubleshoot latency problems in microservice architectures and helps to understand a complete transaction or workflow as it propagates through a distributed system. Every transaction may reflect performance anomalies in an early phase when new services are being introduced by independent teams. Tracing the CodeReady Workspaces application may help analyze the execution of various operations, such as workspace creations, workspace startup, breaking down the duration of sub-operations executions, helping finding bottlenecks and improve the overall state of the platform. Tracers live in applications. They record timing and metadata about operations that take place. They often instrument libraries, so that their use is indiscernible to users. For example, an instrumented web server records when it received a request and when it sent a response. The trace data collected is called a span . A span has a context that contains information such as trace and span identifiers and other kinds of data that can be propagated down the line. 7.1. Tracing API CodeReady Workspaces utilizes OpenTracing API - a vendor-neutral framework for instrumentation. This means that if a developer wants to try a different tracing back end, then rather than repeating the whole instrumentation process for the new distributed tracing system, the developer can simply change the configuration of the tracer back end. 7.2. Tracing back end By default, CodeReady Workspaces uses Jaeger as the tracing back end. Jaeger was inspired by Dapper and OpenZipkin, and it is a distributed tracing system released as open source by Uber Technologies. Jaeger extends a more complex architecture for a larger scale of requests and performance. 7.3. Installing the Jaeger tracing tool The following sections describe the installation methods for the Jaeger tracing tool. Jaeger can then be used for gathering metrics in CodeReady Workspaces. Installation methods available: Section 7.3.1, "Installing Jaeger using OperatorHub on OpenShift 4" Section 7.3.2, "Installing Jaeger using CLI on OpenShift 4" For tracing a CodeReady Workspaces instance using Jaeger, version 1.12.0 or above is required. For additional information about Jaeger, see the Jaeger website . 7.3.1. Installing Jaeger using OperatorHub on OpenShift 4 This section provide information about using Jaeger tracing tool for testing an evaluation purposes in production. To install the Jaeger tracing tool from the OperatorHub interface in OpenShift Container Platform, follow the instructions below. Prerequisites The user is logged in to the OpenShift Container Platform Web Console. A CodeReady Workspaces instance is available in a project. Procedure Open the OpenShift Container Platform console. From the left menu of the main OpenShift Container Platform screen, navigate to Operators OperatorHub . In the Search by keyword search bar, type Jaeger Operator . Click the Jaeger Operator tile. Click the Install button in the Jaeger Operator pop-up window. Select the installation method: A specific project on the cluster where the CodeReady Workspaces is deployed and leave the rest in its default values. Click the Subscribe button. From the left menu of the main OpenShift Container Platform screen, navigate to the Operators Installed Operators section. Red Hat CodeReady Workspaces is displayed as an Installed Operator, as indicated by the InstallSucceeded status. Click the Jaeger Operator name in the list of installed Operators. Navigate to the Overview tab. In the Conditions sections at the bottom of the page, wait for this message: install strategy completed with no errors . Jaeger Operator and additional Elasticsearch Operator is installed. Navigate to the Operators Installed Operators section. Click Jaeger Operator in the list of installed Operators. The Jaeger Cluster page is displayed. In the lower left corner of the window, click Create Instance Click Save . OpenShift creates the Jaeger cluster jaeger-all-in-one-inmemory . Follow the steps in Enabling metrics collection to finish the procedure. 7.3.2. Installing Jaeger using CLI on OpenShift 4 This section provide information about using Jaeger tracing tool for testing an evaluation purposes. To install the Jaeger tracing tool from a CodeReady Workspaces project in OpenShift Container Platform, follow the instructions in this section. Prerequisites The user is logged in to the OpenShift Container Platform web console. A instance of CodeReady Workspaces in an OpenShift Container Platform cluster. Procedure In the CodeReady Workspaces installation project of the OpenShift Container Platform cluster, use the oc client to create a new application for the Jaeger deployment. Using the Workloads Deployments from the left menu of main OpenShift Container Platform screen, monitor the Jaeger deployment until it finishes successfully. Select Networking Routes from the left menu of the main OpenShift Container Platform screen, and click the URL link to access the Jaeger dashboard. Follow the steps in Enabling metrics collection to finish the procedure. 7.4. Enabling metrics collection Prerequisites Installed Jaeger v1.12.0 or above. See instructions at Section 7.3, "Installing the Jaeger tracing tool" Procedure For Jaeger tracing to work, enable the following environment variables in your CodeReady Workspaces deployment: # Activating CodeReady Workspaces tracing modules CHE_TRACING_ENABLED=true # Following variables are the basic Jaeger client library configuration. JAEGER_ENDPOINT="http://jaeger-collector:14268/api/traces" # Service name JAEGER_SERVICE_NAME="che-server" # URL to remote sampler JAEGER_SAMPLER_MANAGER_HOST_PORT="jaeger:5778" # Type and param of sampler (constant sampler for all traces) JAEGER_SAMPLER_TYPE="const" JAEGER_SAMPLER_PARAM="1" # Maximum queue size of reporter JAEGER_REPORTER_MAX_QUEUE_SIZE="10000" To enable the following environment variables: In the yaml source code of the CodeReady Workspaces deployment, add the following configuration variables under spec.server.customCheProperties . customCheProperties: CHE_TRACING_ENABLED: 'true' JAEGER_SAMPLER_TYPE: const DEFAULT_JAEGER_REPORTER_MAX_QUEUE_SIZE: '10000' JAEGER_SERVICE_NAME: che-server JAEGER_ENDPOINT: 'http://jaeger-collector:14268/api/traces' JAEGER_SAMPLER_MANAGER_HOST_PORT: 'jaeger:5778' JAEGER_SAMPLER_PARAM: '1' Edit the JAEGER_ENDPOINT value to match the name of the Jaeger collector service in your deployment. From the left menu of the main OpenShift Container Platform screen, obtain the value of JAEGER_ENDPOINT by navigation to Networking Services . Alternatively, execute the following oc command: The requested value is included in the service name that contains the collector string. Additional resources For additional information about custom environment properties and how to define them in CheCluster Custom Resource, see https://access.redhat.com/documentation/en-us/red_hat_codeready_workspaces/2.15/html-single/installation_guide/index#advanced-configuration-options-for-the-che-server-component.adoc . For custom configuration of Jaeger, see the list of Jaeger client environment variables . 7.5. Viewing CodeReady Workspaces traces in Jaeger UI This section demonstrates how to use the Jaeger UI to overview traces of CodeReady Workspaces operations. Procedure In this example, the CodeReady Workspaces instance has been running for some time and one workspace start has occurred. To inspect the trace of the workspace start: In the Search panel on the left, filter spans by the operation name (span name), tags, or time and duration. Figure 7.1. Using Jaeger UI to trace CodeReady Workspaces Select the trace to expand it and show the tree of nested spans and additional information about the highlighted span, such as tags or durations. Figure 7.2. Expanded tracing tree 7.6. CodeReady Workspaces tracing codebase overview and extension guide The core of the tracing implementation for CodeReady Workspaces is in the che-core-tracing-core and che-core-tracing-web modules. All HTTP requests to the tracing API have their own trace. This is done by TracingFilter from the OpenTracing library , which is bound for the whole server application. Adding a @Traced annotation to methods causes the TracingInterceptor to add tracing spans for them. 7.6.1. Tagging Spans may contain standard tags, such as operation name, span origin, error, and other tags that may help users with querying and filtering spans. Workspace-related operations (such as starting or stopping workspaces) have additional tags, including userId , workspaceID , and stackId . Spans created by TracingFilter also have an HTTP status code tag. Declaring tags in a traced method is done statically by setting fields from the TracingTags class: TracingTags.WORKSPACE_ID.set(workspace.getId()); TracingTags is a class where all commonly used tags are declared, as respective AnnotationAware tag implementations. Additional resources For more information about how to use Jaeger UI, visit Jaeger documentation: Jaeger Getting Started Guide .

Commands:
oc new-app -f / USD{CHE_LOCAL_GIT_REPO}/deploy/openshift/templates/jaeger-all-in-one-template.yml: --> Deploying template " <project_name> /jaeger-template-all-in-one" for "/home/user/crw-projects/crw/deploy/openshift/templates/jaeger-all-in-one-template.yml" to project <project_name> Jaeger (all-in-one) --------- Jaeger Distributed Tracing Server (all-in-one) * With parameters: * Jaeger Service Name=jaeger * Image version=latest * Jaeger Zipkin Service Name=zipkin --> Creating resources deployment.apps "jaeger" created service "jaeger-query" created service "jaeger-collector" created service "jaeger-agent" created service "zipkin" created route.route.openshift.io "jaeger-query" created --> Success Access your application using the route: 'jaeger-query- <project_name> .apps.ci-ln-whx0352-d5d6b.origin-ci-int-aws.dev.rhcloud.com' Run 'oc status' to view your app.
Activating CodeReady Workspaces tracing modules CHE_TRACING_ENABLED=true Following variables are the basic Jaeger client library configuration. JAEGER_ENDPOINT="http://jaeger-collector:14268/api/traces" Service name JAEGER_SERVICE_NAME="che-server" URL to remote sampler JAEGER_SAMPLER_MANAGER_HOST_PORT="jaeger:5778" Type and param of sampler (constant sampler for all traces) JAEGER_SAMPLER_TYPE="const" JAEGER_SAMPLER_PARAM="1" Maximum queue size of reporter JAEGER_REPORTER_MAX_QUEUE_SIZE="10000"
customCheProperties: CHE_TRACING_ENABLED: 'true' JAEGER_SAMPLER_TYPE: const DEFAULT_JAEGER_REPORTER_MAX_QUEUE_SIZE: '10000' JAEGER_SERVICE_NAME: che-server JAEGER_ENDPOINT: 'http://jaeger-collector:14268/api/traces' JAEGER_SAMPLER_MANAGER_HOST_PORT: 'jaeger:5778' JAEGER_SAMPLER_PARAM: '1'
oc get services
TracingTags.WORKSPACE_ID.set(workspace.getId());