## 3 The Main Idea  
This section describes our multi-layer strategy with the final goal of application troubleshooting. One common approach to collecting traces is to use a tracer based on OpenTelemetry, Zipkin, Jaeger, etc. The tracer is configured to forward traces, metrics, and logs from an application to a proxy. The proxy securely, quickly, and reliably sends data to a Trace Manager. A cloud-scale application generates a large number of traces. The proxy should be configured to sample data to reduce the volume. A common strategy is to apply head-based sampling, also known as a rate-based strategy, which randomly samples without going deep into the context (see Figure 1).  
In our case, regardless of the sampling strategy, the tracer allows all error spans. Based on the traces, it collects and reports request, error, and duration (RED) metrics to provide full application observability. RED metrics measure requests (the number of requests being served per second), errors (the number of failed requests per second), and durations (per-minute histogram distributions of the amount of time that each request takes). These metrics can be used for application troubleshooting.  
The Trace Manager collects and samples traces based on their properties, such as type, duration, and error, although other properties can be considered. Then, the Trace Manager visualizes and analyzes the traces. At this stage, the main purpose of the property-based sampling is to preserve all available properties for further analysis. Trace-type sampling involves categorizing traces based on their characteristics or origins within the system. On the other hand, duration-based sampling prioritizes the collection of traces associated with significant delays or performance bottlenecks.  
Error-based sampling identifies and retains traces associated with errors, exceptions, or other abnormal conditions within the application. Combining these approaches allows the system to store more diverse traces, ensuring that valuable information across different categories and performance metrics is captured. By isolating and storing these traces, developers and system administrators have the necessary data to diagnose and address issues effectively, improving overall system reliability and performance.  
However, the basic purpose of the Trace Manager is application troubleshooting in the case of performance degradation. It has diverse tools for different management tasks. One of them is the Traces Browser, which can reveal the context and details of traces. In the Traces Browser, one can search for traces that include spans for a particular operation or examine the spans that belong to a selected trace. It is also possible to view the corresponding RED metrics for troubleshooting purposes.  
Another sophisticated troubleshooting engine is an applications map that overviews how the applications and services are linked. It allows us to focus on a specific service, view RED metrics for each service, and filter out trace traffic for the malfunctioning microservices. Here, noise reduction is performed based on the rate-based sampling of erroneous traces. It preserves all dominant normal and erroneous traces and removes the rare ones. This approach is acceptable when the problem has already been detected, and the goal is to explain it.  
The final layer applies XAI on the sampled data for RCA, bottleneck detection, and optimizations. In particular, rule learning systems which applied to the reduced tracing traffic  
Figure 1: Trace sampling multi-layer design.  
can generate recommendations in the form of rules which should be inspected by system administrators to accelerate the remediation of IT issues.