Remote Demo Dataset

In ClickStack, traces are also automatically correlated with both logs and metrics. Let's explore the trace linked to our selected log to identify the service responsible.

Select `Trace` to visualize the associated trace. Scrolling down through the subsequent view we can see how HyperDX is able to visualize the distributed trace across the microservices, connecting the spans in each service. A payment clearly involves multiple microservices, including those that performance checkout and currency conversions.

By scrolling to the bottom of the view we can see that the `payment` service is causing the error, which in turn propagates back up the call chain.

Searching traces

### Searching traces[​](#searching-traces "Direct link to Searching traces")

We have established users are failing to complete purchases due to a cache issue in the payment service. Let's explore the traces for this service in more detail to see if we can learn more about the root cause.

Switch to the main Search view by selecting `Search`. Switch the data source for `Traces` and select the `Results table` view. **Ensure the timespan is still over the last day.**

This view shows all traces in the last day. We know the issue originates in our payment service, so apply the `payment` filter to the `ServiceName`.

If we apply event clustering to the traces by selecting `Event Patterns`, we can immediately see our cache issue with the `payment` service.

Explore infrastructure for a trace

### Explore infrastructure for a trace[​](#explore-infrastructure-for-a-trace "Direct link to Explore infrastructure for a trace")

Switch to the results view by clicking on `Results table`. Filter to errors using the `StatusCode` filter and `Error` value.

Select a `Error: Visa cache full: cannot add new item.` error, switch to the `Infrastructure` tab and widen the timespan to `1d`.

By correlating traces with metrics we can see that memory and CPU increased with the `payment` service, before collapsing to `0` (we can attribute this to a pod restart) - suggesting the cache issue caused resource issues. We can expect this has impacted payment completion times.

Event deltas for faster resolution

### Event deltas for faster resolution[​](#event-deltas-for-faster-resolution "Direct link to Event deltas for faster resolution")

Event Deltas help surface anomalies by attributing changes in performance or error rates to specific subsets of data—making it easier to quickly pinpoint the root cause.

While we know that the `payment` service has a cache issue, causing an increase in resource consumption, we haven't fully identified the root cause.

Return to the result table view and select the time period containing the errors to limit the data. Ensure you select several hours to the left of the errors and after if possible (the issue may still be occurring):

Remove the errors filter and select `Event Deltas` from the left `Analysis Mode` menu.

The top panel shows the distribution of timings, with colors indicating event density (number of spans). The subset of events outside of the main concentration are typically those worth investigating.

If we select the events with a duration greater than `200ms`, and apply the filter `Filter by selection`, we can limit our analysis to slower events:

With analysis performed on the subset of data, we can see most performance spikes are associated with `visa` transactions.

Using charts for more context

### Using charts for more context[​](#using-charts-for-more-context "Direct link to Using charts for more context")

In ClickStack, we can chart any numeric value from logs, traces, or metrics for greater context.

We have established:

- Our issue resides with the payment service
- A cache is full
- This caused increases in resource consumption
- The issue prevented visa payments from completing - or at least causing them to take a long time to complete.

  

Select `Chart Explorer` from the left menu. Complete the following values to chart the time taken for payments to complete by chart type:

- `Data Source`: `Traces`
- `Metric`: `Maximum`
- `SQL Column`: `Duration`
- `Where`: `ServiceName: payment`
- `Timespan`: `Last 1 day`

  

Clicking `▶️` will show how the performance of payments degraded over time.

If we set `Group By` to `SpanAttributes['app.payment.card_type']` (just type `card` for autocomplete) we can see how the performance of the service degraded for Visa transactions relative to Mastercard:

Note than once the error occurs responses return in `0s`.

Exploring metrics more context

### Exploring metrics more context[​](#exploring-metrics-for-more-context "Direct link to Exploring metrics more context")

Finally, let's plot the cache size as a metric to see how it behaved over time, thus giving us more context.

Complete the following values:

- `Data Source`: `Metrics`
- `Metric`: `Maximum`
- `SQL Column`: `visa_validation_cache.size (gauge)` (just type `cache` for autocomplete)
- `Where`: `ServiceName: payment`
- `Group By`: `<empty>`

We can see how the cache size increased over a 4-5 hr period (likely after a software deployment) before reaching a maximum size of `100,000`. From the `Sample Matched Events` we can see our errors correlate with the cache reaching this limit and, after which it is recorded as having a size of `0` with responses also returning in `0s`.

In summary, by exploring logs, traces and finally metrics we have concluded:

- Our issue resides with the payment service
- A change in service behavior, likely due to a deployment, resulted in a slow increase of a visa cache over a 4-5 hr period - reaching a maximum size of `100,000`.
- This caused increases in resource consumption as the cache grew in size - likely due to a poor implementation
- As the cache grew, the performance of Visa payments degraded
- On reaching the maximum size, the cache rejected payments and reported itself as size `0`.

Using sessions

### Using sessions[​](#using-sessions "Direct link to Using sessions")

Sessions allow us to replay the user experience, offering a visual account of how an error occurred from the user's perspective. While not typically used to diagnose root causes, they're valuable for confirming issues reported to customer support and can serve as a starting point for deeper investigation.

In HyperDX, sessions are linked to traces and logs, providing a complete view of the underlying cause.

For example, if the support team provides the email of a user who encountered a payment issue `[email protected]` - it's often more effective to begin with their session rather than directly searching logs or traces.

Navigate to the `Client Sessions` tab from the left menu before ensuring the data source is set to `Sessions` and the time period is set to the `Last 1 day`:

Search for `SpanAttributes.userEmail: Braulio` to find our customer's session. Selecting the session will show the browser events and associated spans for the customer's session on the left, with the user's browser experience re-rendered to the right:

Replaying sessions

### Replaying sessions[​](#replaying-sessions "Direct link to Replaying sessions")

Sessions can be replayed by pressing the ▶️ button. Switching between `Highlighted` and `All Events` allows varying degrees of span granularity, with the former highlighting key events and errors.

If we scroll to the bottom of the spans we can see a `500` error associated with `/api/checkout`. Selecting the ▶️ button for this specific span moves the replay to this point in the session, allowing us to confirm the customer's experience - payment seems to simply not work with no error rendered.

Selecting the span we can confirm this was caused by an internal error. By clicking the `Trace` tab and scrolling though the connected spans, we're able to confirm the customer indeed was a victim of our cache issue.