Question: "How do I troubleshoot missing X-Ray traces, segments, or services in the service map?"

I want to troubleshoot my missing AWS X-Ray traces, missing segments, and missing services in the service map.

Short descriptionMissing services in the service map are caused by missing traces. The causes that lead to missing traces are:Improper instrumentationX-Ray SDK can't reach the X-Ray daemon over the daemon address or port numberX-Ray daemon can't reach the X-Ray service endpointTracing isn't configured at the individual service levelMissing AWS Identity and Access Management (IAM) permissionsSampling rules configurationMissing configuration in the OpenTelemetry CollectorMissing segments are caused by the absence of open parent segments to the sub segments. For more information, see troubleshooting AWS X-Ray.

ResolutionImproper instrumentationImproper instrumentation when using X-Ray SDKIf your application code isn't instrumented properly for X-Ray SDK to patch the supported libraries and frameworks, then your traces will be missing. Missing traces are also caused by trace IDs that aren't passed to the downstream services.To troubleshoot this, turn on debug-level logging on the SDK to output more detailed logs to your application log file. This allows you to further isolate issues related to the instrumentation and to trace the flow of the trace ID across the application. The logs also indicate if a trace is missing because it's not sampled or because of instrumentation misconfiguration. For more information on debug logging, see X-Ray SDK for Java, X-Ray SDK for Node.js, X-Ray SDK for Python, X-Ray SDK for .NET, X-Ray SDK for Ruby.Note: If the application is using an auto-instrumentation agent for Java, then it doesn't capture traces for asynchronous requests and traces will be missing. Consider manual instrumentation with SDK if you have missing traces.Improper instrumentation when using OpenTelemetry SDKIf you use OpenTelemetry SDK and the application code isn't instrumented properly to patch the supported libraries and frameworks, then your traces will be missing. Missing segments are also caused by trace IDs that aren't passed to the downstream services.To troubleshoot this, do the following:Turn on debug-level logging on the SDK to send more detailed logs to your application log file. This allows you to further isolate issues related to the instrumentation and to trace the flow of the trace ID across the application. Logs indicate if a trace is missing because it's not sampled or because of instrumentation misconfiguration. For more information on debug logging, see OpenTelemetry JavaScript SDK, OpenTelemetry Java SDK, OpenTelemetry Python SDK, OpenTelemetry Ruby SDK, or OpenTelemetry PHP SDK.Make sure that your instrumentation isn't turned off. For more information, see suppressing specific auto-instrumentation.Note: For enriching traces with AWS infrastructure information, make sure that the AWS Detector is supported for that AWS service in the OpenTelemetry SDK. For more information, see using the AWS resource Detectors.X-Ray SDK can't reach the X-Ray daemon over the daemon address or port numberIf X-Ray SDK can't reach the X-Ray daemon over the default or configured daemon port number and address, then traces will be missing. The X-Ray daemon address is passed to SDK with the environmental variable AWS_XRAY_DAEMON_ADDRESS. By default, the X-ray Daemon listens on port 2000 UDP. This port can be changed through the command line option and the configuration file passed to the X-ray daemon. For more information, see using a configuration file to configure the X-Ray daemon.If you are running the X-Ray daemon with debug logs, then identify the UDP port configured for the X-Ray daemon using the -l debug command:./xray -l debugNote: The following example identifies 3000 as the port number.Example output:2023-03-28T15:15:43-07:00 [Debug] Listening on UDP 127.0.0.1:3000If you aren't running the X-Ray Daemon with debug logs, then turn on the option to debug logs and re-run the daemon. For more information, see Using a configuration file.Note: If your X-Ray daemon isn't running on the same machine as your application and X-Ray SDK, then check the inbound rule and ingress rule. Check that the security group inbound rule and network access control list (network ACL) ingress rule are allowing traffic. Traffic must be allowed for X-Ray SDK over the listener port of the X-Ray daemon.X-Ray daemon can't reach the X-Ray service endpointIf your X-Ray daemon can't reach the X-Ray service endpoint, then check the following:If you have proxy configured at the source end, then check whether proxy is inhibiting the traffic from reaching the X-Ray service endpoint.If you're using the X-Ray Amazon Virtual Private Cloud (Amazon VPC) endpoint com.amazonaws.region.xray to connect to the X-Ray service, then check the security group. Make sure that the security group attached has outbound traffic that allows HTTPS traffic to push data to the X-Ray service API. Also, make sure that the security group allows inbound traffic on port 443 to allow X-Ray daemon.Verify that the associated network ACL and X-Ray Amazon VPC endpoint have an egress rule that allows traffic on port 443 for everywhere 0.0.0.0/0. The X-Ray daemon must send data to the X-Ray service endpoint. Also, verify that the network ACL ingress rule allows traffic from the X-Ray daemon to the Amazon VPC endpointTracing isn't configured at the individual service levelWhen X-Ray configuration isn't turned on for certain AWS services, traces will be missing. The following is a list of AWS services that require X-Ray configuration to be turned on at the service level:AWS AppSyncAWS App RunnerAmazon API GatewayAmazon CloudWatch real user monitoring (CloudWatch RUM)AWS LambdaAmazon Simple Notification Service (Amazon SNS)AWS Step FunctionsMissing AWS Identity and Access Management (IAM) permissionsIf you receive Access Denied errors in the X-Ray daemon logs, then check the IAM permissions associated with the IAM role that used by the X-Ray daemon. To send the traces to the X-Ray service endpoint, the X-Ray daemon requires permissions to be defined in the managed policy AWSXRAYDaemonWriteAccess. If you're using a tags-based condition in your IAM policy, then make sure that the IAM policy refers to the correct tag. For more information, see Authorization based on X-Ray tags.If X-Ray daemon is running in Amazon Elastic Container Service (Amazon ECS) as a sidecar container in the task, then check the Amazon ECS task role. Make sure that the Amazon ECS task role has the AWSXRayDaemonWriteAccess policy attached.If X-Ray daemon is deployed in Amazon Elastic Kubernetes Services (Amazon EKS), then check the sidecar container in the pod. Make sure the container is using IAM roles for service accounts (IRSA) - Amazon EKS and the IAM role associated has the AWSXRayDaemonWriteAccess policy attached.If X-ray daemon is configured in Amazon Elastic Compute Cloud (Amazon EC2), then check the associated instance profile. Make sure that the instance profile role associated with that EC2 Instance has the AWSXRayDaemonWriteAccess policy attached.If X-Ray daemon is configured in AWS Lambda, then check the Lambda execution role. Confirm whether the Lambda execution role has the AWSXRayDaemonWriteAccess policy and if it must be attached.If you have active tracing turned on in Amazon SNS and you still can’t see your traces, then configure a resource policy in X-Ray.Sampling rules configurationIf traces aren't sampled by X-Ray SDK and AWS services that support active tracing with sampling configuration, then the traces will be missing. Use sampling rules to determine the requests to record. By default, the X-Ray SDK records the first request each second and five percent of any additional requests. You can customize this, and you can change the sampling rate to decrease or increase the number of traces recorded.Example scenario:Reservoir set to 1. This indicates first request of each second is taken.Sampling rate set to 0.1. This indicates ten percent of additional requests of each second is recorded.Due to the limit on the number of requests recorded by X-Ray SDK, all requests might not get traced. To record more traces, increase the reservoir and sampling rate settings. For more information, see sampling rule examples.Note: If multiple sampling rules are configured and the number of traces sampled fluctuates, then check the sampling rule that's used by X-Ray SDK. For more information, see viewing sampling results.Missing configuration in the OpenTelemetry CollectorIf you're using OpenTelemetry Collector instead of X-Ray daemon, then the required receiver and exporter configuration might be missing. A missing configuration cause the traces to not be sent to the X-Ray service endpoint. Based on your receiver, check the receiver configuration in the config.yaml file to verify there is no missing configuration.Example configurations:X-Ray exporterX-Ray receiverZipKin receiverNote: Check whether your collector has the required IAM permission defined in the managed policy AWSXRayDaemonWriteAccess.