Repository: open-telemetry/opentelemetry-python-contrib

Issue:
Providing Parent in X-Amzn-Trace-Id results in no spans being exported
There's a good chance this is user error. If so, I'd appreciate a pointer to the relevant doc.

**Describe your environment** 
- Python 3.9
- fastapi==0.65.3
- opentelemetry-api==1.4.1
- opentelemetry-exporter-otlp==1.4.1
- opentelemetry-exporter-otlp-proto-grpc==1.4.1
- opentelemetry-instrumentation==0.23b2
- opentelemetry-instrumentation-asgi==0.23b2
- opentelemetry-instrumentation-fastapi==0.23b2
- opentelemetry-proto==1.4.1
- opentelemetry-sdk==1.4.1
- opentelemetry-sdk-extension-aws==0.23b2
- opentelemetry-semantic-conventions==0.23b2
- opentelemetry-util-http==0.23b2

**Steps to reproduce**
Using this sample application:
```
import fastapi
import uvicorn
from opentelemetry import propagate, trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.extension.aws.trace import AwsXRayIdGenerator
from opentelemetry.sdk.extension.aws.trace.propagation.aws_xray_format import AwsXRayFormat
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

app = fastapi.FastAPI()

@app.get("/foo")
async def foo():
    return {"message": "foo"}


# Setup AWS X-Ray Propagator
propagate.set_global_textmap(AwsXRayFormat())

# Setup Tracer
otlp_exporter = OTLPSpanExporter()
span_processor = BatchSpanProcessor(otlp_exporter)

tracer_provider = TracerProvider(id_generator=AwsXRayIdGenerator())
tracer_provider.add_span_processor(span_processor)

trace.set_tracer_provider(tracer_provider)

FastAPIInstrumentor.instrument_app(app)

uvicorn.run(app)
```

Calling: `curl 'http://localhost:8000/foo'` produces a span that is exported by my collector to X-Ray.

Calling: `curl 'http://localhost:8000/foo' -H 'X-Amzn-Trace-Id: Root=1-612fa749-271fa48e4c544863a13425d5;Parent=86153bfee2237b3b'` does not export a span.

**What is the expected behavior?**
My frontend application is producing requests with `X-Amzn-Trace-Id` in the format above. The frontend is separately sending X-Ray data. I'm expecting the trace ID generated by the frontend to be the root of the of the server segment so I can correlate the frontend activity and the backend activity.

**What is the actual behavior?**
When providing the header from the frontend, no span is generated by the backend. The frontend succeeds in generating spans that I see in the X-Ray console. No backend segments are seen.

**Additional context**
Unsure if #445 is related. It's the only thing that I could find in the issues that might be.



Discussion:
FYI @NathanielRN 
I continued playing with this last night and was able to get this to work by adding `;Sampled=1` to the header. My team is exploring moving from using the aws xray sdk to opentelemetry. The behavior of our stack in the past was to not provide `Sampled` from our frontend (we get a small number of requests, so just sample all). The defaults from our instrumentation libraries and the xray sdk seem to use that as a signal to sample.

I'd refine my bug report, as
- Not providing `Sampled` should result in the span being recorded (equivalent of `Sampled=1`)
- The behavior in the absence of `Sampled` should be consistent for `Parent` supplied or not.
Thanks for providing this additional context @andrew-matteson ! It's very helpful 😄 

> Not providing `Sampled` should result in the span being recorded (equivalent of `Sampled=1`)

I looks like you are correct, if `Sampled=` is **not** provided, the trace is assumed to **not** be sampled by the `AwsXrayFormat` propagator when it extracts the context from the header:

https://github.com/open-telemetry/opentelemetry-python-contrib/blob/6f96577989698b9d7834abee2d172d47b24bebae/sdk-extension/opentelemetry-sdk-extension-aws/src/opentelemetry/sdk/extension/aws/trace/propagation/aws_xray_format.py#L154-L158

This behavior is consistent with what is done in other languages such as in the [OTel Java AWS X-Ray Propagator](https://github.com/open-telemetry/opentelemetry-java/blob/29cc44ed08dba28a65fa7d9f0c6824f38c51f6f2/extensions/aws/src/main/java/io/opentelemetry/extension/aws/AwsXrayPropagator.java#L172-L181) or in [OTel JS AWS X-Ray Propagator](https://github.com/open-telemetry/opentelemetry-js-contrib/blob/5fb3313b60a8d74af9ba8bacd260818545e4a1db/propagators/opentelemetry-propagator-aws-xray/src/AWSXRayPropagator.ts#L91-L103).

Your use case makes sense, however I'm not sure if sampling by default is what most users would want. Maybe we can make documentation about this better in the meantime.

> The behavior in the absence of `Sampled` should be consistent for `Parent` supplied or not.

I think based on your configuration, the behavior you saw was expected. In your code you are using `TracerProvider` without a custom `Sampler`, that means you get [the default](https://github.com/open-telemetry/opentelemetry-python/blob/main/opentelemetry-sdk/src/opentelemetry/sdk/trace/__init__.py#L1048) `Sampler`:

```python
class TracerProvider(trace_api.TracerProvider):
    """See `opentelemetry.trace.TracerProvider`."""

    def __init__(
        self,
        sampler: sampling.Sampler = _TRACE_SAMPLER,
```

The default `Sampler` is the `parentbased_alwayson` [sampler](https://github.com/open-telemetry/opentelemetry-python/blob/c1e8a51b752f8d71e3910f122f355b49bfc7dbfa/opentelemetry-sdk/src/opentelemetry/sdk/trace/sampling.py#L384):

```python
def _get_from_env_or_default() -> Sampler:
    trace_sampler = os.getenv(
        OTEL_TRACES_SAMPLER, "parentbased_always_on"
    ).lower()
```

which [will always sample](https://github.com/open-telemetry/opentelemetry-python/blob/c1e8a51b752f8d71e3910f122f355b49bfc7dbfa/opentelemetry-sdk/src/opentelemetry/sdk/trace/sampling.py#L357-L358) _unless the Parent is present to say otherwise_:

```python
DEFAULT_ON = ParentBased(ALWAYS_ON)
"""Sampler that respects its parent span's sampling decision, but otherwise always samples."""
```

So in your 2nd case, when you add the Parent header, your `TracerProvider` chooses to respect the Parent. Maybe if you want the trace to be `Sampled` without setting `Sampled=1` explicitly we could make the `AwsXrayFormat` propagator read from the currently configured `TraceProvider`'s `Sampler` to make the default sampling decision (instead of assuming `sampled = false`.

But to unblock you for now I would suggest using `Sampled=1` 🙂 

Please let me know if that makes sense!
@NathanielRN Thank you! Makes complete sense. I'm unblocked at this point.

Your doc suggestion sounds like the way to go.

Resolving patch:
diff --git a/sdk-extension/opentelemetry-sdk-extension-aws/src/opentelemetry/sdk/extension/aws/trace/propagation/aws_xray_format.py b/sdk-extension/opentelemetry-sdk-extension-aws/src/opentelemetry/sdk/extension/aws/trace/propagation/aws_xray_format.py
--- a/sdk-extension/opentelemetry-sdk-extension-aws/src/opentelemetry/sdk/extension/aws/trace/propagation/aws_xray_format.py
+++ b/sdk-extension/opentelemetry-sdk-extension-aws/src/opentelemetry/sdk/extension/aws/trace/propagation/aws_xray_format.py
@@ -18,9 +18,17 @@
 
 The **AWS X-Ray Propagator** provides a propagator that when used, adds a `trace
 header`_ to outgoing traces that is compatible with the AWS X-Ray backend service.
-This allows the trace context to be propagated when a trace span multiple AWS
+This allows the trace context to be propagated when a trace spans multiple AWS
 services.
 
+The same propagator setup is used to extract a context sent by external systems
+so that child span have the correct parent context.
+
+**NOTE**: Because the parent context parsed from the ``X-Amzn-Trace-Id`` header
+assumes the context is _not_ sampled by default, users should make sure to add
+``Sampled=1`` to their ``X-Amzn-Trace-Id`` headers so that the child spans are
+sampled.
+
 Usage
 -----
 


Test patch: