Repository: elastic/apm-agent-python

Issue:
KeyError: 'otel_attributes'
elastic-apm:  6.9.0

```
KeyError: 'otel_attributes'
  File "elasticapm/contrib/starlette/__init__.py", line 185, in __call__
    self.client.end_transaction()
  File "elasticapm/base.py", line 313, in end_transaction
    transaction = self.tracer.end_transaction(result, name, duration=duration)
  File "elasticapm/traces.py", line 898, in end_transaction
    transaction.end(duration=duration)
  File "elasticapm/traces.py", line 232, in end
    super().end(skip_frames, duration)
  File "elasticapm/traces.py", line 126, in end
    self.compression_buffer.report()
  File "elasticapm/traces.py", line 670, in report
    self.tracer.queue_func(SPAN, self.to_dict())
  File "elasticapm/traces.py", line 580, in to_dict
    attributes = self.context.pop("otel_attributes")
```


Discussion:
apmmachine: 
## :green_heart: Build Succeeded


<!-- BUILD BADGES-->
> _the below badges are clickable and redirect to their specific view in the CI or DOCS_
[![Pipeline View](https://img.shields.io/badge/pipeline-pipeline%20-green)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-python%2Fapm-agent-python-mbp%2FPR-1510/detail/PR-1510/1//pipeline) [![Test View](https://img.shields.io/badge/test-test-green)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-python%2Fapm-agent-python-mbp%2FPR-1510/detail/PR-1510/1//tests) [![Changes](https://img.shields.io/badge/changes-changes-green)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-python%2Fapm-agent-python-mbp%2FPR-1510/detail/PR-1510/1//changes) [![Artifacts](https://img.shields.io/badge/artifacts-artifacts-yellow)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-python%2Fapm-agent-python-mbp%2FPR-1510/detail/PR-1510/1//artifacts) [![preview](https://img.shields.io/badge/docs-preview-yellowgreen)](http://apm-agent-python_1510.docs-preview.app.elstc.co/diff)  [![preview](https://img.shields.io/badge/elastic-observability-blue)](https://ci-stats.elastic.co/app/apm/services/apm-ci/transactions/view?rangeFrom=2022-03-30T11:49:45.530Z&rangeTo=2022-03-30T12:09:45.530Z&transactionName=apm-agent-python/apm-agent-python-mbp/PR-{number}&transactionType=job&latencyAggregationType=avg&traceId=5ff76e8517469254fa1100fecd0da3cb&transactionId=3e752faf4686b077)

<!-- BUILD SUMMARY-->
<details><summary>Expand to view the summary</summary>
<p>

#### Build stats




* Start Time: 2022-03-30T11:59:45.530+0000


* Duration: 24 min 48 sec




#### Test stats :test_tube:

| Test         | Results                         |
| ------------ | :-----------------------------: |
| Failed       | 0  |
| Passed       | 4783  |
| Skipped      | 3209 |
| Total        | 7992   |


</p>
</details>

<!-- TEST RESULTS IF ANY-->


<!-- STEPS ERRORS IF ANY -->












## :green_heart: Flaky test report
Tests succeeded.



## :robot: GitHub comments

To re-run your PR in the CI, just comment with:



- `/test` : Re-trigger the build.


- `/test linters` : Run the Python linters only.


- `/test full` : Run the full matrix of tests.


- `/test benchmark` : Run the APM Agent Python benchmarks tests.


- `run` `elasticsearch-ci/docs` : Re-trigger the docs validation. (use unformatted text in the comment!)




<!--COMMENT_GENERATED_WITH_ID_comment.id-->

Patch:
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index 85cf3af4..462fd2e4 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -29,6 +29,19 @@ endif::[]
 //===== Bug fixes
 //
 
+
+=== Unreleased
+
+// Unreleased changes go here
+// When the next release happens, nest these changes under the "Python Agent version 6.x" heading
+//[float]
+//===== Features
+//
+[float]
+===== Bug fixes
+
+* Fix `otel_attributes`-related regression with older versions of APM Server (<7.16) {pull}1510[#1510]
+
 [[release-notes-6.x]]
 === Python Agent version 6.x
 
diff --git a/docs/opentracing.asciidoc b/docs/opentracing.asciidoc
new file mode 100644
index 00000000..65b12495
--- /dev/null
+++ b/docs/opentracing.asciidoc
@@ -0,0 +1,130 @@
+// TODO This doc should be removed in 7.0.0
+// It continues to exist because older versions point to the latest version on a major
+[[opentracing-bridge]]
+== OpenTracing API
+
+The Elastic APM OpenTracing bridge allows you to create Elastic APM `Transactions` and `Spans`,
+using the OpenTracing API.
+In other words,
+it translates calls to the OpenTracing API to Elastic APM events, which allows for the reuse of existing instrumentation.
+
+The first span of a service will be converted to an Elastic APM
+{apm-guide-ref}/transactions.html[`Transaction`],
+subsequent spans are mapped to Elastic APM
+{apm-guide-ref}/transaction-spans.html[`Span`].
+
+[float]
+[[opentracing-getting-started]]
+=== Getting started
+The first step in getting started with the OpenTracing API bridge is to install the `opentracing` library:
+
+[source,bash]
+----
+pip install elastic-apm[opentracing]
+----
+
+Or if you already have installed `elastic-apm`
+
+
+[source,bash]
+----
+pip install opentracing>=2.0.0
+----
+
+
+[float]
+[[opentracing-init-tracer]]
+=== Initialize tracer
+
+[source,python]
+----
+from elasticapm.contrib.opentracing import Tracer
+
+tracer = Tracer();
+----
+
+`Tracer` accepts the following optional arguments:
+
+  * `client_instance`: an already instantiated Elastic APM client
+  * `config`: a configuration dictionary, which will be used to instantiate a new Elastic APM client,
+     e.g. `{"SERVER_URL": "https://example.org"}`. See <<configuration, configuration>> for more information.
+  * `scope_manager`: a scope manager instance. Defaults to `ThreadLocalScopeManager` (see
+
+
+[float]
+[[opentracing-elastic-apm-tags]]
+=== Elastic APM specific tags
+
+Elastic APM defines some tags which are not included in the OpenTracing API but are relevant in the context of Elastic APM.
+
+- `type` - sets the type of the transaction,
+  for example `request`, `ext` or `db`
+- `user.id` - sets the user id,
+  appears in the "User" tab in the transaction details in the Elastic APM UI
+- `user.email` - sets the user email,
+  appears in the "User" tab in the transaction details in the Elastic APM UI
+- `user.username` - sets the user name,
+  appears in the "User" tab in the transaction details in the Elastic APM UI
+- `result` - sets the result of the transaction. Overrides the default value of `success`.
+
+NOTE: these tags need to be set on the first span of an operation (e.g. an incoming request of your webapp).
+
+[float]
+[[opentracing-caveats]]
+=== Caveats
+Not all features of the OpenTracing API are supported.
+
+[float]
+[[opentracing-scope-managers]]
+==== Scope Managers
+Currently, only the `ThreadLocalScopeManager` is supported.
+Using other scope managers will lead to unpredictable and possibly app-breaking behavior.
+
+[float]
+[[opentracing-instrumentation]]
+==== Instrumentation
+
+It is recommended to not use the built-in instrumentations of Elastic APM together with third-party OpenTracing instrumentations
+like https://pypi.org/project/opentracing_instrumentation/[opentracing_instrumentation] in the same service.
+If you would like to use such instrumentations, we recommend to disable the built-in instrumentation using the <<config-instrument,`instrument`>> config option.
+
+[float]
+[[opentracing-propagation]]
+==== Context propagation
+This bridge only supports the formats `Format.Builtin.TEXT_MAP` and `Format.Builtin.HTTP_HEADERS`.
+`Format.Builtin.BINARY` is currently not supported.
+
+[float]
+[[opentracing-references]]
+==== Span references
+Currently, this bridge only supports `child_of` references.
+Other references,
+like `follows_from` are not supported yet.
+
+[float]
+[[opentracing-baggage]]
+==== Baggage
+The `Span.set_baggage_item(key, value)` method is not supported.
+Baggage items are silently dropped.
+
+[float]
+[[opentracing-logs]]
+==== Logs
+Only exception logging is supported.
+Logging an Exception on the OpenTracing span will create an Elastic APM
+{apm-guide-ref}/errors.html[`Error`].
+Example:
+
+[source,python]
+----
+with tracer.start_active_span("transaction") as tx_scope:
+    try:
+        raise ValueError("oops")
+    except ValueError:
+        exc_type, exc_val, exc_tb = sys.exc_info()[:3]
+        tx_scope.span.log_kv({
+            "python.exception.type": exc_type,
+            "python.exception.val": exc_val,
+            "python.exception.tb": exc_tb
+        })
+----
diff --git a/elasticapm/traces.py b/elasticapm/traces.py
index 8d4df7db..815b9527 100644
--- a/elasticapm/traces.py
+++ b/elasticapm/traces.py
@@ -577,7 +577,7 @@ class Span(BaseSpan):
                         result["otel"]["attributes"] = self.context.pop("otel_attributes")
             else:
                 # Attributes map to labels for older versions
-                attributes = self.context.pop("otel_attributes")
+                attributes = self.context.pop("otel_attributes", {})
                 if attributes and ("tags" not in self.context):
                     self.context["tags"] = {}
                 for key, value in attributes.items():


Test patch:
diff --git a/tests/client/transaction_tests.py b/tests/client/transaction_tests.py
index 6ee0a7dc..1c7db2ed 100644
--- a/tests/client/transaction_tests.py
+++ b/tests/client/transaction_tests.py
@@ -38,6 +38,23 @@ from elasticapm.utils import encoding
 from elasticapm.utils.disttracing import TraceParent
 
 
+@pytest.mark.parametrize("elasticapm_client", [{"server_version": (7, 15)}, {"server_version": (7, 16)}], indirect=True)
+def test_transaction_span(elasticapm_client):
+    elasticapm_client.begin_transaction("test")
+    with elasticapm.capture_span("test", extra={"a": "b"}):
+        pass
+    elasticapm_client.end_transaction("test", constants.OUTCOME.SUCCESS)
+    transactions = elasticapm_client.events[constants.TRANSACTION]
+    assert len(transactions) == 1
+    assert transactions[0]["name"] == "test"
+    assert transactions[0]["type"] == "test"
+    assert transactions[0]["result"] == constants.OUTCOME.SUCCESS
+
+    spans = elasticapm_client.events[constants.SPAN]
+    assert len(spans) == 1
+    assert spans[0]["name"] == "test"
+
+
 @pytest.mark.parametrize(
     "elasticapm_client", [{"transactions_ignore_patterns": ["^OPTIONS", "views.api.v2"]}], indirect=True
 )