Repository: ansible-collections/community.general

Issue:
open telemetry callback error: Transient error StatusCode.UNAVAILABLE 
### Summary

We have configured Ansible withe the Opentelemetry plugin
and the Grafanacloud confugurations suggested on the https://docs.ansible.com/ansible/latest/collections/community/general/opentelemetry_callback.html page, including the token.


### Issue Type

Bug Report

### Component Name

community.general.opentelemetry

### Ansible Version

```console (paste below)
$ ansible --version
ansible [core 2.13.1]
  config file = None
  configured module search path = ['/REDACTED/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /REDACTED/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.17 (main, Jun 15 2023, 07:46:17) [Clang 14.0.3 (clang-1403.0.22.14.1)]
  jinja version = 3.1.2
  libyaml = True
```


### Community.general Version

```console (paste below)
$ ansible-galaxy collection list community.general


# /REDACTED/.ansible/collections/ansible_collections
Collection        Version
----------------- -------
community.general 8.2.0
```


### Configuration

```console (paste below)
$ ansible-config dump --only-changed
CALLBACKS_ENABLED(/REDACTED/ansible.cfg) = ['community.general.opentelemetry']
CONFIG_FILE() = /REDACTED/ansible.cfg
```


### OS / Environment

macosx, docker container with Ubuntu

### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->
```yaml (paste below)
Steps to reproduce the problem:

we configured Ansible like here: https://docs.ansible.com/ansible/latest/collections/community/general/opentelemetry_callback.html
mainly adding this in the ansible.cfg:
[defaults]
callbacksenabled = community.general.opentelemetry [callbackopentelemetry]
enablefromenvironment = ANSIBLEOPENTELEMETRYENABLED

created and configured the environment with the token and ID
export OTELEXPORTEROTLPPROTOCOL="http/protobuf" export OTELEXPORTEROTLPENDPOINT="https://otlp-gateway-prod-eu-west-2.grafana.net/otlp"
export OTELEXPORTEROTLP_HEADERS="Authorization=Basic REDACTED"
3.
Run playbooks and got this:

Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 1s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 2s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 4s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 8s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 16s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 32s.

We also have an open question: https://grafana.slack.com/archives/C038S86TZGF/p1705057683915289
and here: https://forum.ansible.com/t/anyone-managed-to-use-opentelemetry-from-awx-ansible/3005


```


### Expected Results

Running Ansible playbooks should send traces to the grafana cloud



### Actual Results

```console (paste below)

PLAY RECAP *********************************************************************
localhost                  : ok=9    changed=0    unreachable=0    failed=0    skipped=14   rescued=0    ignored=0

Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 1s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 2s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 4s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 8s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 16s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to otlp-gateway-prod-eu-west-2.grafana.net, retrying in 32s.
```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
* [`plugins/callback/opentelemetry.py`](https://github.com/ansible-collections/community.general/blob/main/plugins/callback/opentelemetry.py)

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @v1v
[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: notify --->
This particular implementation is vendor-agnostic.

Would you mind if I ask you to try the same settings but using an OpenTelemetry Collector[^1] with some debug traces[^2]? Pretty much update `OTEL_EXPORTER_OTLP_ENDPOINT` to point to your collector. You can try that locally. 

[^1]: https://opentelemetry.io/docs/collector/
[^2]: https://opentelemetry.io/docs/collector/configuration/
> Would you mind if I ask you to try the same settings but using an OpenTelemetry Collector[1](https://github.com/ansible-collections/community.general/issues/7888#user-content-fn-1-087883021132d17b2f951d22ad2ac4db) with some debug traces[2](https://github.com/ansible-collections/community.general/issues/7888#user-content-fn-2-087883021132d17b2f951d22ad2ac4db)? Pretty much update OTEL_EXPORTER_OTLP_ENDPOINT to point to your collector

I'm in the same situation with `OTEL_EXPORTER_OTLP_ENDPOINT` correctly define. I've other applications in the same network who manage to successfully send their traces.

Resolving patch:
diff --git a/plugins/callback/opentelemetry.py b/plugins/callback/opentelemetry.py
--- a/plugins/callback/opentelemetry.py
+++ b/plugins/callback/opentelemetry.py
@@ -84,6 +84,22 @@
           - section: callback_opentelemetry
             key: disable_attributes_in_logs
         version_added: 7.1.0
+      otel_exporter_otlp_traces_protocol:
+        type: str
+        description:
+          - E(OTEL_EXPORTER_OTLP_TRACES_PROTOCOL) represents the the transport protocol for spans.
+          - See
+            U(https://opentelemetry-python.readthedocs.io/en/latest/sdk/environment_variables.html#envvar-OTEL_EXPORTER_OTLP_TRACES_PROTOCOL).
+        default: grpc
+        choices:
+          - grpc
+          - http/protobuf
+        env:
+          - name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
+        ini:
+          - section: callback_opentelemetry
+            key: otel_exporter_otlp_traces_protocol
+        version_added: 9.0.0
     requirements:
       - opentelemetry-api (Python library)
       - opentelemetry-exporter-otlp (Python library)
@@ -124,7 +140,8 @@
 try:
     from opentelemetry import trace
     from opentelemetry.trace import SpanKind
-    from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
+    from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter as GRPCOTLPSpanExporter
+    from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter as HTTPOTLPSpanExporter
     from opentelemetry.sdk.resources import SERVICE_NAME, Resource
     from opentelemetry.trace.status import Status, StatusCode
     from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
@@ -255,7 +272,15 @@ def finish_task(self, tasks_data, status, result, dump):
         task.dump = dump
         task.add_host(HostData(host_uuid, host_name, status, result))
 
-    def generate_distributed_traces(self, otel_service_name, ansible_playbook, tasks_data, status, traceparent, disable_logs, disable_attributes_in_logs):
+    def generate_distributed_traces(self,
+                                    otel_service_name,
+                                    ansible_playbook,
+                                    tasks_data,
+                                    status,
+                                    traceparent,
+                                    disable_logs,
+                                    disable_attributes_in_logs,
+                                    otel_exporter_otlp_traces_protocol):
         """ generate distributed traces from the collected TaskData and HostData """
 
         tasks = []
@@ -271,7 +296,11 @@ def generate_distributed_traces(self, otel_service_name, ansible_playbook, tasks
             )
         )
 
-        processor = BatchSpanProcessor(OTLPSpanExporter())
+        processor = None
+        if otel_exporter_otlp_traces_protocol == 'grpc':
+            processor = BatchSpanProcessor(GRPCOTLPSpanExporter())
+        else:
+            processor = BatchSpanProcessor(HTTPOTLPSpanExporter())
 
         trace.get_tracer_provider().add_span_processor(processor)
 
@@ -462,6 +491,7 @@ def __init__(self, display=None):
         self.errors = 0
         self.disabled = False
         self.traceparent = False
+        self.otel_exporter_otlp_traces_protocol = None
 
         if OTEL_LIBRARY_IMPORT_ERROR:
             raise_from(
@@ -497,6 +527,8 @@ def set_options(self, task_keys=None, var_options=None, direct=None):
         # See https://github.com/open-telemetry/opentelemetry-specification/issues/740
         self.traceparent = self.get_option('traceparent')
 
+        self.otel_exporter_otlp_traces_protocol = self.get_option('otel_exporter_otlp_traces_protocol')
+
     def v2_playbook_on_start(self, playbook):
         self.ansible_playbook = basename(playbook._file_name)
 
@@ -585,7 +617,8 @@ def v2_playbook_on_stats(self, stats):
             status,
             self.traceparent,
             self.disable_logs,
-            self.disable_attributes_in_logs
+            self.disable_attributes_in_logs,
+            self.otel_exporter_otlp_traces_protocol,
         )
 
     def v2_runner_on_async_failed(self, result, **kwargs):


Test patch: