Repository: open-telemetry/opentelemetry-python-contrib

Issue:
grpc: trailing metadata can't be passed when aborting an AIO server RPC
**Describe your environment**

Python: 3.12.0
`opentelemetry-distro[otlp]==0.42b0`
`opentelemetry-instrumentation-grpc==0.42b0`
`grpcio==1.59.2`

**Steps to reproduce**
Full replication can be found [here](https://github.com/cookiefission/testbench/tree/main/otel/python-grpc-aio-abort-issue).

`server.py`:

```python
import asyncio
import logging

import grpc
import helloworld_pb2
import helloworld_pb2_grpc


class Greeter(helloworld_pb2_grpc.GreeterServicer):
    async def SayHello(
        self,
        request: helloworld_pb2.HelloRequest,
        context: grpc.aio.ServicerContext,
    ) -> helloworld_pb2.HelloReply:
        metadata = (
            ("this", "should"),
            ("work", "nicely")
        )
        await context.abort(code=grpc.StatusCode.ABORTED, details="This is the actual error message", trailing_metadata=metadata)


async def serve() -> None:
    server = grpc.aio.server()
    helloworld_pb2_grpc.add_GreeterServicer_to_server(Greeter(), server)
    listen_addr = "[::]:50051"
    server.add_insecure_port(listen_addr)
    logging.info("Starting server on %s", listen_addr)
    await server.start()
    await server.wait_for_termination()


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    asyncio.run(serve())
```
This server is just adopted from [the grpc examples](https://github.com/grpc/grpc/blob/master/examples/python/helloworld/async_greeter_server.py). The helloworld protobuf artefacts can be found [there](https://github.com/grpc/grpc/blob/master/examples/python/helloworld/helloworld_pb2_grpc.py).

For a client, the [`async_greeter_client.py`](https://github.com/grpc/grpc/blob/master/examples/python/helloworld/async_greeter_client.py) from grpc examples can be used.

This only happens when `trailing_metadata` is passed as an argument to `context.abort`. 

**What is the expected behavior?**

Running the server without opentelemetry, the (expected) error from running the client is:

```
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
        status = StatusCode.ABORTED
        details = "This is the actual error message"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"This is the actual error message", grpc_status:10, created_time:"2023-11-16T15:09:03.552303+00:00"}"
>
```

When running the server under `opentelemetry-instrument`, I expect the same output as when it is run without OTEL. 

**What is the actual behavior?**

Instead, an exception is still raised but the underlying code, details, and trailing_metadata are missing. This makes any nuanced error handling on the client side impossible. 

```
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Unexpected <class 'TypeError'>: _OpenTelemetryServicerContext.abort() got an unexpected keyword argument 'trailing_metadata'"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2023-11-16T15:30:02.017999+00:00", grpc_status:2, grpc_message:"Unexpected <class \'TypeError\'>: _OpenTelemetryServicerContext.abort() got an unexpected keyword argument \'trailing_metadata\'"}"
>
```

**Additional context**

The `abort` method for [`grpc.ServicerContext`](https://grpc.github.io/grpc/python/grpc.html#grpc.ServicerContext.abort) and [`grpc.aio.ServicerContext`](https://grpc.github.io/grpc/python/grpc_asyncio.html#grpc.aio.ServicerContext.abort) is different. It's unclear why. The AIO instrumentation [re-uses](https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumentation/opentelemetry-instrumentation-grpc/src/opentelemetry/instrumentation/grpc/_aio_server.py#L19) the [`_OpenTelemetryServicerContext`](https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumentation/opentelemetry-instrumentation-grpc/src/opentelemetry/instrumentation/grpc/_server.py#L68) from the non-async server instrumentation which is how this bug snuck in, despite there being a test in place for the abort. 



Discussion:

Resolving patch:
diff --git a/instrumentation/opentelemetry-instrumentation-grpc/src/opentelemetry/instrumentation/grpc/_aio_server.py b/instrumentation/opentelemetry-instrumentation-grpc/src/opentelemetry/instrumentation/grpc/_aio_server.py
--- a/instrumentation/opentelemetry-instrumentation-grpc/src/opentelemetry/instrumentation/grpc/_aio_server.py
+++ b/instrumentation/opentelemetry-instrumentation-grpc/src/opentelemetry/instrumentation/grpc/_aio_server.py
@@ -12,13 +12,63 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import grpc
 import grpc.aio
-
-from ._server import (
-    OpenTelemetryServerInterceptor,
-    _OpenTelemetryServicerContext,
-    _wrap_rpc_behavior,
-)
+import wrapt
+
+from opentelemetry.semconv.trace import SpanAttributes
+from opentelemetry.trace.status import Status, StatusCode
+
+from ._server import OpenTelemetryServerInterceptor, _wrap_rpc_behavior
+
+
+# pylint:disable=abstract-method
+class _OpenTelemetryAioServicerContext(wrapt.ObjectProxy):
+    def __init__(self, servicer_context, active_span):
+        super().__init__(servicer_context)
+        self._self_active_span = active_span
+        self._self_code = grpc.StatusCode.OK
+        self._self_details = None
+
+    async def abort(self, code, details="", trailing_metadata=tuple()):
+        self._self_code = code
+        self._self_details = details
+        self._self_active_span.set_attribute(
+            SpanAttributes.RPC_GRPC_STATUS_CODE, code.value[0]
+        )
+        self._self_active_span.set_status(
+            Status(
+                status_code=StatusCode.ERROR,
+                description=f"{code}:{details}",
+            )
+        )
+        return await self.__wrapped__.abort(code, details, trailing_metadata)
+
+    def set_code(self, code):
+        self._self_code = code
+        details = self._self_details or code.value[1]
+        self._self_active_span.set_attribute(
+            SpanAttributes.RPC_GRPC_STATUS_CODE, code.value[0]
+        )
+        if code != grpc.StatusCode.OK:
+            self._self_active_span.set_status(
+                Status(
+                    status_code=StatusCode.ERROR,
+                    description=f"{code}:{details}",
+                )
+            )
+        return self.__wrapped__.set_code(code)
+
+    def set_details(self, details):
+        self._self_details = details
+        if self._self_code != grpc.StatusCode.OK:
+            self._self_active_span.set_status(
+                Status(
+                    status_code=StatusCode.ERROR,
+                    description=f"{self._self_code}:{details}",
+                )
+            )
+        return self.__wrapped__.set_details(details)
 
 
 class OpenTelemetryAioServerInterceptor(
@@ -66,7 +116,7 @@ async def _unary_interceptor(request_or_iterator, context):
                     set_status_on_exception=False,
                 ) as span:
                     # wrap the context
-                    context = _OpenTelemetryServicerContext(context, span)
+                    context = _OpenTelemetryAioServicerContext(context, span)
 
                     # And now we run the actual RPC.
                     try:
@@ -91,7 +141,7 @@ async def _stream_interceptor(request_or_iterator, context):
                     context,
                     set_status_on_exception=False,
                 ) as span:
-                    context = _OpenTelemetryServicerContext(context, span)
+                    context = _OpenTelemetryAioServicerContext(context, span)
 
                     try:
                         async for response in behavior(


Test patch:
diff --git a/instrumentation/opentelemetry-instrumentation-grpc/tests/test_aio_server_interceptor.py b/instrumentation/opentelemetry-instrumentation-grpc/tests/test_aio_server_interceptor.py
--- a/instrumentation/opentelemetry-instrumentation-grpc/tests/test_aio_server_interceptor.py
+++ b/instrumentation/opentelemetry-instrumentation-grpc/tests/test_aio_server_interceptor.py
@@ -88,8 +88,11 @@ async def run_with_test_server(
     channel = grpc.aio.insecure_channel(f"localhost:{port:d}")
 
     await server.start()
-    resp = await runnable(channel)
-    await server.stop(1000)
+
+    try:
+        resp = await runnable(channel)
+    finally:
+        await server.stop(1000)
 
     return resp
 
@@ -514,9 +517,79 @@ async def request(channel):
             request = Request(client_id=1, request_data=failure_message)
             msg = request.SerializeToString()
 
-            with testcase.assertRaises(Exception):
+            with testcase.assertRaises(grpc.RpcError) as cm:
+                await channel.unary_unary(rpc_call)(msg)
+
+            self.assertEqual(
+                cm.exception.code(), grpc.StatusCode.FAILED_PRECONDITION
+            )
+            self.assertEqual(cm.exception.details(), failure_message)
+
+        await run_with_test_server(request, servicer=AbortServicer())
+
+        spans_list = self.memory_exporter.get_finished_spans()
+        self.assertEqual(len(spans_list), 1)
+        span = spans_list[0]
+
+        self.assertEqual(span.name, rpc_call)
+        self.assertIs(span.kind, trace.SpanKind.SERVER)
+
+        # Check version and name in span's instrumentation info
+        self.assertEqualSpanInstrumentationInfo(
+            span, opentelemetry.instrumentation.grpc
+        )
+
+        # make sure this span errored, with the right status and detail
+        self.assertEqual(span.status.status_code, StatusCode.ERROR)
+        self.assertEqual(
+            span.status.description,
+            f"{grpc.StatusCode.FAILED_PRECONDITION}:{failure_message}",
+        )
+
+        # Check attributes
+        self.assertSpanHasAttributes(
+            span,
+            {
+                SpanAttributes.NET_PEER_IP: "[::1]",
+                SpanAttributes.NET_PEER_NAME: "localhost",
+                SpanAttributes.RPC_METHOD: "SimpleMethod",
+                SpanAttributes.RPC_SERVICE: "GRPCTestServer",
+                SpanAttributes.RPC_SYSTEM: "grpc",
+                SpanAttributes.RPC_GRPC_STATUS_CODE: grpc.StatusCode.FAILED_PRECONDITION.value[
+                    0
+                ],
+            },
+        )
+
+    async def test_abort_with_trailing_metadata(self):
+        """Check that we can catch an abort properly when trailing_metadata provided"""
+        rpc_call = "/GRPCTestServer/SimpleMethod"
+        failure_message = "failure message"
+
+        class AbortServicer(GRPCTestServerServicer):
+            # pylint:disable=C0103
+            async def SimpleMethod(self, request, context):
+                metadata = (("meta", "data"),)
+                await context.abort(
+                    grpc.StatusCode.FAILED_PRECONDITION,
+                    failure_message,
+                    trailing_metadata=metadata,
+                )
+
+        testcase = self
+
+        async def request(channel):
+            request = Request(client_id=1, request_data=failure_message)
+            msg = request.SerializeToString()
+
+            with testcase.assertRaises(grpc.RpcError) as cm:
                 await channel.unary_unary(rpc_call)(msg)
 
+            self.assertEqual(
+                cm.exception.code(), grpc.StatusCode.FAILED_PRECONDITION
+            )
+            self.assertEqual(cm.exception.details(), failure_message)
+
         await run_with_test_server(request, servicer=AbortServicer())
 
         spans_list = self.memory_exporter.get_finished_spans()