Repository: open-telemetry/opentelemetry-python-contrib

Issue:
aiopg instrumentation creating nested traces
**Describe your environment**
Python 3.6.12
Macos 11.1
opentelemetry-python-contrib: Tried released version, as well as `main`.
aiohttp==3.7.3
aiopg==1.1.0

 Describe any aspect of your environment relevant to the problem, including your Python version, [platform](https://docs.python.org/3/library/platform.html), version numbers of installed dependencies, information about your cloud hosting provider, etc. If you're reporting a problem with a specific version of a library in this repo, please check whether the problem has been fixed on main.

**Steps to reproduce**
I am trying the `aiopg` instrumentation. I am using it along aiohttp. This may be an usage issue, but I have created a minimal example in [here](https://github.com/cdvv7788/aiopg-instrumentor-bug) (Just install requirements, run, and visit `localhost:4000`...you will need a postgres database named test and jaeger running somewhere).
The trace will keep growing with every request, nesting new traces every time:
![image](https://user-images.githubusercontent.com/5531776/108002651-4494bd80-6fbe-11eb-8ebf-0067905e6ab6.png)


**What is the expected behavior?**
Every connection has a single trace (no nesting)

**What is the actual behavior?**
Every time a request is made, a new trace is nested and sends the whole stack to jaeger

**Notes**
This may be a bad usage issue. If that is the case, please let me know what may be causing it and how to use this properly.



Discussion:
@sartx I saw you worked on this implementation (https://github.com/open-telemetry/opentelemetry-python/pull/801). Can you please check this issue?
Using `pdb` it is entering `traced_execution` at least 3 times in the sample (1 for a single span, 2 nested). If I keep refreshing, the nesting keeps growing. Checking with `opentelemetry.trace.get_current_span()` it is indeed getting a `SELECT` span (Instead of Default, which is what I was expecting) and putting the next one inside.
Hello, I have an idea why it is broken.

In previous versions of opentelemetry `start_as_current_span` had argument `parent`. If it had no value then `DEFAULT` span was used as parent. In current version if `context` was not given as argument in this function `RUNTIME_CONTEXT` is used as context. As far as I understand it is a global variable.
First request uses `RUNTIME_CONTEXT` as parent span and addes own new context to `RUNTIME_CONTEXT`.
While first request is not ended the second one comes to the application and it uses `RUNTIME_CONTEXT` as parent to, but there is still the context from the first request. Old span is closing only after all child spans were closed.

If you want to check it you may try to call new request after previous is ended.
Using contextvar variable for context probably will fix this issue.
Thanks for jumping in. The requests are not running in parallel, they are sequential. Some reference is being kept around in the process, and it will keep nesting.


Resolving patch:
diff --git a/instrumentation/opentelemetry-instrumentation-aiopg/src/opentelemetry/instrumentation/aiopg/aiopg_integration.py b/instrumentation/opentelemetry-instrumentation-aiopg/src/opentelemetry/instrumentation/aiopg/aiopg_integration.py
--- a/instrumentation/opentelemetry-instrumentation-aiopg/src/opentelemetry/instrumentation/aiopg/aiopg_integration.py
+++ b/instrumentation/opentelemetry-instrumentation-aiopg/src/opentelemetry/instrumentation/aiopg/aiopg_integration.py
@@ -86,9 +86,11 @@ def acquire(self):
         async def _acquire(self):
             # pylint: disable=protected-access
             connection = await self.__wrapped__._acquire()
-            return get_traced_connection_proxy(
-                connection, db_api_integration, *args, **kwargs
-            )
+            if not isinstance(connection, AsyncProxyObject):
+                connection = get_traced_connection_proxy(
+                    connection, db_api_integration, *args, **kwargs
+                )
+            return connection
 
     return TracedPoolProxy(pool, *args, **kwargs)
 


Test patch:
diff --git a/tests/opentelemetry-docker-tests/tests/postgres/test_aiopg_functional.py b/tests/opentelemetry-docker-tests/tests/postgres/test_aiopg_functional.py
--- a/tests/opentelemetry-docker-tests/tests/postgres/test_aiopg_functional.py
+++ b/tests/opentelemetry-docker-tests/tests/postgres/test_aiopg_functional.py
@@ -117,6 +117,10 @@ def setUpClass(cls):
         cls._cursor = None
         cls._tracer = cls.tracer_provider.get_tracer(__name__)
         AiopgInstrumentor().instrument(tracer_provider=cls.tracer_provider)
+        cls._dsn = (
+            f"dbname='{POSTGRES_DB_NAME}' user='{POSTGRES_USER}' password='{POSTGRES_PASSWORD}'"
+            f" host='{POSTGRES_HOST}' port='{POSTGRES_PORT}'"
+        )
         cls._pool = async_call(
             aiopg.create_pool(
                 dbname=POSTGRES_DB_NAME,
@@ -185,3 +189,19 @@ def test_callproc(self):
         ):
             async_call(self._cursor.callproc("test", ()))
             self.validate_spans("test")
+
+    def test_instrumented_pool_with_multiple_acquires(self, *_, **__):
+        async def double_acquire():
+            pool = await aiopg.create_pool(dsn=self._dsn)
+            async with pool.acquire() as conn:
+                async with conn.cursor() as cursor:
+                    query = "SELECT 1"
+                    await cursor.execute(query)
+            async with pool.acquire() as conn:
+                async with conn.cursor() as cursor:
+                    query = "SELECT 1"
+                    await cursor.execute(query)
+
+        async_call(double_acquire())
+        spans = self.memory_exporter.get_finished_spans()
+        self.assertEqual(len(spans), 2)