Repository: aio-libs/aiohttp

Issue:
Potential cache/throttle race conditions in Connector._resolve_host with traces
## Long story short

If `traces` are used in [`TCPConnector._resolve_host`](https://github.com/aio-libs/aiohttp/blob/b35c9e3facaa50093feb05309fb7e679ffbb248b/aiohttp/connector.py#L754) the context switch introduced by awaiting their trace API calls can lead to various race conditions.

For example the [DNS cache hit test](https://github.com/aio-libs/aiohttp/blob/b35c9e3facaa50093feb05309fb7e679ffbb248b/aiohttp/connector.py#L779-L780) may evaluate `True` but by the time `traces` is [done](https://github.com/aio-libs/aiohttp/blob/b35c9e3facaa50093feb05309fb7e679ffbb248b/aiohttp/connector.py#L786) it may already be gone again.

The same is true for the throttling logic where between checking that the [key is already throttled](https://github.com/aio-libs/aiohttp/blob/b35c9e3facaa50093feb05309fb7e679ffbb248b/aiohttp/connector.py#L788) and then [accessing that key](https://github.com/aio-libs/aiohttp/blob/b35c9e3facaa50093feb05309fb7e679ffbb248b/aiohttp/connector.py#L792) the key may already be popped.

Similarly in the [opposite case](https://github.com/aio-libs/aiohttp/blob/b35c9e3facaa50093feb05309fb7e679ffbb248b/aiohttp/connector.py#L793) of the above, with traces enabled, the code might [override an already set throttled event](https://github.com/aio-libs/aiohttp/blob/b35c9e3facaa50093feb05309fb7e679ffbb248b/aiohttp/connector.py#L797-L798)

## Steps to reproduce

See [my branch](https://github.com/ojii/aiohttp/blob/93dd3657a667d6367848ee0bb45bbab2ac25472d/tests/test_connector.py#L2263-L2289)

In the tests I mess with the caches from the traces directly to simulate the effects for simplicity.

Potential cache/throttle race conditions in Connector._resolve_host with traces
## Long story short

If `traces` are used in [`TCPConnector._resolve_host`](https://github.com/aio-libs/aiohttp/blob/b35c9e3facaa50093feb05309fb7e679ffbb248b/aiohttp/connector.py#L754) the context switch introduced by awaiting their trace API calls can lead to various race conditions.

For example the [DNS cache hit test](https://github.com/aio-libs/aiohttp/blob/b35c9e3facaa50093feb05309fb7e679ffbb248b/aiohttp/connector.py#L779-L780) may evaluate `True` but by the time `traces` is [done](https://github.com/aio-libs/aiohttp/blob/b35c9e3facaa50093feb05309fb7e679ffbb248b/aiohttp/connector.py#L786) it may already be gone again.

The same is true for the throttling logic where between checking that the [key is already throttled](https://github.com/aio-libs/aiohttp/blob/b35c9e3facaa50093feb05309fb7e679ffbb248b/aiohttp/connector.py#L788) and then [accessing that key](https://github.com/aio-libs/aiohttp/blob/b35c9e3facaa50093feb05309fb7e679ffbb248b/aiohttp/connector.py#L792) the key may already be popped.

Similarly in the [opposite case](https://github.com/aio-libs/aiohttp/blob/b35c9e3facaa50093feb05309fb7e679ffbb248b/aiohttp/connector.py#L793) of the above, with traces enabled, the code might [override an already set throttled event](https://github.com/aio-libs/aiohttp/blob/b35c9e3facaa50093feb05309fb7e679ffbb248b/aiohttp/connector.py#L797-L798)

## Steps to reproduce

See [my branch](https://github.com/ojii/aiohttp/blob/93dd3657a667d6367848ee0bb45bbab2ac25472d/tests/test_connector.py#L2263-L2289)

In the tests I mess with the caches from the traces directly to simulate the effects for simplicity.



Discussion:

Resolving patch:
diff --git a/aiohttp/connector.py b/aiohttp/connector.py
--- a/aiohttp/connector.py
+++ b/aiohttp/connector.py
@@ -778,24 +778,28 @@ async def _resolve_host(self,
 
         if (key in self._cached_hosts) and \
                 (not self._cached_hosts.expired(key)):
+            # get result early, before any await (#4014)
+            result = self._cached_hosts.next_addrs(key)
 
             if traces:
                 for trace in traces:
                     await trace.send_dns_cache_hit(host)
-
-            return self._cached_hosts.next_addrs(key)
+            return result
 
         if key in self._throttle_dns_events:
+            # get event early, before any await (#4014)
+            event = self._throttle_dns_events[key]
             if traces:
                 for trace in traces:
                     await trace.send_dns_cache_hit(host)
-            await self._throttle_dns_events[key].wait()
+            await event.wait()
         else:
+            # update dict early, before any await (#4014)
+            self._throttle_dns_events[key] = \
+                EventResultOrError(self._loop)
             if traces:
                 for trace in traces:
                     await trace.send_dns_cache_miss(host)
-            self._throttle_dns_events[key] = \
-                EventResultOrError(self._loop)
             try:
 
                 if traces:


Test patch:
diff --git a/tests/test_connector.py b/tests/test_connector.py
--- a/tests/test_connector.py
+++ b/tests/test_connector.py
@@ -18,8 +18,9 @@
 from aiohttp import client, web
 from aiohttp.client import ClientRequest, ClientTimeout
 from aiohttp.client_reqrep import ConnectionKey
-from aiohttp.connector import Connection, _DNSCacheTable
+from aiohttp.connector import Connection, TCPConnector, _DNSCacheTable
 from aiohttp.helpers import PY_37
+from aiohttp.locks import EventResultOrError
 from aiohttp.test_utils import make_mocked_coro, unused_port
 from aiohttp.tracing import Trace
 from conftest import needs_unix
@@ -2257,3 +2258,32 @@ def test_next_addrs_single(self, dns_cache_table) -> None:
 
         addrs = dns_cache_table.next_addrs('foo')
         assert addrs == ['127.0.0.1']
+
+
+async def test_connector_cache_trace_race():
+    class DummyTracer:
+        async def send_dns_cache_hit(self, *args, **kwargs):
+            connector._cached_hosts.remove(("", 0))
+
+    token = object()
+    connector = TCPConnector()
+    connector._cached_hosts.add(("", 0), [token])
+
+    traces = [DummyTracer()]
+    assert await connector._resolve_host("", 0, traces) == [token]
+
+
+async def test_connector_throttle_trace_race(loop):
+    key = ("", 0)
+    token = object()
+
+    class DummyTracer:
+        async def send_dns_cache_hit(self, *args, **kwargs):
+            event = connector._throttle_dns_events.pop(key)
+            event.set()
+            connector._cached_hosts.add(key, [token])
+
+    connector = TCPConnector()
+    connector._throttle_dns_events[key] = EventResultOrError(loop)
+    traces = [DummyTracer()]
+    assert await connector._resolve_host("", 0, traces) == [token]