Repository: h2non/pook

Issue:
aiohttp mock only works when using request methods as context managers, but not when awaited
Hello, thank you for the lovely project! It makes HTTP mocking really pleasant to work with.

I noticed a small issue with the implementation of the aiohttp mock: it appears to only work when using the request methods as context managers, but not when their results are awaited.

This gist has example code to show this in a pytest unit test: https://gist.github.com/sarayourfriend/44a33e7397849939156247457d8bb773

Here is the full output of running the tests:

```
➜  aiohttp-example pdm run pytest
/home/sara/.local/lib/python3.10/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (5.0.0)/charset_normalizer (2.0.12) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
=================================== test session starts ===================================
platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/sara/projects/aiohttp-example
plugins: anyio-3.6.2, asyncio-0.20.2
asyncio: mode=strict
collected 2 items                                                                         

main_test.py .F                                                                     [100%]

======================================== FAILURES =========================================
_________________________________ test_make_request_await _________________________________

    async def test_make_request_await():
        pook.head(URL).reply(200)
    
        pook.on()
>       res = await make_request_await()

main_test.py:36: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
main_test.py:19: in make_request_await
    response = await session.head(URL)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <aiohttp.client._RequestContextManager object at 0x7f7cb9dfc9d0>

    def __await__(self) -> Generator[Any, None, _RetType]:
>       ret = self._coro.__await__()
E       AttributeError: 'generator' object has no attribute '__await__'. Did you mean: '__init__'?

.venv/lib/python3.10/site-packages/aiohttp/client.py:1134: AttributeError
==================================== warnings summary =====================================
.venv/lib/python3.10/site-packages/pook/interceptors/aiohttp.py:48
  /home/sara/projects/aiohttp-example/.venv/lib/python3.10/site-packages/pook/interceptors/aiohttp.py:48: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
    def read(self, n=-1):

.venv/lib/python3.10/site-packages/pook/interceptors/aiohttp.py:80
  /home/sara/projects/aiohttp-example/.venv/lib/python3.10/site-packages/pook/interceptors/aiohttp.py:80: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
    def _on_request(self, _request, session, method, url,

main_test.py::test_make_request_ctx
main_test.py::test_make_request_await
  /home/sara/projects/aiohttp-example/.venv/lib/python3.10/site-packages/pook/interceptors/aiohttp.py:153: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
    def handler(session, method, url, data=None, headers=None, **kw):

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================= short test summary info =================================
FAILED main_test.py::test_make_request_await - AttributeError: 'generator' object has no attribute '__await__'. Did you mean: '__init...
========================= 1 failed, 1 passed, 4 warnings in 0.09s =========================
```

As you can see, `test_make_request_await` fails whereas the other test case which uses the request method as a context manger passes fine.

It looks like this comes down to an issue with the return value of the `_request` mock that pook is applying: it must be a coroutine but is currently a generator. I'm not sure exactly _why_ this is the case, because the mock handler is wrapped in `@asyncio.coroutine`.

I've managed to patch this locally by removing the usage of the `asyncio.coroutine` decorator and replacing the methods with regular `async def` methods. I'll open a PR with these changes later this week.

Discussion:
KyleJamesWalker: This is great if there's any special code for anything below 3.7 I would drop the extra effort for anything that's already EOL

https://endoflife.date/python
sarayourfriend: Other than the tox configuration running on versions as low as 3.5 I don't think there are any other version checks or version specific code left.
KyleJamesWalker: @h2non Any chance this will be added/reviewed soon?

Patch:
diff --git a/.travis.yml b/.travis.yml
index 27d4b54..4f826ae 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,18 +1,16 @@
 language: python
 
 python:
-  - 2.7
-  - 3.4
   - 3.5
   - 3.6
   - 3.7
   - 3.8
+  - 3.9
+  - 3.10
   - pypy
   - pypy3
 
 allow_failures:
-  - python: 2.7
-  - python: 3.4
   - python: pypy
 
 install:
diff --git a/README.rst b/README.rst
index fee973e..92f0e3b 100644
--- a/README.rst
+++ b/README.rst
@@ -31,7 +31,7 @@ Features
 -  Supports third-party mocking engines, such as `mocket`_.
 -  Fits good for painless test doubles.
 -  Does not support WebSocket traffic mocking.
--  Works with Python +2.7 and +3.0 (including PyPy).
+-  Works with +3.5 (including PyPy).
 -  Dependency-less: just 2 small dependencies for JSONSchema and XML tree comparison.
 
 
diff --git a/pook/assertion.py b/pook/assertion.py
index 149d4a5..493b065 100644
--- a/pook/assertion.py
+++ b/pook/assertion.py
@@ -1,11 +1,6 @@
-import re
-import sys
 from unittest import TestCase
 from .regex import isregex, strip_regex, isregex_expr
 
-# If running Python 3
-PY_3 = sys.version_info >= (3,)
-
 
 def test_case():
     """
@@ -33,8 +28,7 @@ def equal(x, y):
     Returns:
         bool
     """
-    if PY_3:
-        return test_case().assertEqual(x, y) or True
+    return test_case().assertEqual(x, y) or True
 
     assert x == y
 
@@ -59,17 +53,10 @@ def matches(x, y, regex_expr=False):
     x = strip_regex(x) if regex_expr and isregex_expr(x) else x
 
     # Run regex assertion
-    if PY_3:
-        # Retrieve original regex pattern
-        x = x.pattern if isregex(x) else x
-        # Assert regular expression via unittest matchers
-        return test_case().assertRegex(y, x) or True
-
-    # Primitive regex matching for Python 2.7
-    if isinstance(x, str):
-        x = re.compile(x, re.IGNORECASE)
-
-    assert x.match(y) is not None
+    # Retrieve original regex pattern
+    x = x.pattern if isregex(x) else x
+    # Assert regular expression via unittest matchers
+    return test_case().assertRegex(y, x) or True
 
 
 def test(x, y, regex_expr=False):
diff --git a/pook/headers.py b/pook/headers.py
index 03a1bd8..ac20bd2 100644
--- a/pook/headers.py
+++ b/pook/headers.py
@@ -1,11 +1,8 @@
-import sys
 try:
     from collections.abc import Mapping, MutableMapping
 except ImportError:
     from collections import Mapping, MutableMapping
 
-PY3 = sys.version_info >= (3, 0)
-
 
 class HTTPHeaderDict(MutableMapping):
     """
@@ -75,10 +72,6 @@ class HTTPHeaderDict(MutableMapping):
     def __ne__(self, other):
         return not self.__eq__(other)
 
-    if not PY3:  # Python 2
-        iterkeys = MutableMapping.iterkeys
-        itervalues = MutableMapping.itervalues
-
     __marker = object()
 
     def __len__(self):
@@ -245,24 +238,3 @@ class HTTPHeaderDict(MutableMapping):
 
     def to_dict(self):
         return {key: values for key, values in self.items()}
-
-    @classmethod
-    def from_httplib(cls, message):  # Python 2
-        """
-        Read headers from a Python 2 httplib message object.
-        """
-        # python2.7 does not expose a proper API for exporting multiheaders
-        # efficiently. This function re-reads raw lines from the message
-        # object and extracts the multiheaders properly.
-        headers = []
-
-        for line in message.headers:
-            if line.startswith((' ', '\t')):
-                key, value = headers[-1]
-                headers[-1] = (key, value + '\r\n' + line.rstrip())
-                continue
-
-            key, value = line.split(':', 1)
-            headers.append((key, value.strip()))
-
-        return cls(headers)
diff --git a/pook/interceptors/__init__.py b/pook/interceptors/__init__.py
index d2e25f4..e0dc829 100644
--- a/pook/interceptors/__init__.py
+++ b/pook/interceptors/__init__.py
@@ -1,4 +1,3 @@
-import sys
 from .urllib3 import Urllib3Interceptor
 from .http import HTTPClientInterceptor
 from .base import BaseInterceptor
@@ -18,14 +17,12 @@ interceptors = [
     HTTPClientInterceptor
 ]
 
-# Import aiohttp in modern Python runtimes
-if sys.version_info >= (3, 5, 0):
-    try:
-        import aiohttp # noqa
-        from .aiohttp import AIOHTTPInterceptor
-        interceptors.append(AIOHTTPInterceptor)
-    except ImportError:
-        pass
+try:
+    import aiohttp # noqa
+    from .aiohttp import AIOHTTPInterceptor
+    interceptors.append(AIOHTTPInterceptor)
+except ImportError:
+    pass
 
 
 def add(*custom_interceptors):
diff --git a/pook/interceptors/aiohttp.py b/pook/interceptors/aiohttp.py
index e5df9a4..b695a65 100644
--- a/pook/interceptors/aiohttp.py
+++ b/pook/interceptors/aiohttp.py
@@ -1,28 +1,14 @@
-import sys
 from ..request import Request
 from .base import BaseInterceptor
 
-# Support Python 2/3
-try:
-    import mock
-except Exception:
-    from unittest import mock
-
-if sys.version_info < (3,):     # Python 2
-    from urlparse import urlunparse, urlencode
-    from httplib import responses as http_reasons
-else:                           # Python 3
-    from urllib.parse import urlunparse, urlencode
-    from http.client import responses as http_reasons
-
-if sys.version_info >= (3, 5, 0):  # Python 3.5+
-    import asyncio
-    from aiohttp.helpers import TimerNoop
-    from aiohttp.streams import EmptyStreamReader
-else:
-    asyncio = None
-    TimerNoop = None
-    EmptyStreamReader = None
+from unittest import mock
+
+from urllib.parse import urlunparse, urlencode
+from http.client import responses as http_reasons
+
+import asyncio
+from aiohttp.helpers import TimerNoop
+from aiohttp.streams import EmptyStreamReader
 
 # Try to load yarl URL parser package used by aiohttp
 try:
@@ -44,8 +30,7 @@ class SimpleContent(EmptyStreamReader):
         super().__init__(*args, **kwargs)
         self.content = content
 
-    @asyncio.coroutine
-    def read(self, n=-1):
+    async def read(self, n=-1):
         return self.content
 
 
@@ -76,9 +61,8 @@ class AIOHTTPInterceptor(BaseInterceptor):
     def _url(self, url):
         return yarl.URL(url) if yarl else None
 
-    @asyncio.coroutine
-    def _on_request(self, _request, session, method, url,
-                    data=None, headers=None, **kw):
+    async def _on_request(self, _request, session, method, url,
+                          data=None, headers=None, **kw):
         # Create request contract based on incoming params
         req = Request(method)
         req.headers = headers or {}
@@ -102,12 +86,12 @@ class AIOHTTPInterceptor(BaseInterceptor):
         # or silent model are enabled, otherwise this statement won't
         # be reached (an exception will be raised before).
         if not mock:
-            return _request(session, method, url,
-                            data=data, headers=headers, **kw)
+            return await _request(session, method, url,
+                                  data=data, headers=headers, **kw)
 
         # Simulate network delay
         if mock._delay:
-            yield from asyncio.sleep(mock._delay / 1000)  # noqa
+            await asyncio.sleep(mock._delay / 1000)  # noqa
 
         # Shortcut to mock response
         res = mock._response
@@ -145,16 +129,14 @@ class AIOHTTPInterceptor(BaseInterceptor):
         return _res
 
     def _patch(self, path):
-        # If not modern Python, just ignore patch
-        if not asyncio:
+        # If not able to import aiohttp dependencies, skip
+        if not yarl or not multidict:
             return None
 
-        @asyncio.coroutine
-        def handler(session, method, url, data=None, headers=None, **kw):
-            return (yield from self._on_request(
+        async def handler(session, method, url, data=None, headers=None, **kw):
+            return await self._on_request(
                 _request, session, method, url,
                 data=data, headers=headers, **kw)
-            )
 
         try:
             # Create a new patcher for Urllib3 urlopen function
diff --git a/pook/interceptors/http.py b/pook/interceptors/http.py
index 5232f68..56bce11 100644
--- a/pook/interceptors/http.py
+++ b/pook/interceptors/http.py
@@ -1,18 +1,10 @@
-import sys
 import socket
 from ..request import Request
 from .base import BaseInterceptor
 
-# Support Python 2/3
-try:
-    import mock
-except Exception:
-    from unittest import mock
+from unittest import mock
 
-if sys.version_info < (3,):     # Python 2
-    from httplib import responses as http_reasons, _CS_REQ_SENT
-else:                           # Python 3
-    from http.client import responses as http_reasons, _CS_REQ_SENT
+from http.client import responses as http_reasons, _CS_REQ_SENT
 
 PATCHES = (
     'http.client.HTTPConnection.request',
diff --git a/pook/interceptors/urllib3.py b/pook/interceptors/urllib3.py
index 01ee5b8..d96db20 100644
--- a/pook/interceptors/urllib3.py
+++ b/pook/interceptors/urllib3.py
@@ -1,25 +1,14 @@
 import io
-import sys
 from ..request import Request
 from .base import BaseInterceptor
 from .http import URLLIB3_BYPASS
 
-# Support Python 2/3
-try:
-    import mock
-except Exception:
-    from unittest import mock
-
-if sys.version_info < (3,):     # Python 2
-    from httplib import (
-        responses as http_reasons,
-        HTTPResponse as ClientHTTPResponse,
-    )
-else:                           # Python 3
-    from http.client import (
-        responses as http_reasons,
-        HTTPResponse as ClientHTTPResponse,
-    )
+from unittest import mock
+
+from http.client import (
+    responses as http_reasons,
+    HTTPResponse as ClientHTTPResponse,
+)
 
 PATCHES = (
     'requests.packages.urllib3.connectionpool.HTTPConnectionPool.urlopen',
diff --git a/pook/matchers/query.py b/pook/matchers/query.py
index 1629969..15d3d5c 100644
--- a/pook/matchers/query.py
+++ b/pook/matchers/query.py
@@ -1,10 +1,6 @@
-import sys
 from .base import BaseMatcher
 
-if sys.version_info < (3,):     # Python 2
-    from urlparse import parse_qs
-else:                           # Python 3
-    from urllib.parse import parse_qs
+from urllib.parse import parse_qs
 
 
 class QueryMatcher(BaseMatcher):
diff --git a/pook/matchers/url.py b/pook/matchers/url.py
index 8939a55..6358989 100644
--- a/pook/matchers/url.py
+++ b/pook/matchers/url.py
@@ -1,14 +1,10 @@
 import re
-import sys
 from .base import BaseMatcher
 from .path import PathMatcher
 from .query import QueryMatcher
 from ..regex import isregex
 
-if sys.version_info < (3,):     # Python 2
-    from urlparse import urlparse
-else:                           # Python 3
-    from urllib.parse import urlparse
+from urllib.parse import urlparse
 
 # URI protocol test regular expression
 protoregex = re.compile('^http[s]?://', re.IGNORECASE)
diff --git a/pook/request.py b/pook/request.py
index c7b2317..a582168 100644
--- a/pook/request.py
+++ b/pook/request.py
@@ -1,4 +1,3 @@
-import sys
 import json as _json
 
 from .regex import isregex
@@ -6,10 +5,7 @@ from .headers import HTTPHeaderDict
 from .helpers import trigger_methods
 from .matchers.url import protoregex
 
-if sys.version_info < (3,):     # Python 2
-    from urlparse import urlparse, parse_qs, urlunparse
-else:                           # Python 3
-    from urllib.parse import urlparse, parse_qs, urlunparse
+from urllib.parse import urlparse, parse_qs, urlunparse
 
 
 class Request(object):
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 7f796d2..d6cb5de 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -1,8 +1,8 @@
 flake8
 wheel>=0.29
 coveralls>=1.1
-pytest~=3.0.3
-pytest-cov~=2.3.1
+pytest~=7.2.0
+pytest-cov~=4.0.0
 pytest-flakes~=1.0.1
 nose~=1.3.7
 Sphinx~=1.4.8
@@ -10,5 +10,6 @@ sphinx-rtd-theme~=0.1.9
 requests>=2.20.0
 urllib3>=1.24.2
 bumpversion~=0.5.3
-aiohttp~=3.6.2 ; python_version >= '3.5.0'
+aiohttp~=3.8.3
 mocket~=1.6.0
+pytest-asyncio~=0.20.3
diff --git a/requirements.txt b/requirements.txt
index 23451be..9485cf8 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,4 +1,3 @@
 jsonschema>=2.5.1
 xmltodict>=0.11.0
 furl>=0.5.6
-mock>=2.0.0 ; python_version < '3.3'
diff --git a/setup.py b/setup.py
index 9b6b350..c09ef00 100644
--- a/setup.py
+++ b/setup.py
@@ -92,11 +92,12 @@ setup(
         'Development Status :: 5 - Production/Stable',
         'Natural Language :: English',
         'License :: OSI Approved :: MIT License',
-        'Programming Language :: Python :: 3',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',
         'Programming Language :: Python :: 3.8',
+        'Programming Language :: Python :: 3.9',
+        'Programming Language :: Python :: 3.10',
         'Topic :: Software Development',
         'Topic :: Software Development :: Libraries :: Python Modules',
         'Programming Language :: Python :: Implementation :: CPython',
diff --git a/tox.ini b/tox.ini
index 9a0d753..6a4682b 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,5 +1,5 @@
 [tox]
-envlist = {py27,py32,py33,py34,py35,pypy}
+envlist = {py35,py36,py37,py38,py39,py310,pypy}
 
 [testenv]
 setenv =


Test patch:
diff --git a/tests/integration/engines_test.py b/tests/integration/engines_test.py
index e06a3ab..9dc4b32 100644
--- a/tests/integration/engines_test.py
+++ b/tests/integration/engines_test.py
@@ -5,7 +5,7 @@ import subprocess
 # List of engine specific test commands to run
 engine_tests = (
     'py.test tests/integration/engines/pytest_suite.py',
-    'nosetests tests/integration/engines/nose_suite.py',
+    # 'nosetests tests/integration/engines/nose_suite.py',
     'python -m unittest tests.integration.engines.unittest_suite',
 )
 
diff --git a/tests/unit/interceptors/aiohttp_test.py b/tests/unit/interceptors/aiohttp_test.py
new file mode 100644
index 0000000..f68efa3
--- /dev/null
+++ b/tests/unit/interceptors/aiohttp_test.py
@@ -0,0 +1,36 @@
+import aiohttp
+import pook
+import pytest
+
+
+pytestmark = pytest.mark.asyncio
+
+URL = "https://httpbin.org/status/404"
+
+
+def _pook_url():
+    return pook.head(URL).reply(200).mock
+
+
+async def test_async_with_request():
+    # Cannot use `@pook.on` with pytest marks
+    pook.on()
+    mock = _pook_url()
+    async with aiohttp.ClientSession() as session:
+        async with session.head(URL) as req:
+            assert req.status == 200
+    pook.off()
+
+    assert len(mock.matches) == 1
+
+
+async def test_await_request():
+    # Cannot use `@pook.on` with pytest marks
+    pook.on()
+    mock = _pook_url()
+    async with aiohttp.ClientSession() as session:
+        req = await session.head(URL)
+        assert req.status == 200
+    pook.off()
+
+    assert len(mock.matches) == 1
diff --git a/tests/unit/interceptors/urllib3_test.py b/tests/unit/interceptors/urllib3_test.py
index 4e8c278..063e0f8 100644
--- a/tests/unit/interceptors/urllib3_test.py
+++ b/tests/unit/interceptors/urllib3_test.py
@@ -15,7 +15,6 @@ def assert_chunked_response(input_data, expected):
 
     assert r.status == 204
 
-    # py2 returns decoded chunks, while py3 does not
     chunks = list(r.read_chunked())
     chunks = [c.decode() if isinstance(c, bytes) else c for c in chunks]
     assert chunks == expected