Repository: getsentry/responses
Pull request #582 (2022-09-01 19:33:42)

add docs about beta features

codecov[bot]: # [Codecov](https://app.codecov.io/gh/getsentry/responses/compare/582?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry) Report
Base: **100.00**% // Head: **100.00**% // No change to project coverage :thumbsup:
> Coverage data is based on head [(`9e4d221`)](https://app.codecov.io/gh/getsentry/responses/compare/582?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry) compared to base [(`6144ef5`)](https://codecov.io/gh/getsentry/responses/commit/6144ef5dcc84389e395592e54628ebb0a42e34de?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry).
> Patch has no changes to coverable lines.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff            @@
##            master      #582   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            9         9           
  Lines         2730      2730           
=========================================
  Hits          2730      2730           
```



Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry)

</details>

[:umbrella: View full report at Codecov](https://app.codecov.io/gh/getsentry/responses/compare/582?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry).


Patch:
diff --git a/CHANGES b/CHANGES
index 88e9f0b..005ddea 100644
--- a/CHANGES
+++ b/CHANGES
@@ -1,15 +1,20 @@
 0.22.0
 ------
 
+* Update `requests` dependency to the version of 2.22.0 or higher. See #584.
 * [BETA] Added possibility to record responses to TOML files via `@_recorder.record(file_path="out.toml")` decorator.
+* [BETA] Added possibility to replay responses (populate registry) from TOML files
+  via `responses._add_from_file(file_path="out.toml")` method.
 * Fix type for the `mock`'s patcher object. See #556
 * Fix type annotation for `CallList`
 * Add `passthrough` argument to `BaseResponse` object. See #557
 * Fix `registries` leak. See #563
+* `OriginalResponseShim` is removed. See #585
 * Add support for the `loose` version of `json_params_matcher` via named argument `strict_match`. See #551
 * Add lists support as JSON objects in `json_params_matcher`. See #559
 * Added project links to pypi listing.
 * Fix `MaxRetryError` exception. Replace exception by `RetryError` according to `requests` implementation. See #572.
+* Adjust error message when `Retry` is exhausted. See #580.
 
 0.21.0
 ------
diff --git a/README.rst b/README.rst
index 357bd76..1c8a794 100644
--- a/README.rst
+++ b/README.rst
@@ -17,7 +17,7 @@ A utility library for mocking out the ``requests`` Python library.
 
 ..  note::
 
-    Responses requires Python 3.7 or newer, and requests >= 2.0
+    Responses requires Python 3.7 or newer, and requests >= 2.22.0
 
 
 Table of Contents
@@ -62,7 +62,80 @@ Please ensure to update your code according to the guidance.
      - Use ``responses.mock.assert_all_requests_are_fired``,
        ``responses.mock.passthru_prefixes``, ``responses.mock.target`` instead.
 
+BETA Features
+-------------
+Below you can find a list of BETA features. Although we will try to keep the API backwards compatible
+with released version, we reserve the right to change these APIs before they are considered stable. Please share your feedback via
+`GitHub Issues <https://github.com/getsentry/responses/issues>`_.
 
+Record Responses to files
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You can perform real requests to the server and ``responses`` will automatically record the output to the
+file. Recorded data is stored in `toml <https://toml.io>`_ format.
+
+Apply ``@responses._recorder.record(file_path="out.toml")`` decorator to any function where you perform
+requests to record responses to ``out.toml`` file.
+
+Following code
+
+.. code-block:: python
+
+    import requests
+    from responses import _recorder
+
+
+    def another():
+        rsp = requests.get("https://httpstat.us/500")
+        rsp = requests.get("https://httpstat.us/202")
+
+
+    @_recorder.record(file_path="out.toml")
+    def test_recorder():
+        rsp = requests.get("https://httpstat.us/404")
+        rsp = requests.get("https://httpbin.org/status/wrong")
+        another()
+
+will produce next output:
+
+.. code-block:: toml
+
+    [[responses]]
+
+    [responses.response]
+    method = "GET"
+    url = "https://httpstat.us/404"
+    body = "404 Not Found"
+    status = 404
+    content_type = "text/plain"
+    auto_calculate_content_length = false
+    [[responses]]
+
+    [responses.response]
+    method = "GET"
+    url = "https://httpbin.org/status/wrong"
+    body = "Invalid status code"
+    status = 400
+    content_type = "text/plain"
+    auto_calculate_content_length = false
+    [[responses]]
+
+    [responses.response]
+    method = "GET"
+    url = "https://httpstat.us/500"
+    body = "500 Internal Server Error"
+    status = 500
+    content_type = "text/plain"
+    auto_calculate_content_length = false
+    [[responses]]
+
+    [responses.response]
+    method = "GET"
+    url = "https://httpstat.us/202"
+    body = "202 Accepted"
+    status = 202
+    content_type = "text/plain"
+    auto_calculate_content_length = false
 
 Basics
 ------
diff --git a/responses/__init__.py b/responses/__init__.py
index df372a0..a83b3e2 100644
--- a/responses/__init__.py
+++ b/responses/__init__.py
@@ -24,6 +24,7 @@ from typing import Union
 from typing import overload
 from warnings import warn
 
+import toml as _toml
 from requests.adapters import HTTPAdapter
 from requests.adapters import MaxRetryError
 from requests.exceptions import ConnectionError
@@ -64,6 +65,7 @@ from urllib.parse import urlunsplit
 
 if TYPE_CHECKING:  # pragma: no cover
     # import only for linter run
+    import os
     from unittest.mock import _patch as _mock_patcher
 
     from requests import PreparedRequest
@@ -482,6 +484,32 @@ class BaseResponse(object):
         return True, ""
 
 
+def _form_response(
+    body: Union[BufferedReader, BytesIO],
+    headers: Optional[Mapping[str, str]],
+    status: int,
+) -> HTTPResponse:
+    # The requests library's cookie handling depends on the response object
+    # having an original response object with the headers as the `msg`, so
+    # we give it what it needs.
+    data = BytesIO()
+    data.close()
+
+    orig_response = HTTPResponse(
+        body=data,  # required to avoid "ValueError: Unable to determine whether fp is closed."
+        msg=headers,
+        preload_content=False,
+    )
+    return HTTPResponse(
+        status=status,
+        reason=client.responses.get(status, None),
+        body=body,
+        headers=headers,
+        original_response=orig_response,
+        preload_content=False,
+    )
+
+
 class Response(BaseResponse):
     def __init__(
         self,
@@ -543,14 +571,7 @@ class Response(BaseResponse):
             content_length = len(body.getvalue())
             headers["Content-Length"] = str(content_length)
 
-        return HTTPResponse(
-            status=status,
-            reason=client.responses.get(status, None),
-            body=body,
-            headers=headers,
-            original_response=OriginalResponseShim(headers),
-            preload_content=False,
-        )
+        return _form_response(body, headers, status)
 
     def __repr__(self) -> str:
         return (
@@ -612,14 +633,7 @@ class CallbackResponse(BaseResponse):
         body = _handle_body(body)
         headers.extend(r_headers)
 
-        return HTTPResponse(
-            status=status,
-            reason=client.responses.get(status, None),
-            body=body,
-            headers=headers,
-            original_response=OriginalResponseShim(headers),
-            preload_content=False,
-        )
+        return _form_response(body, headers, status)
 
 
 class PassthroughResponse(BaseResponse):
@@ -627,29 +641,6 @@ class PassthroughResponse(BaseResponse):
         super().__init__(*args, passthrough=True, **kwargs)
 
 
-class OriginalResponseShim(object):
-    """
-    Shim for compatibility with older versions of urllib3
-
-    requests cookie handling depends on responses having a property chain of
-    `response._original_response.msg` which contains the response headers [1]
-
-    Using HTTPResponse() for this purpose causes compatibility errors with
-    urllib3<1.23.0. To avoid adding more dependencies we can use this shim.
-
-    [1]: https://github.com/psf/requests/blob/75bdc998e2d/requests/cookies.py#L125
-    """
-
-    def __init__(self, headers: Any) -> None:
-        self.msg: Any = headers
-
-    def isclosed(self) -> bool:
-        return True
-
-    def close(self) -> None:
-        return
-
-
 class RequestsMock(object):
     DELETE: Literal["DELETE"] = "DELETE"
     GET: Literal["GET"] = "GET"
@@ -680,9 +671,25 @@ class RequestsMock(object):
         self._thread_lock = _ThreadingLock()
 
     def get_registry(self) -> FirstMatchRegistry:
+        """Returns current registry instance with responses.
+
+        Returns
+        -------
+        FirstMatchRegistry
+            Current registry instance with responses.
+
+        """
         return self._registry
 
     def _set_registry(self, new_registry: Type[FirstMatchRegistry]) -> None:
+        """Replaces current registry with `new_registry`.
+
+        Parameters
+        ----------
+        new_registry : Type[FirstMatchRegistry]
+            Class reference of the registry that should be set, eg OrderedRegistry
+
+        """
         if self.registered():
             err_msg = (
                 "Cannot replace Registry, current registry has responses.\n"
@@ -693,6 +700,7 @@ class RequestsMock(object):
         self._registry = new_registry()
 
     def reset(self) -> None:
+        """Resets registry (including type), calls, passthru_prefixes to default values."""
         self._registry = FirstMatchRegistry()
         self._calls.reset()
         self.passthru_prefixes = ()
@@ -752,6 +760,21 @@ class RequestsMock(object):
         response = Response(method=method, url=url, body=body, **kwargs)
         return self._registry.add(response)
 
+    def _add_from_file(self, file_path: "Union[str, bytes, os.PathLike[Any]]") -> None:
+        with open(file_path) as file:
+            data = _toml.load(file)
+
+        for rsp in data["responses"]:
+            rsp = rsp["response"]
+            self.add(
+                method=rsp["method"],
+                url=rsp["url"],
+                body=rsp["body"],
+                status=rsp["status"],
+                content_type=rsp["content_type"],
+                auto_calculate_content_length=rsp["auto_calculate_content_length"],
+            )
+
     def delete(self, *args: Any, **kwargs: Any) -> BaseResponse:
         return self.add(DELETE, *args, **kwargs)
 
@@ -1048,7 +1071,7 @@ class RequestsMock(object):
                 retries = retries.increment(
                     method=response.request.method,  # type: ignore[misc]
                     url=response.url,  # type: ignore[misc]
-                    response=response,  # type: ignore[misc]
+                    response=response.raw,  # type: ignore[misc]
                 )
                 return self._on_request(adapter, request, retries=retries, **kwargs)
             except MaxRetryError as e:
@@ -1129,6 +1152,7 @@ __all__ = [
     # Exposed by the RequestsMock class:
     "activate",
     "add",
+    "_add_from_file",
     "add_callback",
     "add_passthru",
     "_deprecated_assert_all_requests_are_fired",
@@ -1163,6 +1187,7 @@ __all__ = [
 # expose only methods and/or read-only methods
 activate = _default_mock.activate
 add = _default_mock.add
+_add_from_file = _default_mock._add_from_file
 add_callback = _default_mock.add_callback
 add_passthru = _default_mock.add_passthru
 _deprecated_assert_all_requests_are_fired = _default_mock.assert_all_requests_are_fired
diff --git a/responses/registries.py b/responses/registries.py
index 049df23..4b7e233 100644
--- a/responses/registries.py
+++ b/responses/registries.py
@@ -75,9 +75,32 @@ class FirstMatchRegistry(object):
 
 
 class OrderedRegistry(FirstMatchRegistry):
+    """Registry where `Response` objects are dependent on the insertion order and invocation index.
+
+    OrderedRegistry applies the rule of first in - first out. Responses should be invoked in
+    the same order in which they were added to the registry. Otherwise, an error is returned.
+    """
+
     def find(
         self, request: "PreparedRequest"
     ) -> Tuple[Optional["BaseResponse"], List[str]]:
+        """Find the next registered `Response` and check if it matches the request.
+
+        Search is performed by taking the first element of the registered responses list
+        and removing this object (popping from the list).
+
+        Parameters
+        ----------
+        request : PreparedRequest
+            Request that was caught by the custom adapter.
+
+        Returns
+        -------
+        Tuple[Optional["BaseResponse"], List[str]]
+            Matched `Response` object and empty list in case of match.
+            Otherwise, None and a list with reasons for not finding a match.
+
+        """
 
         if not self.registered:
             return None, ["No more registered responses"]
diff --git a/setup.py b/setup.py
index 115d183..03e980c 100644
--- a/setup.py
+++ b/setup.py
@@ -17,7 +17,7 @@ from setuptools.command.test import test as TestCommand
 setup_requires = []
 
 install_requires = [
-    "requests>=2.0,<3.0",
+    "requests>=2.22.0,<3.0",
     "urllib3>=1.25.10",
     "toml",
     "types-toml",


Test patch:
diff --git a/responses/tests/test_recorder.py b/responses/tests/test_recorder.py
index 0d73379..d70d60f 100644
--- a/responses/tests/test_recorder.py
+++ b/responses/tests/test_recorder.py
@@ -3,9 +3,58 @@ from pathlib import Path
 import requests
 import toml
 
+import responses
 from responses import _recorder
 
 
+def get_data(host, port):
+    data = {
+        "responses": [
+            {
+                "response": {
+                    "method": "GET",
+                    "url": f"http://{host}:{port}/404",
+                    "body": "404 Not Found",
+                    "status": 404,
+                    "content_type": "text/plain",
+                    "auto_calculate_content_length": False,
+                }
+            },
+            {
+                "response": {
+                    "method": "GET",
+                    "url": f"http://{host}:{port}/status/wrong",
+                    "body": "Invalid status code",
+                    "status": 400,
+                    "content_type": "text/plain",
+                    "auto_calculate_content_length": False,
+                }
+            },
+            {
+                "response": {
+                    "method": "GET",
+                    "url": f"http://{host}:{port}/500",
+                    "body": "500 Internal Server Error",
+                    "status": 500,
+                    "content_type": "text/plain",
+                    "auto_calculate_content_length": False,
+                }
+            },
+            {
+                "response": {
+                    "method": "PUT",
+                    "url": f"http://{host}:{port}/202",
+                    "body": "OK",
+                    "status": 202,
+                    "content_type": "text/plain",
+                    "auto_calculate_content_length": False,
+                }
+            },
+        ]
+    }
+    return data
+
+
 class TestRecord:
     def setup(self):
         self.out_file = Path("out.toml")
@@ -35,7 +84,7 @@ class TestRecord:
 
         def another():
             requests.get(url500)
-            requests.get(url202)
+            requests.put(url202)
 
         @_recorder.record(file_path=self.out_file)
         def run():
@@ -48,47 +97,46 @@ class TestRecord:
         with open(self.out_file) as file:
             data = toml.load(file)
 
-        assert data == {
-            "responses": [
-                {
-                    "response": {
-                        "method": "GET",
-                        "url": f"http://{httpserver.host}:{httpserver.port}/404",
-                        "body": "404 Not Found",
-                        "status": 404,
-                        "content_type": "text/plain",
-                        "auto_calculate_content_length": False,
-                    }
-                },
-                {
-                    "response": {
-                        "method": "GET",
-                        "url": f"http://{httpserver.host}:{httpserver.port}/status/wrong",
-                        "body": "Invalid status code",
-                        "status": 400,
-                        "content_type": "text/plain",
-                        "auto_calculate_content_length": False,
-                    }
-                },
-                {
-                    "response": {
-                        "method": "GET",
-                        "url": f"http://{httpserver.host}:{httpserver.port}/500",
-                        "body": "500 Internal Server Error",
-                        "status": 500,
-                        "content_type": "text/plain",
-                        "auto_calculate_content_length": False,
-                    }
-                },
-                {
-                    "response": {
-                        "method": "GET",
-                        "url": f"http://{httpserver.host}:{httpserver.port}/202",
-                        "body": "OK",
-                        "status": 202,
-                        "content_type": "text/plain",
-                        "auto_calculate_content_length": False,
-                    }
-                },
-            ]
-        }
+        assert data == get_data(httpserver.host, httpserver.port)
+
+
+class TestReplay:
+    def teardown(self):
+        out_file = Path("out.toml")
+        if out_file.exists():
+            out_file.unlink()
+
+        assert not out_file.exists()
+
+    def test_add_from_file(self):
+        with open("out.toml", "w") as file:
+            toml.dump(get_data("example.com", "8080"), file)
+
+        @responses.activate
+        def run():
+            responses.patch("http://httpbin.org")
+            responses._add_from_file(file_path="out.toml")
+            responses.post("http://httpbin.org/form")
+
+            assert responses.registered()[0].url == "http://httpbin.org/"
+            assert responses.registered()[1].url == "http://example.com:8080/404"
+            assert (
+                responses.registered()[2].url == "http://example.com:8080/status/wrong"
+            )
+            assert responses.registered()[3].url == "http://example.com:8080/500"
+            assert responses.registered()[4].url == "http://example.com:8080/202"
+            assert responses.registered()[5].url == "http://httpbin.org/form"
+
+            assert responses.registered()[0].method == "PATCH"
+            assert responses.registered()[2].method == "GET"
+            assert responses.registered()[4].method == "PUT"
+            assert responses.registered()[5].method == "POST"
+
+            assert responses.registered()[2].status == 400
+            assert responses.registered()[3].status == 500
+
+            assert responses.registered()[3].body == "500 Internal Server Error"
+
+            assert responses.registered()[3].content_type == "text/plain"
+
+        run()
diff --git a/responses/tests/test_responses.py b/responses/tests/test_responses.py
index be87497..53924df 100644
--- a/responses/tests/test_responses.py
+++ b/responses/tests/test_responses.py
@@ -2450,6 +2450,23 @@ class TestMaxRetry:
         run()
         assert_reset()
 
+    def test_max_retries_exceed_msg(self):
+        @responses.activate(registry=registries.OrderedRegistry)
+        def run():
+            url = "https://example.com"
+            responses.get(url, body="Error", status=500)
+            responses.get(url, body="Error", status=500)
+
+            session = self.set_session(total=1)
+
+            with pytest.raises(RetryError) as err:
+                session.get(url)
+
+            assert "too many 500 error responses" in str(err.value)
+
+        run()
+        assert_reset()
+
     def test_adapter_retry_untouched(self):
         """Validate that every new request uses brand-new Retry object"""