Repository: getsentry/responses

Issue:
Add expected response from a file
In your [introductory blog post](http://cramer.io/2014/05/20/mocking-requests-with-responses/) you have the last example with a lot of such calls:

```
responses.add(
            responses.GET, 'http://jenkins.example.com/job/server/2/logText/progressiveHtml/?start=0',
            match_querystring=True,
            adding_headers={'X-Text-Size': '7'},
            body='Foo bar')
```

This is clearly a lack of the feature and the workaround because it is barely practical to stuff one's test suite with big response strings. And a full response file can contain all of status, headers and response body. It can look like this (py3):

```
from io import BytesIO
from http.client import HTTPResponse

import responses


class FakeSocket():

  _stream = None


  def __init__(self, s):
    self._stream = BytesIO(s)

  def makefile(self, *args, **kwargs):
    return self._stream


def parse_response(request_string):
  response = HTTPResponse(FakeSocket(request_string))
  response.begin()
  return response

class ResponseMock(responses.RequestsMock):

  def add_from_file(self, method, url, filename, match_querystring = False, stream = False):  
    with open(filename, 'rb') as f:
      r = parse_response(f.read())

    content_type = r.getheader('content-type', 'text/plain')

    self.add(method, url, r.read(), match_querystring, r.status, r.getheaders(), stream, content_type)
```

Response file can look like this:

```
HTTP/1.1 200 OK
Date: Sat, 22 Nov 2014 12:00:40 GMT
Content-Type: text/xml; charset="utf-8"
Connection: close

Some <em>HTML</em> goes here.
```


Discussion:
codecov[bot]: # [Codecov](https://app.codecov.io/gh/getsentry/responses/compare/581?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry) Report
Base: **100.00**% // Head: **100.00**% // No change to project coverage :thumbsup:
> Coverage data is based on head [(`186d1b7`)](https://app.codecov.io/gh/getsentry/responses/compare/581?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry) compared to base [(`f431d49`)](https://codecov.io/gh/getsentry/responses/commit/f431d49021252657e55253c18f0f8493c61e6ae6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry).
> Patch coverage: 100.00% of modified lines in pull request are covered.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff            @@
##            master      #581   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            9         9           
  Lines         2730      2765   +35     
=========================================
+ Hits          2730      2765   +35     
```


| [Impacted Files](https://app.codecov.io/gh/getsentry/responses/compare/581?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry) | Coverage Δ | |
|---|---|---|
| [responses/\_\_init\_\_.py](https://app.codecov.io/gh/getsentry/responses/compare/581/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry#diff-cmVzcG9uc2VzL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | |
| [responses/tests/test\_recorder.py](https://app.codecov.io/gh/getsentry/responses/compare/581/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry#diff-cmVzcG9uc2VzL3Rlc3RzL3Rlc3RfcmVjb3JkZXIucHk=) | `100.00% <100.00%> (ø)` | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry)

</details>

[:umbrella: View full report at Codecov](https://app.codecov.io/gh/getsentry/responses/compare/581?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry).

beliaev-maksim: @markstory @chadwhitacre 
historical moment :)
we went down from more than 70 issues last year, to 0, once this PR will be merged and new package is released. Implemented lots of new feature and bug fixes for them :D 
Making the library №1 for mocking out requests
chadwhitacre: Awesome progress, @beliaev-maksim! 😍

Patch:
diff --git a/CHANGES b/CHANGES
index 2675334..662cfc3 100644
--- a/CHANGES
+++ b/CHANGES
@@ -1,14 +1,19 @@
 0.22.0
 ------
 
+* Update `requests` dependency to the version of 2.22.0 or higher. See #584.
 * [BETA] Added possibility to record responses to TOML files via `@_recorder.record(file_path="out.toml")` decorator.
+* [BETA] Added possibility to replay responses (populate registry) from TOML files
+  via `responses._add_from_file(file_path="out.toml")` method.
 * Fix type for the `mock`'s patcher object. See #556
+* Fix type annotation for `CallList`
 * Add `passthrough` argument to `BaseResponse` object. See #557
 * Fix `registries` leak. See #563
 * Add support for the `loose` version of `json_params_matcher` via named argument `strict_match`. See #551
 * Add lists support as JSON objects in `json_params_matcher`. See #559
 * Added project links to pypi listing.
 * Fix `MaxRetryError` exception. Replace exception by `RetryError` according to `requests` implementation. See #572.
+* Adjust error message when `Retry` is exhausted. See #580.
 
 0.21.0
 ------
diff --git a/README.rst b/README.rst
index 357bd76..d03dad4 100644
--- a/README.rst
+++ b/README.rst
@@ -17,7 +17,7 @@ A utility library for mocking out the ``requests`` Python library.
 
 ..  note::
 
-    Responses requires Python 3.7 or newer, and requests >= 2.0
+    Responses requires Python 3.7 or newer, and requests >= 2.22.0
 
 
 Table of Contents
diff --git a/responses/__init__.py b/responses/__init__.py
index 3cfafb8..24d664b 100644
--- a/responses/__init__.py
+++ b/responses/__init__.py
@@ -2,8 +2,6 @@ import inspect
 import json as json_module
 import logging
 from collections import namedtuple
-from collections.abc import Sequence
-from collections.abc import Sized
 from functools import wraps
 from http import client
 from itertools import groupby
@@ -18,12 +16,15 @@ from typing import Iterator
 from typing import List
 from typing import Mapping
 from typing import Optional
+from typing import Sequence
+from typing import Sized
 from typing import Tuple
 from typing import Type
 from typing import Union
 from typing import overload
 from warnings import warn
 
+import toml as _toml
 from requests.adapters import HTTPAdapter
 from requests.adapters import MaxRetryError
 from requests.exceptions import ConnectionError
@@ -64,6 +65,7 @@ from urllib.parse import urlunsplit
 
 if TYPE_CHECKING:  # pragma: no cover
     # import only for linter run
+    import os
     from unittest.mock import _patch as _mock_patcher
 
     from requests import PreparedRequest
@@ -217,7 +219,7 @@ def get_wrapped(
     return wrapper
 
 
-class CallList(Sequence, Sized):  # type: ignore[type-arg]
+class CallList(Sequence[Any], Sized):
     def __init__(self) -> None:
         self._calls: List[Call] = []
 
@@ -227,7 +229,7 @@ class CallList(Sequence, Sized):  # type: ignore[type-arg]
     def __len__(self) -> int:
         return len(self._calls)
 
-    def __getitem__(self, idx: int) -> Call:  # type: ignore[override]
+    def __getitem__(self, idx: Union[int, slice]) -> Union[Call, List[Call]]:
         return self._calls[idx]
 
     def add(self, request: "PreparedRequest", response: _Body) -> None:
@@ -415,6 +417,25 @@ class BaseResponse(object):
         return bool(urlsplit(self.url).query)
 
     def _url_matches(self, url: _URLPatternType, other: str) -> bool:
+        """Compares two URLs.
+
+        Compares only scheme, netloc and path. If 'url' is a re.Pattern, then checks that
+        'other' matches the pattern.
+
+        Parameters
+        ----------
+        url : Union["Pattern[str]", str]
+            Reference URL or Pattern to compare.
+
+        other : str
+            URl that should be compared.
+
+        Returns
+        -------
+        bool
+            True, if URLs are identical or 'other' matches the pattern.
+
+        """
         if isinstance(url, str):
             if _has_unicode(url):
                 url = _clean_unicode(url)
@@ -661,9 +682,25 @@ class RequestsMock(object):
         self._thread_lock = _ThreadingLock()
 
     def get_registry(self) -> FirstMatchRegistry:
+        """Returns current registry instance with responses.
+
+        Returns
+        -------
+        FirstMatchRegistry
+            Current registry instance with responses.
+
+        """
         return self._registry
 
     def _set_registry(self, new_registry: Type[FirstMatchRegistry]) -> None:
+        """Replaces current registry with `new_registry`.
+
+        Parameters
+        ----------
+        new_registry : Type[FirstMatchRegistry]
+            Class reference of the registry that should be set, eg OrderedRegistry
+
+        """
         if self.registered():
             err_msg = (
                 "Cannot replace Registry, current registry has responses.\n"
@@ -674,6 +711,7 @@ class RequestsMock(object):
         self._registry = new_registry()
 
     def reset(self) -> None:
+        """Resets registry (including type), calls, passthru_prefixes to default values."""
         self._registry = FirstMatchRegistry()
         self._calls.reset()
         self.passthru_prefixes = ()
@@ -733,6 +771,21 @@ class RequestsMock(object):
         response = Response(method=method, url=url, body=body, **kwargs)
         return self._registry.add(response)
 
+    def _add_from_file(self, file_path: "Union[str, bytes, os.PathLike[Any]]") -> None:
+        with open(file_path) as file:
+            data = _toml.load(file)
+
+        for rsp in data["responses"]:
+            rsp = rsp["response"]
+            self.add(
+                method=rsp["method"],
+                url=rsp["url"],
+                body=rsp["body"],
+                status=rsp["status"],
+                content_type=rsp["content_type"],
+                auto_calculate_content_length=rsp["auto_calculate_content_length"],
+            )
+
     def delete(self, *args: Any, **kwargs: Any) -> BaseResponse:
         return self.add(DELETE, *args, **kwargs)
 
@@ -1029,7 +1082,7 @@ class RequestsMock(object):
                 retries = retries.increment(
                     method=response.request.method,  # type: ignore[misc]
                     url=response.url,  # type: ignore[misc]
-                    response=response,  # type: ignore[misc]
+                    response=response.raw,  # type: ignore[misc]
                 )
                 return self._on_request(adapter, request, retries=retries, **kwargs)
             except MaxRetryError as e:
@@ -1110,6 +1163,7 @@ __all__ = [
     # Exposed by the RequestsMock class:
     "activate",
     "add",
+    "_add_from_file",
     "add_callback",
     "add_passthru",
     "_deprecated_assert_all_requests_are_fired",
@@ -1144,6 +1198,7 @@ __all__ = [
 # expose only methods and/or read-only methods
 activate = _default_mock.activate
 add = _default_mock.add
+_add_from_file = _default_mock._add_from_file
 add_callback = _default_mock.add_callback
 add_passthru = _default_mock.add_passthru
 _deprecated_assert_all_requests_are_fired = _default_mock.assert_all_requests_are_fired
diff --git a/responses/registries.py b/responses/registries.py
index 049df23..4b7e233 100644
--- a/responses/registries.py
+++ b/responses/registries.py
@@ -75,9 +75,32 @@ class FirstMatchRegistry(object):
 
 
 class OrderedRegistry(FirstMatchRegistry):
+    """Registry where `Response` objects are dependent on the insertion order and invocation index.
+
+    OrderedRegistry applies the rule of first in - first out. Responses should be invoked in
+    the same order in which they were added to the registry. Otherwise, an error is returned.
+    """
+
     def find(
         self, request: "PreparedRequest"
     ) -> Tuple[Optional["BaseResponse"], List[str]]:
+        """Find the next registered `Response` and check if it matches the request.
+
+        Search is performed by taking the first element of the registered responses list
+        and removing this object (popping from the list).
+
+        Parameters
+        ----------
+        request : PreparedRequest
+            Request that was caught by the custom adapter.
+
+        Returns
+        -------
+        Tuple[Optional["BaseResponse"], List[str]]
+            Matched `Response` object and empty list in case of match.
+            Otherwise, None and a list with reasons for not finding a match.
+
+        """
 
         if not self.registered:
             return None, ["No more registered responses"]
diff --git a/setup.py b/setup.py
index 115d183..03e980c 100644
--- a/setup.py
+++ b/setup.py
@@ -17,7 +17,7 @@ from setuptools.command.test import test as TestCommand
 setup_requires = []
 
 install_requires = [
-    "requests>=2.0,<3.0",
+    "requests>=2.22.0,<3.0",
     "urllib3>=1.25.10",
     "toml",
     "types-toml",


Test patch:
diff --git a/responses/tests/test_recorder.py b/responses/tests/test_recorder.py
index 0d73379..d70d60f 100644
--- a/responses/tests/test_recorder.py
+++ b/responses/tests/test_recorder.py
@@ -3,9 +3,58 @@ from pathlib import Path
 import requests
 import toml
 
+import responses
 from responses import _recorder
 
 
+def get_data(host, port):
+    data = {
+        "responses": [
+            {
+                "response": {
+                    "method": "GET",
+                    "url": f"http://{host}:{port}/404",
+                    "body": "404 Not Found",
+                    "status": 404,
+                    "content_type": "text/plain",
+                    "auto_calculate_content_length": False,
+                }
+            },
+            {
+                "response": {
+                    "method": "GET",
+                    "url": f"http://{host}:{port}/status/wrong",
+                    "body": "Invalid status code",
+                    "status": 400,
+                    "content_type": "text/plain",
+                    "auto_calculate_content_length": False,
+                }
+            },
+            {
+                "response": {
+                    "method": "GET",
+                    "url": f"http://{host}:{port}/500",
+                    "body": "500 Internal Server Error",
+                    "status": 500,
+                    "content_type": "text/plain",
+                    "auto_calculate_content_length": False,
+                }
+            },
+            {
+                "response": {
+                    "method": "PUT",
+                    "url": f"http://{host}:{port}/202",
+                    "body": "OK",
+                    "status": 202,
+                    "content_type": "text/plain",
+                    "auto_calculate_content_length": False,
+                }
+            },
+        ]
+    }
+    return data
+
+
 class TestRecord:
     def setup(self):
         self.out_file = Path("out.toml")
@@ -35,7 +84,7 @@ class TestRecord:
 
         def another():
             requests.get(url500)
-            requests.get(url202)
+            requests.put(url202)
 
         @_recorder.record(file_path=self.out_file)
         def run():
@@ -48,47 +97,46 @@ class TestRecord:
         with open(self.out_file) as file:
             data = toml.load(file)
 
-        assert data == {
-            "responses": [
-                {
-                    "response": {
-                        "method": "GET",
-                        "url": f"http://{httpserver.host}:{httpserver.port}/404",
-                        "body": "404 Not Found",
-                        "status": 404,
-                        "content_type": "text/plain",
-                        "auto_calculate_content_length": False,
-                    }
-                },
-                {
-                    "response": {
-                        "method": "GET",
-                        "url": f"http://{httpserver.host}:{httpserver.port}/status/wrong",
-                        "body": "Invalid status code",
-                        "status": 400,
-                        "content_type": "text/plain",
-                        "auto_calculate_content_length": False,
-                    }
-                },
-                {
-                    "response": {
-                        "method": "GET",
-                        "url": f"http://{httpserver.host}:{httpserver.port}/500",
-                        "body": "500 Internal Server Error",
-                        "status": 500,
-                        "content_type": "text/plain",
-                        "auto_calculate_content_length": False,
-                    }
-                },
-                {
-                    "response": {
-                        "method": "GET",
-                        "url": f"http://{httpserver.host}:{httpserver.port}/202",
-                        "body": "OK",
-                        "status": 202,
-                        "content_type": "text/plain",
-                        "auto_calculate_content_length": False,
-                    }
-                },
-            ]
-        }
+        assert data == get_data(httpserver.host, httpserver.port)
+
+
+class TestReplay:
+    def teardown(self):
+        out_file = Path("out.toml")
+        if out_file.exists():
+            out_file.unlink()
+
+        assert not out_file.exists()
+
+    def test_add_from_file(self):
+        with open("out.toml", "w") as file:
+            toml.dump(get_data("example.com", "8080"), file)
+
+        @responses.activate
+        def run():
+            responses.patch("http://httpbin.org")
+            responses._add_from_file(file_path="out.toml")
+            responses.post("http://httpbin.org/form")
+
+            assert responses.registered()[0].url == "http://httpbin.org/"
+            assert responses.registered()[1].url == "http://example.com:8080/404"
+            assert (
+                responses.registered()[2].url == "http://example.com:8080/status/wrong"
+            )
+            assert responses.registered()[3].url == "http://example.com:8080/500"
+            assert responses.registered()[4].url == "http://example.com:8080/202"
+            assert responses.registered()[5].url == "http://httpbin.org/form"
+
+            assert responses.registered()[0].method == "PATCH"
+            assert responses.registered()[2].method == "GET"
+            assert responses.registered()[4].method == "PUT"
+            assert responses.registered()[5].method == "POST"
+
+            assert responses.registered()[2].status == 400
+            assert responses.registered()[3].status == 500
+
+            assert responses.registered()[3].body == "500 Internal Server Error"
+
+            assert responses.registered()[3].content_type == "text/plain"
+
+        run()
diff --git a/responses/tests/test_responses.py b/responses/tests/test_responses.py
index be87497..53924df 100644
--- a/responses/tests/test_responses.py
+++ b/responses/tests/test_responses.py
@@ -2450,6 +2450,23 @@ class TestMaxRetry:
         run()
         assert_reset()
 
+    def test_max_retries_exceed_msg(self):
+        @responses.activate(registry=registries.OrderedRegistry)
+        def run():
+            url = "https://example.com"
+            responses.get(url, body="Error", status=500)
+            responses.get(url, body="Error", status=500)
+
+            session = self.set_session(total=1)
+
+            with pytest.raises(RetryError) as err:
+                session.get(url)
+
+            assert "too many 500 error responses" in str(err.value)
+
+        run()
+        assert_reset()
+
     def test_adapter_retry_untouched(self):
         """Validate that every new request uses brand-new Retry object"""