Repository: edgi-govdata-archiving/wayback

Issue:
search_calls_per_second needs to be dialed down
I was running some fairly simple data retrieval in [this Notebook](https://github.com/edsu/foiaonline/blob/main/Notebook.ipynb)(see the Wayback section) and I discovered that I got completely blocked from accessing web.archive.org! Luckily I remembered that there was Internet Archive's #wayback-researchers Slack channel, where I got this reponse.

> Hi edsu - I found your /cdx requests from 4:29UTC. Those requests are limited to an average of 60/min. Over that and we start sending 429s. If 429s are ignored for more than a minute we block the IP at the firewall (no connection) for 1 hour, which is what happened to you.  Subsequent 429s over a given period will double that time each occurrence. If you can keeping your api request < 60/minute you will prevent this from happening.

I thought that the openwayback module's defaults would have prevented me from going over the 60 requests per minute (one per second) and I thought wayback's support for handling 429 responses would have backed off sufficiently fast. I suspect that the goal posts on the server side have changed recently because I had code that worked a month or so ago, which stopped working (resulting in the block).

I was able to get around this by using a custom WaybackSession where I set the `search_calls_per_second` to `0.5`, but I suspect `1.0` would probably work better. Maybe the default could me moved down from 1.5 to 1.0?

Also, perhaps there needs to be some logic to make sure to wait a minute when encountering a 429 as well?

Comments:

Patch:
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 7d0559d..a7aaa14 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -2,7 +2,7 @@
 # etc.) but not necessarily required for _using_ it. Some dev tooling requires
 # newer Python versions than the package itself (>=3.8)
 wheel
-check-wheel-contents ~=0.5.0
+check-wheel-contents ~=0.6.0
 flake8 ~=6.1.0
 twine
 -r requirements-test.txt
diff --git a/wayback/_client.py b/wayback/_client.py
index 3383f53..f26acf0 100644
--- a/wayback/_client.py
+++ b/wayback/_client.py
@@ -347,7 +347,7 @@ class WaybackSession(_utils.DisableAfterCloseSession, requests.Session):
     user_agent : str, optional
         A custom user-agent string to use in all requests. Defaults to:
         `wayback/{version} (+https://github.com/edgi-govdata-archiving/wayback)`
-    search_calls_per_second : int or float, default: 1.5
+    search_calls_per_second : int or float, default: 1
         The maximum number of calls made to the search API per second.
         To disable the rate limit, set this to 0.
     memento_calls_per_second : int or float, default: 30
@@ -366,7 +366,7 @@ class WaybackSession(_utils.DisableAfterCloseSession, requests.Session):
     handleable_errors = (ConnectionError,) + retryable_errors
 
     def __init__(self, retries=6, backoff=2, timeout=60, user_agent=None,
-                 search_calls_per_second=1.5, memento_calls_per_second=30):
+                 search_calls_per_second=1, memento_calls_per_second=30):
         super().__init__()
         self.retries = retries
         self.backoff = backoff
diff --git a/wayback/_utils.py b/wayback/_utils.py
index 00b1a4f..c99bc75 100644
--- a/wayback/_utils.py
+++ b/wayback/_utils.py
@@ -204,7 +204,7 @@ def set_memento_url_mode(url, mode):
 
 
 @contextmanager
-def rate_limited(calls_per_second=2, group='default'):
+def rate_limited(calls_per_second=1, group='default'):
     """
     A context manager that restricts entries to its body to occur only N times
     per second (N can be a float). The current thread will be put to sleep in


Test patch:
diff --git a/wayback/tests/test_client.py b/wayback/tests/test_client.py
index cc5548b..cd8e676 100644
--- a/wayback/tests/test_client.py
+++ b/wayback/tests/test_client.py
@@ -792,7 +792,7 @@ def test_search_rate_limits(self):
                 next(client.search('zew.de'))
         duration_with_limits_custom = time.time() - start_time
 
-        assert 1.3 <= duration_with_limits <= 1.4
+        assert 1.9 <= duration_with_limits <= 2.1
         assert 0.0 <= duration_without_limits <= 0.05
         assert 0.0 <= duration_with_limits_custom <= 1.05