Repository: praw-dev/prawcore

Issue:
Getting 429 responses despite using absurdly high ratelimit_seconds
### Describe the Bug

When mass-removing contributors from a private sub, I am encountering occasional 429's. I can see from the debug logs that the limit remaining (I think from the headers) is ticking down until it reaches zero and then blows up. 

I'm curious how to best handle this. I am initializing the Reddit object with ratelimit_seconds=3000, timeout=120 which I would think is plenty enough given the documentation says the praw maximum wait period is 14 minutes.

Is there something I'm missing? Do I need to be watching the remaining number tick down and insert sleeps before it hits 0? I'm not using any kind of threading, which seems to be the usual cause of stuff like this.

Thanks in advance.

### Desired Result

PRAW limits itself to never get a 429 response, or traps it and tries again later in the background.

### Code to reproduce the bug

```Python
for user in users_to_remove:
        subreddit.contributor.remove(user)
```

### My code does not include sensitive credentials

- [x] Yes, I have removed sensitive credentials from my code.

### Relevant Logs

```Shell
2025-01-24 03:29:56,287 DEBUG:Response: 200 (24 bytes) (rst-3:rem-5.0:used-995 ratelimit) at 1737707396.2870934
2025-01-24 03:29:56,494 DEBUG:Response: 200 (24 bytes) (rst-3:rem-4.0:used-996 ratelimit) at 1737707396.4948132
2025-01-24 03:29:57,271 DEBUG:Response: 200 (24 bytes) (rst-2:rem-3.0:used-997 ratelimit) at 1737707397.2709942
2025-01-24 03:29:57,660 DEBUG:Response: 200 (24 bytes) (rst-2:rem-2.0:used-998 ratelimit) at 1737707397.66084
2025-01-24 03:29:58,622 DEBUG:Response: 200 (24 bytes) (rst-1:rem-1.0:used-999 ratelimit) at 1737707398.6224327
2025-01-24 03:29:59,193 DEBUG:Response: 200 (24 bytes) (rst-0:rem-0.0:used-1000 ratelimit) at 1737707399.1930978
2025-01-24 03:29:59,230 DEBUG:Response: 429 (0 bytes) (rst-0:rem-0.0:used-1000 ratelimit) at 1737707399.23043

raise self.STATUS_EXCEPTIONS[response.status_code](response)
prawcore.exceptions.TooManyRequests: received 429 HTTP response
```

### This code has previously worked as intended

Yes

### Operating System/Environment

Arch Linux, pipenv

### Python Version

3.13.1

### PRAW Version

7.8.1

### Links, references, and/or additional comments?

_No response_

Hints from issue comments:
@LilSpazJoekp: `ratelimit_seconds` is for undocumented (purposefully undocumented) ratelimits. It is the maximum amount of time PRAW will wait before retrying. This only covers instances where Reddit rejects a request with an error similar to "You're doing that too much" for example. This typically happens if you send too many message or leave too many comments in a short period of time.@TheCraiggers: Ah, so is the presumption that the users of the praw library know what the documented limits are for Reddit and code appropriately to not run afoul of them?@bboe: This seems like a bug if it's 429ing on `subreddit.contributor.remove(user)`@bboe: @TheCraiggers will you provide the full stacktrace so we can see the call and lines numbers for the failure please?@TheCraiggers: > [@TheCraiggers](https://github.com/TheCraiggers) will you provide the full stacktrace so we can see the call and lines numbers for the failure please?

My apologies, I should have done that to begin with.

```
Traceback (most recent call last):
  File "/home/foo/remover.py", line 322, in remove_users
    subreddit.contributor.remove(user)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/foo/.local/share/virtualenvs/remover-Kpv4gCij/lib/python3.13/site-packages/praw/models/reddit/subreddit.py", line 1581, in remove
    self.subreddit._reddit.post(url, data=data)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/foo/.local/share/virtualenvs/remover-Kpv4gCij/lib/python3.13/site-packages/praw/util/deprecate_args.py", line 46, in wrapped
    return func(**dict(zip(_old_args, args)), **kwargs)
  File "/home/foo/.local/share/virtualenvs/remover-Kpv4gCij/lib/python3.13/site-packages/praw/reddit.py", line 861, in post
    return self._objectify_request(
           ~~~~~~~~~~~~~~~~~~~~~~~^
        data=data,
        ^^^^^^^^^^
    ...<4 lines>...
        path=path,
        ^^^^^^^^^^
    )
    ^
  File "/home/foo/.local/share/virtualenvs/remover-Kpv4gCij/lib/python3.13/site-packages/praw/reddit.py", line 514, in _objectify_request
    self.request(
    ~~~~~~~~~~~~^
        data=data,
        ^^^^^^^^^^
    ...<4 lines>...
        path=path,
        ^^^^^^^^^^
    )
    ^
  File "/home/foo/.local/share/virtualenvs/remover-Kpv4gCij/lib/python3.13/site-packages/praw/util/deprecate_args.py", line 46, in wrapped
    return func(**dict(zip(_old_args, args)), **kwargs)
  File "/home/foo/.local/share/virtualenvs/remover-Kpv4gCij/lib/python3.13/site-packages/praw/reddit.py", line 963, in request
    return self._core.request(
           ~~~~~~~~~~~~~~~~~~^
        data=data,
        ^^^^^^^^^^
    ...<4 lines>...
        path=path,
        ^^^^^^^^^^
    )
    ^
  File "/home/foo/.local/share/virtualenvs/remover-Kpv4gCij/lib/python3.13/site-packages/prawcore/sessions.py", line 328, in request
    return self._request_with_retries(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^
        data=data,
        ^^^^^^^^^^
    ...<5 lines>...
        url=url,
        ^^^^^^^^
    )
    ^
  File "/home/foo/.local/share/virtualenvs/remover-Kpv4gCij/lib/python3.13/site-packages/prawcore/sessions.py", line 267, in _request_with_retries
    raise self.STATUS_EXCEPTIONS[response.status_code](response)
prawcore.exceptions.TooManyRequests: received 429 HTTP response
```@bboe: Well this is rather interesting. PRAW is written to avoid hitting `TooManyRequests` completely. Are you running multiple programs from the same account or IP by chance?@TheCraiggers: Yes, I have multiple programs, but they don't run at the same time. In fact, the other program doesn't run at all on the day this program runs, explicitly to help prevent ratelimit issues. (Also, it would probably blow up in other fun ways, as one adds users and the other removes them. Having them both run at the same time would create all kinds of problems.)

I just double-checked and yep, the other program's last timestamp was yesterday. Almost exactly 24 hours ago.@LilSpazJoekp: cc: @Watchful1 @bboe: Okay, please turn on full praw and prawcore logging as described here, and run again:

https://praw.readthedocs.io/en/stable/getting_started/logging.html

We'll need to see most of the log info around the time of failure as the retry handler in prawcore should not let you exceed the ratelimit, but something funny is happening if you aren't doing anything else funny.@TheCraiggers: This runs once per week on Fridays so I'll enable the extra logging for praw and prawcore and leave everything else as-is in hopes it retriggers. This happened two weeks ago, was fine last week, and then happened again today. So it's not 100% reproducible.

I'll report back in a week. Fingers crossed we get good data. :)@bboe: Oh okay. That's fascinating, thanks for the heads up on the frequency bit.@Watchful1: I believe this is a known bug on reddit's side.

```
2025-01-24 03:29:58,622 DEBUG:Response: 200 (24 bytes) (rst-1:rem-1.0:used-999 ratelimit) at 1737707398.6224327
2025-01-24 03:29:59,193 DEBUG:Response: 200 (24 bytes) (rst-0:rem-0.0:used-1000 ratelimit) at 1737707399.1930978
2025-01-24 03:29:59,230 DEBUG:Response: 429 (0 bytes) (rst-0:rem-0.0:used-1000 ratelimit) at 1737707399.23043
```
this is saying
> It is now timestamp 1737707398.6224327, you have 1 request remaining and the window will reset in 1 second.

> It is now timestamp 1737707399.1930978, you have 0 requests remaining and the window will reset in 0 seconds.

On the next request, PRAW sees that the window will reset at 1737707399.1930978 + 0, it sees that it's past that time and assumes the window has reset and it can request right away. But then

> It is now timestamp 1737707399.23043, you have 0 requests remaining, the window will reset in 0 seconds, and this request errored.

The issue here is that reddit is rounding down the seconds to reset in the response instead of returning a decimal or rounding up. Reddit could fix it by doing that, or prawcore could fix it by naively adding 1 second to the time till reset all the time.

It's been like this basically forever, but doesn't come up often because it's usually almost impossible to actually use up all the requests sequentially, since request time * 1000 is bigger than 600 seconds. Only some endpoints, like this one, return fast enough for it to be a problem.
@bboe: Thanks @Watchful1 for explaining that as I didn't notice the nuance. If that's the case, then yes, prawcore should sleep at least half a second when there are no more requests.@oussama-gourari: I am still getting HTTP 429 errors after upgrading to `prawcore 3.0.0`, less frequent than it used to be, but it's still there, here is the code I am using:

```Python
import praw
from praw.models.util import BoundedSet

subreddit = reddit.subreddit("test")
seen = BoundedSet(10)
skip_existing = True
while True:
    for submission in subreddit.new(limit=2):
        if submission.id in seen:
            continue
        seen.add(submission.id)
        if skip_existing:
            continue
        # Process submission...
    skip_existing = False
```@LilSpazJoekp: Can you provide some debug logs as recommended [here](https://github.com/praw-dev/prawcore/issues/198)?

Out of curiosity, why aren't you using the stream helper for getting new submissions from a subreddit as they are created ( [`subreddit.stream.submissions()`](https://praw.readthedocs.io/en/stable/code_overview/models/subreddit.html#praw.models.Subreddit.stream))? That does basically what your code is doing and would do it in a more efficient manner when it comes to API requests and rate limit consumption. It would likely help with the 429 issue.@bboe: @oussama-gourari thanks for the feedback. Perhaps I should make it sleep for a full second on the last request.

Would you mind testing out changing this line https://github.com/praw-dev/prawcore/blob/dafd854f0dfe06222fd83f955f381a600a61273f/prawcore/rate_limit.py#L92 from:

```python
                NANOSECONDS / 2, seconds_to_reset * self.NANOSECONDS
```

to

```python
                NANOSECONDS, seconds_to_reset * self.NANOSECONDS
```

If that solves it, I'll make the change.@oussama-gourari: @LilSpazJoekp here are the logs for 2 errors:

```
Fetching: GET https://oauth.reddit.com/r/test/new at 27895.7638001
Data: None
Params: {'limit': 2, 'raw_json': 1}
Response: 200 (2544 bytes) (rst-2:rem-3.0:used-997 ratelimit) at 27896.0487665
Fetching: GET https://oauth.reddit.com/r/test/new at 27896.0506917
Data: None
Params: {'limit': 2, 'raw_json': 1}
Sleeping: 0.20 seconds prior to call
Response: 200 (2541 bytes) (rst-1:rem-2.0:used-998 ratelimit) at 27896.5029816
Fetching: GET https://oauth.reddit.com/r/test/new at 27896.5051973
Data: None
Params: {'limit': 2, 'raw_json': 1}
Response: 200 (2544 bytes) (rst-1:rem-1.0:used-999 ratelimit) at 27896.749761
Fetching: GET https://oauth.reddit.com/r/test/new at 27896.7537152
Data: None
Params: {'limit': 2, 'raw_json': 1}
Sleeping: 0.40 seconds prior to call
Response: 200 (2544 bytes) (rst-0:rem-0.0:used-1000 ratelimit) at 27897.4569648
Fetching: GET https://oauth.reddit.com/r/test/new at 27897.4580548
Data: None
Params: {'limit': 2, 'raw_json': 1}
Sleeping: 0.50 seconds prior to call
Response: 429 (0 bytes) (rst-0:rem-0.0:used-1000 ratelimit) at 27898.1506069
Fetching: GET https://oauth.reddit.com/r/test/new at 27898.1511682
Data: None
Params: {'limit': 2, 'raw_json': 1}
Sleeping: 0.50 seconds prior to call
Response: 200 (2541 bytes) (rst-599:rem-999.0:used-1 ratelimit) at 27898.8959766
Fetching: GET https://oauth.reddit.com/r/test/new at 27898.8970932
Data: None
Params: {'limit': 2, 'raw_json': 1}
Response: 200 (2544 bytes) (rst-598:rem-998.0:used-2 ratelimit) at 27899.8107366
Fetching: GET https://oauth.reddit.com/r/test/new at 27899.8117462
Data: None
Params: {'limit': 2, 'raw_json': 1}
Response: 200 (2544 bytes) (rst-598:rem-997.0:used-3 ratelimit) at 27900.0970624
```

```
Fetching: GET https://oauth.reddit.com/r/test/new at 31495.9588784
Data: None
Params: {'limit': 2, 'raw_json': 1}
Sleeping: 0.20 seconds prior to call
Response: 200 (2471 bytes) (rst-1:rem-2.0:used-998 ratelimit) at 31496.4291136
Fetching: GET https://oauth.reddit.com/r/test/new at 31496.4305374
Data: None
Params: {'limit': 2, 'raw_json': 1}
Response: 200 (2471 bytes) (rst-1:rem-1.0:used-999 ratelimit) at 31496.7038728
Fetching: GET https://oauth.reddit.com/r/test/new at 31496.7061372
Data: None
Params: {'limit': 2, 'raw_json': 1}
Sleeping: 0.40 seconds prior to call
Response: 200 (2474 bytes) (rst-0:rem-0.0:used-1000 ratelimit) at 31497.3250262
Fetching: GET https://oauth.reddit.com/r/test/new at 31497.3266247
Data: None
Params: {'limit': 2, 'raw_json': 1}
Sleeping: 0.50 seconds prior to call
Response: 429 (0 bytes) (rst-0:rem-0.0:used-1000 ratelimit) at 31498.0381291
Fetching: GET https://oauth.reddit.com/r/test/new at 31498.1840852
Data: None
Params: {'limit': 2, 'raw_json': 1}
Sleeping: 0.35 seconds prior to call
Response: 200 (2471 bytes) (rst-599:rem-999.0:used-1 ratelimit) at 31498.8260363
Fetching: GET https://oauth.reddit.com/r/test/new at 31498.8279697
Data: None
Params: {'limit': 2, 'raw_json': 1}
Response: 200 (2471 bytes) (rst-599:rem-998.0:used-2 ratelimit) at 31499.0745937
Fetching: GET https://oauth.reddit.com/r/test/new at 31499.0757566
Data: None
Params: {'limit': 2, 'raw_json': 1}
Sleeping: 0.20 seconds prior to call
Response: 200 (2471 bytes) (rst-598:rem-997.0:used-3 ratelimit) at 31499.5508478
```

As for your second question, the `limit` parameter of the `ListingGenerator` is hardcoded to 100 and not exposed in the `stream_generator` function, and retrieving 100 items is slow enough that the 1000 requests won't be exhausted in 10 min, in my use case I wanted the fastest way possible to learn about new submissions in a subreddit that is not much active , so I thought of reducing the `limit` to make the requests faster and additionally consuming all those 1000 requests.

@bboe No problem, I'll let the script run overnight with that change and see if I catch anything.@Watchful1: > Response: 200 (2474 bytes) (rst-0:rem-0.0:used-1000 ratelimit) at 31497.3250262

> Response: 429 (0 bytes) (rst-0:rem-0.0:used-1000 ratelimit) at 31498.0381291

That's crazy, it's more than a full second after the last request reported 0 seconds till reset and it still failed.@bboe: > That's crazy, it's more than a full second after the last request reported 0 seconds till reset and it still failed.

Not quite a full second, `0.7131029`. If Reddit converts a float to an int, then .99s would be reported as 0, hence why we my need to sleep a full second. I was hoping they'd just call round, but I guess not.@oussama-gourari: @bboe the change solved it for me, no 429 errors at all.@bboe: Awesome. I'll make that update. Thanks @oussama-gourari.@bboe: Woah, I didn't know you could transfer issues cross project. That's cool!@bboe: Version 3.0.1 of prawcore _should_ resolve this issue:

```sh
pip install prawcore==3.0.1
```

`praw` is currently fixed to `prawcore < 3` so the above manual upgrade suggestion breaks dependency resolution. Since this change resulted in a breaking change in `prawcore` we will correspondingly have to do a major version release of `praw` and that might take a bit of time to clean up anything else we want to include in said release.@LilSpazJoekp: To add, we plan to remove all methods that are currently marked as deprecated and any methods that rely on endpoints that Reddit has deprecated. More info on that [here](https://www.reddit.com/r/help/comments/1fojw02/cleaning_up_some_lowusage_features/) and [here](https://www.reddit.com/r/redditdev/comments/1i86sn8/removing_obsolete_endpoints_from_the_data_api/).@Chambrse: fyi if you're developing and you're annoyed by persistent 429s, you can do 

`pip install praw==7.8.1 prawcore==3.0.1 --no-deps
`
I've been streaming /all and it was happening a lot, my little test project has a limited scope so I tried it and so far so good, but obviously you're in no mans land.@bboe: 👍 to @Chambrse's approach. As long as you aren't doing anything with the prawcore's rate limiter in your code, the above will work fine.

Solution patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index ebe3522..86cec6d 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -6,6 +6,10 @@ prawcore follows `semantic versioning <https://semver.org/>`_.
 Unreleased
 ----------
 
+**Fixed**
+
+- Increase half-second delay introduced in ``3.0.0`` to a full second delay.
+
 3.0.0 (2025/01/25)
 ------------------
 
diff --git a/prawcore/rate_limit.py b/prawcore/rate_limit.py
index 59f924c..20d9118 100644
--- a/prawcore/rate_limit.py
+++ b/prawcore/rate_limit.py
@@ -89,7 +89,7 @@ class RateLimiter:
 
         if self.remaining <= 0:
             self.next_request_timestamp_ns = now_ns + max(
-                NANOSECONDS / 2, seconds_to_reset * NANOSECONDS
+                NANOSECONDS, seconds_to_reset * NANOSECONDS
             )
             return
 


Test patch:
diff --git a/tests/unit/test_rate_limit.py b/tests/unit/test_rate_limit.py
index 8a0ac09..4bf93e1 100644
--- a/tests/unit/test_rate_limit.py
+++ b/tests/unit/test_rate_limit.py
@@ -121,7 +121,7 @@ class TestRateLimiter(UnitTest):
         rate_limiter.update(self._headers(0, 100, 0))
         assert rate_limiter.remaining == 0
         assert rate_limiter.used == 100
-        assert rate_limiter.next_request_timestamp_ns == 37.5 * NANOSECONDS
+        assert rate_limiter.next_request_timestamp_ns == 38 * NANOSECONDS
 
     def test_update__no_change_without_headers(self, rate_limiter):
         prev = copy(rate_limiter)