Help for "HTTP Error 403: rate limit exceeded"

Verner: <p>hi, everyone.<br>
I´m try to use the ResNet in pytorch hub to my project in google colab, but the following error appeared:</p>
<pre><code class="lang-auto">/usr/lib/python3.7/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
    647 class HTTPDefaultErrorHandler(BaseHandler):
    648     def http_error_default(self, req, fp, code, msg, hdrs):
--&gt; 649         raise HTTPError(req.full_url, code, msg, hdrs, fp)
    650 
    651 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 403: rate limit exceeded
</code></pre>
<p>I am new in this area and I thank you in advance for all the help.</p>

SimonTommerup: <p>It is probably because the site you are trying to reach has a request limit for security reasons. For example, if you use the Reddit API, you are only allowed to do one request every two seconds if I remember correctly. You could maybe investigate the rules at the Google Colab API and see if it mentions this.</p>
<p>You could try to make an exception handling on HTTPError and make your program sleep for a number of seconds before requesting again.</p>
<p>Maybe these links are helpful:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://cloud.google.com/compute/docs/api-rate-limits">
  <header class="source">
      <img src="https://www.gstatic.com/devrel-devsite/prod/v5f61782021051fb502364887a46a1c5ce2cd6f3d29a3549e907afe67612e9bba/cloud/images/favicons/onecloud/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://cloud.google.com/compute/docs/api-rate-limits" target="_blank" rel="noopener nofollow ugc">Google Cloud</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/362;"><img src="https://cloud.google.com/_static/cloud/images/social-icon-google-cloud-1200-630.png" class="thumbnail" width="690" height="362"></div>

<h3><a href="https://cloud.google.com/compute/docs/api-rate-limits" target="_blank" rel="noopener nofollow ugc">API rate limits  |  Compute Engine Documentation  | ...</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox wikipedia" data-onebox-src="https://en.wikipedia.org/wiki/HTTP_403">
  <header class="source">

      <a href="https://en.wikipedia.org/wiki/HTTP_403" target="_blank" rel="noopener nofollow ugc">en.wikipedia.org</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:220/155;"><img src="//upload.wikimedia.org/wikipedia/commons/thumb/1/14/English_Wikipedia_403.png/220px-English_Wikipedia_403.png" class="thumbnail" width="220" height="155"></div>

<h3><a href="https://en.wikipedia.org/wiki/HTTP_403" target="_blank" rel="noopener nofollow ugc">HTTP 403</a></h3>

<p>The HTTP 403 is an HTTP status code meaning access to the requested resource is forbidden. The server understood the request, but will not fulfill it.
 HTTP 403 provides a distinct error case from HTTP 401; while HTTP 401 is returned when the client has not authenticated, and implies that a successful response may be returned following valid authentication, HTTP 403 is returned when the client is not permitted access to the resource despite providing authentication such as insufficient permission...</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Verner: <p>Hi Simon,<br>
Thanks for answering. I don’t understand how the reboot works? this message has been appearing for a few days.<br>
How do I reset the time? <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>Thanks a lot.</p>

SimonTommerup: <p>I think in order for people to help you better you have to more clearly explain what exactly you’re trying to do.</p>
<p>What I am hinting at is that if you’re programatically trying to request an URL, you may want to do something like</p>
<pre><code class="lang-auto">import time

try:
 # do something
except(HTTPError):
 time.sleep(60)
 # do it again

</code></pre>

Verner: <p>Hi again.<br>
Yes, It’s just that I’m new to this forum. I’m sorry.<br>
I’m learning about CNN and I created a ResNet by calling the Pytorch hub:</p>
<p><code>net = torch.hub.load('pytorch/vision:v0.9.0','resnet18',pretrained=False)</code></p>
<p>This call is made by the training code. “Net” is the model I’m using, which in this case will be an untrained ResNet.<br>
That’s when the error I mentioned appears.<br>
Sorry if it’s a rookie question, but I really am. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
Thanks a lot.</p>

ptrblck: <p>I can reproduce the issue so thanks for posting it here.<br>
Would you mind updating <a href="https://github.com/pytorch/vision/issues/4156">this issue</a>, too?</p>

Verner: <p>Hi everyone,<br>
Thanks ptrblck for answer. Yes, i´ll post there too.<br>
The paleative I’m using to solve this problem is copying the ResNet code from github and pasting it into the colab code. And there I run the code.<br>
I followed the recommendation of user D_L in this <a href="https://discuss.pytorch.org/t/torch-hub-load-gives-httperror-rate-limit-exceeded/124769/5">post</a>.<br>
The same error happens for other networks used by pytorch hub. I am currently using shufflenet which had the same problem.</p>
<p>Thanks a lot.</p>

Moritz_Sontheimer: <p>I ran into the same issue today when using a conda environment with pytorch 1.9.0 installed.</p>
<p>When my python script hits the following line:<br>
model = torch.hub.load(“pytorch/vision:v0.6.0”, “inception_v3”, pretrained=True)<br>
i run into this error   File “web.py”, line 26, in classify_product<br>
product_h1 = Product_h1().getmodel()<br>
File “/home/frontier/01_Programs/00_code/fabric_tag/models.py”, line 97, in <strong>init</strong><br>
model = torch.hub.load(“pytorch/vision:v0.6.0”, “inception_v3”, pretrained=True)<br>
File “/home/frontier/anaconda3/lib/python3.8/site-packages/torch/hub.py”, line 362, in load<br>
repo_or_dir = _get_cache_or_reload(repo_or_dir, force_reload, verbose)<br>
File “/home/frontier/anaconda3/lib/python3.8/site-packages/torch/hub.py”, line 162, in _get_cache_or_reload<br>
_validate_not_a_forked_repo(repo_owner, repo_name, branch)<br>
File “/home/frontier/anaconda3/lib/python3.8/site-packages/torch/hub.py”, line 124, in _validate_not_a_forked_repo<br>
with urlopen(url) as r:<br>
File “/home/frontier/anaconda3/lib/python3.8/urllib/request.py”, line 222, in urlopen<br>
return opener.open(url, data, timeout)<br>
File “/home/frontier/anaconda3/lib/python3.8/urllib/request.py”, line 531, in open<br>
response = meth(req, response)<br>
File “/home/frontier/anaconda3/lib/python3.8/urllib/request.py”, line 640, in http_response<br>
response = self.parent.error(<br>
File “/home/frontier/anaconda3/lib/python3.8/urllib/request.py”, line 569, in error<br>
return self._call_chain(*args)<br>
File “/home/frontier/anaconda3/lib/python3.8/urllib/request.py”, line 502, in _call_chain<br>
result = func(*args)<br>
File “/home/frontier/anaconda3/lib/python3.8/urllib/request.py”, line 649, in http_error_default<br>
raise HTTPError(req.full_url, code, msg, hdrs, fp)<br>
urllib.error.HTTPError: HTTP Error 403: rate limit exceeded</p>
<p>Is there any solution workaround? I saw some people installed another pytorch version…</p>

ptrblck: <p>In the linked issue <a href="https://github.com/pytorch/pytorch/issues/61755#issuecomment-885801511">this workaround</a> was posted, so you could try it out and check if it’s working.</p>

Moritz_Sontheimer: <p>Thank you! This worked</p>