Repository: python-gitlab/python-gitlab

Issue:
Rate limiting & recovery
Would it be possible to add rate limiting into `GitLab._raw_get` (https://github.com/gpocentek/python-gitlab/blob/5f444e4ddf1087a8c4081f9b8d8b3d87d36a0985/gitlab/__init__.py), and tie it to a config option in `~/.python-gitlab.cfg`?

I keep getting `gitlab.exceptions.GitlabListError` from `HTTP 429` caused by gitlab rate limiting.  I expect that I can catch this exception, delay the code, then retry the failed command.  There is no nice way though to know the number of underlying API requests made by python-gitlab, so I cannot precisely figure out the delay, nor easily pre-delay to avoid the `HTTP 429`.  I'm using a few `....list(all=True)`, so even with arbitrary sleeps here and there, I still hit the limit a lot.



Discussion:
@jflemer-ndp did you try to pass a per_page argument to get more than 20 items per API request? I guess you have a lot of data to handle if you reach the limits, but maybe getting larger batch of data at a time would help. Something like `....list(all=True, per_page=100)`

If possible I'd like to avoid implementing an API limitation workaround.

Let me know if this works for you.

Note: my company gitlab seems to limit the number of items per page to 100. Not sure if this is configurable on the server side.

With `per_page=100` I get a little further, but still eventually hit the limit.  I am looping over all the notes of all the tickets.  Since there is no way in the GitLab API to filter what notes I want, this is going to be a lot of objects.

```
proj = gl.projects.get(urllib.quote(project))
for tix in proj.issues.list(all=True, per_page=100):
  for n in tix.notes.list(all=True, per_page=100):
    pass
```

It was worth a try... I'll see what I can do but this might take me some time (bit swamped at the moment).

I happened to find a kinda cool idea about using a decorator for rate limiting here http://blog.gregburek.com/2011/12/05/Rate-limiting-with-decorators/

But that probably wouldn't fit with making it configurable.

If I get fed up with my excessive and arbitrary `sleep`s, I'll try to submit a patch/merge request on this.

@jflemer-ndp Any progress on your implementation of rate limits, yet?

Maybe this library could be used? https://github.com/tomasbasham/ratelimit

Resolving patch:
diff --git a/gitlab/__init__.py b/gitlab/__init__.py
--- a/gitlab/__init__.py
+++ b/gitlab/__init__.py
@@ -23,6 +23,7 @@
 import itertools
 import json
 import re
+import time
 import warnings
 
 import requests
@@ -698,24 +699,35 @@ def copy_dict(dest, src):
         prepped.url = sanitized_url(prepped.url)
         settings = self.session.merge_environment_settings(
             prepped.url, {}, streamed, verify, None)
-        result = self.session.send(prepped, timeout=timeout, **settings)
 
-        if 200 <= result.status_code < 300:
-            return result
+        # obey the rate limit by default
+        obey_rate_limit = kwargs.get("obey_rate_limit", True)
 
-        try:
-            error_message = result.json()['message']
-        except (KeyError, ValueError, TypeError):
-            error_message = result.content
-
-        if result.status_code == 401:
-            raise GitlabAuthenticationError(response_code=result.status_code,
-                                            error_message=error_message,
-                                            response_body=result.content)
-
-        raise GitlabHttpError(response_code=result.status_code,
-                              error_message=error_message,
-                              response_body=result.content)
+        while True:
+            result = self.session.send(prepped, timeout=timeout, **settings)
+
+            if 200 <= result.status_code < 300:
+                return result
+
+            if 429 == result.status_code and obey_rate_limit:
+                wait_time = int(result.headers["Retry-After"])
+                time.sleep(wait_time)
+                continue
+
+            try:
+                error_message = result.json()['message']
+            except (KeyError, ValueError, TypeError):
+                error_message = result.content
+
+            if result.status_code == 401:
+                raise GitlabAuthenticationError(
+                    response_code=result.status_code,
+                    error_message=error_message,
+                    response_body=result.content)
+
+            raise GitlabHttpError(response_code=result.status_code,
+                                  error_message=error_message,
+                                  response_body=result.content)
 
     def http_get(self, path, query_data={}, streamed=False, **kwargs):
         """Make a GET request to the Gitlab server.


Test patch:
diff --git a/tools/python_test_v4.py b/tools/python_test_v4.py
--- a/tools/python_test_v4.py
+++ b/tools/python_test_v4.py
@@ -646,3 +646,28 @@
 
 # events
 gl.events.list()
+
+# rate limit
+settings = gl.settings.get()
+settings.throttle_authenticated_api_enabled = True
+settings.throttle_authenticated_api_requests_per_period = 1
+settings.throttle_authenticated_api_period_in_seconds = 3
+settings.save()
+projects = list()
+for i in range(0, 20):
+    projects.append(gl.projects.create(
+        {'name': str(i) + "ok"}))
+
+error_message = None
+for i in range(20, 40):
+    try:
+        projects.append(
+            gl.projects.create(
+                {'name': str(i) + 'shouldfail'}, obey_rate_limit=False))
+    except gitlab.GitlabCreateError as e:
+        error_message = e.error_message
+        break
+assert 'Retry later' in error_message
+[current_project.delete() for current_project in projects]
+settings.throttle_authenticated_api_enabled = False
+settings.save()