Repository: readthedocs/readthedocs.org

Issue:
Github sync pagination issue
After syncing the repos at https://readthedocs.org/dashboard/import/github/ , the ones I'm interested in are now showing up in the list.

The issue is that I have more than 100 repos and the code is not using github's pagination over here:
- https://github.com/rtfd/readthedocs.org/blob/5c6ff1bd36b1a7e80f6ea7714e9c07c7b2f11769/readthedocs/projects/views/private.py#L460
- https://github.com/rtfd/readthedocs.org/blob/5c6ff1bd36b1a7e80f6ea7714e9c07c7b2f11769/readthedocs/projects/views/private.py#L471

Here is some code I started to implement to replace the session.get(url) call with github_paginate(session, url). I didn't really know where to put it and it's untested.

``` python
import re
NEXT_MATCH = re.compile('<([^>]+)>;\s*rel="next"')
def github_paginate(client, url):
    """
    Scans trough all github paginates results and returns the concatenated
    result.

    See https://developer.github.com/v3/#pagination
    """
    result = []
    while url:
        r = session.get(url)

        result.extend(r.json())

        url = None
        links = r.headers.get('Link')
        if links:
            md = NEXT_MATCH.search(links)
            if md:
                url = md[0]
    return result
```

PS: thanks a lot for readthedocs, it's awesome



Discussion:
Great, thanks for the snippet. I believe requests handles the link headers natively, so it should be pretty simple to do. I was wondering how long it would be until someone had more than 100 repos :)

Yea, here is the links stuff: http://docs.python-requests.org/en/latest/api/#requests.Response.links

Sweet, requests is awesome.

Where would you put the helper function in the current code-base ?


Resolving patch:
diff --git a/readthedocs/projects/utils.py b/readthedocs/projects/utils.py
--- a/readthedocs/projects/utils.py
+++ b/readthedocs/projects/utils.py
@@ -229,3 +229,21 @@ def make_api_project(project_data):
     project = Project(**project_data)
     project.save = _new_save
     return project
+
+
+def github_paginate(client, url):
+    """
+    Scans trough all github paginates results and returns the concatenated
+    list of results.
+
+    :param client: requests client instance
+    :param url: start url to get the data from.
+
+    See https://developer.github.com/v3/#pagination
+    """
+    result = []
+    while url:
+        r = session.get(url)
+        result.extend(r.json())
+        url = r.links.get('next')
+    return result
diff --git a/readthedocs/projects/views/private.py b/readthedocs/projects/views/private.py
--- a/readthedocs/projects/views/private.py
+++ b/readthedocs/projects/views/private.py
@@ -28,6 +28,7 @@
                             UserForm, EmailHookForm, TranslationForm,
                             AdvancedProjectForm, RedirectForm, WebHookForm)
 from projects.models import Project, EmailHook, WebHook
+from projects.utils import github_paginate
 from projects import constants
 from redirects.models import Redirect
 
@@ -457,7 +458,7 @@ def project_import_github(request, sync=False):
                 }
             )
             # Get user repos
-            owner_resp = session.get('https://api.github.com/user/repos?per_page=100')
+            owner_resp = github_paginate(session, 'https://api.github.com/user/repos?per_page=100')
             for repo in owner_resp.json():
                 log.info('Trying %s' % repo['full_name'])
                 oauth_utils.make_github_project(user=request.user, org=None, privacy=repo_type, repo_json=repo)
@@ -465,10 +466,10 @@ def project_import_github(request, sync=False):
             # Get org repos
             resp = session.get('https://api.github.com/user/orgs')
             for org_json in resp.json():
-                org_resp = session.get('https://api.github.com/orgs/%s' % org_json['login'])
+                org_resp = github_paginate(session, 'https://api.github.com/orgs/%s' % org_json['login'])
                 org_obj = oauth_utils.make_github_organization(user=request.user, org_json=org_resp.json())
                 # Add repos
-                org_repos_resp = session.get('https://api.github.com/orgs/%s/repos?type=%s' % (org_json['login'], repo_type))
+                org_repos_resp = github_paginate(session, 'https://api.github.com/orgs/%s/repos?type=%s' % (org_json['login'], repo_type))
                 for repo in org_repos_resp.json():
                     oauth_utils.make_github_project(user=request.user, org=org_obj, privacy=repo_type, repo_json=repo)
 


Test patch: