Repository: VirtusLab/git-machete
Pull request #754 (2022-12-28 14:33:18)

Implement pagination handling for GitHub API

codecov-commenter: # [Codecov](https://codecov.io/gh/VirtusLab/git-machete/pull/754?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VirtusLab) Report
> Merging [#754](https://codecov.io/gh/VirtusLab/git-machete/pull/754?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VirtusLab) (f196594) into [feature/137-github-enterprise-domains](https://codecov.io/gh/VirtusLab/git-machete/commit/45c0ca940a97a1d869d398f47923faf6d84eba05?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VirtusLab) (45c0ca9) will **decrease** coverage by `0.14%`.
> The diff coverage is `41.66%`.

```diff
@@                            Coverage Diff                            @@
##           feature/137-github-enterprise-domains     #754      +/-   ##
=========================================================================
- Coverage                                  70.88%   70.74%   -0.15%     
=========================================================================
  Files                                         10       10              
  Lines                                       3009     3018       +9     
  Branches                                     702      704       +2     
=========================================================================
+ Hits                                        2133     2135       +2     
- Misses                                       612      618       +6     
- Partials                                     264      265       +1     
```


| [Impacted Files](https://codecov.io/gh/VirtusLab/git-machete/pull/754?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VirtusLab) | Coverage Δ | |
|---|---|---|
| [git\_machete/github.py](https://codecov.io/gh/VirtusLab/git-machete/pull/754/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VirtusLab#diff-Z2l0X21hY2hldGUvZ2l0aHViLnB5) | `62.85% <22.22%> (-2.00%)` | :arrow_down: |
| [git\_machete/client.py](https://codecov.io/gh/VirtusLab/git-machete/pull/754/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VirtusLab#diff-Z2l0X21hY2hldGUvY2xpZW50LnB5) | `66.64% <100.00%> (+0.02%)` | :arrow_up: |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VirtusLab). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VirtusLab)


Patch:
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index b7edbb8..1a8a9c5 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -5,6 +5,7 @@
 - added: `push=no` and `rebase=no` branch qualifiers that control push and rebase behaviour of `git machete traverse`
 - added: `machete.github.domain` config key to support GitHub Enterprise domains
 - added: support for per-domain entries in `~/.github-token` file
+- fixed: fetching GitHub PRs when there is more than 30 of them in the given repository
 
 ## New in git-machete 3.13.2
 
diff --git a/git_machete/client.py b/git_machete/client.py
index e8e0241..d36d033 100644
--- a/git_machete/client.py
+++ b/git_machete/client.py
@@ -1461,10 +1461,12 @@ class MacheteClient:
     def sync_annotations_to_github_prs(self) -> None:
         domain = self.__derive_github_domain()
         _, org, repo = self.__derive_remote_and_github_org_and_repo(domain)
+        print('Checking for open GitHub PRs... ', end='', flush=True)
         current_user: Optional[str] = git_machete.github.derive_current_user_login(domain)
         debug('Current GitHub user is ' + (current_user or '<none>'))
-        all_prs: List[GitHubPullRequest] = derive_pull_requests(domain, org, repo)
-        self.__sync_annotations_to_definition_file(all_prs, current_user)
+        all_open_prs: List[GitHubPullRequest] = derive_pull_requests(domain, org, repo)
+        print(fmt('<green><b>OK</b></green>'))
+        self.__sync_annotations_to_definition_file(all_open_prs, current_user)
 
     def __sync_annotations_to_definition_file(self,
                                               prs: List[GitHubPullRequest],
@@ -1967,12 +1969,13 @@ class MacheteClient:
                             verbose: bool = False,
                             fail_on_missing_current_user_for_my_opened_prs: bool = True
                             ) -> None:
-        print(bold('Checking for open GitHub PRs...'))
         org: str
         repo: str
         remote: str
         domain = self.__derive_github_domain()
         remote, org, repo = self.__derive_remote_and_github_org_and_repo(domain)
+        print('Checking for open GitHub PRs... ', end='', flush=True)
+
         current_user: Optional[str] = git_machete.github.derive_current_user_login(domain)
         if not current_user and my_opened_prs:
             msg = ("Could not determine current user name, please check that the GitHub API token provided by one of the: "
@@ -1983,6 +1986,8 @@ class MacheteClient:
             else:
                 raise MacheteException(msg)
         all_open_prs: List[GitHubPullRequest] = derive_pull_requests(domain, org, repo)
+        print(fmt('<green><b>OK</b></green>'))
+
         applicable_prs: List[GitHubPullRequest] = self.__get_applicable_pull_requests(pr_nos,
                                                                                       all_opened_prs_from_github=all_open_prs,
                                                                                       domain=domain,
diff --git a/git_machete/github.py b/git_machete/github.py
index fbf18ee..7063024 100644
--- a/git_machete/github.py
+++ b/git_machete/github.py
@@ -203,6 +203,15 @@ def __fire_github_api_request(domain: str, method: str, path: str,
     try:
         with urllib.request.urlopen(http_request) as response:
             parsed_response_body: Any = json.loads(response.read().decode())
+            # https://docs.github.com/en/rest/guides/using-pagination-in-the-rest-api?apiVersion=2022-11-28#using-link-headers
+            link_header: str = response.info()["link"]
+            if link_header:
+                host_regex = re.escape(host)
+                match = re.search(f'<{host_regex}(/[^>]+)>; rel="next"', link_header)
+                if match:
+                    next_page_path = match.group(1)
+                    debug(f'there is more data to retrieve under {next_page_path}')
+                    return parsed_response_body + __fire_github_api_request(domain, method, next_page_path, token, request_body)
             return parsed_response_body
     except urllib.error.HTTPError as err:
         if err.code == http.HTTPStatus.UNPROCESSABLE_ENTITY:
@@ -315,7 +324,8 @@ def derive_pull_request_by_head(domain: str, org: str, repo: str, head: LocalBra
 
 def derive_pull_requests(domain: str, org: str, repo: str) -> List[GitHubPullRequest]:
     token: Optional[str] = __get_github_token(domain)
-    prs = __fire_github_api_request(domain, 'GET', f'/repos/{org}/{repo}/pulls', token)
+    # As of Dec 2022, GitHub API never returns more than 100 PRs, even if per_page>100.
+    prs = __fire_github_api_request(domain, 'GET', f'/repos/{org}/{repo}/pulls?per_page=100', token)
     return list(map(__parse_pr_json, prs))
 
 


Test patch:
diff --git a/tests/mockers.py b/tests/mockers.py
index 808d43f..844670b 100644
--- a/tests/mockers.py
+++ b/tests/mockers.py
@@ -161,6 +161,9 @@ class MockGitHubAPIResponse:
     def read(self) -> bytes:
         return json.dumps(self.response_data).encode()
 
+    def info(self) -> Dict[str, None]:
+        return {"link": None}
+
 
 class MockGitHubAPIRequest:
     def __init__(self, github_api_state: MockGitHubAPIState) -> None:
diff --git a/tests/test_github.py b/tests/test_github.py
index 955909d..0cabe0d 100644
--- a/tests/test_github.py
+++ b/tests/test_github.py
@@ -1123,7 +1123,7 @@ class TestGithub:
                 'Verify that expected error message has appeared when one of the given pull requests to checkout does not exists.'
 
         # check against user with no open pull requests
-        expected_msg = ("Checking for open GitHub PRs...\n"
+        expected_msg = ("Checking for open GitHub PRs... OK\n"
                         f"Warn: User tester has no open pull request in repository {org}/{repo}\n")
         assert_command(['github', 'checkout-prs', '--by', 'tester'], expected_msg, strip_indentation=False)
 
@@ -1158,14 +1158,14 @@ class TestGithub:
          )
         os.chdir(self.repo_sandbox.local_path)
 
-        expected_msg = ("Checking for open GitHub PRs...\n"
+        expected_msg = ("Checking for open GitHub PRs... OK\n"
                         "Warn: Pull request #5 is already closed.\n"
                         "Pull request #5 checked out at local branch bugfix/remove-n-option\n")
 
         assert_command(['github', 'checkout-prs', '5'], expected_msg, strip_indentation=False)
 
         # Check against multiple PRs
-        expected_msg = 'Checking for open GitHub PRs...\n'
+        expected_msg = 'Checking for open GitHub PRs... OK\n'
 
         assert_command(['github', 'checkout-prs', '3', '12'], expected_msg, strip_indentation=False)
 
@@ -1263,7 +1263,7 @@ class TestGithub:
         )
         os.chdir(self.repo_sandbox.local_path)
         rewrite_definition_file("master")
-        expected_msg = ("Checking for open GitHub PRs...\n"
+        expected_msg = ("Checking for open GitHub PRs... OK\n"
                         "Pull request #2 checked out at local branch comments/add_docstrings\n")
         assert_command(
             ['github', 'checkout-prs', '2'],
@@ -1286,7 +1286,7 @@ class TestGithub:
 
         # Check against closed pull request
         self.repo_sandbox.execute('git branch -D sphinx_export')
-        expected_msg = ("Checking for open GitHub PRs...\n"
+        expected_msg = ("Checking for open GitHub PRs... OK\n"
                         "Warn: Pull request #23 is already closed.\n"
                         "Pull request #23 checked out at local branch sphinx_export\n")
 
@@ -1349,7 +1349,7 @@ class TestGithub:
             .push()
         )
         launch_command('discover')
-        expected_msg = ("Checking for open GitHub PRs...\n"
+        expected_msg = ("Checking for open GitHub PRs... OK\n"
                         "Warn: Pull request #2 comes from fork and its repository is already deleted. "
                         "No remote tracking data will be set up for feature/allow_checkout branch.\n"
                         "Warn: Pull request #2 is already closed.\n"