Repository: python-gitlab/python-gitlab

Issue:
int() argument must be a string, a bytes-like object or a real number, not 'NoneType' while accessing pagination related attributes
## Description of the problem, including code/CLI snippet
Accessing current_page , per_page , total , total_pages attributes , throwing the following error.
**int() argument must be a string, a bytes-like object or a real number, not 'NoneType'**

  
```
  gl = gitlab.Gitlab('https://gitlab.com/', private_token=developer_access_token,
                       pagination="keyset", order_by="id",
                       per_page=100)
    iterator_projects = gl.projects.list(as_list=False, owned=True)
```


## Expected Behavior

It should return the number of pages expected

## Actual Behavior

throwing exceptions.

## Specifications

  - python-gitlab version:2.10.1
  - API version you are using (v3/v4): v4 
  - Gitlab server version (or gitlab.com):gitlab.com



Discussion:
I can't reproduce this but I only have about 90 projects.

You might try doing:

```
gl.enable_debug()
```

As it might help diagnose the issue.
Also if you can paste the full Exception stack trace it would be helpful to see what part of the code is triggering the issue.
I think this is a duplicate of https://github.com/python-gitlab/python-gitlab/issues/1506 and maybe https://github.com/python-gitlab/python-gitlab/issues/1114, that gitlab.com keyset pagination thing again :(
Also , related to pagination i do have specific question.

Can we implement pagination in such way that , calling the required page number ( say for example 3 will return me the 3rd page ) , 

current implementation is that i need to be in the same method , iterate through page 1 , page 2 and page 3. as long as i'm in the same function all is good.

But if i leave this function , and come back to this pagination fuction with next_page number , Is this something possible ? Atleast i could not find it in the code examples.

if you guys , have some ideas , please post it here. Appreciate your support.

I'm more than willing to do code changes as well , if you point me the right souce location. Thanks much.
+1 on this issue, debug log below. Can't seem to fetch the total pages. Seems sometimes Gitlab will not return the header `X-total-pages`. 

python-gitlab version:2.10.1
API version you are using (v3/v4): v4
Gitlab server version (or gitlab.com):13.12.12
```
send: b'GET /api/v4/projects/112 HTTP/1.1\r\nHost: <gitlab.private.url>\r\nUser-Agent: python-gitlab/2.10.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nPRIVATE-TOKEN: private_token\r\nContent-type: application/json\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: max-age=0, private, must-revalidate
header: Content-Encoding: gzip
header: Content-Type: application/json
header: Date: Wed, 22 Dec 2021 17:40:02 GMT
header: Etag: W/"aeeab9b1b4fe9256fb52a73786e6b849"
header: Referrer-Policy: strict-origin-when-cross-origin
header: Server: nginx
header: Strict-Transport-Security: max-age=31536000
header: Vary: Accept-Encoding
header: Vary: Origin
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: X-Gitlab-Feature-Category: projects
header: X-Request-Id: 01FQHKN125FFCY3RACSDX2EKPS
header: X-Runtime: 0.065188
header: Content-Length: 1398
header: Connection: keep-alive
DEBUG:urllib3.connectionpool:https://<gitlab.private.url>:443 "GET /api/v4/projects/112 HTTP/1.1" 200 1398
send: b'GET /api/v4/projects/112/repository/commits?since=2021-01-01+00%3A00%3A00&ref_name=master&per_page=100 HTTP/1.1\r\nHost: <gitlab.private.url>\r\nUser-Agent: python-gitlab/2.10.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nPRIVATE-TOKEN: private_token\r\nContent-type: application/json\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: max-age=0, private, must-revalidate
header: Content-Encoding: gzip
header: Content-Type: application/json
header: Date: Wed, 22 Dec 2021 17:40:03 GMT
header: Etag: W/"84a9393b2e899b627491c499a48ca92b"
header: Link: <https://<gitlab.private.url>/api/v4/projects/112/repository/commits?id=112&order=default&page=2&per_page=100&ref_name=master&since=2021-01-01T00%3A00%3A00%2B00%3A00>; rel="next", <https://<gitlab.private.url>/api/v4/projects/112/repository/commits?id=112&order=default&page=1&per_page=100&ref_name=master&since=2021-01-01T00%3A00%3A00%2B00%3A00>; rel="first", <https://<gitlab.private.url>/api/v4/projects/112/repository/commits?id=112&order=default&page=2&per_page=100&ref_name=master&since=2021-01-01T00%3A00%3A00%2B00%3A00>; rel="last"
header: Referrer-Policy: strict-origin-when-cross-origin
header: Server: nginx
header: Strict-Transport-Security: max-age=31536000
header: Vary: Accept-Encoding
header: Vary: Origin
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: X-Gitlab-Feature-Category: source_code_management
header: X-Next-Page: 2
header: X-Page: 1
header: X-Per-Page: 100
header: X-Prev-Page: 
header: X-Request-Id: 01FQHKN17NPZMT9XGVAHH1QAXX
header: X-Runtime: 0.065319
header: Content-Length: 12033
header: Connection: keep-alive
DEBUG:urllib3.connectionpool:https://<gitlab.private.url>:443 "GET /api/v4/projects/112/repository/commits?since=2021-01-01+00%3A00%3A00&ref_name=master&per_page=100 HTTP/1.1" 200 12033
Traceback (most recent call last):
  File "python_gitlab.py", line 83, in <module>
    get_repo_merges(SESSION, r_id,r_name)
  File "python_gitlab.py", line 32, in get_repo_merges
    print(commits.total)
  File "python/lib/python3.9/site-packages/gitlab/base.py", line 259, in total
    return self._list.total
  File "python/lib/python3.9/site-packages/gitlab/client.py", line 963, in total
    return int(self._total)
```
    
seems to be the issue here:
https://docs.gitlab.com/ee/user/gitlab_com/index.html#pagination-response-headers
> +1 on this issue, debug log below. Can't seem to fetch the total pages. Seems sometimes Gitlab will not return the header `X-total-pages`.

Thank you very much for the debug log and the link. That made it easy to figure out the issue. I have proposed a PR to resolve the issue.

Resolving patch:
diff --git a/gitlab/base.py b/gitlab/base.py
--- a/gitlab/base.py
+++ b/gitlab/base.py
@@ -288,12 +288,12 @@ def per_page(self) -> int:
         return self._list.per_page
 
     @property
-    def total_pages(self) -> int:
+    def total_pages(self) -> Optional[int]:
         """The total number of pages."""
         return self._list.total_pages
 
     @property
-    def total(self) -> int:
+    def total(self) -> Optional[int]:
         """The total number of items."""
         return self._list.total
 
diff --git a/gitlab/client.py b/gitlab/client.py
--- a/gitlab/client.py
+++ b/gitlab/client.py
@@ -917,14 +917,12 @@ def _query(
             self._next_url = next_url
         except KeyError:
             self._next_url = None
-        self._current_page: Optional[Union[str, int]] = result.headers.get("X-Page")
-        self._prev_page: Optional[Union[str, int]] = result.headers.get("X-Prev-Page")
-        self._next_page: Optional[Union[str, int]] = result.headers.get("X-Next-Page")
-        self._per_page: Optional[Union[str, int]] = result.headers.get("X-Per-Page")
-        self._total_pages: Optional[Union[str, int]] = result.headers.get(
-            "X-Total-Pages"
-        )
-        self._total: Optional[Union[str, int]] = result.headers.get("X-Total")
+        self._current_page: Optional[str] = result.headers.get("X-Page")
+        self._prev_page: Optional[str] = result.headers.get("X-Prev-Page")
+        self._next_page: Optional[str] = result.headers.get("X-Next-Page")
+        self._per_page: Optional[str] = result.headers.get("X-Per-Page")
+        self._total_pages: Optional[str] = result.headers.get("X-Total-Pages")
+        self._total: Optional[str] = result.headers.get("X-Total")
 
         try:
             self._data: List[Dict[str, Any]] = result.json()
@@ -965,19 +963,22 @@ def per_page(self) -> int:
             assert self._per_page is not None
         return int(self._per_page)
 
+    # NOTE(jlvillal): When a query returns more than 10,000 items, GitLab doesn't return
+    # the headers 'x-total-pages' and 'x-total'. In those cases we return None.
+    # https://docs.gitlab.com/ee/user/gitlab_com/index.html#pagination-response-headers
     @property
-    def total_pages(self) -> int:
+    def total_pages(self) -> Optional[int]:
         """The total number of pages."""
-        if TYPE_CHECKING:
-            assert self._total_pages is not None
-        return int(self._total_pages)
+        if self._total_pages is not None:
+            return int(self._total_pages)
+        return None
 
     @property
-    def total(self) -> int:
+    def total(self) -> Optional[int]:
         """The total number of items."""
-        if TYPE_CHECKING:
-            assert self._total is not None
-        return int(self._total)
+        if self._total is not None:
+            return int(self._total)
+        return None
 
     def __iter__(self) -> "GitlabList":
         return self


Test patch:
diff --git a/tests/unit/test_gitlab.py b/tests/unit/test_gitlab.py
--- a/tests/unit/test_gitlab.py
+++ b/tests/unit/test_gitlab.py
@@ -16,9 +16,9 @@
 # You should have received a copy of the GNU Lesser General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+import copy
 import pickle
 import warnings
-from copy import deepcopy
 
 import pytest
 import responses
@@ -109,7 +109,7 @@ def _strip_pagination_headers(response):
     """
     https://docs.gitlab.com/ee/user/gitlab_com/index.html#pagination-response-headers
     """
-    stripped = deepcopy(response)
+    stripped = copy.deepcopy(response)
 
     del stripped["headers"]["X-Total-Pages"]
     del stripped["headers"]["X-Total"]
@@ -117,7 +117,6 @@ def _strip_pagination_headers(response):
     return stripped
 
 
-@pytest.mark.xfail(reason="See #1686")
 @responses.activate
 def test_gitlab_build_list_missing_headers(gl, resp_page_1, resp_page_2):
     stripped_page_1 = _strip_pagination_headers(resp_page_1)