Repository: python-gitlab/python-gitlab

Issue:
Request URL is getting too long because of query_parameters
## Description of the problem, including code/CLI snippet

I was using the following solution to iterate over the jobs:

> @duncz there is the solution I used in my project:
> 
> ```py
> scopes = ['pending', 'running', 'failed', 'success']
> jobs = git_project.jobs.list(query_parameters={'scope[]': scopes})
> ```
> 
> Should work.

_Originally posted by @romansvozil in https://github.com/python-gitlab/python-gitlab/issues/806#issuecomment-607449709_

When using it with `as_list=False` the scope is added to each `next_url` again. This results at some point to a 502 Bad Gateway error, because the URL is getting too long.

Here is a small example how I was able to reproduce it:

```py
try:
    for idx, job in enumerate(project.jobs.list(query_parameters={'scope[]': ['failed', 'success']}, as_list=False), start=1):
        if idx == 500:
            break
except Exception as exc:
    print(exc)
```

The URL is getting too long at idx == 400.

I think it started to appear after #1124 was merged. Anyhow it seems related to the brackets in the scope parameter, maybe because they get encoded in the URL.

## Expected Behavior

Iterating through all jobs.

## Actual Behavior

Getting a 502 Bad Gateway after some iterations because the URL is getting too long.

## Specifications

  - python-gitlab version: 2.6.0
  - API version you are using (v3/v4): v4
  - Gitlab server version (or gitlab.com): 13.7.4

Request URL is getting too long because of query_parameters
## Description of the problem, including code/CLI snippet

I was using the following solution to iterate over the jobs:

> @duncz there is the solution I used in my project:
> 
> ```py
> scopes = ['pending', 'running', 'failed', 'success']
> jobs = git_project.jobs.list(query_parameters={'scope[]': scopes})
> ```
> 
> Should work.

_Originally posted by @romansvozil in https://github.com/python-gitlab/python-gitlab/issues/806#issuecomment-607449709_

When using it with `as_list=False` the scope is added to each `next_url` again. This results at some point to a 502 Bad Gateway error, because the URL is getting too long.

Here is a small example how I was able to reproduce it:

```py
try:
    for idx, job in enumerate(project.jobs.list(query_parameters={'scope[]': ['failed', 'success']}, as_list=False), start=1):
        if idx == 500:
            break
except Exception as exc:
    print(exc)
```

The URL is getting too long at idx == 400.

I think it started to appear after #1124 was merged. Anyhow it seems related to the brackets in the scope parameter, maybe because they get encoded in the URL.

## Expected Behavior

Iterating through all jobs.

## Actual Behavior

Getting a 502 Bad Gateway after some iterations because the URL is getting too long.

## Specifications

  - python-gitlab version: 2.6.0
  - API version you are using (v3/v4): v4
  - Gitlab server version (or gitlab.com): 13.7.4



Discussion:

Resolving patch:
diff --git a/gitlab/client.py b/gitlab/client.py
--- a/gitlab/client.py
+++ b/gitlab/client.py
@@ -451,6 +451,10 @@ def http_request(
         post_data: Optional[Dict[str, Any]] = None,
         streamed: bool = False,
         files: Optional[Dict[str, Any]] = None,
+        timeout: Optional[float] = None,
+        obey_rate_limit: bool = True,
+        retry_transient_errors: bool = False,
+        max_retries: int = 10,
         **kwargs: Any,
     ) -> requests.Response:
         """Make an HTTP request to the Gitlab server.
@@ -465,6 +469,14 @@ def http_request(
                               json)
             streamed (bool): Whether the data should be streamed
             files (dict): The files to send to the server
+            timeout (float): The timeout, in seconds, for the request
+            obey_rate_limit (bool): Whether to obey 429 Too Many Request
+                                    responses. Defaults to True.
+            retry_transient_errors (bool): Whether to retry after 500, 502,
+                                           503, or 504 responses. Defaults
+                                           to False.
+            max_retries (int): Max retries after 429 or transient errors,
+                               set to -1 to retry forever. Defaults to 10.
             **kwargs: Extra options to send to the server (e.g. sudo)
 
         Returns:
@@ -496,9 +508,10 @@ def http_request(
         opts = self._get_session_opts(content_type="application/json")
 
         verify = opts.pop("verify")
-        timeout = opts.pop("timeout")
+        opts_timeout = opts.pop("timeout")
         # If timeout was passed into kwargs, allow it to override the default
-        timeout = kwargs.get("timeout", timeout)
+        if timeout is None:
+            timeout = opts_timeout
 
         # We need to deal with json vs. data when uploading files
         if files:
@@ -526,15 +539,7 @@ def http_request(
             prepped.url, {}, streamed, verify, None
         )
 
-        # obey the rate limit by default
-        obey_rate_limit = kwargs.get("obey_rate_limit", True)
-        # do not retry transient errors by default
-        retry_transient_errors = kwargs.get("retry_transient_errors", False)
-
-        # set max_retries to 10 by default, disable by setting it to -1
-        max_retries = kwargs.get("max_retries", 10)
         cur_retries = 0
-
         while True:
             result = self.session.send(prepped, timeout=timeout, **settings)
 
@@ -821,6 +826,9 @@ def __init__(
         self._query(url, query_data, **self._kwargs)
         self._get_next = get_next
 
+        # Remove query_parameters from kwargs, which are saved via the `next` URL
+        self._kwargs.pop("query_parameters", None)
+
     def _query(
         self, url: str, query_data: Optional[Dict[str, Any]] = None, **kwargs: Any
     ) -> None:


Test patch: