Repository: aio-libs/aiohttp

Issue:
[ClientSession behavior] Authorization header dropped during HTTP redirect
🐞 **Describe the behavior**

When using `aiohttp.ClientSession` to make a GET request that carries an `Authorization` header, the header is silently dropped whenever the remote server responds with HTTP 3xx redirections to the initial request.

📋 **Versions**

`python 3.8.1 aiohttp 3.7.4.post0 multidict 4.7.6 yarl 1.5.1`

💡 **To Reproduce**

Script to reproduce the behavior using the Discord HTTP API and a non-sensitive OAuth2 access token (note that the access tokens expires on Jun 14 2021, although the behavior should still be observable through `Response.request_info.headers`):

```py
import asyncio, logging, sys
import aiohttp, multidict, yarl

# Non-sensitive, expires on Jun 14 2021
OAUTH2_ACCESS_TOKEN = 'UHwmFVVCnXiywoXp4WMAauaPLkIDzx'
log = logging.getLogger('debug')

async def make_request(url: str):
    async with aiohttp.ClientSession() as session:
        async with session.get(url, headers={
            'Authorization': f'Bearer {OAUTH2_ACCESS_TOKEN}',  # Set Authorization header
        }) as res:
            log.info(f'Requested URL: {url}')
            log.info(f'Real URL: {res.real_url}')
            log.info(f'Request headers: { {**res.request_info.headers} }')
            log.info(f'Status: {res.status}')

async def main(no_redirect: str, redirected: str):
    logging.basicConfig(
        level=logging.DEBUG, format='%(levelname)-8s [%(name)s] %(message)s',
    )
    log.info(f'aiohttp {aiohttp.__version__}')
    log.info(f'multidict {multidict.__version__}')
    log.info(f'yarl {yarl.__version__}')
    await make_request(no_redirect)
    await asyncio.sleep(1)
    await make_request(redirected)

if __name__ == '__main__':
    asyncio.run(main(sys.argv[1], sys.argv[2]))
```

Save as `redirect.py` and run

```
python3 redirect.py https://discord.com/api/users/@me http://discord.com/api/users/@me
```

Output:

```
INFO     [debug] aiohttp 3.7.4.post0
INFO     [debug] multidict 4.7.6
INFO     [debug] yarl 1.5.1
INFO     [debug] Requested URL: https://discord.com/api/users/@me
INFO     [debug] Real URL: https://discord.com/api/users/@me
INFO     [debug] Request headers: {'Host': 'discord.com', 'Authorization': 'Bearer UHwmFVVCnXiywoXp4WMAauaPLkIDzx', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Python/3.8 aiohttp/3.7.4.post0'}
INFO     [debug] Status: 200
INFO     [debug] Requested URL: http://discord.com/api/users/@me
INFO     [debug] Real URL: https://discord.com/api/users/@me
INFO     [debug] Request headers: {'Host': 'discord.com', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Python/3.8 aiohttp/3.7.4.post0'}
INFO     [debug] Status: 401
```

💡 **Expected behavior**

Provided that the authorization info is valid, I expect that the remote server not return an HTTP 401 Unauthorized response due to `aiohttp` dropping the auth header.

📋 **Additional context**

This was raised a year ago in #4568.

These are the specific lines causing this behavior for `3.7.4.post0`: https://github.com/aio-libs/aiohttp/blob/184274d9b28bbfa06ac60e48bf286a761c6a6cb0/aiohttp/client.py#L610-L612

For the test script, the first URL is HTTPS, for which Discord returns HTTP 200 directly, but for the second, HTTP version of the API, Discord issues an HTTP 301 first to redirect the client to the HTTPS version. `aiohttp` then proceeds to drop the `Authorization` header because it is being redirected to a different origin (from http://discord.com to http**s**://discord.com).

This seems to be done for security considerations, to prevent sending the authorization info to a different location. Notably this is also the behavior for the `requests` library (see [this SO](https://stackoverflow.com/a/60451840/10896407) and this [PR on `requests`](https://github.com/psf/requests/pull/5375)).

However, for `aiohttp`, this is undocumented and is therefore surprising for developers, especially in this scenario when the remote server would like to simply upgrade from HTTP to HTTPS but stay on the same host. It is also surprising because it is different than the behaviors seen on major browsers (even though the goal of `aiohttp` is not to emulate browser behaviors). For browsers, the Fetch API spec decides that the same request info including non-forbidden headers should be reused for all redirections (see [issue 553 at whatwg/fetch](https://github.com/whatwg/fetch/issues/553)).

---

For `aiohttp`, my opinion is that it seems less surprising to at least keep the header when the redirect is to the same host/authority as in the previous request, and is done to upgrade from HTTP to HTTPS (`http://discord.com` to `https://discord.com`), instead of dropping it whenever the origin changes (emulating `curl`, see below):

```py
if (
    url.authority() != parsed_url.authority()
    or url.scheme == 'https' and parsed_url.scheme == 'http'  # downgrade
):
    auth = None
    headers.pop(hdrs.AUTHORIZATION, None)
```

and to perhaps provide some warning when it happens, or to document it somewhere.

---

I set up a test server at `https://aiohttp-issue4568.tonywu.org/` that unconditionally redirects any request to `https://discord.com/api/users/@me`. It is setup to respond correctly to CORS preflights.

On Chrome and Firefox, open the console while at `https://discord.com`, then run the following:

```js
await fetch('https://aiohttp-issue4568.tonywu.org/', {mode: 'cors', headers: {'Authorization': 'Bearer UHwmFVVCnXiywoXp4WMAauaPLkIDzx'}})
```

The fetch will succeed without receiving an HTTP 401, meaning the Authorization header survived through redirections.

Safari on the other hand has the same behavior and drops the Authorization header.

`curl` seems to be more specific: it seems to throw out the header when it is redirected to a different host, but keep it when it is upgrading on the same host:

```
$ curl -V
curl 7.77.0 (x86_64-apple-darwin20.4.0) libcurl/7.77.0 ...
```

```
$ curl -Lv 'http://discord.com/api/users/@me' -H 'Authorization: Bearer UHwmFVVCnXiywoXp4WMAauaPLkIDzx'
...
< HTTP/1.1 301 Moved Permanently
...
* Issue another request to this URL: 'https://discord.com/api/users/@me'
...
> GET /api/users/@me HTTP/2
> Host: discord.com
> user-agent: curl/7.77.0
> accept: */*
> authorization: Bearer UHwmFVVCnXiywoXp4WMAauaPLkIDzx
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200
```

```
$ curl -Lv 'https://aiohttp-issue4568.tonywu.org/' -H 'Authorization: Bearer UHwmFVVCnXiywoXp4WMAauaPLkIDzx'
...
< HTTP/1.1 301 Moved Permanently
...
* Issue another request to this URL: 'https://discord.com/api/users/@me'
...
> GET /api/users/@me HTTP/2
> Host: discord.com
> user-agent: curl/7.77.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 401
```


Discussion:
Thanks for the detailed write-up! This indeed sounds like it was designed with security in mind. We'll need to see what the RFCs say about the scenario you described before making any decisions on changing anything.
I guess the obvious action item is to update the docs and the workaround would be not following the redirect automatically but doing it on the app side rather than in the framework.
If we decide to change the logic of forwarding the auth, this would be a breaking change, meaning that it could only go into aiohttp v4 and won't be backported to the 3.x stream.
It was added in PR #2328 as a fix for the #1699. #1699 is about two different hosts. Moreover, Andrew added a test for this case:
https://github.com/aio-libs/aiohttp/blob/09ac1cbb4c07b4b3e1872374808512c9bc52f2ab/tests/test_client_functional.py#L2336-L2345

Seems like it was designed with the different hosts in mind, and `http->https` redirect is just a corner case that wasn't considered at the time, but I might be wrong.
Fair enough. PRs with a better regression test (per https://pganssle-talks.github.io/xfail-lightning/) followed by the fix is welcome.

Resolving patch:
diff --git a/aiohttp/client.py b/aiohttp/client.py
--- a/aiohttp/client.py
+++ b/aiohttp/client.py
@@ -566,7 +566,16 @@ async def _request(
                         elif not scheme:
                             parsed_url = url.join(parsed_url)
 
-                        if url.origin() != parsed_url.origin():
+                        is_same_host_https_redirect = (
+                            url.host == parsed_url.host
+                            and parsed_url.scheme == "https"
+                            and url.scheme == "http"
+                        )
+
+                        if (
+                            url.origin() != parsed_url.origin()
+                            and not is_same_host_https_redirect
+                        ):
                             auth = None
                             headers.pop(hdrs.AUTHORIZATION, None)
 


Test patch:
diff --git a/tests/test_client_functional.py b/tests/test_client_functional.py
--- a/tests/test_client_functional.py
+++ b/tests/test_client_functional.py
@@ -8,11 +8,13 @@
 import json
 import pathlib
 import socket
+import ssl
 from typing import Any
 from unittest import mock
 
 import pytest
 from multidict import MultiDict
+from yarl import URL
 
 import aiohttp
 from aiohttp import Fingerprint, ServerFingerprintMismatch, hdrs, web
@@ -2333,25 +2335,85 @@ async def test_creds_in_auth_and_url() -> None:
         await session.close()
 
 
-async def test_drop_auth_on_redirect_to_other_host(aiohttp_server: Any) -> None:
-    async def srv1(request):
-        assert request.host == "host1.com"
+@pytest.fixture
+def create_server_for_url_and_handler(
+    aiohttp_server: Any, tls_certificate_authority: Any
+):
+    def create(url: URL, srv: Any):
+        app = web.Application()
+        app.router.add_route("GET", url.path, srv)
+
+        kwargs = {}
+        if url.scheme == "https":
+            cert = tls_certificate_authority.issue_cert(
+                url.host, "localhost", "127.0.0.1"
+            )
+            ssl_ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
+            cert.configure_cert(ssl_ctx)
+            kwargs["ssl"] = ssl_ctx
+        return aiohttp_server(app, **kwargs)
+
+    return create
+
+
+@pytest.mark.parametrize(
+    ["url_from", "url_to", "is_drop_header_expected"],
+    [
+        [
+            "http://host1.com/path1",
+            "http://host2.com/path2",
+            True,
+        ],
+        ["http://host1.com/path1", "https://host1.com/path1", False],
+        ["https://host1.com/path1", "http://host1.com/path2", True],
+    ],
+    ids=(
+        "entirely different hosts",
+        "http -> https",
+        "https -> http",
+    ),
+)
+async def test_drop_auth_on_redirect_to_other_host(
+    create_server_for_url_and_handler: Any,
+    url_from: str,
+    url_to: str,
+    is_drop_header_expected: bool,
+) -> None:
+    url_from, url_to = URL(url_from), URL(url_to)
+
+    async def srv_from(request):
+        assert request.host == url_from.host
         assert request.headers["Authorization"] == "Basic dXNlcjpwYXNz"
-        raise web.HTTPFound("http://host2.com/path2")
+        raise web.HTTPFound(url_to)
 
-    async def srv2(request):
-        assert request.host == "host2.com"
-        assert "Authorization" not in request.headers
+    async def srv_to(request):
+        assert request.host == url_to.host
+        if is_drop_header_expected:
+            assert "Authorization" not in request.headers, "Header wasn't dropped"
+        else:
+            assert "Authorization" in request.headers, "Header was dropped"
         return web.Response()
 
-    app = web.Application()
-    app.router.add_route("GET", "/path1", srv1)
-    app.router.add_route("GET", "/path2", srv2)
+    server_from = await create_server_for_url_and_handler(url_from, srv_from)
+    server_to = await create_server_for_url_and_handler(url_to, srv_to)
 
-    server = await aiohttp_server(app)
+    assert (
+        url_from.host != url_to.host or server_from.scheme != server_to.scheme
+    ), "Invalid test case, host or scheme must differ"
+
+    protocol_port_map = {
+        "http": 80,
+        "https": 443,
+    }
+    etc_hosts = {
+        (url_from.host, protocol_port_map[server_from.scheme]): server_from,
+        (url_to.host, protocol_port_map[server_to.scheme]): server_to,
+    }
 
     class FakeResolver(AbstractResolver):
         async def resolve(self, host, port=0, family=socket.AF_INET):
+            server = etc_hosts[(host, port)]
+
             return [
                 {
                     "hostname": host,
@@ -2366,14 +2428,17 @@ async def resolve(self, host, port=0, family=socket.AF_INET):
         async def close(self):
             pass
 
-    connector = aiohttp.TCPConnector(resolver=FakeResolver())
+    connector = aiohttp.TCPConnector(resolver=FakeResolver(), ssl=False)
+
     async with aiohttp.ClientSession(connector=connector) as client:
         resp = await client.get(
-            "http://host1.com/path1", auth=aiohttp.BasicAuth("user", "pass")
+            url_from,
+            auth=aiohttp.BasicAuth("user", "pass"),
         )
         assert resp.status == 200
         resp = await client.get(
-            "http://host1.com/path1", headers={"Authorization": "Basic dXNlcjpwYXNz"}
+            url_from,
+            headers={"Authorization": "Basic dXNlcjpwYXNz"},
         )
         assert resp.status == 200