Repository: encode/httpx
Pull request: Handle multiple auth headers correctly
Closes #1234 by correctly treating WWW-Authenticate headers as possibly occurring multiple times, and only dealing with “Digest” headers, and just ignoring other types of authentications.

Problem statement:
Auth fails for responses that include multiple WWW-Authenticate headers.
### Checklist

httpx==0.11.1

### Describe the bug
I use httpx request camera mjpeg,The camera response two WWW-Authenticate headers,I used the right username and password,but It response 401. When I user chrome, everything is ok, I doubt the httpx.DigestAuth may have bug when handle digest.
WireShark Data as follows:

**request with httpx**
```
GET /cgi-bin/guest/Video.cgi?media=MJPEG&channel=0 HTTP/1.1
host: 192.168.12.88
user-agent: python-httpx/0.11.1
accept: */*
accept-encoding: gzip, deflate, br
connection: keep-alive

HTTP/1.1 401 Unauthorized
Connection: close
Content-Type: text/html; charset=ISO-8859-1
WWW-Authenticate: Digest realm="server", nonce="4f3b1a0c93d39d1cab4d5d2a8928f85a3cc41bd5e8180e5c37587a9a8aea5097"
WWW-Authenticate: Basic realm="server"
Content-Length: 211

<HTML><HEAD><TITLE>401 Unauthorized</TITLE></HEAD>
<BODY><H1>401 Unauthorized</H1>
Your client does not have permission to get URL /cgi-bin/guest/Video.cgi?media=MJPEG&channel=0 from this server.
</BODY></HTML>

GET /cgi-bin/guest/Video.cgi?media=MJPEG&channel=0 HTTP/1.1
host: 192.168.12.88
user-agent: python-httpx/0.11.1
accept: */*
accept-encoding: gzip, deflate, br
connection: keep-alive
authorization: Digest username="admin", realm="server", nonce="4f3b1a0c93d39d1cab4d5d2a8928f85a3cc41bd5e8180e5c37587a9a8aea5097", uri="/cgi-bin/guest/Video.cgi?media=MJPEG&channel=0", response="43265b82a45736ef943ae501f0234807", algorithm=MD5

HTTP/1.1 401 Unauthorized
Connection: close
Content-Type: text/html; charset=ISO-8859-1
WWW-Authenticate: Digest realm="server", nonce="4f3b1a0ca12e0b92dc8ccca22e84edf7ce355d7825ac0e21eedf8f97d82ec588"
WWW-Authenticate: Basic realm="server"
Content-Length: 211

<HTML><HEAD><TITLE>401 Unauthorized</TITLE></HEAD>
<BODY><H1>401 Unauthorized</H1>
Your client does not have permission to get URL /cgi-bin/guest/Video.cgi?media=MJPEG&channel=0 from this server.
</BODY></HTML>
```

**request with chrome**
```
GET /cgi-bin/guest/Video.cgi?media=MJPEG&channel=0 HTTP/1.1
Host: 192.168.12.88
Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9,en;q=0.8
Cookie: pbAudioFalg=ON; RESOLUTION=720P; LifeTime=0; captcha=4PTT5z50/JVrA

HTTP/1.1 401 Unauthorized
Connection: close
Content-Type: text/html; charset=ISO-8859-1
WWW-Authenticate: Digest realm="server", nonce="4f3b1c6b2dccae706af263fdc5001a702ae3512d3cc4e42ebf823c60431f64b6"
WWW-Authenticate: Basic realm="server"
Content-Length: 211

<HTML><HEAD><TITLE>401 Unauthorized</TITLE></HEAD>
<BODY><H1>401 Unauthorized</H1>
Your client does not have permission to get URL /cgi-bin/guest/Video.cgi?media=MJPEG&channel=0 from this server.
</BODY></HTML>

GET /cgi-bin/guest/Video.cgi?media=MJPEG&channel=0 HTTP/1.1
Host: 192.168.12.88
Connection: keep-alive
Cache-Control: max-age=0
Authorization: Digest username="admin", realm="server", nonce="4f3b1c6b2dccae706af263fdc5001a702ae3512d3cc4e42ebf823c60431f64b6", uri="/cgi-bin/guest/Video.cgi?media=MJPEG&channel=0", response="7411b325f99b1ef61db78f6a1ba3ff9d"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9,en;q=0.8
Cookie: pbAudioFalg=ON; RESOLUTION=720P; LifeTime=0; captcha=4PTT5z50/JVrA

HTTP/1.1 200 OK
Expires: 0
Pragma: no-cache
Cache-Control: no-cache
Content-Type: multipart/x-mixed-replace;boundary="boundary"

--boundary
Content-Type: image/jpeg
Content-Length: 59048

xxxxx
```

Hints from issue discussion:
Hello!

* You're using a rather old-ish HTTPX version (0.11.1). What does this look like on the latest HTTPX release (0.14.2)?
* Can you share a minimal example that reproduces the issue on your side?
* Are there any details about the server/device you're requesting worth sharing? (Make and model, server documentation, etc.)
Perhaps a server-side implementation that's treating the `Authentication` header in a case sensitive way, and so is failing to authenticate correctly?
I switch to the latest version httpx==0.14.2,It seems bug still exists, I found a camera instance on public network, And I wrote a simple example to illustrate this problem， as follows, I used requests and httpx request the same url, the requests can  authentication successfully,but httpx failed. 
```

import httpx
import requests
from requests.auth import HTTPDigestAuth

url = "http://190.156.226.164:8083/cgi-bin/guest/Video.cgi?media=MJPEG&channel=0"


def test_requests():
    auth = HTTPDigestAuth("admin", "admin")
    req = requests.get(url, auth=auth, stream=True)
    print(req.status_code)
    req.close()


def test_httpx():
    auth = httpx.DigestAuth("admin", "admin")
    with httpx.stream("GET", url, auth=auth) as r:
        print(r.status_code)


if __name__ == '__main__':
    test_requests()
    test_httpx()
```
output:
```
200
401
```
> Perhaps a server-side implementation that's treating the `Authentication` header in a case sensitive way, and so is failing to authenticate correctly?

It might be like you said， Is there any way to modify it？
@beruhan To confirm that case sensitivity is causing this issue, you can try adjusting the request exemple so that it sends `authorization` (lowercase) instead of `Authorization`. (I don't have a code snippet off the top of my head right now, but the Requests docs should have some examples of modifying headers before sending the request.)

If this is confirmed then this would be related to #538.
like you said,I modified the requests code to lowercase `Authorization` header,It can also authenticated,
```
GET /cgi-bin/guest/Video.cgi?media=MJPEG&channel=0 HTTP/1.1
Host: 192.168.12.88
User-Agent: python-requests/2.24.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
authorization: Digest username="admin", realm="server", nonce="4f3e773156897b27b411c1e6e28dfc81ba65d826ceaad75908ef1194b749acd2", uri="/cgi-bin/guest/Video.cgi?media=MJPEG&channel=0", response="be2fc243ffba15e79854605b202a301a"

HTTP/1.1 200 OK
Expires: 0
Pragma: no-cache
Cache-Control: no-cache
Content-Type: multipart/x-mixed-replace;boundary="boundary"
```
I don't know way the problem,I debug the authenticate process in requests and httpx as follows:
*requests authenticate process*
```
www-authenticate header received======= Digest realm="server", nonce="4f3e79d568136064295acb9ac2a95ae3a635ba5807cfda9d74549b5a1d4a9947", Basic realm="server"

Authorization header======= Digest username="admin", realm="server", 
nonce="4f3e79d568136064295acb9ac2a95ae3a635ba5807cfda9d74549b5a1d4a9947", uri="/cgi-bin/guest/Video.cgi?media=MJPEG&channel=0", response="e1198d68b1e06ebe68109cb10dce09ea"
```
*httpx authenticate process*
```
www-authenticate header received======= Digest realm="server", nonce="4f3e79d526379f6c6b9c52bb4f74a864d3e4ee32f3ce72e59bc6b817cb804450", Basic realm="server"

Authorization header======= Digest username="admin", realm="server", nonce="4f3e79d526379f6c6b9c52bb4f74a864d3e4ee32f3ce72e59bc6b817cb804450", uri="/cgi-bin/guest/Video.cgi?media=MJPEG&channel=0", response="8e82811163fff43a2522d4f7a15b1f9f"
```
I also modify the code to make camera response header as the same ,
```
www-authenticate header received======= Digest realm="server", nonce="4f3e79d568136064295acb9ac2a95ae3a635ba5807cfda9d74549b5a1d4a9947", Basic realm="server"
Authorization header======= Digest username="admin", realm="server", nonce="4f3e79d568136064295acb9ac2a95ae3a635ba5807cfda9d74549b5a1d4a9947", uri="/cgi-bin/guest/Video.cgi?media=MJPEG&channel=0", response="e1198d68b1e06ebe68109cb10dce09ea"
200
www-authenticate header received======= Digest realm="server", nonce="4f3e79d568136064295acb9ac2a95ae3a635ba5807cfda9d74549b5a1d4a9947", Basic realm="server"
Authorization header======= Digest username="admin", realm="server", nonce="4f3e79d568136064295acb9ac2a95ae3a635ba5807cfda9d74549b5a1d4a9947", uri="/cgi-bin/guest/Video.cgi?media=MJPEG&channel=0", response="1189f82b9463486970d32134d3e9f6d4", algorithm=MD5
401
``` 
Somehow I'd completely missed that this issue is specific to the case when the server sends *two* WWW-Authenticate headers in the response.

And yup, can see how that might pick out a bug on our side, and yup can see how we ought to resolve this.

Leaving a link to a useful part of Mozilla's multiple header handling... https://github.com/bnoordhuis/mozilla-central/blob/master/netwerk/protocol/http/nsHttpHeaderArray.h#L185 for my own later reference in this.

Commit message: Handle multiple auth headers correctly

Patch to review:
diff --git a/httpx/_auth.py b/httpx/_auth.py
index 571584593b..eb110dea3a 100644
--- a/httpx/_auth.py
+++ b/httpx/_auth.py
@@ -112,28 +112,34 @@ def auth_flow(self, request: Request) -> typing.Generator[Request, Response, Non
         response = yield request
 
         if response.status_code != 401 or "www-authenticate" not in response.headers:
-            # If the response is not a 401 WWW-Authenticate, then we don't
+            # If the response is not a 401 then we don't
             # need to build an authenticated request.
             return
 
-        challenge = self._parse_challenge(request, response)
+        for auth_header in response.headers.get_list("www-authenticate"):
+            if auth_header.lower().startswith("digest "):
+                break
+        else:
+            # If the response does not include a 'WWW-Authenticate: Digest ...'
+            # header, then we don't need to build an authenticated request.
+            return
+
+        challenge = self._parse_challenge(request, response, auth_header)
         request.headers["Authorization"] = self._build_auth_header(request, challenge)
         yield request
 
     def _parse_challenge(
-        self, request: Request, response: Response
+        self, request: Request, response: Response, auth_header: str
     ) -> "_DigestAuthChallenge":
         """
         Returns a challenge from a Digest WWW-Authenticate header.
         These take the form of:
         `Digest realm="realm@host.com",qop="auth,auth-int",nonce="abc",opaque="xyz"`
         """
-        header = response.headers["www-authenticate"]
+        scheme, _, fields = auth_header.partition(" ")
 
-        scheme, _, fields = header.partition(" ")
-        if scheme.lower() != "digest":
-            message = "Header does not start with 'Digest'"
-            raise ProtocolError(message, request=request)
+        # This method should only ever have been called with a Digest auth header.
+        assert scheme.lower() == "digest"
 
         header_dict: typing.Dict[str, str] = {}
         for field in parse_http_list(fields):
diff --git a/tests/client/test_auth.py b/tests/client/test_auth.py
index ae35e2cd35..b2b8c01e51 100644
--- a/tests/client/test_auth.py
+++ b/tests/client/test_auth.py
@@ -354,6 +354,21 @@ async def test_digest_auth_returns_no_auth_if_no_digest_header_in_response() ->
     assert len(response.history) == 0
 
 
+def test_digest_auth_returns_no_auth_if_alternate_auth_scheme() -> None:
+    url = "https://example.org/"
+    auth = DigestAuth(username="tomchristie", password="password123")
+    auth_header = b"Token ..."
+
+    client = Client(
+        transport=SyncMockTransport(auth_header=auth_header, status_code=401)
+    )
+    response = client.get(url, auth=auth)
+
+    assert response.status_code == 401
+    assert response.json() == {"auth": None}
+    assert len(response.history) == 0
+
+
 @pytest.mark.asyncio
 async def test_digest_auth_200_response_including_digest_auth_header() -> None:
     url = "https://example.org/"
@@ -509,9 +524,6 @@ async def test_digest_auth_incorrect_credentials() -> None:
     "auth_header",
     [
         b'Digest realm="httpx@example.org", qop="auth"',  # missing fields
-        b'realm="httpx@example.org", qop="auth"',  # not starting with Digest
-        b'DigestZ realm="httpx@example.org", qop="auth"'
-        b'qop="auth,auth-int",nonce="abc",opaque="xyz"',
         b'Digest realm="httpx@example.org", qop="auth,au',  # malformed fields list
     ],
 )
@@ -533,9 +545,6 @@ async def test_async_digest_auth_raises_protocol_error_on_malformed_header(
     "auth_header",
     [
         b'Digest realm="httpx@example.org", qop="auth"',  # missing fields
-        b'realm="httpx@example.org", qop="auth"',  # not starting with Digest
-        b'DigestZ realm="httpx@example.org", qop="auth"'
-        b'qop="auth,auth-int",nonce="abc",opaque="xyz"',
         b'Digest realm="httpx@example.org", qop="auth,au',  # malformed fields list
     ],
 )


Code review comments:
diff_hunk: @@ -354,6 +354,21 @@ async def test_digest_auth_returns_no_auth_if_no_digest_header_in_response() ->
     assert len(response.history) == 0
 
 
+def test_digest_auth_returns_no_auth_if_alternate_auth_scheme() -> None:
+    url = "https://example.org/"
+    auth = DigestAuth(username="tomchristie", password="password123")
+    auth_header = b"Token ..."
+
+    client = Client(
path: tests/client/test_auth.py
text: @user1:
CI is failing, I guess this should be `httpx.Client`?