Repository: urllib3/urllib3

Issue:
Auth header remains during redirects
Requests does it: https://github.com/request/request/pull/1184


Discussion:
Except urllib3 doesn't directly handle authentication the way Requests does. I don't disagree, but there are fundamentally different layers of concern here.
I tend to agree with @sigmavirus24. Our header handling is much more agnostic to what those headers actually are. If we implemented authentication directly, that'd be one thing, but we don't.

I would be interested to see what cURL does here in the specific case where it's given `-H "Authorization: Basic xxxxxxxxx"` as compared to `-U "username:password"`. If it's the case that cURL strips a manually-constructed header on cross-host redirect, then I could be convinced to entertain a PR, although I likely wouldn't spend time on it myself.
@haikuginger actually, cURL preserves authentication on redirect in the face of leaking credential. But it also doesn't automagic redirects like requests does. To do that you need to specify `-L` so sending auth in the header or with `-u` won't automatically give away credentials unless you manually follow it or specify `-L`.
(Since you said you'd be interested)
urllib3 redirects across hosts by default these days, doesn't it? Sounds like a pretty scary surprise-footgun to have.

At minimum, sounds like the redirect feature needs a big fat warning that all headers including authentication headers will be forwarded along. Ideally it feels like request-specific headers need to be stripped for cross-host by default (maybe a `urllib3.Retry` object param?).

(Sorry for helicopter-commenting here, just randomly saw this thread in my filter, please feel free to ignore. :heart:)

Edit: Random scary scenario I'm thinking about: You have an API that for some reason can yield a redirect (it's a thing, [I've even built one in 2010](https://github.com/shazow/302found)), then by default the redirect destination is going to get your secret bearer token and all that jazz. Especially scary as a _default_.
Btw if we decide it's a good idea and nobody wants to do it, I'll put together a quick PR. Should be 4-5 lines of changes in urlopen and Retry.
I agree this is a potential footgun. I'd almost be tempted to turn off redirect following by default.
If I turn off redirect, the script breaks e.g redirect=False.  It needs to redirect just not with auth.  This is a big problem if you use say a webhost server and then have to redirect to Amazon.  A workaround for now is to use a monkey patch to remove auth.  
@YODABear, understandable, but our concerns are for all users, not just for one specific use case. I'd welcome feedback from @Lukasa, @sigmavirus24, and @shazow as to what the Right Thing to Do is considering that we don't have an end-to-end API for HTTP auth. Reopening this issue for that discussion of the broader case.
@shazow there is more discussion of potential problems with this over on the relevant Requests issues and I agree with you that it's a problem. I thought urllib3 had redirect handling off-by-default.

I can also toss together a PR for this.

> If I turn off redirect, the script breaks e.g redirect=False.

That's surprising but I suspect that if you asked for help on [StackOverflow](https://stackoverflow.com) they would help you understand why its breaking and how to fix it while we fix this underlying bug.
@sigmavirus24, please go right ahead.
@sigmavirus24 Right, IMO any changes we make should be configurable back to what we have today via the Retry object. I was imagining something like `retries=Retry(forward_headers_across_hosts=True)` in the spirit of keeping scary flags annoyingly long and explicit.
I doubt I'm the only one who has come across this issue.  Anyone who uses a host with auth and has to redirect with no auth has likely encountered this (so likely issue will be asked by someone else in the future if not fixed). I had a look at some SO posts.   In the mean time, the only viable method seems to be using monkey patches.  Can get quite length depending on how many different hosts you have.  Another possibility could be to urlparse out the auth credentials though monkey patch works fine for now so never tested.
@sigmavirus24, were you still planning on handling this? If not, I can make the change myself.

Resolving patch:
diff --git a/urllib3/poolmanager.py b/urllib3/poolmanager.py
--- a/urllib3/poolmanager.py
+++ b/urllib3/poolmanager.py
@@ -312,8 +312,9 @@ def urlopen(self, method, url, redirect=True, **kw):
 
         kw['assert_same_host'] = False
         kw['redirect'] = False
+
         if 'headers' not in kw:
-            kw['headers'] = self.headers
+            kw['headers'] = self.headers.copy()
 
         if self.proxy is not None and u.scheme == "http":
             response = conn.urlopen(method, url, **kw)
@@ -335,6 +336,14 @@ def urlopen(self, method, url, redirect=True, **kw):
         if not isinstance(retries, Retry):
             retries = Retry.from_int(retries, redirect=redirect)
 
+        # Strip headers marked as unsafe to forward to the redirected location.
+        # Check remove_headers_on_redirect to avoid a potential network call within
+        # conn.is_same_host() which may use socket.gethostbyname() in the future.
+        if (retries.remove_headers_on_redirect
+                and not conn.is_same_host(redirect_location)):
+            for header in retries.remove_headers_on_redirect:
+                kw['headers'].pop(header, None)
+
         try:
             retries = retries.increment(method, url, response=response, _pool=conn)
         except MaxRetryError:
diff --git a/urllib3/util/retry.py b/urllib3/util/retry.py
--- a/urllib3/util/retry.py
+++ b/urllib3/util/retry.py
@@ -19,6 +19,7 @@
 
 log = logging.getLogger(__name__)
 
+
 # Data structure for representing the metadata of requests that result in a retry.
 RequestHistory = namedtuple('RequestHistory', ["method", "url", "error",
                                                "status", "redirect_location"])
@@ -139,6 +140,10 @@ class Retry(object):
         Whether to respect Retry-After header on status codes defined as
         :attr:`Retry.RETRY_AFTER_STATUS_CODES` or not.
 
+    :param iterable remove_headers_on_redirect:
+        Sequence of headers to remove from the request when a response
+        indicating a redirect is returned before firing off the redirected
+        request.
     """
 
     DEFAULT_METHOD_WHITELIST = frozenset([
@@ -146,13 +151,16 @@ class Retry(object):
 
     RETRY_AFTER_STATUS_CODES = frozenset([413, 429, 503])
 
+    DEFAULT_REDIRECT_HEADERS_BLACKLIST = frozenset(['Authorization'])
+
     #: Maximum backoff time.
     BACKOFF_MAX = 120
 
     def __init__(self, total=10, connect=None, read=None, redirect=None, status=None,
                  method_whitelist=DEFAULT_METHOD_WHITELIST, status_forcelist=None,
                  backoff_factor=0, raise_on_redirect=True, raise_on_status=True,
-                 history=None, respect_retry_after_header=True):
+                 history=None, respect_retry_after_header=True,
+                 remove_headers_on_redirect=DEFAULT_REDIRECT_HEADERS_BLACKLIST):
 
         self.total = total
         self.connect = connect
@@ -171,6 +179,7 @@ def __init__(self, total=10, connect=None, read=None, redirect=None, status=None
         self.raise_on_status = raise_on_status
         self.history = history or tuple()
         self.respect_retry_after_header = respect_retry_after_header
+        self.remove_headers_on_redirect = remove_headers_on_redirect
 
     def new(self, **kw):
         params = dict(
@@ -182,6 +191,7 @@ def new(self, **kw):
             raise_on_redirect=self.raise_on_redirect,
             raise_on_status=self.raise_on_status,
             history=self.history,
+            remove_headers_on_redirect=self.remove_headers_on_redirect
         )
         params.update(kw)
         return type(self)(**params)


Test patch:
diff --git a/test/test_retry.py b/test/test_retry.py
--- a/test/test_retry.py
+++ b/test/test_retry.py
@@ -249,3 +249,13 @@ def test_retry_method_not_in_whitelist(self):
         retry = Retry()
         with pytest.raises(ReadTimeoutError):
             retry.increment(method='POST', error=error)
+
+    def test_retry_default_remove_headers_on_redirect(self):
+        retry = Retry()
+
+        assert list(retry.remove_headers_on_redirect) == ['Authorization']
+
+    def test_retry_set_remove_headers_on_redirect(self):
+        retry = Retry(remove_headers_on_redirect=['X-API-Secret'])
+
+        assert list(retry.remove_headers_on_redirect) == ['X-API-Secret']
diff --git a/test/with_dummyserver/test_poolmanager.py b/test/with_dummyserver/test_poolmanager.py
--- a/test/with_dummyserver/test_poolmanager.py
+++ b/test/with_dummyserver/test_poolmanager.py
@@ -109,6 +109,52 @@ def test_too_many_redirects(self):
         except MaxRetryError:
             pass
 
+    def test_redirect_cross_host_remove_headers(self):
+        http = PoolManager()
+        self.addCleanup(http.clear)
+
+        r = http.request('GET', '%s/redirect' % self.base_url,
+                         fields={'target': '%s/headers' % self.base_url_alt},
+                         headers={'Authorization': 'foo'})
+
+        self.assertEqual(r.status, 200)
+
+        data = json.loads(r.data.decode('utf-8'))
+
+        self.assertNotIn('Authorization', data)
+
+    def test_redirect_cross_host_no_remove_headers(self):
+        http = PoolManager()
+        self.addCleanup(http.clear)
+
+        r = http.request('GET', '%s/redirect' % self.base_url,
+                         fields={'target': '%s/headers' % self.base_url_alt},
+                         headers={'Authorization': 'foo'},
+                         retries=Retry(remove_headers_on_redirect=[]))
+
+        self.assertEqual(r.status, 200)
+
+        data = json.loads(r.data.decode('utf-8'))
+
+        self.assertEqual(data['Authorization'], 'foo')
+
+    def test_redirect_cross_host_set_removed_headers(self):
+        http = PoolManager()
+        self.addCleanup(http.clear)
+
+        r = http.request('GET', '%s/redirect' % self.base_url,
+                         fields={'target': '%s/headers' % self.base_url_alt},
+                         headers={'X-API-Secret': 'foo',
+                                  'Authorization': 'bar'},
+                         retries=Retry(remove_headers_on_redirect=['X-API-Secret']))
+
+        self.assertEqual(r.status, 200)
+
+        data = json.loads(r.data.decode('utf-8'))
+
+        self.assertNotIn('X-API-Secret', data)
+        self.assertEqual(data['Authorization'], 'bar')
+
     def test_raise_on_redirect(self):
         http = PoolManager()
         self.addCleanup(http.clear)