Repository: urllib3/urllib3

Issue:
Scheme and host erroneously passed to HTTPConnection request method
I think there is a problem in the use of `httplib.HTTPConnection` method `request` when called at
[line 213 of urllib3/connectionpool.py](https://github.com/shazow/urllib3/blob/master/urllib3/connectionpool.py#L213) where you pass it the full URL, containing the scheme and host, instead of just the path (and query part), as show in [httplib usage examples](http://docs.python.org/library/httplib.html#examples).

This ends up in a wrong HTTP request performed to the server. To see it, you can for instance run

```
python -m SimpleHTTPServer
```

in a shell and then, in another one, run

```
python -c 'from urllib3 import PoolManager; http = PoolManager(); http.request( "GET", "http://localhost:8000/this/is/an/example" )'
```

and compare what the access log in the first shell reports as compared to what happens if you do 

```
curl  "http://localhost:8000/this/is/an/example"
```

I can submit a patch, but I'm not an urllib3 expert so I will probably miss some other place where the same error occurs.



Discussion:
Hi there, thank you for the report!

This was a conscious decision, but perhaps not the correct one.

The goal was to reduce complexity and avoid inexplicit behaviour. That is, when you make a request to "http://localhost:8000/this/is/an/example", that's exactly the request that urllib3 should be making.

The current workaround to achieve what you want is:

``` python
from urllib3 import PoolManager
http = PoolManager()
conn = http.connection_from_url("http://localhost:8000")
response = conn.request("GET", "/this/is/an/example")
```

When we do PoolManager.request, it does the same thing behind the scenes except it doesn't strip away the host like we did here manually. I agree that there should be an option to strip away the host (perhaps even by default).

Should this option be specified in the PoolManager constructor? Such as `PoolManager(strip_host=True)`.

But then when should the stripping occur? If it happens in urlopen, then should we backport the same functionality outside of PoolManager? (ie. into ConnectionPool objects.)

The request should definitely be made with the path (and the query) only, because urllib3 is a HTTP/1.1 client.
[RFC2616](http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1.2):

> To allow for transition to absoluteURIs in all requests in future versions of HTTP, all HTTP/1.1 servers MUST accept the
> absoluteURI form in requests, even though HTTP/1.1 clients will only generate them in requests to proxies.

Excellent info. Thanks :)

This isn't a bug in urllib3. It's doing exactly what it's told.

Btw, if anyone is in dire need, here's a handy basic recipe for doing "proper" url passing with redirection in urllib3:

``` python
import urlparse
import urllib3

http = urllib3.PoolManager()

def request(method, url, conn=None):
    if conn:
        # Request within the current host connection (used for redirect handling)
        if not url.startswith('/'):
            url = '/' + url
        r = conn.request(method, url, redirect=False, assert_same_host=False)
    else:
        p = urlparse.urlparse(url)
        conn = http.connection_from_host(p.hostname, p.port, p.scheme)
        r = conn.request(method, p.path, redirect=False, assert_same_host=False)

    is_redirect = r.get_redirect_location()
    if not is_redirect:
        return r
    print "Redirecting: %s" % is_redirect

    if '://' not in is_redirect:
        # Redirect to same host
        return request('GET', is_redirect, conn)

    return request('GET', is_redirect)
```


Resolving patch:
diff --git a/urllib3/connectionpool.py b/urllib3/connectionpool.py
--- a/urllib3/connectionpool.py
+++ b/urllib3/connectionpool.py
@@ -268,15 +268,16 @@ def _make_request(self, conn, method, url, timeout=_Default,
         log.debug("\"%s %s %s\" %s %s" % (method, url, http_version,
                                           httplib_response.status,
                                           httplib_response.length))
-
         return httplib_response
 
-
     def is_same_host(self, url):
         """
         Check if the given ``url`` is a member of the same host as this
         connection pool.
         """
+        if url.startswith('/'):
+            return True
+
         # TODO: Add optional support for socket.gethostbyname checking.
         scheme, host, port = get_host(url)
 
@@ -284,8 +285,7 @@ def is_same_host(self, url):
             # Use explicit default port for comparison when none is given.
             port = port_by_scheme.get(scheme)
 
-        return (url.startswith('/') or
-                (scheme, host, port) == (self.scheme, self.host, self.port))
+        return (scheme, host, port) == (self.scheme, self.host, self.port)
 
     def urlopen(self, method, url, body=None, headers=None, retries=3,
                 redirect=True, assert_same_host=True, timeout=_Default,
diff --git a/urllib3/poolmanager.py b/urllib3/poolmanager.py
--- a/urllib3/poolmanager.py
+++ b/urllib3/poolmanager.py
@@ -8,9 +8,10 @@
 
 from ._collections import RecentlyUsedContainer
 from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool
-from .connectionpool import get_host, connection_from_url, port_by_scheme
+from .connectionpool import connection_from_url, port_by_scheme
 from .exceptions import HostChangedError
 from .request import RequestMethods
+from .util import parse_url
 
 
 __all__ = ['PoolManager', 'ProxyManager', 'proxy_from_url']
@@ -54,13 +55,15 @@ def __init__(self, num_pools=10, **connection_pool_kw):
         self.connection_pool_kw = connection_pool_kw
         self.pools = RecentlyUsedContainer(num_pools)
 
-    def connection_from_host(self, host, port=80, scheme='http'):
+    def connection_from_host(self, host, port=None, scheme='http'):
         """
         Get a :class:`ConnectionPool` based on the host, port, and scheme.
 
-        Note that an appropriate ``port`` value is required here to normalize
-        connection pools in our container most effectively.
+        If ``port`` isn't given, it will be derived from the ``scheme`` using
+        ``urllib3.connectionpool.port_by_scheme``.
         """
+        port = port or port_by_scheme.get(scheme, 80)
+
         pool_key = (scheme, host, port)
 
         # If the scheme, host, or port doesn't match existing open connections,
@@ -86,26 +89,39 @@ def connection_from_url(self, url):
         Additional parameters are taken from the :class:`.PoolManager`
         constructor.
         """
-        scheme, host, port = get_host(url)
+        u = parse_url(url)
+        return self.connection_from_host(u.host, port=u.port, scheme=u.scheme)
 
-        port = port or port_by_scheme.get(scheme, 80)
+    def handle_redirect(response):
+        pass
 
-        return self.connection_from_host(host, port=port, scheme=scheme)
-
-    def urlopen(self, method, url, **kw):
+    def urlopen(self, method, url, redirect=True, **kw):
         """
-        Same as :meth:`urllib3.connectionpool.HTTPConnectionPool.urlopen`.
+        Same as :meth:`urllib3.connectionpool.HTTPConnectionPool.urlopen`
+        with custom cross-host redirect logic and only sends the request-uri
+        portion of the ``url``.
 
-        ``url`` must be absolute, such that an appropriate
+        The given ``url`` parameter must be absolute, such that an appropriate
         :class:`urllib3.connectionpool.ConnectionPool` can be chosen for it.
         """
-        conn = self.connection_from_url(url)
-        try:
-            return conn.urlopen(method, url, **kw)
+        u = parse_url(url)
+        conn = self.connection_from_host(u.host, port=u.port, scheme=u.scheme)
+
+        kw['assert_same_host'] = False
+        kw['redirect'] = False
+
+        response = conn.urlopen(method, u.request_uri, **kw)
+
+        redirect_location = redirect and response.get_redirect_location()
+        if not redirect_location:
+            return response
+
+        if response.status == 303:
+            method = 'GET'
 
-        except HostChangedError as e:
-            kw['retries'] = e.retries # Persist retries countdown
-            return self.urlopen(method, e.url, **kw)
+        log.info("Redirecting %s -> %s" % (url, redirect_location))
+        kw['retries'] = kw.get('retries', 3) - 1 # Persist retries countdown
+        return self.urlopen(method, redirect_location, **kw)
 
 
 class ProxyManager(RequestMethods):


Test patch:
diff --git a/test/with_dummyserver/test_poolmanager.py b/test/with_dummyserver/test_poolmanager.py
--- a/test/with_dummyserver/test_poolmanager.py
+++ b/test/with_dummyserver/test_poolmanager.py
@@ -32,7 +32,7 @@ def test_cross_host_redirect(self):
         try:
             http.request('GET', '%s/redirect' % self.base_url,
                          fields={'target': cross_host_location},
-                         timeout=0.01, retries=1)
+                         timeout=0.01, retries=0)
             self.fail("Request succeeded instead of raising an exception like it should.")
 
         except MaxRetryError:
@@ -40,7 +40,7 @@ def test_cross_host_redirect(self):
 
         r = http.request('GET', '%s/redirect' % self.base_url,
                          fields={'target': '%s/echo?a=b' % self.base_url_alt},
-                         timeout=0.01, retries=2)
+                         timeout=0.01, retries=1)
 
         self.assertEqual(r._pool.host, self.host_alt)