Repository: urllib3/urllib3

Issue:
Scheme and host erroneously passed to HTTPConnection request method
I think there is a problem in the use of `httplib.HTTPConnection` method `request` when called at
[line 213 of urllib3/connectionpool.py](https://github.com/shazow/urllib3/blob/master/urllib3/connectionpool.py#L213) where you pass it the full URL, containing the scheme and host, instead of just the path (and query part), as show in [httplib usage examples](http://docs.python.org/library/httplib.html#examples).

This ends up in a wrong HTTP request performed to the server. To see it, you can for instance run

```
python -m SimpleHTTPServer
```

in a shell and then, in another one, run

```
python -c 'from urllib3 import PoolManager; http = PoolManager(); http.request( "GET", "http://localhost:8000/this/is/an/example" )'
```

and compare what the access log in the first shell reports as compared to what happens if you do 

```
curl  "http://localhost:8000/this/is/an/example"
```

I can submit a patch, but I'm not an urllib3 expert so I will probably miss some other place where the same error occurs.



Discussion:
Hi there, thank you for the report!

This was a conscious decision, but perhaps not the correct one.

The goal was to reduce complexity and avoid inexplicit behaviour. That is, when you make a request to "http://localhost:8000/this/is/an/example", that's exactly the request that urllib3 should be making.

The current workaround to achieve what you want is:

``` python
from urllib3 import PoolManager
http = PoolManager()
conn = http.connection_from_url("http://localhost:8000")
response = conn.request("GET", "/this/is/an/example")
```

When we do PoolManager.request, it does the same thing behind the scenes except it doesn't strip away the host like we did here manually. I agree that there should be an option to strip away the host (perhaps even by default).

Should this option be specified in the PoolManager constructor? Such as `PoolManager(strip_host=True)`.

But then when should the stripping occur? If it happens in urlopen, then should we backport the same functionality outside of PoolManager? (ie. into ConnectionPool objects.)

The request should definitely be made with the path (and the query) only, because urllib3 is a HTTP/1.1 client.
[RFC2616](http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1.2):

> To allow for transition to absoluteURIs in all requests in future versions of HTTP, all HTTP/1.1 servers MUST accept the
> absoluteURI form in requests, even though HTTP/1.1 clients will only generate them in requests to proxies.

Excellent info. Thanks :)

This isn't a bug in urllib3. It's doing exactly what it's told.

Btw, if anyone is in dire need, here's a handy basic recipe for doing "proper" url passing with redirection in urllib3:

``` python
import urlparse
import urllib3

http = urllib3.PoolManager()

def request(method, url, conn=None):
    if conn:
        # Request within the current host connection (used for redirect handling)
        if not url.startswith('/'):
            url = '/' + url
        r = conn.request(method, url, redirect=False, assert_same_host=False)
    else:
        p = urlparse.urlparse(url)
        conn = http.connection_from_host(p.hostname, p.port, p.scheme)
        r = conn.request(method, p.path, redirect=False, assert_same_host=False)

    is_redirect = r.get_redirect_location()
    if not is_redirect:
        return r
    print "Redirecting: %s" % is_redirect

    if '://' not in is_redirect:
        # Redirect to same host
        return request('GET', is_redirect, conn)

    return request('GET', is_redirect)
```


Resolving patch:
diff --git a/urllib3/util.py b/urllib3/util.py
--- a/urllib3/util.py
+++ b/urllib3/util.py
@@ -6,6 +6,7 @@
 
 
 from base64 import b64encode
+from collections import namedtuple
 
 try:
     from select import poll, POLLIN
@@ -20,94 +21,70 @@
 from .exceptions import LocationParseError
 
 
-def make_headers(keep_alive=None, accept_encoding=None, user_agent=None,
-                 basic_auth=None):
+class Url(namedtuple('Url', ['scheme', 'auth', 'host', 'port', 'path', 'query', 'fragment'])):
     """
-    Shortcuts for generating request headers.
-
-    :param keep_alive:
-        If ``True``, adds 'connection: keep-alive' header.
-
-    :param accept_encoding:
-        Can be a boolean, list, or string.
-        ``True`` translates to 'gzip,deflate'.
-        List will get joined by comma.
-        String will be used as provided.
-
-    :param user_agent:
-        String representing the user-agent you want, such as
-        "python-urllib3/0.6"
-
-    :param basic_auth:
-        Colon-separated username:password string for 'authorization: basic ...'
-        auth header.
-
-    Example: ::
-
-        >>> make_headers(keep_alive=True, user_agent="Batman/1.0")
-        {'connection': 'keep-alive', 'user-agent': 'Batman/1.0'}
-        >>> make_headers(accept_encoding=True)
-        {'accept-encoding': 'gzip,deflate'}
+    Datastructure for representing an HTTP URL. Used as a return value for
+    :func:`parse_url`.
     """
-    headers = {}
-    if accept_encoding:
-        if isinstance(accept_encoding, str):
-            pass
-        elif isinstance(accept_encoding, list):
-            accept_encoding = ','.join(accept_encoding)
-        else:
-            accept_encoding = 'gzip,deflate'
-        headers['accept-encoding'] = accept_encoding
+    slots = ()
 
-    if user_agent:
-        headers['user-agent'] = user_agent
+    def __new__(cls, scheme=None, auth=None, host=None, port=None, path=None, query=None, fragment=None):
+        return super(Url, cls).__new__(cls, scheme, auth, host, port, path, query, fragment)
 
-    if keep_alive:
-        headers['connection'] = 'keep-alive'
-
-    if basic_auth:
-        headers['authorization'] = 'Basic ' + \
-            b64encode(six.b(basic_auth)).decode('utf-8')
-
-    return headers
+    @property
+    def hostname(self):
+        """For backwards-compatibility with urlparse. We're nice like that."""
+        return self.host
 
 
 def split_first(s, delims):
     """
     Given a string and an iterable of delimiters, split on the first found
-    delimiter. Return two split parts.
+    delimiter. Return two split parts and the matched delimiter.
 
     If not found, then the first part is the full input string.
 
+    Example: ::
+
+        >>> split_first('foo/bar?baz', '?/=')
+        ('foo', 'bar?baz', '/')
+        >>> split_first('foo/bar?baz', '123')
+        ('foo/bar?baz', '', None)
+
     Scales linearly with number of delims. Not ideal for large number of delims.
     """
     min_idx = None
+    min_delim = None
     for d in delims:
         idx = s.find(d)
         if idx < 0:
             continue
 
-        if not min_idx:
+        if min_idx is None or idx < min_idx:
             min_idx = idx
-        else:
-            min_idx = min(idx, min_idx)
+            min_delim = d
 
     if min_idx < 0:
-        return s, ''
+        return s, '', None
 
-    return s[:min_idx], s[min_idx+1:]
+    return s[:min_idx], s[min_idx+1:], min_delim
 
 
-def get_host(url):
+def parse_url(url):
     """
-    Given a url, return its scheme, host and port (None if it's not there).
+    Given a url, return a parsed :class:`.Url` namedtuple. Best-effort is
+    performed to parse incomplete urls. Fields not provided will be None.
 
-    For example: ::
+    Partly backwards-compatible with :module:`urlparse`.
 
-        >>> get_host('http://google.com/mail/')
-        ('http', 'google.com', None)
-        >>> get_host('google.com:80')
-        ('http', 'google.com', 80)
+    Example: ::
+
+        >>> parse_url('http://google.com/mail/')
+        Url(scheme='http', host='google.com', port=None, path='/', ...)
+        >>> prase_url('google.com:80')
+        Url(scheme=None, host='google.com', port=80, path=None, ...)
+        >>> prase_url('/foo?bar')
+        Url(scheme=None, host=None, port=None, path='/foo', query='bar', ...)
     """
 
     # While this code has overlap with stdlib's urlparse, it is much
@@ -115,9 +92,13 @@ def get_host(url):
     # Additionally, this imeplementations does silly things to be optimal
     # on CPython.
 
-    scheme = 'http'
+    scheme = None
+    auth = None
     host = None
     port = None
+    path = None
+    fragment = None
+    query = None
 
     # Scheme
     if '://' in url:
@@ -125,11 +106,15 @@ def get_host(url):
 
     # Find the earliest Authority Terminator
     # (http://tools.ietf.org/html/rfc3986#section-3.2)
-    url, _path = split_first(url, ['/', '?', '#'])
+    url, path_, delim = split_first(url, ['/', '?', '#'])
+
+    if delim:
+        # Reassemble the path
+        path = delim + path_
 
     # Auth
     if '@' in url:
-        _auth, url = url.split('@', 1)
+        auth, url = url.split('@', 1)
 
     # IPv6
     if url and url[0] == '[':
@@ -147,10 +132,85 @@ def get_host(url):
 
         port = int(port)
 
-    elif not host:
+    elif not host and url:
         host = url
 
-    return scheme, host, port
+    if not path:
+        return Url(scheme, auth, host, port, path, query, fragment)
+
+    # Fragment
+    if '#' in path:
+        path, fragment = path.split('#', 1)
+
+    # Query
+    if '?' in path:
+        path, query = path.split('?', 1)
+
+    # Paths start with '/'
+    if path and path[0] != '/':
+        path = '/' + path
+
+    return Url(scheme, auth, host, port, path, query, fragment)
+
+
+def get_host(url):
+    """
+    Deprecated. Use :func:`parse_url` instead.
+    """
+    p = parse_url(url)
+    return p.scheme or 'http', p.hostname, p.port
+
+
+def make_headers(keep_alive=None, accept_encoding=None, user_agent=None,
+                 basic_auth=None):
+    """
+    Shortcuts for generating request headers.
+
+    :param keep_alive:
+        If ``True``, adds 'connection: keep-alive' header.
+
+    :param accept_encoding:
+        Can be a boolean, list, or string.
+        ``True`` translates to 'gzip,deflate'.
+        List will get joined by comma.
+        String will be used as provided.
+
+    :param user_agent:
+        String representing the user-agent you want, such as
+        "python-urllib3/0.6"
+
+    :param basic_auth:
+        Colon-separated username:password string for 'authorization: basic ...'
+        auth header.
+
+    Example: ::
+
+        >>> make_headers(keep_alive=True, user_agent="Batman/1.0")
+        {'connection': 'keep-alive', 'user-agent': 'Batman/1.0'}
+        >>> make_headers(accept_encoding=True)
+        {'accept-encoding': 'gzip,deflate'}
+    """
+    headers = {}
+    if accept_encoding:
+        if isinstance(accept_encoding, str):
+            pass
+        elif isinstance(accept_encoding, list):
+            accept_encoding = ','.join(accept_encoding)
+        else:
+            accept_encoding = 'gzip,deflate'
+        headers['accept-encoding'] = accept_encoding
+
+    if user_agent:
+        headers['user-agent'] = user_agent
+
+    if keep_alive:
+        headers['connection'] = 'keep-alive'
+
+    if basic_auth:
+        headers['authorization'] = 'Basic ' + \
+            b64encode(six.b(basic_auth)).decode('utf-8')
+
+    return headers
 
 
 def is_connection_dropped(conn):


Test patch:
diff --git a/test/test_util.py b/test/test_util.py
--- a/test/test_util.py
+++ b/test/test_util.py
@@ -2,7 +2,7 @@
 import logging
 
 from urllib3 import add_stderr_logger
-from urllib3.util import get_host, make_headers, split_first
+from urllib3.util import get_host, make_headers, split_first, parse_url, Url
 from urllib3.exceptions import LocationParseError
 
 
@@ -62,6 +62,25 @@ def test_invalid_host(self):
         for location in invalid_host:
             self.assertRaises(LocationParseError, get_host, location)
 
+    def test_parse_url(self):
+        url_host_map = {
+            'http://google.com/mail': Url('http', None, 'google.com', None, '/mail'),
+            'http://google.com/mail/': Url('http', None, 'google.com', None, '/mail/'),
+            'google.com/mail': Url(None, None, 'google.com', None, '/mail'),
+            'http://google.com/': Url('http', None, 'google.com', None, '/'),
+            'http://google.com': Url('http', None, 'google.com', None, None),
+            '': Url(),
+            '/': Url(path='/'),
+            '?': Url(path='', query=''),
+            '#': Url(path='', fragment=''),
+            '#?/!google.com/?foo#bar': Url(path='', fragment='?/!google.com/?foo#bar'),
+            '/foo': Url(path='/foo'),
+            '/foo?bar=baz': Url(path='/foo', query='bar=baz'),
+            '/foo?bar=baz#banana?apple/orange': Url(path='/foo', query='bar=baz', fragment='banana?apple/orange'),
+        }
+        for url, expected_host in url_host_map.items():
+            returned_host = parse_url(url)
+            self.assertEquals(returned_host, expected_host)
 
     def test_make_headers(self):
         self.assertEqual(
@@ -95,9 +114,11 @@ def test_make_headers(self):
 
     def test_split_first(self):
         test_cases = {
-            ('abcd', 'b'): ('a', 'cd'),
-            ('abcd', 'cb'): ('a', 'cd'),
-            ('abcd', ''): ('abcd', ''),
+            ('abcd', 'b'): ('a', 'cd', 'b'),
+            ('abcd', 'cb'): ('a', 'cd', 'b'),
+            ('abcd', ''): ('abcd', '', None),
+            ('abcd', 'a'): ('', 'bcd', 'a'),
+            ('abcd', 'ab'): ('', 'bcd', 'a'),
         }
         for input, expected in test_cases.iteritems():
             output = split_first(*input)