Repository: urllib3/urllib3

Issue:
tests are stuck when running `make test`
I tried to use google's public dns as suggested by @sigmavirus24, but it didn't help.

``` shell
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.0.17/24 brd 192.168.0.255 scope global enp5s0
       valid_lft forever preferred_lft forever
    inet6 fe80::beae:c5ff:fe65:dd71/64 scope link 
       valid_lft forever preferred_lft forever
$ cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
```

strace:

``` shell
$ strace -p 12676
Process 12676 attached
futex(0xbf8dd0, FUTEX_WAIT_PRIVATE, 0, NULL
```

This is gentoo, wicd and python 2.7. I think I had the same problem with Fedora and NetworkManager.

Also, last couple of lines from output:

```
test_oldapi (test.with_dummyserver.test_proxy_poolmanager.TestHTTPProxyManager) ... ok
test_proxy_conn_fail (test.with_dummyserver.test_proxy_poolmanager.TestHTTPProxyManager) ... ok
test_proxy_pooling (test.with_dummyserver.test_proxy_poolmanager.TestHTTPProxyManager) ... ok
test_proxy_pooling_ext (test.with_dummyserver.test_proxy_poolmanager.TestHTTPProxyManager) ... ok
test_proxy_verified (test.with_dummyserver.test_proxy_poolmanager.TestHTTPProxyManager) ... ok
test_redirect (test.with_dummyserver.test_proxy_poolmanager.TestHTTPProxyManager) ... ok
test_multi_setcookie (test.with_dummyserver.test_socketlevel.TestCookies) ... Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/home/tt/dev/urllib3/dummyserver/server.py", line 76, in run
    self.server = self._start_server()
  File "/home/tt/dev/urllib3/dummyserver/server.py", line 63, in _start_server
    sock.bind((self.host, 0))
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: getsockaddrarg: bad family
```



Discussion:
That's perplexing, bad family feels like a very weird error to raise. I wonder if the 'bind port 0' is causing a problem here.

looks like an IPv6 issue. Applying this, I can run the whole suite.

``` diff
--- a/dummyserver/server.py
+++ b/dummyserver/server.py
@@ -57,7 +57,7 @@ class SocketServerThread(threading.Thread):
         self.ready_event = ready_event

     def _start_server(self):
-        sock = socket.socket(socket.AF_INET6)
+        sock = socket.socket(socket.AF_INET)
         if sys.platform != 'win32':
             sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
         sock.bind((self.host, 0))
```

One test fails still:

```
ERROR: test_source_address (test.with_dummyserver.test_connectionpool.TestConnectionPool)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tt/dev/urllib3/test/with_dummyserver/test_connectionpool.py", line 603, in test_source_address
    r = pool.request('GET', '/source_address')
  File "/home/tt/dev/urllib3/urllib3/request.py", line 68, in request
    **urlopen_kw)
  File "/home/tt/dev/urllib3/urllib3/request.py", line 81, in request_encode_url
    return self.urlopen(method, url, **urlopen_kw)
  File "/home/tt/dev/urllib3/urllib3/connectionpool.py", line 597, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/tt/dev/urllib3/urllib3/util/retry.py", line 222, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/tt/dev/urllib3/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/home/tt/dev/urllib3/urllib3/connectionpool.py", line 349, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python2.7/httplib.py", line 1001, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1035, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 850, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 812, in send
    self.connect()
  File "/home/tt/dev/urllib3/urllib3/connection.py", line 155, in connect
    conn = self._new_conn()
  File "/home/tt/dev/urllib3/urllib3/connection.py", line 134, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/home/tt/dev/urllib3/urllib3/util/connection.py", line 88, in create_connection
    raise err
ProtocolError: ('Connection aborted.', gaierror(-9, 'Address family for hostname not supported'))
-------------------- >> begin captured logging << --------------------
urllib3.util.retry: DEBUG: Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
urllib3.connectionpool: INFO: Starting new HTTP connection (1): localhost
```

So it would seem that whatever machine you're using doesn't support IPv6. That's very surprising. (That or the version of python you're running was compiled without that support.)

@sigmavirus24 you are right. Python is compiled without IPv6 support.

Is this bug then? Shouldn't those tests just skip?

These tests are written expecting IPv6 support since most modern systems should have that. If @shazow and @Lukasa feel that supporting Pythons that don't support IPv6 is something they want to do, then yes this is a bug. In other words, not my call. ;)

Ideally we should skip IPv6 dependent tests on platforms that don't support it. Seems like we can test that pretty easily by attempting to bind an IPv6 socket. Might be worth doing that?

/me quietly adds "ipv6" to $USE in /etc/make.conf, as a fellow Gentoo user.

Umm, yea skipping with a warning makes sense.

Any reason we don't want to do `socket.socket()` and let it choose the default family, rather than forcing ipv6? I guess for coverage purposes? (Should that be yet another tox target?)

Btw, can confirm tests pass on Py3 on Gentoo with ipv6 enabled.

(Also amused that @TomasTomecek working at Redhat is running Gentoo. :P)

>  Seems like we can test that pretty easily by attempting to bind an IPv6 socket.

Even easier, we can check [socket.has_ipv6](https://docs.python.org/2/library/socket.html#socket.has_ipv6).

At this point I'd lean to using ipv4 with a warning if ipv6 is not available, with a warning.


Resolving patch:
diff --git a/dummyserver/server.py b/dummyserver/server.py
--- a/dummyserver/server.py
+++ b/dummyserver/server.py
@@ -13,6 +13,9 @@
 import sys
 import threading
 import socket
+import warnings
+
+from urllib3.exceptions import HTTPWarning
 
 from tornado.platform.auto import set_close_exec
 import tornado.wsgi
@@ -40,6 +43,11 @@
 # Different types of servers we have:
 
 
+class NoIPv6Warning(HTTPWarning):
+    "IPv6 is not available"
+    pass
+
+
 class SocketServerThread(threading.Thread):
     """
     :param socket_handler: Callable which receives a socket argument for one
@@ -57,7 +65,12 @@ def __init__(self, socket_handler, host='localhost', port=8081,
         self.ready_event = ready_event
 
     def _start_server(self):
-        sock = socket.socket(socket.AF_INET6)
+        if socket.has_ipv6:
+            sock = socket.socket(socket.AF_INET6)
+        else:
+            warnings.warn("No IPv6 support. Falling back to IPv4.",
+                          NoIPv6Warning)
+            sock = socket.socket(socket.AF_INET)
         if sys.platform != 'win32':
             sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
         sock.bind((self.host, 0))


Test patch:
diff --git a/test/__init__.py b/test/__init__.py
--- a/test/__init__.py
+++ b/test/__init__.py
@@ -13,7 +13,8 @@
 # Reset. SO suggests this hostname
 TARPIT_HOST = '10.255.255.1'
 
-VALID_SOURCE_ADDRESSES = [('::1', 0), ('127.0.0.1', 0)]
+# (Arguments for socket, is it IPv6 address?)
+VALID_SOURCE_ADDRESSES = [(('::1', 0), True), (('127.0.0.1', 0), False)]
 # RFC 5737: 192.0.2.0/24 is for testing only.
 # RFC 3849: 2001:db8::/32 is for documentation only.
 INVALID_SOURCE_ADDRESSES = [('192.0.2.255', 0), ('2001:db8::1', 0)]
diff --git a/test/with_dummyserver/test_connectionpool.py b/test/with_dummyserver/test_connectionpool.py
--- a/test/with_dummyserver/test_connectionpool.py
+++ b/test/with_dummyserver/test_connectionpool.py
@@ -4,6 +4,7 @@
 import sys
 import unittest
 import time
+import warnings
 
 import mock
 
@@ -35,6 +36,7 @@
 
 import tornado
 from dummyserver.testcase import HTTPDummyServerTestCase
+from dummyserver.server import NoIPv6Warning
 
 from nose.tools import timed
 
@@ -597,7 +599,11 @@ def test_dns_error(self):
         self.assertRaises(MaxRetryError, pool.request, 'GET', '/test', retries=2)
 
     def test_source_address(self):
-        for addr in VALID_SOURCE_ADDRESSES:
+        for addr, is_ipv6 in VALID_SOURCE_ADDRESSES:
+            if is_ipv6 and not socket.has_ipv6:
+                warnings.warn("No IPv6 support: skipping.",
+                              NoIPv6Warning)
+                continue
             pool = HTTPConnectionPool(self.host, self.port,
                     source_address=addr, retries=False)
             r = pool.request('GET', '/source_address')