Repository: urllib3/urllib3

Issue:
Usage of cryptography + idna breaks on wildcard certs
The changes in #930 seem to have broken with some wildcard certificates: https://stackoverflow.com/questions/39521147/why-is-urllib3-idna-complaining-about-a-wildcard-in-an-x509-cert-how-do-i-fix-i

I haven't attempted to reproduce this, but I thought recording it here would be of use.



Discussion:
So we're using idna basically because cryptography does. @reaperhulk, any insight here?

Besides that `idna` was a terrible mistake?

I need to know what version of `cryptography`. I was just able to load the cert in question without problems with both current master and version 1.5.

**title** = "cryptography"
...
**version** = "1.5"

This is running under python 2.7 on an Ubuntu 12.04 VM.

I'm running under a virtual env with the following:

setuptools==26.1.1
td-client==0.5.0
urllib3[secure]==1.17

I also have the required system packages listed in the user guide:
https://cryptography.io/en/latest/installation/#building-cryptography-on-linux

dpkg -l | grep build-essential
ii  build-essential                      11.6ubuntu6                         amd64        Informational list of build-essential packages
dpkg -l | grep libssl-dev
ii  libssl-dev:amd64                     1.0.1f-1ubuntu2.7                   amd64        Secure Sockets Layer toolkit - development files
dpkg -l | grep libffi-dev
ii  libffi-dev:amd64                     3.1~rc1+r3.0.13-12                  amd64        Foreign Function Interface library (development files)
dpkg -l | grep "\spython-dev\s"
ii  python-dev                           2.7.5-5ubuntu3                      amd64        header files and a static library for Python (default)

What happens if you run this script:

``` python
from cryptography import x509
from cryptography.hazmat.backends.openssl.backend import backend


pem = b"""-----BEGIN CERTIFICATE-----
MIIFMjCCBBqgAwIBAgIHK427mrAroDANBgkqhkiG9w0BAQsFADCBtDELMAkGA1UE
BhMCVVMxEDAOBgNVBAgTB0FyaXpvbmExEzARBgNVBAcTClNjb3R0c2RhbGUxGjAY
BgNVBAoTEUdvRGFkZHkuY29tLCBJbmMuMS0wKwYDVQQLEyRodHRwOi8vY2VydHMu
Z29kYWRkeS5jb20vcmVwb3NpdG9yeS8xMzAxBgNVBAMTKkdvIERhZGR5IFNlY3Vy
ZSBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkgLSBHMjAeFw0xNDA0MTAwNjA2MDNaFw0x
NzAxMjAxMDA3MDRaMEAxITAfBgNVBAsTGERvbWFpbiBDb250cm9sIFZhbGlkYXRl
ZDEbMBkGA1UEAwwSKi50cmVhc3VyZWRhdGEuY29tMIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAzmjm0Ogr4IPN2YQAWFcxJGzCegewnT0E3hW5tD7kgFBj
bGpLvHFdqnhd1nGhxYF/CufioUxKSgWtvrw5sL0zVyO1oeURjk5+FFwPMTKd3scW
1v7u2FJrs2uMoYL+Y2DlqWqUQikbX9U3flXcol1XSWD9MZbZLMNeZdKHMRtTi5ew
FVXQusYWG+eCovYtetutZ+qhoaGChirTKwg4lNXeID79/XuAleNQ0rUD+HgD6B1K
ZBXxomMibV48VerPQLzrp3BztXgjroSIqqtZGEcQHloMYfcn1zq7rRmhR5268RqP
sUc6cvijI537IsBvlNaByb2WW5DeLgQKraPt3v2YIQIDAQABo4IBujCCAbYwDwYD
VR0TAQH/BAUwAwEBADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDgYD
VR0PAQH/BAQDAgWgMDYGA1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly9jcmwuZ29kYWRk
eS5jb20vZ2RpZzJzMS0zOS5jcmwwUwYDVR0gBEwwSjBIBgtghkgBhv1tAQcXATA5
MDcGCCsGAQUFBwIBFitodHRwOi8vY2VydGlmaWNhdGVzLmdvZGFkZHkuY29tL3Jl
cG9zaXRvcnkvMHYGCCsGAQUFBwEBBGowaDAkBggrBgEFBQcwAYYYaHR0cDovL29j
c3AuZ29kYWRkeS5jb20vMEAGCCsGAQUFBzAChjRodHRwOi8vY2VydGlmaWNhdGVz
LmdvZGFkZHkuY29tL3JlcG9zaXRvcnkvZ2RpZzIuY3J0MB8GA1UdIwQYMBaAFEDC
vSeOzDSDMKIz1/tss/C0LIDOMC8GA1UdEQQoMCaCEioudHJlYXN1cmVkYXRhLmNv
bYIQdHJlYXN1cmVkYXRhLmNvbTAdBgNVHQ4EFgQU3oO+gpNB3WYMVywxSM2p2E6f
etMwDQYJKoZIhvcNAQELBQADggEBAJLvOqYh7tiPbT+qKyNHuuSjetfUYHWBx29A
t+hXI7Khn1waKxbgUa/QSZiUUD9C097LMhydx/Z8p6bHlkFqIfSOKkxWS736N17C
Tldrz8RSSTByp2OfCdA9Ts9ANToR9R33vR40aOQDL29NyoyTAjNPRCHKmLz4F8sE
1tlKJKysO9p6qA1+B5ZEHOY6McK4tm3L2Jhz0DgMzk3QbBh8aJ/cGfezQQoC+RT4
L3Nx0+yhclrOY9L1PEQ4hBjJCsUKd7XcKfUNx+iXTNpTfmTiTfsBmfNx+V79cu+v
RoSAetF+rkP0MTReO7DD9jybxn+2ttsC1x8HCw6Nc+NK7GqEHok=
-----END CERTIFICATE-----"""

cert = x509.load_pem_x509_certificate(pem, backend)
print(cert.subject)
print(cert.extensions.get_extension_for_class(x509.SubjectAlternativeName))
```

That should load and print info about the certificate you linked to on SO.

Running with urllib3 v1.7

```
~/repos/akamai_pusher/current_version$ ./akamai_pusher.sh                                 
<Name([<NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.11, name=organizationalUnitName)>, value=u'Domain Control Validated')>, <NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.3, name=commonName)>, value=u'*.treasuredata.com')>])>
<Extension(oid=<ObjectIdentifier(oid=2.5.29.17, name=subjectAltName)>, critical=False, value=<SubjectAlternativeName(<GeneralNames([<DNSName(value=*.treasuredata.com)>, <DNSName(value=treasuredata.com)>])>)>)>
```

Okay, I understand this bug now.

https://github.com/shazow/urllib3/blob/0319846b13ad44e1a2a6b3985369cdc15cefe914/urllib3/contrib/pyopenssl.py#L141 is way too simplistic. It assumes any name given to it will be IDNA encodable, which is untrue. In reality this code needs to look like what we do in cryptography (https://github.com/pyca/cryptography/blob/master/src/cryptography/hazmat/backends/openssl/encode_asn1.py#L365-L371).

Ok fab, I'll give that a shot and see if it helps. 


Resolving patch:
diff --git a/urllib3/contrib/pyopenssl.py b/urllib3/contrib/pyopenssl.py
--- a/urllib3/contrib/pyopenssl.py
+++ b/urllib3/contrib/pyopenssl.py
@@ -138,7 +138,19 @@ def _dnsname_to_stdlib(name):
     then on Python 3 we also need to convert to unicode via UTF-8 (the stdlib
     uses PyUnicode_FromStringAndSize on it, which decodes via UTF-8).
     """
-    name = idna.encode(name)
+    def idna_encode(name):
+        """
+        Borrowed wholesale from the Python Cryptography Project. It turns out
+        that we can't just safely call `idna.encode`: it can explode for
+        wildcard names. This avoids that problem.
+        """
+        for prefix in [u'*.', u'.']:
+            if name.startswith(prefix):
+                name = name[len(prefix):]
+                return prefix.encode('ascii') + idna.encode(name)
+        return idna.encode(name)
+
+    name = idna_encode(name)
     if sys.version_info >= (3, 0):
         name = name.decode('utf-8')
     return name


Test patch:
diff --git a/test/contrib/test_pyopenssl.py b/test/contrib/test_pyopenssl.py
--- a/test/contrib/test_pyopenssl.py
+++ b/test/contrib/test_pyopenssl.py
@@ -1,9 +1,13 @@
+# -*- coding: utf-8 -*-
+import unittest
+
 from nose.plugins.skip import SkipTest
 from urllib3.packages import six
 
 try:
     from urllib3.contrib.pyopenssl import (inject_into_urllib3,
-                                           extract_from_urllib3)
+                                           extract_from_urllib3,
+                                           _dnsname_to_stdlib)
 except ImportError as e:
     raise SkipTest('Could not import PyOpenSSL: %r' % e)
 
@@ -18,3 +22,36 @@ def setup_module():
 
 def teardown_module():
     extract_from_urllib3()
+
+
+class TestPyOpenSSLHelpers(unittest.TestCase):
+    """
+    Tests for PyOpenSSL helper functions.
+    """
+    def test_dnsname_to_stdlib_simple(self):
+        """
+        We can convert a dnsname to a native string when the domain is simple.
+        """
+        name = u"उदाहरण.परीक"
+        expected_result = 'xn--p1b6ci4b4b3a.xn--11b5bs8d'
+
+        self.assertEqual(_dnsname_to_stdlib(name), expected_result)
+
+    def test_dnsname_to_stdlib_leading_period(self):
+        """
+        If there is a . in front of the domain name we correctly encode it.
+        """
+        name = u".उदाहरण.परीक"
+        expected_result = '.xn--p1b6ci4b4b3a.xn--11b5bs8d'
+
+        self.assertEqual(_dnsname_to_stdlib(name), expected_result)
+
+    def test_dnsname_to_stdlib_leading_splat(self):
+        """
+        If there's a wildcard character in the front of the string we handle it
+        appropriately.
+        """
+        name = u"*.उदाहरण.परीक"
+        expected_result = '*.xn--p1b6ci4b4b3a.xn--11b5bs8d'
+
+        self.assertEqual(_dnsname_to_stdlib(name), expected_result)