Repository: ansible/ansible

Issue:
SNI support is broken for new versions of urllib3
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
SNI support provided by `ansible.module_utils.urls`

##### ANSIBLE VERSION
Everything since [commit 6e9c09d](https://github.com/ansible/ansible/commit/6e9c09d7f76decbbf50ee9804da69b7c25dba9f9)

##### SUMMARY
[As seen here](https://github.com/ansible/ansible/blob/77af3a68de3d054ddb3be905d96b1d07c2e89adf/lib/ansible/module_utils/urls.py#L148), Ansible currently tests for SNI support by attempting to import `ssl_wrap_socket` from `urllib3.contrib.pyopenssl`, falling back to importing from `requests.packages.urllib3.contrib.pyopenssl`.  However, version 1.19 of urllib3 [removed this function](https://github.com/shazow/urllib3/commit/1db3bba5f5bc83812810fb95803ea21971c6c052) ("As we never exported `ssl_wrap_socket` nobody should be using it."), and requests [updated its vendorized copy of urllib3 to match](https://github.com/kennethreitz/requests/commit/99fa7becf263473c7bfc1998b41c2c6c80a0f499) in version 2.12.0.  As a result, on systems using the latest versions of these packages, Ansible will falsely conclude that SNI is not supported, leading to attempts to connect to SNI-dependent sites (say, with `get_url`) to fail.

##### STEPS TO REPRODUCE
```
- name: Install the latest versions of SNI-related packages
  # Not shown here: Installing the SSL, FFI, and Python development libraries that are needed to build these modules.
  pip: name={{item}} state=latest
  with_items:
    - ndg-httpsclient
    - pyasn1
    - pyopenssl
    - requests
    - urllib3

- name: Try to download a file from an SNI-dependent site
  get_url:
    url: https://downloads.atlassian.com
    dest: /tmp/whatever
```

##### EXPECTED RESULTS
The file would be downloaded.

##### ACTUAL RESULTS
```
TASK [get_url] *****************************************************************
fatal: [sshbase]: FAILED! => {"changed": false, "failed": true, "msg": "Failed to validate the SSL certificate for downloads.atlassian.com:443. Make sure your managed systems have a valid CA certificate installed. If the website serving the url uses SNI you need python >= 2.7.9 on your managed machine or you can install the `urllib3`, `pyopenssl`, `ndg-httpsclient`, and `pyasn1` python modules to perform SNI verification in python >= 2.6. You can use validate_certs=False if you do not need to confirm the servers identity but this is unsafe and not recommended. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible"}
```


Discussion:
pinging @sivel
I've seen.  We are likely going to have to just vendor the old `ssl_wrap_socket` from urllib3 to continue using this functionality.

I don't immediately have time to work on this.
I'll be looking into this over the coming week.

Resolving patch:
diff --git a/lib/ansible/module_utils/urls.py b/lib/ansible/module_utils/urls.py
--- a/lib/ansible/module_utils/urls.py
+++ b/lib/ansible/module_utils/urls.py
@@ -145,14 +145,26 @@
 except ImportError:
     HAS_SSLCONTEXT = False
 
+# SNI Handling for python < 2.7.9 with urllib3 support
 try:
+    # urllib3>=1.15
+    HAS_URLLIB3_SSL_WRAP_SOCKET = False
     try:
-        from urllib3.contrib.pyopenssl import ssl_wrap_socket
+        from urllib3.contrib.pyopenssl import PyOpenSSLContext
     except ImportError:
-        from requests.packages.urllib3.contrib.pyopenssl import ssl_wrap_socket
-    HAS_URLLIB3_SNI_SUPPORT = True
+        from requests.packages.urllib3.contrib.pyopenssl import PyOpenSSLContext
+    HAS_URLLIB3_PYOPENSSLCONTEXT = True
 except ImportError:
-    HAS_URLLIB3_SNI_SUPPORT = False
+    # urllib3<1.15,>=1.6
+    HAS_URLLIB3_PYOPENSSLCONTEXT = False
+    try:
+        try:
+            from urllib3.contrib.pyopenssl import ssl_wrap_socket
+        except ImportError:
+            from requests.packages.urllib3.contrib.pyopenssl import ssl_wrap_socket
+        HAS_URLLIB3_SSL_WRAP_SOCKET = True
+    except ImportError:
+        pass
 
 # Select a protocol that includes all secure tls protocols
 # Exclude insecure ssl protocols if possible
@@ -359,10 +371,13 @@ class NoSSLError(SSLValidationError):
     class CustomHTTPSConnection(httplib.HTTPSConnection):
         def __init__(self, *args, **kwargs):
             httplib.HTTPSConnection.__init__(self, *args, **kwargs)
+            self.context = None
             if HAS_SSLCONTEXT:
                 self.context = create_default_context()
-                if self.cert_file:
-                    self.context.load_cert_chain(self.cert_file, self.key_file)
+            elif HAS_URLLIB3_PYOPENSSLCONTEXT:
+                self.context = PyOpenSSLContext(PROTOCOL)
+            if self.context and self.cert_file:
+                self.context.load_cert_chain(self.cert_file, self.key_file)
 
         def connect(self):
             "Connect to a host on a given (SSL) port."
@@ -380,9 +395,9 @@ def connect(self):
                 self._tunnel()
                 server_hostname = self._tunnel_host
 
-            if HAS_SSLCONTEXT:
+            if HAS_SSLCONTEXT or HAS_URLLIB3_PYOPENSSLCONTEXT:
                 self.sock = self.context.wrap_socket(sock, server_hostname=server_hostname)
-            elif HAS_URLLIB3_SNI_SUPPORT:
+            elif HAS_URLLIB3_SSL_WRAP_SOCKET:
                 self.sock = ssl_wrap_socket(sock, keyfile=self.key_file, cert_reqs=ssl.CERT_NONE, certfile=self.cert_file, ssl_version=PROTOCOL,
                         server_hostname=server_hostname)
             else:
@@ -541,8 +556,8 @@ def build_ssl_validation_error(hostname, port, paths, exc=None):
     if not HAS_SSLCONTEXT:
         msg.append('If the website serving the url uses SNI you need'
                    ' python >= 2.7.9 on your managed machine')
-        if not HAS_URLLIB3_SNI_SUPPORT:
-            msg.append('or you can install the `urllib3`, `pyopenssl`,'
+        if not HAS_URLLIB3_PYOPENSSLCONTEXT or not HAS_URLLIB3_SSL_WRAP_SOCKET:
+            msg.append('or you can install the `urllib3`, `pyOpenSSL`,'
                        ' `ndg-httpsclient`, and `pyasn1` python modules')
 
         msg.append('to perform SNI verification in python >= 2.6.')
@@ -665,7 +680,10 @@ def detect_no_proxy(self, url):
         return True
 
     def _make_context(self, to_add_ca_cert_path):
-        context = create_default_context()
+        if HAS_URLLIB3_PYOPENSSLCONTEXT:
+            context = PyOpenSSLContext(PROTOCOL)
+        else:
+            context = create_default_context()
         if to_add_ca_cert_path:
             context.load_verify_locations(to_add_ca_cert_path)
         return context
@@ -674,7 +692,7 @@ def http_request(self, req):
         tmp_ca_cert_path, to_add_ca_cert_path, paths_checked = self.get_ca_certs()
         https_proxy = os.environ.get('https_proxy')
         context = None
-        if HAS_SSLCONTEXT:
+        if HAS_SSLCONTEXT or HAS_URLLIB3_PYOPENSSLCONTEXT:
             context = self._make_context(to_add_ca_cert_path)
 
         # Detect if 'no_proxy' environment variable is set and if our URL is included
@@ -705,7 +723,7 @@ def http_request(self, req):
                     self.validate_proxy_response(connect_result)
                     if context:
                         ssl_s = context.wrap_socket(s, server_hostname=self.hostname)
-                    elif HAS_URLLIB3_SNI_SUPPORT:
+                    elif HAS_URLLIB3_SSL_WRAP_SOCKET:
                         ssl_s = ssl_wrap_socket(s, ca_certs=tmp_ca_cert_path, cert_reqs=ssl.CERT_REQUIRED, ssl_version=PROTOCOL, server_hostname=self.hostname)
                     else:
                         ssl_s = ssl.wrap_socket(s, ca_certs=tmp_ca_cert_path, cert_reqs=ssl.CERT_REQUIRED, ssl_version=PROTOCOL)
@@ -716,7 +734,7 @@ def http_request(self, req):
                 s.connect((self.hostname, self.port))
                 if context:
                     ssl_s = context.wrap_socket(s, server_hostname=self.hostname)
-                elif HAS_URLLIB3_SNI_SUPPORT:
+                elif HAS_URLLIB3_SSL_WRAP_SOCKET:
                     ssl_s = ssl_wrap_socket(s, ca_certs=tmp_ca_cert_path, cert_reqs=ssl.CERT_REQUIRED, ssl_version=PROTOCOL, server_hostname=self.hostname)
                 else:
                     ssl_s = ssl.wrap_socket(s, ca_certs=tmp_ca_cert_path, cert_reqs=ssl.CERT_REQUIRED, ssl_version=PROTOCOL)


Test patch:
diff --git a/test/integration/targets/uri/tasks/main.yml b/test/integration/targets/uri/tasks/main.yml
--- a/test/integration/targets/uri/tasks/main.yml
+++ b/test/integration/targets/uri/tasks/main.yml
@@ -235,7 +235,7 @@
 - name: install OS packages that are needed for SNI on old python
   package:
     name: "{{ item }}"
-  with_items: "{{ uri_os_packages[ansible_os_family] | default([]) }}"
+  with_items: "{{ uri_os_packages[ansible_os_family].step1 | default([]) }}"
   when: not ansible_python.has_sslcontext and not is_ubuntu_precise|bool
 
 - name: install python modules for Older Python SNI verification
@@ -259,7 +259,7 @@
       - 'sni_host in result.content'
   when: not ansible_python.has_sslcontext and not is_ubuntu_precise|bool
 
-- name: Uninstall ndg-httpsclient and urllib3
+- name: Uninstall ndg-httpsclient
   pip:
     name: "{{ item }}"
     state: absent
@@ -271,7 +271,45 @@
   package:
     name: "{{ item }}"
     state: absent
-  with_items: "{{ uri_os_packages[ansible_os_family] | default([]) }}"
+  with_items: "{{ uri_os_packages[ansible_os_family].step1 | default([]) }}"
+  when: not ansible_python.has_sslcontext and not is_ubuntu_precise|bool
+
+- name: install OS packages that are needed for building cryptography
+  package:
+    name: "{{ item }}"
+  with_items: "{{ uri_os_packages[ansible_os_family].step2 | default([]) }}"
+  when: not ansible_python.has_sslcontext and not is_ubuntu_precise|bool
+
+- name: install urllib3 and pyopenssl via pip
+  pip:
+    name: "{{ item }}"
+    state: latest
+  with_items:
+    - urllib3
+    - PyOpenSSL
+  when: not ansible_python.has_sslcontext and not is_ubuntu_precise|bool
+
+- name: Verify SNI verification succeeds on old python with pip urllib3 contrib
+  uri:
+    url: 'https://{{ sni_host }}'
+    return_content: true
+  when: not ansible_python.has_sslcontext and not is_ubuntu_precise|bool
+  register: result
+
+- name: Assert SNI verification succeeds on old python with pip urllib3 contrib
+  assert:
+    that:
+      - result|success
+      - 'sni_host in result.content'
+  when: not ansible_python.has_sslcontext and not is_ubuntu_precise|bool
+
+- name: Uninstall urllib3 and PyOpenSSL
+  pip:
+    name: "{{ item }}"
+    state: absent
+  with_items:
+    - urllib3
+    - PyOpenSSL
   when: not ansible_python.has_sslcontext and not is_ubuntu_precise|bool
 
 - name: validate the status_codes are correct
diff --git a/test/integration/targets/uri/vars/main.yml b/test/integration/targets/uri/vars/main.yml
--- a/test/integration/targets/uri/vars/main.yml
+++ b/test/integration/targets/uri/vars/main.yml
@@ -1,9 +1,19 @@
 uri_os_packages:
   RedHat:
-    - python-pyasn1
-    - pyOpenSSL
-    - python-urllib3
+    step1:
+      - python-pyasn1
+      - pyOpenSSL
+      - python-urllib3
+    step2:
+      - libffi-devel
+      - openssl-devel
+      - python-devel
   Debian:
-    - python-pyasn1
-    - python-openssl
-    - python-urllib3
+    step1:
+      - python-pyasn1
+      - python-openssl
+      - python-urllib3
+    step2:
+      - libffi-dev
+      - libssl-dev
+      - python-dev
\ No newline at end of file