Repository: ansible/ansible

Issue:
httplib.HTTPSConnection used even when HAS_SSL False
##### Issue Type:
- Bug Report
##### Ansible Version:

```
[jed@jed ~]$ ansible --version
ansible 1.9.2
  configured module search path = None
```
##### Ansible Configuration:
- Stock.
##### Environment:
- Any host OS.
- Managed OS: CoreOS on GCE.
##### Summary:

When Google vendors CoreOS for their platform, they include a Python interpreter for daemons owned by them since we do not ship one in the OS. Conveniently for me, I can then use this interpreter to manage Ansible, and this works fine for a pretty significant deployment at present.

The one roadblock is that [this subclass](https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/urls.py#L314) blindly uses `httplib.HTTPSConnection`, even though `import ssl` was guarded further up. Google hasn't included OpenSSL in its build of Python, so `import ssl` fails (which sets `HAS_SSL` to `False`), which then implies that `httplib.HTTPSConnection` is nonexistent. There are obviously a couple solutions here, including Google recompiling Python with OpenSSL, but it seems like since work was put in to do the `HAS_SSL` guard it should apply to the whole file.

I'm trying to `get_url` a cleartext URL, so I don't need the extra TLS-centric functionality. I began writing a patch to guard this extra functionality under `HAS_SSL` but I quickly lost the plot and broke Ansible inexplicably, so I'm just filing an issue instead.
##### Steps To Reproduce:
1. Launch CoreOS on Google Compute Engine.
2. Attempt to manage it with `ansible_python_interpreter=/usr/share/oem/python/bin/python2.7`.
3. Use a `get_url` play.
##### Expected Results:
- Successful file download.
##### Actual Results:

```
TASK: [kubelet | Install Kubelet] *********************************************
failed: [core1.coreos.systems] => {"failed": true, "parsed": false}
BECOME-SUCCESS-xugzhlvqovsgnxebwkpyiptibpqnkrhn
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /Users/jed/.ssh/config
debug1: /Users/jed/.ssh/config line 6: Applying options for *.coreos.systems
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: auto-mux: Trying existing master
debug1: mux_client_request_session: master session id: 2
Traceback (most recent call last):
  File "<stdin>", line 2201, in <module>
AttributeError: 'module' object has no attribute 'HTTPSConnection'
```



Discussion:
Just realized you don't need GCE to confirm, probably, if you can just make `import ssl` fail. So:

##### Steps to Reproduce
1. Force `import ssl` to fail by hiding the module temporarily or building without OpenSSL.
2. As above.

hi @jedsmith I have the same problem fetching a tar.gz via https using get_url module (ansible 1.9.2)

I can see this error:

`"Request failed: <urlopen error [Errno 8] _ssl.c:492: EOF occurred in violation of protocol>"`

part of the playbook:

get_url: url=**https**://somehost/somefile.tar.gz dest=/tmp

IIRC, the public module ssl wa added after the support for httpsconnection in httplib.  So we're not actually checking for whether python was built against openssl.  We're checking if the installed python is either recent enough to have the ssl module or has an addon package which provides the ssl module (See the python-2.4 documentation which has httpsconnection https://docs.python.org/release/2.4.4/lib/module-httplib.html  but not an ssl module).

I'm not sure if we want to have a separate guard for HTTPSConnection or if we would want to tell people to use the command module with curl or wget instead.  Looking into how hard it would be to code a separate guard now....

I think it will be moderate to hard to implement this.  I have a few other things to look into in urls.py so I'll try to squeeze this in as well but I can't absolutely promise it will get fixed in this round.  As you saw when you tried to add it to the HAS_SSL guard, the code is not all straightforward.

Most people on CoreOS install their own python.  command module with curl and wget is another workaround.  


Resolving patch:
diff --git a/lib/ansible/module_utils/urls.py b/lib/ansible/module_utils/urls.py
--- a/lib/ansible/module_utils/urls.py
+++ b/lib/ansible/module_utils/urls.py
@@ -310,36 +310,45 @@ class NoSSLError(SSLValidationError):
     """Needed to connect to an HTTPS url but no ssl library available to verify the certificate"""
     pass
 
+# Some environments (Google Compute Engine's CoreOS deploys) do not compile
+# against openssl and thus do not have any HTTPS support.
+CustomHTTPSConnection = CustomHTTPSHandler = None
+if hasattr(httplib, 'HTTPSConnection') and hasattr(urllib2, 'HTTPSHandler'):
+    class CustomHTTPSConnection(httplib.HTTPSConnection):
+        def __init__(self, *args, **kwargs):
+            httplib.HTTPSConnection.__init__(self, *args, **kwargs)
+            if HAS_SSLCONTEXT:
+                self.context = create_default_context()
+                if self.cert_file:
+                    self.context.load_cert_chain(self.cert_file, self.key_file)
+
+        def connect(self):
+            "Connect to a host on a given (SSL) port."
+
+            if hasattr(self, 'source_address'):
+                sock = socket.create_connection((self.host, self.port), self.timeout, self.source_address)
+            else:
+                sock = socket.create_connection((self.host, self.port), self.timeout)
+
+            server_hostname = self.host
+            # Note: self._tunnel_host is not available on py < 2.6 but this code
+            # isn't used on py < 2.6 (lack of create_connection)
+            if self._tunnel_host:
+                self.sock = sock
+                self._tunnel()
+                server_hostname = self._tunnel_host
+
+            if HAS_SSLCONTEXT:
+                self.sock = self.context.wrap_socket(sock, server_hostname=server_hostname)
+            else:
+                self.sock = ssl.wrap_socket(sock, keyfile=self.key_file, certfile=self.cert_file, ssl_version=PROTOCOL)
 
-class CustomHTTPSConnection(httplib.HTTPSConnection):
-    def __init__(self, *args, **kwargs):
-        httplib.HTTPSConnection.__init__(self, *args, **kwargs)
-        if HAS_SSLCONTEXT:
-            self.context = create_default_context()
-            if self.cert_file:
-                self.context.load_cert_chain(self.cert_file, self.key_file)
-
-    def connect(self):
-        "Connect to a host on a given (SSL) port."
-
-        if hasattr(self, 'source_address'):
-            sock = socket.create_connection((self.host, self.port), self.timeout, self.source_address)
-        else:
-            sock = socket.create_connection((self.host, self.port), self.timeout)
-        if self._tunnel_host:
-            self.sock = sock
-            self._tunnel()
-        if HAS_SSLCONTEXT:
-            self.sock = self.context.wrap_socket(sock, server_hostname=self.host)
-        else:
-            self.sock = ssl.wrap_socket(sock, keyfile=self.key_file, certfile=self.cert_file, ssl_version=PROTOCOL)
-
-class CustomHTTPSHandler(urllib2.HTTPSHandler):
+    class CustomHTTPSHandler(urllib2.HTTPSHandler):
 
-    def https_open(self, req):
-        return self.do_open(CustomHTTPSConnection, req)
+        def https_open(self, req):
+            return self.do_open(CustomHTTPSConnection, req)
 
-    https_request = urllib2.AbstractHTTPHandler.do_request_
+        https_request = urllib2.AbstractHTTPHandler.do_request_
 
 def generic_urlparse(parts):
     '''
@@ -373,7 +382,10 @@ def generic_urlparse(parts):
         # get the username, password, etc.
         try:
             netloc_re = re.compile(r'^((?:\w)+(?::(?:\w)+)?@)?([A-Za-z0-9.-]+)(:\d+)?$')
-            (auth, hostname, port) = netloc_re.match(parts[1])
+            match = netloc_re.match(parts[1])
+            auth = match.group(1)
+            hostname = match.group(2)
+            port = match.group(3)
             if port:
                 # the capture group for the port will include the ':',
                 # so remove it and convert the port to an integer
@@ -383,6 +395,8 @@ def generic_urlparse(parts):
                 # and then split it up based on the first ':' found
                 auth = auth[:-1]
                 username, password = auth.split(':', 1)
+            else:
+                username = password = None
             generic_parts['username'] = username
             generic_parts['password'] = password
             generic_parts['hostname'] = hostname
@@ -390,7 +404,7 @@ def generic_urlparse(parts):
         except:
             generic_parts['username'] = None
             generic_parts['password'] = None
-            generic_parts['hostname'] = None
+            generic_parts['hostname'] = parts[1]
             generic_parts['port']     = None
     return generic_parts
 
@@ -532,7 +546,8 @@ def http_request(self, req):
             s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
             if https_proxy:
                 proxy_parts = generic_urlparse(urlparse.urlparse(https_proxy))
-                s.connect((proxy_parts.get('hostname'), proxy_parts.get('port')))
+                port = proxy_parts.get('port') or 443
+                s.connect((proxy_parts.get('hostname'), port))
                 if proxy_parts.get('scheme') == 'http':
                     s.sendall(self.CONNECT_COMMAND % (self.hostname, self.port))
                     if proxy_parts.get('username'):
@@ -542,7 +557,7 @@ def http_request(self, req):
                     connect_result = s.recv(4096)
                     self.validate_proxy_response(connect_result)
                     if context:
-                        ssl_s = context.wrap_socket(s, server_hostname=proxy_parts.get('hostname'))
+                        ssl_s = context.wrap_socket(s, server_hostname=self.hostname)
                     else:
                         ssl_s = ssl.wrap_socket(s, ca_certs=tmp_ca_cert_path, cert_reqs=ssl.CERT_REQUIRED, ssl_version=PROTOCOL)
                         match_hostname(ssl_s.getpeercert(), self.hostname)
@@ -661,8 +676,9 @@ def open_url(url, data=None, headers=None, method=None, use_proxy=True,
         handlers.append(proxyhandler)
 
     # pre-2.6 versions of python cannot use the custom https
-    # handler, since the socket class is lacking this method
-    if hasattr(socket, 'create_connection'):
+    # handler, since the socket class is lacking create_connection.
+    # Some python builds lack HTTPS support.
+    if hasattr(socket, 'create_connection') and CustomHTTPSHandler:
         handlers.append(CustomHTTPSHandler)
 
     opener = urllib2.build_opener(*handlers)


Test patch: