Repository: ansible/ansible

Issue:
SSLValidationError in urls.py when urllib3 is installed with ansbile 2.4.0 and pyhon 2.7.13
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
`get_url`

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->

From Fedora 27 currently up-to-date as of 10/20/2017:

```
ansible 2.4.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/xenith/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.13 (default, Aug 16 2017, 12:56:26) [GCC 7.1.1 20170802 (Red Hat 7.1.1-7)]

```



##### OS / ENVIRONMENT

Fedora 27 Beta, up-to-date 2017-10-20 on updates-testing repository (bleeding edge, but not rawhide)

##### SUMMARY
<!--- Explain the problem briefly -->

When and only when `python2-urllib3` *and dependent packages are installed*, `get_url` fails 

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
---
- hosts: all
  connection: local
  become: yes

  tasks:

      - name: Remove urllib3 on Fedora
        package:
            name: python2-urllib3
            state: absent

      - name: Test get_url without urllib3
        get_url:
            url: "https://download.rocket.chat/stable"
            dest: .
            validate_certs: true

      - name: Install urllib3 on Fedora
        package:
            name: python2-urllib3
            state: installed

      - name: Test urllib3 failure
        get_url:
            url: "https://download.rocket.chat/stable"
            dest: .
            validate_certs: true
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->

 Download the file, like it does when urllib3 isn't installed 

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
```
TASK [get_url] *******************************************************************************************************************************************************
task path: /home/xenith/git/RocketChat.Server/tests/urllib_fail.yaml:13
Using module file /usr/lib/python2.7/site-packages/ansible/modules/net_tools/basics/get_url.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: xenith
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/xenith/.ansible/tmp/ansible-tmp-1508524890.54-154445362747043 `" && echo ansible-tmp-1508524890.54-154445362747043="` echo /home/xenith/.ansible/tmp/ansible-tmp-1508524890.54-154445362747043 `" ) && sleep 0'                                                           
<localhost> PUT /tmp/tmpCPcFea TO /home/xenith/.ansible/tmp/ansible-tmp-1508524890.54-154445362747043/get_url.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/xenith/.ansible/tmp/ansible-tmp-1508524890.54-154445362747043/ /home/xenith/.ansible/tmp/ansible-tmp-1508524890.54-154445362747043/get_url.py && sleep 0'                                                                                                                                      
<localhost> EXEC /bin/sh -c 'sudo -H -S  -p "[sudo via ansible, key=eggmjjdikqtvadeojfmmozbynslidgbv] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-eggmjjdikqtvadeojfmmozbynslidgbv; /usr/bin/python /home/xenith/.ansible/tmp/ansible-tmp-1508524890.54-154445362747043/get_url.py; rm -rf "/home/xenith/.ansible/tmp/ansible-tmp-1508524890.54-154445362747043/" > /dev/null 2>&1'"'"' && sleep 0'                                                                                                   
The full traceback is:
  File "/tmp/ansible_t9pKZ8/ansible_modlib.zip/ansible/module_utils/urls.py", line 1044, in fetch_url
    client_key=client_key, cookies=cookies)
  File "/tmp/ansible_t9pKZ8/ansible_modlib.zip/ansible/module_utils/urls.py", line 951, in open_url
    r = urllib_request.urlopen(*urlopen_args)
  File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 427, in open
    req = meth(req)
  File "/tmp/ansible_t9pKZ8/ansible_modlib.zip/ansible/module_utils/urls.py", line 763, in http_request
    build_ssl_validation_error(self.hostname, self.port, paths_checked, e)
  File "/tmp/ansible_t9pKZ8/ansible_modlib.zip/ansible/module_utils/urls.py", line 591, in build_ssl_validation_error
    raise SSLValidationError(' '.join(msg) % (hostname, port, ", ".join(paths)))

fatal: [localhost]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "attributes": null,
            "backup": null,
            "checksum": "",
            "client_cert": null,
            "client_key": null,
            "content": null,
            "delimiter": null,
            "dest": ".",
            "directory_mode": null,
            "follow": false,
            "force": false,
            "force_basic_auth": false,
            "group": null,
            "headers": null,
            "http_agent": "ansible-httpget",
            "mode": null,
            "owner": null,
            "regexp": null,
            "remote_src": null,
            "selevel": null,
            "serole": null,
            "setype": null,
            "seuser": null,
            "sha256sum": "",
            "src": null,
            "timeout": 10,
            "tmp_dest": null,
            "unsafe_writes": null,
            "url": "https://download.rocket.chat/stable",
            "url_password": null,
            "url_username": null,
            "use_proxy": true,
            "validate_certs": true
        }
    },
    "msg": "Failed to validate the SSL certificate for download.rocket.chat:443. Make sure your managed systems have a valid CA certificate installed. You can use validate_certs=False if you do not need to confirm the servers identity but this is unsafe and not recommended. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible. The exception msg was: (\"bad handshake: Error([('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')],)\",)."
}
        to retry, use: --limit @/home/xenith/git/RocketChat.Server/tests/urllib_fail.retry

PLAY RECAP ***********************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=1

Additional info:

Has package versions: https://github.com/ansible/ansible/issues/18528#issuecomment-338284006
https://github.com/ansible/ansible/issues/25402#issuecomment-338286531


Discussion:
cc @jpmens
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
(linking to the comment on other bug that spawned this) https://github.com/ansible/ansible/issues/18528#issuecomment-338284006

And it's original source, just for ref. https://github.com/ansible/ansible-modules-core/issues/1716#issuecomment-338271789
(quoting a formatted version of the error msg here for reference)

> Failed to validate the SSL certificate for download.rocket.chat:443. Make sure your managed systems have a valid CA certificate installed. You can use validate_certs=False if you do not need to confirm the servers identity but this is unsafe and not recommended. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible. 
> 
> The exception msg was:
>   ("bad handshake: Error([('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')],)",).
Yep. Thanks for tracking down the previous issues, that I admittedly sprawled over a few tickets that I thought were the same thing. When I wrote this one I decided to keep it clean and not involve "the story" because I now had a very basic test case that was easily reproducible. 

Also, thanks for taking the time to triage. I'm available for testing if necessary.

Resolving patch:
diff --git a/lib/ansible/module_utils/urls.py b/lib/ansible/module_utils/urls.py
--- a/lib/ansible/module_utils/urls.py
+++ b/lib/ansible/module_utils/urls.py
@@ -700,10 +700,13 @@ def detect_no_proxy(self, url):
         return True
 
     def _make_context(self, to_add_ca_cert_path):
-        if HAS_URLLIB3_PYOPENSSLCONTEXT:
+        if HAS_SSLCONTEXT:
+            context = create_default_context()
+        elif HAS_URLLIB3_PYOPENSSLCONTEXT:
             context = PyOpenSSLContext(PROTOCOL)
         else:
-            context = create_default_context()
+            raise NotImplementedError('Host libraries are too old to support creating an sslcontext')
+
         if to_add_ca_cert_path:
             context.load_verify_locations(to_add_ca_cert_path)
         return context
@@ -712,8 +715,11 @@ def http_request(self, req):
         tmp_ca_cert_path, to_add_ca_cert_path, paths_checked = self.get_ca_certs()
         https_proxy = os.environ.get('https_proxy')
         context = None
-        if HAS_SSLCONTEXT or HAS_URLLIB3_PYOPENSSLCONTEXT:
+        try:
             context = self._make_context(to_add_ca_cert_path)
+        except Exception:
+            # We'll make do with no context below
+            pass
 
         # Detect if 'no_proxy' environment variable is set and if our URL is included
         use_proxy = self.detect_no_proxy(req.get_full_url())


Test patch: