Issue 1:
ISSUE TYPE

Bug Report

COMPONENT NAME

get_url module

ANSIBLE VERSION

Ansible 2.4.1.0 (sorry, don't have the exact ansible --version output at hand right now)

"ansible_python": {
"executable": "/usr/bin/python",
"has_sslcontext": true,
"type": "CPython",
"version": {
"major": 2,
"micro": 12,
"minor": 7,
"releaselevel": "final",
"serial": 0
},
"version_info": [
2,
7,
12,
"final",
0
]
}

CONFIGURATION

nothing changed

OS / ENVIRONMENT

Ubuntu 16.04 in a Vagrant Box running on VirtualBox on Windows. The script runs during Vagrant's Ansible provisioning.

SUMMARY

Ansible's get_url module fails to retrieve data when these conditions are met:

A proxy is defined (through https_proxy environment variable in my case)

The target url is secure i.e. uses https://

Certificate validation is active (which it is by default)

It then says "Connection to proxy failed". In my case there's a cascade of proxies with a local cntlm talking to the corporate squid proxy.

The problem does not occur when you use validate_certs: no. In Ansible the issues seems to arise from

ansible/lib/ansible/module_utils/urls.py

Line 602
in
e946c4b

CONNECT_COMMAND = "CONNECT %s:%s HTTP/1.0\r\nConnection: close\r\n"

That is part of the SSLValidationHandler in that file. And that is only used when using validate_certs: yes. Please see below for details and packet captures.

The problem is pretty much the same as request/request#2766 in the requests library. Similar problems have been reported in Ansible before (see #10941) but that discussion is a bit misleading and eventually led nowhere.

STEPS TO REPRODUCE

You'd need a web proxy like squid. Make sure to export https_proxy=http://whatever-adress-your-proxy-may-have.net:3128 to point to that proxy.

- name: download nodejs script to add repository
get_url:
url: https://deb.nodesource.com/setup_7.x
dest: /tmp/nodejs_setup_7.x
mode: 0755

EXPECTED RESULTS

get_url should connect through the proxy, verify the certificate, fetch the URL, and save the file.

ACTUAL RESULTS

The file is not retrieved. An error is written and the below traceback is printed:

File "/tmp/ansible_akRnA9/ansible_modlib.zip/ansible/module_utils/urls.py", line 1044, in fetch_url
client_key=client_key, cookies=cookies)
File "/tmp/ansible_akRnA9/ansible_modlib.zip/ansible/module_utils/urls.py", line 951, in open_url
r = urllib_request.urlopen(*urlopen_args)
File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 427, in open
req = meth(req)
File "/tmp/ansible_akRnA9/ansible_modlib.zip/ansible/module_utils/urls.py", line 740, in http_request
self.validate_proxy_response(connect_result)
File "/tmp/ansible_akRnA9/ansible_modlib.zip/ansible/module_utils/urls.py", line 682, in validate_proxy_response
raise ProxyError('Connection to proxy failed')

I used tcpdump to compare get_url with

validate_certs: yes

validate_certs: no

and wget

This is the result in packet capture snippets:

validate yes

CONNECT deb.nodesource.com:443 HTTP/1.0
Connection: close

HTTP/1.1 407 Proxy Authentication Required
Server: squid/3.5.27
Mime-Version: 1.0
Date: Wed, 08 Nov 2017 07:25:21 GMT

validate no

CONNECT deb.nodesource.com:443 HTTP/1.0

HTTP/1.1 200 Connection established
Connection: close

wget

CONNECT deb.nodesource.com:443 HTTP/1.1
User-Agent: Wget/1.17.1 (linux-gnu)
Host: deb.nodesource.com:443

HTTP/1.1 200 Connection established
Connection: close

Notice that when using certificate validation get_url sets the Connection: close header. This causes squid to kill the connection right away. There seems to be a violation of rfc 2616.

I wouldn't know what good it does anyway to set the header the way it's done. So I suggest to just get rid of setting the header. After all, it's not set when validation is off.

Issue 2:
I want to assert that certain EC2 security groups are present.

- name: Create EC2 SSH security group.
ec2_group:
name: SSH
description: Allows SSH access.
state: present
region: "{{ aws_region }}"
rules:
- proto: tcp
from_port: 22
to_port: 22
cidr_ip: 0.0.0.0/0

It works the first time:

$ ansible-playbook -i hosts provision.yml

PLAY [localhost] **************************************************************

TASK: [ec2 | Create EC2 SSH security group.] **********************************
changed: [localhost]

PLAY RECAP ********************************************************************
localhost : ok=1 changed=1 unreachable=0 failed=0

But when running it again, it fails:

$ ansible-playbook -i hosts provision.yml

PLAY [localhost] **************************************************************

TASK: [ec2 | Create EC2 SSH security group.] **********************************
failed: [localhost] => {"failed": true, "item": "", "parsed": false}
invalid output was: Traceback (most recent call last):
File "/home/danilo/.ansible/tmp/ansible-1389865891.33-218682145301746/ec2_group", line 1313, in <module>
main()
File "/home/danilo/.ansible/tmp/ansible-1389865891.33-218682145301746/ec2_group", line 192, in main
group = ec2.create_security_group(name, description, vpc_id=vpc_id)
File "/usr/lib/python2.7/site-packages/boto/ec2/connection.py", line 2948, in create_security_group
SecurityGroup, verb='POST')
File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1151, in get_object
raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidGroup.Duplicate</Code><Message>The security group 'SSH' already exists for VPC 'vpc-ab1e15c9'</Message></Error></Errors><RequestID>76a8ffae-6e58-4ab4-aacb-fff220476816</RequestID></Response>

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
to retry, use: --limit @/home/danilo/provision.retry

localhost : ok=0 changed=0 unreachable=0 failed=1

Is there a way to make this module idempotent? When a rule with the same name already exists, the rules should be compared. If they're the same, it should set the status to "not changed". (Otherwise an error could be thrown.)