Repository: ansible/ansible

Issue:
wait_for_connection fails for nodes behind bastion
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
wait_for_connection

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.4.0
  config file = 
  configured module search path = [u'/Users/smikulc/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/smikulc/virtualenvs/ansible/lib/python2.7/site-packages/ansible-2.4.0-py2.7.egg/ansible
  executable location = /Users/smikulc/virtualenvs/ansible/bin/ansible
  python version = 2.7.10 (default, Jul 30 2016, 19:40:32) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)]
```

##### CONFIGURATION
ansible.cfg
```
[defaults]
transport = ssh

[ssh_connection]
ssh_args = "-F ssh_config"
```

ssh_config
```
Host *
    User vagrant
    StrictHostKeyChecking no
    ForwardAgent yes

Host bastion
    HostName 192.168.33.20
    IdentityFile "./.vagrant/machines/bastion/virtualbox/private_key"

Host hidden
    HostName 192.168.33.21
    ProxyJump bastion
    IdentityFile "./.vagrant/machines/hidden/virtualbox/private_key"
```

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->
from macOS Sierra 10.12.3
managing ubuntu 14.04

This should not be platform specific though

##### SUMMARY

wait_for_connection currently uses the transport_test method the connection script (ssh.py, paramiko_ssh.py ...). These methods uses socket.create_socket to test if the transport is working.

Once a node is hidden behind a bastion, a socket will not be able to reach the private network where the node is.

##### STEPS TO REPRODUCE
Reboot a 

```yaml
---
- name: provision servers
  hosts: servers
  tasks:
    - ping:
    - name: shutdown node
      command: shutdown -r now
      become: yes
    - name: wait for it to come back up
      wait_for_connection:
    - ping:
```

see https://github.com/smikulcik/ansible-waitforconnection-bug-demo
for a complete demo

##### EXPECTED RESULTS

This should restart all the nodes in the servers group, wait for them to come back up, and run ping.

##### ACTUAL RESULTS
The play fails on the `wait_for_connection` module.  It never is able to connect to the
hidden node.

<!--- Paste verbatim command output between quotes below -->
```

TASK [ping] *************************************************************************************************
task path: /Users/smikulc/ansibleVagrant/play.yml:5
Using module_utils file /Users/smikulc/virtualenvs/ansible/src/ansible/lib/ansible/module_utils/basic.py
Using module_utils file /Users/smikulc/virtualenvs/ansible/src/ansible/lib/ansible/module_utils/_text.py
Using module_utils file /Users/smikulc/virtualenvs/ansible/src/ansible/lib/ansible/module_utils/pycompat24.py
Using module_utils file /Users/smikulc/virtualenvs/ansible/src/ansible/lib/ansible/module_utils/six/__init__.py
Using module_utils file /Users/smikulc/virtualenvs/ansible/src/ansible/lib/ansible/module_utils/six/_six.py
Using module file /Users/smikulc/virtualenvs/ansible/src/ansible/lib/ansible/modules/system/ping.py
<hidden> ESTABLISH SSH CONNECTION FOR USER: None
<hidden> SSH: EXEC ssh -F ssh_config -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 hidden '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<hidden> (0, '/home/vagrant\n', 'Killed by signal 1.\r\n')
<hidden> ESTABLISH SSH CONNECTION FOR USER: None
<hidden> SSH: EXEC ssh -F ssh_config -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 hidden '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/vagrant/.ansible/tmp/ansible-tmp-1492642574.04-169826551062871 `" && echo ansible-tmp-1492642574.04-169826551062871="` echo /home/vagrant/.ansible/tmp/ansible-tmp-1492642574.04-169826551062871 `" ) && sleep 0'"'"''
<hidden> (0, 'ansible-tmp-1492642574.04-169826551062871=/home/vagrant/.ansible/tmp/ansible-tmp-1492642574.04-169826551062871\n', 'Killed by signal 1.\r\n')
<hidden> PUT /var/folders/xx/7j_s8th92db91lq4wywqcjb80000gn/T/tmpkKxw4b TO /home/vagrant/.ansible/tmp/ansible-tmp-1492642574.04-169826551062871/ping.py
<hidden> SSH: EXEC sftp -b - -F ssh_config -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 '[hidden]'
<hidden> (0, 'sftp> put /var/folders/xx/7j_s8th92db91lq4wywqcjb80000gn/T/tmpkKxw4b /home/vagrant/.ansible/tmp/ansible-tmp-1492642574.04-169826551062871/ping.py\n', 'Killed by signal 1.\r\n')
<hidden> ESTABLISH SSH CONNECTION FOR USER: None
<hidden> SSH: EXEC ssh -F ssh_config -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 hidden '/bin/sh -c '"'"'chmod u+x /home/vagrant/.ansible/tmp/ansible-tmp-1492642574.04-169826551062871/ /home/vagrant/.ansible/tmp/ansible-tmp-1492642574.04-169826551062871/ping.py && sleep 0'"'"''
<hidden> (0, '', 'Killed by signal 1.\r\n')
<hidden> ESTABLISH SSH CONNECTION FOR USER: None
<hidden> SSH: EXEC ssh -F ssh_config -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -tt hidden '/bin/sh -c '"'"'/usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1492642574.04-169826551062871/ping.py; rm -rf "/home/vagrant/.ansible/tmp/ansible-tmp-1492642574.04-169826551062871/" > /dev/null 2>&1 && sleep 0'"'"''
<hidden> (0, '\r\n{"invocation": {"module_args": {"data": null}}, "changed": false, "ping": "pong"}\r\n', 'Connection to 192.168.33.21 closed.\r\nKilled by signal 1.\r\n')
ok: [hidden] => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "data": null
        }
    }, 
    "ping": "pong"
}

TASK [shutdown node] ****************************************************************************************
task path: /Users/smikulc/ansibleVagrant/play.yml:6
Using module_utils file /Users/smikulc/virtualenvs/ansible/src/ansible/lib/ansible/module_utils/basic.py
Using module_utils file /Users/smikulc/virtualenvs/ansible/src/ansible/lib/ansible/module_utils/six/__init__.py
Using module_utils file /Users/smikulc/virtualenvs/ansible/src/ansible/lib/ansible/module_utils/_text.py
Using module_utils file /Users/smikulc/virtualenvs/ansible/src/ansible/lib/ansible/module_utils/pycompat24.py
Using module_utils file /Users/smikulc/virtualenvs/ansible/src/ansible/lib/ansible/module_utils/six/_six.py
Using module file /Users/smikulc/virtualenvs/ansible/src/ansible/lib/ansible/modules/commands/command.py
<hidden> ESTABLISH SSH CONNECTION FOR USER: None
<hidden> SSH: EXEC ssh -F ssh_config -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 hidden '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<hidden> (0, '/home/vagrant\n', 'Killed by signal 1.\r\n')
<hidden> ESTABLISH SSH CONNECTION FOR USER: None
<hidden> SSH: EXEC ssh -F ssh_config -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 hidden '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/vagrant/.ansible/tmp/ansible-tmp-1492642577.57-28996063107867 `" && echo ansible-tmp-1492642577.57-28996063107867="` echo /home/vagrant/.ansible/tmp/ansible-tmp-1492642577.57-28996063107867 `" ) && sleep 0'"'"''
<hidden> (0, 'ansible-tmp-1492642577.57-28996063107867=/home/vagrant/.ansible/tmp/ansible-tmp-1492642577.57-28996063107867\n', 'Killed by signal 1.\r\n')
<hidden> PUT /var/folders/xx/7j_s8th92db91lq4wywqcjb80000gn/T/tmp5bvLZg TO /home/vagrant/.ansible/tmp/ansible-tmp-1492642577.57-28996063107867/command.py
<hidden> SSH: EXEC sftp -b - -F ssh_config -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 '[hidden]'
<hidden> (0, 'sftp> put /var/folders/xx/7j_s8th92db91lq4wywqcjb80000gn/T/tmp5bvLZg /home/vagrant/.ansible/tmp/ansible-tmp-1492642577.57-28996063107867/command.py\n', 'Killed by signal 1.\r\n')
<hidden> ESTABLISH SSH CONNECTION FOR USER: None
<hidden> SSH: EXEC ssh -F ssh_config -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 hidden '/bin/sh -c '"'"'chmod u+x /home/vagrant/.ansible/tmp/ansible-tmp-1492642577.57-28996063107867/ /home/vagrant/.ansible/tmp/ansible-tmp-1492642577.57-28996063107867/command.py && sleep 0'"'"''
<hidden> (0, '', 'Killed by signal 1.\r\n')
<hidden> ESTABLISH SSH CONNECTION FOR USER: None
<hidden> SSH: EXEC ssh -F ssh_config -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -tt hidden '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-ruutvotgsendydxqtaqwnrodarlnzlfc; /usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1492642577.57-28996063107867/command.py; rm -rf "/home/vagrant/.ansible/tmp/ansible-tmp-1492642577.57-28996063107867/" > /dev/null 2>&1'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<hidden> (0, '\x07\r\r\nBroadcast message from vagrant@vagrant-ubuntu-trusty-64\r\r\n\t(unknown) at 22:56 ...\r\r\n\r\r\n\rThe system is going down for reboot NOW!\r\r\n\r\n{"changed": true, "end": "2017-04-19 22:56:21.174975", "stdout": "", "cmd": ["shutdown", "-r", "now"], "rc": 0, "start": "2017-04-19 22:56:21.160405", "stderr": "", "delta": "0:00:00.014570", "invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": false, "_raw_params": "shutdown -r now", "removes": null, "creates": null, "chdir": null}}, "warnings": []}\r\n', 'Connection to 192.168.33.21 closed.\r\nKilled by signal 1.\r\n')
changed: [hidden] => {
    "changed": true, 
    "cmd": [
        "shutdown", 
        "-r", 
        "now"
    ], 
    "delta": "0:00:00.014570", 
    "end": "2017-04-19 22:56:21.174975", 
    "invocation": {
        "module_args": {
            "_raw_params": "shutdown -r now", 
            "_uses_shell": false, 
            "chdir": null, 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "warn": true
        }
    }, 
    "rc": 0, 
    "start": "2017-04-19 22:56:21.160405", 
    "stderr": "", 
    "stderr_lines": [], 
    "stdout": "", 
    "stdout_lines": []
}

TASK [wait for it to come back up] **************************************************************************
task path: /Users/smikulc/ansibleVagrant/play.yml:9
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
attempting transport test to hidden:22
.
.
.
```



Discussion:
cc @dagwieers
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
IMHO it would be best to just remove the call to `transport_test` here: https://github.com/ansible/ansible/blob/52959ebdc18bfe96de083b5317b32c5216abf80a/lib/ansible/plugins/action/wait_for_connection.py#L105-L106 (or at the very least make it optional via a module parameter).
The whole raison d'être for the `wait_for_connection` module is to test for the end-to-end ansible command connectivity. If you just want to see whether a TCP port is open there's `wait_for`.

In my use case I want a reliable upgrade/reboot/wait role that can be used across a range of different scenarios, both hypervisors and virtual machines, with configuration through both .ssh/config (hardcoded for some hypervisors) and `ansible_ssh_common_args` (dynamically generated for some virtual machines). Even figuring out which port on which host the SSH connection will eventually go through is hard, which is why I want `wait_for_connection` instead of `wait_for`. I believe a second `transport_test` code path cannot ever correctly give the same result as just trying to connect and letting OpenSSH figure it out.
Yes, that's the plan. I haven't been able to get around this, but with the upcoming freeze I will have to.

Resolving patch:
diff --git a/lib/ansible/plugins/connection/paramiko_ssh.py b/lib/ansible/plugins/connection/paramiko_ssh.py
--- a/lib/ansible/plugins/connection/paramiko_ssh.py
+++ b/lib/ansible/plugins/connection/paramiko_ssh.py
@@ -130,14 +130,6 @@ class Connection(ConnectionBase):
 
     transport = 'paramiko'
 
-    def transport_test(self, connect_timeout):
-        ''' Test the transport mechanism, if available '''
-        host = self._play_context.remote_addr
-        port = int(self._play_context.port or 22)
-        display.vvv("attempting transport test to %s:%s" % (host, port))
-        sock = socket.create_connection((host, port), connect_timeout)
-        sock.close()
-
     def _cache_key(self):
         return "%s__%s__" % (self._play_context.remote_addr, self._play_context.remote_user)
 
diff --git a/lib/ansible/plugins/connection/ssh.py b/lib/ansible/plugins/connection/ssh.py
--- a/lib/ansible/plugins/connection/ssh.py
+++ b/lib/ansible/plugins/connection/ssh.py
@@ -260,13 +260,6 @@ def __init__(self, *args, **kwargs):
     def _connect(self):
         return self
 
-    def transport_test(self, connect_timeout):
-        ''' Test the transport mechanism, if available '''
-        port = int(self.port or 22)
-        display.vvv("attempting transport test to %s:%s" % (self.host, port))
-        sock = socket.create_connection((self.host, port), connect_timeout)
-        sock.close()
-
     @staticmethod
     def _create_control_path(host, port, user):
         '''Make a hash for the controlpath based on con attributes'''
diff --git a/lib/ansible/plugins/connection/winrm.py b/lib/ansible/plugins/connection/winrm.py
--- a/lib/ansible/plugins/connection/winrm.py
+++ b/lib/ansible/plugins/connection/winrm.py
@@ -90,14 +90,6 @@ def __init__(self, *args, **kwargs):
 
         super(Connection, self).__init__(*args, **kwargs)
 
-    def transport_test(self, connect_timeout):
-        ''' Test the transport mechanism, if available '''
-        host = self._winrm_host
-        port = int(self._winrm_port)
-        display.vvv("attempting transport test to %s:%s" % (host, port))
-        sock = socket.create_connection((host, port), connect_timeout)
-        sock.close()
-
     def set_host_overrides(self, host, hostvars=None):
         '''
         Override WinRM-specific options from host variables.


Test patch: