Repository: ansible/ansible

Issue:
Wrong error message when unable to connect via ssh
## Version:

ansible-playbook 2.0.0 (devel 8f77dd1cf2) last updated 2015/10/26 11:46:26 (GMT -400)
  lib/ansible/modules/core: (detached HEAD 06f301b05b) last updated 2015/10/26 11:46:28 (GMT -400)
  lib/ansible/modules/extras: (detached HEAD 405c3cb2c7) last updated 2015/10/26 11:46:30 (GMT -400)
## Issue:

When the server is not online, the error message is confusing.
## What happens now:

Get the error message:

```
fatal: [10.10.67.157]: FAILED! => {"failed": true, "msg": "ERROR! Timeout (10s) waiting for privilege escalation prompt: "}
```
## What should happen:

```
fatal: [10.10.67.157]: FAILED! => {"failed": true, "msg": "ERROR! Unable to connect to server "}
```



Discussion:
Interesting, with both the adhoc and playbook commands, I get the following for a VM that's offline:

<pre>
    "msg": "ERROR! SSH encountered an unknown error during the connection. We recommend you re-run the command using -vvvv, which will enable SSH debugging output to help diagnose the issue", 
</pre>

Have you changed any settings in your ansible.cfg which might affect the SSH connection params?

This is my ansible.cfg settings:

```
[defaults]
host_key_checking = False
jinja2_extensions = jinja2.ext.do
hash_behaviour=merge
[ssh_connection]
pipelining = True
control_path = /tmp/ansible-ssh-%%h-%%p-%%r
```

@evanccnyc I haven't been able to reproduce this problem either (I ran an ansible command with `-sKU` to enable sudo, but the failure came on the first connection where it just creates the ansible directory without sudo), but I suspect the fix is along the lines of the patch at https://gist.github.com/amenonsen/a8131bf563524f694186

Could you please try it and see?

I did a clean reinstall recently rm -Rf and then git clone and it seems to have gone away. Ill close and reopen if I see it again.

@amenonsen I've seen this error with the beta (v2.0.0-0.4.beta2) and the latest (c64ac90). I get the error even with the patch.

Can this issue be reopened, or should I open a new issue?

My ansible.cfg is:

``` ini
[defaults]
retry_files_save_path = $HOME/.ansible-retries
inventory = ./inventory/
vault_password_file = $HOME/.ansible_vault
remote_user = ubuntu
forks = 25
host_key_checking = False
gathering = smart

[privilege_escalation]
become = True

[ssh_connection]
control_path = %(directory)s/%%h-%%r
pipelining = True
```

@leedm777 Could you perhaps send me output of a problematic run with `-vvvv` and `ANSIBLE_DEBUG=1` please? I'd like to understand what's happening. Also please explain what "unable to connect" means in your specific case. If possible, run `ssh -vvv` (or better still, the exact command that ansible generates, which you'll see in `-vvv` output) on the host and send me that output too.

I'm able to duplicate the issue by adding a bad route to the host, so that it's unreachable.

``` bash
$ sudo route add -host X.X.X.X -iface lo0
```

``` bash
$ ssh -vvv X.X.X.X
OpenSSH_6.9p1, LibreSSL 2.1.7
debug1: Reading configuration data /Users/dlee/.ssh/config
debug1: /Users/dlee/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 20: Applying options for *
debug1: /etc/ssh/ssh_config line 102: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to X.X.X.X [X.X.X.X] port 22.
# wait about 75 seconds
debug1: connect to address X.X.X.X port 22: Operation timed out
ssh: connect to host X.X.X.X port 22: Operation timed out
```

Full playbook output is on [pastebin](http://pastebin.com/e5gad1Gm)

``` bash
$ ansible-playbook --limit X.X.X.X -vvvv -- deploy.yml
# snip
    14 1447106272.11609: executing the command /bin/sh -c 'sudo -H -n -S -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-snhljbvnujrniptfyjnawvhxroahegzd; LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python'"'"'' through the connection
<X.X.X.X> ESTABLISH SSH CONNECTION FOR USER: ubuntu
<X.X.X.X> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<X.X.X.X> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<X.X.X.X> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<X.X.X.X> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=ubuntu)
<X.X.X.X> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<X.X.X.X> SSH: PlayContext set ssh_common_args: ()
<X.X.X.X> SSH: PlayContext set ssh_extra_args: ()
<X.X.X.X> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/root/.ansible/cp/%h-%r)
<X.X.X.X> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/%h-%r X.X.X.X /bin/sh -c 'sudo -H -n -S -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-snhljbvnujrniptfyjnawvhxroahegzd; LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python'"'"''
    14 1447106272.12518: Initial state: awaiting_escalation: BECOME-SUCCESS-snhljbvnujrniptfyjnawvhxroahegzd
    14 1447106272.12924: stderr chunk (state=1):
>>>OpenSSH_6.7p1 Debian-5, OpenSSL 1.0.1k 8 Jan 2015
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
<<<
    14 1447106272.12966: stderr chunk (state=1):
>>>debug1: auto-mux: Trying existing master
debug1: Control socket "/root/.ansible/cp/X.X.X.X-ubuntu" does not exist
debug2: ssh_connect: needpriv 0
debug1: Connecting to X.X.X.X [X.X.X.X] port 22.
debug2: fd 3 setting O_NONBLOCK
<<<
# about a 10 second delay here
    14 1447106282.14050: done running TaskExecutor() for X.X.X.X/TASK: setup 
    14 1447106282.14081: sending task result
    14 1447106282.14197: done sending task result
    39 1447106282.15155: worker 1 has data to read
    39 1447106282.15443: got a result from worker 1: <ansible.executor.task_result.TaskResult object at 0x7fd3d110aa50>
    39 1447106282.15452: sending result: [u'host_task_failed', u'<ansible.executor.task_result.TaskResult object at 0x7fd3d110aa50>']
    39 1447106282.15563: done sending result
     1 1447106282.16237: got result from result worker: [u'host_task_failed', u'<ansible.executor.task_result.TaskResult object at 0x7fd3d1021e90>']
     1 1447106282.16248: marking X.X.X.X as failed
fatal: [X.X.X.X]: FAILED! => {"failed": true, "msg": "ERROR! Timeout (10s) waiting for privilege escalation prompt: "}

#snip
PLAY RECAP *********************************************************************
X.X.X.X              : ok=0    changed=0    unreachable=0    failed=1   
```

OK, so it actually _is_ a timeout waiting for the escalation prompt then. :-) I don't think we can do better here unless we wait for the ssh process to timeout and die, and I don't think we want to do that.

While you are _technically_ correct, the error message is pretty unhelpful in terms of trying to debug what's happening when things go wrong.

You could set `-o ConnectTimeout=X` to reduce the SSH connection timeout.

It would make more sense to me if the privilege escalation prompt timer started once SSH was connected. If not that, it should at least be set to a value greater than the SSH connect timeout.

Thoughts?

Huh. Looks like the `ConnectTimeout` is already set to 10. I guess it's racing with the select timeout, and the select timeout is normally winning.


Resolving patch:
diff --git a/lib/ansible/plugins/connection/ssh.py b/lib/ansible/plugins/connection/ssh.py
--- a/lib/ansible/plugins/connection/ssh.py
+++ b/lib/ansible/plugins/connection/ssh.py
@@ -392,7 +392,10 @@ def _run(self, cmd, in_data, sudoable=True):
             become_error=False, become_nopasswd_error=False
         )
 
-        timeout = self._play_context.timeout
+        # select timeout should be longer than the connect timeout, otherwise
+        # they will race each other when we can't connect, and the connect
+        # timeout usually fails
+        timeout = 2 + self._play_context.timeout
         rpipes = [p.stdout, p.stderr]
         for fd in rpipes:
             fcntl.fcntl(fd, fcntl.F_SETFL, fcntl.fcntl(fd, fcntl.F_GETFL) | os.O_NONBLOCK)


Test patch: