Repository: ansible/ansible

Issue:
Wait_for is failing on the first attempt and not waiting for port
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
wait_for

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.3.0.0
  config file = 
  configured module search path = Default w/o overrides
  python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
[ssh_connection]
scp_if_ssh=True
[ssh_connection]
ssh_args = -o ServerAliveInterval=30 -o ControlMaster=auto -o ControlPersist=10m

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->
Ansible being run on: ubuntu16.04
Target machine: CentOS release 6.5 (Final)

##### SUMMARY
<!--- Explain the problem briefly -->
I have a playbook where I am waiting for a port to come up. If the port isn't up on the initial check it immediately fails.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
The following is from the role where it is actually executed.
```yaml
- name: wait for api to come back up (120 minutes)
  wait_for: port=7778 timeout=7200 delay=61
```

I have recreated in isolation in port_check.yml.
```
- name: Test ansible wait_for
  hosts: all
  gather_facts: yes
  tasks:
    - name: wait for api to come back up (120 minutes)
      # I tried with `local_action: wait_for host="{{inventory_hostname}}" port=7778 timeout=7200` as well as non-local
      wait_for: host="{{inventory_hostname}}" port=7778 timeout=7200
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
The play would wait for the port to come up.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
It failed on the first attempt. After the failure I manually checked the port with nmap within the timeout of 120 minutes and it came up.
<!--- Paste verbatim command output between quotes below -->
Without verbosity, but with timestamps:
```
16:37:16 TASK [act_appliance : wait for api to come back up (120 minutes)] *******
16:38:12 ok: [gin]
16:38:13 fatal: [aregsky1]: FAILED! => {"changed": false, "elapsed": 18280, "failed": true, "msg": "Timeout when waiting for 127.0.0.1:7778"}
```
With -vvvv and timestamps:
```
16:08:05 TASK [act_appliance : wait for api to come back up (120 minutes)] *******
16:08:05 task path: /home/jenkins/workspace/trunk_1-day-regression/framework3/roles/act_appliance/tasks/patch_internal.yml:36
16:08:05 <172.27.4.100> ESTABLISH SSH CONNECTION FOR USER: root
16:08:05 <172.27.4.100> SSH: EXEC sshpass -d15 ssh -C -vvv -o ServerAliveInterval=30 -o ControlMaster=auto -o ControlPersist=10m -o StrictHostKeyChecking=no -o Port=26 -o 'IdentityFile="/home/jenkins/.ssh/brianwilliams/brianwilliams-v2-key"' -o User=root -o ConnectTimeout=10 -o ControlPath=/home/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r 172.27.4.100 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1493064485.79-247085387603234 `" && echo ansible-tmp-1493064485.79-247085387603234="` echo $HOME/.ansible/tmp/ansible-tmp-1493064485.79-247085387603234 `" ) && sleep 0'"'"''
16:08:05 <aregsky1> ESTABLISH SSH CONNECTION FOR USER: root
16:08:05 <aregsky1> SSH: EXEC sshpass -d15 ssh -C -vvv -o ServerAliveInterval=30 -o ControlMaster=auto -o ControlPersist=10m -o StrictHostKeyChecking=no -o Port=26 -o 'IdentityFile="/home/jenkins/.ssh/brianwilliams/brianwilliams-v2-key"' -o User=root -o ConnectTimeout=10 -o ControlPath=/home/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r aregsky1 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1493064485.84-212858276659267 `" && echo ansible-tmp-1493064485.84-212858276659267="` echo $HOME/.ansible/tmp/ansible-tmp-1493064485.84-212858276659267 `" ) && sleep 0'"'"''
16:08:05 <172.27.4.100> PUT /tmp/tmpExkBc2 TO /home/root/.ansible/tmp/ansible-tmp-1493064485.79-247085387603234/wait_for
16:08:05 <172.27.4.100> SSH: EXEC sshpass -d15 scp -C -vvv -o ServerAliveInterval=30 -o ControlMaster=auto -o ControlPersist=10m -o StrictHostKeyChecking=no -o Port=26 -o 'IdentityFile="/home/jenkins/.ssh/brianwilliams/brianwilliams-v2-key"' -o User=root -o ConnectTimeout=10 -o ControlPath=/home/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r /tmp/tmpExkBc2 '[172.27.4.100]:/home/root/.ansible/tmp/ansible-tmp-1493064485.79-247085387603234/wait_for'
16:08:05 <172.27.4.100> ESTABLISH SSH CONNECTION FOR USER: root
16:08:05 <172.27.4.100> SSH: EXEC sshpass -d15 ssh -C -vvv -o ServerAliveInterval=30 -o ControlMaster=auto -o ControlPersist=10m -o StrictHostKeyChecking=no -o Port=26 -o 'IdentityFile="/home/jenkins/.ssh/brianwilliams/brianwilliams-v2-key"' -o User=root -o ConnectTimeout=10 -o ControlPath=/home/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r -tt 172.27.4.100 '/bin/sh -c '"'"'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/root/.ansible/tmp/ansible-tmp-1493064485.79-247085387603234/wait_for; rm -rf "/home/root/.ansible/tmp/ansible-tmp-1493064485.79-247085387603234/" > /dev/null 2>&1 && sleep 0'"'"''
16:08:08 <aregsky1> PUT /tmp/tmpKICtHc TO /root/.ansible/tmp/ansible-tmp-1493064485.84-212858276659267/wait_for
16:08:08 <aregsky1> SSH: EXEC sshpass -d15 scp -C -vvv -o ServerAliveInterval=30 -o ControlMaster=auto -o ControlPersist=10m -o StrictHostKeyChecking=no -o Port=26 -o 'IdentityFile="/home/jenkins/.ssh/brianwilliams/brianwilliams-v2-key"' -o User=root -o ConnectTimeout=10 -o ControlPath=/home/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r /tmp/tmpKICtHc '[aregsky1]:/root/.ansible/tmp/ansible-tmp-1493064485.84-212858276659267/wait_for'
16:08:08 <aregsky1> ESTABLISH SSH CONNECTION FOR USER: root
16:08:08 <aregsky1> SSH: EXEC sshpass -d15 ssh -C -vvv -o ServerAliveInterval=30 -o ControlMaster=auto -o ControlPersist=10m -o StrictHostKeyChecking=no -o Port=26 -o 'IdentityFile="/home/jenkins/.ssh/brianwilliams/brianwilliams-v2-key"' -o User=root -o ConnectTimeout=10 -o ControlPath=/home/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r -tt aregsky1 '/bin/sh -c '"'"'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1493064485.84-212858276659267/wait_for; rm -rf "/root/.ansible/tmp/ansible-tmp-1493064485.84-212858276659267/" > /dev/null 2>&1 && sleep 0'"'"''
16:09:16 ok: [gin] => {"changed": false, "elapsed": 61, "invocation": {"module_args": {"connect_timeout": 5, "delay": 61, "exclude_hosts": null, "host": "127.0.0.1", "path": null, "port": 7778, "search_regex": null, "state": "started", "timeout": 7200}, "module_name": "wait_for"}, "path": null, "port": 7778, "search_regex": null, "state": "started"}
16:09:16 fatal: [aregsky1]: FAILED! => {"changed": false, "elapsed": 18282, "failed": true, "invocation": {"module_args": {"connect_timeout": 5, "delay": 61, "exclude_hosts": null, "host": "127.0.0.1", "path": null, "port": 7778, "search_regex": null, "state": "started", "timeout": 7200}, "module_name": "wait_for"}, "msg": "Timeout when waiting for 127.0.0.1:7778"}
```



Discussion:
cc @gregswift
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
According to the result, the `wait_for` module waited for 18280 seconds before failing.

I notice from your output, that `wait_for` did succeed for the host `gin` in which it says it elapsed 61 seconds.  Although based on your timestamps you are saying that `argesky1` failed at the same time, but the module claims to have waited for `18282` seconds, which is roughly 5 hours.

I'm not entirely sure how that is possible, since the most it should wait is 7261 seconds.
Those timestamps output are from a Jenkins job. The entire Jenkins job I watched manually and took a total of 41 minutes and 13 seconds. So nothing could've waited for 5 hours because ansible-playbook was running for less than that time.
What was the nmap command that you used, that allowed you to verify the port was listening?
If that port isn't up on the first check that task always fails. Here are the failures from the last few builds (unfortunately I wasn't running with -vvvv). All of these builds took under 1 hour yet have almost identically long elapsed times. I originally was running with "strategy: free", but changed that after the first few builds because I thought it may be causing the problem.

```
02:10:35 TASK [act_appliance : wait for api to come back up (120 minutes)] *******
02:11:31 ok: [gin]
02:11:34 fatal: [aregsky1]: FAILED! => {"changed": false, "elapsed": 18281, "failed": true, "msg": "Timeout when waiting for 127.0.0.1:7778"}
```

```
18:55:51 TASK [act_appliance : wait for api to come back up (120 minutes)] *******
18:56:47 ok: [gin]
18:56:47 fatal: [aregsky1]: FAILED! => {"changed": false, "elapsed": 18280, "failed": true, "msg": "Timeout when waiting for 127.0.0.1:7778"}
```

```
16:37:16 TASK [act_appliance : wait for api to come back up (120 minutes)] *******
16:38:12 ok: [gin]
16:38:13 fatal: [aregsky1]: FAILED! => {"changed": false, "elapsed": 18280, "failed": true, "msg": "Timeout when waiting for 127.0.0.1:7778"}
```

\# This one was run with "strategy: free" which is why you see the RETRYING TASKS for gin next to failed for aregsky1
```
15:33:50 TASK [act_appliance : wait for api to come back up (120 minutes)] *******
15:33:54 FAILED - RETRYING: TASK: act_appliance : wait for dedup to be up (345 retries left).
15:34:07 FAILED - RETRYING: TASK: act_appliance : wait for dedup to be up (344 retries left).
15:34:15 FAILED - RETRYING: TASK: act_appliance : wait for dedup to be up (343 retries left).
15:34:27 FAILED - RETRYING: TASK: act_appliance : wait for dedup to be up (342 retries left).
15:34:37 FAILED - RETRYING: TASK: act_appliance : wait for dedup to be up (341 retries left).
15:34:47 FAILED - RETRYING: TASK: act_appliance : wait for dedup to be up (340 retries left).
15:34:55 fatal: [aregsky1]: FAILED! => {"changed": false, "elapsed": 18280, "failed": true, "msg": "Timeout when waiting for 127.0.0.1:7778"}
```
@sivel 
Here was my output from the terminal command with the date beforehand to show it was within 120 minutes of the ansible task.
```
$ date
Fri Apr 21 17:06:55 EDT 2017
$ nmap -p 7778 aregsky1

Starting Nmap 7.40 ( https://nmap.org ) at 2017-04-21 17:06 EDT
Nmap scan report for aregsky1 (<omitted>)
Host is up (0.028s latency).
PORT     STATE    SERVICE
7778/tcp filtered interwise

Nmap done: 1 IP address (1 host up) scanned in 0.41 seconds
```
According to that `namp`, it could not validate that the port was open.  It lists the `STATE` as `filtered`.

> Filtered means that a firewall, filter, or other network obstacle is blocking the port so that Nmap cannot tell whether it is open or closed

It detected that the host was up, but could not verify that the port was open.  That `nmap` output seems to indicate the same thing as `wait_for`.  The port is not open.
To clarify that nmap is coming from my laptop and not the target host itself.

I get the same result for gin (from my machine) and that goes through. Even if the port wasn't viewed as open shouldn't it wait the full 120 minutes for it to open? I'll try to reproduce and check on the target machines themselves.

```
$ nmap -p 7778 gin

Starting Nmap 7.40 ( https://nmap.org ) at 2017-04-25 12:19 EDT
Nmap scan report for gin (172.27.4.100)
Host is up (0.012s latency).
PORT     STATE    SERVICE
7778/tcp filtered interwise

Nmap done: 1 IP address (1 host up) scanned in 0.41 seconds
```
Your `wait_for` is running from the machine itself, communicating with `127.0.0.1:7778`.

I feel like the fact that it is saying that 5 hours elapsed when we know that isn't true, may indicate a problem with the time on that host.

Something like this may assist in seeing if that is a problem:

```
python -c 'import datetime, time; start = datetime.datetime.now(); time.sleep(60); print (datetime.datetime.now()-start).seconds'
```

That should print out `60`

That is roughly the calculation that is done in the `wait_for` module.
Everything looks in order. I also show that the port is open when checked from the machine as localhost. It's important to note I _just_ did this so it doesn't show that the port was open within the timeframe that ansible would've been checking, but I can confirm that that port was open because I was able to get an API key in the timeframe after my last attempt.

```
[root@aregsky1 admin]# python -c 'import datetime, time; start = datetime.datetime.now(); time.sleep(60); print (datetime.datetime.now()-start).seconds'

60
[root@aregsky1 admin]# 
[root@aregsky1 admin]# nmap -p 7778 127.0.0.1

Starting Nmap 5.51 ( http://nmap.org ) at 2017-04-25 12:38 EDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000070s latency).
PORT     STATE SERVICE
7778/tcp open  interwise

Nmap done: 1 IP address (1 host up) scanned in 0.07 seconds
```
Honestly, I'm not sure.  I feel like this has to be time related.  Is it possible that the app you are starting messes with the time while it is starting?

If `wait_for` thinks that a large amount of time has elapsed, it would likely think it has waited for longer than the `timeout` and fail.  Based on the `elapsed` value, I'd have to assume that somehow, that machines time is jumping, causing this issue.
I thought that before as well. For a few builds I had the task changed to this...
```
- name: wait for api to come back up (120 minutes)
  local_action: wait_for host="{{ inventory_hostname }}" port=7778 timeout=7200 delay=61
```
... and believe I still observed the same problem. My logs from that attempt don't have verbosity high though. I will try to reproduce that failure with -vvvv on and report back.

EDIT: Now that I think of it, that may have failed because the port may have been filtered instead of open however. I don't believe it ran 120 minutes still though. Will recreate as soon as I have access to the machine.

Resolving patch:
diff --git a/lib/ansible/modules/utilities/logic/wait_for.py b/lib/ansible/modules/utilities/logic/wait_for.py
--- a/lib/ansible/modules/utilities/logic/wait_for.py
+++ b/lib/ansible/modules/utilities/logic/wait_for.py
@@ -442,7 +442,7 @@ def main():
         except:
             module.fail_json(msg="unknown active_connection_state ("+_connection_state+") defined")
 
-    start = datetime.datetime.now()
+    start = datetime.datetime.utcnow()
 
     if delay:
         time.sleep(delay)
@@ -453,7 +453,7 @@ def main():
         ### first wait for the stop condition
         end = start + datetime.timedelta(seconds=timeout)
 
-        while datetime.datetime.now() < end:
+        while datetime.datetime.utcnow() < end:
             if path:
                 try:
                     f = open(path)
@@ -470,7 +470,7 @@ def main():
             # Conditions not yet met, wait and try again
             time.sleep(params['sleep'])
         else:
-            elapsed = datetime.datetime.now() - start
+            elapsed = datetime.datetime.utcnow() - start
             if port:
                 module.fail_json(msg="Timeout when waiting for %s:%s to stop." % (host, port), elapsed=elapsed.seconds)
             elif path:
@@ -479,7 +479,7 @@ def main():
     elif state in ['started', 'present']:
         ### wait for start condition
         end = start + datetime.timedelta(seconds=timeout)
-        while datetime.datetime.now() < end:
+        while datetime.datetime.utcnow() < end:
             if path:
                 try:
                     os.stat(path)
@@ -487,7 +487,7 @@ def main():
                     e = get_exception()
                     # If anything except file not present, throw an error
                     if e.errno != 2:
-                        elapsed = datetime.datetime.now() - start
+                        elapsed = datetime.datetime.utcnow() - start
                         module.fail_json(msg="Failed to stat %s, %s" % (path, e.strerror), elapsed=elapsed.seconds)
                     # file doesn't exist yet, so continue
                 else:
@@ -506,7 +506,7 @@ def main():
                     except IOError:
                         pass
             elif port:
-                alt_connect_timeout = math.ceil(_timedelta_total_seconds(end - datetime.datetime.now()))
+                alt_connect_timeout = math.ceil(_timedelta_total_seconds(end - datetime.datetime.utcnow()))
                 try:
                     s = _create_connection(host, port, min(connect_timeout, alt_connect_timeout))
                 except:
@@ -517,8 +517,8 @@ def main():
                     if compiled_search_re:
                         data = ''
                         matched = False
-                        while datetime.datetime.now() < end:
-                            max_timeout = math.ceil(_timedelta_total_seconds(end - datetime.datetime.now()))
+                        while datetime.datetime.utcnow() < end:
+                            max_timeout = math.ceil(_timedelta_total_seconds(end - datetime.datetime.utcnow()))
                             (readable, w, e) = select.select([s], [], [], max_timeout)
                             if not readable:
                                 # No new data.  Probably means our timeout
@@ -550,7 +550,7 @@ def main():
 
         else:   # while-else
             # Timeout expired
-            elapsed = datetime.datetime.now() - start
+            elapsed = datetime.datetime.utcnow() - start
             if port:
                 if search_regex:
                     module.fail_json(msg="Timeout when waiting for search string %s in %s:%s" % (search_regex, host, port), elapsed=elapsed.seconds)
@@ -566,7 +566,7 @@ def main():
         ### wait until all active connections are gone
         end = start + datetime.timedelta(seconds=timeout)
         tcpconns = TCPConnectionInfo(module)
-        while datetime.datetime.now() < end:
+        while datetime.datetime.utcnow() < end:
             try:
                 if tcpconns.get_active_connections_count() == 0:
                     break
@@ -575,10 +575,10 @@ def main():
             # Conditions not yet met, wait and try again
             time.sleep(params['sleep'])
         else:
-            elapsed = datetime.datetime.now() - start
+            elapsed = datetime.datetime.utcnow() - start
             module.fail_json(msg="Timeout when waiting for %s:%s to drain" % (host, port), elapsed=elapsed.seconds)
 
-    elapsed = datetime.datetime.now() - start
+    elapsed = datetime.datetime.utcnow() - start
     module.exit_json(state=state, port=port, search_regex=search_regex, path=path, elapsed=elapsed.seconds)
 
 # import module snippets


Test patch: