Repository: ansible/ansible-modules-core

Issue:
wait_for with port & search_regex can run forever
##### Issue Type: Bug Report
##### Ansible Version: 1.9.2 (probably others)
##### Ansible Configuration: stock
##### Environment: N/A
##### Summary:

wait_for won't always honour its timeout values. In particular, if you connect to a socket which dribbles data out at a rate that exceeds the specified timeout, you'll wind up in a loop reading data forever.

The problem lies with this inner loop in the wait_for code:

```
elif port:
    ...
             while 1:
                 data += s.recv(1024)
                 if not data:
                     break
                 elif re.search(search_regex, data, re.MULTILINE):
                     matched = True
                     break
    ...
```

There may be additional issues around the close timeout in the same body of code; I found that killing the nc side of the connection left the ansible playbook permanently in close_wait. This is down to not checking the result of s.recv directly for an EOF: the whole thing should be as follows:

```
elif port:
    ...
             while datetime.datetime.now() < end:
                 response = s.recv(1024)
                 if not response:
                     break
                 data += response
                 elif re.search(search_regex, data, re.MULTILINE):
                     matched = True
                     break
    ...
```
##### Steps To Reproduce:

An example ansible task:

```
- hosts: localhost
  connection: local
  tasks:
    - name: test wait_for
      wait_for:
        timeout: 5
        delay: 1
        connect_timeout: 5
        host: localhost
        port: 9500
        search_regex: 'is synced'
        state: present
```

Run this with the following running in another window:

```
(while : ; do echo n; sleep 2; done) | nc -l 9500
```
##### Expected Results:

A failure after 5 seconds (in the example above).
##### Actual Results:

The playbook will hang indefinitely; either reading data or, should you kill the server process (nc in the example above), stuck in CLOSE_WAIT and looping forever.



Discussion:
cc: @jhoekx @jarv @AnderEnder


Resolving patch:
diff --git a/utilities/logic/wait_for.py b/utilities/logic/wait_for.py
--- a/utilities/logic/wait_for.py
+++ b/utilities/logic/wait_for.py
@@ -18,12 +18,14 @@
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 
-import socket
+import binascii
 import datetime
-import time
-import sys
+import math
 import re
-import binascii
+import select
+import socket
+import sys
+import time
 
 HAS_PSUTIL = False
 try:
@@ -349,6 +351,10 @@ def main():
     state = params['state']
     path = params['path']
     search_regex = params['search_regex']
+    if search_regex is not None:
+        compiled_search_re = re.compile(search_regex, re.MULTILINE)
+    else:
+        compiled_search_re = None
 
     if port and path:
         module.fail_json(msg="port and path parameter can not both be passed to wait_for")
@@ -404,55 +410,72 @@ def main():
             if path:
                 try:
                     os.stat(path)
-                    if search_regex:
-                        try:
-                            f = open(path)
-                            try:
-                                if re.search(search_regex, f.read(), re.MULTILINE):
-                                    break
-                                else:
-                                    time.sleep(1)
-                            finally:
-                                f.close()
-                        except IOError:
-                            time.sleep(1)
-                            pass
-                    else:
-                        break
                 except OSError, e:
-                    # File not present
-                    if e.errno == 2:
-                        time.sleep(1)
-                    else:
+                    # If anything except file not present, throw an error
+                    if e.errno != 2:
                         elapsed = datetime.datetime.now() - start
                         module.fail_json(msg="Failed to stat %s, %s" % (path, e.strerror), elapsed=elapsed.seconds)
+                    # file doesn't exist yet, so continue
+                else:
+                    # File exists.  Are there additional things to check?
+                    if not compiled_search_re:
+                        # nope, succeed!
+                        break
+                    try:
+                        f = open(path)
+                        try:
+                            if re.search(compiled_search_re, f.read()):
+                                # String found, success!
+                                break
+                        finally:
+                            f.close()
+                    except IOError:
+                        pass
             elif port:
+                alt_connect_timeout = math.ceil((end - datetime.datetime.now()).total_seconds())
                 try:
-                    s = _create_connection( (host, port), connect_timeout)
-                    if search_regex:
+                    s = _create_connection((host, port), min(connect_timeout, alt_connect_timeout))
+                except:
+                    # Failed to connect by connect_timeout. wait and try again
+                    pass
+                else:
+                    # Connected -- are there additional conditions?
+                    if compiled_search_re:
                         data = ''
                         matched = False
-                        while 1:
-                            data += s.recv(1024)
-                            if not data:
+                        while datetime.datetime.now() < end:
+                            max_timeout = math.ceil((end - datetime.datetime.now()).total_seconds())
+                            (readable, w, e) = select.select([s], [], [], max_timeout)
+                            if not readable:
+                                # No new data.  Probably means our timeout
+                                # expired
+                                continue
+                            response = s.recv(1024)
+                            if not response:
+                                # Server shutdown
                                 break
-                            elif re.search(search_regex, data, re.MULTILINE):
+                            data += response
+                            if re.search(compiled_search_re, data):
                                 matched = True
                                 break
+
+                        # Shutdown the client socket
+                        s.shutdown(socket.SHUT_RDWR)
+                        s.close()
                         if matched:
-                            s.shutdown(socket.SHUT_RDWR)
-                            s.close()
+                            # Found our string, success!
                             break
                     else:
+                        # Connection established, success!
                         s.shutdown(socket.SHUT_RDWR)
                         s.close()
                         break
-                except:
-                    time.sleep(1)
-                    pass
-            else:
-                time.sleep(1)
-        else:
+
+            # Conditions not yet met, wait and try again
+            time.sleep(1)
+
+        else:   # while-else
+            # Timeout expired
             elapsed = datetime.datetime.now() - start
             if port:
                 if search_regex:
@@ -485,4 +508,5 @@ def main():
 
 # import module snippets
 from ansible.module_utils.basic import *
-main()
+if __name__ == '__main__':
+    main()


Test patch: