Repository: ansible/ansible

Issue:
Command Module: Unexpected failure during module execution.
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
Encountered a python error with stack trace when running a playbook via vagrant from a Ubuntu 16.04 host to a Ubuntu 18.04 guest

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
command

##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
ansible 2.5.4
  config file = /home/omn/projects/ifi/ifi2/ansible.cfg
  configured module search path = [u'/home/omn/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]

```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->

running a playbook via vagrant 2.1.1 from a Ubuntu 16.04 host to a Ubuntu 18.04 guest

##### STEPS TO REPRODUCE
I cannot reproduce the issue, re-running the playbook with no changes does not reproduce the error.

##### EXPECTED RESULTS
Expected to see the task complete successfully or fail. 

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
Instead got a python error and stack trace
<!--- Paste verbatim command output between quotes below -->
```
The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 138, in run
    res = self._execute()
  File "/usr/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 561, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/dist-packages/ansible/plugins/action/command.py", line 24, in run
    results = merge_hash(results, self._execute_module(task_vars=task_vars, wrap_async=wrap_async))
  File "/usr/lib/python2.7/dist-packages/ansible/plugins/action/__init__.py", line 810, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/usr/lib/python2.7/dist-packages/ansible/plugins/action/__init__.py", line 917, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/dist-packages/ansible/plugins/connection/ssh.py", line 984, in exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/dist-packages/ansible/plugins/connection/ssh.py", line 258, in wrapped
    return_tuple = func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/ansible/plugins/connection/ssh.py", line 878, in _run
    return self._bare_run(cmd, in_data, sudoable=sudoable, checkrc=checkrc)
  File "/usr/lib/python2.7/dist-packages/ansible/plugins/connection/ssh.py", line 715, in _bare_run
    events = selector.select(timeout)
  File "/usr/lib/python2.7/dist-packages/ansible/compat/selectors/_selectors2.py", line 464, in select
    maxevents=max_events)
  File "/usr/lib/python2.7/dist-packages/ansible/compat/selectors/_selectors2.py", line 139, in _syscall_wrapper
    raise OSError(errno=errno.ETIMEDOUT)
TypeError: exceptions.OSError does not take keyword arguments

fatal: [staging.bizlock.net]: FAILED! => {
    "msg": "Unexpected failure during module execution.", 
    "stdout": ""
}
```



Discussion:
Files identified in the description:
* [lib/ansible/modules/commands/command.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/commands/command.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
This appears to be a bug in some vendor'ed third party code and not related to any particular module.
This is indeed a bug in the upstream code that we have vendored and has been resolved upstream in https://github.com/SethMichaelLarson/selectors2/commit/f0c2c6c66cfa7662bc52beaf4e2d65adfa25e189

However, that fix is in the 2.0.1 release, and we have vendored 1.1.0

We would need to evaluate whether updating our vendored copy to this version is safe.

I'd recommend against us trying to backport any changes into our vendored copy.

cc @abadger 
I've taken a look at the 2.0.1 release.  I'm a little conflicted as upstream's 2.x series looks like it's no longer a backport of the python stdlib selectors module.

* If we backport we stay closer to the stdlib code which we are using this as a substitute for (which is good as we want to use the stdlib code when it's available so the bugs encountered in the bundled code should be more similar).
* If we update we stay closer to the selectors2 code.  That's good as we're unlikely to write our own selectors module and so staying close to selectors2 allows us to keep updating with his code which may handle more cases than the stdlib selectors code.

My evaluation of the changes selectors2 2.x has made is that the present changes are good, adding some additional support for more platform select/poll implementations but I'm not infallible, the code could be introducing more bugs or abusing the stdlib architecture.

Resolving patch:
diff --git a/lib/ansible/compat/selectors/__init__.py b/lib/ansible/compat/selectors/__init__.py
--- a/lib/ansible/compat/selectors/__init__.py
+++ b/lib/ansible/compat/selectors/__init__.py
@@ -25,6 +25,14 @@
 '''
 # The following makes it easier for us to script updates of the bundled code
 _BUNDLED_METADATA = {"pypi_name": "selectors2", "version": "1.1.0"}
+# Added these bugfix commits from 2.1.0:
+# * https://github.com/SethMichaelLarson/selectors2/commit/3bd74f2033363b606e1e849528ccaa76f5067590
+#   Wrap kqueue.control so that timeout is a keyword arg
+# * https://github.com/SethMichaelLarson/selectors2/commit/6f6a26f42086d8aab273b30be492beecb373646b
+#   Fix formatting of the kqueue.control patch for pylint
+# * https://github.com/SethMichaelLarson/selectors2/commit/f0c2c6c66cfa7662bc52beaf4e2d65adfa25e189
+#   Fix use of OSError exception for py3 and use the wrapper of kqueue.control so retries of
+#   interrupted syscalls work with kqueue
 
 import os.path
 import sys
diff --git a/lib/ansible/compat/selectors/_selectors2.py b/lib/ansible/compat/selectors/_selectors2.py
--- a/lib/ansible/compat/selectors/_selectors2.py
+++ b/lib/ansible/compat/selectors/_selectors2.py
@@ -136,7 +136,7 @@ def _syscall_wrapper(func, recalc_timeout, *args, **kwargs):
                     if expires is not None:
                         current_time = monotonic()
                         if current_time > expires:
-                            raise OSError(errno=errno.ETIMEDOUT)
+                            raise OSError(errno.ETIMEDOUT)
                         if recalc_timeout:
                             if "timeout" in kwargs:
                                 kwargs["timeout"] = expires - current_time
@@ -340,7 +340,7 @@ def select(self, timeout=None):
             timeout = None if timeout is None else max(timeout, 0.0)
             ready = []
             r, w, _ = _syscall_wrapper(self._select, True, self._readers,
-                                       self._writers, timeout)
+                                       self._writers, timeout=timeout)
             r = set(r)
             w = set(w)
             for fd in r | w:
@@ -561,14 +561,14 @@ def register(self, fileobj, events, data=None):
                                        select.KQ_FILTER_READ,
                                        select.KQ_EV_ADD)
 
-                _syscall_wrapper(self._kqueue.control, False, [kevent], 0, 0)
+                _syscall_wrapper(self._wrap_control, False, [kevent], 0, 0)
 
             if events & EVENT_WRITE:
                 kevent = select.kevent(key.fd,
                                        select.KQ_FILTER_WRITE,
                                        select.KQ_EV_ADD)
 
-                _syscall_wrapper(self._kqueue.control, False, [kevent], 0, 0)
+                _syscall_wrapper(self._wrap_control, False, [kevent], 0, 0)
 
             return key
 
@@ -579,7 +579,7 @@ def unregister(self, fileobj):
                                        select.KQ_FILTER_READ,
                                        select.KQ_EV_DELETE)
                 try:
-                    _syscall_wrapper(self._kqueue.control, False, [kevent], 0, 0)
+                    _syscall_wrapper(self._wrap_control, False, [kevent], 0, 0)
                 except SelectorError:
                     pass
             if key.events & EVENT_WRITE:
@@ -587,7 +587,7 @@ def unregister(self, fileobj):
                                        select.KQ_FILTER_WRITE,
                                        select.KQ_EV_DELETE)
                 try:
-                    _syscall_wrapper(self._kqueue.control, False, [kevent], 0, 0)
+                    _syscall_wrapper(self._wrap_control, False, [kevent], 0, 0)
                 except SelectorError:
                     pass
 
@@ -600,8 +600,8 @@ def select(self, timeout=None):
             max_events = len(self._fd_to_key) * 2
             ready_fds = {}
 
-            kevent_list = _syscall_wrapper(self._kqueue.control, True,
-                                           None, max_events, timeout)
+            kevent_list = _syscall_wrapper(self._wrap_control, True,
+                                           None, max_events, timeout=timeout)
 
             for kevent in kevent_list:
                 fd = kevent.ident
@@ -626,6 +626,9 @@ def close(self):
             self._kqueue.close()
             super(KqueueSelector, self).close()
 
+        def _wrap_control(self, changelist, max_events, timeout):
+            return self._kqueue.control(changelist, max_events, timeout)
+
     __all__.append('KqueueSelector')
 
 


Test patch: