Repository: ansible/ansible

Issue:
'MODULE FAILURE' message when notification of password expiration
<!--- Verify first that your issue/request is not already reported in GitHub -->

This was originally submitted as https://github.com/ansible/ansible-modules-core/issues/3439 which has been closed.  @jctanner requested I post this issue in this repo instead.
##### ISSUE TYPE

<!--- Pick one below and delete the rest: -->
- Bug Report
##### COMPONENT NAME

setup, file, likely others (those were the two that I tested)
##### ANSIBLE VERSION

```
ansible 2.0.0.2
Tower 2.4.4
```
##### CONFIGURATION

pipelining=true
timeout=30
##### OS / ENVIRONMENT

Control Host: CentOS Linux release 7.0.1406 (Core)
Target Host: Centos 6.3
##### SUMMARY

When the password of the remote user you are connecting to has a password that is going to expire, you receive a non-descript MODULE FAILURE error

**_Password expiration notice triggering this:_**
Warning: Your password will expire in 5 days on Tue 19 Apr 2016 06:58:59 PM UTC
##### STEPS TO REPRODUCE
- Use an account with a password that is about to expire. In our case, the run was to sudo as root, but not sure that makes a difference.
- Execute a playbook against a target host with that remote user

```
- hosts: webservers
  become: yes

  gather_facts: true


  tasks:
    - name: create dir as someone
      file:
        path: /tmp/water
        state: directory

```

<!--- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
- /tmp/water created as root (assuming user password is not actually expired yet and user has sudo as root rights, which the user did)
- I would expect the playbook to fail if the password had actually expired, but in this case it was simply a warning and the play should have succeeded
- I would also expect a more descriptive error message.  MODULE FAILURE is too generic and did not point to a specific cause

When we did the password change, the error went away.
##### ACTUAL RESULTS

<!--- What actually happened? If possible run with high verbosity (-vvvv) -->

Renamed some specifics to protect the innocent

```
Using /etc/ansible/ansible.cfg as config file
SSH password: 
SUDO password[defaults to SSH password]: 
1 plays in test_download.yml

PLAY ***************************************************************************

TASK [Ensure /tmp/water folder exists] ********************************************
task path: /var/lib/awx/projects/_1063__application_engineering/test_download.yml:8
<server1.mycompany.com> ESTABLISH SSH CONNECTION FOR USER: userid123
<server2.mycompany.com> ESTABLISH SSH CONNECTION FOR USER: userid123
<server2.mycompany.com> SSH: EXEC sshpass -d25 ssh -C -q -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=userid123 -o ConnectTimeout=30 -o ControlPath=/tmp/ansible_tower_vkmTPd/cp/ansible-ssh-%h-%p-%r server2.mycompany.com '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=ciwwtampnqikadsfwwkxwszmozbasafv] password: " -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-ciwwtampnqikadsfwwkxwszmozbasafv; LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python'"'"'"'"'"'"'"'"''"'"''
<server1.mycompany.com> SSH: EXEC sshpass -d24 ssh -C -q -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=userid123 -o ConnectTimeout=30 -o ControlPath=/tmp/ansible_tower_vkmTPd/cp/ansible-ssh-%h-%p-%r server1.mycompany.com '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=guaywsgdbbtlrxouxrraebsustvesxbk] password: " -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-guaywsgdbbtlrxouxrraebsustvesxbk; LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python'"'"'"'"'"'"'"'"''"'"''
fatal: [server2.mycompany.com]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_name": "file"}, "module_stderr": "", "module_stdout": "{\"group\": \"root\", \"uid\": 0, \"changed\": true, \"owner\": \"root\", \"state\": \"directory\", \"gid\": 0, \"mode\": \"0755\", \"invocation\": {\"module_args\": {\"directory_mode\": null, \"force\": false, \"remote_src\": null, \"path\": \"/tmp/water/\", \"owner\": null, \"follow\": false, \"group\": null, \"state\": \"directory\", \"content\": null, \"serole\": null, \"diff_peek\": null, \"setype\": null, \"selevel\": null, \"original_basename\": null, \"regexp\": null, \"validate\": null, \"src\": null, \"seuser\": null, \"recurse\": false, \"delimiter\": null, \"mode\": null, \"backup\": null}}, \"path\": \"/tmp/water/\", \"size\": 4096}\nWarning: Your password will expire in 5 days on Tue 19 Apr 2016 06:58:59 PM UTC\n", "msg": "MODULE FAILURE", "parsed": false}
fatal: [server1.mycompany.com]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_name": "file"}, "module_stderr": "", "module_stdout": "{\"group\": \"root\", \"uid\": 0, \"changed\": true, \"owner\": \"root\", \"state\": \"directory\", \"gid\": 0, \"mode\": \"0755\", \"invocation\": {\"module_args\": {\"directory_mode\": null, \"force\": false, \"remote_src\": null, \"path\": \"/tmp/water/\", \"owner\": null, \"follow\": false, \"group\": null, \"state\": \"directory\", \"content\": null, \"serole\": null, \"diff_peek\": null, \"setype\": null, \"selevel\": null, \"original_basename\": null, \"regexp\": null, \"validate\": null, \"src\": null, \"seuser\": null, \"recurse\": false, \"delimiter\": null, \"mode\": null, \"backup\": null}}, \"path\": \"/tmp/water/\", \"size\": 4096}\nWarning: Your password will expire in 5 days on Tue 19 Apr 2016 06:58:59 PM UTC\n", "msg": "MODULE FAILURE", "parsed": false}

PLAY RECAP *********************************************************************
server1.mycompany.com : ok=0    changed=0    unreachable=0    failed=1   
server2.mycompany.com : ok=0    changed=0    unreachable=0    failed=1  
```



Discussion:
I just tested this and it seems to be a variant on this: http://docs.ansible.com/ansible/intro_configuration.html#pipelining

You can test again with either requiretty turned off in /etc/sudoers or pipelining off to verify whether we're seeing the same thing.

i don't think this is fixable but I'll see if there's a way to better detect the error and give a better message (it might not be possible... It all depends on what information we get back from ssh in this circumstance).

Hmm... so thinking about this more, i'm not sure whether this is the same problem or not.  the output I get is different from yours when I have pipelining and requiretty.  So the question is if you turn pipelining off, do things start working?

I'm unable to replicate your error condition (getting the same output as you do).  Everything seems to work fine unless I have the requiretty and pipelining combo (which gives a different error).

Manual login looks like this:

```
$ ssh rhel6                                                     (15:12:05)
Warning: your password will expire in 98 days
Last login: Wed Apr 27 14:44:25 2016 from 192.168.122.1
$ chage -l badger                                                                          (15:12:32)
Last password change                                    : Jan 01, 2015
Password expires                                        : Aug 03, 2016
Password inactive                                       : never
Account expires                                         : Jan 20, 11761191
Minimum number of days between password change          : 0
Maximum number of days between password change          : 580
Number of days of warning before password expires       : 100
```

Working Ansible run:

```
$ ansible-playbook 15601.yml -i inven -K                        (15:14:18)
SUDO password: 

PLAY [pass-test] ***************************************************************

TASK [create dir as someone] ***************************************************
changed: [pass-test]

PLAY RECAP *********************************************************************
pass-test                  : ok=1    changed=1    unreachable=0    failed=0
$ ansible --version                                             (15:17:34)
ansible 2.0.0.2 (detached HEAD 7de237c5a1) last updated 2016/04/27 15:17:29 (GMT -700)
  lib/ansible/modules/core: (stable-2.0 4b42c6ad59) last updated 2016/04/26 10:34:02 (GMT -700)
  lib/ansible/modules/extras: (stable-2.0 4eb177e545) last updated 2016/04/26 10:34:17 (GMT -700)
  config file = /srv/ansible/troubleshooting/ansible.cfg
  configured module search path = Default w/o overrides
```

Is there anything else you can tell me about your setup that might help to reproduce this?

Pipelining was on and the hosts did not have requiretty set.  Are you just testing with ansible core or are you throwing Tower into the mix? This was occurring when running in Tower.

I was just testing with ansible core.  I don't think tower will make a difference.  But thinking about how you could get the warning after the module's output, maybe there's pam or syslog configuration that causes this.  Do you happen to know if you're changing RHEL/Centos6's defaults for either of those in some way?  I'd like to be able to reproduce so that I know when I've fixed this.

Failing that, I think I can propose fixes for you to test but that's obviously more work for you that I'd like to avoid.

Ah ha!  @alikins thinks it might be that using krb5 might cause the Warning message to be printed to stdout differently so that could create the case where the module prints the json results before the warning message is emitted.


Resolving patch:
diff --git a/lib/ansible/plugins/action/__init__.py b/lib/ansible/plugins/action/__init__.py
--- a/lib/ansible/plugins/action/__init__.py
+++ b/lib/ansible/plugins/action/__init__.py
@@ -324,7 +324,7 @@ def _fixup_perms(self, remote_path, remote_user, execute=False, recursive=True):
             # contain a path to a tmp dir but doesn't know if it needs to
             # exist or not.  If there's no path, then there's no need for us
             # to do work
-            self._display.debug('_fixup_perms called with remote_path==None.  Sure this is correct?')
+            display.debug('_fixup_perms called with remote_path==None.  Sure this is correct?')
             return remote_path
 
         if self._play_context.become and self._play_context.become_user not in ('root', remote_user):
@@ -360,7 +360,7 @@ def _fixup_perms(self, remote_path, remote_user, execute=False, recursive=True):
                     if C.ALLOW_WORLD_READABLE_TMPFILES:
                         # fs acls failed -- do things this insecure way only
                         # if the user opted in in the config file
-                        self._display.warning('Using world-readable permissions for temporary files Ansible needs to create when becoming an unprivileged user which may be insecure. For information on securing this, see https://docs.ansible.com/ansible/become.html#becoming-an-unprivileged-user')
+                        display.warning('Using world-readable permissions for temporary files Ansible needs to create when becoming an unprivileged user which may be insecure. For information on securing this, see https://docs.ansible.com/ansible/become.html#becoming-an-unprivileged-user')
                         res = self._remote_chmod('a+%s' % mode, remote_path, recursive=recursive)
                         if res['rc'] != 0:
                             raise AnsibleError('Failed to set file mode on remote files (rc: {0}, err: {1})'.format(res['rc'], res['stderr']))
@@ -480,21 +480,49 @@ def _remote_expand_user(self, path):
         else:
             return initial_fragment
 
-    def _filter_leading_non_json_lines(self, data):
+    @staticmethod
+    def _filter_non_json_lines(data):
         '''
         Used to avoid random output from SSH at the top of JSON output, like messages from
         tcagetattr, or where dropbear spews MOTD on every single command (which is nuts).
 
-        need to filter anything which starts not with '{', '[', ', '=' or is an empty line.
-        filter only leading lines since multiline JSON is valid.
+        need to filter anything which does not start with '{', '[', or is an empty line.
+        Have to be careful how we filter trailing junk as multiline JSON is valid.
         '''
-        idx = 0
-        for line in data.splitlines(True):
-            if line.startswith((u'{', u'[')):
+        # Filter initial junk
+        lines = data.splitlines()
+        for start, line in enumerate(lines):
+            line = line.strip()
+            if line.startswith(u'{'):
+                endchar = u'}'
                 break
-            idx = idx + len(line)
+            elif line.startswith(u'['):
+                endchar = u']'
+                break
+        else:
+            display.debug('No start of json char found')
+            raise ValueError('No start of json char found')
+
+        # Filter trailing junk
+        lines = lines[start:]
+        lines.reverse()
+        for end, line in enumerate(lines):
+            if line.strip().endswith(endchar):
+                break
+        else:
+            display.debug('No end of json char found')
+            raise ValueError('No end of json char found')
+
+        if end < len(lines) - 1:
+            # Trailing junk is uncommon and can point to things the user might
+            # want to change.  So print a warning if we find any
+            trailing_junk = lines[:end]
+            trailing_junk.reverse()
+            display.warning('Module invocation had junk after the JSON data: %s' % '\n'.join(trailing_junk))
 
-        return data[idx:]
+        lines = lines[end:]
+        lines.reverse()
+        return '\n'.join(lines)
 
     def _strip_success_message(self, data):
         '''
@@ -539,7 +567,7 @@ def _execute_module(self, module_name=None, module_args=None, tmp=None, task_var
         module_args['_ansible_diff'] = self._play_context.diff
 
         # let module know our verbosity
-        module_args['_ansible_verbosity'] = self._display.verbosity
+        module_args['_ansible_verbosity'] = display.verbosity
 
         (module_style, shebang, module_data) = self._configure_module(module_name=module_name, module_args=module_args, task_vars=task_vars)
         if not shebang:
@@ -627,7 +655,7 @@ def _execute_module(self, module_name=None, module_args=None, tmp=None, task_var
 
     def _parse_returned_data(self, res):
         try:
-            data = json.loads(self._filter_leading_non_json_lines(res.get('stdout', u'')))
+            data = json.loads(self._filter_non_json_lines(res.get('stdout', u'')))
         except ValueError:
             # not valid json, lets try to capture error
             data = dict(failed=True, parsed=False)


Test patch:
diff --git a/test/units/plugins/action/test_action.py b/test/units/plugins/action/test_action.py
--- a/test/units/plugins/action/test_action.py
+++ b/test/units/plugins/action/test_action.py
@@ -33,6 +33,8 @@
 except ImportError:
     import __builtin__ as builtins
 
+from nose.tools import eq_, raises
+
 from ansible.release import __version__ as ansible_version
 from ansible import constants as C
 from ansible.compat.six import text_type
@@ -630,3 +632,42 @@ def test_action_base_sudo_only_if_user_differs(self):
             play_context.make_become_cmd.assert_called_once_with("ECHO SAME", executable=None)
         finally:
             C.BECOME_ALLOW_SAME_USER = become_allow_same_user
+
+# Note: Using nose's generator test cases here so we can't inherit from
+# unittest.TestCase
+class TestFilterNonJsonLines(object):
+    parsable_cases = (
+            (u'{"hello": "world"}', u'{"hello": "world"}'),
+            (u'{"hello": "world"}\n', u'{"hello": "world"}'),
+            (u'{"hello": "world"} ', u'{"hello": "world"} '),
+            (u'{"hello": "world"} \n', u'{"hello": "world"} '),
+            (u'Message of the Day\n{"hello": "world"}', u'{"hello": "world"}'),
+            (u'{"hello": "world"}\nEpilogue', u'{"hello": "world"}'),
+            (u'Several\nStrings\nbefore\n{"hello": "world"}\nAnd\nAfter\n', u'{"hello": "world"}'),
+            (u'{"hello": "world",\n"olá": "mundo"}', u'{"hello": "world",\n"olá": "mundo"}'),
+            (u'\nPrecedent\n{"hello": "world",\n"olá": "mundo"}\nAntecedent', u'{"hello": "world",\n"olá": "mundo"}'),
+            )
+
+    unparsable_cases = (
+            u'No json here',
+            u'"olá": "mundo"',
+            u'{"No json": "ending"',
+            u'{"wrong": "ending"]',
+            u'["wrong": "ending"}',
+            )
+
+    def check_filter_non_json_lines(self, stdout_line, parsed):
+        eq_(parsed, ActionBase._filter_non_json_lines(stdout_line))
+
+    def test_filter_non_json_lines(self):
+        for stdout_line, parsed in self.parsable_cases:
+            yield self.check_filter_non_json_lines, stdout_line, parsed
+
+    @raises(ValueError)
+    def check_unparsable_filter_non_json_lines(self, stdout_line):
+        ActionBase._filter_non_json_lines(stdout_line)
+
+    def test_unparsable_filter_non_json_lines(self):
+        for stdout_line in self.unparsable_cases:
+            yield self.check_unparsable_filter_non_json_lines, stdout_line
+