Repository: mitogen-hq/mitogen (language: python)

Issue:
Test fail with `become_user` in make_tmp_path.yml but works locally
When running this PR: https://github.com/dw/mitogen/pull/658

From here, copied in case the link expires: https://dev.azure.com/dw-mitogen/Mitogen/_build/results?buildId=790&view=logs&j=fa8bcf4d-d390-5291-b04f-0e6afdb08a32&t=3ae5371e-f71d-5b96-255d-41f780f14324
```
TASK [Try writing to temp directory for the readonly_homedir user script=from ansible.module_utils.basic import get_module_path
path = get_module_path() + '/foo.txt'
result['path'] = path
open(path, 'w').write("bar")
] ***
task path: /Users/runner/runners/2.164.8/work/1/s/tests/ansible/integration/action/make_tmp_path.yml:131
[task 4254] 01:48:46.938134 D ansible_mitogen.process: will use multiplexer 0 (/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/mitogen_unix_cnuAuJ.sock) to connect to "target"
[task 4254] 01:48:46.939074 D mitogen.unix: client: connecting to /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/mitogen_unix_cnuAuJ.sock
[task 4254] 01:48:46.939765 D mitogen.unix: client: local ID is 10073, remote is 0
[mux  3582] 01:48:46.939811 D mitogen.unix: listener: accepted connection from PID 4254: unix_client.4254
[mux  3582] 01:48:46.941767 D mitogen.parent: starting function call to ssh.localhost: mitogen.parent._proxy_connect(method_name=u'sudo', name=None, kwargs=Kwargs({u'username': u'mitogen__readonly_homedir', u'remote_name': None, u'profiling': False, u'sudo_path': u'sudo', u'python_path': [u'/usr/bin/python'], 'unidirectional': True, u'debug': False, u'password': None, u'sudo_args': [u'-H', u'-S', u'-n'], u'connect_timeout': 10}))
[mux  3582] 01:48:46.943608 D mitogen.[ssh.localhost]: Dispatcher: dispatching (None, u'mitogen.parent', None, u'_proxy_connect', (), Kwargs({'method_name': u'sudo', 'name': None, 'kwargs': Kwargs({u'username': u'mitogen__readonly_homedir', u'password': None, u'profiling': False, u'sudo_path': u'sudo', u'python_path': [u'/usr/bin/python'], 'unidirectional': True, u'debug': False, u'remote_name': None, u'sudo_args': [u'-H', u'-S', u'-n'], u'connect_timeout': 10})}))
[mux  3582] 01:48:46.944395 D mitogen.parent.[ssh.localhost]: creating connection to context 1009 using mitogen.sudo
[mux  3582] 01:48:46.946509 D mitogen.parent.[ssh.localhost]: command line for Connection(None): sudo -u mitogen__readonly_homedir -H -- /usr/bin/python -c "import codecs,os,sys;_=codecs.decode;exec(_(_(\"eNqFkcFLwzAUxu/9K3p7CYtbusEqhYCyg3gYQhF30CFpm2qwS0Lars6/3rdOXDsP3t4v35f3Pd5L2UbYeuq0U4QGnnUD0mWIUFr/QWgSHOuidXPCWcQ5PXPKhuRRjU6cV7ZWJB2CH8JmCB0CBtYHjK9kg6m7UIgQCuk7bSCUpuhF9anytpFZpXp51tZ+lmkzc4fm3RrAOS9tE9G/7JWvtTXPyWLbxyqz1x4ZbtO7Jw5bMf528iBWZCywMU6A7HRj35RJfGuM8jdrmV9FS349rWwuq2QRz2MKNMB2ndeNIhGD9f3jA+f8xQCOkdsCt02DlXglx30X1imDWwafAZ16JQsSxXzJKYMv7bBT6cTZt2HQZXA8Qel+AlZ9fVrrhbv7z/13ymg85e/B5vQb3H+0CQ==\".encode(),\"base64\"),\"zip\"))"
[mux  3582] 01:48:46.966627 D mitogen.parent.[ssh.localhost]: child for Connection(None) started: pid:4255 stdin:30 stdout:30 stderr:28
[mux  3582] 01:48:46.987828 D mitogen.parent.[ssh.localhost]: sudo.mitogen__readonly_homedir: (unrecognized): sudo: runas gid 4294967295: invalid value
[mux  3582] 01:48:46.988303 D mitogen.parent.[ssh.localhost]: sudo.mitogen__readonly_homedir: (unrecognized): sudo: runas_gid=4294967295: invalid value
[mux  3582] 01:48:46.988794 D mitogen.[ssh.localhost]: <Side of sudo.mitogen__readonly_homedir fd 30>: empty read, disconnecting
[mux  3582] 01:48:46.989234 D mitogen.[ssh.localhost]: BootstrapProtocol(sudo.mitogen__readonly_homedir): disconnecting
[mux  3582] 01:48:46.989614 D mitogen.[ssh.localhost]: <Side of sudo.mitogen__readonly_homedir fd 28>: empty read, disconnecting
[mux  3582] 01:48:46.989998 D mitogen.parent.[ssh.localhost]: failing connection sudo.mitogen__readonly_homedir due to EofError(u'EOF on stream; last 100 lines received:\nsudo: runas gid 4294967295: invalid value\nsudo: runas_gid=4294967295: invalid value',)
[mux  3582] 01:48:46.990407 D mitogen.parent.[ssh.localhost]: PopenProcess sudo.mitogen__readonly_homedir pid 4255: exited with return code 1
[mux  3582] 01:48:46.990789 D mitogen.[ssh.localhost]: SetupProtocol(sudo.mitogen__readonly_homedir): disconnecting
[mux  3582] 01:48:46.991178 D mitogen.[ssh.localhost]: SetupProtocol(sudo.mitogen__readonly_homedir): disconnecting
[mux  3582] 01:48:46.991543 D mitogen.[ssh.localhost]: Dispatcher: Message(1006, 0, 0, 101, 1038, '\x80\x02(NX\x0e\x00\x00\x00mitogen.parentNX\x0e\x00\x00\x00_proxy_connect)cmitog'..410) -> {u'msg': 'error occurred on host Mac-1608.local: EOF on stream; last 100 lines received:\nsudo: runas gid 4294967295: invalid value\nsudo: runas_gid=4294967295: invalid value', u'id': None, u'name': None}
[task 4254] 01:48:46.992739 D mitogen: MitogenProtocol(unix_listener.3582): disconnecting
[mux  3582] 01:48:46.993331 D mitogen: <Side of unix_client.4254 fd 123>: empty read, disconnecting
[task 4254] 01:48:46.993378 D mitogen: Waker(fd=57/58): disconnecting
[mux  3582] 01:48:46.993779 D mitogen: MitogenProtocol(unix_client.4254): disconnecting
[task 4254] 01:48:46.993911 D mitogen: Router(Broker(fb90)): stats: 0 module requests in 0 ms, 0 sent (0 ms minify time), 0 negative responses. Sent 0.0 kb total, 0.0 kb avg.
fatal: /Users/runner/runners/2.164.8/work/1/s/tests/ansible/integration/action/make_tmp_path.yml:131: [target]: FAILED! => {
  msg: error occurred on host Mac-1608.local: EOF on stream; last 100 lines received:
  msg: sudo: runas gid 4294967295: invalid value

```
This test works locally on the branch though after manually running `_user_accounts.yml` to set up test users:
```
/tmp/mitogen/tests/ansible$ cd ../.. && python setup.py install && cd - && ANSIBLE_STRATEGY=mitogen_linear ./run_ansible_playbook.py integration/action/make_tmp_path.yml -l target -k -K
```

I tried to get it to work with Mac 10.13,10.14, and 10.15. My Mac locally runs 10.14.4. 


Discussion / hints:
Similar error with `TASK [file path=/tmp/sync-test.out, state=absent] ******************************
`
similar error with `Ensure password sudo succeeds.`
also `TASK [Verify we can login to a non-passworded requiretty account _raw_params=whoami]`

Fix patch:
diff --git a/docs/changelog.rst b/docs/changelog.rst
index 0655e9704..e435fe771 100644
--- a/docs/changelog.rst
+++ b/docs/changelog.rst
@@ -23,6 +23,7 @@ In progress (unreleased)
 
 * :gh:issue:`905` :mod:`ansible_mitogen`: Support templated SSH command
   arguments (e.g. ``ansible_ssh_args``, ``ansible_ssh_extra_args``).
+* :gh:issue:`692` tests: Fix and re-enable several sudo tests
 
 
 v0.3.14 (2024-10-16)


Test patch:
diff --git a/tests/ansible/integration/action/make_tmp_path.yml b/tests/ansible/integration/action/make_tmp_path.yml
index b3f3b519e..c0200691c 100644
--- a/tests/ansible/integration/action/make_tmp_path.yml
+++ b/tests/ansible/integration/action/make_tmp_path.yml
@@ -142,17 +142,16 @@
     # readonly homedir
     #
 
-    # TODO: https://github.com/dw/mitogen/issues/692
-    # - name: "Try writing to temp directory for the readonly_homedir user"
-    #   become: true
-    #   become_user: mitogen__readonly_homedir
-    #   custom_python_run_script:
-    #     script: |
-    #       from ansible.module_utils.basic import get_module_path
-    #       path = get_module_path() + '/foo.txt'
-    #       result['path'] = path
-    #       open(path, 'w').write("bar")
-    #   register: tmp_path
+    - name: Try writing to temp directory for the readonly_homedir user
+      become: true
+      become_user: mitogen__readonly_homedir
+      custom_python_run_script:
+        script: |
+          from ansible.module_utils.basic import get_module_path
+          path = get_module_path() + '/foo.txt'
+          result['path'] = path
+          open(path, 'w').write("bar")
+      register: tmp_path
 
     #
     # modules get the same base dir
diff --git a/tests/ansible/integration/action/synchronize.yml b/tests/ansible/integration/action/synchronize.yml
index 6da81da30..58f54cc82 100644
--- a/tests/ansible/integration/action/synchronize.yml
+++ b/tests/ansible/integration/action/synchronize.yml
@@ -40,11 +40,11 @@
       delegate_to: localhost
       run_once: true
 
-    # TODO: https://github.com/dw/mitogen/issues/692
-    # - file:
-    #     path: /tmp/sync-test.out
-    #     state: absent
-    #   become: true
+    - name: Ensure clean slate
+      become: true
+      file:
+        path: /tmp/sync-test.out
+        state: absent
 
     # exception:   File "/tmp/venv/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 129, in cleanup
     # exception:     self._remove_tmp_path(self._connection._shell.tmpdir)
@@ -70,14 +70,14 @@
               outout={{ outout }}
       when: False
 
-    # TODO: https://github.com/dw/mitogen/issues/692
-    # - file:
-    #     path: "{{item}}"
-    #     state: absent
-    #   become: true
-    #   with_items:
-    #   - /tmp/synchronize-action-key
-    #   - /tmp/sync-test
-    #   - /tmp/sync-test.out
+    - name: Cleanup
+      become: true
+      file:
+        path: "{{ item }}"
+        state: absent
+      with_items:
+        - /tmp/synchronize-action-key
+        - /tmp/sync-test
+        - /tmp/sync-test.out
   tags:
     - synchronize
diff --git a/tests/ansible/integration/become/sudo_password.yml b/tests/ansible/integration/become/sudo_password.yml
index 070346359..8f99bf6f5 100644
--- a/tests/ansible/integration/become/sudo_password.yml
+++ b/tests/ansible/integration/become/sudo_password.yml
@@ -5,10 +5,12 @@
   tasks:
 
     - name: Ensure sudo password absent but required.
-      shell: whoami
       become: true
       become_user: mitogen__pw_required
+      command:
+        cmd: whoami
       register: out
+      changed_when: false
       ignore_errors: true
       when:
         # https://github.com/ansible/ansible/pull/70785
@@ -32,10 +34,12 @@
           or is_mitogen
 
     - name: Ensure password sudo incorrect.
-      shell: whoami
       become: true
       become_user: mitogen__pw_required
+      command:
+        cmd: whoami
       register: out
+      changed_when: false
       vars:
         ansible_become_pass: nopes
       ignore_errors: true
@@ -59,18 +63,27 @@
           or ansible_version.full is version("2.11", ">=", strict=True)
           or is_mitogen
 
-    # TODO: https://github.com/dw/mitogen/issues/692
-    # - name: Ensure password sudo succeeds.
-    #   shell: whoami
-    #   become: true
-    #   become_user: mitogen__pw_required
-    #   register: out
-    #   vars:
-    #     ansible_become_pass: pw_required_password
+    - block:
+        - name: Ensure password sudo succeeds
+          become: true
+          become_user: mitogen__pw_required
+          vars:
+            ansible_become_pass: pw_required_password
+          command:
+            cmd: whoami
+          register: sudo_password_success_whoami
+          changed_when: false
 
-    # - assert:
-    #     that:
-    #       - out.stdout == 'mitogen__pw_required'
+        - assert:
+            that:
+              - sudo_password_success_whoami.stdout == 'mitogen__pw_required'
+            fail_msg: |
+              sudo_password_success_whoami={{ sudo_password_success_whoami }}
+      when:
+        # https://github.com/ansible/ansible/pull/70785
+        - ansible_facts.distribution not in ["MacOSX"]
+          or ansible_version.full is version("2.11", ">=", strict=True)
+          or is_mitogen
   tags:
     - sudo
     - sudo_password
diff --git a/tests/ansible/integration/become/sudo_requiretty.yml b/tests/ansible/integration/become/sudo_requiretty.yml
index 20f106fc6..08b478a28 100644
--- a/tests/ansible/integration/become/sudo_requiretty.yml
+++ b/tests/ansible/integration/become/sudo_requiretty.yml
@@ -3,34 +3,38 @@
 - name: integration/become/sudo_requiretty.yml
   hosts: test-targets
   tasks:
-    # - include_tasks: ../_mitogen_only.yml
+    # AIUI Vanilla Ansible cannot do sudo when requiretty configured
+    - include_tasks: ../_mitogen_only.yml
 
-    # TODO: https://github.com/dw/mitogen/issues/692
-    # - name: Verify we can login to a non-passworded requiretty account
-    #   shell: whoami
-    #   become: true
-    #   become_user: mitogen__require_tty
-    #   register: out
+    - name: Verify we can login to a non-passworded requiretty account
+      become: true
+      become_user: mitogen__require_tty
+      command:
+        cmd: whoami
+      changed_when: false
+      register: sudo_require_tty_whoami
 
-    # - assert:
-    #     that:
-    #       - out.stdout == 'mitogen__require_tty'
+    - assert:
+        that:
+          - sudo_require_tty_whoami.stdout == 'mitogen__require_tty'
+        fail_msg: |
+          sudo_require_tty_whoami={{ sudo_require_tty_whoami }}
 
+    - name: Verify we can login to a passworded requiretty account
+      become: true
+      become_user: mitogen__require_tty_pw_required
+      vars:
+        ansible_become_pass: require_tty_pw_required_password
+      command:
+        cmd: whoami
+      changed_when: false
+      register: sudo_require_tty_password_whoami
 
-    # ---------------
-
-    # TODO: https://github.com/dw/mitogen/issues/692
-    # - name: Verify we can login to a passworded requiretty account
-    #   shell: whoami
-    #   become: true
-    #   become_user: mitogen__require_tty_pw_required
-    #   vars:
-    #     ansible_become_pass: require_tty_pw_required_password
-    #   register: out
-
-    # - assert:
-    #     that:
-    #       - out.stdout == 'mitogen__require_tty_pw_required'
+    - assert:
+        that:
+          - sudo_require_tty_password_whoami.stdout == 'mitogen__require_tty_pw_required'
+        fail_msg: |
+          sudo_require_tty_password_whoami={{ sudo_require_tty_password_whoami }}
   tags:
     - mitogen_only
     - sudo
diff --git a/tests/ansible/integration/playbook_semantics/with_items.yml b/tests/ansible/integration/playbook_semantics/with_items.yml
index 4de29c4b8..891469814 100644
--- a/tests/ansible/integration/playbook_semantics/with_items.yml
+++ b/tests/ansible/integration/playbook_semantics/with_items.yml
@@ -3,30 +3,39 @@
 
 - name: integration/playbook_semantics/with_items.yml
   hosts: test-targets
+  gather_facts: true
   tasks:
+    - block:
+        - name: Spin up a few interpreters
+          become: true
+          vars:
+            ansible_become_user: "mitogen__user{{ item }}"
+          command:
+            cmd: whoami
+          with_sequence: start=1 end=3
+          register: first_run
+          changed_when: false
 
-    # TODO: https://github.com/dw/mitogen/issues/692
-    # - name: Spin up a few interpreters
-    #   shell: whoami
-    #   become: true
-    #   vars:
-    #     ansible_become_user: "mitogen__user{{item}}"
-    #   with_sequence: start=1 end=3
-    #   register: first_run
+        - name: Reuse them
+          become: true
+          vars:
+            ansible_become_user: "mitogen__user{{ item }}"
+          command:
+            cmd: whoami
+          with_sequence: start=1 end=3
+          register: second_run
+          changed_when: false
 
-    # - name: Reuse them
-    #   shell: whoami
-    #   become: true
-    #   vars:
-    #     ansible_become_user: "mitogen__user{{item}}"
-    #   with_sequence: start=1 end=3
-    #   register: second_run
-
-    # - name: Verify first and second run matches expected username.
-    #   assert:
-    #     that:
-    #     - first_run.results[item|int].stdout == ("mitogen__user%d" % (item|int + 1))
-    #     - first_run.results[item|int].stdout == second_run.results[item|int].stdout
-    #   with_sequence: start=0 end=2
-  tags:
-    - custom_python_new_style_module
+        - name: Verify first and second run matches expected username.
+          vars:
+            user_expected: "mitogen__user{{ item | int + 1 }}"
+          assert:
+            that:
+              - first_run.results[item | int].stdout == user_expected
+              - second_run.results[item | int].stdout == user_expected
+          with_sequence: start=0 end=2
+      when:
+        # https://github.com/ansible/ansible/pull/70785
+        - ansible_facts.distribution not in ["MacOSX"]
+          or ansible_version.full is version("2.11", ">=", strict=True)
+          or is_mitogen
diff --git a/tests/image_prep/_user_accounts.yml b/tests/image_prep/_user_accounts.yml
index 0b6d5e610..ad5a4ef57 100644
--- a/tests/image_prep/_user_accounts.yml
+++ b/tests/image_prep/_user_accounts.yml
@@ -73,6 +73,7 @@
       - user:
           name: "mitogen__{{item}}"
           shell: /bin/bash
+          group: staff
           groups: |
             {{
                 ['com.apple.access_ssh'] +
diff --git a/tests/image_prep/ansible.cfg b/tests/image_prep/ansible.cfg
index 0745aed13..da778786a 100644
--- a/tests/image_prep/ansible.cfg
+++ b/tests/image_prep/ansible.cfg
@@ -6,6 +6,7 @@ retry_files_enabled = false
 display_args_to_stdout = True
 no_target_syslog = True
 host_key_checking = False
+stdout_callback = yaml
 
 [inventory]
 unparsed_is_fatal = true