Repository: ansible/ansible

Issue:
user_module corrupts shadow password on MacOS X
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
user_module

##### ANSIBLE VERSION
```
ansible 2.4.3.0
  config file = /home/rleigh/.ansible.cfg
  configured module search path = [u'/home/rleigh/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/rleigh/venv/ansible/local/lib/python2.7/site-packages/ansible
  executable location = /home/rleigh/venv/ansible/bin/ansible
  python version = 2.7.14 (default, Sep 23 2017, 22:06:14) [GCC 7.2.0]
```

##### CONFIGURATION
```
ALLOW_WORLD_READABLE_TMPFILES(/home/rleigh/.ansible.cfg) = True
DEFAULT_HOST_LIST(/home/rleigh/.ansible.cfg) = [u'/home/rleigh/ci-test/ansible/inventory']
DEFAULT_ROLES_PATH(/home/rleigh/.ansible.cfg) = [u'/home/rleigh/ansible/ci-playbooks/roles']
```

##### OS / ENVIRONMENT
OS running ansible: Ubuntu 18.04 (bionic)
OS being managed: MacOS X 10.13 (high sierra), as well as various Linux, BSD and other systems (which work just fine).

##### SUMMARY
Updating the password with the `user` module results in a broken account which can't be logged into.  This was working fine with Ansible 2.2 and MacOS X 10.12 as far as I can recall, though I don't know which might have changed incompatibly.

##### STEPS TO REPRODUCE
- Create test password
- Create account `ci-admin` on MacOS X using the GUI or at initial OS installation time, using test password
- Update account with tasks below
- Account is broken as a result

```yaml
---
- name: ci-admin-user (unix) | Create ci-admin group
  group:
    name: ci-admin
    state: present
  become: yes

- name: ci-admin-user (unix) | Create ci-admin user
  user:
    name: ci-admin
    comment: "CI System Administrator for Ansible use"
    group: ci-admin
    password: "{{ ci_admin_password_hash }}"
    state: present
  become: yes
```

Where `ci_admin_password_hash` is in the form `$6$RdOcIY.fBNdZRKJJ$QCMbASbrMechNW6bsTz.yQkbWPQEn1/tsiPWhUvZTyKYaaFhFBjuw3aXR7AvUB17u7SuhWue9rZ7sLPMUK9wA1` (note: not the actual hash, it's representative only).  Use a hashed copy of the test password already used above, so that you can log in with the same password before and after the change.

##### EXPECTED RESULTS
I expected that the user would be created if missing, and updated if present.

##### ACTUAL RESULTS
The password was updated, but it was no longer possible to log into this user account with the password.

```
TASK [ci-admin-user : ci-admin-user (unix) | Create ci-admin group] ***********************************************************************************
task path: /home/rleigh/ansible/ci-playbooks/roles/ci-admin-user/tasks/unix.yml:2
Using module file /home/rleigh/venv/ansible/local/lib/python2.7/site-packages/ansible/modules/system/group.py
<ci-test.example.org> ESTABLISH SSH CONNECTION FOR USER: ci-admin
<ci-test.example.org> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o User=ci-admin -o ConnectTimeout=10 -o ControlPath=/home/rleigh/.ansible/cp/5bc0d52b20 ci-test.example.org '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''                                         
<ci-test.example.org> (0, '/Users/ci-admin\n', '')
<ci-test.example.org> ESTABLISH SSH CONNECTION FOR USER: ci-admin
<ci-test.example.org> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o User=ci-admin -o ConnectTimeout=10 -o ControlPath=/home/rleigh/.ansible/cp/5bc0d52b20 ci-test.example.org '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /Users/ci-admin/.ansible/tmp/ansible-tmp-1519125255.14-153699103275302 `" && echo ansible-tmp-1519125255.14-153699103275302="` echo /Users/ci-admin/.ansible/tmp/ansible-tmp-1519125255.14-153699103275302 `" ) && sleep 0'"'"''                                                                                                               
<ci-test.example.org> (0, 'ansible-tmp-1519125255.14-153699103275302=/Users/ci-admin/.ansible/tmp/ansible-tmp-1519125255.14-153699103275302\n', '')                                                                                                                                                   
<ci-test.example.org> PUT /tmp/tmpcyXhzu TO /Users/ci-admin/.ansible/tmp/ansible-tmp-1519125255.14-153699103275302/group.py
<ci-test.example.org> SSH: EXEC sshpass -d10 sftp -o BatchMode=no -b - -C -o ControlMaster=auto -o ControlPersist=60s -o User=ci-admin -o ConnectTimeout=10 -o ControlPath=/home/rleigh/.ansible/cp/5bc0d52b20 '[ci-test.example.org]'                                                        
<ci-test.example.org> (0, 'sftp> put /tmp/tmpcyXhzu /Users/ci-admin/.ansible/tmp/ansible-tmp-1519125255.14-153699103275302/group.py\n', '')
<ci-test.example.org> ESTABLISH SSH CONNECTION FOR USER: ci-admin
<ci-test.example.org> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o User=ci-admin -o ConnectTimeout=10 -o ControlPath=/home/rleigh/.ansible/cp/5bc0d52b20 ci-test.example.org '/bin/sh -c '"'"'chmod u+x /Users/ci-admin/.ansible/tmp/ansible-tmp-1519125255.14-153699103275302/ /Users/ci-admin/.ansible/tmp/ansible-tmp-1519125255.14-153699103275302/group.py && sleep 0'"'"''                                     
<ci-test.example.org> (0, '', '')
<ci-test.example.org> ESTABLISH SSH CONNECTION FOR USER: ci-admin
<ci-test.example.org> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o User=ci-admin -o ConnectTimeout=10 -o ControlPath=/home/rleigh/.ansible/cp/5bc0d52b20 -tt ci-test.example.org '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=gpwidxlemiifjhjvcermdcwpaemqfnoi] password: " -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-gpwidxlemiifjhjvcermdcwpaemqfnoi; /usr/bin/python /Users/ci-admin/.ansible/tmp/ansible-tmp-1519125255.14-153699103275302/group.py; rm -rf "/Users/ci-admin/.ansible/tmp/ansible-tmp-1519125255.14-153699103275302/" > /dev/null 2>&1'"'"'"'"'"'"'"'"' && sleep 0'"'"''                                                                                                              
<ci-test.example.org> (0, '\r\n\r\n{"name": "ci-admin", "changed": true, "system": false, "state": "present", "gid": 501, "invocation": {"module_args": {"state": "present", "gid": null, "name": "ci-admin", "system": false}}}\r\n', 'Shared connection to ci-test.example.org closed.\r\n')
changed: [ci-test.example.org] => {
    "changed": true, 
    "gid": 501, 
    "invocation": {
        "module_args": {
            "gid": null, 
            "name": "ci-admin", 
            "state": "present", 
            "system": false
        }
    }, 
    "name": "ci-admin", 
    "state": "present", 
    "system": false
}

TASK [ci-admin-user : ci-admin-user (unix) | Create ci-admin user] ************************************************************************************
task path: /home/rleigh/ansible/ci-playbooks/roles/ci-admin-user/tasks/unix.yml:8
Using module file /home/rleigh/venv/ansible/local/lib/python2.7/site-packages/ansible/modules/system/user.py
<ci-test.example.org> ESTABLISH SSH CONNECTION FOR USER: ci-admin
<ci-test.example.org> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o User=ci-admin -o ConnectTimeout=10 -o ControlPath=/home/rleigh/.ansible/cp/5bc0d52b20 ci-test.example.org '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''                                         
<ci-test.example.org> (0, '/Users/ci-admin\n', '')
<ci-test.example.org> ESTABLISH SSH CONNECTION FOR USER: ci-admin
<ci-test.example.org> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o User=ci-admin -o ConnectTimeout=10 -o ControlPath=/home/rleigh/.ansible/cp/5bc0d52b20 ci-test.example.org '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /Users/ci-admin/.ansible/tmp/ansible-tmp-1519125263.5-208748598895782 `" && echo ansible-tmp-1519125263.5-208748598895782="` echo /Users/ci-admin/.ansible/tmp/ansible-tmp-1519125263.5-208748598895782 `" ) && sleep 0'"'"''                                                                                                                  
<ci-test.example.org> (0, 'ansible-tmp-1519125263.5-208748598895782=/Users/ci-admin/.ansible/tmp/ansible-tmp-1519125263.5-208748598895782\n', '')                                                                                                                                                     
<ci-test.example.org> PUT /tmp/tmpMtf5XA TO /Users/ci-admin/.ansible/tmp/ansible-tmp-1519125263.5-208748598895782/user.py
<ci-test.example.org> SSH: EXEC sshpass -d10 sftp -o BatchMode=no -b - -C -o ControlMaster=auto -o ControlPersist=60s -o User=ci-admin -o ConnectTimeout=10 -o ControlPath=/home/rleigh/.ansible/cp/5bc0d52b20 '[ci-test.example.org]'                                                        
<ci-test.example.org> (0, 'sftp> put /tmp/tmpMtf5XA /Users/ci-admin/.ansible/tmp/ansible-tmp-1519125263.5-208748598895782/user.py\n', '')
<ci-test.example.org> ESTABLISH SSH CONNECTION FOR USER: ci-admin
<ci-test.example.org> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o User=ci-admin -o ConnectTimeout=10 -o ControlPath=/home/rleigh/.ansible/cp/5bc0d52b20 ci-test.example.org '/bin/sh -c '"'"'chmod u+x /Users/ci-admin/.ansible/tmp/ansible-tmp-1519125263.5-208748598895782/ /Users/ci-admin/.ansible/tmp/ansible-tmp-1519125263.5-208748598895782/user.py && sleep 0'"'"''                                        
<ci-test.example.org> (0, '', '')
<ci-test.example.org> ESTABLISH SSH CONNECTION FOR USER: ci-admin
<ci-test.example.org> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o User=ci-admin -o ConnectTimeout=10 -o ControlPath=/home/rleigh/.ansible/cp/5bc0d52b20 -tt ci-test.example.org '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=bebrypcjmpmzxkdrtcephxymtufcvytv] password: " -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-bebrypcjmpmzxkdrtcephxymtufcvytv; /usr/bin/python /Users/ci-admin/.ansible/tmp/ansible-tmp-1519125263.5-208748598895782/user.py; rm -rf "/Users/ci-admin/.ansible/tmp/ansible-tmp-1519125263.5-208748598895782/" > /dev/null 2>&1'"'"'"'"'"'"'"'"' && sleep 0'"'"''                                                                                                                 
<ci-test.example.org> (0, '\r\n\r\n{"comment": "CI System Administrator for Ansible use", "shell": "/bin/bash", "group": 501, "name": "ci-admin", "changed": true, "state": "present", "invocation": {"module_args": {"comment": "CI System Administrator for Ansible use", "ssh_key_bits": 0, "update_password": "always", "non_unique": false, "force": false, "ssh_key_type": "rsa", "ssh_key_passphrase": null, "createhome": true, "uid": null, "home": null, "append": false, "skeleton": null, "ssh_key_comment": "ansible-generated on mackerel.local", "group": "ci-admin", "system": false, "state": "present", "local": null, "shell": null, "expires": null, "ssh_key_file": null, "groups": null, "move_home": false, "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "name": "ci-admin", "seuser": null, "remove": false, "login_class": null, "generate_ssh_key": null}}, "home": "/Users/ci-admin", "move_home": false, "password": "NOT_LOGGING_PASSWORD", "append": false, "uid": 501}\r\n', 'Shared connection to ci-test.example.org closed.\r\n')        
changed: [ci-test.example.org] => {
    "append": false, 
    "changed": true, 
    "comment": "CI System Administrator for Ansible use", 
    "group": 501, 
    "home": "/Users/ci-admin", 
    "invocation": {
        "module_args": {
            "append": false, 
            "comment": "CI System Administrator for Ansible use", 
            "createhome": true, 
            "expires": null, 
            "force": false, 
            "generate_ssh_key": null, 
            "group": "ci-admin", 
            "groups": null, 
            "home": null, 
            "local": null, 
            "login_class": null, 
            "move_home": false, 
            "name": "ci-admin", 
            "non_unique": false, 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "remove": false, 
            "seuser": null, 
            "shell": null, 
            "skeleton": null, 
            "ssh_key_bits": 0, 
            "ssh_key_comment": "ansible-generated on mackerel.local", 
            "ssh_key_file": null, 
            "ssh_key_passphrase": null, 
            "ssh_key_type": "rsa", 
            "state": "present", 
            "system": false, 
            "uid": null, 
            "update_password": "always"
        }
    }, 
    "move_home": false, 
    "name": "ci-admin", 
    "password": "NOT_LOGGING_PASSWORD", 
    "shell": "/bin/bash", 
    "state": "present", 
    "uid": 501
}
```

Checked account details with 

```
sudo plutil -p /var/db/dslocal/nodes/Default/users/ci-admin.plist > acct-plist.new
defaults read /var/db/dslocal/nodes/Default/users/ci-admin.plist ShadowHashData|tr -dc 0-9a-f|xxd -r -p|plutil -convert xml1 - -o - > shadow.new
```

and diffed with the state before running ansible.  This is a representative example of the changes:

```
--- shadow.orig 2018-02-20 11:01:31.000000000 +0000
+++ shadow.new  2018-02-20 11:14:55.000000000 +0000
@@ -6,39 +6,39 @@
        <dict>
                <key>entropy</key>
                <data>
-               mTSEiACImPlSDdZjEZzx/QV8AdCMBirpCkGhjitzo+U8/oqrtQjkxDrVgqqK
-               81vmjWezRtmZroRXsice1wIFjacIlJoDEkb2S/W+zVNEwHP+l0XcrvAvwTnU
-               gMLZ63r7k7bwgymXcDEl3uo1qnQ74XJdc+fR2psrP+50SUl21Cs=
+               OF2McmIkESnx2KAwVgSkkXB5+AsTIpEAUKwvDfTbnJbSQkejEiabXmXa2Y+U
+               fciHflhlagDeX+ipJV7rnD0OmSLSlpH6lYapWbDXvyku8yD+kBNIjWRzymF/
+               akU+vG7oB7k9SLmudGDCsnJPuEHUXnGJkVvMMZyow8LDLuHl+RM=
                </data>
                <key>iterations</key>
-               <integer>59513</integer>
+               <integer>51893</integer>
                <key>salt</key>
                <data>
-               7fuyJ9cXLN3Kw3VxPIU3UeCKaZEb0QXf7mCVxB0Np5w=
+               Vlfc7gkUd8Dq3HKinSRoG7v20KoI55xvRqY+LdUG+wU=
                </data>
        </dict>
        <key>SRP-RFC5054-4096-SHA512-PBKDF2</key>
        <dict>
                <key>iterations</key>
-               <integer>50545</integer>
+               <integer>57873</integer>
                <key>salt</key>
                <data>
-               Vp4KGFfnwKAFjB3UVsVXXdgEj4IoIfjcX0WOfjRaDo8=
+               qAO7UO8zVEC0ANIUcHYoXrgPV0Bx0v2352u9V2rlVMU=
                </data>
                <key>verifier</key>
                <data>
-               XZuGlPGAf/xW87wcOrct4dSMHDANl8DugtK4nHrxUu+G6Dqm2oHDY9frqF0e
-               JXrfI3tNRq6u7DkjkaJgzyZ5U+7Jj0hAV9Ba7omBLvFWtiFVHHDzMmboaDwW
-               nCLb5/W8e2f1AzEXN8QA01Hlq5n4cl0E7JXaJB56tBy8lxsxC14AfdVSBr0K
-               i4fsKBRBOac/bYsqoxqpzUBnaWnPAnpgi6D4PXb7oBRfEU4JANj88IYYQV3F
-               fq36UZ8p4VEI+C/m5gQ1Kgn+vFXh2te8DuDJBAVIA9n5MI6oAnMWYARWboaX
-               4ZAqRLCzo+qKHQjNLbb8ztxWfRB4uBBPbhrn0Zi/grWSwoSk5fTLo5NO0l6G
-               sjsWcAZJu4zpatzlXO2A6IsmVc4tXNmL4CT8E6b8y6dsfEqFJc8x6BVx8kxo
-               7KRwh2usjWK4FjzG1dNyi8BZaqSoGE7NVisgbuBN4ljMTj2S89jlYinbTaC9
-               R8UyYxuXV8kX0+zwbDfkqRnvQsREbiM1npTtdaQZCwjsUNDlqPDrOOx8oYcd
-               ip5buDyN7CKfxC/pOk2Axop6JJOedzbxyQmpXAqnQYHUSMIixFsM4vCEjLMS
-               OZo/8K3CUKFMmt2i0sfcJnDvl3xf/3fATauy8geDXftSPXqjL18fUNOggz/Z
-               Jq04UWUE5oo41k0+rhPPA4U=
+               qRd8uml+rKo+/6kZVQ5Ze8n6yljzm6sb2FpI/1AjptGiOtmveO4EIlNJ0tU7
+               EJlf7l37epXBbQ/zxF+Om6zYozMMzSwWRCA7GLiLGE2b47HgwcSFsJHVCv7i
+               3azvEok3RWyEJjCCre2qfFUP8LvHM2QYhK5nnUgegHlLjcpRkyXMZ3Rsdn0v
+               lIc+73LBD6nrXh5oSe4beRT0tHpKj3/B7tErPosjXlGsnEy3yKG/nJ4L5Soi
+               sf6BZVoB0+T7WS3MvEr0lYFwH0SbCKZSIq14ZxW3mzGirrdqMm8Dj2G2Melu
+               5+/J3EnmnKMmuE2lFpAz2fm//Rv3V36pIZCnTRdfCUVrIkRizNxEnMFfEO+1
+               Ncm2C87tIGs0DaN88bw4kLcQ8o/ulxSjdYybUowDoc2wElGh6PIJJRkJA8gB
+               tpDZ3mRcDT7XT2pho4gOUP418kXqnRKh3Uqg4peRKU91TYPvgmEo6lbDd+fK
+               0INPFP23njz6kV3DbXaDFqnB9WhRHJJO+kymRgm0auLwy40trxaJNKm8qPS0
+               q+73j3CjnTmHLPN8fSKGhKgNV7A+DjRez0v5i8aA2LvS4CSuQlpPgRhAzPJM
+               CUfRkZqOSH6xsoW3rkPm4kku4tMVrPIxgYVI/odGjMAjWXtSVLKnL+vYduRo
+               8O9KenhEJqp5jBLf7nw0s+E=
                </data>
        </dict>
 </dict>
```

On the face of it, it doesn't look too different, but in practice it's broken.


Discussion:
Files identified in the description:
* [lib/ansible/modules/system/user.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/system/user.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @sfromm
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Also tested with

```
ansible 2.6.0
  config file = /home/rleigh/.ansible.cfg
  configured module search path = [u'/home/rleigh/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/rleigh/venv/ansible-dev/local/lib/python2.7/site-packages/ansible-2.6.0-py2.7.egg/ansible
  executable location = /home/rleigh/venv/ansible-dev/bin/ansible
  python version = 2.7.14+ (default, Feb  6 2018, 19:12:18) [GCC 7.3.0]
```

with the same results (current git `devel` @ `78e900cd7`).
While I provided a diff of the user shadow password data, it may be the case that something else was modified in addition to this file.  I'm not a MacOS X auth expert, so I don't know what else could be at fault.  But I'm happy to do further testing and provide more data.
Retested with the ansible 2.5.0 release today; the bug is present in this release as well.

Also tested current `develop` (commit d5cfc54ef4b06).  It's still present here as well.
On Mac, the password must be a plain text password, it cannot be an encrypted hash.

Per the documentation:

> Note on Darwin system, this value has to be cleartext. Beware of security issues.

needs_info
What's the reason for this restriction?  The plist has exactly the same information that's encoded in the shadow db, just split out into separate elements (salt, iterations, hash etc.).  So long as you're using a compatible hash like SHA512, it should be able to use the same hash on Linux, BSD and MacOS X.
> What's the reason for this restriction?

It is a limitation of the `dscl` command line utility.  It only accepts plain text passwords, and handles the encryption itself.

Manipulation of the databases outside of `dscl` is not recommended.  We use the recommended approach for manipulating users on Mac.
OK.  Is this a change to the module?  I could have sworn it previously worked with 2.1/2.2, but I could be wrong.

As a suggestion, would it be worth having separate e.g. `password` and `password-hash` (or `password` and `password-plaintext`) properties for this module so that there's no ambiguity over what the input data is?  This would allow the module to error out on Darwin if provided a hash, rather than hashing the hash and messing up the plist file.  It would also permit the use of plaintext passwords on other systems (ansible could handle the hashing).
Just to confirm, using a plaintext password appears to be working correctly, thanks.
it is clearly stated in the docs for the option:

`Note on Darwin system, this value has to be cleartext. Beware of security issues.`

but i'll update it to clarify for those that don't know OS X is a Darwin system.

Resolving patch:
diff --git a/lib/ansible/modules/system/user.py b/lib/ansible/modules/system/user.py
--- a/lib/ansible/modules/system/user.py
+++ b/lib/ansible/modules/system/user.py
@@ -79,12 +79,10 @@
         version_added: "2.0"
     password:
         description:
-            - Optionally set the user's password to this crypted value.  See
-              the user example in the github examples directory for what this looks
-              like in a playbook. See U(http://docs.ansible.com/ansible/faq.html#how-do-i-generate-crypted-passwords-for-the-user-module)
+            - Optionally set the user's password to this crypted value.
+            - On Darwin/OS X systems, this value has to be cleartext. Beware of security issues.
+            - See U(http://docs.ansible.com/ansible/faq.html#how-do-i-generate-crypted-passwords-for-the-user-module)
               for details on various ways to generate these password values.
-              Note on Darwin system, this value has to be cleartext.
-              Beware of security issues.
     state:
         description:
             - Whether the account should exist or not, taking action if the state is different from what is stated.


Test patch: