Repository: ansible/ansible

Issue:
Race condition in password lookup
##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
 - Bug Report


##### COMPONENT NAME
password lookup module

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
# ansible --version
ansible 2.4.2.0
  config file = /root/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
```

##### CONFIGURATION
```
# ansible-config dump --only-changed
DEFAULT_HOST_LIST(/root/ansible.cfg) = [u'/root/hosts', u'/etc/ansible/hosts']
RETRY_FILES_SAVE_PATH(/root/ansible.cfg) = /root/.ansible/retry-files
```

<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->

##### OS / ENVIRONMENT
Centos 7.4
ansible-2.4.2.0-2.el7.noarch

##### SUMMARY
Race condition in paswword lookup module if task is being run for multiple hosts. As a workaround it's possible to use 2 steps of acquiring passwords: generate first and then fill the variable to store in playbook
Example output:
```
TASK [mongodb : Generate root password for MongoDB and save it to /etc/trinity/passwords] ***
ok: [controller1] => {"ansible_facts": {"tmp_pwd": "2f6eEfb6SoEmp76QCK9S"}, "changed": false}
ok: [controller2] => {"ansible_facts": {"tmp_pwd": "F9UwFND06kdK89CiaGP3"}, "changed": false}

TASK [mongodb : Get root password for MongoDB from /etc/trinity/passwords] *****
ok: [controller1] => {"ansible_facts": {"mongo_root_pwd": "2f6eEfb6SoEmp76QCK9S"}, "changed": false}
ok: [controller2] => {"ansible_facts": {"mongo_root_pwd": "2f6eEfb6SoEmp76QCK9S"}, "changed": false}
```
As it's seen `tmp_pwd` differs and only one var is being stored on disk.

##### STEPS TO REPRODUCE
Run playbook with password lookup with several target hosts.

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: Generate root password for MongoDB and save it to /etc/trinity/passwords
  set_fact:
    tmp_pwd: "{{ lookup('password',
                          '/etc/trinity/passwords/mongo/root.txt
                           chars=ascii_letters,digits,hexdigits') }}"

- name: Get root password for MongoDB from /etc/trinity/passwords
  set_fact:
    mongo_root_pwd: "{{ lookup('password',
                          '/etc/trinity/passwords/mongo/root.txt
                           chars=ascii_letters,digits,hexdigits') }}"
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
Passwords should be consistent

##### ACTUAL RESULTS
See above

##### MAIN SUSPECT
https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/lookup/password.py#L280

Steps of reading and checking if password is available in file and generating/saving one are split witch opens door for race condition on concurrent runs.

I think it's worth to use `f = os.open(b_path, os.O_CREAT | os.O_EXCL)` and use f as global var or wrap it in the class



Discussion:
Files identified in the description:
* [lib/ansible/plugins/lookup/password.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/lookup/password.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->

Resolving patch:
diff --git a/lib/ansible/plugins/lookup/password.py b/lib/ansible/plugins/lookup/password.py
--- a/lib/ansible/plugins/lookup/password.py
+++ b/lib/ansible/plugins/lookup/password.py
@@ -91,6 +91,9 @@
 
 import os
 import string
+import time
+import shutil
+import hashlib
 
 from ansible.errors import AnsibleError, AnsibleAssertionError
 from ansible.module_utils._text import to_bytes, to_native, to_text
@@ -266,6 +269,40 @@ def _write_password_file(b_path, content):
         f.write(b_content)
 
 
+def _get_lock(b_path):
+    """Get the lock for writing password file."""
+    first_process = False
+    b_pathdir = os.path.dirname(b_path)
+    lockfile_name = to_bytes("%s.ansible_lockfile" % hashlib.md5(b_path).hexdigest())
+    lockfile = os.path.join(b_pathdir, lockfile_name)
+    if not os.path.exists(lockfile) and b_path != to_bytes('/dev/null'):
+        try:
+            makedirs_safe(b_pathdir, mode=0o700)
+            fd = os.open(lockfile, os.O_CREAT | os.O_EXCL)
+            os.close(fd)
+            first_process = True
+        except OSError as e:
+            if e.strerror != 'File exists':
+                raise
+
+    counter = 0
+    # if the lock is got by other process, wait until it's released
+    while os.path.exists(lockfile) and not first_process:
+        time.sleep(2 ** counter)
+        if counter >= 2:
+            raise AnsibleError("Password lookup cannot get the lock in 7 seconds, abort..."
+                               "This may caused by un-removed lockfile"
+                               "you can manually remove it from controller machine at %s and try again" % lockfile)
+        counter += 1
+    return first_process, lockfile
+
+
+def _release_lock(lockfile):
+    """Release the lock so other processes can read the password file."""
+    if os.path.exists(lockfile):
+        os.remove(lockfile)
+
+
 class LookupModule(LookupBase):
     def run(self, terms, variables, **kwargs):
         ret = []
@@ -276,7 +313,10 @@ def run(self, terms, variables, **kwargs):
             b_path = to_bytes(path, errors='surrogate_or_strict')
             chars = _gen_candidate_chars(params['chars'])
 
-            changed = False
+            changed = None
+            # make sure only one process finishes all the job first
+            first_process, lockfile = _get_lock(b_path)
+
             content = _read_password_file(b_path)
 
             if content is None or b_path == to_bytes('/dev/null'):
@@ -294,6 +334,10 @@ def run(self, terms, variables, **kwargs):
                 content = _format_content(plaintext_password, salt, encrypt=params['encrypt'])
                 _write_password_file(b_path, content)
 
+            if first_process:
+                # let other processes continue
+                _release_lock(lockfile)
+
             if params['encrypt']:
                 password = do_encrypt(plaintext_password, params['encrypt'], salt=salt)
                 ret.append(password)


Test patch:
diff --git a/test/units/plugins/lookup/test_password.py b/test/units/plugins/lookup/test_password.py
--- a/test/units/plugins/lookup/test_password.py
+++ b/test/units/plugins/lookup/test_password.py
@@ -29,6 +29,7 @@
 from ansible.errors import AnsibleError
 from ansible.module_utils.six import text_type
 from ansible.module_utils.six.moves import builtins
+from ansible.module_utils._text import to_bytes
 from ansible.plugins.loader import PluginLoader
 from ansible.plugins.lookup import password
 from ansible.utils import encrypt
@@ -372,6 +373,14 @@ def setUp(self):
         self.fake_loader = DictDataLoader({'/path/to/somewhere': 'sdfsdf'})
         self.password_lookup = password.LookupModule(loader=self.fake_loader)
         self.os_path_exists = password.os.path.exists
+        self.os_open = password.os.open
+        password.os.open = lambda path, flag: None
+        self.os_close = password.os.close
+        password.os.close = lambda fd: None
+        self.os_remove = password.os.remove
+        password.os.remove = lambda path: None
+        self.makedirs_safe = password.makedirs_safe
+        password.makedirs_safe = lambda path, mode: None
 
         # Different releases of passlib default to a different number of rounds
         self.sha256 = passlib.registry.get_crypt_handler('pbkdf2_sha256')
@@ -380,6 +389,10 @@ def setUp(self):
 
     def tearDown(self):
         password.os.path.exists = self.os_path_exists
+        password.os.open = self.os_open
+        password.os.close = self.os_close
+        password.os.remove = self.os_remove
+        password.makedirs_safe = self.makedirs_safe
         passlib.registry.register_crypt_handler(self.sha256, force=True)
 
     @patch.object(PluginLoader, '_get_paths')
@@ -425,7 +438,7 @@ def test_encrypt(self, mock_get_paths, mock_write_file):
     @patch('ansible.plugins.lookup.password._write_password_file')
     def test_password_already_created_encrypt(self, mock_get_paths, mock_write_file):
         mock_get_paths.return_value = ['/path/one', '/path/two', '/path/three']
-        password.os.path.exists = lambda x: True
+        password.os.path.exists = lambda x: x == to_bytes('/path/to/somewhere')
 
         with patch.object(builtins, 'open', mock_open(read_data=b'hunter42 salt=87654321\n')) as m:
             results = self.password_lookup.run([u'/path/to/somewhere chars=anything encrypt=pbkdf2_sha256'], None)
@@ -436,7 +449,7 @@ def test_password_already_created_encrypt(self, mock_get_paths, mock_write_file)
     @patch('ansible.plugins.lookup.password._write_password_file')
     def test_password_already_created_no_encrypt(self, mock_get_paths, mock_write_file):
         mock_get_paths.return_value = ['/path/one', '/path/two', '/path/three']
-        password.os.path.exists = lambda x: True
+        password.os.path.exists = lambda x: x == to_bytes('/path/to/somewhere')
 
         with patch.object(builtins, 'open', mock_open(read_data=b'hunter42 salt=87654321\n')) as m:
             results = self.password_lookup.run([u'/path/to/somewhere chars=anything'], None)
@@ -452,3 +465,27 @@ def test_only_a(self, mock_get_paths, mock_write_file):
         results = self.password_lookup.run([u'/path/to/somewhere chars=a'], None)
         for result in results:
             self.assertEquals(result, u'a' * password.DEFAULT_LENGTH)
+
+    def test_lock_been_held(self):
+        # pretend the lock file is here
+        password.os.path.exists = lambda x: True
+        try:
+            with patch.object(builtins, 'open', mock_open(read_data=b'hunter42 salt=87654321\n')) as m:
+                # should timeout here
+                results = self.password_lookup.run([u'/path/to/somewhere chars=anything'], None)
+                self.fail("Lookup didn't timeout when lock already been held")
+        except AnsibleError:
+            pass
+
+    def test_lock_not_been_held(self):
+        # pretend now there is password file but no lock
+        password.os.path.exists = lambda x: x == to_bytes('/path/to/somewhere')
+        try:
+            with patch.object(builtins, 'open', mock_open(read_data=b'hunter42 salt=87654321\n')) as m:
+                # should not timeout here
+                results = self.password_lookup.run([u'/path/to/somewhere chars=anything'], None)
+        except AnsibleError:
+            self.fail('Lookup timeouts when lock is free')
+
+        for result in results:
+            self.assertEqual(result, u'hunter42')