Repository: ansible/ansible

Issue:
[1.9.2] race in initial creation of accelerated keys dir with multiple target hosts
As described at https://groups.google.com/forum/#!topic/ansible-project/Wh0zm1ZY5l8 there is a race in how accelerate mode attempts to create the ACCELERATE_KEYS_DIR path when it doesn't exist and executing against multiple target hosts.

The problem is at https://github.com/ansible/ansible/blob/stable-1.9/lib/ansible/utils/__init__.py#L130-L131

```
    key_path = os.path.expanduser(C.ACCELERATE_KEYS_DIR)
    if not os.path.exists(key_path):
        os.makedirs(key_path, mode=0700)
```

When running with multiple target hosts, each fork will run through this section of code. It is possible for each of the forks to evaluate the initial conditional before any have executed and returned from `os.makedirs(key_path, mode=0700)`, this means that for one fork, it will succeed in completing this block, while the others may fail. On subsequent runs, the directory now exists, so until it is deleted, or the play is run again in a pristine environment, the error will disappear.

Possible solutions include using a lock, or simply wrapping the call in a try/except that swallons any EEXIST errors.

While also less likely for the actual file used to store the per-host key:
https://github.com/ansible/ansible/blob/stable-1.9/lib/ansible/utils/__init__.py#L142-L147

It still remains possible for a similar race, when using delegate_to, to attempt to open and write the file multiple times thus resulting in the key being generated and used correctly sometimes and throwing an exception in others about trying to create the file multiple times when it already exists. A lock would probably be the safest thing here, as would want to wait until the file was created and could be read before allowing other forks to continue to read it's contents in parallel.

Alternatively a lock around the `utils.key_for_hostname(host)` at https://github.com/ansible/ansible/blob/stable-1.9/lib/ansible/runner/connection_plugins/accelerate.py#L48 also appears as though it would solve the problem.



Discussion:

Resolving patch:
diff --git a/lib/ansible/utils/encrypt.py b/lib/ansible/utils/encrypt.py
--- a/lib/ansible/utils/encrypt.py
+++ b/lib/ansible/utils/encrypt.py
@@ -20,6 +20,8 @@
 
 import os
 import stat
+import tempfile
+import multiprocessing
 import time
 import warnings
 
@@ -68,6 +70,9 @@
 
 __all__ = ['do_encrypt']
 
+_LOCK = multiprocessing.Lock()
+
+
 def do_encrypt(result, encrypt, salt_size=None, salt=None):
     if PASSLIB_AVAILABLE:
         try:
@@ -95,8 +100,15 @@ def key_for_hostname(hostname):
 
     key_path = os.path.expanduser(C.ACCELERATE_KEYS_DIR)
     if not os.path.exists(key_path):
-        os.makedirs(key_path, mode=0o700)
-        os.chmod(key_path, int(C.ACCELERATE_KEYS_DIR_PERMS, 8))
+        # avoid race with multiple forks trying to create paths on host
+        # but limit when locking is needed to creation only
+        with(_LOCK):
+            if not os.path.exists(key_path):
+                # use a temp directory and rename to ensure the directory
+                # searched for only appears after permissions applied.
+                tmp_dir = tempfile.mkdtemp(dir=os.path.dirname(key_path))
+                os.chmod(tmp_dir, int(C.ACCELERATE_KEYS_DIR_PERMS, 8))
+                os.rename(tmp_dir, key_path)
     elif not os.path.isdir(key_path):
         raise AnsibleError('ACCELERATE_KEYS_DIR is not a directory.')
 
@@ -107,19 +119,26 @@ def key_for_hostname(hostname):
 
     # use new AES keys every 2 hours, which means fireball must not allow running for longer either
     if not os.path.exists(key_path) or (time.time() - os.path.getmtime(key_path) > 60*60*2):
-        key = AesKey.Generate(size=256)
-        fd = os.open(key_path, os.O_WRONLY | os.O_CREAT, int(C.ACCELERATE_KEYS_FILE_PERMS, 8))
-        fh = os.fdopen(fd, 'w')
-        fh.write(str(key))
-        fh.close()
-        return key
-    else:
-        if stat.S_IMODE(os.stat(key_path).st_mode) != int(C.ACCELERATE_KEYS_FILE_PERMS, 8):
-            raise AnsibleError('Incorrect permissions on the key file for this host. Use `chmod 0%o %s` to correct this issue.' % (int(C.ACCELERATE_KEYS_FILE_PERMS, 8), key_path))
-        fh = open(key_path)
-        key = AesKey.Read(fh.read())
-        fh.close()
-        return key
+        # avoid race with multiple forks trying to create key
+        # but limit when locking is needed to creation only
+        with(_LOCK):
+            if not os.path.exists(key_path) or (time.time() - os.path.getmtime(key_path) > 60*60*2):
+                key = AesKey.Generate()
+                # use temp file to ensure file only appears once it has
+                # desired contents and permissions
+                with tempfile.NamedTemporaryFile(mode='w', dir=os.path.dirname(key_path), delete=False) as fh:
+                    tmp_key_path = fh.name
+                    fh.write(str(key))
+                os.chmod(tmp_key_path, int(C.ACCELERATE_KEYS_FILE_PERMS, 8))
+                os.rename(tmp_key_path, key_path)
+                return key
+
+    if stat.S_IMODE(os.stat(key_path).st_mode) != int(C.ACCELERATE_KEYS_FILE_PERMS, 8):
+        raise AnsibleError('Incorrect permissions on the key file for this host. Use `chmod 0%o %s` to correct this issue.' % (int(C.ACCELERATE_KEYS_FILE_PERMS, 8), key_path))
+    fh = open(key_path)
+    key = AesKey.Read(fh.read())
+    fh.close()
+    return key
 
 def keyczar_encrypt(key, msg):
     return key.Encrypt(msg.encode('utf-8'))


Test patch: