Repository: ansible/ansible

Issue:
[1.9.2] race in initial creation of accelerated keys dir with multiple target hosts
As described at https://groups.google.com/forum/#!topic/ansible-project/Wh0zm1ZY5l8 there is a race in how accelerate mode attempts to create the ACCELERATE_KEYS_DIR path when it doesn't exist and executing against multiple target hosts.

The problem is at https://github.com/ansible/ansible/blob/stable-1.9/lib/ansible/utils/__init__.py#L130-L131

```
    key_path = os.path.expanduser(C.ACCELERATE_KEYS_DIR)
    if not os.path.exists(key_path):
        os.makedirs(key_path, mode=0700)
```

When running with multiple target hosts, each fork will run through this section of code. It is possible for each of the forks to evaluate the initial conditional before any have executed and returned from `os.makedirs(key_path, mode=0700)`, this means that for one fork, it will succeed in completing this block, while the others may fail. On subsequent runs, the directory now exists, so until it is deleted, or the play is run again in a pristine environment, the error will disappear.

Possible solutions include using a lock, or simply wrapping the call in a try/except that swallons any EEXIST errors.

While also less likely for the actual file used to store the per-host key:
https://github.com/ansible/ansible/blob/stable-1.9/lib/ansible/utils/__init__.py#L142-L147

It still remains possible for a similar race, when using delegate_to, to attempt to open and write the file multiple times thus resulting in the key being generated and used correctly sometimes and throwing an exception in others about trying to create the file multiple times when it already exists. A lock would probably be the safest thing here, as would want to wait until the file was created and could be read before allowing other forks to continue to read it's contents in parallel.

Alternatively a lock around the `utils.key_for_hostname(host)` at https://github.com/ansible/ansible/blob/stable-1.9/lib/ansible/runner/connection_plugins/accelerate.py#L48 also appears as though it would solve the problem.

[1.9.2] race in initial creation of accelerated keys dir with multiple target hosts
As described at https://groups.google.com/forum/#!topic/ansible-project/Wh0zm1ZY5l8 there is a race in how accelerate mode attempts to create the ACCELERATE_KEYS_DIR path when it doesn't exist and executing against multiple target hosts.

The problem is at https://github.com/ansible/ansible/blob/stable-1.9/lib/ansible/utils/__init__.py#L130-L131

```
    key_path = os.path.expanduser(C.ACCELERATE_KEYS_DIR)
    if not os.path.exists(key_path):
        os.makedirs(key_path, mode=0700)
```

When running with multiple target hosts, each fork will run through this section of code. It is possible for each of the forks to evaluate the initial conditional before any have executed and returned from `os.makedirs(key_path, mode=0700)`, this means that for one fork, it will succeed in completing this block, while the others may fail. On subsequent runs, the directory now exists, so until it is deleted, or the play is run again in a pristine environment, the error will disappear.

Possible solutions include using a lock, or simply wrapping the call in a try/except that swallons any EEXIST errors.

While also less likely for the actual file used to store the per-host key:
https://github.com/ansible/ansible/blob/stable-1.9/lib/ansible/utils/__init__.py#L142-L147

It still remains possible for a similar race, when using delegate_to, to attempt to open and write the file multiple times thus resulting in the key being generated and used correctly sometimes and throwing an exception in others about trying to create the file multiple times when it already exists. A lock would probably be the safest thing here, as would want to wait until the file was created and could be read before allowing other forks to continue to read it's contents in parallel.

Alternatively a lock around the `utils.key_for_hostname(host)` at https://github.com/ansible/ansible/blob/stable-1.9/lib/ansible/runner/connection_plugins/accelerate.py#L48 also appears as though it would solve the problem.



Discussion:

Resolving patch:
diff --git a/lib/ansible/runner/connection_plugins/accelerate.py b/lib/ansible/runner/connection_plugins/accelerate.py
--- a/lib/ansible/runner/connection_plugins/accelerate.py
+++ b/lib/ansible/runner/connection_plugins/accelerate.py
@@ -21,6 +21,7 @@
 import socket
 import struct
 import time
+import threading
 from ansible.callbacks import vvv, vvvv
 from ansible.errors import AnsibleError, AnsibleFileNotFound
 from ansible.runner.connection_plugins.ssh import Connection as SSHConnection
@@ -35,6 +36,8 @@
 # multiple of the value to speed up file reads.
 CHUNK_SIZE=1044*20
 
+_LOCK = threading.Lock()
+
 class Connection(object):
     ''' raw socket accelerated connection '''
 
@@ -111,6 +114,15 @@ def _execute_accelerate_module(self):
     def connect(self, allow_ssh=True):
         ''' activates the connection object '''
 
+        # ensure only one fork tries to setup the connection, in case the
+        # first task for multiple hosts is delegated to the same host.
+        if not self.is_connected:
+            with(_LOCK):
+                return self._connect(allow_ssh)
+
+        return self
+
+    def _connect(self, allow_ssh=True):
         try:
             if not self.is_connected:
                 wrong_user = False
@@ -150,7 +162,7 @@ def connect(self, allow_ssh=True):
                 res = self._execute_accelerate_module()
                 if not res.is_successful():
                     raise AnsibleError("Failed to launch the accelerated daemon on %s (reason: %s)" % (self.host,res.result.get('msg')))
-                return self.connect(allow_ssh=False)
+                return self._connect(allow_ssh=False)
             else:
                 raise AnsibleError("Failed to connect to %s:%s" % (self.host,self.accport))
         self.is_connected = True
diff --git a/lib/ansible/utils/__init__.py b/lib/ansible/utils/__init__.py
--- a/lib/ansible/utils/__init__.py
+++ b/lib/ansible/utils/__init__.py
@@ -50,6 +50,8 @@
 import sys
 import subprocess
 import contextlib
+import threading
+import tempfile
 
 from vault import VaultLib
 
@@ -63,6 +65,7 @@
 PRINT_CODE_REGEX = re.compile(r'(?:{[{%]|[%}]})')
 CODE_REGEX = re.compile(r'(?:{%|%})')
 
+_LOCK = threading.Lock()
 
 try:
     # simplejson can be much faster if it's available
@@ -128,8 +131,15 @@ def key_for_hostname(hostname):
 
     key_path = os.path.expanduser(C.ACCELERATE_KEYS_DIR)
     if not os.path.exists(key_path):
-        os.makedirs(key_path, mode=0700)
-        os.chmod(key_path, int(C.ACCELERATE_KEYS_DIR_PERMS, 8))
+        # avoid race with multiple forks trying to create paths on host
+        # but limit when locking is needed to creation only
+        with(_LOCK):
+            if not os.path.exists(key_path):
+                # use a temp directory and rename to ensure the directory
+                # searched for only appears after permissions applied.
+                tmp_dir = tempfile.mkdtemp(dir=os.path.dirname(key_path))
+                os.chmod(tmp_dir, int(C.ACCELERATE_KEYS_DIR_PERMS, 8))
+                os.rename(tmp_dir, key_path)
     elif not os.path.isdir(key_path):
         raise errors.AnsibleError('ACCELERATE_KEYS_DIR is not a directory.')
 
@@ -140,19 +150,25 @@ def key_for_hostname(hostname):
 
     # use new AES keys every 2 hours, which means fireball must not allow running for longer either
     if not os.path.exists(key_path) or (time.time() - os.path.getmtime(key_path) > 60*60*2):
-        key = AesKey.Generate(size=256)
-        fd = os.open(key_path, os.O_WRONLY | os.O_CREAT, int(C.ACCELERATE_KEYS_FILE_PERMS, 8))
-        fh = os.fdopen(fd, 'w')
-        fh.write(str(key))
-        fh.close()
-        return key
-    else:
-        if stat.S_IMODE(os.stat(key_path).st_mode) != int(C.ACCELERATE_KEYS_FILE_PERMS, 8):
-            raise errors.AnsibleError('Incorrect permissions on the key file for this host. Use `chmod 0%o %s` to correct this issue.' % (int(C.ACCELERATE_KEYS_FILE_PERMS, 8), key_path))
-        fh = open(key_path)
-        key = AesKey.Read(fh.read())
-        fh.close()
-        return key
+        # avoid race with multiple forks trying to create key
+        # but limit when locking is needed to creation only
+        with(_LOCK):
+            if not os.path.exists(key_path) or (time.time() - os.path.getmtime(key_path) > 60*60*2):
+                key = AesKey.Generate()
+                # use temp file to ensure file only appears once it has
+                # desired contents and permissions
+                with tempfile.NamedTemporaryFile(mode='w', dir=os.path.dirname(key_path), delete=False) as fh:
+                    tmp_key_path = fh.name
+                    fh.write(str(key))
+                os.chmod(tmp_key_path, int(C.ACCELERATE_KEYS_FILE_PERMS, 8))
+                os.rename(tmp_key_path, key_path)
+                return key
+
+    if stat.S_IMODE(os.stat(key_path).st_mode) != int(C.ACCELERATE_KEYS_FILE_PERMS, 8):
+        raise errors.AnsibleError('Incorrect permissions on the key file for this host. Use `chmod 0%o %s` to correct this issue.' % (int(C.ACCELERATE_KEYS_FILE_PERMS, 8), key_path))
+
+    with open(key_path) as fh:
+        return AesKey.Read(fh.read())
 
 def encrypt(key, msg):
     return key.Encrypt(msg.encode('utf-8'))


Test patch: