Repository: ansible/ansible

Issue:
OSError: [Errno 2] No such file or directory: xxx/ziploader_cache
##### ISSUE TYPE
- Bug Report
##### ANSIBLE VERSION

```
ansible 2.1.0.0
  config file = /home/ansible/infra-ansible-configuration/ansible.cfg
  configured module search path = Default w/o overrides
```
##### CONFIGURATION

```
[defaults]
host_key_checking = False
lookup_plugins = plugins/lookup/
gathering = smart
fact_caching = redis
fact_caching_timeout = 3600

[ssh_connection]
control_path = %(directory)s/%%h-%%r
ssh_args = -o ControlMaster=yes
```
##### OS / ENVIRONMENT

```
Ubuntu 14.04.4
Linux xxx 3.19.0-61-generic #69~14.04.1-Ubuntu SMP Thu Jun 9 09:09:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
```
##### SUMMARY

When using any module we sometimes - about half the time in fact, get the error shown below. Sometimes this will happen many times in a row until finally the run works.
##### STEPS TO REPRODUCE

<!---
For bugs, show exactly how to reproduce the problem.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->

Just a simple copy of a local file to remote will trigger this on my systems about half the time.

<!--- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS

I expect the file to be coped over.
##### ACTUAL RESULTS

<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->

```
fatal: [REDACTED]: FAILED! => {"failed": true, "msg": "Unexpected failure during module execution.", "stdout": ""}
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 124, in run
    res = self._execute()
  File "/usr/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 446, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/dist-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/dist-packages/ansible/plugins/action/__init__.py", line 583, in _execute_module
    (module_style, shebang, module_data) = self._configure_module(module_name=module_name, module_args=module_args, task_vars=task_vars)
  File "/usr/lib/python2.7/dist-packages/ansible/plugins/action/__init__.py", line 149, in _configure_module
    (module_data, module_style, module_shebang) = modify_module(module_name, module_path, module_args, task_vars=task_vars, module_compression=self._play_context.module_compression)
  File "/usr/lib/python2.7/dist-packages/ansible/executor/module_common.py", line 718, in modify_module
    (module_data, module_style, shebang) = _find_snippet_imports(module_name, module_data, module_path, module_args, task_vars, module_compression)
  File "/usr/lib/python2.7/dist-packages/ansible/executor/module_common.py", line 586, in _find_snippet_imports
    os.mkdir(lookup_path)
OSError: [Errno 2] No such file or directory: '/home/ansible/.ansible/tmp/ansible-local-tmpvqsFYh/ziploader_cache'
```



Discussion:
@khushil 
1. how are you invoking ansible-playbook / ansible? 
2. Is it running in a for loop or a cronjob?
3. On average, how many runs before the error occurs?

- And how are you installing Ansible?  Is it from the Ansible Ubuntu PPA?  (I fixed a bug like this in 2.1.0rc2 so just want to make sure you aren't getting code from prior to that fix).
- In the play where this is happening, is localhost one of the hosts that is being run against?
- Can you give us more information about your playbook/config/etc?  From what you've said so far this seems to be a race condition so it isn't happening all the time.  But it is odd that you're seeing it 50% of the time and we haven't seen it once.  Maybe there's some configuration setting or other thing about your environment that we need to do to in order to reproduce this in our testing.
- If possible, can you use ANSIBLE_DEBUG=1 to run your task and capture all of the output to a file.  If the task fails, it would be helpful for us to see this verbose output to see what's happening in your run (It will be a large amount of information so uploading the file somewhere or putting it in a gist would be best).

Thanks for the additional info.

We run via ansible-playbook

Not running in a loop or CRON but I notice this happen most during busy parts of the day when mutiple playbooks will be running at the same time as Ansible is part of our CI/CD process.

Busy times it can fail consecutive times before it works. Sometimes just runs.

We're using Ansible from the Ubuntu PPA.

Localhost isn't one of the hosts the play is being run against.

It's actually no happening way more frequently. Pretty standard system.  Typical playbook gets a dynamic inventory from ec2.py and runs against a tag_ansible_group sub-group. Is there anyway to change where those temp files and cache directories are created? Sometime like local_temp?

I assume your referring to https://docs.ansible.com/ansible/playbooks_debugger.html

@khushil Actually, it's just an environment variable that turns on _very_ verbose output from ansible.  If you're running as part of a shell script, it would be something like:
  ANSIBLE_DEBUG=1 ansible-playbook my_playbook.yml &> logfile.$$

(Or if your CI is already capturing all the output from running ansible-playbook, you don't need the shell redirection at the end).

Note that this output is very verbose so you don't want to run with it in general.  Just enough to capture what's failing.

@khushil Since you're running it from CI/CD I have a question -- what kind of environment and cleanup is your CI/CD system doing?  Is each ansible run in an isolated environment or do they share localhost?  Is there a part of your CI/CD that tries to cleanup the ansible directories?  Maybe something that rm -rf's /home/ansible/ or one of it's subdirectories?

@khushil If you're unsure about whether anything is cleaning up the /home/ansible directories, one thing to try might be setting `local_tmp      = /tmp/foo_directory` in the ansible.cfg file.  Since that's an unexpected directory it probably won't get cleaned up by any potential current scripts.

@khushil we encountered this issue and AFAICT it was due to Jenkins and custom workspaces, just curious to see what your setup is.

@abadger we tried setting `local_tmp = /tmp` a while ago which does leave directories and cache items behind.

```
drwxrwxr-x 2 ansible ansible 4096 Jun 30 13:58 /tmp/ansible-local-tmpWxvk9U/ziploader_cache
drwxrwxr-x 2 ansible ansible 4096 Jun 30 13:57 /tmp/ansible-local-tmpx7fvr3/ziploader_cache
drwxrwxr-x 2 ansible ansible 4096 Jun 30 11:31 /tmp/ansible-local-tmpxshsOS/ziploader_cache
drwxrwxr-x 2 ansible ansible 4096 Jun 30 12:35 /tmp/ansible-local-tmpyxomJZ/ziploader_cache
drwxrwxr-x 2 ansible ansible 4096 Jun 30 12:31 /tmp/ansible-local-tmpzI7ncI/ziploader_cache
drwxrwxr-x 2 ansible ansible 4096 Jun 30 12:25 /tmp/ansible-local-tmpZm0ywa/ziploader_cache
```

Our assumption here was that the entropy used by `tempfile.mkdtemp()` is not enough and it starts colliding.
To prove this assumption I did force the cache directory to be unique for each execution by including the current PID and a timestamp in the name like:
`/tmp/ansible-3306-1467366110.27`

It is configured via `local_tmp = /tmp/ansible-$ANSIBLE_PID-$ANSIBLE_EPOCH/` where the two ENV variables are set before it reads in the `constants.py` library file.

Since we run it with this "hack" it didn't fall over again. But since there was a weekend in between the testing I'll watch it over the day now before calling it.

@acaire we're triggering a run via Jenkins. But it is called via ssh on a remote host so it should be not bound to Jenkins in this problem case.

@MerlinDMC Hmm... `tempfile.mkdtemp()` shouldn't collide.  It does an `os._mkdir()` which will fail if the file or directory is already used rather than returning an in-use name.  I could add the PID as part of the temp dir prefix if your use of the PID in the environment variable proves to be what's needed to fix this, though.  (ie: the default path would then look like: /home/ansible/.ansible/tmp/ansible-local-16705-tmpvqsFYh/ziploader_cache  where 16705 is the PID of this run of ansible.)

@MerlinDMC and to be clear... My understanding is that:
-  Simply setting `local_tmp =/tmp`       Did not resolve the issue.
- If you use `local_tmp = /tmp/ansible-$ANSIBLE_PID` (without the ANSIBLE_EPOCH) it does resolve the issue?

@abadger I'll run it again over the day tomorrow without the epoch.

Yes I agree that it should not collide. But the fact that it didn't break again over a couple of days is still interesting.

I'm unsure on how to try to make this a completely repeatable failure case as well.

@MerlinDMC Yeah, I agree.  I just want to make sure that adding the PID inside of ansible code will fix things and it's not something different like /home/ansible/.ansible/tmp/ being deleted out from under us that's the problem.


Resolving patch:
diff --git a/lib/ansible/constants.py b/lib/ansible/constants.py
--- a/lib/ansible/constants.py
+++ b/lib/ansible/constants.py
@@ -76,7 +76,8 @@ def get_config(p, section, key, env_var, default, boolean=False, integer=False,
             value = shell_expand(value)
             if not os.path.exists(value):
                 os.makedirs(value, 0o700)
-            value = tempfile.mkdtemp(prefix='ansible-local-tmp', dir=value)
+            prefix = 'ansible-local-%s' % os.getpid()
+            value = tempfile.mkdtemp(prefix=prefix, dir=value)
         elif ispathlist:
             if isinstance(value, string_types):
                 value = [shell_expand(x, expand_relative_paths=expand_relative_paths) \


Test patch: