Repository: ansible/ansible

Issue:
Atomic_move issue on python3 with /tmp on tmpfs
##### ISSUE TYPE
- Bug Report
##### COMPONENT NAME

modules_utils/basic.py

<!--- Name of the plugin/task/feature -->
##### ANSIBLE VERSION

```
ansible 2.3.0 (devel 3098cc2eb9) last updated 2016/10/24 08:45:16 (GMT +200)
  lib/ansible/modules/core: (detached HEAD ce44f0a0ae) last updated 2016/10/22 20:00:33 (GMT +200)
  lib/ansible/modules/extras: (detached HEAD 89a8c18c6a) last updated 2016/10/22 20:00:33 (GMT +200)
  config file = /home/misc/.ansible.cfg
  configured module search path = ['/home/misc/checkout/git/ansible-modules-core', '/home/misc/checkout/git/ansible-modules-extras']
```
##### OS / ENVIRONMENT

Local: RHEL 7
Remote: Freebsd 11, with python 3

/tmp is mounted with a tmpfs
##### SUMMARY

Using blockinfile, or copy on the aformentioned setup result into this traceback:

```
Traceback (most recent call last):
  File \"/tmp/ansible_gtdr4scw/ansible_modlib.zip/ansible/module_utils/basic.py\", line 1942, in atomic_move
OSError: [Errno 18] Cross-device link: b'/tmp/ansible-tmp-1477294473.69-143769266568884/source' -> b'/home/misc/toto'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/tmp/ansible_gtdr4scw/ansible_module_copy.py\", line 364, in <module>
    main()
  File \"/tmp/ansible_gtdr4scw/ansible_module_copy.py\", line 343, in main
    module.atomic_move(b_mysrc, dest, unsafe_writes=module.params['unsafe_writes'])
  File \"/tmp/ansible_gtdr4scw/ansible_modlib.zip/ansible/module_utils/basic.py\", line 1959, in atomic_move
  File \"/usr/local/lib/python3.4/tempfile.py\", line 278, in mkstemp
    return _mkstemp_inner(dir, prefix, suffix, flags)
  File \"/usr/local/lib/python3.4/tempfile.py\", line 205, in _mkstemp_inner
    file = _os.path.join(dir, pre + name + suf)
TypeError: can't concat bytes to str
```

I can be reproduce with this playbook

```
- hosts: all
  tasks:
  - copy:
      dest: /home/misc/toto
      content: toto
```

Without the /tmp on tmpfs, it work fine. And I am testing this on Linux to see if it fail too.



Discussion:
Ahhh... I think python-3.4 is the problem.  IIRC we finally decided to make python3.5 our default because of this bug in tempfile.mkstemp

@abadger Do we need to:
- Document this
- Detect if we are running on < py3.5

It's documented here: https://github.com/ansible/ansible/blob/devel/docsite/rst/dev_guide/developing_modules_python3.rst  but unfortunately it seems that it is not live yet.  pinged dharmabumstead on slack to see if we can build the docs site.

Detection is an open question.  We don't detect incompatible python2 versions.  bcoca and I talked about it and it would be dead simple to hardcode a check into `AnsibleModule.__init__`.  Making it flexible (like allowing us to specify a module supports only python2.6+ or that a module supports python2 but not python3) is also be possible but requires some thought.  We had talked about doing that controller-side via metadata before but maybe we don't need to do that; AnsibleModule is good enough.

This is the upstream bug for tempfile with bytes that was fixed in python-3.5: http://bugs.python.org/issue24230

So, what is the status of 3.5 in non EOL distribution ? 

For freebsd, python 3.5 is here (albeit not on my arm board), and python 2 is still here. RHEL/Centos is still python 2, and 3.5 will likely be ok for next release. Fedora is 3.5 on 24 and 25, so soon on all supported release. 

Debian has 3.5 in testing, and python 2 is still here. Centos/RHEL are on python 2 for now. Arch is 3.5, Gentos is 3.5. So the only issue would be Ubuntu, which is 3.4 on trusty (but still python2 by default), or 3.5 on xenial (latest LTS). 

SO I suspect the only people using python 3 would be either: 
- be able to use python2 without trouble (because installed by default)
- have a default version that work.

Given that python 3 is still experimental, I am not sure we need to add a check for that, especially since that requires specific configuration (ie, /tmp in tmpfs). Maybe we could just catch the execption and show the warning, rather than testing it by default in **init** ? 


Resolving patch:
diff --git a/lib/ansible/module_utils/basic.py b/lib/ansible/module_utils/basic.py
--- a/lib/ansible/module_utils/basic.py
+++ b/lib/ansible/module_utils/basic.py
@@ -1960,13 +1960,21 @@ def atomic_move(self, src, dest, unsafe_writes=False):
                 except (OSError, IOError):
                     e = get_exception()
                     self.fail_json(msg='The destination directory (%s) is not writable by the current user. Error was: %s' % (os.path.dirname(dest), e))
+                except TypeError:
+                    # We expect that this is happening because python3.4.x and
+                    # below can't handle byte strings in mkstemp().  Traceback
+                    # would end in something like:
+                    #     file = _os.path.join(dir, pre + name + suf)
+                    # TypeError: can't concat bytes to str
+                    self.fail_json(msg='Failed creating temp file for atomic move.  This usually happens when using Python3 less than Python3.5.  Please use Python2.x or Python3.5 or greater.', exception=sys.exc_info())
+
                 b_tmp_dest_name = to_bytes(tmp_dest_name, errors='surrogate_or_strict')
 
                 try:
                     try:
                         # close tmp file handle before file operations to prevent text file busy errors on vboxfs synced folders (windows host)
                         os.close(tmp_dest_fd)
-                        # leaves tmp file behind when sudo and  not root
+                        # leaves tmp file behind when sudo and not root
                         if switched_user and os.getuid() != 0:
                             # cleanup will happen by 'rm' of tempdir
                             # copy2 will preserve some metadata


Test patch: