Repository: ansible/ansible

Issue:
file operations fail on shared/synced folders that do not implement correct filesystem semantics
##### Issue Type:

“Bug Report”, 
##### Ansible Version:

1.7.2
##### Environment:

Ubuntu 14.04.1 VM on Virtual Box 4.3.14  on Windows 8.1 
I know the issue is likely specific to vboxfs 
##### Summary:

The copy modules fails on virtual box shared folders. specifically the os.rename() fails.  I tried replacing the os.rename with shutils.copy2 and it worked fine.  

I notice there is a lot of working being done in atomic_move. There is a fallback strategy that seems to attempt to copy the source file to a temp file in the destination directory, and then rename the temp file. Why not just do the copy straight to the destination and let cleanup() handle the stuff in the temp folder?

I tested a few scenarios if I try to copy to some place besides the shared folder, the module works. So the source doesn't appear to be what is blocking the rename, but it's the destination. I tried hard coding an alternate destination at the os.rename, in case a previous operation was taking a lock, but still kept receiving error 26.

When I try opening up the repl, I get a slightly different error, but an error none the same..

> > > import os
> > > os.rename('test.txt', '/vagrant/test.txt')
> > > Traceback (most recent call last):
> > >   File "<stdin>", line 1, in <module>
> > > OSError: [Errno 18] Invalid cross-device link

It looks like os.rename doesn't work across partitions, and this is the root of the problem. (maybe this is why it tries to copy to the save folder as the destination, then renames).

https://mail.python.org/pipermail/python-list/2005-February/342893.html
https://bugs.launchpad.net/python-keystoneclient/+bug/1265190
##### Steps To Reproduce:

install virtualbox. 

put all the files from  https://gist.github.com/dopry/4bc4f2d852b8055a7b68 in a folder.

vagrant up 
vagrant ssh 

from inside the vm..

echo "test" > test.txt
ansible web -i /vagrant/ansible/inventory/vagrant/hosts.py -m copy -a "src=test.txt dest=/vagrant/test.txt"
##### Expected Results:

test.txt is copied to  /vagrant.
##### Actual Results:

vagrant.dev | FAILED >> {
    "failed": true,
    "md5sum": "bbfa1f311a5828452b953d1335cbf027",
    "msg": "could not cleanup /vagrant/.ansible_tmpE8ytJhtest.txt: [Errno 26] Text file busy: '/vagrant/.ansible_tmpE8ytJhtest.txt'{\"msg\": \"Could not replace file: /home/vagrant/.ansible/tmp/ansible-tmp-1415586483.17-87111780037684/source to /vagrant/test.txt: [Errno 26] Text file busy\", \"failed\": true}\r\n",
    "parsed": false
}



Discussion:
If someone else is experiencing this problem I worked around it by copying to the remote home then moving with command.  (the ansible tmp dir is in the remote home, I assume it will generally be the same partition)
- copy:   src="test.txt" dest= "test.txt"
- command: mv test.txt /vagrant/test.txt"

There is no win_copy module at this time, so this is why the module cannot be used on Windows.

It seems we should instate a check to make sure modules implemented in Python don't get attempted to be executed in this way.

This has nothing to do with win_copy or connecting to a Windows host; it appears to be an issue when copying a file to a Linux host, where the destination is on a virtualbox shared folder.

Ansible is running in and connecting to a linux VM. The VM is running on
windows.  It's trying to copy a file from the VM's root filesystem to the
/vagrant folder. the /vagrant folder has a vboxfs filesystem mounted to it.
os.rename seems to fail when copying from the root to the vboxsf partition.
It also fails when renaming a file on the vboxsf partition. If you want I
can setup a repo just to demonstrate the bug for you. I haven't confirmed
if this happens if linux or mac are hosting the VM.

On Mon, Nov 10, 2014 at 3:55 PM, Michael DeHaan notifications@github.com
wrote:

> There is no win_copy module at this time, so this is why the module cannot
> be used on Windows.
> 
> It seems we should instate a check to make sure modules implemented in
> Python don't get attempted to be executed in this way.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/ansible/ansible/issues/9526#issuecomment-62453258.

## 

Darrel O'Pry
The Spry Group, LLC.
http://www.spry-group.com
718-355-9767 x101

Thank you @dopry for this information! This problem has been driving me mad for about the past three hours. Win7 x64, VirtualBox 4.3.20, Linux guest Ubuntu 14.04, synced_folder, Ansible running within the guest, it can't copy one particular file from Linux to vboxfs.

```
==> default: <default> PUT /tmp/tmpGKnnSx TO /root/.ansible/tmp/ansible-tmp-1417932523.39-220512482662826/source ==> default: <default> PUT /tmp/tmp4eB2Z2 TO /root/.ansible/tmp/ansible-tmp-1417932523.39-220512482662826/copy
==> default: <default> EXEC ['/bin/sh', '-c', u'LANG=C LC_CTYPE=C /usr/bin/python /root/.ansible/tmp/ansible-tmp-1417932523.39-220512482662826/copy; rm -rf /root/.ansible/tmp/ansible-tmp-1417932523.39-220512482662826/ >/dev/null 2>&1']
==> default: failed: [default] => {"failed": true}
==> default: msg: Could not replace file: /root/.ansible/tmp/ansible-tmp-1417932523.39-220512482662826/source to /var/www/sites/local.example.com/settings.php: [Errno 26] Text file busy
==> default:
==> default: FATAL: all hosts have already failed -- aborting
111
```

Fetch and template are not equivalent. Fetch does not interpolate variables
and expressions.
On Dec 8, 2014 2:33 AM, "fk0" notifications@github.com wrote:

> Hi, I've run into this issue and was able to work it around by just
> replacing
> template: src=/path/to/srcfiles dst=/path/to/dstfiles
> by
> fetch: src=/path/to/srcfiles dst=/path/to/dstfiles
> 
> hope it help somebody
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/ansible/ansible/issues/9526#issuecomment-66033222.

having a similar problem copying file from local client to remote server, which is using a network attached storage device - using **ansible 1.8.2** -

``` yml
    - name: upload ping file
      copy: src="templates/ping.html" dest="/mnt/files"
```

fails with -

> Could not replace file: /home/me/.ansible/tmp/ansible-tmp-1420234876.35-71225400803213/source to /mnt/files/ping.html: [Errno 95] Operation not supported

but if I do a `copy` to remote server and then manually `move` it to the network storage, it works fine -

``` yml
    - name: upload ping file
      copy: src="templates/ping.html" dest="/tmp"
    - name: move ping file
      command: mv /tmp/ping.html /mnt/files
```

(The downside is more complexity in the playbook to make this idempotent)

I'm having the exact same issue but using the `template` module. Any ideas for at least a workaround using the `template` module?

Having a very similar issue with 1.8.2 on a Ubuntu 14.04.1 virtual machine running in Virtualbox, while using the `template` module.

The destination path is an NFS share exposed in the VM via Vagrant. The action actually looks like it succeeds (the file is created with the correct variables populated) but the task reports that it's failing.

In the playbook:

```
- name: create local.settings.php
  template: src=local.settings.php.j2  dest=/var/www/site/docroot/sites/default/local.settings.php mode=0644 backup=yes
  sudo: yes
```

The output:

```
TASK: [post_install | create local.settings.php] ****************** 
failed: [localhost] => {"failed": true, "parsed": false}
Traceback (most recent call last):
  File "/home/vagrant/.ansible/tmp/ansible-tmp-1423143555.31-218436313464603/copy", line 1824, in <module>
    main()
  File "/home/vagrant/.ansible/tmp/ansible-tmp-1423143555.31-218436313464603/copy", line 238, in main
    module.atomic_move(src, dest)
  File "/home/vagrant/.ansible/tmp/ansible-tmp-1423143555.31-218436313464603/copy", line 1624, in atomic_move
    os.chown(dest, os.getuid(), os.getgid())
OSError: [Errno 1] Operation not permitted: '/var/www/site/docroot/sites/default/local.settings.php'
Exception OSError: (2, 'No such file or directory', '/var/www/site/docroot/sites/default/.ansible_tmpp1DIX7local.settings.php') in <bound method _TemporaryFileWrapper.__del__ of <closed file '<fdopen>', mode 'w+b' at 0x7fa03ecbc300>> ignored

FATAL: all hosts have already failed -- aborting
```

does anyone have this issue outside of vagrant synced folders? is is only with the virtualbox backend? and only when physical host is windows?

Most of the file plugins (non win_) use the atomic_move function to ensure consistency and atomicity of its results. This function  requires a posix filesystem (which is an ansible requirement), NTFS is not compliant and can cause issues.

@bcoca yes, I do, I'll give some additional context for my comment above -https://github.com/ansible/ansible/issues/9526#issuecomment-68566699 -

That was trying to copy a file from local OS X machine to remote Ubuntu 14 guest running on VMWare ESXi host. (I don't have specific details about the storage device mounted under `/mnt/files` but could retrieve it, if that would help?)

yes, as I run a full linux stack with virtualbox and am not able to reproduce this problem

Just to add, I was seeing the problem with an Ubuntu host (14.10), virtualbox and an Ubuntu VM (14.04). The vm is provisioned through Vagrant and the destination path is a Vagrant NFS share.

The workaround I had to use was an Ansible shell command to touch the file just before trying to overwrite it with the template module.

```
- name: create empty local.settings.php if required
  shell: cd /var/www/site/docroot/sites/default; touch local.settings.php
  when: not local_settings_php.stat.exists

- name: create local.settings.php
  template: src=local.settings.php.j2  dest=/var/www/site/docroot/sites/default/local.settings.php mode=0644 backup=yes
```

I believe happens when copying across any mount point on a POSIX system.
This is not limited to vagrant or virtual box.
On Feb 6, 2015 10:57 AM, "George Buckerfield" notifications@github.com
wrote:

> Just to add, I was seeing the problem with an Ubuntu host (14.10),
> virtualbox and an Ubuntu VM (14.04). The vm is provisioned through Vagrant
> and the destination path is a Vagrant NFS share.
> 
> The workaround I had to use was an Ansible shell command to touch the file
> just before trying to overwrite it with the template module.
> - name: create empty local.settings.php if required
>   shell: cd /var/www/site/docroot/sites/default; touch local.settings.php
>   when: not local_settings_php.stat.exists
> - name: create local.settings.php
>   template: src=local.settings.php.j2  dest=/var/www/site/docroot/sites/default/local.settings.php mode=0644 backup=yes
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/ansible/ansible/issues/9526#issuecomment-73248376.

@dorpy, the code accounts for that, if a direct rename (which doesn't work across filesystems) fails, it copies a temporary file to the target directory and then does a rename there.

Yeah, but that process fails. I believe I documented that in earlier
comments. It's a bit hard to check my notes now.  I'm on holiday in the
Dominican Republic. I'll be stateside next week and can follow up then if
the examples fail.
On Feb 6, 2015 2:36 PM, "Brian Coca" notifications@github.com wrote:

> @dorpy, the code accounts for that, if a direct rename (which failes
> across filesystems) fails, it copies a temporary file to the target
> directory and then does a rename there.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/ansible/ansible/issues/9526#issuecomment-73288672.

https://bugzilla.gnome.org/show_bug.cgi?id=656225

Same issue with glib.

I'm seeing a similar problem when using the `authorized_key:` module on a `vboxsf`-mounted filesystem: `Could not replace file: /www/default/tmph3jXcn to /www/default/keys: [Errno 26] Text file busy`

I got around the `copy:` problem above by using the `synchronize:` module to sync the `copy:`'ed tree from `/tmp/www/` to `/www/` conditionally `when: ansible_devices.sda is defined and ansible_devices.sda.model == 'VBOX HARDDISK'`

However, the `authorized_keys:` issue might be harder to solve, as there's no equivalent operation... it might have to be a `lineinfile:` instead.

To clarify, that's running Ansible 1.8.4 on Ubuntu Linux (trusty32) inside a Vagrant VM running on Windows 7.  `/www` is a mount of type `vboxsf`. While it is running inside a Windows-hosted VM, Ansible's running in the Linux VM, talking to itself using an `ansible_connection=local` host.  This is to get around the problem of not having Ansible for Windows to provision a Vagrant VM directly.

a gist reproducing the problem: https://gist.github.com/zsoldosp/777a6fd7d242c2c1dabd/ 

(**edit** after @dopry's request for more info)
Host: Win7 x64, Vagrant 1.7.2, VirtualBox 4.3.24 r98716
Guest: It's in the Vagrantfile, ubuntu/trusty64 (14.04), ansible 1.8.2 

Download the gist, and run `vagrant up` (or `vagrant provision` when running multiple times). Currently it gives the ErrNo 26: text file is busy... error

@zsoldosp: what host os/guest os combination are you using?

For me it's on Windows 7 host, Ubuntu 1404 guest 

I seem to be fighting ansible every time I turn around. Running on Windows 7 to a Vagrant with shared folders just seems to be broken on several layers - the 'every file is executable' problem and now writing out simple text files from a template... will it end after this?

@jensenbox ansible was designed with posix systems in mind, if you use non posix systems it is predictable you'll hit some issues. That said, we are working on making it easier for people to do so.

i hope that posix-oriented-ness is not carved in stone and that some day ansible team will reconsider supporting windows :)

we are adding windows as a target, there are no plans to support windows as a controller/master

@bcoca I think I have at least narrowed down the problem to something that is mildly fixable...

So it is in fact a 'cross partition' issue. The template task is creating a temp file of the result in the current users home directory and then doing an atomic os.rename.

Because /vagrant is technically on a different filesystem it is failing - but with what I suspect to be an invalid error code. 

```
msg: Could not replace file: /root/.ansible/tmp/ansible-tmp-1427476564.34-155580542794406/source to /vagrant/src/officepools/local_settings.py: [Errno 26] Text file busy
```

There are two points in the following code that could be returning the same error here: https://github.com/ansible/ansible/blob/60f972dfe4bc58180c666f820ef2d602acf917e4/v2/ansible/module_utils/basic.py#L1336

I even went as far to have the templated file store its result on the current posix filesystem then copy the file to the non-posix directory and the copy function fails as well. Pretty much the same error:

```
msg: Could not replace file: /root/.ansible/tmp/ansible-tmp-1427477026.96-38878336250870/source to /vagrant/src/officepools/local_settings.py: [Errno 26] Text file busy
```

Turns out that they are both using the exact same module base (nice and DRY)

So I tried the 'command' to have the os move it and all was good.

This:

```
- template: src=local_settings.py.jinja2 dest="{{ directories.src }}/{{ names.codename }}/local_settings.py"
```

becomes this:

```
- template: src=local_settings.py.jinja2 dest=/root/local_settings.py.temp
- command: mv /root/local_settings.py.temp {{ directories.src }}/{{ names.codename }}/local_settings.py
```

and all is good. 

Seems like Ansible could be modified pretty easily to fix at least this corner case.

@jensenbox thanks for the tip on using command, that's worked for me in bumping into this as well.  Any ideas on how to make this idempotent?  

@jensenbox so if we incorporate error #26 'file too busy' and attempt to copy it non-atomically it should solve this issue? I would like to narrow it down to non-posix file systems as that is a valid error, though rare in the posix case.


Resolving patch:
diff --git a/lib/ansible/module_utils/basic.py b/lib/ansible/module_utils/basic.py
--- a/lib/ansible/module_utils/basic.py
+++ b/lib/ansible/module_utils/basic.py
@@ -1356,8 +1356,9 @@ def atomic_move(self, src, dest):
             # Optimistically try a rename, solves some corner cases and can avoid useless work, throws exception if not atomic.
             os.rename(src, dest)
         except (IOError,OSError), e:
-            # only try workarounds for errno 18 (cross device), 1 (not permitted) and 13 (permission denied)
-            if e.errno != errno.EPERM and e.errno != errno.EXDEV and e.errno != errno.EACCES:
+            # only try workarounds for errno 18 (cross device), 1 (not permitted),  13 (permission denied)
+            # and 26 (text file busy) which happens on vagrant synced folders
+            if e.errno not in [errno.EPERM, errno.EXDEV, errno.EACCES, errno.ETXTBSY]:
                 self.fail_json(msg='Could not replace file: %s to %s: %s' % (src, dest, e))
 
             dest_dir = os.path.dirname(dest)


Test patch: