Repository: ansible/ansible

Issue:
Copy module silently truncates files if target server tmp location is to small.
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
Copy module

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.4.2.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/l066731/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Aug 29 2016, 10:12:21) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]
```

##### CONFIGURATION
```
ALLOW_WORLD_READABLE_TMPFILES(/etc/ansible/ansible.cfg) = True
ANSIBLE_PIPELINING(/etc/ansible/ansible.cfg) = True
ANSIBLE_SSH_ARGS(/etc/ansible/ansible.cfg) = -C -o ControlMaster=auto -o ControlPersist=240s
ANSIBLE_SSH_CONTROL_PATH_DIR(/etc/ansible/ansible.cfg) = ~/.ansible/cp
CACHE_PLUGIN(/etc/ansible/ansible.cfg) = redis
CACHE_PLUGIN_CONNECTION(/etc/ansible/ansible.cfg) = localhost:6379:0
CACHE_PLUGIN_TIMEOUT(/etc/ansible/ansible.cfg) = 10000
COMMAND_WARNINGS(/etc/ansible/ansible.cfg) = False
DEFAULT_ALLOW_UNSAFE_LOOKUPS(/etc/ansible/ansible.cfg) = True
DEFAULT_CALLBACK_WHITELIST(/etc/ansible/ansible.cfg) = ['timer', 'skippy', 'profile_roles', 'junit']
DEFAULT_FORKS(/etc/ansible/ansible.cfg) = 15
DEFAULT_GATHERING(/etc/ansible/ansible.cfg) = smart
DEFAULT_GATHER_SUBSET(/etc/ansible/ansible.cfg) = all
DEFAULT_GATHER_TIMEOUT(/etc/ansible/ansible.cfg) = 60
DEFAULT_HASH_BEHAVIOUR(/etc/ansible/ansible.cfg) = replace
DEFAULT_LOAD_CALLBACK_PLUGINS(/etc/ansible/ansible.cfg) = True
DEFAULT_LOG_PATH(/etc/ansible/ansible.cfg) = /var/log/ansible/ansible.log
DEFAULT_MANAGED_STR(/etc/ansible/ansible.cfg) = Ansible managed: {file} %Y-%m-%d %H:%M:%S
DEFAULT_PRIVATE_ROLE_VARS(/etc/ansible/ansible.cfg) = True
DEFAULT_SCP_IF_SSH(/etc/ansible/ansible.cfg) = smart
DEFAULT_SFTP_BATCH_MODE(/etc/ansible/ansible.cfg) = False
DEFAULT_SSH_TRANSFER_METHOD(/etc/ansible/ansible.cfg) = smart
DEFAULT_TRANSPORT(/etc/ansible/ansible.cfg) = smart
DEFAULT_UNDEFINED_VAR_BEHAVIOR(/etc/ansible/ansible.cfg) = True
DEPRECATION_WARNINGS(/etc/ansible/ansible.cfg) = False
DISPLAY_ARGS_TO_STDOUT(/etc/ansible/ansible.cfg) = True
DISPLAY_SKIPPED_HOSTS(/etc/ansible/ansible.cfg) = True
ERROR_ON_MISSING_HANDLER(/etc/ansible/ansible.cfg) = True
SHOW_CUSTOM_STATS(/etc/ansible/ansible.cfg) = True
```

##### OS / ENVIRONMENT
Running with master and target server on Red Hat Enterprise Linux Server 7.2 (Maipo)

##### SUMMARY
The Copy module seems to silently fail if the tmp location runs out of space when copying a large file. The copy module will simply fill the diskmount for the defined tmp location on the target server then move the truncated file to the desired path without notifying the user of the failed copy.

This can be caught by checking the hash's or file size of the target file, but this seems like it's something that the copy module itself should be able to do to validate that a copy actually occurred correctly.

##### STEPS TO REPRODUCE
Set up an environment with a target server with a /home diskmount of 2gb size
Configure ansible to use ~/.ansible/tmp for tmp when copying files (I assume this works if you set the tmp directory to any location with a limited size mount, but I'll use /home for the sake of simplicity)

Generate a 3gb random file `dd if=/dev/urandom of=random bs=1M count=3072` and place it in the files directory of a role.
Execute the following role:

```yaml
- name: copy random file to /random
  copy:
    src: random
    dest: /random
    force: yes
  register: copied

- name: debug copied
  debug:
    var: copied

- name: debug random file stat
  local_action: stat path="{{ role_path }}/files/random"
  register: local_file

- name: debug random file details
  local_action: debug msg="{{ local_file.stat }}"
```

##### EXPECTED RESULTS
The sha1sum's of the files copied to the server should match if the copy is successful. If the copy is *not* successful then the copy task should fail.

If the target diskmount hasn't got sufficient space for the copied file, the copy task should fail when it runs out of space, not silently truncating the file.

##### ACTUAL RESULTS
The copy task is "successful" (changed) but the sha1sum of each of the files is random depending how much of the file actually copied to the server before the disk capped storage space.

Below trimmed down execution on 5 servers with varying amount of stuff already on the /home directory:
```yaml
PLAY [all] ***********************************************************************************************************************************************************************************

TASK [test : copy random file to /random src=random, force=True, mode=a=rwX, dest=/random, group=root, owner=root] ***************************************************************
changed: [auslslnxsd4958]
changed: [auslslnxsd5505]
changed: [auslslnxsd3899]
changed: [auslslnxsd5502]
changed: [auslslnxsd4960]

TASK [test : debug copied var=copied] ****************************************************************************************************************************************************
ok: [auslslnxsd4958] => {
    "copied": {
        "checksum": "0f1d62e25d9487eed4c302a7320d3dd2e37c97cd", 
        "dest": "/random", 
        "size": 1761767424, 
    }
}
ok: [auslslnxsd5505] => {
    "copied": {
        "checksum": "5aa686f6fc2038a57bc053c4f7256d1a0144c2a8", 
        "dest": "/random",
        "size": 1913671680
    }
}
ok: [auslslnxsd3899] => {
    "copied": {
        "checksum": "96febf0131e1e8e70d883bd74e3b66ee1bcbd67d", 
        "dest": "/random", 
        "size": 1911713792
    }
}
ok: [auslslnxsd5502] => {
    "copied": {
        "checksum": "7bd512c18837afd438df718d9304c94a12bacd7a", 
        "dest": "/random", 
        "size": 1913860096 
    }
}
ok: [auslslnxsd4960] => {
    "copied": {
        "checksum": "b761ec98e3578742c2911851fa8f712cbb531de2", 
        "dest": "/random", 
        "size": 1915072512 
    }
}

TASK [test : local stat of random file path={{ role_path }}/files/random] ****************************************************************************************************************
ok: [auslslnxsd4958 -> localhost]
ok: [auslslnxsd5505 -> localhost]
ok: [auslslnxsd3899 -> localhost]
ok: [auslslnxsd5502 -> localhost]
ok: [auslslnxsd4960 -> localhost]


TASK [test : debug random file stat msg={{ local_file.stat }}] **************************************************************************************************************************
ok: [auslslnxsd4958 -> localhost] => {
    "msg": {
        "checksum": "592f1b9adaab949bd30b10f9b1a893bac792dd19", 
        "size": 3221225472 
    }
}
ok: [auslslnxsd5505 -> localhost] => {
    "msg": {
        "checksum": "592f1b9adaab949bd30b10f9b1a893bac792dd19", 
        "size": 3221225472 
    }
}
ok: [auslslnxsd3899 -> localhost] => {
    "msg": {
        "checksum": "592f1b9adaab949bd30b10f9b1a893bac792dd19", 
        "size": 3221225472 
    }
}
ok: [auslslnxsd5502 -> localhost] => {
    "msg": {
        "checksum": "592f1b9adaab949bd30b10f9b1a893bac792dd19", 
        "size": 3221225472 
    }
}
ok: [auslslnxsd4960 -> localhost] => {
    "msg": {
        "checksum": "592f1b9adaab949bd30b10f9b1a893bac792dd19", 
        "size": 3221225472 
    }
}
```


Discussion:
Files identified in the description:
* [lib/ansible/modules/files/copy.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/files/copy.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Ok, from my testing I have narrowed this down to 2 configuration options that produce this behavior:

```
[ssh_connection]
sftp_batch_mode = false
pipelining = true
```

Any combination other than this allows it to succeed.

It seems that when using `sftp_batch_mode = false`, we don't get a proper failure from sftp.  And when using `pipelining = True`, it seems that later because we don't need to create a temporary file for the module, it doesn't fail due to being out of disk space.

I think we just need to look into how we cause `sftp_batch_mode = false` to properly fail.
```
        # sftp batch mode allows us to correctly catch failed transfers, but can
        # be disabled if the client side doesn't support the option. However,
        # sftp batch mode does not prompt for passwords so it must be disabled
        # if not using controlpersist and using sshpass
```

It seems like this may be a known issue.  Disabling `BatchMode` on sftp prevents us from catching failed transfers.
I probably should be more attentive to notifications, I just wasted about 2 hours of my life coming up with the exact same analysis by toggling settings around.

I had a cursory look at the code which references sftp_batch_mode and can't actually see anything immediately obvious that could allow this failure to happen.

The strange thing about your comment 

>  it seems that later because we don't need to create a temporary file for the module, it doesn't fail due to being out of disk space.

Is that the resulting file on the server ends up being the same size as the amount of disk space left on the diskmount where temp files are written. So, if you're correct in saying that we don't need to write the temporary file, then I'm not sure what's going on truncation, because there should be no reason why the copy can't just push the entire file onto the remote server anyway.

Although, looking at the code I suspect isn't going to help me a huge amount unless I go back a few commits. It looks like there's a fairly broad re-write of copy #31677 as part of something which seems to touch a lot of remote_tmp... Although the PR isn't hugely descriptive about what the change is, I might find that this issue get's fixed through whatever changes are being made there.
The copied file does indeed end up taking up all of the disk space, and does need to be transferred.

However, the ansible module (the python code) does not need to be transferred as a temporary file due to pipelining.

We never push files directly to their final location, always to a temp dir first, largely for the purpose of atomic writes, and a number of other edge cases.
Makes sense to me.

As a temporary workaround I've set the following:

```yaml
  vars:
    ansible_ssh_pipelining: no
    sftp_batch_mode: yes
```

on all the tasks that are copying big files (there are only a few in my use case, and it's just to work around weird firewalling things preventing access to the original host, so not such a big change) and I've needed to update the remote tmp directory to a directory we are setting up before we start these tasks as /tmp is a tmpfs which is only 2gb as well....

But that really only fixes my specific issue. The fact that the copy module doesn't compare the checksums (or at least the file sizes) of the file on the source host  and the file on the remote host seems like it's a missing check that would catch this sort of failure however it occurs in future, but by the looks of it, unless you hand roll it yourself, there is no validation done that the copy was in fact successful.

Resolving patch:
diff --git a/lib/ansible/modules/files/copy.py b/lib/ansible/modules/files/copy.py
--- a/lib/ansible/modules/files/copy.py
+++ b/lib/ansible/modules/files/copy.py
@@ -83,6 +83,11 @@
     type: bool
     default: 'yes'
     version_added: "2.4"
+  checksum:
+    description:
+      - SHA1 checksum of the file being transferred. Used to valdiate that the copy of the file was successful.
+      - If this is not provided, ansible will use the local calculated checksum of the src file.
+    version_added: '2.5'
 extends_documentation_fragment:
     - files
     - validate
@@ -265,6 +270,7 @@ def main():
             directory_mode=dict(type='raw'),
             remote_src=dict(type='bool'),
             local_follow=dict(type='bool'),
+            checksum=dict(),
         ),
         add_file_common_args=True,
         supports_check_mode=True,
@@ -281,6 +287,7 @@ def main():
     follow = module.params['follow']
     mode = module.params['mode']
     remote_src = module.params['remote_src']
+    checksum = module.params['checksum']
 
     if not os.path.exists(b_src):
         module.fail_json(msg="Source %s not found" % (src))
@@ -299,6 +306,13 @@ def main():
 
     changed = False
 
+    if checksum and checksum_src != checksum:
+        module.fail_json(
+            msg='Copied file does not match the expected checksum. Transfer failed.',
+            checksum=checksum_src,
+            expected_checksum=checksum
+        )
+
     # Special handling for recursive copy - create intermediate dirs
     if original_basename and dest.endswith(os.sep):
         dest = os.path.join(dest, original_basename)
diff --git a/lib/ansible/plugins/action/copy.py b/lib/ansible/plugins/action/copy.py
--- a/lib/ansible/plugins/action/copy.py
+++ b/lib/ansible/plugins/action/copy.py
@@ -291,6 +291,9 @@ def _copy_file(self, source_full, source_rel, content, content_tempfile, dest, t
                     original_basename=source_rel,
                 )
             )
+            if not self._task.args.get('checksum'):
+                new_module_args['checksum'] = local_checksum
+
             if lmode:
                 new_module_args['mode'] = lmode
 


Test patch: