Repository: ansible/ansible

Issue:
Got MemoryError when copy huge local file to remote
<!--- Verify first that your issue/request is not already reported in GitHub -->
##### ISSUE TYPE

<!--- Pick one below and delete the rest: -->
- Bug Report
##### ANSIBLE VERSION

<!--- Paste verbatim output from “ansible --version” between quotes below -->

```
ansible 2.1.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
```
##### CONFIGURATION

<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
##### OS / ENVIRONMENT

<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->

Ubuntu 14.04.4 LTS
##### SUMMARY

<!--- Explain the problem briefly -->

Got MemoryError when copy huge local file to remote.
##### STEPS TO REPRODUCE

<!---
For bugs, show exactly how to reproduce the problem.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->

```
$ cat huge_copy.yml 

---
- hosts: all
  become: yes

  tasks:
    - name: Copy huge file
      copy: src=5gb dest=dst

$ dd if=/dev/zero of=5gb bs=1G count=5
5+0 records in
5+0 records out
5368709120 bytes (5.4 GB) copied, 206.341 s, 26.0 MB/s

$ ansible-playbook -i hosts huge_copy.yml
```

<!--- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS

<!--- What did you expect to happen when running the steps above? -->

Copied successfully.
##### ACTUAL RESULTS

<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->

```
$ ansible-playbook -i hosts huge_copy.yml -vvv
Using /etc/ansible/ansible.cfg as config file

PLAYBOOK: huge_copy.yml ********************************************************
1 plays in huge_copy.yml

PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
<192.168.80.129> ESTABLISH SSH CONNECTION FOR USER: ansible
<192.168.80.129> SSH: EXEC ssh -C -q -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ansible -o ConnectTimeout=10 -o ControlPath=/home/y/.ansible/cp/ansible-ssh-%h-%p-%r 192.168.80.129 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1466567672.43-71840509019686 `" && echo ansible-tmp-1466567672.43-71840509019686="` echo $HOME/.ansible/tmp/ansible-tmp-1466567672.43-71840509019686 `" ) && sleep 0'"'"''
<192.168.80.129> PUT /tmp/tmp3Bk56k TO /home/ansible/.ansible/tmp/ansible-tmp-1466567672.43-71840509019686/setup
<192.168.80.129> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ansible -o ConnectTimeout=10 -o ControlPath=/home/y/.ansible/cp/ansible-ssh-%h-%p-%r '[192.168.80.129]'
<192.168.80.129> ESTABLISH SSH CONNECTION FOR USER: ansible
<192.168.80.129> SSH: EXEC ssh -C -q -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ansible -o ConnectTimeout=10 -o ControlPath=/home/y/.ansible/cp/ansible-ssh-%h-%p-%r -tt 192.168.80.129 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-rmwgnaejatjdbjwnltokljxsdjjlhagx; LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/ansible/.ansible/tmp/ansible-tmp-1466567672.43-71840509019686/setup; rm -rf "/home/ansible/.ansible/tmp/ansible-tmp-1466567672.43-71840509019686/" > /dev/null 2>&1'"'"'"'"'"'"'"'"' && sleep 0'"'"''
ok: [192.168.80.129]

TASK [Copy huge file] **********************************************************
task path: /home/y/nfvdv4-single/huge_copy.yml:6
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 124, in run
    res = self._execute()
  File "/usr/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 446, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/dist-packages/ansible/plugins/action/copy.py", line 156, in run
    source_full = self._loader.get_real_file(source_full)
  File "/usr/lib/python2.7/dist-packages/ansible/parsing/dataloader.py", line 334, in get_real_file
    data = f.read()
MemoryError

fatal: [192.168.80.129]: FAILED! => {"failed": true, "msg": "Unexpected failure during module execution.", "stdout": ""}

NO MORE HOSTS LEFT *************************************************************
    to retry, use: --limit @huge_copy.retry

PLAY RECAP *********************************************************************
192.168.80.129             : ok=1    changed=0    unreachable=0    failed=1  
```



Discussion:

Resolving patch:
diff --git a/lib/ansible/parsing/dataloader.py b/lib/ansible/parsing/dataloader.py
--- a/lib/ansible/parsing/dataloader.py
+++ b/lib/ansible/parsing/dataloader.py
@@ -331,11 +331,11 @@ def get_real_file(self, file_path):
 
         try:
             with open(to_bytes(real_path), 'rb') as f:
-                data = f.read()
-                if self._vault.is_encrypted(data):
+                if self._vault.is_encrypted(f):
                     # if the file is encrypted and no password was specified,
                     # the decrypt call would throw an error, but we check first
                     # since the decrypt function doesn't know the file name
+                    data = f.read()
                     if not self._vault_password:
                         raise AnsibleParserError("A vault password must be specified to decrypt %s" % file_path)
 
diff --git a/lib/ansible/parsing/vault/__init__.py b/lib/ansible/parsing/vault/__init__.py
--- a/lib/ansible/parsing/vault/__init__.py
+++ b/lib/ansible/parsing/vault/__init__.py
@@ -110,6 +110,12 @@ def is_encrypted(self, data):
         :returns: True if it is recognized.  Otherwise, False.
         """
 
+        if hasattr(data, 'read'):
+            current_position = data.tell()
+            header_part = data.read(len(b_HEADER))
+            data.seek(current_position)
+            return self.is_encrypted(header_part)
+
         if to_bytes(data, errors='strict', encoding='utf-8').startswith(b_HEADER):
             return True
         return False


Test patch: