Repository: ansible/ansible

Issue:
copy module doesn't follow symbolic directory
##### ISSUE TYPE

Bug Report
##### COMPONENT NAME:

copy action plugin
##### ANSIBLE VERSION

2.0.1.0
##### SUMMARY

I am copying a folder, foo, which has a subfolder, foo1, which is a symbolic link to another folder say /root/myfoo1. copy module won't copy it to remote location.

copy module slower than expected when recursive
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
copy module

##### ANSIBLE VERSION
```
ansible 2.2.1.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrid
```

##### CONFIGURATION


##### OS / ENVIRONMENT
Debian Sid -> Centos 6

##### SUMMARY
The copy module documentation states:
> The “copy” module recursively copy facility does not scale to lots (>hundreds) of files.
and I'm finding incredible slowdowns at barely 50 files.

If I list each file individually using with_items in my task however, ansible completes an order of magnitude faster.

I'm confident that there is a large speed boost to be gained somewhere here.

##### STEPS TO REPRODUCE
```
copy: src=directory dest=somewhere
```
vs
```
file: path=somewhere state=directory

copy: src={{ item }} dest=directory/{{ item }}
with_items:
  - file1
  - file2
  - etc
```

##### EXPECTED RESULTS
Using the copy module recursively _should_ take a similar amount of time to listing the files using with_items?

##### ACTUAL RESULTS
Using the copy module recursively takes at least 10x longer.


Discussion:
@justinligraph with which version of Ansible are you seeing this occur?

I can confirm this bug in both 2.0.1.0 (Ubuntu repo) as well as the devel branch from git (pulled today).  

I have attached a small self contained demo.  It copies a directory that includes a couple symlinks to /tmp/ of the local machine.  One symlink is pointed to a file (called "link_to_file"), one is pointed to a directory (called "link_to_dir").  The directory that "link_to_dir" points to never gets copied (dir2), however the file that "link_to_file" points to does (file2).  
[bugtest.tar.gz](https://github.com/ansible/ansible/files/230957/bugtest.tar.gz)

I've tried profiling this but all the time is spent on ansible/plugins/strategy._wait_on_pending_results()

I've tried ANSIBLE_KEEP_REMOTE_FILES=1, hoping I could profile the resultant script. That leaves copy.py but no source directory? Maybe that's a separate bug...
To put some numbers to it:
synchronize:
real	0m4.311s
user	0m2.408s
sys	0m0.276s

copy recursive:
real	9m36.888s
user	1m25.800s
sys	0m15.456s

copy with_items:
real	0m42.066s
user	0m8.348s
sys	0m1.352s

synchronize hasn't been an option for us historically (bugs around become, ssh keys etc.) but 40 seconds is still definitely acceptable. 10 minutes for the benefit of not having to write files out in the task yaml is... silly.

Now I don't want to do this for obvious reasons, but something like:

```
name: Find files to install.
delegate_to: localhost
connection: local
run_once: true
find:
  paths: "{{ role_path }}/files/webservice"
  file_type: "file"
  recurse: yes
register: webservice
become: false

name: copy
copy: 
  src: "{{ item.path }}"
  dest: "/var/www/python/webservice/{{ item.path[(role_path + '/files/webservice/').__len__():] }}"
with_items: "{{ webservice.files }}"
when: item.path is defined
```
...matches a simple with_items loop:
real	0m43.873s
user	0m9.160s
sys	0m1.348s

and gives us the benefit of both. (Directory creation not shown but taken into account for timings) 

There are separate reasons why we're not using synchronize but I'm not entirely sure if they're merely historical at this point. It's something we probably want to move to, but still I think there's room for improvement here.
@PriceChild can you show us the directory structure, the filenames, their sizes and their content types?

needs_info
Hi @jctanner https://github.com/PriceChild/ansible-issue21513 hopefully provides everything you need to see this issue in action.

It's hopefully a sanitised version of my real data. filenames randomly changed & random bytes filling each file to their original size.

It seems like it's the later files slowing things down... the first ~45 speed by just as quick as the with_items version but then things get silly.
Recording some information so I don't lose it again if I get distracted before finishing this:

I believe the slowness is because we end up walking subtrees of the directory many many times instead of only once.  I believe it was introduced with this commit: 29fda4be1eb73d3af4212997222b5e9ae0b03594  which fixed this bug: https://github.com/ansible/ansible/issues/13013

Resolving patch:
diff --git a/lib/ansible/plugins/action/copy.py b/lib/ansible/plugins/action/copy.py
--- a/lib/ansible/plugins/action/copy.py
+++ b/lib/ansible/plugins/action/copy.py
@@ -113,7 +113,7 @@ def run(self, tmp=None, task_vars=None):
                 sz = len(source.rsplit('/', 1)[0]) + 1
 
             # Walk the directory and append the file tuples to source_files.
-            for base_path, sub_folders, files in os.walk(to_bytes(source)):
+            for base_path, sub_folders, files in os.walk(to_bytes(source), followlinks=True):
                 for file in files:
                     full_path = to_text(os.path.join(base_path, file), errors='surrogate_or_strict')
                     rel_path = full_path[sz:]
@@ -121,10 +121,6 @@ def run(self, tmp=None, task_vars=None):
                         rel_path = rel_path[1:]
                     source_files.append((full_path, rel_path))
 
-                # recurse into subdirs
-                for sf in sub_folders:
-                    source_files += self._get_recursive_files(os.path.join(source, to_text(sf)), sz=sz)
-
             # If it's recursive copy, destination is always a dir,
             # explicitly mark it so (note - copy module relies on this).
             if not self._connection._shell.path_has_trailing_slash(dest):
@@ -326,22 +322,6 @@ def run(self, tmp=None, task_vars=None):
 
         return result
 
-    def _get_recursive_files(self, topdir, sz=0):
-        ''' Recursively create file tuples for sub folders '''
-        r_files = []
-        for base_path, sub_folders, files in os.walk(to_bytes(topdir)):
-            for fname in files:
-                full_path = to_text(os.path.join(base_path, fname), errors='surrogate_or_strict')
-                rel_path = full_path[sz:]
-                if rel_path.startswith('/'):
-                    rel_path = rel_path[1:]
-                r_files.append((full_path, rel_path))
-
-                for sf in sub_folders:
-                    r_files += self._get_recursive_files(os.path.join(topdir, to_text(sf)), sz=sz)
-
-        return r_files
-
     def _create_content_tempfile(self, content):
         ''' Create a tempfile containing defined content '''
         fd, content_tempfile = tempfile.mkstemp()


Test patch:
diff --git a/test/integration/targets/copy/tasks/main.yml b/test/integration/targets/copy/tasks/main.yml
--- a/test/integration/targets/copy/tasks/main.yml
+++ b/test/integration/targets/copy/tasks/main.yml
@@ -258,3 +258,48 @@
   assert:
     that:
     - replace_follow_result.checksum == target_file_result.stdout
+
+- name: create a test dir to copy
+  file:
+    path: '{{ output_dir }}/top_dir'
+    state: directory
+
+- name: create a test dir to symlink to
+  file:
+    path: '{{ output_dir }}/linked_dir'
+    state: directory
+
+- name: create a file in the test dir
+  copy:
+    dest: '{{ output_dir }}/linked_dir/file1'
+    content: 'hello world'
+
+- name: create a link to the test dir
+  file:
+    path: '{{ output_dir }}/top_dir/follow_link_dir'
+    src: '{{ output_dir }}/linked_dir'
+    state: link
+
+- name: copy the directory's link
+  copy:
+    src: '{{ output_dir }}/top_dir'
+    dest: '{{ output_dir }}/new_dir'
+    follow: True
+
+- name: stat the copied path
+  stat:
+    path: '{{ output_dir }}/new_dir/top_dir/follow_link_dir'
+  register: stat_dir_result
+
+- name: stat the copied path
+  stat:
+    path: '{{ output_dir }}/new_dir/top_dir/follow_link_dir/file1'
+  register: stat_file_in_dir_result
+
+- name: assert that the directory exists
+  assert:
+    that:
+    - stat_dir_result.stat.exists
+    - stat_dir_result.stat.isdir
+    - stat_file_in_dir_result.stat.exists
+    - stat_file_in_dir_result.stat.isreg