Repository: conan-io/conan

Issue:
Symbolic links to directories are not properly copied during packaging
To help us debug your issue please explain:

- [x] I've read the [CONTRIBUTING guide](https://raw.githubusercontent.com/conan-io/conan/develop/.github/CONTRIBUTING.md).
- [x] I've specified the Conan version, operating system version and any tool that can be relevant.
- [x] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

Hello,

I encountered an interesting error while building Qt as a framework on macOS.
As frameworks can actually contain different version of the library, it is common to have symlinks inside the bundle pointing to the "Current" version and the respective headers, resources, etc.

After my test package failed, I did some digging and it turned out that the ```self.copy``` called inside the ```package``` method is ignoring some symlinks.

I managed to create a test recipe to reproduce the issue.
```python
import os
from conans import ConanFile, CMake, tools


class SymlinkBugTestConan(ConanFile):
    name = "symlink-bug-test"
    version = "0.1.0"
    license = "<Put the package license here>"
    url = "<Package recipe repository url here, for issues about the package>"
    description = "<Description of Symlinkbugtest here>"

    settings = "os", "compiler", "build_type", "arch"
    generators = "cmake"
    prefix_folder = 'dist'

    def build(self):
        self.run('mkdir "%s"' % self.prefix_folder)
        self.run('mkdir "%s"' % os.path.join(self.prefix_folder, 'QtCore.framework'))
        self.run('mkdir "%s"' % os.path.join(self.prefix_folder, 'QtCore.framework', 'Versions'))
        self.run('mkdir "%s"' % os.path.join(self.prefix_folder, 'QtCore.framework', 'Versions', '5'))
        self.run('mkdir "%s"' % os.path.join(self.prefix_folder, 'QtCore.framework', 'Versions', '5', 'Headers'))
        self.run('mkdir "%s"' % os.path.join(self.prefix_folder, 'QtCore.framework', 'Versions', '5', 'Resources'))

        self.run('touch "%s"' % os.path.join(self.prefix_folder, 'QtCore.framework', 'Versions', '5', 'Headers', 'a'))
        self.run('touch "%s"' % os.path.join(self.prefix_folder, 'QtCore.framework', 'Versions', '5', 'Resources', 'a'))
        self.run('touch "%s"' % os.path.join(self.prefix_folder, 'QtCore.framework', 'Versions', '5', 'QtCore'))

        self.run('ln -s "%s" "%s"' % (
            '5',
            os.path.join(self.prefix_folder, 'QtCore.framework', 'Versions', 'Current')))

        self.run('ln -s "%s" "%s"' % (
            os.path.join('Versions', 'Current', 'Headers'),
            os.path.join(self.prefix_folder, 'QtCore.framework', 'Headers')))

        self.run('ln -s "%s" "%s"' % (
            os.path.join('Versions', 'Current', 'QtCore'),
            os.path.join(self.prefix_folder, 'QtCore.framework', 'QtCore')))

        self.run('ln -s "%s" "%s"' % (
            os.path.join('Versions', 'Current', 'Resources'),
            os.path.join(self.prefix_folder, 'QtCore.framework', 'Resources')))

    def package(self):
        self.copy("*", dst="", src=self.prefix_folder,
                  symlinks=True)
```

Listing the build output contains the correct symlinks:
```
tree $CONAN_USER_HOME/.conan/data/symlink-bug-test/0.1.0/user/testing/build/<package-id>/dist/QtCore.framework
$CONAN_USER_HOME/.conan/data/symlink-bug-test/0.1.0/user/testing/build/<package-id>/dist/QtCore.framework
├── Headers -> Versions/Current/Headers
├── QtCore -> Versions/Current/QtCore
├── Resources -> Versions/Current/Resources
└── Versions
    ├── 5
    │   ├── Headers
    │   │   └── a
    │   ├── QtCore
    │   └── Resources
    │       └── a
    └── Current -> 5

7 directories, 4 files
```

Now, when I list the output package, there is a symlink missing:
```
tree $CONAN_USER_HOME/.conan/data/symlink-bug-test/0.1.0/user/testing/package/<package-id>/QtCore.framework/
$CONAN_USER_HOME/.conan/data/symlink-bug-test/0.1.0/user/testing/package/<package-id>/QtCore.framework/
├── Headers -> Versions/Current/Headers
├── QtCore -> Versions/Current/QtCore
└── Versions
    ├── 5
    │   ├── Headers
    │   │   └── a
    │   ├── QtCore
    │   └── Resources
    │       └── a
    └── Current -> 5

6 directories, 4 files
```

I could not determine an exact pattern by which specific symlinks are skipped, since my build host actually skipped the ```Headers``` directory, while my development machine (when running this recipe) skips the ```Resources``` directory.

I managed to find a temporary workaround for the issue, by using:
```
from distutils.dir_util import copy_tree
copy_tree(os.path.join(self.build_folder, self.prefix_folder),
          os.path.join(self.package_folder),
          preserve_symlinks=True)
```

As I've had issues with ```os.walk``` in the past, I suspect that the error might be in the ```_filter_files``` function of ```file_copier.py```.

In any case, it would be extremely useful if an additional function to copy whole directory trees without patterns is implemented.

Software information:
I'm currently using Conan ```version 1.2.3``` on
```
ProductName:	Mac OS X
ProductVersion:	10.13.4
BuildVersion:	17E199
```
with
```
Xcode 9.3
Build version 9E145
```
and ```Python 3.6.5``` installed from brew


Discussion:
This is interesting. Tried to reproduce in Ubuntu, but seems to work fine, output of package:

```bash
└── QtCore.framework
    ├── Headers -> Versions/Current/Headers
    ├── QtCore -> Versions/Current/QtCore
    ├── Resources -> Versions/Current/Resources
    └── Versions
        ├── 5
        │   ├── Headers
        │   │   └── a
        │   ├── QtCore
        │   └── Resources
        │       └── a
        └── Current -> 5
```
Working both in python 2 and python 3.
@memsharded this sure is interesting because I'm able to reproduce the issue on these clean VMs:
- ```Mac OS X 10.10.5 (14F2511)``` with Python ```3.6.5``` installed from brew
    - Both folder symlinks (Headers, Resources) are not present in the package
- ```Manjaro 17``` with Python ```3.6.2```
    - Only the Resources symlink is not present
Leaving a comment for information: we are experiencing the exact same problem while packaging Qt 5.9.5

Our package method in the recipe is trivial:

    def package(self):
        self.copy("*", symlinks=True)

Yet, several symlinks are actually not copied over in the package, preventing its use.
Hello @memsharded,

I think I managed to pinpoint the cause for this bug.
After debugging conan for some time, I came to the conclusion that ```_filter_files``` is actually correct.
It's output, ```file_copier.py:80``` in my case is:
```
----------LINK FOLDERS
['QtCore.framework/Resources', 'QtCore.framework/Versions/Current', 'QtCore.framework/Headers']
```

The issue occurs in the ```_link_folders``` method, specifically the if statement that checks if the ```dest_dir``` exists: ```            if os.path.exists(os.path.join(dst, dest_dir)):```

This is the case because because the list item that is currently being processed, in this case ```QtCore.framework/Resources``` depends on a link that was not yet processed, ```QtCore.framework/Versions/Current```, so the ```exists``` check returns false, and there is no ```else``` to handle this case.

If I just switch the places of the list items before passing them to the ```_link_folders``` method,
```link_folders[0], link_folders[1] = link_folders[1], link_folders[0]```, then everything is correctly executed.

I can only speculate as to why you weren't able to reproduce it on Ubuntu, but my best bet is that the traversal order of ```os.walk``` used in ```_filter_files``` might be platform specific.

Resolving patch:
diff --git a/conans/client/file_copier.py b/conans/client/file_copier.py
--- a/conans/client/file_copier.py
+++ b/conans/client/file_copier.py
@@ -138,18 +138,21 @@ def _filter_files(self, src, pattern, links, excludes, ignore_case):
     def _link_folders(src, dst, linked_folders):
         for linked_folder in linked_folders:
             link = os.readlink(os.path.join(src, linked_folder))
-            # The link is relative to the directory of the "linker_folder"
-            dest_dir = os.path.join(os.path.dirname(linked_folder), link)
-            if os.path.exists(os.path.join(dst, dest_dir)):
-                with tools.chdir(dst):
-                    try:
-                        # Remove the previous symlink
-                        os.remove(linked_folder)
-                    except OSError:
-                        pass
-                    # link is a string relative to linked_folder
-                    # e.j: os.symlink("test/bar", "./foo/test_link") will create a link to foo/test/bar in ./foo/test_link
-                    os.symlink(link, linked_folder)
+            dst_link = os.path.join(dst, linked_folder)
+            try:
+                # Remove the previous symlink
+                os.remove(dst_link)
+            except OSError:
+                pass
+            # link is a string relative to linked_folder
+            # e.j: os.symlink("test/bar", "./foo/test_link") will create a link to foo/test/bar in ./foo/test_link
+            os.symlink(link, dst_link)
+        # Remove empty links
+        for linked_folder in linked_folders:
+            dst_link = os.path.join(dst, linked_folder)
+            abs_path = os.path.realpath(dst_link)
+            if not os.path.exists(abs_path):
+                os.remove(dst_link)
 
     @staticmethod
     def _copy_files(files, src, dst, keep_path, symlinks):


Test patch:
diff --git a/conans/test/functional/file_copier_test.py b/conans/test/functional/file_copier_test.py
--- a/conans/test/functional/file_copier_test.py
+++ b/conans/test/functional/file_copier_test.py
@@ -35,45 +35,43 @@ def basic_test(self):
         self.assertEqual("Hello1 sub", load(os.path.join(folder2, "texts/sub1/file1.txt")))
         self.assertNotIn("subdir2", os.listdir(os.path.join(folder2, "texts")))
 
+    @unittest.skipUnless(platform.system() != "Windows", "Requires Symlinks")
     def basic_with_linked_dir_test(self):
-        if platform.system() == "Linux" or platform.system() == "Darwin":
-            folder1 = temp_folder()
-            sub1 = os.path.join(folder1, "subdir1")
-            sub2 = os.path.join(folder1, "subdir2")
-            os.makedirs(sub1)
-            os.symlink("subdir1", sub2)
-            save(os.path.join(sub1, "file1.txt"), "Hello1")
-            save(os.path.join(sub1, "file2.c"), "Hello2")
-            save(os.path.join(sub1, "sub1/file1.txt"), "Hello1 sub")
-
-            for links in (False, True):
-                folder2 = temp_folder()
-                copier = FileCopier(folder1, folder2)
-                copier("*.txt", "texts", links=links)
-                if links:
-                    self.assertEqual(os.readlink(os.path.join(folder2, "texts/subdir2")), "subdir1")
-                self.assertEqual("Hello1", load(os.path.join(folder2, "texts/subdir1/file1.txt")))
-                self.assertEqual("Hello1 sub", load(os.path.join(folder2, "texts/subdir1/sub1/file1.txt")))
-                self.assertEqual("Hello1", load(os.path.join(folder2, "texts/subdir2/file1.txt")))
-                self.assertEqual(['file1.txt', 'sub1'].sort(), os.listdir(os.path.join(folder2, "texts/subdir2")).sort())
-
-            for links in (False, True):
-                folder2 = temp_folder()
-                copier = FileCopier(folder1, folder2)
-                copier("*.txt", "texts", "subdir1", links=links)
-                self.assertEqual("Hello1", load(os.path.join(folder2, "texts/file1.txt")))
-                self.assertEqual("Hello1 sub", load(os.path.join(folder2, "texts/sub1/file1.txt")))
-                self.assertNotIn("subdir2", os.listdir(os.path.join(folder2, "texts")))
+        folder1 = temp_folder()
+        sub1 = os.path.join(folder1, "subdir1")
+        sub2 = os.path.join(folder1, "subdir2")
+        os.makedirs(sub1)
+        os.symlink("subdir1", sub2) # @UndefinedVariable
+        save(os.path.join(sub1, "file1.txt"), "Hello1")
+        save(os.path.join(sub1, "file2.c"), "Hello2")
+        save(os.path.join(sub1, "sub1/file1.txt"), "Hello1 sub")
 
+        for links in (False, True):
+            folder2 = temp_folder()
+            copier = FileCopier(folder1, folder2)
+            copier("*.txt", "texts", links=links)
+            if links:
+                self.assertEqual(os.readlink(os.path.join(folder2, "texts/subdir2")), "subdir1") # @UndefinedVariable
+            self.assertEqual("Hello1", load(os.path.join(folder2, "texts/subdir1/file1.txt")))
+            self.assertEqual("Hello1 sub", load(os.path.join(folder2, "texts/subdir1/sub1/file1.txt")))
+            self.assertEqual("Hello1", load(os.path.join(folder2, "texts/subdir2/file1.txt")))
+            self.assertEqual(['file1.txt', 'sub1'].sort(), os.listdir(os.path.join(folder2, "texts/subdir2")).sort())
+
+        for links in (False, True):
+            folder2 = temp_folder()
+            copier = FileCopier(folder1, folder2)
+            copier("*.txt", "texts", "subdir1", links=links)
+            self.assertEqual("Hello1", load(os.path.join(folder2, "texts/file1.txt")))
+            self.assertEqual("Hello1 sub", load(os.path.join(folder2, "texts/sub1/file1.txt")))
+            self.assertNotIn("subdir2", os.listdir(os.path.join(folder2, "texts")))
+
+    @unittest.skipUnless(platform.system() != "Windows", "Requires Symlinks")
     def linked_folder_missing_error_test(self):
-        if platform.system() != "Linux" and platform.system() != "Darwin":
-            return
- 
         folder1 = temp_folder()
         sub1 = os.path.join(folder1, "subdir1")
         sub2 = os.path.join(folder1, "subdir2")
         os.makedirs(sub1)
-        os.symlink("subdir1", sub2)
+        os.symlink("subdir1", sub2)  # @UndefinedVariable
         save(os.path.join(sub1, "file1.txt"), "Hello1")
         save(os.path.join(sub1, "file2.c"), "Hello2")
         save(os.path.join(sub1, "sub1/file1.txt"), "Hello1 sub")
@@ -81,23 +79,20 @@ def linked_folder_missing_error_test(self):
         folder2 = temp_folder()
         copier = FileCopier(folder1, folder2)
         copier("*.cpp", links=True)
-        self.assertEqual(len(os.listdir(folder2)), 0)
+        self.assertEqual(os.listdir(folder2), [])
         copier("*.txt", links=True)
         self.assertEqual(sorted(os.listdir(folder2)), sorted(["subdir1", "subdir2"]))
-        self.assertEqual(os.readlink(os.path.join(folder2, "subdir2")), "subdir1")
+        self.assertEqual(os.readlink(os.path.join(folder2, "subdir2")), "subdir1")  # @UndefinedVariable
         self.assertEqual("Hello1", load(os.path.join(folder2, "subdir1/file1.txt")))
         self.assertEqual("Hello1", load(os.path.join(folder2, "subdir2/file1.txt")))
 
+    @unittest.skipUnless(platform.system() != "Windows", "Requires Symlinks")
     def linked_relative_test(self):
-        if platform.system() != "Linux" and platform.system() != "Darwin":
-            return
-
         folder1 = temp_folder()
         sub1 = os.path.join(folder1, "foo/other/file")
-        os.makedirs(sub1)
         save(os.path.join(sub1, "file.txt"), "Hello")
         sub2 = os.path.join(folder1, "foo/symlink")
-        os.symlink("other/file", sub2)
+        os.symlink("other/file", sub2)  # @UndefinedVariable
 
         folder2 = temp_folder()
         copier = FileCopier(folder1, folder2)
diff --git a/conans/test/functional/symlink_package_test.py b/conans/test/functional/symlink_package_test.py
--- a/conans/test/functional/symlink_package_test.py
+++ b/conans/test/functional/symlink_package_test.py
@@ -6,10 +6,8 @@
 
 class SymlinkPackageTest(unittest.TestCase):
 
+    @unittest.skipUnless(platform.system() in ("Linux", "Darwin"), "Requires Symlinks")
     def test_symlink_created(self):
-        if platform.system() != "Linux" and platform.system() != "Darwin":
-            return
-
         conanfile = """from conans import ConanFile
 import os
 
diff --git a/conans/test/integration/symlinks_test.py b/conans/test/integration/symlinks_test.py
--- a/conans/test/integration/symlinks_test.py
+++ b/conans/test/integration/symlinks_test.py
@@ -1,9 +1,10 @@
+import os
+import platform
 import unittest
+
 from conans.test.utils.tools import TestClient, TestServer
 from conans.util.files import load, save
 from conans.model.ref import PackageReference, ConanFileReference
-import os
-import platform
 
 conanfile = """
 from conans import ConanFile
@@ -20,6 +21,10 @@ def build(self):
         os.symlink("file1.txt", "file1.txt.1")
         save("version1/file2.txt", "Hello2")
         os.symlink("version1", "latest")
+        os.symlink("latest", "edge")
+        os.symlink("empty_folder", "broken_link")
+        os.makedirs("other_empty_folder")
+        os.symlink("other_empty_folder", "other_link")
 
     def package(self):
         self.copy("*.txt*", links=True)
@@ -33,7 +38,7 @@ def package(self):
 ., * -> .
 """
 
-
+@unittest.skipUnless(platform.system() != "Windows", "Requires Symlinks")
 class SymLinksTest(unittest.TestCase):
 
     def _check(self, client, ref, build=True):
@@ -57,11 +62,11 @@ def _check(self, client, ref, build=True):
             filepath = os.path.join(base, "latest/file2.txt")
             file1 = load(filepath)
             self.assertEqual("Hello2", file1)
+            filepath = os.path.join(base, "edge/file2.txt")
+            file1 = load(filepath)
+            self.assertEqual("Hello2", file1)
 
     def basic_test(self):
-        if platform.system() == "Windows":
-            return
-
         client = TestClient()
         client.save({"conanfile.py": conanfile,
                      "conanfile.txt": test_conanfile})
@@ -77,9 +82,6 @@ def basic_test(self):
         self._check(client, ref)
 
     def package_files_test(self):
-        if platform.system() == "Windows":
-            return
-
         client = TestClient()
         conanfile = """
 from conans import ConanFile
@@ -95,10 +97,12 @@ def package(self):
         file2 = os.path.join(client.current_folder, "version1/file2.txt")
         file11 = os.path.join(client.current_folder, "file1.txt.1")
         latest = os.path.join(client.current_folder, "latest")
+        edge = os.path.join(client.current_folder, "edge")
         save(file1, "Hello1")
         os.symlink("file1.txt", file11)
         save(file2, "Hello2")
         os.symlink("version1", latest)
+        os.symlink("latest", edge)
         client.run("export-pkg ./recipe Hello/0.1@lasote/stable")
         ref = PackageReference.loads("Hello/0.1@lasote/stable:"
                                      "5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9")
@@ -106,9 +110,6 @@ def package(self):
         self._check(client, ref, build=False)
 
     def export_and_copy_test(self):
-        if platform.system() == "Windows":
-            return
-
         lib_name = "libtest.so.2"
         lib_contents = "TestLib"
         link_name = "libtest.so"
@@ -144,9 +145,6 @@ def export_and_copy_test(self):
         self._check(client, package_ref)
 
     def upload_test(self):
-        if platform.system() == "Windows":
-            return
-
         test_server = TestServer()
         servers = {"default": test_server}
         client = TestClient(servers=servers, users={"default": [("lasote", "mypass")]})