Repository: conan-io/conan

Issue:
[question] Are there environment difference between the "developer workflow" and `conan create`?
While moving one of our _fat_ dependencies from prebuilt binaries (packaged through Conan ; ) to a recipe actually building the thing, we are encountering a complication we never encountered for our other recipes:

We wrote the recipe, following the recommendend [Package Development Flow](https://docs.conan.io/en/latest/developing_packages/package_dev_flow.html#package-development-flow). The end to end flow works without any error, notably `conan build` command.

Yet, when moving to the final step of calling `conan create` on the exact same recipe, the process fails during the `build()`  step.

Are there known difference between the environments used by the "developer workflow" and `conan create` command that we could investigate to find the source of this failure?


Discussion:
### Gory details

Due to the nature of this problem appearing in the middle of a 3rd party "mega build", we cannot really isolate a minimal example to quickly demonstrate the issue (hence the very vague question, sorry for that)
This problem is appearing while building Qt 5.9.6 on OSX 10.12, using Conan 1.7.1

The error when using `conan create` ends with:

<details>
<summary>(... and there is so much more spewing before that)</summary>
<p>
/Users/adn/.conan/data/qt/5.9.6/ag/testing/build/3cfade032d90e40671f831047a4b021865f8396f/qt-everywhere-opensource-src-5.9.6/qtwebengine/src/3rdparty/chromium/tools/gn/._builder_record.cc:1:190: error: expected unqualified-id
<U+0000><U+0005><U+0016><U+0007><U+0000><U+0002><U+0000><U+0000>Mac OS X        <U+0000><U+0002><U+0000><U+0000><U+0000>   <U+0000><U+0000><U+0000>2<U+0000><U+0000><U+0000><B0><U+0000><U+0000><U+0000><U+0002><U+0000><U+0000><U+0000><E2><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000>ATTR<U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><E2><U+0000><U+0000><U+0000><98><U+0000><U+0000><U+0000>J<U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0001><U+0000><U+0000><U+0000><98><U+0000><U+0000><U+0000>J<U+0000><U+0000><U+0015>com.apple.quarantine<U+0000>q/0081;5b8eaa09;Google\x20Chrome.app;057AC61C-1740-47C1-BFBF-345B35B9491F
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               ^
/Users/adn/.conan/data/qt/5.9.6/ag/testing/build/3cfade032d90e40671f831047a4b021865f8396f/qt-everywhere-opensource-src-5.9.6/qtwebengine/src/3rdparty/chromium/tools/gn/._builder_record.cc:1:226: warning: null character ignored [-Wnull-character]
<U+0000><U+0005><U+0016><U+0007><U+0000><U+0002><U+0000><U+0000>Mac OS X        <U+0000><U+0002><U+0000><U+0000><U+0000>   <U+0000><U+0000><U+0000>2<U+0000><U+0000><U+0000><B0><U+0000><U+0000><U+0000><U+0002><U+0000><U+0000><U+0000><E2><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000>ATTR<U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><E2><U+0000><U+0000><U+0000><98><U+0000><U+0000><U+0000>J<U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0000><U+0001><U+0000><U+0000><U+0000><98><U+0000><U+0000><U+0000>J<U+0000><U+0000><U+0015>com.apple.quarantine<U+0000>q/0081;5b8eaa09;Google\x20Chrome.app;057AC61C-1740-47C1-BFBF-345B35B9491F<U+0000>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   ^
96 warnings and 10 errors generated.
[207/463] CXX base/message_loop/message_pump_mac.o
/Users/adn/.conan/data/qt/5.9.6/ag/testing/build/3cfade032d90e40671f831047a4b021865f8396f/qt-everywhere-opensource-src-5.9.6/qtwebengine/src/3rdparty/chromium/base/message_loop/message_pump_mac.mm:643:53: warning: 'NSAnyEventMask' is deprecated: first deprecated in macOS 10.12 [-Wdeprecated-declarations]
      NSEvent* event = [NSApp nextEventMatchingMask:NSAnyEventMask
                                                    ^~~~~~~~~~~~~~
                                                    NSEventMaskAny
/System/Library/Frameworks/AppKit.framework/Headers/NSEvent.h:158:26: note: 'NSAnyEventMask' has been explicitly marked deprecated here
static const NSEventMask NSAnyEventMask                 API_DEPRECATED_WITH_REPLACEMENT("NSEventMaskAny", macosx(10.0, 10.12)) = NSUIntegerMax;
                         ^
/Users/adn/.conan/data/qt/5.9.6/ag/testing/build/3cfade032d90e40671f831047a4b021865f8396f/qt-everywhere-opensource-src-5.9.6/qtwebengine/src/3rdparty/chromium/base/message_loop/message_pump_mac.mm:665:48: warning: 'NSApplicationDefined' is deprecated: first deprecated in macOS 10.12 [-Wdeprecated-declarations]
  [NSApp postEvent:[NSEvent otherEventWithType:NSApplicationDefined
                                               ^~~~~~~~~~~~~~~~~~~~
                                               NSEventTypeApplicationDefined
/System/Library/Frameworks/AppKit.framework/Headers/NSEvent.h:79:26: note: 'NSApplicationDefined' has been explicitly marked deprecated here
static const NSEventType NSApplicationDefined   API_DEPRECATED_WITH_REPLACEMENT("NSEventTypeApplicationDefined", macosx(10.0, 10.12)) = NSEventTypeApplicationDefined;
                         ^
2 warnings generated.

[210/463] CXX base/metrics/field_trial.o

ninja: build stopped: subcommand failed.
Command '['/Users/adn/.conan/data/qt/5.9.6/ag/testing/build/3cfade032d90e40671f831047a4b021865f8396f/qtwebengine/src/3rdparty/ninja/ninja', '-C', '/Users/adn/.conan/data/qt/5.9.6/ag/testing/build/3cfade032d90e40671f831047a4b021865f8396f/qtwebengine/src/3rdparty/chromium/tools/gn/out/Release', 'gn']' returned non-zero exit status 1
Project ERROR: GN build error!

make[3]: *** [sub-gn-pro-make_first] Error 3
make[2]: *** [sub-buildtools-make_first] Error 2
make[1]: *** [sub-src-make_first] Error 2
make: *** [module-qtwebengine-make_first] Error 2

qt/5.9.6@ag/testing:
qt/5.9.6@ag/testing: ERROR: Package '3cfade032d90e40671f831047a4b021865f8396f' build failed
qt/5.9.6@ag/testing: WARN: Build folder /Users/adn/.conan/data/qt/5.9.6/ag/testing/build/3cfade032d90e40671f831047a4b021865f8396f

ERROR: qt/5.9.6@ag/testing: Error in build() method, line 91
	self.run("make -j {cpu_count} -s".format(cpu_count=tools.cpu_count()))
	ConanException: Error 512 while executing make -j 8 -s
</p>
</details>

Do you have `test_package` conanfile? Could we see the generated `conanbuildinfo.txt` and `conaninfo.txt`? and the applied profile?
Thank you @lasote for taking a look at this issue.

I have no `test_package` for this recipe yet. 

Do you want the files when calling `conan create` ?

Here is the (uninteresting) `conanbuildinfo.txt` found in the build folder.

<details>
<summary>conanbuildinfo.txt</summary>
<p>

```ini
[includedirs]


[libdirs]


[bindirs]


[resdirs]


[builddirs]


[libs]


[defines]


[cppflags]


[cflags]


[sharedlinkflags]


[exelinkflags]


[sysroot]
```
</p>
</details>

Is there a `conaninfo.txt` when calling `conan create`?

As for my (default) profile:

<details>
<summary>default</summary>
<p>

```ini
[settings]
os=Macos
os_build=Macos
arch=x86_64
arch_build=x86_64
compiler=apple-clang
compiler.version=9.0
compiler.libcxx=libc++
build_type=Release
[options]
[build_requires]
[env]
```
</p>
</details>
> Is there a conaninfo.txt when calling conan create?

Sorry I was interested in the files when you call `conan install` in the local development flow. But honestly, I don't think I'm going to find anything but just in case. We know there can be slight differences between the local methods and the create but I'm not sure if big enough to break your build like that. Are you using the `AutotoolsBuildEnvironment` or calling directly the command line?
After long on-and-off periods of trying to understand the problem, I think I can finally provide a description of what happens. The actual problem is actually triggered by the combination of different elements: OS X [a], Qt(ninja) [b], and Conan.

# Reproducing the issue
 Note that the situation described should only be the same on **OS X platforms**:

Here is a repository that allows to illustrate the issue:
https://github.com/Adnn/issue_conan-3529

The trivial `conanfile.py`

```
from conans import ConanFile, tools

class Issue3529Conan(ConanFile):
    name = "issue3529"
    version = "0.0.1"
    description = "see: https://github.com/conan-io/conan/issues/3529"
    settings = "os", "compiler", "build_type", "arch"

    no_copy_source = False

    def source(self):
        tools.get("https://github.com/Adnn/issue_conan-3529/releases/download/0.0.1/qtxml.tar.bz2")
```

Let's clone it and move into the root of the repo
```
git clone https://github.com/Adnn/issue_conan-3529.git
cd issue_conan-3529/
```

### Developer workflow

```
mkdir source
conan source .  -sf=source
mkdir build
cp -r source/* build/ # we simulate a recipe that needs to copy the source
```

    ls -la source/qt-everywhere-opensource-src-5.9.6/qtxmlpatterns/

> drwxr-xr-x   3 adn  wheel  102  3 Oct 15:40 .
> drwxr-xr-x   3 adn  wheel  102  3 Oct 15:40 ..
> drwxr-xr-x  20 adn  wheel  680  3 Oct 15:40 qtxmlpatterns

    ls -la build/qt-everywhere-opensource-src-5.9.6/qtxmlpatterns/

> drwxr-xr-x   4 adnn  wheel   136  3 Oct 15:39 .
> drwxr-xr-x   3 adnn wheel   102  3 Oct 15:39 ..
> -rwxr-xr-x   1 adnn  wheel   226  3 Oct 10:58 ._qtxmlpatterns
> drwxr-xr-x  38 adnn  wheel  1292  3 Oct 10:58 qtxmlpatterns

Here is the difference: 
The source folder contains a `._` prefixed version of each file and folder it contains, whereas the build one does not.

### Normal (create) workflow

```
conan create . com/testing
ls -la ~/.conan/data/issue3529/0.0.1/com/testing/build/$(HASH)/qt-everywhere-opensource-src-5.9.6/
```

> drwxr-xr-x   4 adn  staff   136  3 Oct 15:51 .
> drwxr-xr-x   5 adn  staff   170  3 Oct 15:51 ..
> -rwxr-xr-x   1 adn  staff   226  3 Oct 10:58 ._qtxmlpatterns
> drwxr-xr-x  38 adn  staff  1292  3 Oct 10:58 qtxmlpatterns

With the `conan create` workflow, the source folder **that is copied inside the build folder** also contains the `._` prefixed version of the folder.

### Command line extraction

```
mkdir dl && cd dl
wget https://github.com/Adnn/issue_conan-3529/releases/download/0.0.1/qtxml.tar.bz2
tar -xf qtxml.tar.bz2
ls -la qt-everywhere-opensource-src-5.9.6/
```

> drwxr-xr-x   3 adn  wheel  102  3 Oct 15:56 .
> drwxr-xr-x   4 adn  wheel  136  3 Oct 15:56 ..
> drwxr-xr-x@ 20 adn  wheel  680  3 Oct 15:56 qtxmlpatterns

With command line `tar` extraction, **no** `._` file.

# Some explanations

Those extra  `._` file are storage for extended attributes, that is created by OS X builtin `tar` command (as well as others). See this [superuser entry](https://superuser.com/q/61185/375719).

Because of that, they would only exist for archives created on OS X[a].

When extracting those files with `tar`, they are of course not expanded (and used to apply the correct attributes to the actual file), as seen in the example above.

When retrieved with Conan's `tools.get()`, those are not properly handled, and the files are left around.
Now, in cases were those file would be present is source archives, their existence are hardly noticed by the user (as I was fully unaware of them until today). Even if using `tools.get()`, those extra hidden files don't usually get in the way, and the compilation still works just fine.

But it seems Qt [b] WebEngine module, which relies on *ninja* to build some parts, gets really thrown off by those files. That is why my initial Qt recipe fails to build: *ninja* trips on those `._` files that are present in the copied source folder.

---

Finally, back to the initial topic of this issue:

> why is there a difference in behaviour between between the "developer workflow" and `conan create`

Because, to fully reproduce the expected recipe that would need to copy the source folder to modify bits of it, I added a manual `cp` step to the developer workflow, **because I could not find any conan developer command to do this copy otherwise**.

And the `cp` shipped with OS X took care of handling those `._` (applying the attributes, and not copying them over to destination).

So, here it is, a recipe that was working perfectly in "developer workflow" is breaking at build time with `conan create`.


# Potential directions from here

This issue took more time than I'd like to admit to understand, and I keep hope that there are potential things to do so others would not have to go through this.

Of course, one important solution is to not include those files in the archive, which seems possible by defining `COPYFILE_DISABLE` env var before calling `tar` (or creating the archives on a non-OS X machine).
Yet, it seems reasonable to assume that if people are not aware of it, they will unawarely produce such archives.

Another direction would be for `tools.get()` to handle those extra files at least for OS X systems. I don't know if Python implementations make any provision to help with that. Or perhaps give warning if they are found, so users are given a direction to look into (and developers encouraged to purge their source archives ).

What would seem important to me though, would be to give the recipes developers the ability to call the "source copy logic" from a developer command (either `conan source` or `conan build` I guess). In my situation, if I was able use the same copy logic in my developer workflow than was implicitly used by `conan create`, I would probably have spent less time barking the wrong trees (as the developer workflow would have failed directly).

If you read up to there: congratulations ; )
Wow, many thanks for your investigation!
So, If I understood correctly:

- We cannot avoid these files to exist within a tar file because the tar file could have been compressed by a third party, right? like the `qtxml.tar.bz2`. So the `COPYFILE_DISABLE` var won't help us much.

- The goal is to unzip the file (by using the `tools.get`) but without generating these "._" files.

Am I right?
Hi @lasote, and thanks a lot for taking time to look into that (despite a *lengthy* comment),

Your description seems right!

Ideally, source archives should not contain those files, because they are expected to be platform-agnostic.

Yet as you pointed out, Conan has no control over archives produced by 3rd parties. (Just a personal gut-feeling: it seems to me that a non-negligible part of developers working on OS X are unaware of this `tar` peculiarity on the platform).
The `COPYFILE_DISABLE` variable would allow to produce archives files without those `._` files in them, but once the archive is produced, it cannot help.

* A potential direction would indeed be for `tools.get` to take care of those `._`.  On OS X, that should always be a good policy: if an archive is expected to work with the builtin `tar` command, it means these `._` are not expected to exist in the expanded destination anyway.
Yet, if someone on another OS decides that he wants to actually store (and retrieve) files prefixed with `._` in the archive, that might break. This is unlikely of course, but to be aware of.

* Should Conan instead issue **warning** on the CLI that those files are present in the archives, and that they go against "platform-agnostic" good practices? This way, someone in my current situation (responsible for both archiving the sources and developing the recipe) would be informed that he should clean up the archives. Yet, it could be an issue for some recipe developer who rely on an already existing archive he cannot control.

* Should there be a new option in the recipe (eg. `archived_dotunderscore`) with 3 values:
    1. warn(default): to just issue warning if the files exist, so by default people are encouraged to produce "clean" archives.
    2. appledouble: the `._` file are correctly handled on OSX, and simply discarded on other OSes.
    3. keep: the `._` files are extracted normally (the current behaviour)

---

And some related remark, which could close the only gap I found between `conan create` and the *developer workflow*:

Do you think it would be possible to add a `--copy ` flag to `conan build` command?
It would run Conan's logic responsible for copying the content of source folder over to the build folder, like in a call to `conan create` with the default `no_copy_source=False`.
To summarize, investigate if we should remove (or not extract) the `._ ` osx files from the downloaded tgz`s.
I've been reading around and those files are intended to store only metadata about associated files. They are created and deleted on the fly (creating tar file, copying files,...) depending on the underlying storage, if it cannot store this metadata those files will be there. All the comments agree that those files are just metadata and can be deleted (Macos provides a command for it: `dot_clean`, and also the env variable you0ve talked about).

People agree that **all** the `._*` files can be deleted, but I think we can be more conservative and delete just the files that matches another one with the same name in the same directory (just in case). So a function following these lines would be enough:

```python
def dot_clean(folder):
    files = os.listdir(folder)
    for f in files:
        full_name = os.path.join(folder, f)
        if os.path.isdir(full_name):
            dot_clean(full_name)
        elif f.startswith("._"):
            if os.path.exists(os.path.join(folder, f[2:])):
                os.remove(full_name)
```

I would delete these files after extraction and also in the local cache after copying sources, to check that nobody is using them inside the recipe during creation. And I wouldn't provide an opt-out (do you think someone will ever need it?)

----
There are also some other _weird_ files associated to Mac like `.DS_Store` and `.Trashes` (dir). Also, in other osys we can find things like `Thumbs.db`,... shall we take care of all of them too?

Resolving patch:
diff --git a/conans/client/conan_api.py b/conans/client/conan_api.py
--- a/conans/client/conan_api.py
+++ b/conans/client/conan_api.py
@@ -922,12 +922,12 @@ def read_profile(self, profile=None):
 
     @api_method
     def get_path(self, reference, package_id=None, path=None, remote_name=None):
-        from conans.client.local_file_getter import get_path
         reference = ConanFileReference.loads(reference)
         if not path:
             path = "conanfile.py" if not package_id else "conaninfo.txt"
 
         if not remote_name:
+            from conans.client.local_file_getter import get_path
             return get_path(self._client_cache, reference, package_id, path), path
         else:
             remote = self.get_remote_by_name(remote_name)
diff --git a/conans/client/tools/apple.py b/conans/client/tools/apple.py
--- a/conans/client/tools/apple.py
+++ b/conans/client/tools/apple.py
@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
-
+import os
 import subprocess
 
 
@@ -131,3 +131,14 @@ def strip(self):
     def libtool(self):
         """path to libtool"""
         return self.find('libtool')
+
+
+def apple_dot_clean(folder):
+    files = os.listdir(folder)
+    for f in files:
+        full_name = os.path.join(folder, f)
+        if os.path.isdir(full_name):
+            apple_dot_clean(full_name)
+        elif f.startswith("._"):
+            if os.path.exists(os.path.join(folder, f[2:])):
+                os.remove(full_name)
\ No newline at end of file
diff --git a/conans/client/tools/files.py b/conans/client/tools/files.py
--- a/conans/client/tools/files.py
+++ b/conans/client/tools/files.py
@@ -363,3 +363,5 @@ def unix2dos(filepath):
 
 def dos2unix(filepath):
     _replace_with_separator(filepath, "\n")
+
+


Test patch:
diff --git a/conans/test/functional/tgz_macos_dot_files.py b/conans/test/functional/tgz_macos_dot_files.py
new file mode 100644
--- /dev/null
+++ b/conans/test/functional/tgz_macos_dot_files.py
@@ -0,0 +1,103 @@
+import os
+import platform
+import shutil
+import subprocess
+import tempfile
+import textwrap
+import unittest
+
+from conans.client.remote_manager import uncompress_file
+from conans.model.ref import ConanFileReference, PackageReference
+from conans.paths import EXPORT_SOURCES_TGZ_NAME
+from conans.test.utils.tools import TestBufferConanOutput
+from conans.test.utils.tools import TestClient, TestServer
+
+
+@unittest.skipUnless(platform.system() == "Darwin", "Requires OSX")
+class TgzMacosDotFilesTest(unittest.TestCase):
+
+    def _add_macos_metadata_to_file(self, filepath):
+        subprocess.call(["xattr", "-w", "name", "value", filepath])
+
+    def _test_for_metadata_in_zip_file(self, tgz, annotated_file, dot_file_expected):
+        tmp_folder = tempfile.mkdtemp()
+        try:
+            uncompress_file(src_path=tgz,
+                            dest_folder=tmp_folder,
+                            output=TestBufferConanOutput())
+            self.assertTrue(os.path.exists(os.path.join(tmp_folder, annotated_file)))
+            self.assertEqual(dot_file_expected,
+                             os.path.exists(os.path.join(tmp_folder, "._" + annotated_file)))
+        finally:
+            shutil.rmtree(tmp_folder)
+
+    def _test_for_metadata(self, folder, annotated_file, dot_file_expected):
+        """ We want to check if the file has metadata associated: Mac creates the
+            ._ files at the moment of creating a tar file in order to send the
+            metadata associated to every file. """
+        self.assertTrue(os.path.exists(os.path.join(folder, annotated_file)))
+
+        tmp_folder = tempfile.mkdtemp()
+        try:
+            tgz = os.path.join(tmp_folder, 'compressed.tgz')
+            subprocess.call(["tar", "-zcvf", tgz, "-C", folder, "."])
+            self._test_for_metadata_in_zip_file(tgz, annotated_file, dot_file_expected)
+        finally:
+            shutil.rmtree(tmp_folder)
+
+    def test_dot_files(self):
+        """ Check behavior related to ._ files in Macos OS
+
+            Macos has the ability to store metadata associated to files. This metadata can be
+            stored in the HFS+ (Apple native) or Unix/UFS volumes, but if the store does
+            not have this capability it will be placed in a ._ file. So these files will
+            automatically be created by the OS when it is creating a package in order to
+            send this information.
+
+            Nevertheless, Conan is using the Python libraries to copy, tar and untar files
+            and this metadata is lost when using them. It can avoid some problems like
+            #3529 but there is missing information that can be valuable at some point in time.
+
+            This test is here just to be sure that the behavior is not changed without
+            noticing.
+            """
+
+        conanfile = textwrap.dedent("""\
+            from conans import ConanFile
+            
+            class Lib(ConanFile):
+                name = "lib"
+                version = "version"
+                exports_sources = "file.txt"
+                
+                def package(self):
+                    self.copy("file.txt")
+            """)
+
+        default_server = TestServer()
+        servers = {"default": default_server}
+        t = TestClient(path_with_spaces=False, servers=servers,
+                       users={"default": [("lasote", "mypass")]})
+        t.save(files={'conanfile.py': conanfile, 'file.txt': "content"})
+        self._add_macos_metadata_to_file(os.path.join(t.current_folder, 'file.txt'))
+        t.run("create . lasote/channel")
+
+        # Check if the metadata travels through the Conan commands
+        pkg_ref = PackageReference.loads(
+            "lib/version@lasote/channel:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9")
+        pkg_folder = t.client_cache.package(pkg_ref)
+
+        # 1) When copied to the package folder, the metadata is lost
+        self._test_for_metadata(pkg_folder, 'file.txt', dot_file_expected=False)
+
+        # 2) If we add metadata to a file, it will be there
+        self._add_macos_metadata_to_file(os.path.join(pkg_folder, 'file.txt'))
+        self._test_for_metadata(pkg_folder, 'file.txt', dot_file_expected=True)
+
+        # 3) In the upload process, the metadata is lost again
+        ref = ConanFileReference.loads("lib/version@lasote/channel")
+        export_folder = t.client_cache.export(ref)
+        tgz = os.path.join(export_folder, EXPORT_SOURCES_TGZ_NAME)
+        self.assertFalse(os.path.exists(tgz))
+        t.run("upload lib/version@lasote/channel")
+        self._test_for_metadata_in_zip_file(tgz, 'file.txt', dot_file_expected=False)
diff --git a/conans/test/unittests/client/tools/test_files.py b/conans/test/unittests/client/tools/test_files.py
new file mode 100644
--- /dev/null
+++ b/conans/test/unittests/client/tools/test_files.py
@@ -0,0 +1,44 @@
+# coding=utf-8
+
+import os
+import unittest
+
+from conans.client.tools.files import save, chdir
+from conans.client.tools import apple_dot_clean
+from conans.test.utils.test_files import temp_folder
+
+
+class DotCleanTest(unittest.TestCase):
+
+    def _run_test(self, tuples):
+        tmp_folder = temp_folder()
+        with chdir(tmp_folder):
+            for f, _ in tuples:
+                save(f, "")
+
+            apple_dot_clean(".")
+
+            for f, expected_after in tuples:
+                self.assertEqual(expected_after, os.path.exists(f))
+
+    def test_removal_normal(self):
+        files = [("file.txt", True),
+                 ("._file.txt", False),
+                 ("folder/file.txt", True),
+                 ("folder/._file.txt", False),]
+        self._run_test(tuples=files)
+
+    def test_only_remove_matching_ones(self):
+        files = [("file.txt", True),
+                 ("._file.txt", False),
+                 ("._other.txt", True)]
+        self._run_test(tuples=files)
+
+    def test_handle_dirs(self):
+        files = [("folder/file.txt", True),
+                 ("folder/._file.txt", False),
+                 ("._folder/file.txt", True),
+                 ("._folder/._file.txt", False),
+                 ("._other/._file.txt", True),
+                 ("._other2/file.txt", True)]
+        self._run_test(tuples=files)