Repository: conan-io/conan

Issue:
[feature] Automatic directory management in source unpack support
Debian's `dpkg-source` has a nice feature when unpacking upstream tarballs to ensure that the target directory always has a specific name. It will do the following:

 - Create the target directory that should be created
 - Unpack within that directory
 - If the top-level contents of the archive was just a single directory, move the contents of that directory one level up and remove that directory

The following situations exist then:

 - Archive contains a directory tree in the following form:
    `dir/`
    `dir/CMakeLists.txt`
    `dir/sourcefile.c`
  Calling `dpkg-source -x` would create a directory of the expected name (in Debian's case `package-version`) and that directory would contain the `CMakeLists.txt` file and the `sourcefile.c` file - regardless of the name `dir/` **within** the archive.

 - Archive contains a directory tree in the following form:
   `CMakeLists.txt`
   `sourcefile.c`
   Calling `dpkg-source -x` would create the same result as before, even though the archive doesn't contain a subdirectory.

I think something like that would be interesting for conan, because I see the following pattern very often in recipies:

 - unpack source archive
 - rename directory to `_source_subfolder` (or something otherwise deterministic)

Since the rename step requires the user to know the directory name within the source archive, and projects might change that name between releases, this can add quite a bit of complexity to the conan file.

Since this changes the behavior of unzip / get I would suggest adding a keyword argument `autodir` (default `False`) that introduces this new behavior.

Example usage could be:

    tools.get(url, sha256=hash, destination='_source_subfolder', autodir=True)

This would ensure that the contents of the downloaded archive is available in the directory `_source_subfolder` according to the described algorithm, regardless of how the archive was organized internally.

I could implement this and provide a pull request if this is something you'd be interested in adding to conan.

- [x] I've read the [CONTRIBUTING guide](https://github.com/conan-io/conan/blob/develop/.github/CONTRIBUTING.md).



Discussion:
Hi! I've seen this behavior on desktop tools too, and it is useful in general terms, but for an automated tool... I think it is not so annoying to have that _extra_ directory. I would like to hear more feedback from the community that is contributing with recipes to Conan Center to know if it would be useful or they will still rename the directory to `_source_subfolder` (cc/ @conan-io/community-leaders ).

If it is really useful, as an opt-in, it won't break anything and we can consider including it.

Thanks for the suggestion!
Just fyi, because I feel that I probably wasn't completely clear in my initial posting based on your response: the main issue I see is not the fact that there might be an extra directory (there'll be an extra directory anyway, even with this scheme), but that the name of the extra directory has to be constructed every time from the package version, adding a lot of boiler-plate code - especially if upstream doesn't follow the packagename-version scheme. (Most packages do, but some don't.)

Anyway, thanks for considering this, and as I said, I'd be willing to implement this if you approve.
I see, it is about providing a `destination` that is the final directory, so the result of.

```
tools.get(url='https://.../lib-2.0.tar.gz', destination='_source_subfolder',...)
```

would be:

```
.../_souce_subfolder/
├───CMakeLists.txt
├───sourcefile.c
```

instead of:

```
.../_souce_subfolder/lib-2.0/
├───CMakeLists.txt
├───sourcefile.c
```

----

 From the UX perspective, I find it a useful feature, although the `tarfile` library, which is the one we are using to unzip the files, doesn't have that feature available ([docs here](https://docs.python.org/3/library/tarfile.html#tarfile.TarFile.extractall)):

```
TarFile.extractall(path=".", members=None, *, numeric_owner=False)
```

> Extract all members from the archive to the current working directory or directory path.

As a Conan developer, I prefer to keep things simple and, whenever it is possible, to keep the same signature as the underlying functions we are calling instead of adding a layer of functionality on top of it. That's my only concern, but I'll tag this as a feature request, and share it with the team, so we can prioritize it. Nevertheless, if you want to contribute to Conan and you create the PR we will consider it.

👍 @chris-se, waiting for your PR, if you need help writing the feature or the tests, just ping me. Thanks!
Thanks! I'll start working on it.
thanks for the suggestion. for the time being, I was thinking about the same. I have simple implementation (not for conan, for my own scripts):
```
def extract(filename):
    if filename.endswith(".zip"):
        f = zipfile.ZipFile(filename, 'r')
        names = f.namelist()
    else:
        f = tarfile.open(filename, 'r')
        names = f.getnames()
    dstdir = TMP
    dirname = os.path.commonprefix(names)
    if not dirname:
        dirname = os.path.splitext(filename)[0]
        dstdir = os.path.join(TMP, dirname)
    dirname = os.path.join(TMP, dirname)
    if os.path.isdir(dirname):
        logging.info('extracting "%s" into "%s"... (cached)' % (filename, dirname))
    else:
        logging.info('extracting "%s" into "%s"...' % (filename, dirname))
        f.extractall(path=dstdir)
    return dirname
```
It'd be really great to have that for the CCI. Some archive names are harder to decipher, like for example ones from direct commit hash links - you either pass the hash to the recipe or glob downloaded files, both ways are inconvenient. 
It would be awesome to have that!
As there has not been progress here, and we want to release 1.23, removing the milestone. When we have something to review, we will plan the release to be merged in. Thanks!

Resolving patch:
diff --git a/conans/client/tools/files.py b/conans/client/tools/files.py
--- a/conans/client/tools/files.py
+++ b/conans/client/tools/files.py
@@ -55,7 +55,8 @@ def human_size(size_bytes):
     return "%s%s" % (formatted_size, suffix)
 
 
-def unzip(filename, destination=".", keep_permissions=False, pattern=None, output=None):
+def unzip(filename, destination=".", keep_permissions=False, pattern=None, output=None,
+          strip_root=False):
     """
     Unzip a zipped file
     :param filename: Path to the zip file
@@ -67,6 +68,7 @@ def unzip(filename, destination=".", keep_permissions=False, pattern=None, outpu
     :param pattern: Extract only paths matching the pattern. This should be a
     Unix shell-style wildcard, see fnmatch documentation for more details.
     :param output: output
+    :param flat: If all the contents are in a single dir, flat that directory.
     :return:
     """
     output = default_output(output, 'conans.client.tools.files.unzip')
@@ -74,7 +76,7 @@ def unzip(filename, destination=".", keep_permissions=False, pattern=None, outpu
     if (filename.endswith(".tar.gz") or filename.endswith(".tgz") or
             filename.endswith(".tbz2") or filename.endswith(".tar.bz2") or
             filename.endswith(".tar")):
-        return untargz(filename, destination, pattern)
+        return untargz(filename, destination, pattern, strip_root)
     if filename.endswith(".gz"):
         with gzip.open(filename, 'rb') as f:
             file_content = f.read()
@@ -84,7 +86,7 @@ def unzip(filename, destination=".", keep_permissions=False, pattern=None, outpu
     if filename.endswith(".tar.xz") or filename.endswith(".txz"):
         if six.PY2:
             raise ConanException("XZ format not supported in Python 2. Use Python 3 instead")
-        return untargz(filename, destination, pattern)
+        return untargz(filename, destination, pattern, strip_root)
 
     import zipfile
     full_path = os.path.normpath(os.path.join(get_cwd(), destination))
@@ -103,10 +105,21 @@ def print_progress(_, __):
             pass
 
     with zipfile.ZipFile(filename, "r") as z:
-        if not pattern:
-            zip_info = z.infolist()
-        else:
-            zip_info = [zi for zi in z.infolist() if fnmatch(zi.filename, pattern)]
+        zip_info = z.infolist()
+        if pattern:
+            zip_info = [zi for zi in zip_info if fnmatch(zi.filename, pattern)]
+        if strip_root:
+            names = [n.replace("\\", "/") for n in z.namelist()]
+            common_folder = os.path.commonprefix(names).split("/", 1)[0]
+            if not common_folder and len(names) > 1:
+                raise ConanException("The zip file contains more than 1 folder in the root")
+            if len(names) == 1 and len(names[0].split("/", 1)) == 1:
+                raise ConanException("The zip file contains a file in the root")
+
+            for member in zip_info:
+                name = member.filename.replace("\\", "/")
+                member.filename = name.split("/", 1)[1]
+
         uncompress_size = sum((file_.file_size for file_ in zip_info))
         if uncompress_size > 100000:
             output.info("Unzipping %s, this can take a while" % human_size(uncompress_size))
@@ -139,14 +152,29 @@ def print_progress(_, __):
         output.writeln("")
 
 
-def untargz(filename, destination=".", pattern=None):
+def untargz(filename, destination=".", pattern=None, strip_root=False):
     import tarfile
     with tarfile.TarFile.open(filename, 'r:*') as tarredgzippedFile:
-        if not pattern:
+        if not pattern and not strip_root:
             tarredgzippedFile.extractall(destination)
         else:
-            members = list(filter(lambda m: fnmatch(m.name, pattern),
-                                  tarredgzippedFile.getmembers()))
+            members = tarredgzippedFile.getmembers()
+
+            if strip_root:
+                names = [n.replace("\\", "/") for n in tarredgzippedFile.getnames()]
+                common_folder = os.path.commonprefix(names).split("/", 1)[0]
+                if not common_folder and len(names) > 1:
+                    raise ConanException("The tgz file contains more than 1 folder in the root")
+                if len(names) == 1 and len(names[0].split("/", 1)) == 1:
+                    raise ConanException("The tgz file contains a file in the root")
+
+                for member in members:
+                    name = member.name.replace("\\", "/")
+                    member.name = name.split("/", 1)[1]
+                    member.path = member.name
+            if pattern:
+                members = list(filter(lambda m: fnmatch(m.name, pattern),
+                                      tarredgzippedFile.getmembers()))
             tarredgzippedFile.extractall(destination, members=members)
 
 
diff --git a/conans/client/tools/net.py b/conans/client/tools/net.py
--- a/conans/client/tools/net.py
+++ b/conans/client/tools/net.py
@@ -8,7 +8,7 @@
 
 def get(url, md5='', sha1='', sha256='', destination=".", filename="", keep_permissions=False,
         pattern=None, requester=None, output=None, verify=True, retry=None, retry_wait=None,
-        overwrite=False, auth=None, headers=None):
+        overwrite=False, auth=None, headers=None, strip_root=False):
     """ high level downloader + unzipper + (optional hash checker) + delete temporary zip
     """
 
@@ -23,7 +23,7 @@ def get(url, md5='', sha1='', sha256='', destination=".", filename="", keep_perm
              retry_wait=retry_wait, overwrite=overwrite, auth=auth, headers=headers,
              md5=md5, sha1=sha1, sha256=sha256)
     unzip(filename, destination=destination, keep_permissions=keep_permissions, pattern=pattern,
-          output=output)
+          output=output, strip_root=strip_root)
     os.unlink(filename)
 
 


Test patch:
diff --git a/conans/test/functional/command/config_install_test.py b/conans/test/functional/command/config_install_test.py
--- a/conans/test/functional/command/config_install_test.py
+++ b/conans/test/functional/command/config_install_test.py
@@ -4,7 +4,6 @@
 import textwrap
 import time
 import unittest
-import zipfile
 
 import pytest
 import six
@@ -17,7 +16,7 @@
 from conans.errors import ConanException
 from conans.test.assets.genconanfile import GenConanfile
 from conans.test.utils.test_files import temp_folder
-from conans.test.utils.tools import TestClient, StoppableThreadBottle
+from conans.test.utils.tools import TestClient, StoppableThreadBottle, zipdir
 from conans.util.files import load, mkdir, save, save_files, make_file_read_only
 
 win_profile = """[settings]
@@ -72,17 +71,6 @@
 """
 
 
-def zipdir(path, zipfilename):
-    with zipfile.ZipFile(zipfilename, 'w', zipfile.ZIP_DEFLATED) as z:
-        for root, _, files in os.walk(path):
-            for f in files:
-                file_path = os.path.join(root, f)
-                if file_path == zipfilename:
-                    continue
-                relpath = os.path.relpath(file_path, path)
-                z.write(file_path, relpath)
-
-
 class ConfigInstallTest(unittest.TestCase):
     def setUp(self):
         self.client = TestClient()
diff --git a/conans/test/unittests/util/files/strip_root_extract_test.py b/conans/test/unittests/util/files/strip_root_extract_test.py
new file mode 100644
--- /dev/null
+++ b/conans/test/unittests/util/files/strip_root_extract_test.py
@@ -0,0 +1,191 @@
+import os
+import tarfile
+import unittest
+
+import six
+
+from conans.client.tools import untargz, unzip
+from conans.client.tools.files import chdir, save
+from conans.test.utils.test_files import temp_folder
+from conans.errors import ConanException
+from conans.model.manifest import gather_files
+from conans.test.functional.command.config_install_test import zipdir
+from conans.util.files import gzopen_without_timestamps
+
+
+class ZipExtractPlainTest(unittest.TestCase):
+
+    def test_plain_zip(self):
+        tmp_folder = temp_folder()
+        with chdir(tmp_folder):
+            ori_files_dir = os.path.join(tmp_folder, "subfolder-1.2.3")
+            file1 = os.path.join(ori_files_dir, "file1")
+            file2 = os.path.join(ori_files_dir, "folder", "file2")
+            file3 = os.path.join(ori_files_dir, "file3")
+
+            save(file1, "")
+            save(file2, "")
+            save(file3, "")
+
+        zip_file = os.path.join(tmp_folder, "myzip.zip")
+        zipdir(tmp_folder, zip_file)
+
+        # Tgz unzipped regularly
+        extract_folder = temp_folder()
+        unzip(zip_file, destination=extract_folder, strip_root=False)
+        self.assertTrue(os.path.exists(os.path.join(extract_folder, "subfolder-1.2.3")))
+        self.assertTrue(os.path.exists(os.path.join(extract_folder, "subfolder-1.2.3", "file1")))
+        self.assertTrue(os.path.exists(os.path.join(extract_folder, "subfolder-1.2.3", "folder",
+                                                    "file2")))
+        self.assertTrue(os.path.exists(os.path.join(extract_folder, "subfolder-1.2.3", "file3")))
+
+        # Extract without the subfolder
+        extract_folder = temp_folder()
+        unzip(zip_file, destination=extract_folder, strip_root=True)
+        self.assertFalse(os.path.exists(os.path.join(extract_folder, "subfolder-1.2.3")))
+        self.assertTrue(os.path.exists(os.path.join(extract_folder, "file1")))
+        self.assertTrue(os.path.exists(os.path.join(extract_folder, "folder", "file2")))
+        self.assertTrue(os.path.exists(os.path.join(extract_folder, "file3")))
+
+    def test_invalid_flat(self):
+        tmp_folder = temp_folder()
+        with chdir(tmp_folder):
+            # Not a single dir containing everything
+            file1 = os.path.join(tmp_folder, "subfolder-1.2.3", "folder2", "file1")
+            file2 = os.path.join(tmp_folder, "other-1.2.3", "folder", "file2")
+
+            save(file1, "")
+            save(file2, "")
+
+        zip_folder = temp_folder()
+        zip_file = os.path.join(zip_folder, "file.zip")
+        zipdir(tmp_folder, zip_file)
+
+        # Extract without the subfolder
+        extract_folder = temp_folder()
+        with six.assertRaisesRegex(self, ConanException, "The zip file contains more than 1 folder "
+                                                         "in the root"):
+            unzip(zip_file, destination=extract_folder, strip_root=True)
+
+    def test_invalid_flat_single_file(self):
+        tmp_folder = temp_folder()
+        with chdir(tmp_folder):
+            save("file1", "contentsfile1")
+
+        zip_folder = temp_folder()
+        zip_file = os.path.join(zip_folder, "file.zip")
+        zipdir(tmp_folder, zip_file)
+
+        # Extract without the subfolder
+        extract_folder = temp_folder()
+        with six.assertRaisesRegex(self, ConanException, "The zip file contains a file in the root"):
+            unzip(zip_file, destination=extract_folder, strip_root=True)
+
+
+class TarExtractPlainTest(unittest.TestCase):
+
+    def _compress_folder(self, folder, tgz_path):
+        # Create a tar.gz file with the files in the folder
+        with open(tgz_path, "wb") as tgz_handle:
+            tgz = gzopen_without_timestamps("name", mode="w", fileobj=tgz_handle)
+
+            files, _ = gather_files(folder)
+            for filename, abs_path in files.items():
+                info = tarfile.TarInfo(name=filename)
+                with open(os.path.join(folder, filename), 'rb') as file_handler:
+                    tgz.addfile(tarinfo=info, fileobj=file_handler)
+            tgz.close()
+
+    def test_plain_tgz(self):
+
+        tmp_folder = temp_folder()
+        with chdir(tmp_folder):
+            # Create a couple of files
+            ori_files_dir = os.path.join(tmp_folder, "subfolder-1.2.3")
+            file1 = os.path.join(ori_files_dir, "file1")
+            file2 = os.path.join(ori_files_dir, "folder", "file2")
+            file3 = os.path.join(ori_files_dir, "file3")
+
+            save(file1, "")
+            save(file2, "")
+            save(file3, "")
+
+        tgz_folder = temp_folder()
+        tgz_file = os.path.join(tgz_folder, "file.tar.gz")
+        self._compress_folder(tmp_folder, tgz_file)
+
+        # Tgz unzipped regularly
+        extract_folder = temp_folder()
+        untargz(tgz_file, destination=extract_folder, strip_root=False)
+        self.assertTrue(os.path.exists(os.path.join(extract_folder, "subfolder-1.2.3")))
+        self.assertTrue(os.path.exists(os.path.join(extract_folder, "subfolder-1.2.3", "file1")))
+        self.assertTrue(os.path.exists(os.path.join(extract_folder, "subfolder-1.2.3", "folder",
+                                                    "file2")))
+        self.assertTrue(os.path.exists(os.path.join(extract_folder, "subfolder-1.2.3", "file3")))
+
+        # Extract without the subfolder
+        extract_folder = temp_folder()
+        untargz(tgz_file, destination=extract_folder, strip_root=True)
+        self.assertFalse(os.path.exists(os.path.join(extract_folder, "subfolder-1.2.3")))
+        self.assertTrue(os.path.exists(os.path.join(extract_folder, "file1")))
+        self.assertTrue(os.path.exists(os.path.join(extract_folder, "folder", "file2")))
+        self.assertTrue(os.path.exists(os.path.join(extract_folder, "file3")))
+
+    def test_plain_tgz_common_base(self):
+
+        tmp_folder = temp_folder()
+        with chdir(tmp_folder):
+            # Create a couple of files
+            ori_files_dir = os.path.join(tmp_folder, "subfolder-1.2.3")
+            file1 = os.path.join(ori_files_dir, "folder", "file1")
+            file2 = os.path.join(ori_files_dir, "folder", "file2")
+            file3 = os.path.join(ori_files_dir, "folder", "file3")
+
+            save(file1, "")
+            save(file2, "")
+            save(file3, "")
+
+        tgz_folder = temp_folder()
+        tgz_file = os.path.join(tgz_folder, "file.tar.gz")
+        self._compress_folder(tmp_folder, tgz_file)
+
+        # Tgz unzipped regularly
+        extract_folder = temp_folder()
+        untargz(tgz_file, destination=extract_folder, strip_root=True)
+        self.assertFalse(os.path.exists(os.path.join(extract_folder, "subfolder-1.2.3")))
+        self.assertTrue(os.path.exists(os.path.join(extract_folder, "folder", "file1")))
+        self.assertTrue(os.path.exists(os.path.join(extract_folder, "folder", "file2")))
+        self.assertTrue(os.path.exists(os.path.join(extract_folder, "folder", "file3")))
+
+    def test_invalid_flat(self):
+        tmp_folder = temp_folder()
+        with chdir(tmp_folder):
+            # Not a single dir containing everything
+            file1 = os.path.join(tmp_folder, "subfolder-1.2.3", "folder2", "file1")
+            file2 = os.path.join(tmp_folder, "other-1.2.3", "folder", "file2")
+
+            save(file1, "")
+            save(file2, "")
+
+        tgz_folder = temp_folder()
+        tgz_file = os.path.join(tgz_folder, "file.tar.gz")
+        self._compress_folder(tmp_folder, tgz_file)
+
+        extract_folder = temp_folder()
+        with six.assertRaisesRegex(self, ConanException, "The tgz file contains more than 1 folder "
+                                                         "in the root"):
+            untargz(tgz_file, destination=extract_folder, strip_root=True)
+
+    def test_invalid_flat_single_file(self):
+        tmp_folder = temp_folder()
+        with chdir(tmp_folder):
+            save("file1", "contentsfile1")
+
+        zip_folder = temp_folder()
+        tgz_file = os.path.join(zip_folder, "file.tar.gz")
+        self._compress_folder(tmp_folder, tgz_file)
+
+        # Extract without the subfolder
+        extract_folder = temp_folder()
+        with six.assertRaisesRegex(self, ConanException, "The tgz file contains a file in the root"):
+            unzip(tgz_file, destination=extract_folder, strip_root=True)
diff --git a/conans/test/unittests/util/tools_test.py b/conans/test/unittests/util/tools_test.py
--- a/conans/test/unittests/util/tools_test.py
+++ b/conans/test/unittests/util/tools_test.py
@@ -21,6 +21,7 @@
 from conans.client.conan_api import ConanAPIV1
 from conans.client.conf import get_default_settings_yml, get_default_client_conf
 from conans.client.output import ConanOutput
+from conans.client.tools import chdir
 from conans.client.tools.files import replace_in_file, which
 from conans.client.tools.oss import OSInfo
 from conans.client.tools.win import vswhere
@@ -29,7 +30,7 @@
 from conans.model.settings import Settings
 from conans.test.utils.mocks import ConanFileMock, TestBufferConanOutput
 from conans.test.utils.test_files import temp_folder
-from conans.test.utils.tools import StoppableThreadBottle, TestClient
+from conans.test.utils.tools import StoppableThreadBottle, TestClient, zipdir
 from conans.tools import get_global_instances
 from conans.util.env_reader import get_env
 from conans.util.files import load, md5, mkdir, save
@@ -709,6 +710,33 @@ def error_url():
         # Not found error
         self.assertEqual(str(out).count("Waiting 0 seconds to retry..."), 2)
 
+    def test_get_unzip_strip_root(self):
+        """Test that the strip_root mechanism from the underlying unzip
+          is called if I call the tools.get by checking that the exception of an invalid zip to
+          flat is raised"""
+
+        zip_folder = temp_folder()
+
+        def mock_download(*args, **kwargs):
+            tmp_folder = temp_folder()
+            with chdir(tmp_folder):
+                ori_files_dir = os.path.join(tmp_folder, "subfolder-1.2.3")
+                file1 = os.path.join(ori_files_dir, "file1")
+                file2 = os.path.join(ori_files_dir, "folder", "file2")
+                # !!! This file is not under the root "subfolder-1.2.3"
+                file3 = os.path.join("file3")
+                save(file1, "")
+                save(file2, "")
+                save(file3, "")
+                zip_file = os.path.join(zip_folder, "file.zip")
+                zipdir(tmp_folder, zip_file)
+
+        with six.assertRaisesRegex(self, ConanException, "The zip file contains more than 1 "
+                                                         "folder in the root"):
+            with patch('conans.client.tools.net.download', new=mock_download):
+                with chdir(zip_folder):
+                    tools.get("file.zip", strip_root=True)
+
     @attr('slow')
     @pytest.mark.slow
     @attr('local_bottle')
diff --git a/conans/test/utils/tools.py b/conans/test/utils/tools.py
--- a/conans/test/utils/tools.py
+++ b/conans/test/utils/tools.py
@@ -8,6 +8,7 @@
 import threading
 import time
 import uuid
+import zipfile
 from collections import OrderedDict
 from contextlib import contextmanager
 
@@ -896,3 +897,14 @@ def stop(self):
     def run_server(self):
         self.start()
         time.sleep(1)
+
+
+def zipdir(path, zipfilename):
+    with zipfile.ZipFile(zipfilename, 'w', zipfile.ZIP_DEFLATED) as z:
+        for root, _, files in os.walk(path):
+            for f in files:
+                file_path = os.path.join(root, f)
+                if file_path == zipfilename:
+                    continue
+                relpath = os.path.relpath(file_path, path)
+                z.write(file_path, relpath)