Repository: pypa/pip

Issue:
Pip processes corrupt packages containing  using ../ chars placing files outside target directory
- Pip version: tested on pip versions 6.0.8 and  8.0.2,
- Python version: 2.7
- Operating System: Linux / Windows
### Description:

pip processes invalid packages .tar/.zip archives containing "../" directory traversal chars and places package files outside of target directories even after pip complains and throws errors and aborts the install.

Pip processes corrupt packages containing  using ../ chars placing files outside target directory
- Pip version: tested on pip versions 6.0.8 and  8.0.2,
- Python version: 2.7
- Operating System: Linux / Windows
### Description:

pip processes invalid packages .tar/.zip archives containing "../" directory traversal chars and places package files outside of target directories even after pip complains and throws errors and aborts the install.



Discussion:
This has been open quite a while, and it's something Sonatype (who own Nexus) say is still present in the latest versions of Pip.  For this reason, any organisation running Nexus IQ at their perimeter finds all versions of the Pip package quarantined.

Not wishing to be rude, but is there a reason it's never been fixed in 2 and a half years?  I work with people who could maybe work on a patch, but I'd like to know in advance if there's some genuine blocker, or whether it's just fallen through the cracks.
Mainly just that no-one has offered a PR, I suspect. That plus the fact that we don't really offer any guarantees if you feed invalid package files to pip, so fixing this one specific issue is probably only the tip of the iceberg, and we don't have anything like the resources to provide any guarantees here.

But if you want to produce a PR, there's no reason I can see that we'd reject it.
Is there any backwards compatibility concern here?
i raised a PR simply add a warning message if the output tar file is placed outside of target location 
FYI, someone is working on a PR for this issue, PR #6313. However, the PR doesn't prevent files from being placed outside the target directory. It just logs a warning and says the behavior is deprecated.

Personally, I think we should simply prevent this from happening as it's a security issue and can cause data loss. What do people on this issue think?

Also, as a secondary point, if pip is already complaining by throwing an error and aborting the install as the OP said, I don't see what we gain by logging a warning on top of that.
I agree pip should not log a warning but simply not process those files.
Cool, let's stick with aborting.
This has been open quite a while, and it's something Sonatype (who own Nexus) say is still present in the latest versions of Pip.  For this reason, any organisation running Nexus IQ at their perimeter finds all versions of the Pip package quarantined.

Not wishing to be rude, but is there a reason it's never been fixed in 2 and a half years?  I work with people who could maybe work on a patch, but I'd like to know in advance if there's some genuine blocker, or whether it's just fallen through the cracks.
Mainly just that no-one has offered a PR, I suspect. That plus the fact that we don't really offer any guarantees if you feed invalid package files to pip, so fixing this one specific issue is probably only the tip of the iceberg, and we don't have anything like the resources to provide any guarantees here.

But if you want to produce a PR, there's no reason I can see that we'd reject it.
Is there any backwards compatibility concern here?
i raised a PR simply add a warning message if the output tar file is placed outside of target location 
FYI, someone is working on a PR for this issue, PR #6313. However, the PR doesn't prevent files from being placed outside the target directory. It just logs a warning and says the behavior is deprecated.

Personally, I think we should simply prevent this from happening as it's a security issue and can cause data loss. What do people on this issue think?

Also, as a secondary point, if pip is already complaining by throwing an error and aborting the install as the OP said, I don't see what we gain by logging a warning on top of that.
I agree pip should not log a warning but simply not process those files.
Cool, let's stick with aborting.

Resolving patch:
diff --git a/src/pip/_internal/utils/unpacking.py b/src/pip/_internal/utils/unpacking.py
--- a/src/pip/_internal/utils/unpacking.py
+++ b/src/pip/_internal/utils/unpacking.py
@@ -85,6 +85,18 @@ def has_leading_dir(paths):
     return True
 
 
+def is_within_directory(directory, target):
+    # type: ((Union[str, Text]), (Union[str, Text])) -> bool
+    """
+    Return true if the absolute path of target is within the directory
+    """
+    abs_directory = os.path.abspath(directory)
+    abs_target = os.path.abspath(target)
+
+    prefix = os.path.commonprefix([abs_directory, abs_target])
+    return prefix == abs_directory
+
+
 def unzip_file(filename, location, flatten=True):
     # type: (str, str, bool) -> None
     """
@@ -107,6 +119,12 @@ def unzip_file(filename, location, flatten=True):
                 fn = split_leading_dir(name)[1]
             fn = os.path.join(location, fn)
             dir = os.path.dirname(fn)
+            if not is_within_directory(location, fn):
+                message = (
+                    'The zip file ({}) has a file ({}) trying to install '
+                    'outside target directory ({})'
+                )
+                raise InstallationError(message.format(filename, fn, location))
             if fn.endswith('/') or fn.endswith('\\'):
                 # A directory
                 ensure_dir(fn)
@@ -166,6 +184,14 @@ def untar_file(filename, location):
                 # https://github.com/python/mypy/issues/1174
                 fn = split_leading_dir(fn)[1]  # type: ignore
             path = os.path.join(location, fn)
+            if not is_within_directory(location, path):
+                message = (
+                    'The tar file ({}) has a file ({}) trying to install '
+                    'outside target directory ({})'
+                )
+                raise InstallationError(
+                    message.format(filename, path, location)
+                )
             if member.isdir():
                 ensure_dir(path)
             elif member.issym():


Test patch:
diff --git a/tests/unit/test_utils_unpacking.py b/tests/unit/test_utils_unpacking.py
--- a/tests/unit/test_utils_unpacking.py
+++ b/tests/unit/test_utils_unpacking.py
@@ -2,10 +2,19 @@
 import shutil
 import stat
 import sys
+import tarfile
 import tempfile
 import time
+import zipfile
 
-from pip._internal.utils.unpacking import untar_file, unzip_file
+import pytest
+
+from pip._internal.exceptions import InstallationError
+from pip._internal.utils.unpacking import (
+    is_within_directory,
+    untar_file,
+    unzip_file,
+)
 
 
 class TestUnpackArchives(object):
@@ -71,6 +80,27 @@ def confirm_files(self):
                 "mode: %s, expected mode: %s" % (mode, expected_mode)
             )
 
+    def make_zip_file(self, filename, file_list):
+        """
+        Create a zip file for test case
+        """
+        test_zip = os.path.join(self.tempdir, filename)
+        with zipfile.ZipFile(test_zip, 'w') as myzip:
+            for item in file_list:
+                myzip.writestr(item, 'file content')
+        return test_zip
+
+    def make_tar_file(self, filename, file_list):
+        """
+        Create a tar file for test case
+        """
+        test_tar = os.path.join(self.tempdir, filename)
+        with tarfile.open(test_tar, 'w') as mytar:
+            for item in file_list:
+                file_tarinfo = tarfile.TarInfo(item)
+                mytar.addfile(file_tarinfo, 'file content')
+        return test_tar
+
     def test_unpack_tgz(self, data):
         """
         Test unpacking a *.tgz, and setting execute permissions
@@ -90,3 +120,70 @@ def test_unpack_zip(self, data):
         test_file = data.packages.joinpath("test_zip.zip")
         unzip_file(test_file, self.tempdir)
         self.confirm_files()
+
+    def test_unpack_zip_failure(self):
+        """
+        Test unpacking a *.zip with file containing .. path
+        and expect exception
+        """
+        files = ['regular_file.txt', os.path.join('..', 'outside_file.txt')]
+        test_zip = self.make_zip_file('test_zip.zip', files)
+        with pytest.raises(InstallationError) as e:
+            unzip_file(test_zip, self.tempdir)
+        assert 'trying to install outside target directory' in str(e.value)
+
+    def test_unpack_zip_success(self):
+        """
+        Test unpacking a *.zip with regular files,
+        no file will be installed outside target directory after unpack
+        so no exception raised
+        """
+        files = [
+            'regular_file1.txt',
+            os.path.join('dir', 'dir_file1.txt'),
+            os.path.join('dir', '..', 'dir_file2.txt'),
+        ]
+        test_zip = self.make_zip_file('test_zip.zip', files)
+        unzip_file(test_zip, self.tempdir)
+
+    def test_unpack_tar_failure(self):
+        """
+        Test unpacking a *.tar with file containing .. path
+        and expect exception
+        """
+        files = ['regular_file.txt', os.path.join('..', 'outside_file.txt')]
+        test_tar = self.make_tar_file('test_tar.tar', files)
+        with pytest.raises(InstallationError) as e:
+            untar_file(test_tar, self.tempdir)
+        assert 'trying to install outside target directory' in str(e.value)
+
+    def test_unpack_tar_success(self):
+        """
+        Test unpacking a *.tar with regular files,
+        no file will be installed outside target directory after unpack
+        so no exception raised
+        """
+        files = [
+            'regular_file1.txt',
+            os.path.join('dir', 'dir_file1.txt'),
+            os.path.join('dir', '..', 'dir_file2.txt'),
+        ]
+        test_tar = self.make_tar_file('test_tar.tar', files)
+        untar_file(test_tar, self.tempdir)
+
+
+@pytest.mark.parametrize('args, expected', [
+    # Test the second containing the first.
+    (('parent/sub', 'parent/'), False),
+    # Test the first not ending in a trailing slash.
+    (('parent', 'parent/foo'), True),
+    # Test target containing `..` but still inside the parent.
+    (('parent/', 'parent/foo/../bar'), True),
+    # Test target within the parent
+    (('parent/', 'parent/sub'), True),
+    # Test target outside parent
+    (('parent/', 'parent/../sub'), False),
+])
+def test_is_within_directory(args, expected):
+    result = is_within_directory(*args)
+    assert result == expected