Repository: conda/conda-build

Issue:
Windows builds failing if archive contains symlink
<!--
Thanks for opening an issue! A few things to keep in mind:

- This issue tracker is for issues with or enhancements for conda-build.

For conda issues, you should open an issue at https://github.com/conda/conda/issues
For Anaconda issues, you should open an issue at https://github.com/ContinuumIO/anaconda-issues/issues
For an issue with a particular conda-forge package, you should open an issue on the 
corresponding feedstock: https://github.com/conda-forge
-->

### Actual Behavior

In Windows-based builds on AppVeyor, the extraction step crashes if the archive contains a symlink.

For example:

```
Extracting download
Traceback (most recent call last):
  File "C:\Miniconda36-x64\Scripts\conda-build-script.py", line 10, in <module>
    sys.exit(main())
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\cli\main_build.py", line 456, in main
    execute(sys.argv[1:])
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\cli\main_build.py", line 447, in execute
    verify=args.verify, variants=args.variants)
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\api.py", line 208, in build
    notest=notest, need_source_download=need_source_download, variants=variants)
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\build.py", line 2311, in build_tree
    notest=notest,
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\build.py", line 1407, in build
    try_download(m, no_download_source=False, raise_error=True)
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\render.py", line 617, in try_download
    source.provide(metadata)
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\source.py", line 653, in provide
    timeout=metadata.config.timeout, locking=metadata.config.locking)
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\source.py", line 148, in unpack
    tar_xf(src_path, tmpdir)
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\utils.py", line 718, in tar_xf
    libarchive.extract_file(tarball, flags)
  File "C:\Miniconda36-x64\lib\site-packages\libarchive\extract.py", line 70, in extract_file
    extract_entries(archive, flags)
  File "C:\Miniconda36-x64\lib\site-packages\libarchive\extract.py", line 50, in extract_entries
    write_header(write_p, entry._entry_p)
  File "C:\Miniconda36-x64\lib\site-packages\libarchive\ffi.py", line 91, in check_int
    raise archive_error(args[0], retcode)
  File "C:\Miniconda36-x64\lib\site-packages\libarchive\ffi.py", line 75, in archive_error
    raise ArchiveError(msg, errno(archive_p), retcode, archive_p)
libarchive.exception.ArchiveError: Can't create '\\?\C:\bld\tmp5lgmn3gr\nibabel-2.3.2\doc\downloads' (errno=22, retcode=-25, archive_p=429904390704)
```

Note that [`nibabel/doc/downloads`](https://github.com/nipy/nibabel/blob/master/doc/downloads) is a symlink to `nibabel/doc/source/downloads`.

<!-- What actually happens? -->

### Expected Behavior

I would expect any of the following:

1) Symlinks replaced with empty files
2) Symlinks replaced with copies of the linked files/directories (probably cleaned to only permit targets within the archive)

### Steps to Reproduce

I expect, on Windows:

```Python
git clone https://github.com/conda-forge/nibabel-feedstock.git
cd nibabel-feedstock
conda build recipe
```


##### `Output of conda info`
<!-- Between the ticks below, paste the output of 'conda info' -->
```
     active environment : base
    active env location : C:\Miniconda36-x64
            shell level : 1
       user config file : C:\Users\appveyor\.condarc
 populated config files : C:\Users\appveyor\.condarc
          conda version : 4.6.0
    conda-build version : 3.17.5
         python version : 3.6.5.final.0
       base environment : C:\Miniconda36-x64  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Miniconda36-x64\pkgs
                          C:\Users\appveyor\.conda\pkgs
                          C:\Users\appveyor\AppData\Local\conda\conda\pkgs
       envs directories : C:\Miniconda36-x64\envs
                          C:\Users\appveyor\.conda\envs
                          C:\Users\appveyor\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.6.0 requests/2.18.4 CPython/3.6.5 Windows/2012ServerR2 Windows/6.3.9600
          administrator : True
             netrc file : None
           offline mode : False
```

Ref: conda-forge/conda-smithy#1002, #3163.

Known affected builds: conda-forge/nibabel-feedstock#14, conda-forge/openpmd-api-feedstock#20
Windows builds failing if archive contains symlink
<!--
Thanks for opening an issue! A few things to keep in mind:

- This issue tracker is for issues with or enhancements for conda-build.

For conda issues, you should open an issue at https://github.com/conda/conda/issues
For Anaconda issues, you should open an issue at https://github.com/ContinuumIO/anaconda-issues/issues
For an issue with a particular conda-forge package, you should open an issue on the 
corresponding feedstock: https://github.com/conda-forge
-->

### Actual Behavior

In Windows-based builds on AppVeyor, the extraction step crashes if the archive contains a symlink.

For example:

```
Extracting download
Traceback (most recent call last):
  File "C:\Miniconda36-x64\Scripts\conda-build-script.py", line 10, in <module>
    sys.exit(main())
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\cli\main_build.py", line 456, in main
    execute(sys.argv[1:])
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\cli\main_build.py", line 447, in execute
    verify=args.verify, variants=args.variants)
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\api.py", line 208, in build
    notest=notest, need_source_download=need_source_download, variants=variants)
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\build.py", line 2311, in build_tree
    notest=notest,
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\build.py", line 1407, in build
    try_download(m, no_download_source=False, raise_error=True)
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\render.py", line 617, in try_download
    source.provide(metadata)
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\source.py", line 653, in provide
    timeout=metadata.config.timeout, locking=metadata.config.locking)
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\source.py", line 148, in unpack
    tar_xf(src_path, tmpdir)
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\utils.py", line 718, in tar_xf
    libarchive.extract_file(tarball, flags)
  File "C:\Miniconda36-x64\lib\site-packages\libarchive\extract.py", line 70, in extract_file
    extract_entries(archive, flags)
  File "C:\Miniconda36-x64\lib\site-packages\libarchive\extract.py", line 50, in extract_entries
    write_header(write_p, entry._entry_p)
  File "C:\Miniconda36-x64\lib\site-packages\libarchive\ffi.py", line 91, in check_int
    raise archive_error(args[0], retcode)
  File "C:\Miniconda36-x64\lib\site-packages\libarchive\ffi.py", line 75, in archive_error
    raise ArchiveError(msg, errno(archive_p), retcode, archive_p)
libarchive.exception.ArchiveError: Can't create '\\?\C:\bld\tmp5lgmn3gr\nibabel-2.3.2\doc\downloads' (errno=22, retcode=-25, archive_p=429904390704)
```

Note that [`nibabel/doc/downloads`](https://github.com/nipy/nibabel/blob/master/doc/downloads) is a symlink to `nibabel/doc/source/downloads`.

<!-- What actually happens? -->

### Expected Behavior

I would expect any of the following:

1) Symlinks replaced with empty files
2) Symlinks replaced with copies of the linked files/directories (probably cleaned to only permit targets within the archive)

### Steps to Reproduce

I expect, on Windows:

```Python
git clone https://github.com/conda-forge/nibabel-feedstock.git
cd nibabel-feedstock
conda build recipe
```


##### `Output of conda info`
<!-- Between the ticks below, paste the output of 'conda info' -->
```
     active environment : base
    active env location : C:\Miniconda36-x64
            shell level : 1
       user config file : C:\Users\appveyor\.condarc
 populated config files : C:\Users\appveyor\.condarc
          conda version : 4.6.0
    conda-build version : 3.17.5
         python version : 3.6.5.final.0
       base environment : C:\Miniconda36-x64  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Miniconda36-x64\pkgs
                          C:\Users\appveyor\.conda\pkgs
                          C:\Users\appveyor\AppData\Local\conda\conda\pkgs
       envs directories : C:\Miniconda36-x64\envs
                          C:\Users\appveyor\.conda\envs
                          C:\Users\appveyor\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.6.0 requests/2.18.4 CPython/3.6.5 Windows/2012ServerR2 Windows/6.3.9600
          administrator : True
             netrc file : None
           offline mode : False
```

Ref: conda-forge/conda-smithy#1002, #3163.

Known affected builds: conda-forge/nibabel-feedstock#14, conda-forge/openpmd-api-feedstock#20


Discussion:
Hey.

We enable 'developer mode' in our internal CI which is the correct thing to do.

Our team will not have time to look into this unfortunately. The options are:

1. Ask AppVeyor if they can enable developer mode.
2. Move to Azure if it allows / enables developer mode.
3. Implement a fallback in conda-build so that should unpacking fail with a symlink related permissions error, the python tarfile module gets used instead.
4. Implement a fallback in libarchive so that should unpacking fail with a symlink related permissions error, the C code therein falls back to copying instead. This is trickier than it would initially appear because at the point when a symlink is created it is not necessarily the case that the target of the symlink exists due to the ordering of the tarfile, so you may not have any file to copy. This gets worse with symlinks to symlinks to files so must be handled carefully as a post-process. This would be a continuation of my PR at libarchive/libarchive#1030

Cheers!
Overall, I believe if 1 & 2 are not possible, 3 would be pretty easy to implement.
Thanks for reporting! Since the latest updates a week ago, this breaks my windows builds.

Is it possible to work-around this or is someone able to implement suggestion 3? :)
We worked around by making our Python 2-only dependency a regular dependency and going back to `noarch`, so it could build on Linux and install on Windows. It's... not ideal.
I currently work-around by downgrading, which I luckily saw today from others linking here.

Still, won't be a solution for too long and has to be refreshed every rerender.
Hey.

We enable 'developer mode' in our internal CI which is the correct thing to do.

Our team will not have time to look into this unfortunately. The options are:

1. Ask AppVeyor if they can enable developer mode.
2. Move to Azure if it allows / enables developer mode.
3. Implement a fallback in conda-build so that should unpacking fail with a symlink related permissions error, the python tarfile module gets used instead.
4. Implement a fallback in libarchive so that should unpacking fail with a symlink related permissions error, the C code therein falls back to copying instead. This is trickier than it would initially appear because at the point when a symlink is created it is not necessarily the case that the target of the symlink exists due to the ordering of the tarfile, so you may not have any file to copy. This gets worse with symlinks to symlinks to files so must be handled carefully as a post-process. This would be a continuation of my PR at libarchive/libarchive#1030

Cheers!
Overall, I believe if 1 & 2 are not possible, 3 would be pretty easy to implement.
Thanks for reporting! Since the latest updates a week ago, this breaks my windows builds.

Is it possible to work-around this or is someone able to implement suggestion 3? :)
We worked around by making our Python 2-only dependency a regular dependency and going back to `noarch`, so it could build on Linux and install on Windows. It's... not ideal.
I currently work-around by downgrading, which I luckily saw today from others linking here.

Still, won't be a solution for too long and has to be refreshed every rerender.

Resolving patch:
diff --git a/conda_build/index.py b/conda_build/index.py
--- a/conda_build/index.py
+++ b/conda_build/index.py
@@ -41,7 +41,7 @@
 from .conda_interface import MatchSpec, VersionOrder, human_bytes, context
 from .conda_interface import CondaError, CondaHTTPError, get_index, url_path
 from .conda_interface import download, TemporaryDirectory
-from .utils import glob, get_logger, FileNotFoundError, PermissionError
+from .utils import glob, get_logger, FileNotFoundError, PermissionError, tar_xf_file, tar_xf_getnames
 
 try:
     from conda.base.constants import CONDA_TARBALL_EXTENSIONS
@@ -706,7 +706,7 @@ def _cache_recipe(tarball, all_paths, recipe_cache_path):
             )
     recipe_path = next((p for p in recipe_path_search_order if p in all_paths), None)
     if recipe_path:
-        recipe_yaml_binary = _tar_xf_file(tarball, recipe_path)
+        recipe_yaml_binary = tar_xf_file(tarball, recipe_path)
     else:
         recipe_yaml_binary = '{}'
     try:
@@ -725,7 +725,7 @@ def _cache_recipe(tarball, all_paths, recipe_cache_path):
 
 def _cache_about_json(tar_path, about_cache_path):
     try:
-        binary_about_json = _tar_xf_file(tar_path, 'info/about.json')
+        binary_about_json = tar_xf_file(tar_path, 'info/about.json')
     except KeyError:
         log.debug("%s has no file info/about.json" % tar_path)
         binary_about_json = b'{}'
@@ -736,7 +736,7 @@ def _cache_about_json(tar_path, about_cache_path):
 
 def _cache_recipe_log(tar_path, recipe_log_path):
     try:
-        binary_recipe_log = _tar_xf_file(tar_path, 'info/recipe_log.json')
+        binary_recipe_log = tar_xf_file(tar_path, 'info/recipe_log.json')
     except KeyError:
         log.debug("%s has no file info/recipe_log.json (this is OK)" % tar_path)
         binary_recipe_log = b'{}'
@@ -748,11 +748,11 @@ def get_run_exports(tar_or_folder_path):
     run_exports = {}
     if os.path.isfile(tar_or_folder_path):
         try:
-            binary_run_exports = _tar_xf_file(tar_or_folder_path, 'info/run_exports.json')
+            binary_run_exports = tar_xf_file(tar_or_folder_path, 'info/run_exports.json')
             run_exports = json.loads(binary_run_exports.decode("utf-8"))
         except KeyError:
             try:
-                binary_run_exports = _tar_xf_file(tar_or_folder_path, 'info/run_exports.yaml')
+                binary_run_exports = tar_xf_file(tar_or_folder_path, 'info/run_exports.yaml')
                 run_exports = yaml.safe_load(binary_run_exports)
             except KeyError:
                 log.debug("%s has no run_exports file (this is OK)" % tar_or_folder_path)
@@ -777,7 +777,7 @@ def _cache_run_exports(tar_path, run_exports_cache_path):
 
 def _cache_paths_json(tar_path, paths_cache_path):
     try:
-        binary_paths_json = _tar_xf_file(tar_path, 'info/paths.json')
+        binary_paths_json = tar_xf_file(tar_path, 'info/paths.json')
     except KeyError:
         log.debug("%s has no file info/paths.json" % tar_path)
         binary_paths_json = b'{}'
@@ -795,7 +795,7 @@ def _cache_icon(tar_path, recipe_json, all_paths, icon_cache_path):
     app_icon_path = recipe_json.get('app', {}).get('icon') or 'info/icon.png'
     if app_icon_path in all_paths:
         icon_cache_path += splitext(app_icon_path)[-1]
-        binary_icon_data = _tar_xf_file(tar_path, 'info/icon.png')
+        binary_icon_data = tar_xf_file(tar_path, 'info/icon.png')
         with open(icon_cache_path, 'wb') as fh:
             fh.write(binary_icon_data)
 
@@ -848,50 +848,6 @@ def _tmp_chdir(dest):
         os.chdir(curdir)
 
 
-def _tar_xf(tarball, dir_path):
-    flags = libarchive.extract.EXTRACT_TIME | \
-            libarchive.extract.EXTRACT_PERM | \
-            libarchive.extract.EXTRACT_SECURE_NODOTDOT | \
-            libarchive.extract.EXTRACT_SECURE_SYMLINKS | \
-            libarchive.extract.EXTRACT_SECURE_NOABSOLUTEPATHS
-    if not os.path.isabs(tarball):
-        tarball = os.path.join(os.getcwd(), tarball)
-    with _tmp_chdir(dir_path):
-        libarchive.extract_file(tarball, flags)
-
-
-def _tar_xf_file(tarball, entries):
-    from conda_build.utils import ensure_list
-    entries = ensure_list(entries)
-    if not os.path.isabs(tarball):
-        tarball = os.path.join(os.getcwd(), tarball)
-    result = None
-    n_found = 0
-    with libarchive.file_reader(tarball) as archive:
-        for entry in archive:
-            if entry.name in entries:
-                n_found += 1
-                for block in entry.get_blocks():
-                    if result is None:
-                        result = bytes(block)
-                    else:
-                        result += block
-                break
-    if n_found != len(entries):
-        raise KeyError()
-    return result
-
-
-def _tar_xf_getnames(tarball):
-    if not os.path.isabs(tarball):
-        tarball = os.path.join(os.getcwd(), tarball)
-    result = []
-    with libarchive.file_reader(tarball) as archive:
-        for entry in archive:
-            result.append(entry.name)
-    return result
-
-
 def _collect_commits(package_order, hotfix_source_repo, cutoff_time):
     commit_info = {}
 
@@ -1157,10 +1113,10 @@ def _extract_to_cache(self, subdir, fn):
 
             log.debug("hashing, extracting, and caching %s" % tar_path)
             try:
-                binary_index_json = _tar_xf_file(tar_path, 'info/index.json')
+                binary_index_json = tar_xf_file(tar_path, 'info/index.json')
                 index_json = json.loads(binary_index_json.decode('utf-8'))
 
-                all_paths = set(_tar_xf_getnames(tar_path))
+                all_paths = set(tar_xf_getnames(tar_path))
                 _cache_about_json(tar_path, about_cache_path)
                 _cache_run_exports(tar_path, run_exports_cache_path)
                 binary_paths_json = _cache_paths_json(tar_path, paths_cache_path)
diff --git a/conda_build/utils.py b/conda_build/utils.py
--- a/conda_build/utils.py
+++ b/conda_build/utils.py
@@ -706,6 +706,76 @@ def relative(f, d='lib'):
                        '.tar.z', '.tgz', '.whl', '.zip', '.rpm', '.deb')
 
 
+def _tar_xf_fallback(tarball, dir_path, mode='r:*'):
+    if tarball.lower().endswith('.tar.z'):
+        uncompress = external.find_executable('uncompress')
+        if not uncompress:
+            uncompress = external.find_executable('gunzip')
+        if not uncompress:
+            sys.exit("""\
+uncompress (or gunzip) is required to unarchive .z source files.
+""")
+        check_call_env([uncompress, '-f', tarball])
+        tarball = tarball[:-2]
+    if not PY3 and tarball.endswith('.tar.xz'):
+        unxz = external.find_executable('unxz')
+        if not unxz:
+            sys.exit("""\
+unxz is required to unarchive .xz source files.
+""")
+
+        check_call_env([unxz, '-f', '-k', tarball])
+        tarball = tarball[:-3]
+    t = tarfile.open(tarball, mode)
+    members = t.getmembers()
+    for i, member in enumerate(members, 0):
+        if os.path.isabs(member.name):
+            member.name = os.path.relpath(member.name, '/')
+        if not os.path.realpath(member.name).startswith(os.getcwd()):
+            member.name = member.name.replace("../", "")
+        if not os.path.realpath(member.name).startswith(os.getcwd()):
+            sys.exit("tarball contains unsafe path: " + member.name)
+        members[i] = member
+
+    if not PY3:
+        t.extractall(path=dir_path.encode(codec))
+    else:
+        t.extractall(path=dir_path)
+    t.close()
+
+
+def tar_xf_file(tarball, entries):
+    from conda_build.utils import ensure_list
+    entries = ensure_list(entries)
+    if not os.path.isabs(tarball):
+        tarball = os.path.join(os.getcwd(), tarball)
+    result = None
+    n_found = 0
+    with libarchive.file_reader(tarball) as archive:
+        for entry in archive:
+            if entry.name in entries:
+                n_found += 1
+                for block in entry.get_blocks():
+                    if result is None:
+                        result = bytes(block)
+                    else:
+                        result += block
+                break
+    if n_found != len(entries):
+        raise KeyError()
+    return result
+
+
+def tar_xf_getnames(tarball):
+    if not os.path.isabs(tarball):
+        tarball = os.path.join(os.getcwd(), tarball)
+    result = []
+    with libarchive.file_reader(tarball) as archive:
+        for entry in archive:
+            result.append(entry.name)
+    return result
+
+
 def tar_xf(tarball, dir_path):
     flags = libarchive.extract.EXTRACT_TIME | \
             libarchive.extract.EXTRACT_PERM | \
@@ -714,8 +784,17 @@ def tar_xf(tarball, dir_path):
             libarchive.extract.EXTRACT_SECURE_NOABSOLUTEPATHS
     if not os.path.isabs(tarball):
         tarball = os.path.join(os.getcwd(), tarball)
-    with tmp_chdir(dir_path):
-        libarchive.extract_file(tarball, flags)
+    try:
+        with tmp_chdir(dir_path):
+            libarchive.extract_file(tarball, flags)
+    except libarchive.exception.ArchiveError:
+        # try again, maybe we are on Windows and the archive contains symlinks
+        # https://github.com/conda/conda-build/issues/3351
+        # https://github.com/libarchive/libarchive/pull/1030
+        if tarball.endswith(('.tar', '.tar.gz', '.tgz', '.tar.bz2', '.tar.z', '.tar.xz')):
+            _tar_xf_fallback(tarball, dir_path)
+        else:
+            raise
 
 
 def file_info(path):


Test patch: