Repository: conda/conda-build

Issue:
Is a subprocess call necessary to delete one file?
Hi, I noticed in recent versions of `conda-build` (which I like a lot!) that it sometimes hangs for a long time without any progress message. After a quick look in my task explorer, I noticed that `python` was spawning a lot of `cmd.exe` subprocesses, each with a command line similar to `cmd.exe /c del <somefile>`.

Then I found [this comment](https://github.com/conda/conda-build/blob/2e597a1f07a7cd60c7d55ee3422a40522ee0501b/conda_build/utils.py#L1036) in `conda_build.utils` (in the `rm_rf` function):

> \# native windows delete is potentially much faster

I fully agree for whole directory trees, but is this also the case for a single file? In the current `rm_rf` function, even for a single file, a new `cmd.exe` subprocess is created. And processes on Windows are much slower than on Linux. Especially if you have virus scanners around.

For example on my laptop, I tested the following snippets:

```python
In [1]: import os

In [2]: import subprocess

In [3]: def create_file():
   ...:     with open('dummy.txt', 'w') as f:
   ...:         f.write('')
   ...:

In [4]: %%timeit
   ...: create_file()
   ...: os.unlink('dummy.txt')
   ...:
1.18 ms ± 30.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [5]: %%timeit
   ...: create_file()
   ...: subprocess.check_call('del dummy.txt', shell=True)
   ...:
626 ms ± 18.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

So deleting a single file via subprocess and not via builtins is up to several hundred times slower!

Were there other reasons for falling back to shell instead of using `os.unlink` (or similar) ?


Discussion:
Aside from speed, I've also found the subprocess call to be much more reliable than os.remove or os.unlink.  The python functions tend to leave files around from my experience.  I'd say that's the main reason I prefer it.  This is pretty bad, though.  If it's only one single file, it's probably not worth fixing it, but it sounds like you're seeing loads of files deleted this way.  I'll try to fix this.  It would help me track this down in the code if you could isolate which files (or at least some small subset of them) are being deleted this way.
This happens during/after the console output

    Fixing permissions
    Fixing permissions
    updating: pkgxy-version-py36_0.tar.bz2

I can see command lines like

    C:\WINDOWS\system32\cmd.exe /c "del C:\some\path\_h_env\info\paths.json"
    C:\WINDOWS\system32\cmd.exe /c "del C:\some\path\_h_env\info\files"
    ...

I think the common pattern here is the `_h_env`... when conda cleans up these intermediate files. For large packages however this takes ages....

Thanks, that helps.  I'll work on it very soon.

Resolving patch:
diff --git a/conda_build/build.py b/conda_build/build.py
--- a/conda_build/build.py
+++ b/conda_build/build.py
@@ -704,12 +704,6 @@ def post_process_files(m, initial_prefix_files):
     return new_files
 
 
-def remove_prefix_file(filepath, prefix):
-    if not os.path.isabs(filepath):
-        filepath = os.path.abspath(os.path.normpath(os.path.join(prefix, filepath)))
-    utils.rm_rf(filepath)
-
-
 def bundle_conda(output, metadata, env, **kw):
     log = utils.get_logger(__name__)
     log.info('Packaging %s', metadata.dist())
@@ -806,8 +800,8 @@ def bundle_conda(output, metadata, env, **kw):
             # the test belongs to the parent recipe.  Don't include it in subpackages.
             utils.rm_rf(test_dest_path)
     # here we add the info files into the prefix, so we want to re-collect the files list
-    files = set(utils.prefix_files(metadata.config.host_prefix)) - initial_files
-    files = utils.filter_files(files, prefix=metadata.config.host_prefix)
+    prefix_files = set(utils.prefix_files(metadata.config.host_prefix))
+    files = utils.filter_files(prefix_files - initial_files, prefix=metadata.config.host_prefix)
 
     with TemporaryDirectory() as tmp:
         tmp_path = os.path.join(tmp, os.path.basename(output_filename))
@@ -868,10 +862,10 @@ def order(f):
                      verbose=metadata.config.verbose, locking=metadata.config.locking,
                     timeout=metadata.config.timeout)
 
-    # remove info files from host prefix. We do not remove the actual package's files as subsequent
-    # builds may well need them. In other words, the caller manages the files in output['checksums']
-    for f in files:
-        remove_prefix_file(f, metadata.config.host_prefix)
+    # clean out host prefix so that this output's files don't interfere with other outputs
+    #   We have a backup of how things were before any output scripts ran.  That's
+    #   restored elsewhere.
+    utils.rm_rf(metadata.config.host_prefix)
 
     return final_output
 
@@ -1868,7 +1862,7 @@ def build_tree(recipe_list, config, build_only=False, post=False, notest=False,
             # add the failed one back in at the beginning - but its deps may come before it
             recipe_list.extendleft([recipe])
             for pkg, matchspec in zip(e.packages, e.matchspecs):
-                pkg_name = pkg.split(' ')[0]
+                pkg_name = pkg.split(' ')[0].split('=')[0]
                 # if we hit missing dependencies at test time, the error we get says that our
                 #    package that we just built needs to be built.  Very confusing.  Bomb out
                 #    if any of our output metadatas are in the exception list of pkgs.
@@ -1888,14 +1882,14 @@ def build_tree(recipe_list, config, build_only=False, post=False, notest=False,
 and 'x' means 'x' or one of 'x' dependencies isn't built
 for Python 3.5 and needs to be rebuilt."""
 
-                recipe_glob = glob(os.path.join(recipe_parent_dir, pkg))
+                recipe_glob = glob(os.path.join(recipe_parent_dir, pkg_name))
                 # conda-forge style.  meta.yaml lives one level deeper.
                 if not recipe_glob:
-                    recipe_glob = glob(os.path.join(recipe_parent_dir, '..', pkg))
-                feedstock_glob = glob(os.path.join(recipe_parent_dir, pkg + '-feedstock'))
+                    recipe_glob = glob(os.path.join(recipe_parent_dir, '..', pkg_name))
+                feedstock_glob = glob(os.path.join(recipe_parent_dir, pkg_name + '-feedstock'))
                 if not feedstock_glob:
                     feedstock_glob = glob(os.path.join(recipe_parent_dir, '..',
-                                                       pkg + '-feedstock'))
+                                                       pkg_name + '-feedstock'))
                 available = False
                 if recipe_glob or feedstock_glob:
                     for recipe_dir in recipe_glob + feedstock_glob:


Test patch: