Repository: conda/conda-build

Issue:
Bus error with conda-build 2.0 in Constructor installation
We are building our own Conda distribution (similar to miniconda) using [constructor](https://github.com/conda/constructor/).

When we build *Python* (3.5.2 or 3.6.0b4) with *conda-build 1.21.11*, everything works fine and as expected.  When we build it with *conda-build 2.0.x*, the Conda installer generated by Constructor fails with:

```
./emsconda.sh: Zeile 161: 23713 Bus error $PYTHON -E -s $PREFIX/pkgs/.install.py $INST_OPT
```

Context from the installer:

```bash
install_dist()
{
    # This function installs a conda package into prefix, but without linking
    # the conda packages.  It untars the package and calls a simple script
    # which does the post extract steps (update prefix files, run 'post-link',
    # and creates the conda metadata).  Note that this is all done without
    # conda.
    echo "installing: $1 ..."
    PKG=$PREFIX/pkgs/$1.tar.bz2
    tar xjf $PKG -C $PREFIX --no-same-owner || exit 1
    if [[ $1 == 'python-3.5.2-6' ]]; then
        $PYTHON -E -V
        if (( $? )); then
            echo "ERROR:
cannot execute native linux-64 binary, output from 'uname -a' is:" >&2
            uname -a
            exit 1
        fi
    fi
    $PYTHON -E -s $PREFIX/pkgs/.install.py $INST_OPT || exit 1
    rm $PKG
}

install_dist python-3.5.2-6
```

The error happens when `.install.py:update_prefix()` tries to open `lib/libpython3.5m.so.1.0` for writing (obviously, Python does not not like its lib being edited while it is running).

The reason why it works with conda-build 1.21 but not with 2.0 is that 1.21 inserts prefixes only in the executable files, but not in the `*.so` files.  2.0 also inserts prefixes in `*.so` which the installer then tries to fix (and then fails).

Output of `conda info`:
```
Current conda install:

               platform : linux-64
          conda version : 4.2.12
       conda is private : False
      conda-env version : 2.5.1
    conda-build version : 2.0.0 (or 1.21.11 in the working case)
         python version : 3.5.2.final.0
       requests version : 2.11.1
       root environment : /tmp/py3  (writable)
    default environment : /tmp/py3
       envs directories : /tmp/py3/envs
          package cache : /tmp/py3/pkgs
           channel URLs : file:///tmp/py3/conda-bld/linux-64
                          file:///tmp/py3/conda-bld/noarch
            config file : /tmp/py3/.condarc
           offline mode : False
```

I can provide the installers (a working one and a failing one) and also the recipes if necessary.



Discussion:
Thanks, but I disagree a bit with your diagnosis.  I think that the
difference between conda-build 1 and 2 that is affecting this is that
conda-build 1 was opt-in for prefix replacement, whereas conda-build 2 is
opt-out.

The docs reflect this, though they should be clearer about the different
behavior:

http://conda.pydata.org/docs/building/meta-yaml.html#binary-relocation

ignore_prefix_files in lib/libpython3.5m.so.1.0 seems like it might work?

http://conda.pydata.org/docs/building/meta-yaml.html#ignore-prefix-files

Let me know if these bits help you sort it out.  Also, conda-build 2.0.0 is
pretty old.  There are lots of bug fixes between that and the current
release (2.0.10) - please update before investigating further.

On Wed, Nov 23, 2016 at 3:22 PM, Stefan Scherfke <notifications@github.com>
wrote:

> We are building our own Conda distribution (similar to miniconda) using
> constructor <https://github.com/conda/constructor/>.
>
> When we build *Python* (3.5.2 or 3.6.0b4) with *conda-build 1.21.11*,
> everything works fine and as expected. When we build it with *conda-build
> 2.0.x*, the Conda installer generated by Constructor fails with:
>
> ./emsconda.sh: Zeile 161: 23713 Bus error $PYTHON -E -s $PREFIX/pkgs/.install.py $INST_OPT
>
> Context from the installer:
>
> install_dist()
> {
>     # This function installs a conda package into prefix, but without linking
>     # the conda packages.  It untars the package and calls a simple script
>     # which does the post extract steps (update prefix files, run 'post-link',
>     # and creates the conda metadata).  Note that this is all done without
>     # conda.
>     echo "installing: $1 ..."
>     PKG=$PREFIX/pkgs/$1.tar.bz2
>     tar xjf $PKG -C $PREFIX --no-same-owner || exit 1
>     if [[ $1 == 'python-3.5.2-6' ]]; then
>         $PYTHON -E -V
>         if (( $? )); then
>             echo "ERROR:cannot execute native linux-64 binary, output from 'uname -a' is:" >&2
>             uname -a
>             exit 1
>         fi
>     fi
>     $PYTHON -E -s $PREFIX/pkgs/.install.py $INST_OPT || exit 1
>     rm $PKG
> }
>
> install_dist python-3.5.2-6
>
> The error happens when .install.py:update_prefix() tries to open
> lib/libpython3.5m.so.1.0 for writing (obviously, Python does not not like
> its lib being edited while it is running).
>
> The reason why it works with conda-build 1.21 but not with 2.0 is that
> 1.21 inserts prefixes only in the executable files, but not in the *.so
> files. 2.0 also inserts prefixes in *.so which the installer then tries
> to fix (and then fails).
>
> Output of conda info:
>
> Current conda install:
>
>                platform : linux-64
>           conda version : 4.2.12
>        conda is private : False
>       conda-env version : 2.5.1
>     conda-build version : 2.0.0 (or 1.21.11 in the working case)
>          python version : 3.5.2.final.0
>        requests version : 2.11.1
>        root environment : /tmp/py3  (writable)
>     default environment : /tmp/py3
>        envs directories : /tmp/py3/envs
>           package cache : /tmp/py3/pkgs
>            channel URLs : file:///tmp/py3/conda-bld/linux-64
>                           file:///tmp/py3/conda-bld/noarch
>             config file : /tmp/py3/.condarc
>            offline mode : False
>
> I can provide the installers (a working one and a failing one) and also
> the recipes if necessary.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/conda/conda-build/issues/1551>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AACV-RpDH9FHm7xr4H26QMYL7enE2HgGks5rBK6rgaJpZM4K7Dzf>
> .
>

Thanks for your feedback. 

> ignore_prefix_files in lib/libpython3.5m.so.1.0 seems like it might work?

Adding `lib/libpython3.5m.so.1.0` and all `lib/python3.5/lib-dynload/*.so` to `build:ignore_prefix_files` helped indeed. It would be cool though if we could use Wildcards there so that we don't have to list every single `*.so` in `lib-dynload`.

I wonder why the [conda forge Python](https://github.com/conda-forge/python-feedstock/blob/master/recipe/meta.yaml) works without `ignore_prefix_files`?

> Also, conda-build 2.0.0 is pretty old. There are lots of bug fixes between that and the current release (2.0.10) - please update before investigating further.

We are using the lastest versions, of course. I just wanted to narrow down the issue to the last version were it worked and the earliest version were it stopped working. ;-)

Thanks for your help, again! :)
I'm guessing that conda-forge is still using conda-build 1.

I'll look into adding globs for ignore_prefix_files soon.  That's a good
idea.

On Thu, Nov 24, 2016 at 3:31 AM, Stefan Scherfke <notifications@github.com>
wrote:

> Thanks for your feedback.
>
> ignore_prefix_files in lib/libpython3.5m.so.1.0 seems like it might work?
>
> Adding lib/libpython3.5m.so.1.0 and all lib/python3.5/lib-dynload/*.so to
> build:ignore_prefix_files helped indeed. It would be cool though if we
> could use Wildcards there so that we don't have to list every single *.so
> in lib-dynload.
>
> I wonder why the conda forge Python
> <https://github.com/conda-forge/python-feedstock/blob/master/recipe/meta.yaml>
> works without ignore_prefix_files?
>
> Also, conda-build 2.0.0 is pretty old. There are lots of bug fixes between
> that and the current release (2.0.10) - please update before investigating
> further.
>
> We are using the lastest versions, of course. I just wanted to narrow down
> the issue to the last version were it worked and the earliest version were
> it stopped working. ;-)
>
> Thanks for your help, again! :)
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/conda/conda-build/issues/1551#issuecomment-262729907>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AACV-f6etPH9pF3s_SIu1MAH_BEopBjYks5rBVlegaJpZM4K7Dzf>
> .
>


Resolving patch:
diff --git a/conda_build/metadata.py b/conda_build/metadata.py
--- a/conda_build/metadata.py
+++ b/conda_build/metadata.py
@@ -1,5 +1,6 @@
 from __future__ import absolute_import, division, print_function
 
+import glob
 import logging
 import os
 import re
@@ -175,6 +176,14 @@ def parse(data, config, path=None):
     return sanitize(res)
 
 
+def expand_globs(path_list, root_dir):
+    files = []
+    for path in path_list:
+        files.extend(glob.glob(os.path.join(root_dir, path)))
+    # list comp is getting rid of absolute prefix, to match relative paths used in file list
+    return [f.replace(root_dir + os.path.sep, '') for f in files]
+
+
 trues = {'y', 'on', 'true', 'yes'}
 falses = {'n', 'no', 'false', 'off'}
 
@@ -685,17 +694,18 @@ def has_prefix_files(self):
             if any('\\' in i for i in ret):
                 raise RuntimeError("build/has_prefix_files paths must use / "
                                    "as the path delimiter on Windows")
-        return ret
+        return expand_globs(ret, self.config.build_prefix)
 
     def ignore_prefix_files(self):
         ret = self.get_value('build/ignore_prefix_files', False)
         if type(ret) not in (list, bool):
-            raise RuntimeError('build/ignore_prefix_files should be boolean or a list of paths')
+            raise RuntimeError('build/ignore_prefix_files should be boolean or a list of paths '
+                               '(optionally globs)')
         if sys.platform == 'win32':
             if type(ret) is list and any('\\' in i for i in ret):
                 raise RuntimeError("build/ignore_prefix_files paths must use / "
                                    "as the path delimiter on Windows")
-        return ret
+        return expand_globs(ret, self.config.build_prefix) if type(ret) is list else ret
 
     def always_include_files(self):
         files = ensure_list(self.get_value('build/always_include_files', []))
@@ -704,7 +714,8 @@ def always_include_files(self):
                                 "as the path delimiter on Windows")
         if on_win:
             files = [f.replace("/", "\\") for f in files]
-        return files
+
+        return expand_globs(files, self.config.build_prefix)
 
     def include_recipe(self):
         return self.get_value('build/include_recipe', True)
@@ -717,7 +728,7 @@ def binary_has_prefix_files(self):
             if any('\\' in i for i in ret):
                 raise RuntimeError("build/binary_has_prefix_files paths must use / "
                                    "as the path delimiter on Windows")
-        return ret
+        return expand_globs(ret, self.config.build_prefix)
 
     def skip(self):
         return self.get_value('build/skip', False)


Test patch:
diff --git a/tests/test_metadata.py b/tests/test_metadata.py
--- a/tests/test_metadata.py
+++ b/tests/test_metadata.py
@@ -3,7 +3,7 @@
 
 from conda_build.conda_interface import MatchSpec
 
-from conda_build.metadata import select_lines, handle_config_version
+from conda_build.metadata import select_lines, handle_config_version, expand_globs
 from .utils import testing_workdir, test_config, test_metadata
 
 
@@ -107,3 +107,12 @@ def test_numpy(self):
         self.assertRaises(RuntimeError,
                           handle_config_version,
                           MatchSpec('numpy x.x'), None)
+
+
+def test_expand_globs(testing_workdir):
+    files = ['abc', 'acb']
+    for f in files:
+        with open(f, 'w') as _f:
+            _f.write('weee')
+    assert expand_globs(files, testing_workdir) == files
+    assert expand_globs(['a*'], testing_workdir) == files