Repository: spack/spack

Issue:
Installation error: Python can't find _iconv symbol
I have been trying to build packages with spack on mac os Mojave for awhile now and I have been struggling.  I've simplified my software stack as much as possible to reproduce one of the issues I keep seeing.

I created an environment (called `nexus`, here) with the following commands:
```
$ spack env create nexus nexus.yaml
```
Where nexus.yaml is:
```
spack:
  specs:
    - python
```

From here, I activate, concretize and install:
```
$ spack env activate nexus
$ spack concretize
==> Warning: clang@11.0.0-apple cannot build optimized binaries for "skylake". Using best target possible: "x86_64"
==> Warning: clang@11.0.0-apple cannot build optimized binaries for "skylake". Using best target possible: "x86_64"
==> Concretized python
 -   hpjs6nz  python@3.7.4%clang@11.0.0-apple+bz2+ctypes+dbm+lzma~nis~optimizations patches=210df3f28cde02a8135b58cc4168e70ab91dbf9097359d05938f1e2843875e57 +pic+pyexpat+pythoncmd+readline~shared+sqlite3+ssl~tix~tkinter~ucs4~uuid+zlib arch=darwin-mojave-x86_64
 -   ysmjykg      ^bzip2@1.0.8%clang@11.0.0-apple+shared arch=darwin-mojave-x86_64
 -   yn2yzi3          ^diffutils@3.7%clang@11.0.0-apple arch=darwin-mojave-x86_64
 -   jg6ekuh              ^libiconv@1.16%clang@11.0.0-apple arch=darwin-mojave-x86_64
 -   l4gxq46      ^expat@2.2.9%clang@11.0.0-apple~libbsd arch=darwin-mojave-x86_64
 -   3t4dwsv      ^gdbm@1.18.1%clang@11.0.0-apple arch=darwin-mojave-x86_64
 -   h2bjtz2          ^readline@8.0%clang@11.0.0-apple arch=darwin-mojave-x86_64
 -   zbgxvxi              ^ncurses@6.1%clang@11.0.0-apple~symlinks~termlib arch=darwin-mojave-x86_64
 -   f3libie                  ^pkgconf@1.6.3%clang@11.0.0-apple arch=darwin-mojave-x86_64
 -   msyeqht      ^gettext@0.20.1%clang@11.0.0-apple+bzip2+curses+git~libunistring+libxml2+tar+xz arch=darwin-mojave-x86_64
 -   w354vrb          ^libxml2@2.9.9%clang@11.0.0-apple~python arch=darwin-mojave-x86_64
 -   b7n722h              ^xz@5.2.4%clang@11.0.0-apple arch=darwin-mojave-x86_64
 -   xj36ejk              ^zlib@1.2.11%clang@11.0.0-apple+optimize+pic+shared arch=darwin-mojave-x86_64
 -   2e6sifw          ^tar@1.32%clang@11.0.0-apple arch=darwin-mojave-x86_64
 -   7yg7qah      ^libffi@3.2.1%clang@11.0.0-apple arch=darwin-mojave-x86_64
 -   ugvknwc      ^openssl@1.1.1d%clang@11.0.0-apple+systemcerts arch=darwin-mojave-x86_64
 -   l6fc2zw          ^perl@5.30.0%clang@11.0.0-apple+cpanm+shared+threads arch=darwin-mojave-x86_64
 -   hsmq6ou      ^sqlite@3.30.1%clang@11.0.0-apple~column_metadata+fts~functions~rtree arch=darwin-mojave-x86_64

==> Updating view at /Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view
```

The install process reported no errors whatsoever.  Ran for awhile and then afterwards, `python` executable is active and available, but spack itself is broken:

```Traceback (most recent call last):
  File "/Users/corey.adams/spack/spack/bin/spack", line 63, in <module>
    import spack.main  # noqa
  File "/Users/corey.adams/spack/spack/lib/spack/spack/main.py", line 29, in <module>
    import spack.architecture
  File "/Users/corey.adams/spack/spack/lib/spack/spack/architecture.py", line 69, in <module>
    import spack.compiler
  File "/Users/corey.adams/spack/spack/lib/spack/spack/compiler.py", line 19, in <module>
    import spack.spec
  File "/Users/corey.adams/spack/spack/lib/spack/spack/spec.py", line 105, in <module>
    import spack.repo
  File "/Users/corey.adams/spack/spack/lib/spack/spack/repo.py", line 36, in <module>
    import spack.caches
  File "/Users/corey.adams/spack/spack/lib/spack/spack/caches.py", line 15, in <module>
    import spack.fetch_strategy
  File "/Users/corey.adams/spack/spack/lib/spack/spack/fetch_strategy.py", line 42, in <module>
    import spack.util.web as web_util
  File "/Users/corey.adams/spack/spack/lib/spack/spack/util/web.py", line 18, in <module>
    from six.moves.urllib.request import urlopen, Request
  File "/Users/corey.adams/spack/spack/lib/spack/external/six.py", line 92, in __get__
    result = self._resolve()
  File "/Users/corey.adams/spack/spack/lib/spack/external/six.py", line 160, in _resolve
    module = _import_module(self.mod)
  File "/Users/corey.adams/spack/spack/lib/spack/external/six.py", line 82, in _import_module
    __import__(name)
  File "/Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/python3.7/urllib/request.py", line 2583, in <module>
    from _scproxy import _get_proxy_settings, _get_proxies
ImportError: dlopen(/Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/python3.7/lib-dynload/_scproxy.cpython-37m-darwin.so, 2): Symbol not found: _iconv
  Referenced from: /usr/lib/libcups.2.dylib
  Expected in: /Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/libiconv.2.dylib
 in /usr/lib/libcups.2.dylib
```

I am using clang11:
```
$ spack config get compilers
compilers:
- compiler:
    spec: clang@11.0.0-apple
    paths:
      cc: /usr/bin/clang
      cxx: /usr/bin/clang++
      f77:
      fc:
    flags: {}
    operating_system: mojave
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
```


I note a few things here:
The library that causes the issue doesn't directly link against `libcups`:
```
$ otool -L /Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/python3.7/lib-dynload/_scproxy.cpython-37m-darwin.so
/Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/python3.7/lib-dynload/_scproxy.cpython-37m-darwin.so:
	/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 1061.40.2)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1673.126.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)
```

I am not enough of a linking expert to track down where the dependency is coming from.  I also note that the library it finds for libcups that spack has build does *not* link to a library that has `_iconv` symbols defined:
```
corey.adams@cadams2 : ~/NEXT
$ otool -L /Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/libiconv.2.dylib
/Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/libiconv.2.dylib:
	/Users/corey.adams/spack/spack/opt/spack/darwin-mojave-x86_64/clang-11.0.0-apple/libiconv-1.16-jg6ekuhbklculym7pyjpx6l5jetym7wp/lib/libiconv.2.dylib (compatibility version 9.0.0, current version 9.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)

corey.adams@cadams2 : ~/NEXT
$ nm /Users/corey.adams/spack/spack/opt/spack/darwin-mojave-x86_64/clang-11.0.0-apple/libiconv-1.16-jg6ekuhbklculym7pyjpx6l5jetym7wp/lib/libiconv.2.dylib | grep _iconv
0000000000003360 T _iconv_canonicalize
```

However, the library available at system level does:
```
$ otool -L /usr/lib/libcups.2.dylib
/usr/lib/libcups.2.dylib:
	/usr/lib/libcups.2.dylib (compatibility version 2.0.0, current version 2.13.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
	/System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos (compatibility version 5.0.0, current version 6.0.0)
	/System/Library/Frameworks/GSS.framework/Versions/A/GSS (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1575.17.0)
	/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 963.255.3)
	/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 58286.255.3)
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)

corey.adams@cadams2 : ~/NEXT
$ nm /usr/lib/libiconv.2.dylib | grep _iconv
0000000000002360 T _iconv
00000000000f0da0 s _iconv_2VersionNumber
00000000000f0d70 s _iconv_2VersionString
000000000000267a T _iconv_canonicalize
0000000000002382 T _iconv_close
0000000000001049 T _iconv_open
000000000000238f T _iconvctl
0000000000002488 T _iconvlist
```

I don't have anything special on LD_LIBRARY_PATH - it is unset before spack env activate.

Somehow, spack seems to be building a package that links against the system version of libcups (which in turn links to system iconv) while at run time it is picking up it's own libcups (and it's own libiconv) which is missing symbols that it needs.

This is macOS mojave.

Thoughts?


Discussion:
That's odd, I'm also using Spack Environments on macOS (Catalina) and using my Spack-installed Python to run Spack. I haven't encountered any issues like this.

@gartung is our macOS linking expert, maybe he has some ideas on how to debug this.
@gartung any thoughts? spack on mac has been killing me, I'd really love to get this working.


@chissg Have you run into this in the ups environment?
I googled the partial error messages and got this
https://github.com/Roche/pyreadstat/issues/2
which suggested seeing what libraries python is loading with

DYLD_PRINT_LIBRARIES=YES python -c "from _scproxy import _get_proxy_settings, _get_proxies"
This is another one
https://github.com/openai/universe/issues/139
It suggests that you need to check the scons build to see where it picked up libintl from.
> It suggests that you need to check the scons build to see where it picked up libintl from.

This surprises me, as Spack doesn't depend on SCons, so I would be surprised to find that is causes your error. If you remove SCons from your environment, can you no longer reproduce the error?
The very first thing spack installs in that environment is libiconv

```
mac-133717:spack gartung$ spack install
==> Installing environment nexus
==> Installing libiconv
==> Searching for binary cache of libiconv
==> No binary for libiconv found: installing from source
==> Fetching https://ftp.gnu.org/gnu/libiconv/libiconv-1.16.tar.gz
######################################################################## 100.0%
==> Staging archive: /var/folders/sb/_59sd1t97k149f3mrk_hqhbr0000gp/T/gartung/spack-stage/spack-stage-libiconv-1.16-jg6ekuhbklculym7pyjpx6l5jetym7wp/libiconv-1.16.tar.gz
==> Created stage in /var/folders/sb/_59sd1t97k149f3mrk_hqhbr0000gp/T/gartung/spack-stage/spack-stage-libiconv-1.16-jg6ekuhbklculym7pyjpx6l5jetym7wp
==> No patches needed for libiconv
==> Building libiconv [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Executing phase: 'build'
```
Because diffutils depends on it

```
 -   hpjs6nz  python@3.7.4%clang@11.0.0-apple+bz2+ctypes+dbm+lzma~nis~optimizations patches=210df3f28cde02a8135b58cc4168e70ab91dbf9097359d05938f1e2843875e57 +pic+pyexpat+pythoncmd+readline~shared+sqlite3+ssl~tix~tkinter~ucs4~uuid+zlib arch=darwin-mojave-x86_64
 -   ysmjykg      ^bzip2@1.0.8%clang@11.0.0-apple+shared arch=darwin-mojave-x86_64
 -   yn2yzi3          ^diffutils@3.7%clang@11.0.0-apple arch=darwin-mojave-x86_64
 -   jg6ekuh              ^libiconv@1.16%clang@11.0.0-apple arch=darwin-mojave-x86_64
```
There is no way to link diffutils without libiconv. To keep libiconv from being linked into the environment you will need to have a packages.yaml that declares diffutils or bzip2 as external picked up from the system. 
There are a number of libraries that should be picked up from the system on macOS to avoid situations like these.
The error is happening when the six package tries to load urllib in this environment.
I tried; It didn't work.

I couldn't directly remove scons from the environment, because spack itself will not run with the environment active.  I removed the environment, uninstalled scons, and then recreated the environment with only python.  It reinstalled python + it's dependencies, but I get the same error.

Awesome, so the issue has nothing to do with SCons, that simplifies things. I'm still surprised you're encountering this issue and I'm not.
I am concretizing and building with the following packages.yaml to avoid building libiconv

```
packages:
  perl:
    buildable: False
    paths:
      perl: /usr
  tar:
    buildable: False
    paths:
      tar: /usr
  libiconv:
    buildable: False
    paths:
      libiconv: /usr
```
Ok, so it sounds like there is a potential solution which is to make `diffutils` or `bzip2` into external, non-buildable packages?

Here is the dump for python's libraries: https://gist.github.com/coreyjadams/7a5b642d1a1115d729b19d01197863f6

I will try using the system packages as you have above to see if that works.

Using the suggested packages.yaml I can at least run spack in the activated environment. However if I run 
${prefix}/bin/python3 -c "from _scproxy import _get_proxy_settings, _get_proxies" 
I get
```
dyld: unloaded: /Users/gartung/test/spack/opt/spack/darwin-mojave-x86_64/clang-11.0.0-apple/python-3.7.4-lzdvkfn3it4xzowyuvrzft2aj3zfpoqi/lib/python3.7/lib-dynload/_scproxy.cpython-37m-darwin.so
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: dlopen(/Users/gartung/test/spack/opt/spack/darwin-mojave-x86_64/clang-11.0.0-apple/python-3.7.4-lzdvkfn3it4xzowyuvrzft2aj3zfpoqi/lib/python3.7/lib-dynload/_scproxy.cpython-37m-darwin.so, 2): Symbol not found: _sqlite3_intarray_bind
  Referenced from: /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
  Expected in: /Users/gartung/test/spack/var/spack/environments/nexus/.spack-env/view/lib/libsqlite3.dylib
 in /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
```

If I prepend /usr/lib to DYLD_LIBRARY_PATH in the environment with
```
export DYLD_LIBRARY_PATH=/usr/lib:$DYLD_LIBRARY_PATH
```

 then the same command runs without error. 

If I deactivate the environment I can also run the command without error.

The apparent function of lib/python3.7/lib-dynload/_scproxy.cpython-37m-darwin.so is to dlopen all of the C libraries referenced in the default python packages checking for missing symbols. 
The suggestion found following these two link is to use DYLD_FALLBACK_LIBRARY_PATH instead of setting DYLD_LIBRARY_PATH in the environment

https://superuser.com/questions/1446292/anaconda-stopped-working-when-added-dyld-library-path-to-bashrc
https://stackoverflow.com/questions/3146274/is-it-ok-to-use-dyld-library-path-on-mac-os-x-and-whats-the-dynamic-library-s
@adamjstewart Is there a way to chose which variable is set in the environment?
I used the suggested packages from the system and got the same error as before, in that python itself worked but I could not run spack.

Prepending `/usr/lib/` to `DYLD_LIBRARY_PATH` did allow me to run spack, and python still runs, but I have not yet tried anything else on top of this.
@hartzell is gonna love this, this is just like #3955. Might also be related to #13245.

@goxberry reported the exact same problem and opened a PR to fix the issue, see #9221

The thing that still confuses me about this issue, and makes it different from #3955 and #9221, is that those issues involve `(DY)LD_LIBRARY_PATH` causing _system_ software to break, whereas your issue is with _Spack-installed_ software breaking. The RPATHs seem fine, so I'm not sure why you're having these issues. I don't activate my environment or use modules, so that explains why I didn't encounter the same issue with `(DY)LD_LIBRARY_PATH` though.
For what it's worth, as far as I can tell all spack-installed software works.  At least, I haven't yet seen it breaking even with these issues.  It's spack itself that is failing once I installed these packages.


But Spack is just a Python library, it doesn't link to anything. It seems like it's the `python` executable that is broken, not Spack...
@adamjstewart do you run with SIP enabled? This prevents setting DYLD_LIBRARY_PATH to anything other than the system paths. I run with SIP disabled because our build scripts are nested and expect the environment to be set by the parent script.

Perhaps DYLD_LIBRARY_PATH should not be set at all by 'spack env activate'

I am having flashbacks to building ROOT with spack and using Xquartz for the X11 libraries. Some of the X11 libraries depend on system library that ROOT depends on but I did not declare them as external packages. ROOT happily built against the spack-built libraries but when running interactively it used DYLD_LIBRARY_PATH and would crash with missing symbol errors expected from system libraries. 
Yes, I run with SIP enabled.
Ah, I run with SIP disabled.  Perhaps that's the difference in your system's success and mine's failure.
Sorry for the bump, but ran into this myself today. Just for info, I have spack installed under my home area (so theoretically everything it installs is outside SIP) on a Catalina system with SIP enabled. Simplest case to reproduce is pretty much as reported:

```
$ spack env create foo
$ spack env activate -p foo
[foo] $ spack install cmake
...
[foo] $ cmake --version
cmake version 3.15.4

CMake suite maintained and supported by Kitware (kitware.com/cmake).
[foo] $ spack install python
...
[foo] $ cmake --version
dyld: Symbol not found: _iconv
  Referenced from: /usr/lib/libarchive.2.dylib
  Expected in: /Users/<ME>/Software/spack/var/spack/environments/foo/.spack-env/view/lib/libiconv.2.dylib
 in /usr/lib/libarchive.2.dylib
zsh: abort      cmake --version
[foo] $ DYLD_LIBRARY_PATH="" cmake --version
cmake version 3.15.4

CMake suite maintained and supported by Kitware (kitware.com/cmake).
[foo] $
```

@gartung's Python test `python3 -c "from _scproxy import _get_proxy_settings, _get_proxies` also fails in the main environment, but works on unsetting `DYLD_LIBRARY_PATH`.

So the remedy for me seems, at this stage, to be unsetting `DYLD_LIBRARY_PATH` in the environment. I'll see if I can trace the behaviour of SIP vs `DYLD_LIBRARY_PATH` a bit better though.
Yes, environments are basically unusable on mac (without the `-V` flag) because the DYLD_LIBRARY_PATH setting breaks every non-spack app.
Hi all, sorry to append to this issue. But I am finding identical error messages:
```console
    from _scproxy import _get_proxy_settings, _get_proxies
ImportError: dlopen(/Users/LDianaAmorim/Documents/opt/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/python-3.7.6-pqvyilqugzurz6l7qxcmwoqeqo42ckgu/lib/python3.7/lib-dynload/_scproxy.cpython-37m-darwin.so, 2): Symbol not found: _iconv
  Referenced from: /usr/lib/libarchive.2.dylib
  Expected in: /Users/LDianaAmorim/Documents/opt/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/libiconv-1.16-gj3jnj46xug7rpjxwaepe6uwx35jdphz/lib/libiconv.2.dylib
 in /usr/lib/libarchive.2.dylib
```

when I try to run `python setup.py install` on https://github.com/yt-project/yt git repo.

I tried using and not using spack environments.
I tried updating Xcode and command lines, and then re-installing spack.

But none of the fixes you mentioned in the previous messages (setting the DLYB paths), nor the one described here https://github.com/qgis/QGIS-Mac-Packager/issues/28, helped.

I was wondering if you had any other suggestions of what I could try.
Or if you think that my issue in particular has to do with `yt` and not `spack` installation of python.
Thanks
Hi all and sorry for my late reply. I hadn't understood your suggestions here fully before.
Currently, with macOS Catalina 10.15.4 and spack installed python I still get the above mentioned issue.
But there is a simple work-around, as now I can understand you had explained above, that is the replacement:

```bash
$ cp  /usr/lib/libiconv.2.dylib spack/opt/spack/darwin-catalina-x86_64/clang-11.0.3-apple/libiconv-1.16-s3pw3k55ma2ikm3alvpb6ocaqkvzdper/lib/
```
Cheers,
Diana
Is it possible these `python` and `gettext` issues are related? #16075 The former depends on the latter and both have problems with `libiconv` on macOS.

Looks to me like the spack `gettext` packages forgets to `depends_on('iconv')`. It does expose a `--with-libiconv-prefix` and I guess it picks up the system lib along some paths and specifying the spack lib along other packages, crashing things here and there.

I just checked this on ubuntu and even there, libiconv is picked up directly since its provided luckily by a transitive `gettext` dependency. Will send a PR in a few minutes.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/gettext/package.py b/var/spack/repos/builtin/packages/gettext/package.py
--- a/var/spack/repos/builtin/packages/gettext/package.py
+++ b/var/spack/repos/builtin/packages/gettext/package.py
@@ -27,6 +27,7 @@ class Gettext(AutotoolsPackage, GNUMirrorPackage):
     # Optional variants
     variant('libunistring', default=False, description='Use libunistring')
 
+    depends_on('iconv')
     # Recommended dependencies
     depends_on('ncurses',  when='+curses')
     depends_on('libxml2',  when='+libxml2')
@@ -50,7 +51,8 @@ def configure_args(self):
 
         config_args = [
             '--disable-java',
-            '--disable-csharp',
+            '--disable-icsharp',
+            '--with-libiconv-prefix={0}'.format(spec['iconv'].prefix),
             '--with-included-glib',
             '--with-included-gettext',
             '--with-included-libcroco',


Test patch: