Repository: spack/spack

Issue:
spack install curl+libssh2 fails for Mac OS 10.13.4

Summary

The installation for curl for spack fails for Mac OS 10.13.4 (at least).  The default preferred version is 7.56.0. This package installed correctly when macports is used with default version 7.59.

### Expected Result
The package would be installed with no errors.

### Actual Result

The installation fails with the message

configure: error: one or more libs available at link-time are not available run-time. Libs used at link-time: -lssh2 -lssh2  -lssl -lcrypto -lssl -lcrypto  -lldap -lz

[LOG.txt](https://github.com/spack/spack/files/1917450/LOG.txt)

[spack-build.txt](https://github.com/spack/spack/files/1917459/spack-build.txt)

The above is really spack-build.out. I changed out to txt to make your interface happy.

### Steps to reproduce the issue

run
spack install curl
from a terminal on a Macintosh system running OS 10.13.4/

### Information on your system

This includes:

 MacPro Late 2013 OS 10.13.4 3.5 GHz 6-Core Xeon E5
This is a standard configuration. The compile set is clang@9.1.0-apple



Discussion:
I can reproduce this issue and it seems that `configure` picks up `libssh2` correctly
```
...
configure: Added /Users/healther/Documents/git.nosync/spack/opt/spack/darwin-highsierra-x86_64/clang-9.0.0-apple/libssh2-1.8.0-scq7flagnxeosn6w5h7ndsvifegsaog2/lib to LD_LIBRARY_PATH
...
```
For now I haven't been able to find a solution, do you have an idea whether this worked at some point in the past or not? I'm uncertain whether I ever installed curl via spack...

edit: should have explicitly pointed out: The error goes away if you build with `~libssh2`, so it is the picking up of `libssh2` that is the problem
Does curl without libssh2 (`spack install curl~libssh2`) work?
Yes
```
spack install curl~libssh2
```
works without problem. I also tried to use `libssh` (not yet in `spack`, although I'll have a PR for that tomorrow or so), which since `1.58` is an alternative to `libssh2`, but it also didn't work. Same error.
HI Heather: Thanks for the response. I have been using curl built by macports for many years, so I suspect that I went through this version at some point. The current version of curl on macports is 7.59, so it is possible this newer version doesn’t have this problem. I actually tried to get spack to use this version, but it didn’t like it claiming there was no trusted version to download. Macports gets its version from https://curl.haxx.se, and spack wants to use this same site. I actually ran into this problem trying to install lmod. I gave up on this for now but the spack environment setup seems to do what I need, namely switching between compiler versions %gcc@7.3.0<mailto:%25gcc@7.3.0> and %intel@18.0.2<mailto:%25intel@18.0.2>. The next thing I would try is to download curl 7.56 and see what happens with a manual build. This may be something as simple as turning off testing for the configure. I noted that this problem occurred both for the gcc and intel compilers.

I saw that Junghans’ note and I can try to build curl this way without libssh2, but what I would really like to do is to configure spack to use /opt/local for any packages I already have via macports. I would probably build the intel versions separately via spack, but the gcc versions should work with what macports built. Actually the libssh2 question makes me wonder why this library is needed in the first place. It is a system library  and part of the OS. The OS in fact provides OpenSSH_7.6p1, LibreSSL 2.6.2, so I would assume that the spack build could use this. I am new to using spack and am only starting the figure things out. It looks like some amount of package customization may be needed to get things working properly in my environment.

While I was typing this message I ran “spack install curl~libssh2” and that built with no complaints. I assume that whatever this package wanted from libssh2 it either disabled or used an existing system library.


John W. Grove
CCS-2 MS D413, PO Box 1663
Los Alamos National Laboratory
Los Alamos, NM 87545
(505) 667-0723
jgrove@lanl.gov
From: healther <notifications@github.com>
Reply-To: spack/spack <reply@reply.github.com>
Date: Tuesday, April 17, 2018 at 10:02 AM
To: spack/spack <spack@noreply.github.com>
Cc: "John W. Grove" <jgrove@lanl.gov>, Author <author@noreply.github.com>
Subject: Re: [spack/spack] spack install curl fails for Mac OS 10.13.4 (#7777)


I can reproduce this issue and it seems that configure picks up libssh2 correctly

...

configure: Added /Users/healther/Documents/git.nosync/spack/opt/spack/darwin-highsierra-x86_64/clang-9.0.0-apple/libssh2-1.8.0-scq7flagnxeosn6w5h7ndsvifegsaog2/lib to LD_LIBRARY_PATH

...

For now I haven't been able to find a solution, do you have an idea whether this worked at some point in the past or not? I'm uncertain whether I ever installed curl via spack...

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/spack/spack/issues/7777#issuecomment-382047947>, or mute the thread<https://github.com/notifications/unsubscribe-auth/Af6iu2dgIZUCzjd4W7MZVNK08DqoLU0oks5tphIjgaJpZM4TXVQM>.

AFAIK there is no support for spack to automatically pick up all installed packages by another package manager. But if you are ready to do it manually you can add these packages as external and `buildable: False` to your `packages.yaml` (see [here](http://spack.readthedocs.io/en/latest/build_settings.html#external-packages) for reference.

Just FYI: you can do `spack -k install curl@7.59.0` and it would download even the non-trusted version (-k) this is of course not recommended. Or you could add the version via `spack edit curl` and adding the new version. Neither of those solutions worked for me though
> I have been using curl built by macports for many years, so I suspect that I went through this version at some point.

For what it's worth, the reason that the Macports build works and Spack doesn't is that Macports explicitly disables `libssh2` support:
https://github.com/macports/macports-ports/blob/master/net/curl/Portfile#L52

Homebrew's package has some notes about curl + openssl being fussy:
https://github.com/Homebrew/homebrew-core/blob/master/Formula/curl.rb#L62,L64

I'll see what I can do.
Thanks healther:  This is just what I am looking at now. I wasn’t expecting this to be automatic.  I am trying to configure the syntax for packages.yaml and I have to say the examples are somewhat confusing. The specification for paths in the example using something like a: path. In my case I want to substitute the blas dependency on openblas by one of two things the macports built version of openblas or the intel version of the math kernel library. Right now I am trying to install hypre which is trying to build openblas =-0.2.20 using %intel@18.0.2<mailto:%25intel@18.0.2>. This fails with the message

icc -O2 -DMAX_STACK_ALLOC=2048 -wd981 -DF_INTERFACE_INTEL -fPIC -DNO_WARMUP -DMAX_CPU_NUMBER=12 -DASMNAME=_ -DASMFNAME=__ -DNAME=_ -DCNAME= -DCHAR_NAME=\"_\" -DCHAR_CNAME=\"\" -DNO_AFFINITY -I.. -all_load -headerpad_max_install_names -install_name "/private/var/folders/jl/9jpdtxpj67v2z2kn9x6bh0d8000c82/T/jgrove/spack-stage/spack-stage-Zbc1wG/OpenBLAS-0.2.20/exports/../libopenblas_sandybridge-r0.2.20.dylib" -dynamiclib -o ../libopenblas_sandybridge-r0.2.20.dylib ../libopenblas_sandybridge-r0.2.20.a -Wl,-exported_symbols_list,osx.def  -L/opt/intel/compilers_and_libraries_2018.2.164/mac/compiler/lib -L/usr/lib  -lSystem -lpthread  /opt/intel/compilers_and_libraries_2018.2.164/mac/compiler/lib/libifport.a /opt/intel/compilers_and_libraries_2018.2.164/mac/compiler/lib/libifcoremt.a /opt/intel/compilers_and_libraries_2018.2.164/mac/compiler/lib/libimf.a /opt/intel/compilers_and_libraries_2018.2.164/mac/compiler/lib/libsvml.a /opt/intel/compilers_and_libraries_2018.2.164/mac/compiler/lib/libipgo.a /opt/intel/compilers_and_libraries_2018.2.164/mac/compiler/lib/libirc.a /opt/intel/compilers_and_libraries_2018.2.164/mac/compiler/lib/libsvml.a /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/9.1.0/lib/darwin/libclang_rt.osx.a -lifcore
Undefined symbols for architecture x86_64:
  "_MAIN__", referenced from:
      _main in libifcoremt.a(for_main.o)
  "__FTN_ALLOC", referenced from:
      _do_ftn_alloc in libifcoremt.a(for_ftnalloc.o)
ld: symbol(s) not found for architecture x86_64
make[1]: *** [Makefile:113: libopenblas_sandybridge-r0.2.20.dylib] Error 1

I am not sure where this package got the name sandybridge, which I believe is an error. I think it should be using darwin-highsierra-x86_64. The undefined reference is probably due to missing libraries. Generally to compile fortran with icc you need to load not just libimf and  ifcore but ifport too. I think that the intel blas are located in libmkl_core.dylib or libmkl_core.a. So I want to configure package.yaml to use this path. If there a more detailed documentation for the format for this file?


John W. Grove
CCS-2 MS D413, PO Box 1663
Los Alamos National Laboratory
Los Alamos, NM 87545
(505) 667-0723
jgrove@lanl.gov
From: healther <notifications@github.com>
Reply-To: spack/spack <reply@reply.github.com>
Date: Tuesday, April 17, 2018 at 12:33 PM
To: spack/spack <spack@noreply.github.com>
Cc: "John W. Grove" <jgrove@lanl.gov>, Author <author@noreply.github.com>
Subject: Re: [spack/spack] spack install curl fails for Mac OS 10.13.4 (#7777)


AFAIK there is no support for spack to automatically pick up all installed packages by another package manager. But if you are ready to do it manually you can add these packages as external and buildable: False to your packages.yaml (see here<http://spack.readthedocs.io/en/latest/build_settings.html#external-packages> for reference.

Just FYI: you can do spack -k install curl@7.59.0 and it would download even the non-trusted version (-k) this is of course not recommended. Or you could add the version via spack edit curl and adding the new version. Neither of those solutions worked for me though

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/spack/spack/issues/7777#issuecomment-382096497>, or mute the thread<https://github.com/notifications/unsubscribe-auth/Af6iu50SfN66J3iZClydg7JDbE6LsqyZks5tpjVegaJpZM4TXVQM>.

That is nice to know. Spack builds this too if you disable ssh2.

John W. Grove
CCS-2 MS D413, PO Box 1663
Los Alamos National Laboratory
Los Alamos, NM 87545
(505) 667-0723
jgrove@lanl.gov
From: "Adam J. Stewart" <notifications@github.com>
Reply-To: spack/spack <reply@reply.github.com>
Date: Tuesday, April 17, 2018 at 12:55 PM
To: spack/spack <spack@noreply.github.com>
Cc: "John W. Grove" <jgrove@lanl.gov>, Author <author@noreply.github.com>
Subject: Re: [spack/spack] spack install curl fails for Mac OS 10.13.4 (#7777)


I have been using curl built by macports for many years, so I suspect that I went through this version at some point.

For what it's worth, the reason that the Macports build works and Spack doesn't is that Macports explicitly disables libssh2 support:
https://github.com/macports/macports-ports/blob/master/net/curl/Portfile#L52

Homebrew's package has some notes about curl + openssl being fussy:
https://github.com/Homebrew/homebrew-core/blob/master/Formula/curl.rb#L62,L64

I'll see what I can do.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/spack/spack/issues/7777#issuecomment-382103445>, or mute the thread<https://github.com/notifications/unsubscribe-auth/Af6iu4rrZHBAKXzmact3CrQ4UWP53PYOks5tpjqggaJpZM4TXVQM>.

#7800 disables the `libssh2` variant by default. Also added the latest version. If anyone can figure out how to get `curl+libssh2` to build on macOS, let me know.
Just for reference: 
Homebrew (works):
```
./configure --disable-debug --disable-dependency-tracking --disable-silent-rules --prefix=/Users/healther/Documents/git.nosync/spack/opt/spack/darwin-highsierra-x86_64/clang-9.0.0-apple/curl-7.59.0-67jge3wsgrsa4bkp3qeq7lo3vsdwsvtt --with-darwinssl --without-ca-bundle --without-ca-path --with-libssh2 --without-libmetalink --without-gssapi --without-librtmp --disable-ares
```
spack (does not work)
```
'/Users/healther/Documents/git.nosync/spack/var/spack/stage/curl-7.59.0-67jge3wsgrsa4bkp3qeq7lo3vsdwsvtt/curl-7.59.0/configure' '--prefix=/Users/healther/Documents/git.nosync/spack/opt/spack/darwin-highsierra-x86_64/clang-9.0.0-apple/curl-7.59.0-67jge3wsgrsa4bkp3qeq7lo3vsdwsvtt' '--with-zlib=/Users/healther/Documents/git.nosync/spack/opt/spack/darwin-highsierra-x86_64/clang-9.0.0-apple/zlib-1.2.11-t74siw3tvuz3vfb3rje5mkbiwneuzfb5' '--with-ssl=/Users/healther/Documents/git.nosync/spack/opt/spack/darwin-highsierra-x86_64/clang-9.0.0-apple/openssl-1.0.2n-qi2qegltmpe2qvs6eumbinrxinu4avh5' '--without-nghttp2' '--with-libssh2' '--without-libssh'
```

BUT after installing `curl --with-libssh2` via homebrew spack's install also goes through without complaint. Only after removing `libssh2` from homebrew will spack's configure fail again. So my working hypothesis right now is that homebrew works because they install stuff in semi-official and definitely-global directories
The underlying problem isn't resolved

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/curl/package.py b/var/spack/repos/builtin/packages/curl/package.py
--- a/var/spack/repos/builtin/packages/curl/package.py
+++ b/var/spack/repos/builtin/packages/curl/package.py
@@ -23,6 +23,7 @@
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 ##############################################################################
 from spack import *
+import sys
 
 
 class Curl(AutotoolsPackage):
@@ -49,21 +50,37 @@ class Curl(AutotoolsPackage):
     version('7.43.0', '11bddbb452a8b766b932f859aaeeed39')
     version('7.42.1', '296945012ce647b94083ed427c1877a8')
 
-    variant('nghttp2', default=False, description='build nghttp2 library (requires C++11)')
-    variant('libssh2', default=False, description='enable libssh2 support')
+    variant('nghttp2',    default=False, description='build nghttp2 library (requires C++11)')
+    variant('libssh2',    default=False, description='enable libssh2 support')
+    variant('libssh',     default=False, description='enable libssh support')  # , when='7.58:')
+    variant('darwinssl',  default=sys.platform == 'darwin', description="use Apple's SSL/TLS implementation")
 
-    depends_on('openssl')
+    conflicts('+libssh', when='@:7.57.99')
+    # on OSX and --with-ssh the configure steps fails with
+    # one or more libs available at link-time are not available run-time
+    # unless the libssh are installed externally (e.g. via homebrew), even
+    # though spack isn't supposed to know about such a libssh installation.
+    # C.f. https://github.com/spack/spack/issues/7777
+    conflicts('platform=darwin', when='+libssh2')
+    conflicts('platform=darwin', when='+libssh')
+    conflicts('platform=linux', when='+darwinssl')
+
+    depends_on('openssl', when='~darwinssl')
     depends_on('zlib')
     depends_on('nghttp2', when='+nghttp2')
     depends_on('libssh2', when='+libssh2')
+    depends_on('libssh', when='+libssh')
 
     def configure_args(self):
         spec = self.spec
 
-        args = [
-            '--with-zlib={0}'.format(spec['zlib'].prefix),
-            '--with-ssl={0}'.format(spec['openssl'].prefix)
-        ]
+        args = ['--with-zlib={0}'.format(spec['zlib'].prefix)]
+        if self.satisfies('+darwinssl'):
+            args.append('--with-darwinssl')
+        else:
+            args.append('--with-ssl={0}'.format(spec['openssl'].prefix))
+
         args += self.with_or_without('nghttp2')
         args += self.with_or_without('libssh2')
+        args += self.with_or_without('libssh')
         return args
diff --git a/var/spack/repos/builtin/packages/libssh/package.py b/var/spack/repos/builtin/packages/libssh/package.py
new file mode 100644
--- /dev/null
+++ b/var/spack/repos/builtin/packages/libssh/package.py
@@ -0,0 +1,37 @@
+##############################################################################
+# Copyright (c) 2013-2018, Lawrence Livermore National Security, LLC.
+# Produced at the Lawrence Livermore National Laboratory.
+#
+# This file is part of Spack.
+# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved.
+# LLNL-CODE-647188
+#
+# For details, see https://github.com/spack/spack
+# Please also see the NOTICE and LICENSE files for our notice and the LGPL.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License (as
+# published by the Free Software Foundation) version 2.1, February 1999.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and
+# conditions of the GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+##############################################################################
+from spack import *
+
+
+class Libssh(CMakePackage):
+    """libssh: the SSH library"""
+
+    homepage = "https://www.libssh.org"
+    url      = "https://red.libssh.org/attachments/download/218/libssh-0.7.5.tar.xz"
+
+    version('0.7.5', 'd3fc864208bf607ad87cdee836894feb')
+
+    depends_on('openssl')
+    depends_on('zlib')


Test patch: