Repository: spack/spack

Issue:
nlopt build problem - CMakeLists.txt missing
This is a new package to my spack build.  I have not added any variant args just ```spack install nlopt```

### Expected Result

clean build

### Actual Result

==> Installing nlopt
curl: (37) Couldn't open file MIRROR_DIR/mirror/nlopt/nlopt-2.4.2.tar.gz
==> Fetching file://MIRROR_DIR/mirror/nlopt/nlopt-2.4.2.tar.gz
==> Fetching from file://MIRROR_DIR/mirror/nlopt/nlopt-2.4.2.tar.gz failed.
==> Fetching https://github.com/stevengj/nlopt/releases/download/nlopt-2.4.2/nlopt-2.4.2.tar.gz
==> Staging archive: WORKING_DIR/var/spack/stage/nlopt-2.4.2-wei4zixpd7mb2b7zn2uniak5svelp6wb/nlopt-2.4.2.tar.gz
==> Created stage in WORKING_DIR/var/spack/stage/nlopt-2.4.2-wei4zixpd7mb2b7zn2uniak5svelp6wb
==> No patches needed for nlopt
==> Building nlopt [CMakePackage]
==> Executing phase: 'cmake'
==> Error: ProcessError: Command exited with status 1:
    'cmake' 'WORKING_DIR/var/spack/stage/nlopt-2.4.2-wei4zixpd7mb2b7zn2uniak5svelp6wb/nlopt-2.4.2' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:PATH=WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/nlopt-2.4.2-wei4zixpd7mb2b7zn2uniak5svelp6wb' '-DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo' '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=FALSE' '-DCMAKE_INSTALL_RPATH:STRING=WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/nlopt-2.4.2-wei4zixpd7mb2b7zn2uniak5svelp6wb/lib;WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/nlopt-2.4.2-wei4zixpd7mb2b7zn2uniak5svelp6wb/lib64;WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-52dgd34kwi53hcwhmkggokoplqxzzmfp/lib;WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/bzip2-1.0.6-egkg2fbrcljkai2d23pgrq7wzeg2k5p5/lib;WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/gdbm-1.14.1-nu73hb3bwwumnr5pe7n2ovwpvn3lj32c/lib;WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/readline-7.0-6a33hh42gihqevhhwm7ddsklhu7xwkgk/lib;WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/ncurses-6.0-tymhqxlvcy5eduud5ijnohpkuinoi7n4/lib;WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/openssl-1.0.2n-b2pwlezjpb63d4lvpnb6hppps7qw4cap/lib;WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/zlib-1.2.11-euf4keckd7zopxedqa5pp37smh4v2aa4/lib;WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/sqlite-3.18.0-ss65uo3pidaok5xk6ggbrastms2ludx4/lib;WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/pcre-8.41-tnraxjn7calk2xer5ructusju54toa5j/lib' '-DCMAKE_PREFIX_PATH:STRING=WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-52dgd34kwi53hcwhmkggokoplqxzzmfp;WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/swig-3.0.12-eww74pu5pubgf2fhsu24t7m227qooxlx;WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/cmake-3.7.2-2emsjuf7ycy6ai6l7ln4zkwgybanaug3;WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-numpy-1.14.2-foa5vlkdxayrm4cv3wrusczloinaxkls' '-DPYTHON_EXECUTABLE=WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-52dgd34kwi53hcwhmkggokoplqxzzmfp/bin/python2.7'

1 error found in build log:
     1    ==> Executing phase: 'cmake'
     2    ==> 'cmake' 'WORKING_DIR/var/spack/stage/nlopt-2.4.2-wei4zixp
          d7mb2b7zn2uniak5svelp6wb/nlopt-2.4.2' '-G' 'Unix Makefiles' '-DCMAKE_
          INSTALL_PREFIX:PATH=WORKING_DIR/opt/spack/linux-centos7-x86_6
          4/gcc-5.4.0/nlopt-2.4.2-wei4zixpd7mb2b7zn2uniak5svelp6wb' '-DCMAKE_BU
          ILD_TYPE:STRING=RelWithDebInfo' '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' '-
          DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=FALSE' '-DCMAKE_INSTALL_RPATH
          :STRING=WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/
          nlopt-2.4.2-wei4zixpd7mb2b7zn2uniak5svelp6wb/lib;WORKING_DIR/
          opt/spack/linux-centos7-x86_64/gcc-5.4.0/nlopt-2.4.2-wei4zixpd7mb2b7z
          n2uniak5svelp6wb/lib64;WORKING_DIR/opt/spack/linux-centos7-x8
          6_64/gcc-5.4.0/python-2.7.13-52dgd34kwi53hcwhmkggokoplqxzzmfp/lib;/sc
          1/apps/spack/wip/opt/spack/linux-centos7-x86_64/gcc-5.4.0/bzip2-1.0.6
          -egkg2fbrcljkai2d23pgrq7wzeg2k5p5/lib;WORKING_DIR/opt/spack/l
          inux-centos7-x86_64/gcc-5.4.0/gdbm-1.14.1-nu73hb3bwwumnr5pe7n2ovwpvn3
          lj32c/lib;WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.
          0/readline-7.0-6a33hh42gihqevhhwm7ddsklhu7xwkgk/lib;WORKING_DIR
          ip/opt/spack/linux-centos7-x86_64/gcc-5.4.0/ncurses-6.0-tymhqxlvcy5ed
          uud5ijnohpkuinoi7n4/lib;WORKING_DIR/opt/spack/linux-centos7-x
          86_64/gcc-5.4.0/openssl-1.0.2n-b2pwlezjpb63d4lvpnb6hppps7qw4cap/lib;/
          sc1/apps/spack/wip/opt/spack/linux-centos7-x86_64/gcc-5.4.0/zlib-1.2.
          11-euf4keckd7zopxedqa5pp37smh4v2aa4/lib;WORKING_DIR/opt/spack
          /linux-centos7-x86_64/gcc-5.4.0/sqlite-3.18.0-ss65uo3pidaok5xk6ggbras
          tms2ludx4/lib;WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-
          5.4.0/pcre-8.41-tnraxjn7calk2xer5ructusju54toa5j/lib' '-DCMAKE_PREFIX
          _PATH:STRING=WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5
          .4.0/python-2.7.13-52dgd34kwi53hcwhmkggokoplqxzzmfp;WORKING_DIR
          ip/opt/spack/linux-centos7-x86_64/gcc-5.4.0/swig-3.0.12-eww74pu5pubgf
          2fhsu24t7m227qooxlx;WORKING_DIR/opt/spack/linux-centos7-x86_6
          4/gcc-5.4.0/cmake-3.7.2-2emsjuf7ycy6ai6l7ln4zkwgybanaug3;WORKING_DIR
          ack/wip/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-numpy-1.14.2-foa5
          vlkdxayrm4cv3wrusczloinaxkls' '-DPYTHON_EXECUTABLE=WORKING_DIR
          p/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-52dgd34kwi53
          hcwhmkggokoplqxzzmfp/bin/python2.7'
  >> 3    CMake Error: The source directory "WORKING_DIR/var/spack/stag
          e/nlopt-2.4.2-wei4zixpd7mb2b7zn2uniak5svelp6wb/nlopt-2.4.2" does not
          appear to contain CMakeLists.txt.
     4    Specify --help for usage, or press the help button on the CMake GUI.

See build log for details:
  WORKING_DIR/var/spack/stage/nlopt-2.4.2-wei4zixpd7mb2b7zn2uniak5svelp6wb/nlopt-2.4.2/spack-build.out

### Information on your system

linux-centos7-x86_64/gcc-5.4.0


Discussion:
It looks like this package was added by @shuds13 in #6499.

The problem is that the latest release (which is installed by default) uses Autotools, not CMake. To make things better, the develop branch only supports CMake, not Autotools.

The solution is to support both build systems. See `arpack-ng`, `dyninst`, `glog`, `metis`, `opennurbs`, and `superlu` for examples of how to do this. If anyone wants to submit a PR to fix this, I would be more than happy to review/merge.
MAYBE - I am unsure of the configure_args part.  It builds without errors, but I do not see any "*.so" files though spack tells me ```4rrglbb nlopt@2.4.2%gcc~cxx~guile~matlab~octave+python+shared``` implying the shared libraries were built...
```
from spack import *


class Nlopt(Package):
    """NLopt is a free/open-source library for nonlinear optimization,
    providing a common interface for a number of different free optimization
    routines available online as well as original implementations of various
    other algorithms."""

    homepage = "https://nlopt.readthedocs.io"
    url      = "https://github.com/stevengj/nlopt/releases/download/nlopt-2.4.2/nlopt-2.4.2.tar.gz"

    version('develop', git='https://github.com/stevengj/nlopt.git', branch='master')
    version('2.4.2', 'd0b8f139a4acf29b76dbae69ade8ac54')

    variant('shared', default=True, description='Enables the build of shared libraries')
    variant('python', default=True, description='Build python wrappers')
    variant('guile',  default=False, description='Enable Guile support')
    variant('octave', default=False, description='Enable GNU Octave support')
    variant('cxx',    default=False,  description='Build the C++ routines')

    # Note: matlab is licenced - spack does not download automatically
    variant("matlab", default=False, description="Build the Matlab bindings.")

    depends_on('cmake@3.0:', type='build')
    depends_on('python', when='+python')
    depends_on('py-numpy', when='+python', type=('build', 'run'))
    depends_on('swig', when='+python')
    depends_on('guile', when='+guile')
    depends_on('octave', when='+octave')
    depends_on('matlab', when='+matlab')

    def install(self, spec, prefix):
        if spec.satisfies('@2.4.2'):
            configure('--prefix=%s' % prefix)
            make
            make('install')
            return

        cmake_args = ['-DBUILD_SHARED_LIBS=TRUE']
        if '+python' in spec:
            cmake_args.append("-DPYTHON_EXECUTABLE=%s" % spec['python'].command.path)
        if '-shared' in spec:
            cmake_args.append('-DBUILD_SHARED_LIBS:Bool=OFF')
        if '+cxx' in spec:
            cmake_args.append('-DNLOPT_CXX:BOOL=ON')
        if '+matlab' in spec:
            cmake_args.append("-DMatlab_ROOT_DIR=%s" % spec['matlab'].command.path)
        cmake_args.extend(std_cmake_args)

        with working_dir('spack-build', create=True):
            cmake('.', *cmake_args)
            make
            make('install')

    def configure_args(self):
        options = []
        options.extend('CFLAGS=-DBUILD_SHARED_LIBS=TRUE')
        return options
```
Further evidence that my nlopt build is garbage is that when referenced for the r-nloptr package build, a ton of these processes are created but never seem to complete
```
/bin/sh ./configure --prefix=/tmp/peetsv/spack-stage/spack-stage-DGRBlx/nloptr --enable-shared
 --enable-static --without-octave --without-matlab --without-guile --without-python
 --with-cxx CC=WORKING_DIR/lib/spack/env/gcc/gcc CFLAGS=-g -O2
 CPP=WORKING_DIR/lib/spack/env/gcc/gcc -E CPPFLAGS=-I/usr/local/include
 CXX=WORKING_DIR/lib/spack/env/gcc/g++ CXXFLAGS=-g -O2
 CXXCPP=WORKING_DIR/lib/spack/env/gcc/g++ -E
```
for instance:
```
$ ps -ef | grep -c configure
975
```
When it got over 1200, I interrupted the ```spack install r-nloptr```
```
==> Installing r-nloptr
==> Using cached archive: WORKING_DIR/var/spack/cache/r-nloptr/r-nloptr-1.0.4.tar.gz
==> Staging archive: WORKING_DIR/var/spack/stage/r-nloptr-1.0.4-l3qyfzbiju6qosl5kqnzdjojgurhbdjf/nloptr_1.0.4.tar.gz
==> Created stage in WORKING_DIR/var/spack/stage/r-nloptr-1.0.4-l3qyfzbiju6qosl5kqnzdjojgurhbdjf
==> No patches needed for r-nloptr
==> Building r-nloptr [RPackage]
==> Executing phase: 'install'
==> Error: Exception occurred in writer daemon!
Traceback (most recent call last):
  File "WORKING_DIR/lib/spack/llnl/util/tty/log.py", line 452, in _writer_daemon

==> Error: Keyboard interrupt.
    rlist, _, xlist = select.select(istreams, [], [], 0)
```
I added this package (as a user). It would be a good idea to support both build methods. I was not able to get it to build shared libraries on our XC40 system regardless of Spack. Other systems were OK.
So, to be more specific --- I added
```
depends_on('nlopt', type='build')
```
to the r-nloptr package.py and got the following error:
```
==> Installing r-nloptr
==> Using cached archive: WORKING_DIR/var/spack/cache/r-nloptr/r-nloptr-1.0.4.tar.gz
==> Staging archive: WORKING_DIR/var/spack/stage/r-nloptr-1.0.4-l3qyfzbiju6qosl5kqnzdjojgurhbdjf/nloptr_1.0.4.tar.gz
==> Created stage in WORKING_DIR/var/spack/stage/r-nloptr-1.0.4-l3qyfzbiju6qosl5kqnzdjojgurhbdjf
==> No patches needed for r-nloptr
==> Building r-nloptr [RPackage]
==> Executing phase: 'install'
==> Error: ProcessError: Command exited with status 1:
    'WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/r-3.3.2-qf7qio5ehqhix7yi5epqwovdmoatrg55/bin/R' 'CMD' 'INSTALL' '--library=WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/r-nloptr-1.0.4-l3qyfzbiju6qosl5kqnzdjojgurhbdjf/rlib/R/library' 'WORKING_DIR/var/spack/stage/r-nloptr-1.0.4-l3qyfzbiju6qosl5kqnzdjojgurhbdjf/nloptr'

2 errors found in build log:
     61    WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/r-3.3.2
           -qf7qio5ehqhix7yi5epqwovdmoatrg55/rlib/R/etc/Makeconf:159: warning:
           ignoring old recipe for target `.f90.o'
     62    WORKING_DIR/lib/spack/env/gcc/g++ -IWORKING_DIR/opt/
           spack/linux-centos7-x86_64/gcc-5.4.0/r-3.3.2-qf7qio5ehqhix7yi5epqwov
           dmoatrg55/rlib/R/include -DNDEBUG  -I/usr/local/include    -fpic  -g
            -O2  -c dummy.cpp -o dummy.o
     63    WORKING_DIR/lib/spack/env/gcc/gcc -IWORKING_DIR/opt/
           spack/linux-centos7-x86_64/gcc-5.4.0/r-3.3.2-qf7qio5ehqhix7yi5epqwov
           dmoatrg55/rlib/R/include -DNDEBUG  -I/usr/local/include   -IWORKING_DIR
           s/spack/wip/opt/spack/linux-centos7-x86_64/gcc-5.4.0/nlopt-2.4.2-4rr
           glbb4xtjqvporzruh36yaxuuhuto2/include   -fpic  -g -O2  -c nloptr.c -
           o nloptr.o
     64    WORKING_DIR/lib/spack/env/gcc/g++ -shared -LWORKING_DIR
           wip/opt/spack/linux-centos7-x86_64/gcc-5.4.0/r-3.3.2-qf7qio5ehqhix7y
           i5epqwovdmoatrg55/rlib/R/lib -L/usr/local/lib64 -o nloptr.so dummy.o
            nloptr.o -LWORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5
           .4.0/nlopt-2.4.2-4rrglbb4xtjqvporzruh36yaxuuhuto2/lib -lnlopt -lm -L
           WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/r-3.3.2
           -qf7qio5ehqhix7yi5epqwovdmoatrg55/rlib/R/lib -lR
     65    /usr/bin/ld: WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-
           5.4.0/nlopt-2.4.2-4rrglbb4xtjqvporzruh36yaxuuhuto2/lib/libnlopt.a(ge
           neral.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be
            used when making a shared object; recompile with -fPIC
     66    WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/nlopt-2
           .4.2-4rrglbb4xtjqvporzruh36yaxuuhuto2/lib/libnlopt.a: could not read
            symbols: Bad value
  >> 67    collect2: error: ld returned 1 exit status
  >> 68    make: *** [nloptr.so] Error 1
     69    ERROR: compilation failed for package ‘nloptr’
     70    * removing ‘WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc
           -5.4.0/r-nloptr-1.0.4-l3qyfzbiju6qosl5kqnzdjojgurhbdjf/rlib/R/librar
           y/nloptr’

See build log for details:
  WORKING_DIR/var/spack/stage/r-nloptr-1.0.4-l3qyfzbiju6qosl5kqnzdjojgurhbdjf/nloptr/spack-build.out
```
The detail log ends with:
```
/usr/bin/ld: WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/nlopt-2.4.2-4rrglbb4xtjqvporzruh36yaxuuhuto2/lib/libnlopt.a(general.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/nlopt-2.4.2-4rrglbb4xtjqvporzruh36yaxuuhuto2/lib/libnlopt.a: could not read symbols: Bad value
collect2: error: ld returned 1 exit status
make: *** [nloptr.so] Error 1
ERROR: compilation failed for package ‘nloptr’
* removing ‘WORKING_DIR/opt/spack/linux-centos7-x86_64/gcc-5.4.0/r-nloptr-1.0.4-l3qyfzbiju6qosl5kqnzdjojgurhbdjf/rlib/R/library/nloptr’
```
Good news - no runaway invocations of the configure script ;-)
Alright, I got part of it --- the shared libs are being created with this
```
    def install(self, spec, prefix):
        if spec.satisfies('@2.4.2'):
            options = ['--enable-shared']
            configure('--prefix=%s' % prefix, *options)
            make
            make('install')
            return
```
I will be putting through a PR for nlopt soon.
Although Steven G. Johnson is the author of NLOpt, the URLs and git addresses shown above, related to `stevengij`,  are NOT OFFICIAL releases:
```
    homepage = "https://nlopt.readthedocs.io"
    url      = "https://github.com/stevengj/nlopt/releases/download/nlopt-2.4.2/nlopt-2.4.2.tar.gz"

    version('develop', git='https://github.com/stevengj/nlopt.git', branch='master')
    version('2.4.2', 'd0b8f139a4acf29b76dbae69ade8ac54')
```
I would recommend we switch to official releases; and then see how the Autotools vs. CMake stuff plays out.  Official tarballs can be downloaded from:

http://ab-initio.mit.edu/nlopt/nlopt-2.4.2.tar.gz

@shuds13  Does anyone here need the `develop` version?  If not, would it be easiest for now to use the Autotools build that comes with the tarball?

A bit more sleuthing... OK, the stevengj/nlopt IS the official GitHub repo.  Look at the tags...

v2.4.1 uses Autotools-only.  v2.4.2 contains Autotools and CMake builds.  The master branch is CMake-only.  So... looks like we'll need the dual-build system setup, if we want to support versions before 2.4.2.



To the previous question, I don't need the develop version. 2.4.2 is fine for me.
@citibeth I believe the v2.4.2 tarball does not contain the Cmake instructions.   As I noted in #8149 I tried to use the CMake build on each of the tarballs to the same result as I reported above.  It appears to me that the v2.4.2 tarball does not have the necessary bits to build via CMake.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/nlopt/package.py b/var/spack/repos/builtin/packages/nlopt/package.py
--- a/var/spack/repos/builtin/packages/nlopt/package.py
+++ b/var/spack/repos/builtin/packages/nlopt/package.py
@@ -32,11 +32,11 @@ class Nlopt(CMakePackage):
     other algorithms."""
 
     homepage = "https://nlopt.readthedocs.io"
-    url      = "https://github.com/stevengj/nlopt/releases/download/nlopt-2.4.2/nlopt-2.4.2.tar.gz"
+    url      = "https://github.com/stevengj/nlopt/archive/v2.5.0.tar.gz"
     git      = "https://github.com/stevengj/nlopt.git"
 
     version('develop', branch='master')
-    version('2.4.2', 'd0b8f139a4acf29b76dbae69ade8ac54')
+    version('2.5.0', 'ada08c648bf9b52faf8729412ff6dd6d')
 
     variant('shared', default=True, description='Enables the build of shared libraries')
     variant('python', default=True, description='Build python wrappers')


Test patch: