Repository: spack/spack

Issue:
`spack install` output is being inserted into autoconf generated `configure` script
I'm trying to add a newer release of the kallisto package, ideally 0.46.2.

Kallisto is a CMakePackage and/but is "interesting" (may you live in interesting times) because it has vendored its own copy of the `htslib` library and runs `autoheader` and `autoconf` and `configure` in the `htslib` dir via this bit of the toplevel `CMakeLists.txt`:

```
include(ExternalProject)
ExternalProject_Add(htslib
    PREFIX ${PROJECT_SOURCE_DIR}/ext/htslib
    SOURCE_DIR ${PROJECT_SOURCE_DIR}/ext/htslib
    BUILD_IN_SOURCE 1
    CONFIGURE_COMMAND autoheader && autoconf && ${PROJECT_SOURCE_DIR}/ext/htslib/configure
        --prefix=${PREFIX} --disable-bz2 --disable-lzma --disable-libcurl
    BUILD_COMMAND make lib-static
    INSTALL_COMMAND ""
)
```

Anything newer than the version we have in the tree (0.43.1) dies with a similar problem, it seems like the output of the `spack` command is being included in the generated files.

Here's the first few lines of the generated `configure` script

```
#! /bin/sh
# Guess values for system-dependent variables and create Makefiles.
# Generated by GNU Autoconf 2.69 for HTSlib make3: Entering directory `/tmp/ghartzell/spack-stage/spack-stage-kallisto-0.46.2-2cya5t4j2epry6ttce7w2bm5eeoqurjf/spack-src/ext/htslib'
1.4.1
make3: Leaving directory `/tmp/ghartzell/spack-stage/spack-stage-kallisto-0.46.2-2cya5t4j2epry6ttce7w2bm5eeoqurjf/spack-src/ext/htslib'.
#
# Report bugs to <samtools-help@lists.sourceforge.net>.
#
```

The build dies when cmake tries to build the vendored `htslib` lib and stubs its toe on the line tht only contains `1.4.1`:

```
[...]
==> Error: ProcessError: Command exited with status 2:
    'make' '-j16'

4 errors found in build log:
     109    configure.ac:26: 1.4.1
     110    configure.ac:26: make3: Leaving directory `/tmp/ghartzell/spack-stage/spack-stage-kallisto-0.46.2-2cya5t4j2epry6ttce7w2bm5eeoqurjf/spack-src/ext/htslib'
     111    configure.ac:26: warning: AC_INIT: not a literal: make3: Entering directory `/tmp/ghartzell/spack-stage/spack-stage-kallisto-0.46.2-2cya5t4j2epry6ttce7w2bm5eeoqurjf/spac
            k-src/ext/htslib'
     112    configure.ac:26: 1.4.1
     113    configure.ac:26: make3: Leaving directory `/tmp/ghartzell/spack-stage/spack-stage-kallisto-0.46.2-2cya5t4j2epry6ttce7w2bm5eeoqurjf/spack-src/ext/htslib'
     114    /tmp/ghartzell/spack-stage/spack-stage-kallisto-0.46.2-2cya5t4j2epry6ttce7w2bm5eeoqurjf/spack-src/ext/htslib/configure: line 4: 1.4.1: command not found
  >> 115    /tmp/ghartzell/spack-stage/spack-stage-kallisto-0.46.2-2cya5t4j2epry6ttce7w2bm5eeoqurjf/spack-src/ext/htslib/configure: line 34: syntax error near unexpected token `('
     116    /tmp/ghartzell/spack-stage/spack-stage-kallisto-0.46.2-2cya5t4j2epry6ttce7w2bm5eeoqurjf/spack-src/ext/htslib/configure: line 34: `  case `(set -o) 2>/dev/null` in #('
  >> 117    make[2]: *** [/tmp/ghartzell/spack-stage/spack-stage-kallisto-0.46.2-2cya5t4j2epry6ttce7w2bm5eeoqurjf/spack-src/ext/htslib/src/htslib-stamp/htslib-configure] Error 2
     118    make[2]: Leaving directory `/tmp/ghartzell/spack-stage/spack-stage-kallisto-0.46.2-2cya5t4j2epry6ttce7w2bm5eeoqurjf/spack-build'
  >> 119    make[1]: *** [CMakeFiles/htslib.dir/all] Error 2
     120    make[1]: *** Waiting for unfinished jobs....
[...]
```

On the theory that I was picking up old CentOS versions of the tools, I tried (unconditionally, for now) adding dependencies on the usual autotools suspects, to no avail.  Here's the changes:

```
$ git diff
diff --git a/var/spack/repos/builtin/packages/kallisto/package.py b/var/spack/repos/builtin/packages/kallisto/package.py
index 337b50f18..81d509cb4 100644
--- a/var/spack/repos/builtin/packages/kallisto/package.py
+++ b/var/spack/repos/builtin/packages/kallisto/package.py
@@ -13,8 +13,14 @@ class Kallisto(CMakePackage):
     homepage = "http://pachterlab.github.io/kallisto"
     url      = "https://github.com/pachterlab/kallisto/archive/v0.43.1.tar.gz"

+    version('0.46.2', sha256='c447ca8ddc40fcbd7d877d7c868bc8b72807aa8823a8a8d659e19bdd515baaf2')
     version('0.43.1', sha256='7baef1b3b67bcf81dc7c604db2ef30f5520b48d532bf28ec26331cb60ce69400')

     depends_on('zlib')
     depends_on('hdf5')
     depends_on('mpich')
+
+    depends_on('autoconf', type='build')
+    depends_on('automake', type='build')
+    depends_on('libtool',  type='build')
+    depends_on('m4',       type='build')
$
```


Discussion:
@hartzell Can it be a viable solution to patch Kallisto's CMake to avoid external projects and use a Spack built `htslib`?
Unlike some other packages, Kallisto doesn't seem to have an official option to use an external `htlslib`.  [Other folks have complained](https://github.com/pachterlab/kallisto/issues/161) a while back that there should be a way to use an external one (e.g. EasyBuild's), although it seems that the root of their problem at that version it seems that Cmake wasn't automagically running the auto-tools.

Hacking it up might work, technically, but I suspect that the bioinformaticians would have to waste a lot of time explaining *why* they're using a non-standard version of kallisto, and "are you sure it's giving the same answer and the normal one?" and etc.....

It's annoying that they vendor'ed the library, too bad they didn't have a good package manager (cough-Spack-cough).

I'm sure (hah hah hah...) that it's going to be a case of Spack's std{in,out,err} trickery getting in the way of the configure scripts' std{in,out,err} trickery.

I suspect that the eventual fix is going to be to patch whatever Kallisto's vendor'ed `htslib` is doing.

But, I was hoping I wasn't the first person to stumble on this and could just re-use the previous solution....

Or, failing that, a suggestion about which bit of std{in,out,err} trickery might be my stumbling block.

~Is there a flag/option to `spack` or `spack install` that disables the std{in,out,err} trickery?~

Never mind, see below....
I take back any grumbling I did in the `spack` commands direction, *mea culpa* and *etc*.

It seems to be a CMake and autotools interaction, I can trigger it by building by hand.

Spack builds it successfully with `^cmake@3.14.5` and fails with `^cmake@3.15.5` and `^cmake@3.16.21` (the default).

Now just to figure out what to do with this juicy tidbit.
Sigh. Sorry for the false hope, I'm back to blaming Spack.

I have proof in my iTerm scroll back buffer that I built it successfully once, but I can't replicate it.  Perhaps there's something racy, but it still fails with `-jobs 1`.

I had been playing with having Spack run it's bit up through the stage of running cmake then building the rest by hand, perhaps I had left something hanging around from that? Confused, but pressing on.

I had spack stage the tarball and set up an environment for me with all of the dependencies including cmake@3.16.2 (`spack stage kallisto; spack build-env kallisto bash`) and then ran the commands from kallisto's INSTALL.md (`mkdir build; cd build; cmake ..; make`) and it built successfully.

Back to looking at Spack's CMakePackage.
@adamjstewart -- You've played a lot with CMake, any thoughts on what might be causing it's output to end up in autoconf's `configure`?
Next step, I can follow kallisto's `INSTALL.md` and build it by hand successfully (with a `spack build-env kallisto bash` environment) if I just use `cmake ..`, but I copy/paste the cmake invocation that `spack install -v --until cmake kallisto` uses, I get the familiar crash:

```
'cmake' '/tmp/ghartzell/spack-stage/spack-stage-kallisto-0.46.2-2cya5t4j2epry6ttce7w2bm5eeoqurjf/spack-src' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:PATH=/home/ghartzell/tmp/spack-update-kallisto/opt/spack/linux-centos7-skylake_avx512/gcc-8.2.0/kallisto-0.46.2-2cya5t4j2epry6ttce7w2bm5eeoqurjf' '-DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo' '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=FALSE' '-DCMAKE_INSTALL_RPATH:STRING=/home/ghartzell/tmp/spack-update-kallisto/opt/spack/linux-centos7-skylake_avx512/gcc-8.2.0/kallisto-0.46.2-2cya5t4j2epry6ttce7w2bm5eeoqurjf/lib;/home/ghartzell/tmp/spack-update-kallisto/opt/spack/linux-centos7-skylake_avx512/gcc-8.2.0/kallisto-0.46.2-2cya5t4j2epry6ttce7w2bm5eeoqurjf/lib64;/home/ghartzell/tmp/spack-update-kallisto/opt/spack/linux-centos7-skylake_avx512/gcc-8.2.0/hdf5-1.10.6-hvo5omf3klxs2lhmbaufjuyzalomtvoa/lib;/home/ghartzell/tmp/spack-update-kallisto/opt/spack/linux-centos7-skylake_avx512/gcc-8.2.0/mpich-3.3.2-caxsvonveofogph6egb36ivlekau3dp3/lib;/home/ghartzell/tmp/spack-update-kallisto/opt/spack/linux-centos7-skylake_avx512/gcc-8.2.0/libpciaccess-0.13.5-jaezs7g5lcf5whkstrhy6lifyfgekux7/lib;/home/ghartzell/tmp/spack-update-kallisto/opt/spack/linux-centos7-skylake_avx512/gcc-8.2.0/libxml2-2.9.9-d2x6qqpzdkuf65moitkgjevagpbrfof7/lib;/home/ghartzell/tmp/spack-update-kallisto/opt/spack/linux-centos7-skylake_avx512/gcc-8.2.0/libiconv-1.16-mtrpapxd5y62i7hyydroch236qi6ttml/lib;/home/ghartzell/tmp/spack-update-kallisto/opt/spack/linux-centos7-skylake_avx512/gcc-8.2.0/xz-5.2.4-jjka3ksvf266znwongyct5ebdcwnzvyp/lib;/home/ghartzell/tmp/spack-update-kallisto/opt/spack/linux-centos7-skylake_avx512/gcc-8.2.0/zlib-1.2.11-s3oedluluhezqxgxgjx6g3tllytxygkx/lib' '-DCMAKE_PREFIX_PATH:STRING=/home/ghartzell/tmp/spack-update-kallisto/opt/spack/linux-centos7-skylake_avx512/gcc-8.2.0/cmake-3.16.2-boy5cb3fzwvxkxdsum4g4ldyhcqqq2wr;/home/ghartzell/tmp/spack-update-kallisto/opt/spack/linux-centos7-skylake_avx512/gcc-8.2.0/hdf5-1.10.6-hvo5omf3klxs2lhmbaufjuyzalomtvoa;/home/ghartzell/tmp/spack-update-kallisto/opt/spack/linux-centos7-skylake_avx512/gcc-8.2.0/zlib-1.2.11-s3oedluluhezqxgxgjx6g3tllytxygkx;/home/ghartzell/tmp/spack-update-kallisto/opt/spack/linux-centos7-skylake_avx512/gcc-8.2.0/automake-1.16.1-rkb7xalzsg244tppdjuehaxgexj4upg2;/home/ghartzell/tmp/spack-update-kallisto/opt/spack/linux-centos7-skylake_avx512/gcc-8.2.0/m4-1.4.18-dc6pg74a7pjst7sswy6hxk7tld7gkgzl;/home/ghartzell/tmp/spack-update-kallisto/opt/spack/linux-centos7-skylake_avx512/gcc-8.2.0/libtool-2.4.6-6gaoj3ergcctxf3g7d6rka3patrifrod;/home/ghartzell/tmp/spack-update-kallisto/opt/spack/linux-centos7-skylake_avx512/gcc-8.2.0/mpich-3.3.2-caxsvonveofogph6egb36ivlekau3dp3;/home/ghartzell/tmp/spack-update-kallisto/opt/spack/linux-centos7-skylake_avx512/gcc-8.2.0/autoconf-2.69-wh2wedgv7dzysdtm4rbkxehmk2ckylo3'
```
After copying that cmake invocation into a safe place and brute forcing to see which bit was causing the problem, it seems like 

`-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON`

is the culprit.

If I run `cmake .. -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON` I get the expected crash, and if I remove that line from the "standard invocation", I can build it.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/kallisto/package.py b/var/spack/repos/builtin/packages/kallisto/package.py
--- a/var/spack/repos/builtin/packages/kallisto/package.py
+++ b/var/spack/repos/builtin/packages/kallisto/package.py
@@ -13,8 +13,37 @@ class Kallisto(CMakePackage):
     homepage = "http://pachterlab.github.io/kallisto"
     url      = "https://github.com/pachterlab/kallisto/archive/v0.43.1.tar.gz"
 
+    version('0.46.2', sha256='c447ca8ddc40fcbd7d877d7c868bc8b72807aa8823a8a8d659e19bdd515baaf2')
     version('0.43.1', sha256='7baef1b3b67bcf81dc7c604db2ef30f5520b48d532bf28ec26331cb60ce69400')
 
     depends_on('zlib')
     depends_on('hdf5')
     depends_on('mpich')
+
+    # htslib isn't built in time to be used....
+    parallel = False
+
+    # v0.44.0 vendored a copy of htslib and uses auto* to build
+    # its configure script.
+    depends_on('autoconf', type='build', when='@0.44.0:')
+    depends_on('automake', type='build', when='@0.44.0:')
+    depends_on('libtool',  type='build', when='@0.44.0:')
+    depends_on('m4',       type='build', when='@0.44.0:')
+
+    # Including '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' in the cmake args
+    # causes bits of cmake's output to end up in the autoconf-generated
+    # configure script.
+    # See https://github.com/spack/spack/issues/15274 and
+    # https://github.com/pachterlab/kallisto/issues/253
+    @property
+    def std_cmake_args(self):
+        """Call the original std_cmake_args and then filter the verbose
+        setting.
+        """
+        a = super(Kallisto, self).std_cmake_args
+        if (self.spec.version >= Version('0.44.0')):
+            args = [i for i in a if i != '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON']
+        else:
+            args = a
+
+        return args


Test patch: