Repository: spack/spack

Issue:
cryptsetup: UUID library not found
New cryptsetup package was merged into develop just yesterday (#12762).

I was quite sure it was working when merged.  I tested a couple hours before, by rebasing my develop fork and merging.  And it worked!   But today, working from develop HEAD (currently at 3f06d5c12), it looks broken:

```console
$ spack install cryptsetup
[snip]
1 error found in build log:
     138    checking for linux/keyctl.h... yes
     139    checking whether __NR_add_key is declared... yes
     140    checking whether __NR_keyctl is declared... yes
     141    checking whether __NR_request_key is declared... yes
     142    checking for key_serial_t... no
     143    checking for uuid_clear in -luuid... no
  >> 144    configure: error: You need the uuid library.
```

I thought I was going mad, but by trial and error, I found that if I do a `git revert b95b4bb9e`:

```
$ spack install cryptsetup
[snip]
==> Successfully installed cryptsetup
```

b95b4bb9e refers to #12794  -- "add compilation option to sqlite" and it merged just 10 minutes before #12762

I don't currently understand how #12794 could be breaking cryptsetup.  It seems impossible, since it only added a variant that is off by default.

@Sinan81  @hartzell 


Discussion:
I did a diff of `spack spec cryptsetup`, and the only difference is the presence of the new variant `~column_metadata` which is off (by default) as expected.

```diff
$ diff spec_revert_b95b4bb9e.txt spec_HEAD.txt
37c37
<             ^sqlite@3.28.0%gcc@4.8.5+fts~functions~rtree arch=linux-centos7-x86_64
---
>             ^sqlite@3.28.0%gcc@4.8.5~column_metadata+fts~functions~rtree arch=linux-centos7-x86_64     
```

I remain thoroughly confused.  Here's the full spec for cryptsetup, from current HEAD (3f06d5c):

```console
$ ./bin/spack spec cryptsetup 
Input spec
--------------------------------
cryptsetup

Concretized
--------------------------------
cryptsetup@2.2.1%gcc@4.8.5 patches=f75251d72542ad46c9331c739f11f543c434f66950dc0c2303cdbdb7401263c5 arch=linux-centos7-x86_64
    ^autoconf@2.69%gcc@4.8.5 arch=linux-centos7-x86_64
        ^m4@1.4.18%gcc@4.8.5 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-centos7-x86_64
            ^libsigsegv@2.11%gcc@4.8.5 arch=linux-centos7-x86_64
        ^perl@5.26.2%gcc@4.8.5+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-centos7-x86_64
            ^gdbm@1.18.1%gcc@4.8.5 arch=linux-centos7-x86_64
                ^readline@7.0%gcc@4.8.5 arch=linux-centos7-x86_64
                    ^ncurses@6.1%gcc@4.8.5~symlinks~termlib arch=linux-centos7-x86_64
                        ^pkgconf@1.6.1%gcc@4.8.5 arch=linux-centos7-x86_64
    ^automake@1.16.1%gcc@4.8.5 arch=linux-centos7-x86_64
    ^gettext@0.19.8.1%gcc@4.8.5+bzip2+curses+git~libunistring+libxml2 patches=9acdb4e73f67c241b5ef32505c9ddf7cf6884ca8ea661692f21dca28483b04b8 +tar+xz arch=linux-centos7-x86_64
        ^bzip2@1.0.8%gcc@4.8.5+shared arch=linux-centos7-x86_64
            ^diffutils@3.7%gcc@4.8.5 arch=linux-centos7-x86_64
        ^libxml2@2.9.9%gcc@4.8.5~python arch=linux-centos7-x86_64
            ^libiconv@1.15%gcc@4.8.5 arch=linux-centos7-x86_64
            ^xz@5.2.4%gcc@4.8.5 arch=linux-centos7-x86_64
            ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
        ^tar@1.31%gcc@4.8.5 arch=linux-centos7-x86_64
    ^json-c@0.13.1%gcc@4.8.5 arch=linux-centos7-x86_64
    ^libtool@2.4.6%gcc@4.8.5 arch=linux-centos7-x86_64
    ^libuuid@1.0.3%gcc@4.8.5 arch=linux-centos7-x86_64
    ^lvm2@2.03.05%gcc@4.8.5+pkgconfig arch=linux-centos7-x86_64
        ^libaio@0.3.110%gcc@4.8.5 arch=linux-centos7-x86_64
    ^popt@1.16%gcc@4.8.5 arch=linux-centos7-x86_64
    ^util-linux@2.29.2%gcc@4.8.5 arch=linux-centos7-x86_64
        ^python@2.7.16%gcc@4.8.5+bz2+ctypes+dbm+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4~uuid+zlib arch=linux-centos7-x86_64
            ^expat@2.2.5%gcc@4.8.5+libbsd arch=linux-centos7-x86_64
                ^libbsd@0.9.1%gcc@4.8.5 arch=linux-centos7-x86_64
            ^libffi@3.2.1%gcc@4.8.5 arch=linux-centos7-x86_64
            ^openssl@1.1.1c%gcc@4.8.5+systemcerts arch=linux-centos7-x86_64
            ^sqlite@3.28.0%gcc@4.8.5~column_metadata+fts~functions~rtree arch=linux-centos7-x86_64
```
It appears that commenting out this **one line** in sqlite/package.py fixes cryptsetup:

```diff
diff --git a/var/spack/repos/builtin/packages/sqlite/package.py b/var/spack/repos/builtin/packages/sqlite/package.py
index 5bdff6524..65239ff43 100644
--- a/var/spack/repos/builtin/packages/sqlite/package.py
+++ b/var/spack/repos/builtin/packages/sqlite/package.py
@@ -34,7 +34,7 @@ class Sqlite(AutotoolsPackage):
             '(unsafe for <3.26.0.0 due to Magellan).')
 
     variant('rtree', default=False, description='Build with Rtree module')
-    variant('column_metadata', default=False, description="Build with COLUMN_METADATA")
+    #variant('column_metadata', default=False, description="Build with COLUMN_METADATA")
 
     # See https://blade.tencent.com/magellan/index_en.html
     conflicts('+fts', when='@:3.25.99.99')
```

I remain thoroughly confused!
I'm still confused, but I note that doing `spack spec -f` yields an interesting difference:   python and util-linux get rebuilt with a different hash -- although the spec appears to be identical.

The first file in the diff is the HEAD of develop, with broken cryptsetup.
The second file in the diff is with sqlite's `variant('column_metadata', ...` line commented out.
```diff
$ diff spack/cryptsetup.spec spack-fix/cryptsetup.spec 
7c7
< cxzzd3j  cryptsetup@2.2.1%gcc@4.8.5 patches=f75251d72542ad46c9331c739f11f543c434f66950dc0c2303cdbdb7401263c5 arch=linux-centos7-x86_64
---
> hvsni6g  cryptsetup@2.2.1%gcc@4.8.5 patches=f75251d72542ad46c9331c739f11f543c434f66950dc0c2303cdbdb7401263c5 arch=linux-centos7-x86_64
31,32c31,32
< cfuai4l      ^util-linux@2.29.2%gcc@4.8.5 arch=linux-centos7-x86_64
< fjyvsa3          ^python@2.7.16%gcc@4.8.5+bz2+ctypes+dbm+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4~uuid+zlib arch=linux-centos7-x86_64
---
> 5hhcjnl      ^util-linux@2.29.2%gcc@4.8.5 arch=linux-centos7-x86_64
> b2t7osf          ^python@2.7.16%gcc@4.8.5+bz2+ctypes+dbm+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4~uuid+zlib arch=linux-centos7-x86_64
37c37
< evahqif              ^sqlite@3.28.0%gcc@4.8.5~column_metadata+fts~functions~rtree arch=linux-centos7-x86_64
---
> alqtjlt              ^sqlite@3.28.0%gcc@4.8.5+fts~functions~rtree arch=linux-centos7-x86_64
```
>   `>> 144    configure: error: You need the uuid library.`

If Spack is failing to find this library (even given that it is listed as a dependency) then I'm wondering if this is an instance of https://github.com/spack/spack/issues/12709: can you try manually setting the `implicit_rpaths` entry for `gcc@4.8.5` in your `compilers.yaml` or try out the bugfix branch at https://github.com/spack/spack/pull/12789?

> I'm still confused, but I note that doing spack spec -f yields an interesting difference: python and util-linux get rebuilt with a different hash -- although the spec appears to be identical.

The hash of any package depends on the hash of all of its dependencies. Since `python` depends on `sqlite` and `util-linux` depends on `python`, all three hashes will change when you add a variant to `sqlite`.
> all three hashes will change when you add a variant to sqlite.

Thanks, that much makes sense.

> try out the bugfix branch at #12789?

I grabbed that branch and merged it into the HEAD of develop (in my fork), and tried the build install again -- but still get the same error.

> If Spack is failing to find this library

But the weirdness of this is that it is *not* failing to find it, when this line is removed from `sqlite/package.py`:
```
variant('column_metadata', default=False, description="Build with COLUMN_METADATA")
```

It seems like there is something fundamentally wrong, if the addition of a variant that defaults to `False` and *should* not trigger any if tests, can cause a change in the some other package.  

Just to be pedantic, I changed the line to:
```
variant('foobats', default=False, description='nothing but foobats')
```
and (of course) I still get the cryptsetup error.  *But it works without this line!*  That's weird, right??

@hartzell Am I doing something wrong?  Could you try to replicate my observation?

(By the way, just to be sure it wasn't some kind of env contamination, I've repeated all my tests after doing `rm -rf /tmp/spack* ~/.spack*` and an `env -i bash` to get a clean environment.)
@odoublewen -- running a build now....
I also see the failure and can replicate by going into the build dir after the failure, starting a fresh shell, `. ../spack-build-env.txt`, then `./configure`

It's finding libuuid ok (scan through the config output) but blows up when it tries this:

```
checking for uuid_clear in -luuid... no
```

Opening the config.log, searching down for clear and reading a bit and :scream:, :scream:, :scream:.  Hate computers....

It's blowing up when trying to link it's little test program **because it needs `-lintl`**.

```
[...]
configure:13863: result: no
configure:13894: checking for uuid_clear in -luuid
configure:13919: /home/ghartzell/owen/spack/lib/spack/env/gcc/gcc -o conftest -g -O2   conftest.c -luuid   >&5
/home/ghartzell/owen/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/util-linux-2.29.2-daxzq7g4g5xgqc75rsbcgijfjiafu4al/lib/libuuid.so: undefined reference to `libintl_gettext'
collect2: error: ld returned 1 exit status
configure:13919: $? = 1
configure: failed program was:
[...]
```
#$#@%$#%@$%@$#%!!!!!

Gotta run in a bit, but can poke about to figure out the least smelly way to get that to happen.  Perhaps I can just shove a `LIBS=-lintl` somewhere... :confused:  But seriously, I think that something will need to be done for/to the `libuuid` package.
Thanks... But, also, can you confirm my observation that the cryptsetup build works when [this line](https://github.com/Sinan81/spack/blob/931d65cc14dc5ac80c405acc165d9ecc2bd9f169/var/spack/repos/builtin/packages/sqlite/package.py#L37) is commented out?  That's what is making me think I am crazy.

Does commenting out that line rescue cryptsetup for you too?  If so, why??? 😱
You don't appear to be crazy, it works for me with that line commented out also.  I'll see if I can make up^H^H^H^H^H^Hfigure out an explanation over the weekend....
FWIW, with that line commented out, compiling the test program succeeds.  Doesn't speak to *why* though....

```
configure:13894: checking for uuid_clear in -luuid
configure:13919: /home/ghartzell/owen/spack/lib/spack/env/gcc/gcc -o conftest -g -O2   conftest.c -luuid   >&5
configure:13919: $? = 0
configure:13928: result: yes
```
Here's my hunch.  I'm wondering what determines the sort order of the various PATHs that get set up in the build env.  I'm thinking the order is semi-deterministic.  It's reliable with the same spec-hash, but change one hash in the deps and the order might be different. 

I have no data to back this up, just a thought that occurred as I was brushing my teeth.
I had a similar thought, that the presence of the variant, even if it's `False`, in that file changes the ordering of various libraries.

But I can't explain how `/home/ghartzell/owen/spack/lib/spack/env/gcc/gcc -o conftest -g -O2   conftest.c -luuid`, which doesn't obviously involve any other libraries, sometimes turns up a missing symbol and sometimes doesn't.

When I changed the sqlite package definition, sqlite, python, util-linux, and cryptsetup were rebuilt.  None of them are blatantly involved in linking that simple little test program.

I don't understand the details of the change that @scheibelp refered to.
Well, nuts.

I just logged back in and reran `(module purge; spack install cryptsetup)` and it worked.  Also tried a simple `spack install cryptsetup` (leaving various bits in from an upstream repo in my environment) and it also built.  But, looks like the `module purge` isn't enough to get the spack-built compiler out of my environment, presumably it's picking it up from one of my compilers.yaml files.
Try as I may, I can't get it to fail again.  I've removed everything and am rebuilding from scratch.
Just rebuilt from scratch, successfully.   Haven't changed anything (no pulls, no edits, no etc...).

:confused:
Well, I'm glad at least that you were able to replicate (1) the  uuid link error and (2) the lack thereof when twiddling that totally unrelated line in the sqlite dep.  

Yesterday *was* Friday the 13th + full moon. 🌕 

Seems more likely that there is some underlying fragility with the way uuid (and other?) libs are being autoconfig'ed, and as long as it's not a full moon with a hash that ends with a prime number, we're fine 😕 

My nightly spack-tree build Jenkins job has died on the cryptsetup for the past two nights.  We'll see what happens tonight.   And, also, I am, again, retrying the build from a clean slate, will update later today:

```console
$ rm -rf spack ~/.spack /tmp/spack*
$ git clone git@github.com:spack/spack.git
$ cd spack
$ env -i bash
$ ./bin/spack install cryptsetup
```
```console
==> Installing cryptsetup
==> Searching for binary cache of cryptsetup
==> Warning: No Spack mirrors are currently configured
==> No binary for cryptsetup found: installing from source
==> Fetching https://www.kernel.org/pub/linux/utils/cryptsetup/v2.2/cryptsetup-2.2.1.tar.xz
######################################################################## 100.0%
==> Staging archive: /tmp/osolberg/spack-stage/cryptsetup-2.2.1-cxzzd3jsk6zrzo4tjz7etec7jbbxk43l/cryptsetup-2.2.1.tar.xz
==> Created stage in /tmp/osolberg/spack-stage/cryptsetup-2.2.1-cxzzd3jsk6zrzo4tjz7etec7jbbxk43l
==> Applied patch /home/osolberg/temp/spack/var/spack/repos/builtin/packages/cryptsetup/autotools-libintl.patch
==> Building cryptsetup [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Error: ProcessError: Command exited with status 1:
    '/tmp/osolberg/spack-stage/cryptsetup-2.2.1-cxzzd3jsk6zrzo4tjz7etec7jbbxk43l/spack-src/configure' '--prefix=/home/osolberg/temp/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/cryptsetup-2.2.1-cxzzd3jsk6zrzo4tjz7etec7jbbxk43l' 'systemd_tmpfilesdir=/home/osolberg/temp/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/cryptsetup-2.2.1-cxzzd3jsk6zrzo4tjz7etec7jbbxk43l/tmpfiles.d'

1 error found in build log:
     138    checking for linux/keyctl.h... yes
     139    checking whether __NR_add_key is declared... yes
     140    checking whether __NR_keyctl is declared... yes
     141    checking whether __NR_request_key is declared... yes
     142    checking for key_serial_t... no
     143    checking for uuid_clear in -luuid... no
  >> 144    configure: error: You need the uuid library.
```
😢 😢 😢 
More coffee, more clarity.  The reason that my builds were succeeding was that I'd left the sqlite variant commented out.  I just reinstated it and builds are failing wonderfully.
Ah ha-ish.  The test for `uuid_clear` is linking against a `libuuid.so` that's included in the `util-linux` package, **not** the one from the `uuid` package.

```
configure:13894: checking for uuid_clear in -luuid
configure:13919: /home/ghartzell/owen/spack/lib/spack/env/gcc/gcc -o conftest -g -O2   conftest.c -luuid   >&5
/home/ghartzell/owen/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/util-linux-2.29.2-daxzq7g4g5xgqc75rsbcgijfjiafu4al/lib/libuuid.so: undefined reference to `libintl_gettext'
collect2: error: ld returned 1 exit status
```

At a guess, the times that it works it's picking up the `libuuid` package and having happy linkage.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/cryptsetup/package.py b/var/spack/repos/builtin/packages/cryptsetup/package.py
--- a/var/spack/repos/builtin/packages/cryptsetup/package.py
+++ b/var/spack/repos/builtin/packages/cryptsetup/package.py
@@ -20,7 +20,7 @@ class Cryptsetup(AutotoolsPackage):
     depends_on('lvm2', type=('build', 'link'))
     depends_on('popt', type=('build', 'link'))
     depends_on('json-c', type=('build', 'link'))
-    depends_on('util-linux', type=('build', 'link'))
+    depends_on('util-linux~libuuid', type=('build', 'link'))
     depends_on('gettext', type=('build', 'link'))
 
     depends_on('autoconf', type='build')
diff --git a/var/spack/repos/builtin/packages/util-linux/package.py b/var/spack/repos/builtin/packages/util-linux/package.py
--- a/var/spack/repos/builtin/packages/util-linux/package.py
+++ b/var/spack/repos/builtin/packages/util-linux/package.py
@@ -21,9 +21,17 @@ class UtilLinux(AutotoolsPackage):
     depends_on('python@2.7:')
     depends_on('pkgconfig')
 
+    # Make it possible to disable util-linux's libuuid so that you may
+    # reliably depend_on(`libuuid`).
+    variant('libuuid', default=True, description='Build libuuid')
+
     def url_for_version(self, version):
         url = "https://www.kernel.org/pub/linux/utils/util-linux/v{0}/util-linux-{1}.tar.gz"
         return url.format(version.up_to(2), version)
 
     def configure_args(self):
-        return ['--disable-use-tty-group']
+        config_args = [
+            '--disable-use-tty-group',
+        ]
+        config_args.extend(self.enable_or_disable('libuuid'))
+        return config_args


Test patch: