Repository: spack/spack

Issue:
Try to make `ascent+conduit+python` compile properly
I have been trying to compile `ascent+python` with CUDA on SUMMIT, and have encountered a whole bunch of issues. This fixes some but still does not compile properly.... It's a part-PR, part-bug report issue.

### Command used

`spack install ascent~test+python+cuda~shared ^conduit~hdf5_compat`

The `hdf5_compat` is explained below.

### Issue breakdown

- HDF5 version: doing `spack install ascent+python` uses conduit, which by default has `hdf5_compat` enabled. I get this error:

```
     1630    /sw/summit/gcc/7.4.0/bin/g++ -O2 -g -DNDEBUG -rdynamic -pthread CM
             akeFiles/conduit_blueprint_verify.dir/conduit_blueprint_verify_exe
             .cpp.o -o conduit_blueprint_verify  -Wl,-rpath,/tmp/potterg/spack-
             stage/spack-stage-conduit-0.7.2-cqfd7knohz2u3dymmfskceclnenqocdx/s
             pack-build-cqfd7kn/lib:/autofs/nccs-svm1_home1/potterg/spack/opt/s
             pack/linux-rhel7-power9le/gcc-7.4.0/hdf5-1.8.22-w3vpzki7mkj224lsga
             6lhn43pexori77/lib:::::::::::::::::::::: ../../lib/libconduit_rela
             y.so ../../lib/libconduit_blueprint.so ../../lib/libconduit.so -lr
             t -lpthread /autofs/nccs-svm1_home1/potterg/spack/opt/spack/linux-
             rhel7-power9le/gcc-7.4.0/hdf5-1.8.22-w3vpzki7mkj224lsga6lhn43pexor
             i77/lib/libhdf5.so.10.4.0 -ldl -lmpiprofilesupport -lmpi_ibm
  >> 1631    ../../lib/libconduit_relay.so: undefined reference to `H5Oget_info
             1'
  >> 1632    ../../lib/libconduit_relay.so: undefined reference to `H5Oget_info
             _by_name1'
```

where conduit tries to use a function that isn't available below hdf5 1.10.2 but spack installs hdf5 1.8. I am not sure why this option is here, but I'm now using `^conduit~hdf5_compat` in all further commands. This is not fixed here

- Moving on, I now get the following

```
     650    /sw/summit/gcc/7.4.0/bin/g++ -O2 -g -DNDEBUG -rdynamic CMakeFiles/t
            _rapidjson_smoke.dir/t_rapidjson_smoke.cpp.o -o t_rapidjson_smoke
            ../../lib/libgtest_main.a ../../lib/libgtest.a -lpthread -lpthread
     651    make[2]: Leaving directory `/tmp/potterg/spack-stage/spack-stage-co
            nduit-0.7.2-vv222qykcw3fxlrkndwfyspbtqgdmaew/spack-build-vv222qy'
     652    make[2]: Leaving directory `/tmp/potterg/spack-stage/spack-stage-co
            nduit-0.7.2-vv222qykcw3fxlrkndwfyspbtqgdmaew/spack-build-vv222qy'
     653    [ 21%] Built target t_civetweb_smoke
     654    [ 21%] Built target blt_gtest_smoke
  >> 655    /usr/bin/ld: /autofs/nccs-svm1_home1/potterg/spack/opt/spack/linux-
            rhel7-power9le/gcc-7.4.0/hdf5-1.10.7-dtx3nzujzgg5pavhkslezvjekmppxx
            5v/lib/libhdf5.a(H5PLint.c.o): undefined reference to symbol 'dlsym
            @@@@GLIBC_2.17'
  >> 656    //usr/lib64/libdl.so.2: error adding symbols: DSO missing from comm
            and line
  >> 657    collect2: error: ld returned 1 exit status
  >> 658    make[2]: *** [tests/thirdparty/t_hdf5_smoke] Error 1
```

This is exactly https://github.com/spack/spack/pull/7385 so I used mostly the same fix.

- Moving on, I get the following

```
     656    [ 21%] Built target t_libb64_smoke
     657    /autofs/nccs-svm1_home1/potterg/spack/opt/spack/linux-rhel7-power9l
            e/gcc-7.4.0/hdf5-1.10.7-dtx3nzujzgg5pavhkslezvjekmppxx5v/lib/libhdf
            5.a(H5Zdeflate.c.o): In function `H5Z__filter_deflate':
  >> 658    /tmp/potterg/spack-stage/spack-stage-hdf5-1.10.7-dtx3nzujzgg5pavhks
            lezvjekmppxx5v/spack-src/src/H5Zdeflate.c:106: undefined reference
            to `inflateInit_'
```

So let's add a `-lz` in there. At this point, `conduit` compiles

- Moving on, I get the following

```
==> Error: NameError: global name 'site_packages_dir' is not defined

/autofs/nccs-svm1_home1/potterg/spack/var/spack/repos/builtin/packages/ascent/package.py:217, in install:
        214            print("Installing Ascent...")
        215            make("install")
        216            # install copy of host config for provenance
  >>    217            install(host_cfg_fname, prefix)
```
I re-used https://github.com/spack/spack/pull/23156 but it's probably not the right fix...?

- I'm currently stuck with a `uses too much data for compiler-generated constants; please recompile with -Xptxas --disable-optimizer-constants` error.

<details>

<summary>Full error</summary>

```
  >> 595    make[2]: *** [lib/libascent_mpi.a] Error 255
     596    make[2]: Leaving directory `/tmp/potterg/spack-stage/spack-stage-as
            cent-0.7.1-lsiqyvxlznnoza2gncv2humxr4c77lts/spack-src/spack-build'
  >> 597    make[1]: *** [ascent/CMakeFiles/ascent_mpi.dir/all] Error 2
     598    make[1]: *** Waiting for unfinished jobs....
     599    [ 56%] Linking CXX static library ../lib/libascent.a
     600    cd /tmp/potterg/spack-stage/spack-stage-ascent-0.7.1-lsiqyvxlznnoza
            2gncv2humxr4c77lts/spack-src/spack-build/ascent && /autofs/nccs-svm
            1_home1/potterg/spack/opt/spack/linux-rhel7-power9le/gcc-7.4.0/cmak
            e-3.20.5-fpqhvbibm5p25btq7jvz5h3f2wcsvqeq/bin/cmake -P CMakeFiles/a
            scent.dir/cmake_clean_target.cmake
     601    manual device link step for Ascent
     602    cd /tmp/potterg/spack-stage/spack-stage-ascent-0.7.1-lsiqyvxlznnoza
            2gncv2humxr4c77lts/spack-src/spack-build/ascent && /sw/summit/cuda/
            11.4.0/bin/nvcc --device-link -Xnvlink=--suppress-stack-size-warnin
            g --generate-code=arch=compute_70,code=sm_70 /tmp/potterg/spack-sta
            ge/spack-stage-ascent-0.7.1-lsiqyvxlznnoza2gncv2humxr4c77lts/spack-
            src/spack-build/ascent/CMakeFiles/ascent.dir/runtimes/ascent_data_o
            bject.cpp.o /tmp/potterg/spack-stage/spack-stage-ascent-0.7.1-lsiqy
            vxlznnoza2gncv2humxr4c77lts/spack-src/spack-build/ascent/CMakeFiles
            /ascent.dir/runtimes/ascent_vtkh_data_adapter.cpp.o /tmp/potterg/sp
            ack-stage/spack-stage-ascent-0.7.1-lsiqyvxlznnoza2gncv2humxr4c77lts
            /spack-src/spack-build/ascent/CMakeFiles/ascent.dir/runtimes/ascent
            _vtkh_collection.cpp.o /tmp/potterg/spack-stage/spack-stage-ascent-
            0.7.1-lsiqyvxlznnoza2gncv2humxr4c77lts/spack-src/spack-build/ascent
            /CMakeFiles/ascent.dir/runtimes/flow_filters/ascent_runtime_vtkh_fi
            lters.cpp.o /tmp/potterg/spack-stage/spack-stage-ascent-0.7.1-lsiqy
            vxlznnoza2gncv2humxr4c77lts/spack-src/spack-build/ascent/CMakeFiles
            /ascent.dir/runtimes/flow_filters/ascent_runtime_vtkh_utils.cpp.o /
            tmp/potterg/spack-stage/spack-stage-ascent-0.7.1-lsiqyvxlznnoza2gnc
            v2humxr4c77lts/spack-src/spack-build/ascent/CMakeFiles/ascent.dir/r
            untimes/flow_filters/ascent_runtime_rendering_filters.cpp.o /tmp/po
            tterg/spack-stage/spack-stage-ascent-0.7.1-lsiqyvxlznnoza2gncv2humx
            r4c77lts/spack-src/spack-build/ascent/CMakeFiles/ascent.dir/runtime
            s/flow_filters/ascent_runtime_rover_filters.cpp.o /tmp/potterg/spac
            k-stage/spack-stage-ascent-0.7.1-lsiqyvxlznnoza2gncv2humxr4c77lts/s
            pack-src/spack-build/ascent/CMakeFiles/ascent.dir/runtimes/flow_fil
            ters/ascent_runtime_conduit_to_vtkm_parsing.cpp.o /tmp/potterg/spac
            k-stage/spack-stage-ascent-0.7.1-lsiqyvxlznnoza2gncv2humxr4c77lts/s
            pack-src/spack-build/ascent/CMakeFiles/ascent.dir/runtimes/ascent_m
            ain_runtime.cpp.o /tmp/potterg/spack-stage/spack-stage-ascent-0.7.1
            -lsiqyvxlznnoza2gncv2humxr4c77lts/spack-src/spack-build/ascent/CMak
            eFiles/ascent.dir/runtimes/flow_filters/ascent_runtime_blueprint_fi
            lters.cpp.o /tmp/potterg/spack-stage/spack-stage-ascent-0.7.1-lsiqy
            vxlznnoza2gncv2humxr4c77lts/spack-src/spack-build/lib/librover.a /a
            utofs/nccs-svm1_home1/potterg/spack/opt/spack/linux-rhel7-power9le/
            gcc-7.4.0/vtk-h-0.7.1-5vxkfcn7ukur47iyv27bpombjbbkhzsl/lib/libvtkh_
            rendering.a /autofs/nccs-svm1_home1/potterg/spack/opt/spack/linux-r
            hel7-power9le/gcc-7.4.0/vtk-h-0.7.1-5vxkfcn7ukur47iyv27bpombjbbkhzs
            l/lib/libvtkh_compositing.a /autofs/nccs-svm1_home1/potterg/spack/o
            pt/spack/linux-rhel7-power9le/gcc-7.4.0/vtk-h-0.7.1-5vxkfcn7ukur47i
            yv27bpombjbbkhzsl/lib/libvtkh_filters.a /autofs/nccs-svm1_home1/pot
            terg/spack/opt/spack/linux-rhel7-power9le/gcc-7.4.0/vtk-h-0.7.1-5vx
            kfcn7ukur47iyv27bpombjbbkhzsl/lib/libvtkh_core.a /autofs/nccs-svm1_
            home1/potterg/spack/opt/spack/linux-rhel7-power9le/gcc-7.4.0/vtk-h-
            0.7.1-5vxkfcn7ukur47iyv27bpombjbbkhzsl/lib/libvtkh_lodepng.a /autof
            s/nccs-svm1_home1/potterg/spack/opt/spack/linux-rhel7-power9le/gcc-
            7.4.0/vtk-h-0.7.1-5vxkfcn7ukur47iyv27bpombjbbkhzsl/lib/libvtkh_util
            s.a /autofs/nccs-svm1_home1/potterg/spack/opt/spack/linux-rhel7-pow
            er9le/gcc-7.4.0/vtk-m-1.6.0-nnolfl66fv3znuummzweoiiattimnee2/lib/li
            bvtkm_io-1.6.a /autofs/nccs-svm1_home1/potterg/spack/opt/spack/linu
            x-rhel7-power9le/gcc-7.4.0/vtk-m-1.6.0-nnolfl66fv3znuummzweoiiattim
            nee2/lib/libvtkm_rendering-1.6.a /autofs/nccs-svm1_home1/potterg/sp
            ack/opt/spack/linux-rhel7-power9le/gcc-7.4.0/vtk-m-1.6.0-nnolfl66fv
            3znuummzweoiiattimnee2/lib/libvtkm_filter_common-1.6.a /autofs/nccs
            -svm1_home1/potterg/spack/opt/spack/linux-rhel7-power9le/gcc-7.4.0/
            vtk-m-1.6.0-nnolfl66fv3znuummzweoiiattimnee2/lib/libvtkm_filter_con
            tour-1.6.a /autofs/nccs-svm1_home1/potterg/spack/opt/spack/linux-rh
            el7-power9le/gcc-7.4.0/vtk-m-1.6.0-nnolfl66fv3znuummzweoiiattimnee2
            /lib/libvtkm_filter_gradient-1.6.a /autofs/nccs-svm1_home1/potterg/
            spack/opt/spack/linux-rhel7-power9le/gcc-7.4.0/vtk-m-1.6.0-nnolfl66
            fv3znuummzweoiiattimnee2/lib/libvtkm_filter_extra-1.6.a /autofs/ncc
            s-svm1_home1/potterg/spack/opt/spack/linux-rhel7-power9le/gcc-7.4.0
            /vtk-m-1.6.0-nnolfl66fv3znuummzweoiiattimnee2/lib/libvtkm_worklet-1
            .6.a /autofs/nccs-svm1_home1/potterg/spack/opt/spack/linux-rhel7-po
            wer9le/gcc-7.4.0/vtk-m-1.6.0-nnolfl66fv3znuummzweoiiattimnee2/lib/l
            ibvtkm_source-1.6.a /autofs/nccs-svm1_home1/potterg/spack/opt/spack
            /linux-rhel7-power9le/gcc-7.4.0/vtk-m-1.6.0-nnolfl66fv3znuummzweoii
            attimnee2/lib/libvtkm_cont-1.6.a /autofs/nccs-svm1_home1/potterg/sp
            ack/opt/spack/linux-rhel7-power9le/gcc-7.4.0/vtk-m-1.6.0-nnolfl66fv
            3znuummzweoiiattimnee2/lib/libvtkmdiympi_nompi.a -lcudadevrt -lcuda
            rt_static --output-file bcal_vtkm.o
     603    nvlink error   : Entry function '_ZN4vtkm4cont4cuda8internal19TaskS
            trided1DLaunchINS_4exec4cuda8internal13TaskStrided1DIKNS_7worklet17
            particleadvection21ParticleAdvectWorkletEKNS_8internal10InvocationI
            NSC_17FunctionInterfaceIFvNSC_19ArrayPortalImplicitINSC_12IndexFunc
            torEEEPKNS9_14IntegratorBase10ExecObjectENS9_23ParticleExecutionObj
            ectINS_8ParticleEEENSF_INS0_8internal15ConstantFunctorIiEEEEEEENSE_
            IFvNS8_15WorkletMapField7FieldInENS8_8internal11WorkletBase10ExecOb
            jectESZ_SW_EEENSE_IFvNS_12placeholders3ArgILi1EEENS13_ILi2EEENS13_I
            Li3EEENS13_ILi4EEEEEELi1ESH_SS_SH_NS0_20DeviceAdapterTagCudaEEEEEEE
            vT_i' uses too much data for compiler-generated constants; please r
            ecompile with -Xptxas --disable-optimizer-constants
  >> 604    make[2]: *** [lib/libascent.a] Error 255
     605    make[2]: Leaving directory `/tmp/potterg/spack-stage/spack-stage-as
            cent-0.7.1-lsiqyvxlznnoza2gncv2humxr4c77lts/spack-src/spack-build'
  >> 606    make[1]: *** [ascent/CMakeFiles/ascent.dir/all] Error 2
     607    make[1]: Leaving directory `/tmp/potterg/spack-stage/spack-stage-as
            cent-0.7.1-lsiqyvxlznnoza2gncv2humxr4c77lts/spack-src/spack-build'
  >> 608    make: *** [all] Error 2
```

</details>

The only similar issue I could find was https://github.com/ECP-WarpX/WarpX/issues/2004 but because it said not to follow the recommanded compiler I don't know how to go further.

### Additional infos

* **Spack:** 0.16.2-3697-258f757
* **Python:** 2.7.5
* **Platform:** linux-rhel7-power9le
* **Concretizer:** original
* I'm using `CUDA 11.4.0`

Thanks a lot for some feedback or help


Discussion:
thanks @gpotter2 

For the HDF5 zlib issue -- I have seen this before. For some builds hdf5 doens't report all of its dependent libs (in our case with our cmake logic). `-lz` is likely wrong here, it should actually be explicitly linked to a spack built zlib.  It might very well be that hdf5 did link against a system zlib, I'll try to understand this better. 

I'll look into fixin the `-ldl` issue. 

For the cuda linking issue: We have seen this and we haven't yet moved to use cuda 11, I will try to find a path forward. 

Finally, Ascent doesn't yet support compiling against a statically built python.  Our use of cuda requires the ascent lib be statically linked. It might possible to have python be a shared lib, but we haven't paved that path yet. 

Thanks a lot for looking into this @cyrush !
I'll retry using CUDA 10.

Edit: with an additional `-lz`, it works. It sounds likely that it hook to a system zlib. I'm using this profile on summit:
[warpx.profile.txt](https://github.com/spack/spack/files/6957803/warpx.profile.txt)

@spackbot run pipeline
I've started that [pipeline](https://gitlab.spack.io//spack/spack/-/pipelines/56686) for you!
For this case, I am working on seeing if we can build with shared python + static most everything else. This will involve relaxing some of the constraints in the ascent package. 

@gpotter2  is spack using the same version of cmake, etc?  Do you have a spack env, compilers.yaml + packages.yaml to share?
I only use spack for this intent so I did not setup a spack environment. The `compilers.yaml` is what spack auto-detected.

[compilers.yaml.txt](https://github.com/spack/spack/files/6964957/compilers.yaml.txt)
[packages.yaml.txt](https://github.com/spack/spack/files/6964958/packages.yaml.txt)

```
$ spack find
==> 61 installed packages
-- linux-rhel7-power9le / gcc@7.4.0 -----------------------------
adios@1.13.1         libmd@1.0.3           py-setuptools@57.4.0
ascent@develop       libpng@1.6.37         python@3.8.11
autoconf@2.69        libsigsegv@2.13       readline@8.1
automake@1.16.3      libtool@2.4.6         snappy@1.1.8
berkeley-db@18.1.40  libxml2@2.9.10        spectrum-mpi@10.3.1.2-20200121
boost@1.76.0         lz4@1.9.3             spectrum-mpi@10.3.1.2-20200121
bzip2@1.0.8          m4@1.4.19             spectrum-mpi@10.3.1.2-20200121
c-blosc@1.21.0       mpark-variant@1.4.0   sqlite@3.35.5
cmake@3.21.1         ncurses@6.2           sz@1.4.12.3
conduit@0.7.2        nlohmann-json@3.9.1   tar@1.34
cuda@10.1.243        openblas@0.3.17       util-linux-uuid@2.36.2
cuda@11.4.0          openssl@1.1.1k        vtk-h@0.7.1
diffutils@3.7        perl@5.34.0           vtk-h@0.7.1
expat@2.4.1          pkgconf@1.7.4         vtk-m@1.6.0
gdbm@1.19            py-cython@0.29.23     vtk-m@1.6.0
gettext@0.21         py-cython@0.29.23     xz@5.2.5
hdf5@1.10.7          py-mpi4py@3.0.3       zfp@0.5.5
libbsd@0.11.3        py-numpy@1.19.5       zlib@1.2.11
libfabric@1.12.1     py-pip@21.1.2         zstd@1.5.0
libffi@3.3           py-pip@21.1.2
libiconv@1.16        py-setuptools@50.3.2
```
I'll like to try to understand the `-lz` issue a bit more. 

Adding a `-lz` scares me b/c it's going to pick up a system version and cause sadness in the future. 

Here is my guess: 
I suspect when hdf5 is built as static, there is some assumption (in its built system or elsewhere) that libz is also static, leading hdf5 to not report it as a dependency. Since it does't report it, we don't detect in downstream cmake logic.  However, in this case -- libz is actually shared. 

This is just a guess at this point. 



I found the source of the missing `-ldl`, and maybe the `-lz` as well

When hdf5 is built with cmake (recent change in spack), the hdf5 settings file is reporting the libs using a cmake separator `;`

```
Extra libraries: m;dl
```

As we parse this, we get `-lmdl` which is wrong.

Here is what it looks like when hdf5 is not built with cmake:
```
 Extra libraries: -lz -ldl -lm 
```
(yes, the fact that `-lz` is in there for some cases does mean the system linking case could already be haunting us)

I'll fix this in conduit@develop. 

@gpotter2  In conduit, I merged: https://github.com/LLNL/conduit/pull/816 -- which should fix the linking issues you saw w/o any changes to the spack recipe.  If possible can you see if building with `conduit@develop` fixes things? 

We still need to fix the ascent py site packages issue, I can create a PR for that. 


> @gpotter2  In conduit, I merged: https://github.com/LLNL/conduit/pull/816 -- which should fix the linking issues you saw w/o any changes to the spack recipe.  If possible can you see if building with `conduit@develop` fixes things? 

It seems to work !

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/ascent/package.py b/var/spack/repos/builtin/packages/ascent/package.py
--- a/var/spack/repos/builtin/packages/ascent/package.py
+++ b/var/spack/repos/builtin/packages/ascent/package.py
@@ -93,27 +93,25 @@ class Ascent(Package, CudaPackage):
     # Certain CMake versions have been found to break for our use cases
     depends_on("cmake@3.14.1:3.14.99,3.18.2:", type='build')
     depends_on("conduit~python", when="~python")
-    depends_on("conduit+python", when="+python+shared")
-    depends_on("conduit~shared~python", when="~shared")
-    depends_on("conduit~python~mpi", when="~python~mpi")
-    depends_on("conduit+python~mpi", when="+python+shared~mpi")
-    depends_on("conduit~shared~python~mpi", when="~shared~mpi")
+    depends_on("conduit+python", when="+python")
+    depends_on("conduit+mpi", when="+mpi")
+    depends_on("conduit~mpi", when="~mpi")
 
     #######################
     # Python
     #######################
     # we need a shared version of python b/c linking with static python lib
     # causes duplicate state issues when running compiled python modules.
-    depends_on("python+shared", when="+python+shared")
-    extends("python", when="+python+shared")
-    depends_on("py-numpy", when="+python+shared", type=('build', 'run'))
-    depends_on("py-pip", when="+python+shared", type=('build', 'run'))
+    depends_on("python+shared", when="+python")
+    extends("python", when="+python")
+    depends_on("py-numpy", when="+python", type=('build', 'run'))
+    depends_on("py-pip", when="+python", type=('build', 'run'))
 
     #######################
     # MPI
     #######################
     depends_on("mpi", when="+mpi")
-    depends_on("py-mpi4py", when="+mpi+python+shared")
+    depends_on("py-mpi4py", when="+mpi+python")
 
     #######################
     # BabelFlow
@@ -185,7 +183,11 @@ def install(self, spec, prefix):
         with working_dir('spack-build', create=True):
             py_site_pkgs_dir = None
             if "+python" in spec:
-                py_site_pkgs_dir = site_packages_dir
+                try:
+                    py_site_pkgs_dir = site_packages_dir
+                except NameError:
+                    # spack's site_packages_dir won't exist in a subclass
+                    pass
 
             host_cfg_fname = self.create_host_config(spec,
                                                      prefix,


Test patch: