Repository: spack/spack

Issue:
Installation issue: Trilinos does not have variant `Trilinos_ENABLE_THREAD_SAFE`
### Steps to reproduce the issue

Trilinos can be built to be thread safe with `-DTrilinos_ENABLE_THREAD_SAFE`. However, there is no variant that enables this.

https://docs.trilinos.org/files/TrilinosBuildReference.html#enabling-disabling-thread-safety

### Error message

Code that uses TPetra has various memory error due to it not being thread safe, e.g.,
```
Kokkos::Impl::SharedAllocationRecord 'Tpetra::CrsGraph::ptr' failed decrement count = 
```


### Information on your system

```
spack debug report
* **Spack:** 0.22.0.dev0 (23eec9d0378c818e73426bef68c3571932ad84a8)
* **Python:** 3.6.8
* **Platform:** linux-rhel8-icelake
* **Concretizer:** clingo
```

### Additional information

@jwillenbring  @keitat  @kuberry  @psakievich  @sethrj


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have run `spack maintainers <name-of-the-package>` and **@mentioned** any maintainers
- [X] I have uploaded the build log and environment files
- [X] I have searched the issues of this repo and believe this is not a duplicate


Discussion:
I have tried to add this myself to the existing cmake flags here
https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/trilinos/package.py#L618 
like this
```
        options.extend(
            [
                define("Trilinos_VERBOSE_CONFIGURE", False),
                define("Trilinos_ENABLE_THREAD_SAFE", True),

```
However, my code that uses TPetra+kokkos continues to not worked as expected. However: this code works without issue with Trilinos 13,14, and 15 when setting `OMP_NUM_THREADS=1`. Further, on another HPC that builds Trilinos via `easy_build`, that flag works as expected and produces a thread-safe Trilinos.
https://github.com/Chrismarsh/easy_build/blob/master/trilinos-14.4.0-gompi-2020a.eb#L42

it isn't clear to me why this isn't working.


The configure args from the stage dir suggests it's being passed through correctly:
```
-DTrilinos_VERBOSE_CONFIGURE:BOOL=OFF -DTrilinos_ENABLE_THREAD_SAFE:BOOL=ON -DBUILD_SHARED_LIBS:BOOL=ON -DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF -DTrilinos_ENABLE_ALL_PACKAGES:BOOL=OFF -DTrilinos_ENABLE_CXX11:BOOL=ON -DTrilinos_ENABLE_DEBUG:BOOL=OFF -DTrilinos_ENABLE_EXAMPLES:BOOL=OFF -DTrilinos_ENABLE_SECONDARY_TESTED_CODE:BOOL=ON -DTrilinos_ENABLE_TESTS:BOOL=OFF -DTrilinos_ENABLE_Fortran:BOOL=ON -DTrilinos_ENABLE_OpenMP:BOOL=OFF -DTrilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON -DTrilinos_ENABLE_PyTrilinos2:BOOL=OFF -DTrilinos_ENABLE_TESTS:BOOL=OFF -DCMAKE_CXX_STANDARD:STRING=17 -DTrilinos_ENABLE_Adelus:BOOL=OFF -DTrilinos_ENABLE_Amesos:BOOL=ON -DTrilinos_ENABLE_Amesos2:BOOL=ON -DTrilinos_ENABLE_Anasazi:BOOL=ON -DTrilinos_ENABLE_AztecOO:BOOL=ON -DTrilinos_ENABLE_Belos:BOOL=ON -DTrilinos_ENABLE_Epetra:BOOL=ON -DTrilinos_ENABLE_EpetraExt:BOOL=ON -DTrilinos_ENABLE_FEI:BOOL=OFF -DTrilinos_ENABLE_Gtest:BOOL=OFF -DTrilinos_ENABLE_Ifpack:BOOL=ON -DTrilinos_ENABLE_Ifpack2:BOOL=ON -DTrilinos_ENABLE_Intrepid:BOOL=OFF -DTrilinos_ENABLE_Intrepid2:BOOL=OFF -DTrilinos_ENABLE_Isorropia:BOOL=OFF -DTrilinos_ENABLE_Kokkos:BOOL=ON -DTrilinos_ENABLE_MiniTensor:BOOL=OFF -DTrilinos_ENABLE_Mesquite:BOOL=OFF -DTrilinos_ENABLE_ML:BOOL=ON -DTrilinos_ENABLE_MueLu:BOOL=ON -DTrilinos_ENABLE_NOX:BOOL=OFF -DTrilinos_ENABLE_Pamgen:BOOL=OFF -DTrilinos_ENABLE_Panzer:BOOL=OFF -DTrilinos_ENABLE_Pike:BOOL=OFF -DTrilinos_ENABLE_Piro:BOOL=OFF -DTrilinos_ENABLE_Phalanx:BOOL=OFF -DTrilinos_ENABLE_ROL:BOOL=OFF -DTrilinos_ENABLE_Rythmos:BOOL=OFF -DTrilinos_ENABLE_Sacado:BOOL=ON -DTrilinos_ENABLE_SCOREC:BOOL=OFF -DTrilinos_ENABLE_Shards:BOOL=OFF -DTrilinos_ENABLE_ShyLU:BOOL=OFF -DTrilinos_ENABLE_STK:BOOL=OFF -DTrilinos_ENABLE_Stokhos:BOOL=OFF -DTrilinos_ENABLE_Stratimikos:BOOL=OFF -DTrilinos_ENABLE_Teko:BOOL=OFF -DTrilinos_ENABLE_Tempus:BOOL=OFF -DTrilinos_ENABLE_Thyra:BOOL=OFF -DTrilinos_ENABLE_Tpetra:BOOL=ON -DTrilinos_ENABLE_TrilinosCouplings:BOOL=OFF -DTrilinos_ENABLE_Triutils:BOOL=ON -DTrilinos_ENABLE_Zoltan:BOOL=OFF -DTrilinos_ENABLE_Zoltan2:BOOL=OFF -DEpetraExt_BUILD_BTF:BOOL=OFF -DEpetraExt_BUILD_EXPERIMENTAL:BOOL=OFF -DEpetraExt_BUILD_GRAPH_REORDERINGS:BOOL=OFF -DAmesos2_ENABLE_Basker:BOOL=OFF -DAmesos2_ENABLE_LAPACK:BOOL=ON -DAmesos2_ENABLE_MUMPS:BOOL=OFF -DTrilinos_ENABLE_SEACASExodus:BOOL=OFF -DTrilinos_ENABLE_SEACASIoss:BOOL=OFF -DTrilinos_ENABLE_SEACASChaco:BOOL=OFF -DTrilinos_ENABLE_SEACASNemslice:BOOL=OFF -DTPL_ENABLE_ADIOS2:BOOL=OFF -DTPL_ENABLE_Boost:BOOL=OFF -DTPL_ENABLE_CUDA:BOOL=OFF -DTPL_ENABLE_HDF5:BOOL=OFF -DTPL_ENABLE_HYPRE:BOOL=OFF -DTPL_ENABLE_MUMPS:BOOL=OFF -DTPL_ENABLE_AMD:BOOL=OFF -DTPL_ENABLE_UMFPACK:BOOL=OFF -DTPL_ENABLE_SuperLU:BOOL=OFF -DTPL_ENABLE_SuperLUDist:BOOL=OFF -DTPL_ENABLE_X11:BOOL=OFF -DTPL_ENABLE_STRUMPACK:BOOL=OFF -DTPL_ENABLE_BLAS:BOOL=ON -DBLAS_INCLUDE_DIRS:STRING= -DBLAS_ROOT:STRING=/home/chm003/project-ords/spack/opt/spack/linux-rhel8-icelake/gcc-9.3.0/openblas-0.3.26-evmeob57vf5wh2dxlogstt4jvnronzdq -DBLAS_LIBRARY_NAMES:STRING=openblas -DBLAS_LIBRARY_DIRS:STRING=/home/chm003/project-ords/spack/opt/spack/linux-rhel8-icelake/gcc-9.3.0/openblas-0.3.26-evmeob57vf5wh2dxlogstt4jvnronzdq/lib64 -DTPL_ENABLE_CGNS:BOOL=OFF -DTPL_ENABLE_LAPACK:BOOL=ON -DLAPACK_INCLUDE_DIRS:STRING= -DLAPACK_ROOT:STRING=/home/chm003/project-ords/spack/opt/spack/linux-rhel8-icelake/gcc-9.3.0/openblas-0.3.26-evmeob57vf5wh2dxlogstt4jvnronzdq -DLAPACK_LIBRARY_NAMES:STRING=openblas -DLAPACK_LIBRARY_DIRS:STRING=/home/chm003/project-ords/spack/opt/spack/linux-rhel8-icelake/gcc-9.3.0/openblas-0.3.26-evmeob57vf5wh2dxlogstt4jvnronzdq/lib64 -DTPL_ENABLE_Matio:BOOL=OFF -DTPL_ENABLE_METIS:BOOL=OFF -DTPL_ENABLE_Netcdf:BOOL=OFF -DTPL_ENABLE_SCALAPACK:BOOL=OFF -DTPL_ENABLE_Zlib:BOOL=ON -DZlib_INCLUDE_DIRS:STRING=/home/chm003/project-ords/spack/opt/spack/linux-rhel8-icelake/gcc-9.3.0/zlib-1.3-yamyh2hu3oepzsjfssm56i6y76krrgqv/include -DZlib_ROOT:STRING=/home/chm003/project-ords/spack/opt/spack/linux-rhel8-icelake/gcc-9.3.0/zlib-1.3-yamyh2hu3oepzsjfssm56i6y76krrgqv -DZlib_LIBRARY_NAMES:STRING=z -DZlib_LIBRARY_DIRS:STRING=/home/chm003/project-ords/spack/opt/spack/linux-rhel8-icelake/gcc-9.3.0/zlib-1.3-yamyh2hu3oepzsjfssm56i6y76krrgqv/lib -DTPL_ENABLE_Pnetcdf:BOOL=OFF -DTPL_ENABLE_HWLOC:BOOL=ON -DHWLOC_INCLUDE_DIRS:STRING=/home/chm003/project-ords/spack/opt/spack/linux-rhel8-icelake/gcc-9.3.0/hwloc-2.9.1-7auwnwyiom4jwdcq5jncgxohfbakatbc/include -DHWLOC_ROOT:STRING=/home/chm003/project-ords/spack/opt/spack/linux-rhel8-icelake/gcc-9.3.0/hwloc-2.9.1-7auwnwyiom4jwdcq5jncgxohfbakatbc -DHWLOC_LIBRARY_NAMES:STRING=hwloc -DHWLOC_LIBRARY_DIRS:STRING=/home/chm003/project-ords/spack/opt/spack/linux-rhel8-icelake/gcc-9.3.0/hwloc-2.9.1-7auwnwyiom4jwdcq5jncgxohfbakatbc/lib -DTPL_ENABLE_MPI:BOOL=ON -DCMAKE_C_COMPILER:STRING=/fs/ssm/main/opt/intelcomp/master/inteloneapi_2023.2.0_ubuntu-22.04-amd64-64/mpi/2021.10.0/bin/mpiicc -DCMAKE_CXX_COMPILER:STRING=/fs/ssm/main/opt/intelcomp/master/inteloneapi_2023.2.0_ubuntu-22.04-amd64-64/mpi/2021.10.0/bin/mpiicpc -DCMAKE_Fortran_COMPILER:STRING=/fs/ssm/main/opt/intelcomp/master/inteloneapi_2023.2.0_ubuntu-22.04-amd64-64/mpi/2021.10.0/bin/mpiifort -DMPI_BASE_DIR:STRING=/fs/ssm/main/opt/intelcomp/master/inteloneapi_2023.2.0_ubuntu-22.04-amd64-64 -DTPL_ENABLE_ParMETIS:BOOL=OFF -DTPL_ENABLE_Cholmod:BOOL=OFF -DTeuchos_ENABLE_COMPLEX:BOOL=OFF -DTeuchos_ENABLE_FLOAT:BOOL=OFF -DTpetra_INST_OPENMP:BOOL=OFF -DTpetra_INST_DOUBLE:BOOL=ON -DTpetra_INST_COMPLEX_DOUBLE:BOOL=OFF -DTpetra_INST_COMPLEX_FLOAT:BOOL=OFF -DTpetra_INST_FLOAT:BOOL=OFF -DTpetra_INST_SERIAL:BOOL=ON -DTpetra_INST_INT_INT:BOOL=OFF -DTpetra_INST_INT_LONG:BOOL=OFF -DTpetra_INST_INT_LONG_LONG:BOOL=ON -DKokkos_ARCH_SKX:BOOL=ON -DKokkos_ENABLE_CUDA:BOOL=OFF -DKokkos_ENABLE_OPENMP:BOOL=OFF -DCMAKE_INSTALL_NAME_DIR:STRING=/home/chm003/project-ords/spack/opt/spack/linux-rhel8-icelake/gcc-9.3.0/trilinos-15.0.0-n4aqeajdyvaalvcebbow5nyy3v4hcwbm/lib
```
and
```
~/project-ords/spack/opt/spack/linux-rhel8-icelake/gcc-9.3.0/trilinos-15.0.0-n4aqeajdyvaalvcebbow5nyy3v4hcwbm/.spack/archived-files/spack-build-n4aqeaj $ ag Trilinos_ENABLE_THREAD_SAFE
CMakeCache.txt
4749:Trilinos_ENABLE_THREAD_SAFE:BOOL=ON
```
And the spec in case there is something in there that would conflict with the parallel build.
```
Concretized
--------------------------------
[+]  trilinos@15.0.0%gcc@9.3.0~adelus~adios2+amesos+amesos2+anasazi+aztec~basker+belos~boost~chaco~complex~cuda~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings~exodus+explicit_template_instantiation~float+fortran~gtest~hdf5~hypre+ifpack+ifpack2~intrepid~intrepid2~ipo~isorropia+kokkos~mesquite~minitensor+ml+mpi+muelu~mumps~nox~openmp~pamgen~panzer~phalanx~piro~python~rocm~rocm_rdc~rol~rythmos+sacado~scorec~shards+shared~shylu~stk~stokhos~stratimikos~strumpack~suite-sparse~superlu~superlu-dist~teko~tempus~test~thyra+tpetra~trilinoscouplings~wrapper~x11~zoltan~zoltan2 build_system=cmake build_type=Release cxxstd=17 generator=make gotype=long_long arch=linux-rhel8-icelake
[+]      ^cmake@3.27.9%gcc@9.3.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rhel8-icelake
[+]          ^curl@8.6.0%gcc@9.3.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-rhel8-icelake
[+]              ^nghttp2@1.57.0%gcc@9.3.0 build_system=autotools arch=linux-rhel8-icelake
[+]              ^openssl@3.2.1%gcc@9.3.0~docs+shared build_system=generic certs=mozilla arch=linux-rhel8-icelake
[+]                  ^ca-certificates-mozilla@2023-05-30%gcc@9.3.0 build_system=generic arch=linux-rhel8-icelake
[+]          ^ncurses@6.4%gcc@9.3.0~symlinks+termlib abi=none build_system=autotools arch=linux-rhel8-icelake
[+]          ^zlib@1.3%gcc@9.3.0+optimize+pic+shared build_system=makefile arch=linux-rhel8-icelake
[+]      ^gcc-runtime@9.3.0%gcc@9.3.0 build_system=generic arch=linux-rhel8-icelake
[+]      ^gmake@4.4.1%gcc@9.3.0~guile build_system=generic arch=linux-rhel8-icelake
[+]      ^hwloc@2.9.1%gcc@9.3.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-rhel8-icelake
[+]          ^libpciaccess@0.17%gcc@9.3.0 build_system=autotools arch=linux-rhel8-icelake
[+]              ^libtool@2.4.7%gcc@9.3.0 build_system=autotools arch=linux-rhel8-icelake
[+]                  ^findutils@4.9.0%gcc@9.3.0 build_system=autotools patches=440b954 arch=linux-rhel8-icelake
[+]                  ^m4@1.4.19%gcc@9.3.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-rhel8-icelake
[+]                      ^libsigsegv@2.14%gcc@9.3.0 build_system=autotools arch=linux-rhel8-icelake
[+]              ^util-macros@1.19.3%gcc@9.3.0 build_system=autotools arch=linux-rhel8-icelake
[+]          ^libxml2@2.10.3%gcc@9.3.0+pic~python+shared build_system=autotools arch=linux-rhel8-icelake
[+]              ^libiconv@1.17%gcc@9.3.0 build_system=autotools libs=shared,static arch=linux-rhel8-icelake
[+]              ^xz@5.4.6%gcc@9.3.0~pic build_system=autotools libs=shared,static arch=linux-rhel8-icelake
[+]          ^pkgconf@1.9.5%gcc@9.3.0 build_system=autotools arch=linux-rhel8-icelake
[e]      ^intel-oneapi-mpi@2021.10.0%gcc@9.3.0+envmods~external-libfabric~generic-names~ilp64 build_system=generic arch=linux-rhel8-icelake
[+]      ^kokkos@4.1.00%gcc@9.3.0~aggressive_vectorization~compiler_warnings~cuda~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning~wrapper build_system=cmake build_type=Release cxxstd=17 generator=make intel_gpu_arch=none arch=linux-rhel8-icelake
[+]      ^openblas@0.3.26%gcc@9.3.0~bignuma~consistent_fpcsr+fortran~ilp64~ipo+locking+pic+shared build_system=cmake build_type=Release generator=make symbol_suffix=none threads=none arch=linux-rhel8-icelake
[+]          ^perl@5.38.0%gcc@9.3.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-rhel8-icelake
[+]              ^berkeley-db@18.1.40%gcc@9.3.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-rhel8-icelake
[+]              ^bzip2@1.0.8%gcc@9.3.0~debug~pic+shared build_system=generic arch=linux-rhel8-icelake
[+]                  ^diffutils@3.9%gcc@9.3.0 build_system=autotools arch=linux-rhel8-icelake
[+]              ^gdbm@1.23%gcc@9.3.0 build_system=autotools arch=linux-rhel8-icelake
[+]                  ^readline@8.2%gcc@9.3.0 build_system=autotools patches=bbf97f1 arch=linux-rhel8-icelake



```
It sounds like you've done everything right... but I have no idea what Trilinos means by "thread safe" and am not sure I'd believe it if I did 😅
Actually seeing now that you say it's working on a standalone build... one big difference is that Spack uses an external Kokkos. Is there a "Kokkos_THREAD_SAFE" you have to enable upstream as well?
In this case, I fill a crs in parallel:
```
	typedef Tpetra::CrsMatrix<> crs_matrix_type;
	RCP<crs_matrix_type> m_matrix;
...
	m_matrix->sumIntoGlobalValues(global_row_idx, tuple(global_col_idx), tuple(val));

```
"it works" correctly when the "thread safe" is enabled 🙃 

>Actually seeing now that you say it's working on a standalone build... one big difference is that Spack uses an external Kokkos. Is there a "Kokkos_THREAD_SAFE" you have to enable upstream as well?

It's not obvious that there is such a flag. I must admit I assumed kokkos is thread safe given it's a parallel lib. However, in this case, I understand the error to be coming from the Tepetra methods that are wrapped in kokkos+teuchos parallel objects. 

```

2.2   Enabling/disabling thread safety

By default, many Trilinos classes are not thread-safe. However, some of these classes can be made thread safe by configuring with:

-D Trilinos_ENABLE_THREAD_SAFE:BOOL=ON

This will set the default value Teuchos_ENABLE_THREAD_SAFE=ON which makes the Teuchos Memory Management classes (Teuchos::RCP, Teuchos::Ptr, Teuchos::Array, Teuchos::ArrayView, and Teuchos::ArrayRCP) thread-safe. See documentation for other Trilinos packages for what parts become thread safe when setting this option.

```
https://gitlab.osti.gov/jmwille/Trilinos/-/issues/229#note_13308

I have determined that enabling variant `+openmp`, which implies `-DTrilinos_ENABLE_OpenMP` alongside adding the above-mentioned `-DTrilinos_ENABLE_THREAD_SAFE` resolves the problem.

I have not tested the other threading kokkos options to determine if they exhibit this behaviour as well. It is also not clear to me if `-DTrilinos_ENABLE_OpenMP` implies `-DTrilinos_ENABLE_THREAD_SAFE` or not. 

My perspective as a user is that this is a bug in the Trilinos config, and that thread safe Teuchos should occur regardless of what kokkos backends are selected. There might be a technical reason that is not the case.

I can simply enable openmp and move forward with my code, however I think a change should be made to the spack build to avoid this in the future. I'd rather not test out every permutation of the kokkos backends but I can if needed. What do you think?


You can make "threadsafe" a conditional `variant` that only shows up when using `+openmp`. The trilinos configure is a nightmare with billions of possible combinations, only a handful of which are known to work or be broken and only on a handful of compilers.
I am just rebuilding to confirm, but would this work for you? 

```
    # Makes the Teuchos Memory Management classes (Teuchos::RCP, Teuchos::Ptr, Teuchos::Array, 
    # Teuchos::ArrayView, and Teuchos::ArrayRCP) thread-safe. Requires at least the OMP kokkos backend
    # to be enabled. Without, this seemingly does nothing
    variant("threadsafe", default=True, when="+openmp", description="Enable threadsafe in Teuchos")
```

I think defaulting to `true` is sensible. Do you agree?
I'm not sure that's the right behavior.: I thought the typical OpenMP use case is to build on one thread, run on many. I think the Trilinos devs should chime in about the possible performance ramifications.
Sorry, I don't follow. What is not the right behaviour?

>I thought the typical OpenMP use case is to build on one thread, run on many

When you say build, are you referring to filling the matrix with values?


Sorry, by "right behavior" I mean with respect to the default variants, since the default TriBITS value for "thread safe" is "false" , even when OpenMP is enabled.

I'm just concerned that thread safety could add an unacceptable overhead to existing code that build on a single thread and use MPI for parallel execution.
Mmh that's fair. Then let's keep it `default=False`. If you agree I will put together a PR
That would be great. Thank you for investigating this and adding the new variant!

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/trilinos/package.py b/var/spack/repos/builtin/packages/trilinos/package.py
--- a/var/spack/repos/builtin/packages/trilinos/package.py
+++ b/var/spack/repos/builtin/packages/trilinos/package.py
@@ -102,6 +102,13 @@ class Trilinos(CMakePackage, CudaPackage, ROCmPackage):
     variant("uvm", default=False, when="@13.2: +cuda", description="Turn on UVM for CUDA build")
     variant("wrapper", default=False, description="Use nvcc-wrapper for CUDA build")
 
+    # Makes the Teuchos Memory Management classes (Teuchos::RCP, Teuchos::Ptr, Teuchos::Array,
+    # Teuchos::ArrayView, and Teuchos::ArrayRCP) thread-safe. Requires at least the OMP kokkos
+    # backend to be enabled. Without, this seemingly does nothing
+    variant(
+        "threadsafe", default=False, when="+openmp", description="Enable threadsafe in Teuchos"
+    )
+
     # TPLs (alphabet order)
     variant("adios2", default=False, description="Enable ADIOS2")
     variant("boost", default=False, description="Compile with Boost")
@@ -619,6 +626,7 @@ def define_enable(suffix, value=None):
                 define_trilinos_enable(
                     "EXPLICIT_INSTANTIATION", "explicit_template_instantiation"
                 ),
+                define_from_variant("Trilinos_ENABLE_THREAD_SAFE", "threadsafe"),
             ]
         )
 


Test patch: