Repository: spack/spack

Issue:
Installation issue: Trilinos on macos BigSur and m1 cpu
### Steps to reproduce the issue

```spack install -j8 --fail-fast --only dependencies dealii```

Trilinos fails to compile.

### Information on your system

* **Spack:** 0.16.2-4210-b12f38383c
* **Python:** 3.9.6
* **Platform:** darwin-bigsur-m1
* **Concretizer:** original

compilers.yaml

```
compilers:
- compiler:
    spec: apple-clang@12.0.5
    paths:
      cc: /usr/bin/clang
      cxx: /usr/bin/clang++
      f77: /opt/homebrew/bin/gfortran
      fc: /opt/homebrew/bin/gfortran
    flags: {}
    operating_system: bigsur
    target: aarch64
    modules: []
    environment: {}
    extra_rpaths: []
```

packages.yaml

```
packages:
   dealii: 
      variants: ~examples +simplex ~nanoflann ~gmsh ~oce +trilinos ~arborx
   hdf5:
      variants: +fortran +cxx +hl
   petsc:
      variants: +mumps +suite-sparse
   cgns:
      variants: +fortran
   trilinos:
      variants: +rol
   all:
      providers:
         mpi: [openmpi]
         blas: [openblas]
         lapack: [openblas]
```

### Additional information

[spack-build-out.txt](https://github.com/spack/spack/files/7147193/spack-build-out.txt)
[spack-build-env.txt](https://github.com/spack/spack/files/7147194/spack-build-env.txt)

@keitat  @kuberry  @sethrj

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have run `spack maintainers <name-of-the-package>` and **@mentioned** any maintainers
- [X] I have uploaded the build log and environment files
- [X] I have searched the issues of this repo and believe this is not a duplicate


Discussion:
@cpraveen I think you need to open up a separate issue for the kokkos install... it looks unrelated to the trilinos failure, which for ease of reading is a link error:
```
/Users/praveen/Applications/spack/opt/spack/darwin-bigsur-m1/apple-clang-12.0.5/openmpi-4.1.1-qhymis22q3hsyedqepj3zxz53dz57acp/bin/mpic++ -O2 -g -DNDEBUG -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk -mmacosx-version-min=11.5.2 -dynamiclib -Wl,-headerpad_max_install_names -compatibility_version 13.0.0 -current_version 13.0.0 -o libepetra.13.0.dylib -install_name @rpath/libepetra.13.dylib CMakeFiles/epetra.dir/Epetra_BLAS.cpp.o CMakeFiles/epetra.dir/Epetra_BasicDirectory.cpp.o CMakeFiles/epetra.dir/Epetra_BasicRowMatrix.cpp.o CMakeFiles/epetra.dir/Epetra_BlockMap.cpp.o CMakeFiles/epetra.dir/Epetra_BlockMapData.cpp.o CMakeFiles/epetra.dir/Epetra_C_wrappers.cpp.o CMakeFiles/epetra.dir/Epetra_CompObject.cpp.o CMakeFiles/epetra.dir/Epetra_CrsGraph.cpp.o CMakeFiles/epetra.dir/Epetra_CrsGraphData.cpp.o CMakeFiles/epetra.dir/Epetra_CrsMatrix.cpp.o CMakeFiles/epetra.dir/Epetra_CrsSingletonFilter.cpp.o CMakeFiles/epetra.dir/Epetra_Data.cpp.o CMakeFiles/epetra.dir/Epetra_DistObject.cpp.o CMakeFiles/epetra.dir/Epetra_Export.cpp.o CMakeFiles/epetra.dir/Epetra_FECrsGraph.cpp.o CMakeFiles/epetra.dir/Epetra_FECrsMatrix.cpp.o CMakeFiles/epetra.dir/Epetra_FEVbrMatrix.cpp.o CMakeFiles/epetra.dir/Epetra_FEVector.cpp.o CMakeFiles/epetra.dir/Epetra_Flops.cpp.o CMakeFiles/epetra.dir/Epetra_Fortran_wrappers.cpp.o CMakeFiles/epetra.dir/Epetra_Import.cpp.o CMakeFiles/epetra.dir/Epetra_Import_Util.cpp.o CMakeFiles/epetra.dir/Epetra_IntMultiVector.cpp.o CMakeFiles/epetra.dir/Epetra_IntSerialDenseMatrix.cpp.o CMakeFiles/epetra.dir/Epetra_IntSerialDenseVector.cpp.o CMakeFiles/epetra.dir/Epetra_IntVector.cpp.o CMakeFiles/epetra.dir/Epetra_JadMatrix.cpp.o CMakeFiles/epetra.dir/Epetra_LAPACK.cpp.o CMakeFiles/epetra.dir/Epetra_LinearProblem.cpp.o CMakeFiles/epetra.dir/Epetra_LocalMap.cpp.o CMakeFiles/epetra.dir/Epetra_Map.cpp.o CMakeFiles/epetra.dir/Epetra_MapColoring.cpp.o CMakeFiles/epetra.dir/Epetra_MultiVector.cpp.o CMakeFiles/epetra.dir/Epetra_Object.cpp.o CMakeFiles/epetra.dir/Epetra_OffsetIndex.cpp.o CMakeFiles/epetra.dir/Epetra_RowMatrixTransposer.cpp.o CMakeFiles/epetra.dir/Epetra_SerialComm.cpp.o CMakeFiles/epetra.dir/Epetra_SerialCommData.cpp.o CMakeFiles/epetra.dir/Epetra_SerialDenseMatrix.cpp.o CMakeFiles/epetra.dir/Epetra_SerialDenseSVD.cpp.o CMakeFiles/epetra.dir/Epetra_SerialDenseSolver.cpp.o CMakeFiles/epetra.dir/Epetra_SerialDenseVector.cpp.o CMakeFiles/epetra.dir/Epetra_SerialDistributor.cpp.o CMakeFiles/epetra.dir/Epetra_SerialSpdDenseSolver.cpp.o CMakeFiles/epetra.dir/Epetra_SerialSymDenseMatrix.cpp.o CMakeFiles/epetra.dir/Epetra_Time.cpp.o CMakeFiles/epetra.dir/Epetra_Util.cpp.o CMakeFiles/epetra.dir/Epetra_VbrMatrix.cpp.o CMakeFiles/epetra.dir/Epetra_Vector.cpp.o CMakeFiles/epetra.dir/Epetra_LongLongSerialDenseMatrix.cpp.o CMakeFiles/epetra.dir/Epetra_LongLongSerialDenseVector.cpp.o CMakeFiles/epetra.dir/Epetra_LongLongVector.cpp.o CMakeFiles/epetra.dir/Epetra_dcrsmm.f.o CMakeFiles/epetra.dir/Epetra_dcrsmv.f.o CMakeFiles/epetra.dir/Epetra_dcrssm.f.o CMakeFiles/epetra.dir/Epetra_dcrssv.f.o CMakeFiles/epetra.dir/Epetra_MpiComm.cpp.o CMakeFiles/epetra.dir/Epetra_MpiCommData.cpp.o CMakeFiles/epetra.dir/Epetra_MpiDistributor.cpp.o   -L/opt/homebrew/Cellar/gcc/11.2.0/lib/gcc/11/gcc/aarch64-apple-darwin20/11.1.0  -L/opt/homebrew/Cellar/gcc/11.2.0/lib/gcc/11  -Wl,-rpath,/private/var/folders/tc/17nf7v3j53jgg4db9vxl40g00000gn/T/praveen/spack-stage/spack-stage-trilinos-13.0.1-sbqbhthus2snrmj2voosa4hifd237pey/spack-build-sbqbhth/packages/teuchos/kokkoscomm/src -Wl,-rpath,/private/var/folders/tc/17nf7v3j53jgg4db9vxl40g00000gn/T/praveen/spack-stage/spack-stage-trilinos-13.0.1-sbqbhthus2snrmj2voosa4hifd237pey/spack-build-sbqbhth/packages/teuchos/kokkoscompat/src -Wl,-rpath,/private/var/folders/tc/17nf7v3j53jgg4db9vxl40g00000gn/T/praveen/spack-stage/spack-stage-trilinos-13.0.1-sbqbhthus2snrmj2voosa4hifd237pey/spack-build-sbqbhth/packages/teuchos/remainder/src -Wl,-rpath,/private/var/folders/tc/17nf7v3j53jgg4db9vxl40g00000gn/T/praveen/spack-stage/spack-stage-trilinos-13.0.1-sbqbhthus2snrmj2voosa4hifd237pey/spack-build-sbqbhth/packages/teuchos/numerics/src -Wl,-rpath,/private/var/folders/tc/17nf7v3j53jgg4db9vxl40g00000gn/T/praveen/spack-stage/spack-stage-trilinos-13.0.1-sbqbhthus2snrmj2voosa4hifd237pey/spack-build-sbqbhth/packages/teuchos/comm/src -Wl,-rpath,/private/var/folders/tc/17nf7v3j53jgg4db9vxl40g00000gn/T/praveen/spack-stage/spack-stage-trilinos-13.0.1-sbqbhthus2snrmj2voosa4hifd237pey/spack-build-sbqbhth/packages/teuchos/parameterlist/src -Wl,-rpath,/private/var/folders/tc/17nf7v3j53jgg4db9vxl40g00000gn/T/praveen/spack-stage/spack-stage-trilinos-13.0.1-sbqbhthus2snrmj2voosa4hifd237pey/spack-build-sbqbhth/packages/teuchos/parser/src -Wl,-rpath,/private/var/folders/tc/17nf7v3j53jgg4db9vxl40g00000gn/T/praveen/spack-stage/spack-stage-trilinos-13.0.1-sbqbhthus2snrmj2voosa4hifd237pey/spack-build-sbqbhth/packages/teuchos/core/src -Wl,-rpath,/private/var/folders/tc/17nf7v3j53jgg4db9vxl40g00000gn/T/praveen/spack-stage/spack-stage-trilinos-13.0.1-sbqbhthus2snrmj2voosa4hifd237pey/spack-build-sbqbhth/packages/kokkos/core/src ../../teuchos/kokkoscomm/src/libteuchoskokkoscomm.13.0.dylib ../../teuchos/kokkoscompat/src/libteuchoskokkoscompat.13.0.dylib ../../teuchos/remainder/src/libteuchosremainder.13.0.dylib ../../teuchos/numerics/src/libteuchosnumerics.13.0.dylib ../../teuchos/comm/src/libteuchoscomm.13.0.dylib ../../teuchos/parameterlist/src/libteuchosparameterlist.13.0.dylib ../../teuchos/parser/src/libteuchosparser.13.0.dylib ../../teuchos/core/src/libteuchoscore.13.0.dylib /Users/praveen/Applications/spack/opt/spack/darwin-bigsur-m1/apple-clang-12.0.5/openblas-0.3.17-x2mcgm4keqeutwcd6m67ajh3mlfeauxa/lib/libopenblas.dylib /Users/praveen/Applications/spack/opt/spack/darwin-bigsur-m1/apple-clang-12.0.5/openblas-0.3.17-x2mcgm4keqeutwcd6m67ajh3mlfeauxa/lib/libopenblas.dylib ../../kokkos/core/src/libkokkoscore.3.1.1.dylib /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk/usr/lib/libdl.tbd /Users/praveen/Applications/spack/opt/spack/darwin-bigsur-m1/apple-clang-12.0.5/hwloc-2.5.0-izaq5ziq26wzvs2cpbozhshli42d66zd/lib/libhwloc.dylib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lgfortran -lgcc_s.1.1 -lgcc -lm -lgcc_s.1.1 -lgcc -lgcc 
ld: warning: could not create compact unwind for _epetra_sccssm5_: registers 27 and 28 not saved contiguously in frame
ld: warning: could not create compact unwind for _epetra_dcrssv_: registers 25 and 26 not saved contiguously in frame
Undefined symbols for architecture arm64:
  "_epetra_dcrsmm__", referenced from:
      Epetra_CrsMatrix::GeneralMM(double**, int, double**, int, int) const in Epetra_CrsMatrix.cpp.o
      Epetra_CrsMatrix::GeneralMTM(double**, int, double**, int, int) const in Epetra_CrsMatrix.cpp.o
  "_epetra_dcrsmv__", referenced from:
      Epetra_CrsMatrix::GeneralMV(double*, double*) const in Epetra_CrsMatrix.cpp.o
      Epetra_CrsMatrix::GeneralMTV(double*, double*) const in Epetra_CrsMatrix.cpp.o
  "_epetra_dcrssm__", referenced from:
      Epetra_CrsMatrix::GeneralSM(bool, bool, bool, double**, int, double**, int, int) const in Epetra_CrsMatrix.cpp.o
  "_epetra_dcrssv__", referenced from:
      Epetra_CrsMatrix::GeneralSV(bool, bool, bool, double*, double*) const in Epetra_CrsMatrix.cpp.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [packages/epetra/src/libepetra.13.0.dylib] Error 1
make[1]: *** [packages/epetra/src/CMakeFiles/epetra.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
```
This looks like a Fortran name mangling issue: it looks like the fortran compiler is generating `_epetra_dcrssv_` but the C++ code in Trilinos expects `_epetra_dcrssv__` (note the second underscore). I don't know anyone personally who has successfully built Trilinos on the Apple M1. It might be worth trying a minimal install of trilinos (Epetra only) and posting on the [Trilinos issue tracker](https://github.com/trilinos/Trilinos/issues).
I tried this

```
spack install -j8 trilinos ~amesos~amesos2~anasazi~aztec~belos~ifpack~ifpack2~kokkos~muelu~tpetra~rol~sacado
```

which gives same error as before.

Without fortran

```
spack install -j8 trilinos ~amesos~amesos2~anasazi~aztec~belos~ifpack~ifpack2~kokkos~muelu~tpetra~rol~sacado~fortran
```

it compiles. I dont use trilinos from fortran. Do I lose anything if I disable fortran ?
@cpraveen Interesting... this should probably be reported in the Trilinos (or TriBITS) bug tracker for sure. Fortran is used for building internal components (such as the Epetra solvers you see) rather than for making Trilinos available to downstream Fortran code (which is the purview of the ForTrilinos project). I'm honestly not sure what capabilities are lost by disabling Fortran internally, but it's probably limited to some of the older solvers and first-generation packages like Epetra. If that gives you a path forward, that's great!
I had to disable fortran and aztec to compile it.

If I disable only fortran, it fails while compiling aztecoo where it seems to need some fortran stuff. I think I need aztec since the deal.II package asks for aztec to be built in trilinos.

@sethrj 
Here is the output when I disable fortran only

```
spack install -j8 --fail-fast --only dependencies dealii ^trilinos~fortran+aztec
```

These are some errors

```
  >> 2577    /var/folders/tc/17nf7v3j53jgg4db9vxl40g00000gn/T/praveen/spack-stage/spack-stage-trilinos-13.0.1-fukx2afzjviqs46g
             x6u7tmbykpckzivk/spack-src/packages/aztecoo/src/az_domain_decomp.c:513:7: error: implicit declaration of function
              'az_fnroot_c' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
     2578          AZ_FNROOT_F77(&root,bindx2,&(bindx2[N+1]),mask, &nlvl,
     2579          ^
     2580    /var/folders/tc/17nf7v3j53jgg4db9vxl40g00000gn/T/praveen/spack-stage/spack-stage-trilinos-13.0.1-fukx2afzjviqs46g
             x6u7tmbykpckzivk/spack-src/packages/aztecoo/src/az_aztec.h:611:42: note: expanded from macro 'AZ_FNROOT_F77'
     2581    #   define AZ_FNROOT_F77                 az_fnroot_c
     2582                                             ^
  >> 2583    /var/folders/tc/17nf7v3j53jgg4db9vxl40g00000gn/T/praveen/spack-stage/spack-stage-trilinos-13.0.1-fukx2afzjviqs46g
             x6u7tmbykpckzivk/spack-src/packages/aztecoo/src/az_domain_decomp.c:515:7: error: implicit declaration of function
              'az_rcm_c' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
     2584          AZ_RCM_F77(&root,bindx2,&(bindx2[N+1]),mask,&((*ordering)[total]),
     2585          ^
     2586    /var/folders/tc/17nf7v3j53jgg4db9vxl40g00000gn/T/praveen/spack-stage/spack-stage-trilinos-13.0.1-fukx2afzjviqs46g
             x6u7tmbykpckzivk/spack-src/packages/aztecoo/src/az_aztec.h:613:42: note: expanded from macro 'AZ_RCM_F77'
     2587    #   define AZ_RCM_F77                    az_rcm_c
     2588                                             ^
     2589    2 errors generated.
  >> 2590    make[2]: *** [packages/aztecoo/src/CMakeFiles/aztecoo.dir/az_domain_decomp.c.o] Error 1
  >> 2591    make[1]: *** [packages/aztecoo/src/CMakeFiles/aztecoo.dir/all] Error 2
```

[spack-build-out.txt](https://github.com/spack/spack/files/7153186/spack-build-out.txt)
[spack-build-env.txt](https://github.com/spack/spack/files/7153189/spack-build-env.txt)

Thanks @cpraveen , I'll add conflicts for `+aztec~fortran` and `@:13.0.1 +fortran target=m1`.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/trilinos/package.py b/var/spack/repos/builtin/packages/trilinos/package.py
--- a/var/spack/repos/builtin/packages/trilinos/package.py
+++ b/var/spack/repos/builtin/packages/trilinos/package.py
@@ -203,6 +203,7 @@ class Trilinos(CMakePackage, CudaPackage):
         conflicts('~stratimikos')
         conflicts('@:12 gotype=long')
 
+    conflicts('+aztec', when='~fortran')
     conflicts('+basker', when='~amesos2')
     conflicts('+ifpack2', when='~belos')
     conflicts('+intrepid', when='~sacado')
@@ -272,6 +273,9 @@ class Trilinos(CMakePackage, CudaPackage):
     conflicts('+stokhos', when='%xl')
     conflicts('+stokhos', when='%xl_r')
 
+    # Fortran mangling fails on Apple M1 (see spack/spack#25900)
+    conflicts('@:13.0.1 +fortran', when='target=m1')
+
     # ###################### Dependencies ##########################
 
     # Explicit dependency variants


Test patch: