OpenFOAM Installation with Spack (CentOS7)

# Spack Installation

https://spack.readthedocs.io/en/latest/getting_started.html

# Prep of Compiler and MPI with Spack

## Latest gcc installation

- The default gcc on CentOS7 is too old. so if you use this gcc, you must run into many problems.
- To avoid this problem, First of all let's install the latest gcc on Spack

```bash
spack install gcc
```

```bash
$ spack find gcc
==> 1 installed package
-- linux-centos7-x86_64_v3 / gcc@4.8.5 --------------------------
gcc@12.1.0
```

```bash
spack load gcc
```

```terminal
$ gcc --version
gcc (Spack GCC) 12.1.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
```


## Register the installed gcc to Spack

```bash
spack compiler find
```

This case, you can see `gcc@12.1.0`

```terminal
$ spack compilers
==> Available compilers
-- gcc centos7-x86_64 -------------------------------------------
gcc@12.1.0  gcc@4.8.5
```


## Intel OneAPI Installation

```terminal
$ spack list filter intel-oneapi
==> 27 packages.
intel-oneapi-advisor            intel-oneapi-dpct       intel-oneapi-mpi        libnetfilter-cttimeout      r-genefilter
intel-oneapi-ccl                intel-oneapi-dpl        intel-oneapi-tbb        libnetfilter-queue          r-rhdf5filters
intel-oneapi-compilers          intel-oneapi-inspector  intel-oneapi-vpl        perl-xml-filter-buffertext  ufo-filters
intel-oneapi-compilers-classic  intel-oneapi-ipp        intel-oneapi-vtune      py-filterpy
intel-oneapi-dal                intel-oneapi-ippcp      libnetfilter-conntrack  py-pandocfilters
intel-oneapi-dnn                intel-oneapi-mkl        libnetfilter-cthelper   r-annotationfilter
```

```terminal
spack install intel-oneapi-compilers %gcc@12.1.0
spack load intel-oneapi-compilers
```

```terminal
$ icx --version
Intel(R) oneAPI DPC++/C++ Compiler 2022.1.0 (2022.1.0.20220316)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/...(snip)...
Configuration file: /home/...(snip)...
```

## Register Intel OneAPI Compiler to Spack

```bash:command
spack compiler find
```

```terminal
$ spack compilers
==> Available compilers
-- dpcpp centos7-x86_64 -----------------------------------------
dpcpp@2022.1.0

-- gcc centos7-x86_64 -------------------------------------------
gcc@12.1.0  gcc@4.8.5

-- intel centos7-x86_64 -----------------------------------------
intel@2021.6.0

-- oneapi centos7-x86_64 ----------------------------------------
oneapi@2022.1.0
```

## IntelMPI Installation

```bash:command
spack install intel-oneapi-mpi %oneapi@2022.1.0
spack load intel-oneapi-mpi
```

```bash:command
$ mpirun --version
mpirun (Open MPI) 4.1.4

Report bugs to http://www.open-mpi.org/community/help/
```

# OpenFOAM Installation with Spack

- Compiler: `gcc@gcc@12.1.0`
    - If intel-oneapi-compiler, Error was got. So latest gcc is selected.
- MPI: `intel-oneapi-mpi@2021.6.0`
    - As the result of the initial test, IntelMPI can get good performance, compared with OpenMPI

## Check the hash of intel-oneapi-mpi

Please get the hash value of the installed oneapi with the following command.
In this case, hash value is `5onbafk`

```bash:command
spack find -l intel-oneapi-mpi
```

```terminal
==> 1 installed packages
-- linux-centos7-skylake_avx512 / oneapi@2022.1.0 ---------------
5onbafk intel-oneapi-mpi@2021.6.0
```

Define hash-value to a variable

```bash:command
MPI_HASH="5onbafk"
```

## OpenFOAM Installation

```bash:command
spack install openfoam %gcc@12.1.0 ^/${MPI_HASH}
```

## Load OpenFOAM

```bash:command
spack load openfoam
```

## Check the OpenFOAM

```terminal
$ interFoam -help

Usage: interFoam [OPTIONS]
Options:
  -case <dir>       Specify case directory to use (instead of cwd)
  -decomposeParDict <file>
                    Use specified file for decomposePar dictionary
  -dry-run          Check case set-up only using a single time step
  -dry-run-write    Check case set-up and write only using a single time step
  -parallel         Run in parallel
  -postProcess      Execute functionObjects only
  -doc              Display documentation in browser
  -help             Display short help and exit
  -help-full        Display full help and exit

Solver for two incompressible, isothermal immiscible fluids using VOF
phase-fraction based interface capturing.
With optional mesh motion and mesh topology changes including adaptive
re-meshing.

Using: OpenFOAM-v2206 (2206) - visit www.openfoam.com
Build: _76d719d1e6-20220624
Arch:  LSB;label=32;scalar=64
```


Appendix
====================

OpenFOAM information on Spack
-------------------------------------------

you can see library-dependencies and variants for the compile.

```terminal
$ spack info openfoam
Package:   openfoam

Description:
    OpenFOAM is a GPL-opensource C++ CFD-toolbox. This offering is supported
    by OpenCFD Ltd, producer and distributor of the OpenFOAM software via
    www.openfoam.com, and owner of the OPENFOAM trademark. OpenCFD Ltd has
    been developing and releasing OpenFOAM since its debut in 2004.

Homepage: https://www.openfoam.com/

Preferred version:  
    2206           https://sourceforge.net/projects/openfoam/files/v2206/OpenFOAM-v2206.tgz

Safe versions:  
    develop        [git] https://develop.openfoam.com/Development/openfoam.git on branch develop
    master         [git] https://develop.openfoam.com/Development/openfoam.git on branch master
    2206           https://sourceforge.net/projects/openfoam/files/v2206/OpenFOAM-v2206.tgz
    2112_220610    https://sourceforge.net/projects/openfoam/files/v2112/OpenFOAM-v2112_220610.tgz
    2112           https://sourceforge.net/projects/openfoam/files/v2112/OpenFOAM-v2112.tgz
    2106_220610    https://sourceforge.net/projects/openfoam/files/v2106/OpenFOAM-v2106_220610.tgz
    2106_211215    https://sourceforge.net/projects/openfoam/files/v2106/OpenFOAM-v2106_211215.tgz
    2106           https://sourceforge.net/projects/openfoam/files/v2106/OpenFOAM-v2106.tgz
    2012_220610    https://sourceforge.net/projects/openfoam/files/v2012/OpenFOAM-v2012_220610.tgz
    2012_210414    https://sourceforge.net/projects/openfoam/files/v2012/OpenFOAM-v2012_210414.tgz
    2012           https://sourceforge.net/projects/openfoam/files/v2012/OpenFOAM-v2012.tgz
    2006_220610    https://sourceforge.net/projects/openfoam/files/v2006/OpenFOAM-v2006_220610.tgz
    2006_201012    https://sourceforge.net/projects/openfoam/files/v2006/OpenFOAM-v2006_201012.tgz
    2006           https://sourceforge.net/projects/openfoam/files/v2006/OpenFOAM-v2006.tgz
    1912_220610    https://sourceforge.net/projects/openfoam/files/v1912/OpenFOAM-v1912_220610.tgz
    1912_200506    https://sourceforge.net/projects/openfoam/files/v1912/OpenFOAM-v1912_200506.tgz
    1912_200403    https://sourceforge.net/projects/openfoam/files/v1912/OpenFOAM-v1912_200403.tgz
    1912           https://sourceforge.net/projects/openfoam/files/v1912/OpenFOAM-v1912.tgz
    1906_200312    https://sourceforge.net/projects/openfoam/files/v1906/OpenFOAM-v1906_200312.tgz
    1906_191103    https://sourceforge.net/projects/openfoam/files/v1906/OpenFOAM-v1906_191103.tgz
    1906           https://sourceforge.net/projects/openfoam/files/v1906/OpenFOAM-v1906.tgz
    1812_200312    https://sourceforge.net/projects/openfoam/files/v1812/OpenFOAM-v1812_200312.tgz
    1812_191001    https://sourceforge.net/projects/openfoam/files/v1812/OpenFOAM-v1812_191001.tgz
    1812_190531    https://sourceforge.net/projects/openfoam/files/v1812/OpenFOAM-v1812_190531.tgz
    1812           https://sourceforge.net/projects/openfoam/files/v1812/OpenFOAM-v1812.tgz
    1806           https://sourceforge.net/projects/openfoam/files/v1806/OpenFOAM-v1806.tgz
    1712           https://sourceforge.net/projects/openfoam/files/v1712/OpenFOAM-v1712.tgz
    1706           https://sourceforge.net/projects/openfoam/files/v1706/OpenFOAM-v1706.tgz
    1612           https://sourceforge.net/projects/openfoam/files/v1612+/OpenFOAM-v1612+.tgz

Deprecated versions:  
    None

Variants:
    Name [Default]    When    Allowed values    Description
    ==============    ====    ==============    ==================================================

    float32 [off]     --      on, off           Use single-precision
    int64 [off]       --      on, off           With 64-bit labels
    kahip [off]       --      on, off           With kahip decomposition
    knl [off]         --      on, off           Use KNL compiler settings
    metis [off]       --      on, off           With metis decomposition
    mgridgen [off]    --      on, off           With mgridgen support
    paraview [off]    --      on, off           Build paraview plugins and runtime post-processing
    scotch [on]       --      on, off           With scotch/ptscotch decomposition
    source [on]       --      on, off           Install library/application sources and tutorials
    spdp [off]        --      on, off           Use single/double mixed precision
    vtk [off]         --      on, off           With VTK runTimePostProcessing
    zoltan [off]      --      on, off           With zoltan renumbering

Build Dependencies:
    adios2  boost  cgal  cmake  fftw-api  flex  kahip  m4  metis  mpi  paraview  parmgridgen  scotch  vtk  zlib  zoltan

Link Dependencies:
    adios2  boost  cgal  fftw-api  flex  kahip  metis  mpi  paraview  scotch  vtk  zlib  zoltan

Run Dependencies:
    None
```


How to distinguish applications with the same version and different compile conditions on Spack
-----------------------------------------------------------------

### How to check the hash of an installed application and the variant-conditions on Spack

- You can see hash for each openfoam with `-l` option.
- You can also see Variant-Conditions for the installation with `-v` option

```bash:STEP1
$ spack find -lv openfoam
==> 5 installed packages
-- linux-centos7-skylake_avx512 / gcc@12.1.0 --------------------
izv4we3 openfoam@2206~float32~int64~kahip~knl~metis~mgridgen~paraview+scotch+source~spdp~vtk~zoltan
pgmobng openfoam@2206~float32~int64~kahip~knl~metis~mgridgen~paraview+scotch+source~spdp~vtk~zoltan
ovt7fb4 openfoam@2206~float32~int64~kahip~knl~metis~mgridgen~paraview+scotch+source~spdp~vtk~zoltan
sgalfui openfoam@2206~float32~int64~kahip~knl~metis~mgridgen~paraview+scotch+source~spdp~vtk~zoltan

-- linux-centos7-skylake_avx512 / oneapi@2022.1.0 ---------------
bs6ib3o openfoam@2206~float32~int64~kahip~knl~metis~mgridgen~paraview+scotch+source~spdp~vtk~zoltan
```

### Check the lib-dependencies

- if you want to check Library dependencies for a certain openfoam, please get the hash of the application above. for example `izv4we3`
- The hash you want to check is decided, you can see lib-dependencies with the following commands.
- For example, This is useful if you want to switch to openfoam with different MPI-lib

```bash
HASH="izv4we3"
```

```bash
$ spack find -ld /${HASH}
==> 1 installed package
-- linux-centos7-skylake_avx512 / gcc@12.1.0 --------------------
izv4we3 openfoam@2206
xxvcgxf     adios2@2.8.1
uxeczyl         bzip2@1.0.8
26nu5db             diffutils@3.8
e2jcaca                 libiconv@1.16
bbo5g7r         c-blosc@1.21.1
ptkahsc             cmake@3.23.2
fov7vip                 ncurses@6.2
wkherf6                     pkgconf@1.8.0
jri63p6                 openssl@1.1.1p
itda7co                     ca-certificates-mozilla@2022-03-29
pytjagw                     perl@5.34.1
6vpregl                         berkeley-db@18.1.40
uwcdtdl                         gdbm@1.19
sdbhx7q                             readline@8.1.2
iffomx3                         zlib@1.2.12
oh3fgkz             lz4@1.9.3
p3h5hwm             snappy@1.1.8
ub4z64d             zstd@1.5.2
5onbafk         intel-oneapi-mpi@2021.6.0
m6zrw3m         libfabric@1.14.1
5fii44x         libffi@3.4.2
tktzhgt         libpng@1.6.37
ievzj2o         sz@2.1.12.2
k33yqv7         zfp@0.5.5
aqltvxd     boost@1.79.0
chmrkhm     cgal@4.13
otkgdzw         eigen@3.4.0
adxc7gc         gmp@6.2.1
ps5hqpj             autoconf@2.69
s3o4eaq                 m4@1.4.19
bb74ie4                     libsigsegv@2.13
gi7vs7a             automake@1.16.5
pcqswzt             libtool@2.4.7
dwe6krm         mpfr@4.1.0
uunlpwt             autoconf-archive@2022.02.11
q5xx7dm             texinfo@6.5
kdfbbt2     fftw@3.3.10
xopdau4     flex@2.6.4
jzsrd5k         bison@3.8.2
wzibbwh         findutils@4.9.0
au3gxwj         gettext@0.21
zyqemyn             libxml2@2.9.13
ys73qyv                 xz@5.2.5
xiyb6ww             tar@1.34
mdkeqw7                 pigz@2.7
3jxn4if         help2man@1.47.16
tzdw73i     scotch@7.0.1
```

### Sample: Check mpi dependencies at once.

```bash
$ spack find -ld openfoam | grep -e openfoam@2206 -e mpi
izv4we3 openfoam@2206
5onbafk         intel-oneapi-mpi@2021.6.0
pgmobng openfoam@2206
7dvvkku         openmpi@4.1.4
ovt7fb4 openfoam@2206
2aoek3i         openmpi@4.1.4
sgalfui openfoam@2206
sm6zwel         openmpi@4.1.4
bs6ib3o openfoam@2206
sm6zwel         openmpi@4.1.4
```