WRF Installation with Spack (CentOS7)

WRF Installation with Spack
=====================================================

Spack Installatiion
-------------------------------

- [Official Manual](https://spack.readthedocs.io/en/latest/index.html)

```bash
sudo yum install -y libffi-devel
sudo yum install python3
```

```bash
git clone https://github.com/spack/spack
```

```bash
SPACK_DIR="\${HOME}/spack"
echo "export SPACK_ROOT=${SPACK_DIR}" >> ~/.bashrc
echo "source \${SPACK_ROOT}/share/spack/setup-env.sh" >> ~/.bashrc
tail ~/.bashrc
```

gcc update with Spack
-------------------------------

```bash
spack install gcc
```

```bash
spack compiler info
```

```bash
spack load gcc
```

```bash
$ gcc --version
gcc (Spack GCC) 12.1.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
```

```bash
$ spack compiler info gcc
gcc@4.8.5:
        paths:
                cc = /usr/bin/gcc
                cxx = /usr/bin/g++
                f77 = /usr/bin/gfortran
                fc = /usr/bin/gfortran
        modules  = []
        operating system  = centos7
gcc@12.1.0:
        paths:
                cc = /home/xdg-059/xdu-059001/spack/opt/spack/linux-centos7-x86_64_v3/gcc-4.8.5/gcc-12.1.0-y5ycepb3qudsyinthkyz7xf4w45s54qz/bin/gcc
                cxx = /home/xdg-059/xdu-059001/spack/opt/spack/linux-centos7-x86_64_v3/gcc-4.8.5/gcc-12.1.0-y5ycepb3qudsyinthkyz7xf4w45s54qz/bin/g++
                f77 = /home/xdg-059/xdu-059001/spack/opt/spack/linux-centos7-x86_64_v3/gcc-4.8.5/gcc-12.1.0-y5ycepb3qudsyinthkyz7xf4w45s54qz/bin/gfortran
                fc = /home/xdg-059/xdu-059001/spack/opt/spack/linux-centos7-x86_64_v3/gcc-4.8.5/gcc-12.1.0-y5ycepb3qudsyinthkyz7xf4w45s54qz/bin/gfortran
        modules  = []
        operating system  = centos7
```

```bash
spack find compiler
```



WRF installation with Spack
-------------

### Spack対応アプリ確認

```terminal
$ spack list | grep wrf
wrf
wrf-io
```

### 使用可能なパラメータの確認

```terminal
$ spack info wrf
Package:   wrf

Description:
    The Weather Research and Forecasting (WRF) Model is a next-generation
    mesoscale numerical weather prediction system designed for both
    atmospheric research and operational forecasting applications.

Homepage: https://www.mmm.ucar.edu/weather-research-and-forecasting-model

Preferred version:  
    4.3.3      https://github.com/wrf-model/WRF/archive/v4.3.3.tar.gz

Safe versions:  
    4.3.3      https://github.com/wrf-model/WRF/archive/v4.3.3.tar.gz
    4.3.2      https://github.com/wrf-model/WRF/archive/v4.3.2.tar.gz
    4.3.1      https://github.com/wrf-model/WRF/archive/v4.3.1.tar.gz
    4.2        https://github.com/wrf-model/WRF/archive/v4.2.tar.gz
    4.0        https://github.com/wrf-model/WRF/archive/v4.0.tar.gz
    3.9.1.1    https://github.com/wrf-model/WRF/archive/V3.9.1.1.tar.gz

Deprecated versions:  
    None

Variants:
    Name [Default]            When     Allowed values          Description
    ======================    =====    ====================    ===========================================

    build_type [dmpar]        --       serial, smpar,          
                                       dmpar, dm+sm            
    chem [off]                @4:      on, off                 Enable WRF-Chem
    compile_type [em_real]    --       em_real,                
                                       em_quarter_ss,          
                                       em_b_wave, em_les,      
                                       em_heldsuarez,          
                                       em_tropical_cyclone,    
                                       em_hill2d_x,            
                                       em_squall2d_x,          
                                       em_squall2d_y,          
                                       em_grav2d_x,            
                                       em_seabreeze2d_x,       
                                       em_scm_xy               
    nesting [basic]           --       no_nesting, basic,      
                                       preset_moves,           
                                       vortex_following        
    pnetcdf [on]              --       on, off                 Parallel IO support through Pnetcdf library

Build Dependencies:
    hdf5  jasper  jemalloc  libpng  libtirpc  libtool  m4  mpi  netcdf-c  netcdf-fortran  parallel-netcdf  perl  pkgconfig  tcsh  time  zlib

Link Dependencies:
    hdf5  jasper  jemalloc  libpng  libtirpc  mpi  netcdf-c  netcdf-fortran  parallel-netcdf  perl  zlib

Run Dependencies:
    None
```

WRFの`Variants`の各変数をまとめると、以下のようになる

**`build_type=<並列タイプ>`**

- `serial`: 逐次実行（シングルスレッド）
- `smpar`: OpenMP環境でのSMP上での並列実行（マルチスレッド）
- `dmpar`: MPI環境での分散並列実行
- `dm+sm`: MPIとOpenMP環境を併用した分散実行
 
**`compile_type=<コンパイルタイプ>`**

- `em_real`: （3dリアルケース）
- `em_quarter_ss`: （3d理想ケース）
- `em_b_wave`: （3d理想ケース）
- `em_les`: （3d理想ケース）
- `em_heldsuarez`: （3d理想ケース）
- `em_tropical_cyclone`: （3d理想ケース）
- `em_hill2d_x`: （ 2d理想ケース）
- `em_squall2d_x`: （2d理想ケース）
- `em_s`: 2d理想的なケース）
- `em_grav2d_x`: （2d理想的なケース）
- `em_seabreeze2d_x`: （2d理想的なケース）
- `em_scm_xy`: （1d理想的なケース）

**`nesting=<nest type>`**

- `no_nesting`
- `basic`
- `preset_moves`
- `vortex_following`

### Installation with Spack

`spack info wrf`のコマンドの結果より使用するvariantsを定義する

```bash
BUILD_TYPE="dmpar"
NESTING="basic"
COMPILE_TYPE="em_real"
```

インストールの開始

```bash
spack -d install -v -j 16 wrf %gcc@12.1.0 \
    build_type=${BUILD_TYPE} \
    nesting=${NESTING} \
    compile_type=${COMPILE_TYPE} \
    ^openmpi@4.1.4+cxx fabrics=auto
```

長時間の場合は以下のように`nohup`を使うと便利

```bash
nohup spack -d install -v -j 16 wrf %gcc@12.1.0 \
    build_type=${BUILD_TYPE} \
    nesting=${NESTING} \
    compile_type=${COMPILE_TYPE} \
    ^openmpi@4.1.4+cxx fabrics=auto &
```


数十分まっていると以下のような画面になって、インストールが開始される

```terminal
  :
  :
(snip)
  :

If you REALLY want Grib2 output from WRF, modify the arch/Config.pl script.
Right now you are not getting the Jasper lib, from the environment, compiled into WRF.

------------------------------------------------------------------------
Please select from among the following Linux x86_64 options:

  1. (serial)   2. (smpar)   3. (dmpar)   4. (dm+sm)   PGI (pgf90/gcc)
  5. (serial)   6. (smpar)   7. (dmpar)   8. (dm+sm)   PGI (pgf90/pgcc): SGI MPT
  9. (serial)  10. (smpar)  11. (dmpar)  12. (dm+sm)   PGI (pgf90/gcc): PGI accelerator
 13. (serial)  14. (smpar)  15. (dmpar)  16. (dm+sm)   INTEL (ifort/icc)
                                         17. (dm+sm)   INTEL (ifort/icc): Xeon Phi (MIC architecture)
 18. (serial)  19. (smpar)  20. (dmpar)  21. (dm+sm)   INTEL (ifort/icc): Xeon (SNB with AVX mods)
 22. (serial)  23. (smpar)  24. (dmpar)  25. (dm+sm)   INTEL (ifort/icc): SGI MPT
 26. (serial)  27. (smpar)  28. (dmpar)  29. (dm+sm)   INTEL (ifort/icc): IBM POE
 30. (serial)               31. (dmpar)                PATHSCALE (pathf90/pathcc)
 32. (serial)  33. (smpar)  34. (dmpar)  35. (dm+sm)   GNU (gfortran/gcc)
 36. (serial)  37. (smpar)  38. (dmpar)  39. (dm+sm)   IBM (xlf90_r/cc_r)
 40. (serial)  41. (smpar)  42. (dmpar)  43. (dm+sm)   PGI (ftn/gcc): Cray XC CLE
 44. (serial)  45. (smpar)  46. (dmpar)  47. (dm+sm)   CRAY CCE (ftn $(NOOMP)/cc): Cray XE and XC
 48. (serial)  49. (smpar)  50. (dmpar)  51. (dm+sm)   INTEL (ftn/icc): Cray XC
 52. (serial)  53. (smpar)  54. (dmpar)  55. (dm+sm)   PGI (pgf90/pgcc)
 56. (serial)  57. (smpar)  58. (dmpar)  59. (dm+sm)   PGI (pgf90/gcc): -f90=pgf90
 60. (serial)  61. (smpar)  62. (dmpar)  63. (dm+sm)   PGI (pgf90/pgcc): -f90=pgf90
 64. (serial)  65. (smpar)  66. (dmpar)  67. (dm+sm)   INTEL (ifort/icc): HSW/BDW
 68. (serial)  69. (smpar)  70. (dmpar)  71. (dm+sm)   INTEL (ifort/icc): KNL MIC
 72. (serial)  73. (smpar)  74. (dmpar)  75. (dm+sm)   FUJITSU (frtpx/fccpx): FX10/FX100 SPARC64 IXfx/Xlfx

==> [2022-07-12-23:36:20.957535] Warning: Selected build option GNU (gfortran/gcc).
Enter selection [1-75] : ------------------------------------------------------------------------
Compile for nesting? (1=basic, 2=preset moves, 3=vortex following) [default 1]: 
Configuration successful! 
------------------------------------------------------------------------
testing for fseeko and fseeko64
fseeko64 is supported
------------------------------------------------------------------------

# Settings for    Linux x86_64 ppc64le, gfortran compiler with gcc   (dmpar)
#
DESCRIPTION     =       GNU ($SFC/$SCC)
DMPARALLEL      =        1
OMPCPP          =       # -D_OPENMP
OMP             =       # -fopenmp
OMPCC           =       # -fopenmp
SFC             =       gfortran
SCC             =       gcc
CCOMP           =       gcc
DM_FC           =       mpif90 -f90=$(SFC)
DM_CC           =       mpicc -cc=$(SCC)
FC              =       time $(DM_FC)
CC              =       $(DM_CC) -DFSEEKO64_OK 
LD              =       $(FC)
RWORDSIZE       =       $(NATIVE_RWORDSIZE)
PROMOTION       =       #-fdefault-real-8
ARCH_LOCAL      =       -DNONSTANDARD_SYSTEM_SUBR  -DWRF_USE_CLM $(NETCDF4_IO_OPTS)
CFLAGS_LOCAL    =       -w -O3 -c  # -DRSL0_ONLY
LDFLAGS_LOCAL   =       
CPLUSPLUSLIB    =       
ESMF_LDFLAG     =       $(CPLUSPLUSLIB)
FCOPTIM         =       -O2 -ftree-vectorize -funroll-loops
FCREDUCEDOPT    =       $(FCOPTIM)
FCNOOPT         =       -O0
FCDEBUG         =       # -g $(FCNOOPT) # -ggdb -fbacktrace -fcheck=bounds,do,mem,pointer -ffpe-trap=invalid,zero,overflow
FORMAT_FIXED    =       -ffixed-form
FORMAT_FREE     =       -ffree-form -ffree-line-length-none
FCSUFFIX        =       
FCCOMPAT        =       
BYTESWAPIO      =       -fconvert=big-endian -frecord-marker=4
FCBASEOPTS_NO_G =       -w $(FORMAT_FREE) $(BYTESWAPIO) $(FCCOMPAT)
FCBASEOPTS      =       $(FCBASEOPTS_NO_G) $(FCDEBUG)
MODULE_SRCH_FLAG =     
TRADFLAG        =      -traditional-cpp $(NETCDF4_IO_OPTS)
CPP             =      /lib/cpp -P -nostdinc
AR              =      ar
ARFLAGS         =      ru
M4              =      m4 -G
RANLIB          =      ranlib
RLFLAGS         =
CC_TOOLS        =      $(SCC) 

   :
(snip)
   :
   :

------------------------------------------------------------------------
Settings listed above are written to configure.wrf.
If you wish to change settings, please edit that file.
If you wish to change the default options, edit the file:
     arch/configure.defaults
NetCDF users note:
 This installation of NetCDF supports large file support.  To DISABLE large file
 support in NetCDF, set the environment variable WRFIO_NCD_NO_LARGE_FILE_SUPPORT
 to 1 and run configure again. Set to any other value to avoid this message.
  

Testing for NetCDF, C and Fortran compiler

This installation of NetCDF is 64-bit
                 C compiler is 64-bit
           Fortran compiler is 64-bit
              It will build in 64-bit

standard location of RPC
*****************************************************************************
This build of WRF will use NETCDF4 with HDF5 compression
*****************************************************************************
 
==> [2022-07-12-23:36:23.527927] wrf: Executing phase: 'build'
==> [2022-07-12-23:36:23.532815] '/home/xdg-059/xdu-059001/spack/opt/spack/linux-centos7-skylake_avx512/gcc-12.1.0/tcsh-6.24.00-umqmhs76i62mqa7dhlzfb3hembcbslvd/bin/csh' './compile' '-j' '10' 'em_real'

   :
(snip)
   :
   :

==> [2022-07-12-23:55:40.140236] NO MODULE WRITTEN: list of enabled module files is empty
==> [2022-07-12-23:55:40.369065]        WRITE: wrf@4.3.3%gcc@12.1.0~chem+pnetcdf build_type=dmpar compile_type=em_real nesting=basic patches=01c0f71,68548f6,b3f063c,e07c39c,e4971f6,e6f3db3,f3dd50d,fa78635 arch=linux-centos7-skylake_avx512/4q65lk4 [/home/xdg-059/xdu-059001/spack/share/spack/modules/linux-centos7-skylake_avx512/wrf-4.3.3-gcc-12.1.0-4q65lk4]
==> [2022-07-12-23:55:41.560348] Writing manifest file: No manifest from binary
==> [2022-07-12-23:55:51.693038] wrf: Successfully installed wrf-4.3.3-4q65lk4b7juhqrqpfp4r72f5aht7uq2s
  Fetch: 0.33s.  Build: 19m 19.43s.  Total: 19m 19.77s.
   :
(snip)
   :
   :
```