PermafrostExecutorSiwenna

Instructions for installing and configuring a WMT executor on siwenna.
--Mpiper (talk) 14:52, 22 February 2018 (MST)
==Set install directory==
The install directory for this executor is /home/csdms/wmt/permafrost.
 install_dir=/home/csdms/wmt/permafrost
 mkdir -p $install_dir
==Install Python==
==Install the CSDMS software stack==
A limiting factor in setting up this executor
is the componentized version of ILAMB can only be run with `pymt=0.2`;
in particular,
the BMI I wrote for ILAMB is incompatible
with the `model_metadata` package in newer versions of PyMT.
Further,
as of 2018-03-20,
wind back the `netcdf-fortran` package so that HDF5 1.8.18 is used.
There are problems with HDF5 1.10.1 and ESMF, used in PyMT.
To ensure all the packages meet the same dependencies
(looking at you, numpy),
simultaneously install `cca-tools`, `netcdf-fortran`, `pymt=0.2` and `csdms-ilamb`
using the csdms-stack conda channel (the Bakery),
 conda install cca-tools netcdf-fortran=4.4.4=5 pymt==0.2 csdms-ilamb -c csdms-stack -c defaults -c conda-forge
Install the `babelizer` for components that need to be built from source.
 conda install -c csdms-stack babelizer
Optionally install IPython for testing.
 conda install ipython
Note that when running IPython remotely on siwenna,
it's helpful to set
 export MPLBACKEND=Agg
===Install executor software===
Install the `wmt-exe` package from source.
 mkdir -p $install_dir/opt && cd $install_dir/opt
 git clone https://github.com/csdms/wmt-exe
 cd wmt-exe
 python setup.py develop
Create a site configuration file that describes the executor and symlink it to the executor's etc/ diectory.
 python setup.py configure --wmt-prefix=$install_dir --launch-dir='/scratch/$USER/wmt/permafrost' --exec-dir='/scratch/$USER/wmt/permafrost'
 ln -s "$(realpath wmt.cfg)" $install_dir/conda/etc
Here I've configured the executor to do stuff in the /scratch directory instead of the default ~/.wmt.
==Install and test CSDMS components==
Each section below
describes how to install and test a particular CSDMS component.
Currently installed components:
# BenchmarkIngestTool
# CMIP
# CRUAKTemp
# FrostNumberGeoModel
# FrostNumberModel
# ILAMB
# KuGeoModel
# KuModel
# ModelIngestTool
===ILAMB===
We installed ILAMB above.
Test it by starting a Python session
in $install_dir/test
and executing the setup and IRF methods.
 from pymt.components import ILAMB
 
 comp = ILAMB()
 comp.get_component_name()
 comp.setup('.')
At this point,
stop and edit bmi_ilamb.yaml,
updating the `ilamb_root` attribute with the appropriate location on siwenna,
/home/csdms/ilamb.
All other attributes can be left unchanged with their default values.
 comp.initialize('bmi_ilamb.yaml')
 comp.get_start_time()
 comp.get_end_time()
 comp.get_current_time()
 comp.update()
 comp.finalize()
ILAMB is run at the call to `update`.
The default run takes about 75 min.
===PBS===
Download and install the `pbs-executor` software from source.
 cd $install_dir/opt
 git clone https://github.com/permamodel/pbs-executor
 cd pbs-executor
 python setup.py develop
Babelize the PBS components.
 cd $install_dir
 mkdir -p build && cd build
 bmi-babelize $install_dir/opt/pbs-executor --prefix=$install_dir/conda
Test a component (here, ModelIngestTool) by starting a Python session
in $install_dir/test
and executing the setup and IRF methods.
 from pymt.components import ModelIngestTool
 
 comp = ModelIngestTool()
 comp.get_component_name()
 comp.setup('.')
At this point,
suspend the IPython session and copy a sample config file and data from the `pbs-executor` repo.
(This could be improved.)
 cd $install_dir/opt/pbs-executor/pbs_executor/data
 cp ingest.yaml sftlf_fx_PBS-test_historical_r9i0p0.nc $install_dir/test
 cd -
Restore the IPython session and continue.
 comp.initialize('ingest.yaml')
 comp.update()
 comp.finalize()
View the index.html file that results.
===Permamodel===
Install the permamodel components from the Bakery
using the `csdms-permamodel` metapackage.
 conda install csdms-permamodel -c csdms-stack
This metapackage contains
* csdms-permamodel-ku
* csdms-permamodel-frostnumber
* csdms-permamodel-kugeo
* csdms-permamodel-frostnumbergeo
* csdms-cruaktemp
Test a component (here, FnM) by starting a Python session
in $install_dir/test
and executing the setup and IRF methods.
 from pymt.components import FrostNumberModel
 
 comp = FrostNumberModel()
 comp.get_component_name()
 # args = comp.setup('.')
 # comp.initialize(*args)
 comp.setup('.')
 comp.initialize('frostnumber_model.cfg')
 comp.get_start_time()
 comp.get_end_time()
 comp.get_current_time()
 comp.update()
 comp.finalize()
Check the ouput of the run
 $ cat frostnumber_output.dat
 Year: 2000 output=('0.396', '-1.000', '-1.000')
===CMIP===
Download and install the `cmip` software from source.
 cd $install_dir/opt
 git clone https://github.com/permamodel/cmip
 cd cmip
 python setup.py develop
Babelize the CMIP tool.
 cd $install_dir
 mkdir -p build && cd build
 bmi-babelize $install_dir/opt/cmip --prefix=$install_dir/conda
Download the sample config file
and test the babelized CMIP component by starting a Python session in $install_dir/test
and executing the setup and IRF methods.
 from pymt.components import CMIP
 
 comp = CMIP()
 comp.get_component_name()
 comp.initialize('sample_cmip-siwenna.cfg')
 comp.get_start_time()
 comp.get_end_time()
 comp.get_current_time()
 comp.update()
 comp.get_current_time()
 comp.update_until(comp.get_end_time())
 comp.get_current_time()
 comp.finalize()