PbsServerDiluvium

Instructions for installing and configuring a WMT server on diluvium for the PBS project.
--Mpiper (talk) 11:16, 30 July 2018 (MDT)
==Set install directory==
The install directory for this server is /data/web/htdocs/wmt/api/pbs.
 install_dir=/data/web/htdocs/wmt/api/pbs
 mkdir -p $install_dir
==Install Python==
==Install and configure server==
Get the server install script and run it.
 curl -O https://raw.githubusercontent.com/csdms/wmt/master/scripts/install
 python ./install --with-python=$(which python) $(pwd)
This makes a fresh set of database files.
Users will have to create a new login.
(This is something that needs to be changed.
I can't copy over the users.db file from another WMT instance
because the client sets up the user tag,
used to distinguish public and private models,
when the user creates a login.)
Set write permissions on the wsgi script:
 chmod a+w $install_dir/opt/wmt-api/wmt/scripts/wmt_wsgi_main.py
Edit conf/wmt.ini to include paths for this server.
 [paths]
 bin = bin
 templates = templates
 files = files
 uploads = files/uploads
 downloads = files/downloads
 static = static
 logs = logs
 database = %(db)s/wmt.db
 user_db = %(db)s/users.db
 submission_db = %(db)s/submission.db
 pickup = /data/ftp/pub/users/wmt
 db = db
 opt = opt
 
 [url]
 scheme = https
 netloc = csdms.colorado.edu
 path = wmt/api-permafrost
 
 [pickup]
 scheme = https
 netloc = csdms.colorado.edu
 path = pub/users/wmt
 
 [passlib]
 schemes = sha512_crypt, sha256_crypt
 sha256_crypt__default_rounds = 100000
 sha512_crypt__default_rounds = 100000
 
 [user]
 name = <your-name>
 email = <your-email>
Create files that describe the executor.
 cd $install_dir
 mkdir -p db/hosts/beach.colorado.edu/db
 cd db/hosts/beach.colorado.edu/db
 touch env.json info.json
Add info to the files.
 # env.json
 {
 "PATH": [ "/home/csdms/wmt/permafrost/conda/bin", "/bin", "/usr/bin" ]
 }
 # info.json
 {
 "name": "beach.colorado.edu",
 "username": "",
 "password": "",
 "wmt_prefix": "/home/csdms/wmt/permafrost/conda",
 "launcher": "qsub"
 }
I made a similar set of files for siwenna and blanca.
==Install software used by the server==
Install packages used by the WMT server software.
 conda install ipython numpy # for debugging
 conda install passlib paramiko pyyaml
 pip install wsgilog
 pip install requests-toolbelt
Note that WMT server software has already been installed
(at the install step above) into opt/wmt-api.
Install the PBS server package.
 cd $install_dir/opt
 git clone https://github.com/permamodel/pbs-server.git
 cd pbs-server
 python setup.py install
If permissions on the server have been set (see below),
root permission will be needed to install in the Python site-packages directory.
Use instead
 sudo $(which python) setup.py install
==Install component metadata==
Clone the `wmt-metadata` repository.
 cd $install_dir/opt
 git clone https://github.com/csdms/wmt-metadata.git
 cd wmt-metadata
Carefully read the README for this repo.
There are several environment variables that need to be set.
 export wmt_project=pbs
 export wmt_executor="siwenna.colorado.edu"
 export wmt_executor_username=wmtuser
 export wmt_executor_path=/home/csdms/wmt/$wmt_project
 export wmt_server_path=/data/web/htdocs/wmt/api/$wmt_project
Retrieve component metadata from the executor and install it on the server
with the install-components.sh script.
 bash ./scripts/install-components.sh
To run this script,
a login is required on the executor,
and sudo privileges may be required on the server.
As of the writing of this article, the following components are installed:
* BenchmarkIngestTool
* ILAMB
* ModelIngestTool
Set write permissions on the db/components directory and the parameters file for ILAMB (because of its `refresh` method):
 chmod a+w $install_dir/db/components
 chmod a+w $install_dir/db/components/ILAMB/db/parameters.json
==Activate the server==
The WMT server needs almost all of its files
to have `nobody:nobody` as the owner.
The only exceptions are
* packages installed into the opt directory, although everything under opt/wmt-api/wmt must also have `nobody` as the group owner, or models won't save correctly;
* the db/components directory and everything underneath it may have `$USER:nobody` as owner.
 cd $install_dir
 sudo chown -R nobody:nobody bin
 sudo chown -R nobody:nobody conda
 sudo chown -R nobody:nobody conf
 (more)
 sudo chown -R nobody:nobody opt/wmt-api/wmt
 sudo chown -R mapi8461:nobody db/components
Edit the Apache config file,
/usr/local/adm/config/apache/conf/httpd.conf,
to add the new server URL.
 # PBS
 WSGIDaemonProcess wmt-api-pbs threads=16 maximum-requests=10000 python-path=/data/web/htdocs/wmt/api/pbs/conda/lib/python2.7/site-packages
 WSGIProcessGroup wmt-api-pbs
 WSGIScriptAlias /wmt/api-pbs /data/web/htdocs/wmt/api/pbs/bin/wmt_wsgi_main.py/ process-group=wmt-api-pbs
 Alias /wmt/api-pbs/static /data/web/htdocs/wmt/api/pbs/static
 <Directory /data/web/htdocs/wmt/api/pbs/>
 Require all granted
 </Directory>
Restart the web server.
 sudo /usr/local/httpd/bin/apachectl -k restart
If, at a later point, components are updated,
or other changes are made,
instead of restarting the web server,
just touch the bin/wmt_wsgi_main.py file.
 cd $install_dir
 touch bin/wmt_wsgi_main.py