Package: cloudmesh-iu
Romeo Tensorflow InstallationGetting an account on julietPlease get a futuresystems.org account and upload the public key of your
machine to the portal. If you have issues please e-mailhelp@futuresystems.org. Please note that you must have a basic
understanding of ssh as this is a requirement to log in on any computer
these days. If you do not, please read up onssh,ssh-keygen,ssh-add,ssh keycahin.Test your access while logging intojuliet.futuersystems.orgPlease note that the account creation takes one business day.Request to be added to the GPU allowed usersPlease send mail to the GPU allowed users by sending a mail tohelp@futuresystems.orgwith yourfuturesystemsusername.
Such asPlease add me to the Romeo users allowed to use GPU's

username: <PUT YOUR USERNAME HERE>
reservation: lijguo_11Install cloudmeshTo easily accessromeo, you can use either cloudmesh or manage it via
bashrc files, or just do it by hand. The latter option is discouraged as
you may need to do it several times, and it takes a considerable effort
to activate, for example, ajupyterlabnotebook. We recommend that you
have python 3.8 or newer installed on your computer. And use the 64-bit
version. Please download it from python.org. Note that the default
version for windows is 32-bit, which does not work, so locate the 64-bit
version.$pipinstallcloudmesh-installer
$cloudmesh-installerinstalliu
$cmshelpAfter you did thecms helpcommand, you need to add theiu:block to
your cloudmesh.yaml file.cloudmesh:
  ...
  iu:
    user: gvonlasz
    host: r-003
    gpu: 0
    port: 5888
    reservation: lijguo_11You will need to modify the values for user to your futuresystems
username. The host, gpu, and port number will be given to you by the
person that you will be working with on Romeo.Please note that when others use the same port number this will not
work, so make sure you use a port number that is unique. If you find
conflicts, please negotiate with the other users.For class use your use of GPU's will be especially regulated and youneed to coordinate with your classmates the usage. Research projectstypically have priorities and could mean that the access to romeo islimited. In these cases we recommend that you use colab if possible.One line command to start Jupyter labIn a terminal execute$cmsiulabThis command will call a number of cloudmesh commands to allocate a
reservation on romeo, start port forwarding, start jupyterlab and view
the jupyter lab in a browser. It opens 2 additional terminals for the
allocation and port forwarding. To kill all of it, please remove all the
windows created by the command and call in your terminal$cmsiukillSubcommands for more controllThe one ine command is implemented using a number of subcommands.
They are explained next. If the one line command works, please use that,otherwise try to use the subcommands.Using cloudmesh to access romeo jupyterlab notebooksIn a terminal execute$cmsiuallocateThis gives you an interactive allocation in which you can startjupyterlabin the background. Next start in a new terminaljupyterlabwith$cmsiulabTo connect to it open in a new terminal a port that forwards to the
jupyterlab instance:$cmsiuportFinally, you can say in a terminal on your local machine:$cmsiuviewIf you can to kill the jupyterlab, please use:$cmsiukilland start new. Close the windows and start over.For improvement suggestions, look at the source codeand propose changes via pull requests.Installation via bashrc scriptsThis installation is significantly more involved but works for Windows
machines using gitbash. For all others, we do recommend that you use the
cloudmesh installationQuickstartThis quickstart assumes you have done all the steps discussed throughout
the document (this includes setting up your bashrc files, and installing
tensorflow). We include it here so you have an easy way to remember once
you have set up your environment how to start a notebook.Once you have set up the environment as discussed previously, you need 3 terminalsterminal 1:r-allocateterminal 2:r-jupyterterminal 3:r-port file:// ....# copy the line from terminal 2 with the file://browser: copy the url with local host in it in your browserYou will see the jupyter notebookAccountMake sure you get a futuresystems account onhttps://futuresystems.org. You will have to declare a project you
work on with us. Often this is created by the faculty member,
not the students.Make sure you have created an ssh key withssh-keygenin a
shell. On Windows use gitbash and install it the default way on your
machine. Linux and macOS have the ssh commands build inUpload the public key in~/.ssh/id_rsa.pubinto the public key
field whne going to your futuresystems account and edit it. The
account information link is placed on the bottom of the pageNow you have to wait a while till your key gets populated to juliet
and romeo. This is a process done automatically every 10 minutes, but
a system administrator has to activate your account which requires
sending a help ticketSetupThe setup is a bit complex, follow the instructions carefully. We assume
you use bash, zsh, or gitbash (in case of Windows). Other shells are not
discussed here.Host machine setupPlace the following in your .bashrc, or .zprofile or .pash_profile
(depends on your computer):# ##############################################
# BEGIN ROMEO SETUP
# ##############################################
# chose your own favourite port and host 
JPORT="9100"
JHOST="r-003"
JLOG="${HOME}/log-juliet-jupyter.txt"
JMOUNT="${HOME}/DESKTOP"
JUSER="<Your FutureSystems User Name on Juliet>"
JULIET="${JUSER}@juliet.futuresystems.org"
# its in dir juliet, please create it first

# FUNTIONS
function r-port {
    RPORT=`grep "file:" ${JLOG}`
    ssh -L ${JPORT}:r-003:${JPORT} -i ${RPORT} ${JULIET}
}

function r-open {
    RHTML=`grep "127." ${JLOG} | tail -1 | sed 's/or //g'`
    echo
    echo ${RHTML}
    echo
    /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome "${RHTML}"
}

alias r-allocate='ssh ${JULIET} "salloc -p romeo --reservation=lijguo_11"'
alias r-install='ssh -t ${JULIET} "ssh r-003 \"curl -Ls http://cloudmesh.github.io/get/romeo/tf | sh\""'
alias romeo='ssh -t ${JULIET} "ssh ${JHOST}"'

alias r='ssh -t ${JULIET} "ssh ${JHOST}"'
alias j='ssh ${JULIET}'

function r-start-jupyter {
    echo "pkill -u ${JUSER} jupyter-lab; ~/ENV3/bin/jupyter-lab --port ${JPORT} --ip 0.0.0.0 --no-browser" | ssh ${JULIET} "ssh ${JHOST}"
}

alias r-ps='echo "ps -aux| fgrep ${JUSER}" | ssh ${JULIET} "ssh ${JHOST}"'
alias r-kill='echo "echo; hostname; echo; pkill -u ${JUSER} jupyter-lab" | ssh ${JULIET} "ssh ${JHOST}"'

function r-jupyter {
    rm -f ${JLOG}
    r-start-jupyter 2>&1 | tee ${JLOG}
}

alias j-mount="cd ${JMOUNT}; sshfs ${JULIET}:shared ${JULIET} -o auto_cache ; cd ${JMOUNT}/${JULIET}"
alias j-umount="cd ${JMOUNT}; umount ${JULIET}"

alias p-mount="cd ${HOME}; sshfs ${JULIET}:ENV3 RPYTHON -o auto_cache"
alias p-umount="cd ${HOME}; umount RPYTHON"

# ##############################################
# END ROMEO SETUP
# ##############################################This provides the following commands to your-allocateto get an allocation, call once. When you close the window the
allocation is terminated and none of the commands will work wellr-installTensorflow software stack installation in your home dir on romeo ~/ENV3You have to do this only oncerThis logs you into romeojThis logs you into julietr-psThis dos a ps on romeor-killThis kills all jupyter processes on romeor-jupyterThis starts a jupyter lab notebookTo use it you need to call in a new window after you copy the line with
file:// in itr-port file://....This will establish a connection to the notebookNext, you can pates and copy the line with http:// and local host into
your browserSetup.bashrcon julietOn juliet you must include the following in your bashrc fileif ! [ "$HOSTNAME" = j-login1 ]; then
    VCUDA=10.1
    VCUDNN=v7.6.5

    VMODULE=10.1-${VCUDNN}
    module load cuda/${VCUDA}
    module load cudnn/${VMODULE}
    export CUDNN_INCLUDE_DIR=/opt/cudnn-${VCUDA}-linux-x64-${VCUDNN}/cuda/include/
    export CUDNN_LIB_DIR=/opt/cudnn-${VCUDA}-linux-x64-${VCUDNN}/cuda/lib64/
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-${VCUDA}/extras/CUPTI/lib64

fiSSHFSSometimes it is beneficial to use your local browsers to access files
on romeo. We do this at this time just via juliet and enable sharing
with sshfs. This tool is available for many OSes, and you need to install
it before using it.Gregor has placed the following additional lines in hos. bashrc file on
his local computer:alias j-mount="cd ${HOME}/Desktop; sshfs juliet:shared juliet -o auto_cache ; cd ${HOME}/Desktop/juliet"
alias j-umount="cd ${HOME}/Desktop; umount juliet"Once you say j-mount it mounts the dir juliet:~/share to a local
directory ~/Desktop/juliet. As the files on juliet are shared with romeo
they are also available there.In the terminal you simply can sayj-mountfor mount and
```j-unmount`` for ummounting.Using RomeoTo login you can now sayrCheck GPU AvailabilityTo check the availability of the GPU's saynvidia-smiTest ScriptTo test if this all works, please copy the following into a notebookand executeimportosimportwarningsimporttensorflowastfimportloggingwithwarnings.catch_warnings():warnings.filterwarnings("ignore",category=FutureWarning)logging.getLogger('tensorflow').setLevel(logging.FATAL)os.environ["TF_CPP_MIN_LOG_LEVEL"]="2"iftf.test.is_gpu_available():print("------------------------------")print("GPU AVAILABLE")names=tf.test.gpu_device_name()print("------------------------------")print("GPU Device Names")print(names)The output of the code looks like---
GPU AVAILABLE
------------------------------
GPU Device Names
/device:GPU:0GPU Test CodeThis test code may run for a long time, so you may want to interrupt
it after a while. It will put some load on the CUDA Cards and if you
usenvidia-smiyou will see the load reported.git clone git@github.com:vibhatha/mpi4tf.git
pip3 install mpi4tf
cd mpi4tf
python3 examples/model_parallel/model_parallel_v2.pyA convenient way to watch the changing load is to use in another
terminal to use the watch command$watch-n5nvidia-smiThis will repeat the monitor every 5 seconds. Please make sure you
kill this program and do not run it continuously as thenvidia-smiprogram creates unnecessary load if not absolutely neededExtended GPU Setup On Romeo for PytorchUsing Pytorch to do distributed training with MPI backend is
documented herehttps://github.com/vibhatha/PytorchExamples