Package: ultimate-utils
Ultimate-utilsUlitmate-utils (or uutils) is collection of useful code that Brando has collected through the years that has been useful accross his projects.
Mainly for machine learning and programming languages tasks.Installing Ultimate-utilsWARNING: YOU HAVE TO INSTALL PYTORCH ON YOUR OWN (WITH CUDA IF YOU NEED A GPU)Standard pip install [Recommended]If you are going to use a gpu the do this first before continuing
(or check the offical website:https://pytorch.org/get-started/locally/):pipinstalltorchtorchvisiontorchaudio--extra-index-urlhttps://download.pytorch.org/whl/cu113pipinstalltorchtorchvisiontorchaudio--extra-index-urlhttps://download.pytorch.org/whl/cu113--upgradeOtherwise, just doing the following should work.pipinstallultimate-utilsIf that worked, then you should be able to import is as follows:importuutilsnote the import statement is shorter than the library name (ultimate-utilsvsuutils).Note, for an older version of uutils you might need to downgrade pytorch related stuff by doing:pipuninstalltorchtorchvisiontorchaudio
pipinstalltorch==1.9.1+cu111torchvision==0.10.1+cu111torchaudio==0.9.1-fhttps://download.pytorch.org/whl/torch_stable.htmlManual installation [for Development]To use uutils first get the code from this repo (e.g. fork it on github):git clone git@github.com:brando90/ultimate-utils.gitThen install it in development mode in your python env with python >=3.9
(readmodules_in_python.mdto learn about python envs).
E.g. create your env with conda:conda create -n uutils_env python=3.9
conda activate uutils_envThen install uutils in edibable mode and all it's depedencies with pip in the currently activated conda environment:pip install -e ~/ultimate-utils/ultimate-utils-proj-srcNo error should show up from pip.
To test the installation uutils do:python -c "import uutils; uutils.hello()"
python -c "import uutils; uutils.torch_uu.hello()"it should print something like the following:hello from uutils __init__.py in:
<module 'uutils' from '/Users/brando/ultimate-utils/ultimate-utils-proj-src/uutils/__init__.py'>


hello from torch_uu __init__.py in:
<module 'uutils.torch_uu' from '/Users/brando/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/__init__.py'>To test (any) pytorch do:python -c "import uutils; uutils.torch_uu.gpu_test_torch_any_device()"output:(meta_learning_a100) [miranda9@hal-dgx diversity-for-predictive-success-of-meta-learning]$ python -c "import uutils; uutils.torch_uu.gpu_test()"
device name: A100-SXM4-40GB
Success, no Cuda errors means it worked see:
out=tensor([[ 0.5877],
        [-3.0269]], device='cuda:0')
(meta_learning_a100) [miranda9@hal-dgx diversity-for-predictive-success-of-meta-learning]$ python -c "import uutils; uutils.torch_uu.gpu_test_torch_any_device()"
device name: A100-SXM4-40GB
Success, torch works with whatever device is shown in the output tensor:
out=tensor([[-1.9061],
        [ 1.3525]], device='cuda:0')GPU TEST: To test if pytorch works with gpu do (it should fail if no gpus are available):python -c "import uutils; uutils.torch_uu.gpu_test()"output should be something like this:(meta_learning_a100) [miranda9@hal-dgx diversity-for-predictive-success-of-meta-learning]$ python -c "import uutils; uutils.torch_uu.gpu_test()"
device name: A100-SXM4-40GB
Success, no Cuda errors means it worked see:
out=tensor([[ 0.5877],
        [-3.0269]], device='cuda:0')[Adavanced] If using pygraphviz functionsIf you plan to use the functions that depend onpygraphvizyou will likely need to installgraphvizfirst.
On mac,brew install graphviz.
On Ubuntu,sudo apt install graphviz.Then installpygraphvizwithpip install pygraphvizIf the previous steps didn't work you can also try installing using conda
(which seems to install bothpygraphviz andgraphviz`):conda install -y -c conda-forge pygraphvizto see details on that approach see the following stack overflow link question:https://stackoverflow.com/questions/67509980/how-does-one-install-pygraphviz-on-a-hpc-cluster-without-errors-even-when-graphvTo test if pygraphviz works do:python -c "import pygraphviz"Nothing should return if successful.ContributingFeel free to push code with pull request.
Please include at least 1 self-contained test (that works) before pushing.How modules are imported in a python projectRead themodules_in_python.mdto have an idea of the above development/editable installation commands.Executing tensorboard experiment logs from remotevisualize the remote logs using pycharm and my code (TODO: have the download be automatic...perhaps not needed)Download the code from the cluster using pycharm remoteThen copy paste theremote path(from pycharm, browse remote)Using the copied path runtbb path2loge.g.tbbb /home/miranda9/data/logs/logs_Mar06_11-15-02_jobid_0_pid_3657/tbto havetbbbwork as the command add to your.zshrc(or.bashrc):alias tbb="sh ${HOME}/ultimate-utils/run_tb.sh"then the commandtbb path2logshould work.ref: see fileshttps://github.com/brando90/ultimate-utils/blob/master/run_tb.shhttps://github.com/brando90/ultimate-utils/blob/master/ultimate-utils-proj-src/execute_tensorboard.pyPushing to pypiFor full details see~/ultimate-utils/tutorials_for_myself/pushing_to_pypi/README.mdFor quick push do:cd~/ultimate-utils/
rm-rfbuild
rm-rfdistcd~/ultimate-utils/
pythonsetup.pysdistbdist_wheel
twinecheckdist/*
twineuploaddist/*then:cd~/ultimate-utils/
rm-rfbuild
rm-rfdistTesting pip install with dockerCreate & pull image:docker run -ti continuumio/miniconda3then dopip install ultimate-utilsit should be installed. You can import and test the print as mentioned above.CitationIf you use this implementation consider citing us:@software{brando2021ultimateutils,
    author={Brando Miranda},
    title={Ultimate Utils - the Ultimate Utils library for Machine Learning and Artificial Intelligence},
    url={https://github.com/brando90/ultimate-utils},
    year={2021}
}A permanent link lives here:https://www.ideals.illinois.edu/handle/2142/112797Docker containerIf you have issues with docker & space
runwarning this will remove all your docker images:dockersystemprune-afTo build the docker image form the computer that will run the docker container# for adm/x86dockerpullbrandojazz/ultimate-utils:test# for adm/x86dockerpullbrandojazz/ultimate-utils:test_armor do the standard docker build for the image for an x86 machine:# for adm/x86dockerbuild-tbrandojazz/ultimate-utils:test~/ultimate-utils/# for arm/m1dockerbuild-f~/ultimate-utils/Dockerfile_arm-tbrandojazz/ultimate-utils:test_arm~/ultimate-utils/

dockerlogin# for adm/x86dockerpushbrandojazz/ultimate-utils:test# for arm/m1dockerpushbrandojazz/ultimate-utils:test_arm

dockerimagesRun container:dockerrun-tibrandojazz/ultimate-utils:test_armbashor in development mode:dockerrun-v~/ultimate-utils:/home/bot/ultimate-utils\-v~/data:/home/bot/data\-tibrandojazz/ultimate-utils:test_armbashInside the docker container:python-c"import uutils; uutils.torch_uu.gpu_test_torch_any_device()"AppendixData storage at UIUC vision cluster IChome->/home/miranda9(exportHOME=/home/miranda9,echo$HOME)code/git->home
ssh->home
pushonsave->home
.bashrc->home
conda->shared(miniconda)data&logs&ckpts->home(mkdir~/data)wandb->homebutdeletefolderaftereachrun(rm-rf$WANDB_DIR)My goal is to put the large heavy stuff (e.g. conda, data, ) at/shared/rsaas/miranda9/.
Warning: due to the vpn if you run one of this commands and you lose connection you will have to do it again and might
have half a transfer of files.
So run them in a job.sub command or re pull them from git then do a soft link.mv~/data/shared/rsaas/miranda9
ln-s/shared/rsaas/miranda9/data~/datamv~/miniconda3/shared/rsaas/miranda9
ln-s/shared/rsaas/miranda9/miniconda3~/miniconda3

mv~/data_folder_fall2020_spring2021/shared/rsaas/miranda9/
ln-s/shared/rsaas/miranda9/data_folder_fall2020_spring2021~/data_folder_fall2020_spring2021# --# mv ~/diversity-for-predictive-success-of-meta-learning /shared/rsaas/miranda9cd/shared/rsaas/miranda9/
gitclonegit@github.com:brando90/diversity-for-predictive-success-of-meta-learning.git# ln -s file1 link1ln-s/shared/rsaas/miranda9/diversity-for-predictive-success-of-meta-learning~/diversity-for-predictive-success-of-meta-learningmv~/Does-MAML-Only-Work-via-Feature-Re-use-A-Data-Set-Centric-Perspective/shared/rsaas/miranda9
ln-s/shared/rsaas/miranda9/Does-MAML-Only-Work-via-Feature-Re-use-A-Data-Set-Centric-Perspective~/Does-MAML-Only-Work-via-Feature-Re-use-A-Data-Set-Centric-Perspectivemv~/ultimate-anatome/shared/rsaas/miranda9
ln-s/shared/rsaas/miranda9/ultimate-anatome~/ultimate-anatomemv~/ultimate-aws-cv-task2vec/shared/rsaas/miranda9
ln-s/shared/rsaas/miranda9/ultimate-aws-cv-task2vec~/ultimate-aws-cv-task2vec# mv ~/ultimate-utils /shared/rsaas/miranda9cd/shared/rsaas/miranda9/
gitclonegit@github.com:brando90/ultimate-utils.git
ln-s/shared/rsaas/miranda9/ultimate-utils~/ultimate-utilsmv~/pycoq/shared/rsaas/miranda9
ln-s/shared/rsaas/miranda9/pycoq~/pycoqmv~/rfs/shared/rsaas/miranda9
ln-s/shared/rsaas/miranda9/rfs~/rfsmv~/automl-meta-learning/shared/rsaas/miranda9
ln-s/shared/rsaas/miranda9/automl-meta-learning~/automl-meta-learningmv~/wandb/shared/rsaas/miranda9
ln-s/shared/rsaas/miranda9/wandb~/wandbcd/shared/rsaas/miranda9/
gitclonegit@github.com:brando90/proverbot9001.git
ln-s/shared/rsaas/miranda9/proverbot9001~/proverbot9001to check real path (for soft links) do e.g.realpath /home.SNAP serversLogging in/sshing:sshbrando9@whale.stanford.edu
mkdir/dfs/scratch0/brando9cd/dfs/scratch0/brando9# ln -s file1 link1#ln -s /dfs/scratch0/brando9 /afs/cs.stanford.edu/u/brando9/dfs/scratch0/brando9# -- Titan X, 12 GBsshbrando9@hyperion1.stanford.edu
sshbrando9@hyperion3.stanford.edu# -- 2080, 11 GBsshbrando9@turing1.stanford.edu
sshbrando9@turing2.stanford.edu
sshbrando9@turing3.stanford.edu
sshbrando9@turing4.stanford.edu# -- A4000 16 GB [SK]mercury1
mercury2# -- RTX 8000 48GB, local storage 11TBsshbrando9@hyperturing1.stanford.edu
sshbrando9@hyperturing2.stanford.edu
sshbrando9@hyperturing3.stanford.edu
sshbrando9@hyperturing4.stanford.edu# -- a100 80GB, local storage 56TBsshbrando9@ampere1.stanford.edu
...
sshbrando9@ampere4.stanford.edu# -- a100 80GB, local storage ... [SK]sshbrando9@...stanford.eduMy folder set up:chmoda+x~/diversity-for-predictive-success-of-meta-learning/main.sh#chmod a+x /shared/rsaas/miranda9/diversity-for-predictive-success-of-meta-learning/main.shHOME->/dfs/scratch0/brando9/sothatckpting/loggingworks(wanbdlogstolocallfs)AFS->/afs/cs.stanford.edu/u/brando9,exportAFS=/afs/cs.stanford.edu/u/brando9,aliasafs='cd $AFS'(wherecodeis,needtofixdata->symlinktodfsdatafolder)bashrc.user->afswithsymlynktodfs(sinceHOMEisatdfs),cat~/.bashrc.user
ssh->fornowinafsbecausepreviousstuffbrokewhenitwasindfs,needtotestalittlemore
code/git->codein/afs/cs.stanford.edu/u/brando9,sothatpush/sftponsaveworksinpycharm.Symlinkthemtoafssothattheyarevisibleathome=dfs/...
pushonsave->rootofprojs/afs/cs.stanford.edu/u/brando9,makesureifyouhaverootsetautomaticallythatyougivetherelativepathonthedeploymentmapping(avoidputtingrootofhpctwicebyaccident)wandb->tolocallfsofcluster,sincethatfilereallydoesntmattertome,justhastobesomewheresowandbworks,seeecho$LOCAL_MACHINE_PWDor/andls$LOCAL_MACHINE_PWD,(ifitfailsputtodfs)# conda ->  dfs and two back upsconda->/dfs/scratch0/brando9soanyserverhasaccesstoit,plustheyarebigsodontwanttooverwhelmafs(doessymlinkingcondatoafsmakessense?),ls/dfs/scratch0/brando9/miniconda/envs&python-c"import uutils;uutils.get_home_pwd_local_machine_snap()"shouldworkdata->/dfs/scratch0/brando9/butwithasymlinkto/afs/cs.stanford.edu/u/brando9/data,TODO:https://intellij-support.jetbrains.com/hc/en-us/requests/4447850# ln -s file1 link1ln-s/afs/cs.stanford.edu/u/brando9/data/dfs/scratch0/brando9/dataGetting started:# - create homemkdir/dfs/scratch0/brando9
ls/dfs/scratch0/brando9# note, export sets it for current shell and all subshells created# - cd to home dfscd/dfs/scratch0/brando9exportHOME=/dfs/scratch0/brando9# - to edit .bashrc.uservim/afs/cs.stanford.edu/u/brando9/.bashrc.user# ln -s file1 link1ln-s/afs/cs.stanford.edu/u/brando9/.bashrc.user~/.bashrc.user# - get hostname without stanford to set lfs for wandb# export LOCAL_MACHINE_PWD=$(python -c "import uutils;uutils.get_home_pwd_local_machine_snap()")exportHOSTNAME=$(hostname)exportLOCAL_MACHINE_PWD="/lfs/${HOSTNAME::-13}/0/brando9"echoLOCAL_MACHINE_PWD=$LOCAL_MACHINE_PWDInstalling conda:# - in dfs when home is dfsecho$HOMEwgethttps://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh-O~/miniconda.sh
bash~/miniconda.sh-b-p$HOME/miniconda
ls-lah~exportPATH="$HOME/miniconda/bin:$PATH"echo$PATHcondasource~/miniconda/bin/activate
condainit
condainitbash
condaupdate-nbase-cdefaultsconda
condainstallconda-build

condacreate-nmetalearning_gpupython=3.9
condaactivatemetalearning_gpu
condacreate-niit_synthesispython=3.9
condaactivateiit_synthesis
condalist# - in afs when home is dfs but miniconda is installed to afs# WARNING (DISK QUOTA, DONT USE)echo$AFSwgethttps://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh-O$AFS/miniconda.sh
bash$AFS/miniconda.sh-b-p$AFS/miniconda
ls-lah$AFSexportPATH="$AFS/miniconda/bin:$PATH"echo$PATHcondasource$AFS/miniconda/bin/activate
condainit
condainitbash
condaupdate-nbase-cdefaultsconda
condainstallconda-build

condacreate-nmetalearning_gpupython=3.9
condaactivatemetalearning_gpu
condainfo-e

condacreate-niit_synthesispython=3.9
condaactivateiit_synthesis
condainfo-e

condalist# - installing full anacondaecho$HOMEwgethttps://repo.anaconda.com/archive/Anaconda3-2022.10-Linux-x86_64.sh-O~/anaconda.sh#wget https://repo.continuum.io/conda/Anaconda3-latest-Linux-x86_64.sh -O ~/anaconda.sh#bash ~/anaconda.sh -b -p $HOME/anacondanohupbash~/anaconda.sh-b-p$HOME/anaconda>anaconda_install.out&tail-fanaconda_install.out
ls-lah~exportPATH="$HOME/anaconda/bin:$PATH"echo$PATHcondasource~/anaconda/bin/activate
condainit
condainitbash
condaupdate-nbase-cdefaultsconda
condainstallconda-build

condacreate-nmetalearning_gpupython=3.9
condaactivatemetalearning_gpu

nohupsh~/diversity-for-predictive-success-of-meta-learning/install.sh>div_install.out&tail-fdiv_install.out

nohupsh~/diversity-for-predictive-success-of-meta-learning/install.sh>div_install_miniconda.out&tail-fdiv_install_miniconda.out

condacreate-niit_synthesispython=3.9
condaactivateiit_synthesis
condalist# - making a backup# cp -R <source_folder> <destination_folder>nohupcp-Ranacondaanaconda_backup&nohupcp-Rminicondaminiconda_backup&# /dfs/scratch0/brando9nohupcp-R/dfs/scratch0/brando9/anaconda/dfs/scratch0/brando9/anaconda_backup&nohupcp-R/dfs/scratch0/brando9/miniconda/dfs/scratch0/brando9/miniconda_backup&nohuppipinstall-e~/ultimate-utils/&nohuppipinstall-e~/diversity-for-predictive-success-of-meta-learning/&mkdir/lfs/madmax/0/brando9
nohupcp-R~/anaconda/lfs/madmax/0/brando9/anaconda_backup&nohupcp-R~/miniconda/lfs/madmax/0/brando9/miniconda_backup&# - in case you need the path if you broke itexportPATH=/afs/cs.stanford.edu/u/brando9/miniconda/bin:/afs/cs.stanford.edu/u/brando9/miniconda/bin:/afs/cs.stanford.edu/u/brando9/miniconda/condabin:/afs/cs.stanford.edu/u/brando9/miniconda/bin:/dfs/scratch0/brando9/miniconda/bin:/usr/local/cuda-11.1/bin:/home/miranda9/miniconda3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/afs/cs/software/sbin:/afs/cs/software/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/dfs/scratch0/brando9/my_bins# Use this command to check your afs quota and usagefslq~/
fslq$AFS# - try using pyenvcondadeactivatemetalearning_gpu
virtualenv-p/usr/local/bin/python3.pmetalearning_gpu_pyenvsourcemetalearning_gpu_pyenv/bin/activate
whichpip

nohupsh~/diversity-for-predictive-success-of-meta-learning/install.sh>div_install_pyenv.out&tail-fdiv_install_pyenv.outGit cloning your code & SHH keys:echo$HOMEmkdir/afs/cs.stanford.edu/u/brando9/.sshcd/afs/cs.stanford.edu/u/brando9/.ssh#touch ~/.ssh/id_ed25519ssh-keygen-ted25519-C"brandojazz@gmail.com"# copy paste the bellow path (needs absolute path to $HOME unfortuantely)### (no for now) /dfs/scratch0/brando9/.ssh/id_ed25519# just enter send it to /afs/cs.stanford.edu/u/brando9/.ssh/id_ed25519# press enter twice or type passphrase twiceeval"$(ssh-agent-s)"### (no for now) ssh-add /dfs/scratch0/brando9/.ssh/id_ed25519ssh-add/afs/cs.stanford.edu/u/brando9/.ssh/id_ed25519# add public key to your github### (no for now) cat /dfs/scratch0/brando9/.ssh/id_ed25519.pubcat/afs/cs.stanford.edu/u/brando9/.ssh/id_ed25519.pub# copy contents of terminal# paste to your github keys under settings under SSH and GPG keysgitclone projs:gitclonegit@github.com:brando90/ultimate-utils.git
gitclonegit@github.com:brando90/diversity-for-predictive-success-of-meta-learning.git
gitclonegit@github.com:brando90/pycoq.git
gitclonegit@github.com:FormalML/iit-term-synthesis.git
gitclonegit@github.com:brando90/proverbot9001.git
gitcheckoutdevelop
mkdir/dfs/scratch0/brando9/data
mkdir/dfs/scratch2/brando9/data

/afs/cs.stanford.edu/u/brando9
/dfs/scratch0/brando9# - DFS setup# ln -s file1 link1ln-s/afs/cs.stanford.edu/u/brando9/ultimate-utils/dfs/scratch0/brando9/ultimate-utils
ln-s/afs/cs.stanford.edu/u/brando9/diversity-for-predictive-success-of-meta-learning/dfs/scratch0/brando9/diversity-for-predictive-success-of-meta-learningln-s/afs/cs.stanford.edu/u/brando9/pycoq/dfs/scratch0/brando9/pycoqln-s/afs/cs.stanford.edu/u/brando9/iit-term-synthesis/dfs/scratch0/brando9/iit-term-synthesisln-s/dfs/scratch0/brando9/data/afs/cs.stanford.edu/u/brando9/dataln-s/afs/cs.stanford.edu/u/brando9/proverbot9001/dfs/scratch0/brando9/proverbot9001# lfs conda lfs -> dfs conda# ln -s file1 link1ln-s/dfs/scratch0/brando9/anaconda/dfs/scratch0/brando9/proverbot9001# LFS setup# ln -s file1 link1echo$HOMEln-s/afs/cs.stanford.edu/u/brando9/ultimate-utils$HOME/ultimate-utils
ln-s/afs/cs.stanford.edu/u/brando9/diversity-for-predictive-success-of-meta-learning$HOME/diversity-for-predictive-success-of-meta-learningln-s/afs/cs.stanford.edu/u/brando9/pycoq$HOME/pycoqln-s/afs/cs.stanford.edu/u/brando9/iit-term-synthesis$HOME/iit-term-synthesis# ln -s /dfs/scratch0/brando9/data $HOME/datamkdir-p$HOME/data
ln-s/afs/cs.stanford.edu/u/brando9/proverbot9001$HOME/proverbot9001Using kerberos tmux (https://ilwiki.stanford.edu/doku.php?id=hints:long-jobs):
note:need to run krbtmux per serverto reattach and other tmux commands use tmux prefix not krbtmux# - run expt with krbtmuxssh...

top-ubrando9
tmuxls
pkill-9tmux-ubrando9;pkill-9krbtmux-ubrando9;pkill-9reauth-ubrando9;pkill-9python-ubrando9;pkill-9wandb-service*-ubrando9;# - start krbtmuxkrbtmux
reauthcddiversity-for-predictive-success-of-meta-learning#source ~/.bashrc.user# now you have a krbtmux that won't die, so you can 0. perhaps running reauth inside the main_krbtmux.sh script 1. run a python job for each krbtmux or 2. run multiple python jobs inside this krbtmux session# - run exptshmain_krbtmux.sh&# - end of run expttmuxdetach
tmuxattach-t0## --# if the above doesn't work then I will have to have a script that sets up the environment variables correctly for me & then manually type the python main:# sh main_setup_jobid_and_out_files.sh# python main.py &# sh echo_jobid_and_out_files.sh# type password, later how to pass it from terminal automatically https://unix.stackexchange.com/questions/724880/how-to-run-a-job-in-the-background-using-tmux-without-interacting-with-tmux-lik# note that settip up the jobid & outfile for err and stdout can be done within python too if the current sh main_krbtmux.sh & doesn't work and we move to python main.py &# -- some useful tmux commands & keystrokestmuxnew-smysession
tmuxkill-session-tmysession
tmuxswitch-t<sessionnameornumber>
tmuxattach-t<sessionnumber>
tmuxls
tmuxdetach

C-b)=nextsession
C-b(=previoussession
C-b[=toscrollhistoryC-bd=detachfromtmuxwithoutclosing/killingthesessionbutreturnnormallytotheterminal:)todo:https://unix.stackexchange.com/questions/724880/how-to-run-a-job-in-the-background-using-tmux-without-interacting-with-tmux-likUsing nohup for long running jobs in snap TODO:https://unix.stackexchange.com/questions/724902/how-does-one-send-new-commands-to-run-to-an-already-running-nohup-process-or-runTODObellowdoesn't work##nohup sh -c 'echo$SU_PASSWORD|/afs/cs/software/bin/reauth;python-u~/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_diversity_with_task2vec.py--manual_loads_namediversity_ala_task2vec_hdb1_mio>$OUT_FILE2>$ERR_FILE'>$PWD/main.sh.nohup.out$SLURM_JOBID&todo: perhaps opening akrbtmuxand running commands there e.g.python main.sh &would work? (perhaps we don't even need nohup)Using gpus snap:https://ilwiki.stanford.edu/doku.php?id=hints:gpusourcecuda11.1# To see Cuda version in usenvcc-VCheck disk space/quota:df-h/dfs/scratch0TODO, to reuse code, have something that checks
the name of the cluster (if it's one of SNAPS, i.e. put the prefixes of snap ampere etc
and run the code bellow, else do nothing let slurm, condor do it):importosos.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"os.environ["CUDA_VISIBLE_DEVICES"]="0"# likely as above, set HOME on the script you are running to log correctlyCreating folders in scratch with lots of space.cd/dfs/scratch0/mkdirbrando9cd/dfs/scratch0/brando9check quota in afs(metalearning_gpu)brando9~$fslq$AFSVolumeNameQuotaUsed%UsedPartition
user.brando95000000196330139%6%weird bug activate doesn't workexportPATH="$HOME/miniconda/bin:$PATH"echo$PATHsource~/miniconda/bin/activate
condaactivatemetalearning_gpu
conda#mv activate.c~ activate.c#mv deactivate.c~ deactivate.c#mv conda.c~ conda.c##gcc -Wall activate.c -o activate#gcc -Wall deactivate.c -o deactivate#gcc -Wall conda.c -o condashactivate.c
shdeactivate.c
pythonconda.cSeems current meta-learning uses:torch                     1.9.1+cu111              pypi_0    pypilet's try for iit# pip3 install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html
#pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html
pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113
pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113 --upgradeInstalling ruby (snap)mkdir~/.rbenvcd~/.rbenv
gitclonehttps://github.com/rbenv/rbenv.git.exportPATH="$HOME/.rbenv/bin:$PATH"eval"$(rbenvinit-)"echo'export PATH="$HOME/.rbenv/bin:$PATH"'>>~/.bashrc.userecho'eval "$(rbenv init -)"'>>~/.bashrc.user#    exec $SHELLbash

rbenv-v# - install ruby-buildmkdir~/.ruby-buildcd~/.ruby-build
gitclonehttps://github.com/rbenv/ruby-build.git.exportPATH="$HOME/.ruby-build/bin:$PATH"echo'export PATH="$HOME/.ruby-build/bin:$PATH"'>>~/.bashrc.user#    exec $SHELLbash

ruby-build--version# - install ruby without sudo -- now that ruby build was installmkdir-p~/.local#    ruby-build 3.1.2 ~/.local/rbenvinstall3.1.2
rbenvglobal3.1.2

ruby-vInstallation opam (snap)Conda install failed so I did this one:# - opam (snap, no sudo)# ref: https://askubuntu.com/questions/339/how-can-i-install-a-package-without-root-accessapt-getdownloadopam#apt-get download opam_1.2.2-4_amd64#ls | lessmkdir-p~/.local
dpkg-xopam_1.2.2-4_amd64.deb~/.local/binexportPATH="$HOME/.local/bin:$PATH"echo'export PATH="$HOME/.local/bin:$PATH"'>>~/.bashrc.user

tr':''\n'<<<"$PATH"opam--versionLFSidea2:uselfs,simplest
-repos:symlinklfs->afs-condasymlink:lfs->afs-later,migratetominiconda(overwinterbreak)-data:lfs->dfs(effortless)-mybins/locals:inlfs-opam-ruby
-deploymentpath:toampere4->dfs(effortless)-sincedataispointingtodfs(effortless)nothingtobedone-onceitchangestolfsthenwechangethedeploymentpath-onlyneedhyperion1toworkforrepos/codesendingsincecodeisinafs&lfs->afsforthoseonthesnapconfig
-[soundsconfusing,solvingaproblemIdon'thave,maybecreatingthedatasetstodfsisfine-onlyneeddatainpycharmtopointtolfs(notdfs)]# ln -s file1 link1ln-s/afs/cs.stanford.edu/u/brando9/.bashrc.user/dfs/scratch0/brando9/.bashrc.user
ln-s/afs/cs.stanford.edu/u/brando9/.bashrc.lfs/dfs/scratch0/brando9/.bashrc.lfsCPUref:https://ilwiki.stanford.edu/doku.php?id=snap-servers:snap-servers#compute_servers