Package: convml-tt
Studying convective organisation with neural networksThis repository contains code to generate training data, train and interprete
the neural network used inL. Denby
(2020)collected in a python module calledconvml_tt. From versionv0.7.0it
was rewritten to usepytorch-lightningrather
thanfastai v1to adopt best-practices and make it
easier to modify and carry out further research on the technique.Getting startedTo use theconvml_ttcodebase you will first need to installpytorchwhich can most easily by done withconda(or usemambawhich iscondare-implemented in c++ and is orders of magnitude faster - once installed just
replacecondawithmambain the commands below).Once conda is installed you can create a conda environment:condacreate-nconvml-tt
condaactivateconvml-ttInto this conda environment you the need to install pytorch. Depending on
whether you have access to a GPU or not you will need to install different
pytorch packages:2a. For GPU-based trained and inference:condainstallpytorch"torchvision>=0.4.0"pytorch-cuda-cpytorch-cnvidia(to check that is working you can runpython -c 'import torch; print(torch.cuda.is_available())')2b. For CPU-based training and inference:condainstallpytorch"torchvision>=0.4.0"cpuonly-cpytorchWith the environment set up and pytorch installed you can now installconvml-ttdirectly frompypiusing pip (note if you
are planning on modifying theconvml-ttfunctionality you will want to
download theconvml-ttsource code and install from a local copy instead
of from pypi. Seedevelopment instructionsfor more details):python-mpipinstallconvml-ttYou will now haveconvml-ttavailable whenever you activate theconvml-ttconda environment. You will have thebasecomponents ofconvml-ttinstalled which enable training the model on a existing triplet-dataset
and making predictions with a trained model. Functionality to create training
data is contained in a separate package calledconvml-dataTrainingBelow are details on how to obtain training data and how to train the modelTraining dataExample datasetA few example training datasets can be downloaded using the following
commandpython-mconvml_tt.data.examplesModel trainingYou can use the CLI (Command Line Interface) to train the modelpython-mconvml_tt.trainerdata_dirwheredata_diris the path of the dataset you want to use. There are a number
of optional command flags available, for example to train with one GPU use
the training process toweights & biasesuse--log-to-wandb. For a list of all the available flags use the-h.Training can also be done interactively in for example a jupyter notebook, you
can see some simple examples how what commands to use by looking at the
automated tests intests/.Finally there detailed notes on how to train on the ARC3 HPC cluster at
University of Leeds are indoc/README.ARC3.md, on theJASMINanalysis cluster and onGoogle Colab.Model interpretationThere are currently two types of plots that I use for interpreting the
embeddings that the model produces. These are a dendrogram with examples
plotted for each class of the leaf nodes of the dendrogram and a scatter plot
of two dimensions annotated with example tiles so the actual tiles can be
visualised.There is an example of how to make these plots and how to easily generate an
embedding (or encoding) vector for each example tile inexample_notebooks/model_interpretation. Again this notebook expects the
directory layout mentioned above.