Package: pimms-learn
PIMMSPIMMS stands for Proteomics Imputation Modeling Mass Spectrometry
and is a hommage to our dear British friends
who are missing as part of the EU for far too long already
(Pimms is also a British summer drink).The pre-print is availableon biorxiv.PIMMSwas calledvaepduring development.Before entire refactoring has to been completed the imported package will bevaep.We provide functionality as a python package, an excutable workflow and notebooks.The models can be used with the scikit-learn interface in the spirit of other scikit-learn imputers. You can try this in colab.Python packageFor interactive use of the models provided in PIMMS, you can use ourpython packagepimms-learn.
The interface is similar to scikit-learn.pip install pimms-learnThen you can use the models on a pandas DataFrame with missing values. Try this in the tutorial on Colab:Notebooks as scripts using papermillIf you want to run a model on your prepared data, you can run notebooks prefixed with01_, i.e.project/01_*.ipynbafter cloning the repository. Using jupytext also python percentage script versions
are saved.cd project # project folder as pwd
papermill 01_0_split_data.ipynb --help-notebook
papermill 01_1_train_vae.ipynb --help-notebookMisstyped argument names won't throw an error when using papermillSetup for PIMMS comparison workflowThe package is available as a standalone software on pypi. However, running the entire snakemake workflow in enabled using
conda (or mamba) and pip to setup the environment. For a detailed description of setting up
conda (or mamba), seeinstructions on setting up a virtual environment.Download the repositorygit clone https://github.com/RasmussenLab/pimms.git
cd pimmsUsing conda (or mamba), install the dependencies and the package in editable mode# from main folder of repository (containing environment.yml)
conda env create -n pimms -f environment.yml # slower
mamba env create -n pimms -f environment.yml # faster, less then 5minsIf on Mac M1, M2 or having otherwise issue using your accelerator (e.g. GPUs): Install the pytorch dependencies first, then the rest of the environment.Install development dependenciesCheck how to install pytorch for your systemhere.select the version compatible with your cuda version if you have an nvidia gpucondacreate-nvaep_manuelpython=3.8pip
condaactivatevaep_manuel
condaupdatepytorch==1.13.1torchvision==0.14.1torchaudio==0.13.1pytorch-cuda=11.7-cpytorch-cnvidia# might be differentpipinstall.# pimms-learnpipinstallpapermilljupyterlab# use run notebook interactive or as a scriptcdproject
papermill04_1_train_pimms_models.ipynb04_1_train_pimms_models_test.ipynb# second notebook is outputpython04_1_train_pimms_models.ipynb# just execute the code# jupyter lab # open 04_1_train_pimms_models.ipynbEntire development installationcondacreate-npimms_dev-cpytorch-cnvidia-cfastai-cbioconda-cplotly-cconda-forge--filerequirements.txt--filerequirements_R.txt--filerequirements_dev.txt
pipinstall-e.# other pip dependencies missingsnakemake--configfileconfig/single_dev_dataset/example/config.yaml-F-nor if you want to update an existing environmentconda update  -c defaults -c conda-forge -c fastai -c bioconda -c plotly --file requirements.txt --file requirements_R.txt --file requirements_dev.txtor using the environment.yml file (can fail on certain systems)conda env create -f environment.ymlTroubleshootingTrouble shoot your R installation by opening jupyter lab# in projects folder
jupyter lab # open 01_1_train_NAGuideR.ipynb

## Run Demo

Change to the [`project` folder](./project) and see it's [README](project/README.md)

> Currently there are only notebooks and scripts under `project`, 
> but shared functionality will be added under `vaep` folder-package: This can 
> then be imported using `import vaep`. See [`vaep/README.md`](vaep/README.md)

You can subselect models by editing the config file:  [`config.yaml`](project/config/single_dev_dataset/proteinGroups_N50/config.yaml) file.conda activate pimms # activate virtual environment
cd project # go to project folder
pwd # so be in ./pimms/project
snakemake -c1 -p -n # dryrun demo workflow
snakemake -c1 -pThe demo will run an example on a small data set of 50 HeLa samples (protein groups):
  1. it describes the data and does create the splits based on the [example data](project/data/dev_datasets/HeLa_6070/README.md)
     - see `01_0_split_data.ipynb`
  2. it runs the three semi-supervised models next to some default heuristic methods
     - see `01_1_train_collab.ipynb`, `01_1_train_dae.ipynb`, `01_1_train_vae.ipynb`
  3. it creates an comparison
     - see `01_2_performance_plots.ipynb`

The results are written to `./pimms/project/runs/example`, including `html` versions of the 
notebooks for inspection, having the following structure:│   01_0_split_data.html
│   01_0_split_data.ipynb
│   01_1_train_collab.html
│   01_1_train_collab.ipynb
│   01_1_train_dae.html
│   01_1_train_dae.ipynb
│   01_1_train_vae.html
│   01_1_train_vae.ipynb
│   01_2_performance_plots.html
│   01_2_performance_plots.ipynb
│   data_config.yaml
│   tree_folder.txt
|---data
|---figures
|---metrics
|---models
|---predsThe predictions of the three semi-supervised models can be found under `./pimms/project/runs/example/preds`.
To combine them with the observed data you can run

```python
# ipython or python session
# be in ./pimms/project
folder_data = 'runs/example/data'
data = vaep.io.datasplits.DataSplits.from_folder(
    folder_data, file_format='pkl')
observed = pd.concat([data.train_X, data.val_y, data.test_y])
# load predictions for missing values of a certain model
model = 'vae'
fpath_pred = f'runs/example/preds/pred_real_na_{model}.csv '
pred = pd.read_csv(fpath_pred, index_col=[0, 1]).squeeze()
df_imputed = pd.concat([observed, pred]).unstack()
# assert no missing values for retained features
assert df_imputed.isna().sum().sum() == 0
df_imputedAvailable imputation methodsPackages either are based on this repository, or were referenced by NAGuideR (Table S1).
From the brief description in the table the exact procedure is not always clear.MethodPackagesourcestatusnameCFpimmspipCollaborative FilteringDAEpimmspipDenoising AutoencoderVAEpimmspipVariational AutoencoderZERO--replace NA with 0MINIMUM--replace NA with global minimumCOLMEDIANe1071CRANreplace NA with column medianROWMEDIANe1071CRANreplace NA with row medianKNN_IMPUTEimputeBIOCONDUCTORk nearest neighbor imputationSEQKNNSeqKnntar fileSequential k- nearest neighbor imputationstarts with feature with least missing values and re-use imputed values for not yet imputed featuresBPCApcaMethodsBIOCONDUCTORBayesian PCA missing value imputationSVDMETHODpcaMethodsBIOCONDUCTORreplace NA initially with zero, use k most significant eigenvalues using Singular Value Decomposition for imputation until convergenceLLSpcaMethodsBIOCONDUCTORLocal least squares imputation of a feature based on k most correlated featuresMLEnormCRANMaximum likelihood estimationQRILCimputeLCMDCRANquantile regression imputation of left-censored data, i.e. by random draws from a truncated distribution which parameters were estimated by quantile regressionMINDETimputeLCMDCRANreplace NA with q-quantile minimum in a sampleMINPROBimputeLCMDCRANreplace NA by random draws from q-quantile minimum centered distributionIRMVIMCRANiterativ robust model-based imputation (one feature at at time)IMPSEQrrcovNACRANSequential imputation of missing values by minimizing the determinant of the covariance matrix with imputed valuesIMPSEQROBrrcovNACRANSequential imputation of missing values using robust estimatorsMICE-NORMmiceCRANMultivariate Imputation by Chained Equations (MICE) using Bayesian linear regressionMICE-CARTmiceCRANMultivariate Imputation by Chained Equations (MICE) using regression treesTRKNN-scripttruncation k-nearest neighbor imputationRFmissForestCRANRandom Forest imputation (one feature at a time)PI--Downshifted normal distribution (per sample)GSIMP-scriptQRILC initialization and iterative Gibbs sampling with generalized linear models (glmnet)MSIMPUTEmsImputeBIOCONDUCTORMissing at random algorithm using low rank approximationMSIMPUTE_MNARmsImputeBIOCONDUCTORMissing not at random algorithm using low rank approximationgrrDreamAI-Fails to installRigde regressionGMSGMSimputetar fileFails on WindowsLasso modelBuild status