Package: inplace-abn
In-Place Activated BatchNormIn-Place Activated BatchNorm for Memory-Optimized Training of DNNsIn-Place Activated BatchNorm (InPlace-ABN) is a novel approach to reduce the memory required for training deep networks.
It allows for up to 50% memory savings in modern architectures such as ResNet, ResNeXt and Wider ResNet by redefining
BN + non linear activation as a single in-place operation, while smartly dropping or recomputing intermediate buffers as
needed.This repository contains aPyTorchimplementation of the InPlace-ABN layer, as well as some
training scripts to reproduce the ImageNet classification results reported in our paper.OverviewInstallationTraining on ImageNetWe have now also released the inference code for semantic segmentation, together with the Mapillary Vistas trained model leading to#1 position on the Mapillary Vistas Semantic Segmentation leaderboard. More information can be found at the bottom of this page.CitationIf you use In-Place Activated BatchNorm in your research, please cite:@inproceedings{rotabulo2017place,title={In-Place Activated BatchNorm for Memory-Optimized Training of DNNs},author={Rota Bul\`o, Samuel and Porzi, Lorenzo and Kontschieder, Peter},booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},year={2018}}OverviewWhen processing a BN-Activation-Convolution sequence in the forward pass, most deep learning frameworks need to store
two big buffers,i.e.the inputxof BN and the inputzof Conv.
This is necessary because the standard implementations of the backward passes of BN and Conv depend on their inputs to
calculate the gradients.
Using Inplace-ABN to replace the BN-Activation sequence, we can safely discardx, thus saving up to 50% GPU memory at
training time.
To achieve this, we rewrite the backward pass of BN in terms of its outputy, which is in turn reconstructed fromzby inverting the activation function.The parametrization for the scaling factor of BN changed compared to standard BN, in order to ensure an invertible transformation. Specifically, the scaling factor becomes.RequirementsTo install PyTorch, please refer tohttps://github.com/pytorch/pytorch#installation.NOTE 1: our coderequiresPyTorch v1.1 or laterNOTE 2: we are only able to provide support for Linux platforms and CUDA versions >= 10.0NOTE 3: in general, it is not possible to load weights from a network trained with standard BN into an InPlace-ABN network without severe performance degradation, due to the different handling of BN scaling parametersTo install the package containing the iABN layers:pipinstallinplace_abnNote that some parts of InPlace-ABN have native C++/CUDA implementations, meaning that the command above will need to
compile them.Alternatively, to download and install the latest version of our library, also obtaining a copy of the Imagenet / Vistas
scripts:gitclonehttps://github.com/mapillary/inplace_abn.gitcdinplace_abn
pythonsetup.pyinstallcdscripts
pipinstall-rrequirements.txtThe last of the commands above will install some additional libraries required by the Imagenet / Vistas scripts.Training on ImageNet-1kHere you can find the results from our arXiv paper (top-1 / top-5 scores) with corresponding, trained models and md5 checksums, respectively. The model files provided below are made available under thelicense attached to ImageNet.NetworkBatch224224, 10-crops320Trained models (+md5)ResNeXt101, Std-BN25677.04 / 93.5078.72 / 94.4777.92 / 94.28448438885986d14db5e870b95f814f91ResNeXt101, InPlace-ABN51278.08 / 93.7979.52 / 94.6679.38 / 94.673b7a221cbc076410eb12c8dd361b7e4eResNeXt152, InPlace-ABN25678.28 / 94.0479.73 / 94.8279.56 / 94.672c8d572587961ed74611d534c5b2e9ceWideResNet38, InPlace-ABN25679.72 / 94.7881.03 / 95.4380.69 / 95.271c085ab70b789cc1d6c1594f7a761007ResNeXt101, InPlace-ABN sync25677.70 / 93.7879.18 / 94.6078.98 / 94.560a85a21847b15e5a242e17bf3b753849DenseNet264, InPlace-ABN25678.57 / 94.1779.72 / 94.9379.49 / 94.890b413d67b725619441d0646d663865bfResNet50v1, InPlace-ABN sync51275.53 / 92.5977.04 / 93.5776.60 / 93.492522ca639f7fdfd7c0089ba1f5f6c2e8ResNet34v1, InPlace-ABN sync51273.27 / 91.3475.19 / 92.6674.87 / 92.4261515c1484911c3cc753d405131e1ddaResNet101v1, InPlace-ABN sync51277.07 / 93.4578.58 / 94.4078.25 / 94.191552ae0f3d610108df702135f56bd27bData preparationOur script usestorchvision.datasets.ImageFolderfor loading ImageNet data, which expects folders organized as follows:root/train/[class_id1]/xxx.{jpg,png,jpeg}
root/train/[class_id1]/xxy.{jpg,png,jpeg}
root/train/[class_id2]/xxz.{jpg,png,jpeg}
...

root/val/[class_id1]/asdas.{jpg,png,jpeg}
root/val/[class_id1]/123456.{jpg,png,jpeg}
root/val/[class_id2]/__32_.{jpg,png,jpeg}
...Images can have any name, as long as the extension is that of a recognized image format.
Class ids are also free-form, but they are expected to match between train and validation data.
Note that the training data in the standard ImageNet distribution is already given in the required format, while
validation images need to be split into class sub-folders as described above.TrainingThe main training script isscripts/train_imagenet.py: this supports training on ImageNet, or any other dataset
formatted as described above, while keeping a log of relevant metrics in Tensorboard format and periodically saving
snapshots.
Most training parameters can be specified as ajson-formatted configuration file (lookherefor a complete list of configurable parameters).
All parameters not explicitly specified in the configuration file are set to their defaults, also available inscripts/imagenet/config.py.Our arXiv results can be reproduced by runningscripts/train_imagenet.pywith the configuration files inscripts/experiments.
As an example, the command to trainResNeXt101with InPlace-ABN, Leaky ReLU andbatch_size = 512is:cdscripts
python-mtorch.distributed.launch--nproc_per_node<n.GPUspernode>train_imagenet.py--log-dir/path/to/tensorboard/logsexperiments/resnext101_ipabn_lr_512.json/path/to/imagenet/rootValidationValidation is run byscripts/train_imagenet.pyat the end of every training epoch.
To validate a trained model, you can use thescripts/test_imagenet.pyscript, which allows for 10-crops validation and
transferring weights across compatible networks (e.g.fromResNeXt101with ReLU toResNeXt101with Leaky
ReLU).
This script accepts the same configuration files asscripts/train_imagenet.py, but note that thescale_valandcrop_valparameters are ignored in favour of the--scaleand--cropcommand-line arguments.As an example, to validate theResNeXt101trained above using 10-crops of size224from images scaled to256pixels, you can run:cdscripts
python-mtorch.distributed.launch--nproc_per_node<n.GPUspernode>test_imagenet.py--crop224--scale256--ten_cropsexperiments/resnext101_ipabn_lr_512.json/path/to/checkpoint/path/to/imagenet/rootUsage for Semantic Segmentation on Cityscapes and Mapillary VistasWe have successfully used InPlace-ABN with a DeepLab3 segmentation head that was trained on top of the WideResNet38
model above.
Due to InPlace-ABN, we can significantly increase the amount of input data to this model, which eventually allowed us to
obtain #1 positions onCityscapes,Mapillary Vistas,AutoNUE,KittiandScanNetsegmentation leaderboards.
The training settings mostly follow the description in ourpaper.Mapillary Vistas pre-trained modelWe release our WideResNet38 + DeepLab3 segmentation model trained on the Mapillary Vistas research set.
This is the model used to reach #1 position on the MVD semantic segmentation leaderboard.
The segmentation model file provided below is made available under aCC BY-NC-SA 4.0 license.NetworkmIOUTrained model (+md5)WideResNet38 + DeepLab353.42913f78486a34aa1577a7cd295e8a33bbTo use this, please download the.pth.tarmodel file linked above and run thetest_vistas.pyscript as follows:cdscripts
pythontest_vistas.py/path/to/model.pth.tar/path/to/input/folder/path/to/output/folderThe script will process all.png,.jpgand.jpegimages from the input folder and write the predictions in the
output folder as.pngimages.
For additional options,e.g.test time augmentation, please consult the script's help message.The results on the test data written above were obtained by employing only scale 1.0 + flipping.ChangelogUpdate 04 Jul. 2019: version 1.0.0Complete rewrite of the CUDA code following the most recent native BN implementation from PytorchImproved synchronized BN implementation, correctly handling different per-GPU batch sizes and Pytorch distributed groupsThe iABN layers are now packaged in an installable python library to simplify use in other projectsThe Imagenet / Vistas scripts are still available in thescriptsfolderRequires now PyTorch 1.1Update 08 Jan. 2019:Enabled multiprocessing and inplace ABN synchronization over multiple processes (previously using threads). It now requires to use DistributedDataParallel instead of DataParallelAdded compatibility with fp16 (currently allows fp16 input but requires the module to stay in fp32 mode)Requires now PyTorch 1.0Update Feb. 2019:Added ResNet34v1, ResNet50v1 and ResNet101v1 ImageNet-1k pre-trained modelsWe have modified the imagenet training code and BN synchronization in order to work with multiple processes. We have also added compatibility of our Inplace ABN module with fp16.