Package: loss-landscape
Visualizing the Loss Landscape of Neural NetsThis repository is a fork of theoriginal repositoryby the authors of the paperHao Li, Zheng Xu, Gavin Taylor, Christoph Studer and Tom Goldstein.Visualizing the Loss Landscape of Neural Nets. NIPS, 2018.We add simple and easy to use installation and running instructions.Aninteractive 3D visualizerfor loss surfaces has been provided bytelesens.Given a network architecture and its pre-trained parameters, this tool calculates and visualizes the loss surface along random direction(s) near the optimal parameters.
The calculation can be done in parallel with multiple GPUs per node, and multiple nodes.
The random direction(s) and loss surface values are stored in HDF5 (.h5) files after they are produced.SetupInstallationTested on Ubuntu 16.04.6 LTS with Conda 4.8.3.Option 1Runconda env create python=3.8 -f env.yml(created withconda env export -f env.yml --no-builds)Option 2Runconda create python=3.8 --name loss_landscape --file env_explicit.txt(created withconda list --explicit > env_explicit.txt)TroubleshootingIf none of the above options work:
Try to install the packages manually.
The most important packages are listed in the section Environment.Environmentpytorchopenmpimpi4pynumpyh5pymatplotlibscipyscikit-learnseabornWhat exactly do I need to do to make it work?If you have a new dataset: add a new folderdatasets/{your_dataset_name}.Add you data todatasets/{your_dataset_name}/data.Add the model definitions to a file indatasets/{your_dataset_name}/models.Add your trained network to a file indatasets/{your_dataset_name}/trained_nets/{your_model_with_hyper_parameters}.Add a filedata_loader.pyindatasets/{your_dataset_name}and implement the methodget_data_loaders(). You can find documentation indata_loader.py.Add a filemodel_loader.pyindatasets/{your_dataset_name}and implement the methodload(). Also add to the file a dictionary calledmodelscontaining a mapping between the name of your model and the model function. You can find documentation inmodel_loader.py.Examples for running itLocally without GPUImplicit (short version):pythonplot_surface.py--nametest_plot--modelresnet56--datasetcifar10--x=-1:1:51--y=-1:1:51--plot\--model_filedatasets/cifar10/trained_nets/resnet56_sgd_lr=0.1_bs=128_wd=0.0005/model_300.t7Explicit (long version):pythonplot_surface.py--nametest_plot--modelresnet56--datasetcifar10--x=-1:1:51--y=-1:1:51--plot\--model_filedatasets/cifar10/trained_nets/resnet56_sgd_lr=0.1_bs=128_wd=0.0005/model_300.t7\--dir_typeweights--xnormfilter--xignorebiasbn--ynormfilter--yignorebiasbnOn a server with 4 GPUs and 16 CPUsImplicit (short version):nohuppythonplot_surface.py--nametest_plot--modelinit_baseline_vgglike--datasetcinic10--x=-1:1:51--y=-1:1:51--plot\--model_filedatasets/cinic10/trained_nets/init_baseline_vgglike_sgd_lr=0.1_bs=128_wd=0.0005_mom=0.9_save_epoch=1_ngpu=4/model_10.t7\--cuda--ngpu4--threads8--batch_size8192>nohup.out&Explicit (long version):nohuppythonplot_surface.py--nametest_plot--modelinit_baseline_vgglike--datasetcinic10--x=-1:1:51--y=-1:1:51--plot\--model_filedatasets/cinic10/trained_nets/init_baseline_vgglike_sgd_lr=0.1_bs=128_wd=0.0005_mom=0.9_save_epoch=1_ngpu=4/model_10.t7\--cuda--ngpu4--threads8--batch_size8192\--dir_typeweights--xnormfilter--xignorebiasbn--ynormfilter--yignorebiasbn>nohup.out&Please find the description of all the possible parameters inplot_surface.py.
More examples can be found inplot_examples.sh.Make sure you do not use mpi when you run it on a single machine.Pretrained ModelsThe code accepts pre-trained PyTorch models for the CIFAR-10 and CINIC-10 datasets out of the box, but other datasets can also be added.
To load the pre-trained model correctly, the model file should containstate_dict, which is saved from thestate_dict()method.
The default path for pre-trained networks iscifar10/trained_nets.
Some of the pre-trained models and plotted figures can be downloaded here:VGG-9(349 MB)ResNet-56(10 MB)ResNet-56-noshort(20 MB)DenseNet-121(75 MB)Data preprocessingThe data pre-processing method used for visualization should be consistent with the one used for model training.
No data augmentation (random cropping or horizontal flipping) is used in calculating the loss values.Troubleshootinglibgfortran 4.0.0 does not seem to be compatible with linux.
Make sure you don't update the dependencies to include this.CitationIf you find this code useful in your research, please cite:@inproceedings{visualloss,
  title={Visualizing the Loss Landscape of Neural Nets},
  author={Li, Hao and Xu, Zheng and Taylor, Gavin and Studer, Christoph and Goldstein, Tom},
  booktitle={Neural Information Processing Systems},
  year={2018}
}