Package: auto-LiRPA
auto_LiRPA: Automatic Linear Relaxation based Perturbation Analysis for Neural NetworksWhat's New?Our neural network verification toolα,β-CROWN(alpha-beta-CROWN) (usingauto_LiRPAas its core library)wonVNN-COMP 2022. Our library supports the large CIFAR100, TinyImageNet and ImageNet models in VNN-COMP 2022. (09/2022)Implementation ofgeneral cutting planes(GCP-CROWN), support of more activation functions and improved performance and scalability. (09/2022)Our neural network verification toolα,β-CROWN(alpha-beta-CROWN)wonVNN-COMP 2021with the highest total score, outperforming 11 SOTA verifiers. α,β-CROWN uses theauto_LiRPAlibrary as its core bound computation library. (09/2021)Optimized CROWN/LiRPAbound (α-CROWN) for ReLU,sigmoid,tanh, andmaxpoolactivation functions, which can significantly outperform regular CROWN bounds. Seesimple_verification.pyfor an example. (07/31/2021)Handle split constraints for ReLU neurons (β-CROWN) for complete verifiers. (07/31/2021)A memory efficient GPU implementation of backward (CROWN) bounds for
convolutional layers. (10/31/2020)Certified defense models for downscaled ImageNet, TinyImageNet, CIFAR-10, LSTM/Transformer. (08/20/2020)Adding support tocomplex vision modelsincluding DenseNet, ResNeXt and WideResNet. (06/30/2020)Loss fusion, a technique that reduces training cost of tight LiRPA bounds
(e.g. CROWN-IBP) to the same asympototic complexity of IBP, making LiRPA based certified
defense scalable to large datasets (e.g., TinyImageNet, downscaled ImageNet). (06/30/2020)Multi-GPUsupport to scale LiRPA based training to large models and datasets. (06/30/2020)Initial release. (02/28/2020)Introductionauto_LiRPAis a library for automatically deriving and computing bounds with
linear relaxation based perturbation analysis (LiRPA) (e.g.CROWNandDeepPoly) for
neural networks, which is an useful tool for formal robustness verification. We
generalize existing LiRPA algorithms for feed-forward neural networks to a
graph algorithm on general computational graphs, defined by PyTorch.
Additionally, our implementation is also automaticallydifferentiable,
allowing optimizing network parameters to shape the bounds into certain
specifications (e.g., certified defense). You can finda video ▶️ introduction
here.Our library supports the following algorithms:Backward mode LiRPA bound propagation (CROWN/DeepPoly)Backward mode LiRPA bound propagation with optimized bounds (α-CROWN)Backward mode LiRPA bound propagation with split constraints (β-CROWN)Generalized backward mode LiRPA bound propagation with general cutting plane constraints (GCP-CROWN)Forward mode LiRPA bound propagation (Xu et al., 2020)Forward mode LiRPA bound propagation with optimized bounds (similar toα-CROWN)Interval bound propagation (IBP)Hybrid approaches, e.g., Forward+Backward, IBP+Backward (CROWN-IBP),α,β-CROWN(alpha-beta-CROWN)Our library allows automatic bound derivation and computation for general
computational graphs, in a similar manner that gradients are obtained in modern
deep learning frameworks -- users only define the computation in a forward
pass, andauto_LiRPAtraverses through the computational graph and derives
bounds for any nodes on the graph.  Withauto_LiRPAwe free users from
deriving and implementing LiPRA for most common tasks, and they can simply
apply LiPRA as a tool for their own applications.  This is especially useful
for users who are not experts of LiRPA and cannot derive these bounds manually
(LiRPA is significantly more complicated than backpropagation).Technical Background in 1 MinuteDeep learning frameworks such as PyTorch represent neural networks (NN) as
a computational graph, where each mathematical operation is a node and edges
define the flow of computation:Normally, the inputs of a computation graph (which defines a NN) are data and
model weights, and PyTorch goes through the graph and produces model prediction
(a bunch of numbers):Ourauto_LiRPAlibrary conducts perturbation analysis on a computational
graph, where the input data and model weights are defined within some
user-defined ranges.  We get guaranteed output ranges (bounds):InstallationPython 3.7+ and PyTorch 1.8+ are required.
PyTorch 1.11 is recommended, although other recent versions might also work.
It is highly recommended to have a pre-installed PyTorch
that matches your system and our version requirement.
SeePyTorch Get Started.
Then you can installauto_LiRPAvia:gitclonehttps://github.com/KaidiXu/auto_LiRPAcdauto_LiRPA
pythonsetup.pyinstallIf you intend to modify this library, usepython setup.py developinstead.Optionally, you may build and install native CUDA modules (CUDA toolkit required):pythonauto_LiRPA/cuda_utils.pyinstallQuick StartFirst define your computation as ann.Moduleand wrap it usingauto_LiRPA.BoundedModule(). Then, you can call thecompute_boundsfunction
to obtain certified lower and upper bounds under input perturbations:fromauto_LiRPAimportBoundedModule,BoundedTensor,PerturbationLpNorm# Define computation as a nn.Module.classMyModel(nn.Module):defforward(self,x):# Define your computation here.model=MyModel()my_input=load_a_batch_of_data()# Wrap the model with auto_LiRPA.model=BoundedModule(model,my_input)# Define perturbation. Here we add Linf perturbation to input data.ptb=PerturbationLpNorm(norm=np.inf,eps=0.1)# Make the input a BoundedTensor with the pre-defined perturbation.my_input=BoundedTensor(my_input,ptb)# Regular forward propagation using BoundedTensor works as usual.prediction=model(my_input)# Compute LiRPA bounds using the backward mode bound propagation (CROWN).lb,ub=model.compute_bounds(x=(my_input,),method="backward")Checkoutexamples/vision/simple_verification.pyfor a complete but very basic example.We also provide aGoogle Colab Demoincluding an example of computing verification
bounds for a 18-layer ResNet model on CIFAR-10 dataset. Once the ResNet model
is defined as usual in Pytorch, obtaining provable output bounds is as easy as
obtaining gradients through autodiff. Bounds are efficiently computed on GPUs.More Working ExamplesWe providea wide range of examplesof usingauto_LiRPA:Basic Bound Computation andRobustness Verificationof Neural NetworksBasicCertified Adversarial DefenseTrainingLarge-scale Certified Defense Training onImageNetCertified Adversarial Defense Training on Sequence Data withLSTMCertifiably Robust Language Classifier usingTransformersCertified Robustness againstModel Weight Perturbationsauto_LiRPAhas also be used in the following works:α,β-CROWN for complete neural network verificationFast certified robust trainingFull DocumentationsFor more documentations, please refer to:Documentation homepageAPI documentationAdding custom operatorsGuidefor reproducingour NeurIPS 2020 paperPublicationsPlease kindly cite our papers if you use theauto_LiRPAlibrary. FullBibTeX entriescan be foundhere.The general LiRPA based bound propagation algorithm was originally proposed in our paper:Automatic Perturbation Analysis for Scalable Certified Robustness and Beyond.
NeurIPS 2020
Kaidi Xu*, Zhouxing Shi*, Huan Zhang*, Yihan Wang, Kai-Wei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, Cho-Jui Hsieh (* Equal contribution)Theauto_LiRPAlibrary is further extended to allow optimized bound (α-CROWN), split constraints (β-CROWN) and general constraints (GCP-CROWN):Fast and Complete: Enabling Complete Neural Network Verification with Rapid and Massively Parallel Incomplete Verifiers.
ICLR 2021.
Kaidi Xu*, Huan Zhang*, Shiqi Wang, Yihan Wang, Suman Jana, Xue Lin and Cho-Jui Hsieh (* Equal contribution).Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Complete and Incomplete Neural Network Verification.
NeurIPS 2021.
Shiqi Wang*, Huan Zhang*, Kaidi Xu*, Suman Jana, Xue Lin, Cho-Jui Hsieh and Zico Kolter (* Equal contribution).GCP-CROWN: General Cutting Planes for Bound-Propagation-Based Neural Network Verification.
Huan Zhang*, Shiqi Wang*, Kaidi Xu*, Linyi Li, Bo Li, Suman Jana, Cho-Jui Hsieh and Zico Kolter (* Equal contribution).Certified robust training usingauto_LiRPAis improved to allow much shorter warmup and faster training:Fast Certified Robust Training with Short Warmup.
NeurIPS 2021.
Zhouxing Shi*, Yihan Wang*, Huan Zhang, Jinfeng Yi and Cho-Jui Hsieh (* Equal contribution).Developers and CopyrightKaidi XuZhouxing ShiHuan ZhangYihan WangShiqi WangTeam lead:Huan Zhang (huan@huan-zhang.com), CMUMain developers:Zhouxing Shi (zshi@cs.ucla.edu), UCLAKaidi Xu (kx46@drexel.edu), Drexel UniversityContributors:Yihan Wang (yihanwang@ucla.edu), UCLAShiqi Wang (sw3215@columbia.edu), Columbia UniversityLinyi Li (linyi2@illinois.edu), UIUCJinqi (Kathryn) Chen (jinqic@cs.cmu.edu), CMUZhuolin Yang (zhuolin5@illinois.edu), UIUCWe thank thecommitsandpull requestsfrom community contributors.Our library is released under the BSD 3-Clause license.