Package: bound-propagation
Bound propagationLinear and interval bound propagation in Pytorch with easy-to-use API and GPU support.
Initially made as an alternative tothe original CROWN implementationwhich featured only Numpy, lots of for-loops, and a cumbersome API.
This library is comparable toauto_LiRPAbut differs in design philosophy - auto_LiRPA parses the ONNX computation graph and operates on generic computation nodes.
The downside of this approach is a giant violation of the Single Resposibility Principle and the Open/Closed Principle of theSOLIDdesign principles.
For this library, we instead assume that the network is structured in a tree of nn.Modules, which we map to BoundModules using a visitor and abstract factory pattern exploiting Python's type system.
This allows better separation of concerns and extensibility (seeNew Modulesbelow), and improved mental map of the library.To install:pip install bound-propagationSupported bound propagation methods:Interval Bound Propagation (IBP)CROWNCROWN-IBPFor the examples below assume the following network definition:fromtorchimportnnfrombound_propagationimportBoundModelFactory,HyperRectangleclassNetwork(nn.Sequential):def__init__(self):in_size=30classes=10super().__init__(nn.Linear(in_size,16),nn.Tanh(),nn.Linear(16,16),nn.Tanh(),nn.Linear(16,classes))net=Network()factory=BoundModelFactory()net=factory.build(net)The method also works withnn.Sigmoid,nn.ReLU,nn.Identity, and custom non-linear functions:Exp,Log,Reciprocal,Sin,Cos,Pow,UnivariateMonomial,MultivariateMonomial,Clamp.
The following bivariate functions are supported:Add,Sub,Div, andMul, and their vectorized equivalents.
In addition, we have the following convenience functions:Residual,Cat,Parallel,Select,Flip,FixedLinear, andElementwiseLinear.
Initial efforts wrt. probability functions such as normal distribution PDF and CDF, and the error function has begun (consultprobability.pyfor more info).Interval boundsTo get interval bounds for either IBP, CROWN, or CROWN-IBP:x=torch.rand(100,30)epsilon=0.1input_bounds=HyperRectangle.from_eps(x,epsilon)ibp_bounds=net.ibp(input_bounds)crown_bounds=net.crown(input_bounds).concretize()crown_ibp_bounds=net.crown_ibp(input_bounds).concretize()alpha_crown_bounds=net.crown(input_bounds,alpha=True).concretize()alpha_crown_ibp_bounds=net.crown_ibp(input_bounds,alpha=True).concretize()The parameteralpha=Trueenables alpha-CROWN, which means bounds are optimized using projected gradient descent at the cost of more computation.Linear boundsTo get linear bounds for either CROWN or CROWN-IBP:x=torch.rand(100,30)epsilon=0.1input_bounds=HyperRectangle.from_eps(x,epsilon)crown_bounds=net.crown(input_bounds)crown_ibp_bounds=net.crown_ibp(input_bounds)alpha_crown_bounds=net.crown(input_bounds,alpha=True)alpha_crown_ibp_bounds=net.crown_ibp(input_bounds,alpha=True)If lower or upper bounds are not needed, then you can addbound_lower=Falseorbound_lower=Trueto avoid the unnecessary computation.
This also works for interval bounds with CROWN and CROWN-IBP.
IBP accepts the two parameters for compatibility but ignores them since both bounds are necessary for the forward propagation.
Parameteralpha=Trueis explained underInterval bounds.New ModulesTo show how to design a new module, we use a residual module as a running example (seeresidual.pyfor the full code).
Adding a new module revolves around the assumption that each module as one input and one output, but a module can have submodules such as the case for residual.classResidual(nn.Module):def__init__(self,subnetwork):super().__init__()self.subnetwork=subnetworkdefforward(self,x):returnself.subnetwork(x)+xNext, we build the corresponding BoundModule. The first parameter of the BoundModule is the nn.Module, in this case Residual, and the second is a BoundModelFactory.
Why factory? Because we do not know the structure of the subnetwork, and this could contain new network types defined by users of this library.
So the factory allows us to defer defining how to construct the BoundModule for the subnetwork, and the construction happens by callingfactory.build(subnetwork)(possibly multiple times if there are multiple subnetworks).classBoundResidual(BoundModule):def__init__(self,model,factory,**kwargs):super().__init__(model,factory,**kwargs)self.subnetwork=factory.build(model.subnetwork)defpropagate_size(self,in_size):out_size=self.subnetwork.propagate_size(in_size)assertin_size==out_sizereturnout_sizeA small note here is the need to implementpropagate_size- some modules may need to know the input and output size without this being available on the nn.Module.
Hence we propagate the size, and even though we know the output size from the input size forBoundResidual, we still propagate through the subnetwork to store this info throughout the bound graph.Next, we consider linear relaxation for linear bounds - if you only use IBP, you can skip these. BoundModules must implementneed_relaxationas a property andclear_relaxationas a method, and if the module requires relaxation (i.e. not linear and no submodules) it must implementbackward_relaxationtoo.
Note here that we let the subnetwork handle theneed_relaxationsinceResidualitself does not need relaxation but the subnetwork might contain some activation/non-linear function. Similarly, we refer to subnetwork forclear_relaxation.
Thebackward_relaxationmethod is for adding relaxations to the bound graph using CROWN where the top-level function continually callsbackward_relaxationuntil no more relaxation is needed as determined byneed_relaxation.
In this case the method may be called more than once since the subnetwork may have multiple layers needing relaxations, e.g. nn.Sequential with multiple layers of activation functions.
If a module needs relaxation, it starts the backward relaxation chain with identity linear bounds (and zero bias accumulation) and self as the return tuple.
The return signature ofbackward_relaxationislinear_bounds, relaxation_modulesince the top-level method needs a way of referring to the module being relaxed.@propertydefneed_relaxation(self):returnself.subnetwork.need_relaxationdefclear_relaxation(self):self.subnetwork.clear_relaxation()defbackward_relaxation(self,region):assertself.subnetwork.need_relaxationreturnself.subnetwork.backward_relaxation(region)Finally, we get to the IBP forward and CROWN backward propagations themselves.
IBP forward is straightforward using interval arithmetic with the exception that youmust callbounds.regionas the first parameter onIntervalBounds(andLinearBoundstoo).
If not the memory usage will be excessive.
IBP forward must also propagatesave_relaxationand if a relaxation is needed for a module the relaxation must be constructed when IBP forward is called - this is to support CROWN-IBP.
Similarly, IBP forward must also propagatesave_input_boundsand save input ifTrue, which is to improve.crown_backwardhas two components: updating linear bounds backwards and potentially adding to the bias accumulator.
The structure depends largely on how the nn.Module is, but addition can be done as shown below forResidual.
I recommend that you look atactivation.py,linear.py, andcat.pyto see how designcrown_backward.
Note that lower and upper will never interact, and that both can beNonebecause we can avoid unnecessary computations if either is not needed.defibp_forward(self,bounds,save_relaxation=False,save_input_bounds=False):residual_bounds=self.subnetwork.ibp_forward(bounds,save_relaxation=save_relaxation,save_input_bounds=save_input_bounds)returnIntervalBounds(bounds.region,bounds.lower+residual_bounds.lower,bounds.upper+residual_bounds.upper)defcrown_backward(self,linear_bounds,optimize):residual_linear_bounds=self.subnetwork.crown_backward(linear_bounds,optimize)iflinear_bounds.lowerisNone:lower=Noneelse:lower=(linear_bounds.lower[0]+residual_linear_bounds.lower[0],residual_linear_bounds.lower[1])iflinear_bounds.upperisNone:upper=Noneelse:upper=(linear_bounds.upper[0]+residual_linear_bounds.upper[0],residual_linear_bounds.upper[1])returnLinearBounds(linear_bounds.region,lower,upper)If you use alpha-CROWN and the module contains submodules (or optimizable parameters itself), then the bound module must implement method for collecting bound parameters, projecting gradients, and clipping bound parameters, to support PGD.defbound_parameters(self):formoduleinself.bound_sequential:yield frommodule.bound_parameters()defclip_params(self):formoduleinself.bound_sequential:module.clip_params()defproject_grads(self):formoduleinself.bound_sequential:module.project_grads()The last task is to make a factory and register your new module (alternatively subclass the factory and add to the dictionary of nn.Modules and BoundModules).factory=BoundModelFactory()factory.register(Residual,BoundResidual)AuthorsFrederik Baymler Mathiesen- PhD student @ TU DelftCiting@misc{Mathiesen2022,
  author = {Frederik Baymler Mathiesen},
  title = {Bound Propagation},
  year = {2022},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/Zinoex/bound_propagation}}
}Funding and supportTU DelftCopyright notice:Technische Universiteit Delft hereby disclaims all copyright
interest in the program “bound_propagation”
(bound propagation methods for Pytorch)
written by the Frederik Baymler Mathiesen. Theun Baller, Dean of Mechanical, Maritime and Materials Engineering© 2022, Frederik Baymler Mathiesen, HERALD Lab, TU Delft