Sound and Complete Neural Network Repair with Minimality and Locality Guarantees

1 INTRODUCTION . Deep neural networks ( DNNs ) have demonstrated impressive performances on a wide variety of applications ranging from transportation Bojarski et al . ( 2016 ) to health care Shahid et al . ( 2019 ) . However , DNNs are not perfect . In many cases , especially when the DNNs are used in safety-critical contexts , it is important to correct erroneous outputs of a DNN as they are discovered after training . For instance , a neural network in charge of giving control advisories to the pilots in an aircraft collision avoidance system , such as the ACAS Xu network from Julian et al . ( 2019 ) , may produce an incorrect advisory for certain situations and cause the aircraft to turn towards the incoming aircraft , thereby jeopardizing the safety of both airplanes . In this paper , we consider the problem of neural network repair , i.e . given a trained neural network and a set of buggy inputs ( inputs on which the neural network produces incorrect predictions ) , repair the network so that the resulting network on those buggy inputs behave according to some given correctness specification . Ideally , the changes to the neural network function should be small so that the outputs on other inputs are either unchanged or altered in a small way . Existing efforts on neural network repair roughly fall into the following three categories . 1 . Retraining/fine-tuning . The idea is to retrain or fine-tune the network with the newly identified buggy inputs and the corresponding corrected outputs . Methods include counterexample-guided data augmentation Dreossi et al . ( 2018 ) ; Ren et al . ( 2020 ) , editable training Sinitsin et al . ( 2020 ) and training input selection Ma et al . ( 2018 ) . One major weakness of these approaches is the lack of formal guarantees – at the end of retraining/fine-tuning , there is no guarantee that the given buggy inputs are fixed and no new bugs are introduced . In addition , retraining can be very expensive and requires access to the original training data which is impractical in cases where the neural network is obtained from a third party or the training data is private . Fine-tuning , on the other hand , often faces the issue of catastrophic forgetting Kirkpatrick et al . ( 2017 ) . 2 . Direct weight modification . These approaches directly manipulate the weights in a neural network to fix the buggy inputs . The repair problem is typically cast into an optimization problem or a verification problem . For example , Dong et al . ( 2020 ) proposes to minimize a loss defined on the buggy inputs . Goldberger et al . ( 2020 ) uses an SMT solver to identify minimal weight changes to the output layer of the network so that the undesirable behaviors are removed . In general , the optimization-based approach can not guarantee removal of the buggy behaviors , and the verificationbased approach does not scale beyond networks of a few hundred neurons . In addition , both approaches suffer from substantial accuracy drops on normal inputs since weight changes may be a poor proxy for changes in the function space . 3 . Architecture extension . The third category of approaches extends the given NN architecture , such as by introducing more weight parameters , to facilitate more efficient repairs . The so-called Decoupled DNN architecture Sotoudeh & Thakur ( 2021 ) is the only work we know that falls into this category . Their idea is to decouple the activations of the network from values of the network by augmenting the original network . Their construction allows the formulation of any single-layer repair as an linear programming ( LP ) problem . The decoupling , however , causes the repaired network to become discontinuous ( in the functional sense ) . In addition , it still can not isolate the output change to a single buggy input from the rest of the inputs . In addition to the aforementioned limitations , a common weakness that is shared amongst these methods is that the induced changes , as a result of either retraining or direct weight modification , are global . This means that a correct behavior on another input , regardless of how far it is away from the buggy input , may not be preserved by the repair . Worse still , the repair on a new buggy input can end up invalidating the repair on a previous buggy input . The fundamental issue here is that limiting the changes to a few weights or a single layer only poses a structural constraint ( often for ease of computation ) ; it does not limit the changes on the input-output mapping of the neural network . It is known that even a single weight change can have a global effect on the output of a neural network . In this paper , we propose REASSURE , a novel methodology for neural network repair with locality , minimality , soundness and completeness guarantees . Our methodology targets continuous piecewise linear ( CPWL ) neural networks , specifically those that use the ReLU activation functions . The key idea of our approach is to leverage the CPWL property of ReLU networks to synthesize a patch network tailored to the linear region where the buggy input resides , which when combined with the original network , provably corrects the behavior on the buggy input . Our approach is both sound and complete – the repaired network is guaranteed to fix the buggy input , and a patch is guaranteed to be found for any buggy input . Moreover , our approach preserves the CPWL nature of ReLU networks , automatically generalizes the repair to all the points including other undetected buggy inputs inside the repair region , is minimal in terms of changes in the function space , and guarantees that outputs on inputs away from the repair region are unaltered . Figure 1 provides an illustrative comparison of our approach with other methods . Table 1 compares our approach with representative related works in terms of theoretical guarantees . We summarize our contributions below . • We present REASSURE , the first sound and complete repair methodology for ReLU networks with strong theoretical guarantees . • Our technique synthesizes a patch network , which when combined with the original neural network , provably corrects the behavior on the buggy input . This approach is a significant departure from existing methods that rely on retraining or direct weight manipulation . • Our technique is both effective and efficient – across a set of benchmarks , REASSURE can efficiently correct a set of buggy inputs or buggy areas with little or no change to the accuracy and overall functionality of the network . 2 BACKGROUND . 2.1 DEEP NEURAL NETWORKS . An R-layer feed-forward DNN f = κR ◦ σ ◦ κR−1 ◦ ... ◦ σ ◦ κ1 : X → Y is a composition of linear functions κr , r = 1 , 2 , ... , R and activation function σ , where X ⊆ Rm is a bounded input domain and Y ⊆ Rn is the output domain . Weights and biases of linear function { κr } r=1,2 , ... , R are parameters of the DNN . We call the first R− 1 layers hidden layers and the R-th layer the output layer . We use zij to denote the i-th neuron ( before activation ) in the j-th hidden layer . In this paper , we focus on ReLU DNNs , i.e . DNNs that use only the ReLU activation functions . It is known that an Rm → R function is representable by a ReLU DNN if and only if it is a continuous piecewise linear ( CPWL ) function Arora et al . ( 2016 ) . The ReLU function is defined as σ ( x ) = max ( x , 0 ) . We say that σ ( x ) is activated if σ ( x ) = x . 2.2 LINEAR REGIONS . A linear region A is the set of inputs that correspond to the same activation pattern in a ReLU DNN f Serra et al . ( 2017 ) . Geometrically , this corresponds to a convex polytope , which is an intersection of half spaces , in the input space X on which f is linear . We use f |A to denote the part of f on A . 2.3 CORRECTNESS SPECIFICATION . A correctness specification Φ = ( Φin , Φout ) is a tuple of two polytopes , where Φin is the union of some linear regions and Φout is a convex polytope . A DNN f is said to meet a specification Φ = ( Φin , Φout ) , denoted as f |= Φ , if and only if ∀x ∈ Φin , f ( x ) ∈ Φout . Example 1 . For a classification problem , we can formally write the specification that “ the prediction of any point in an area A is class k ” as Φ = ( Φin , Φout ) , where Φin = A and Φout = { y ∈ Rn | yk ≥ yi , ∀i 6= k } 1 . 2.4 PROBLEM DEFINITION . In this paper , we consider the following two repair problems . Definition 1 ( Area repair ) . Given a correctness specification Φ = ( Φin , Φout ) and a ReLU DNN f 6|= Φ , the area repair problem is to find a modified ReLU DNN f̂ such that f̂ |= Φ . 1Note that here y is the output of the layer right before the softmax layer in a classification network . Note that we do not require f̂ to have the same structure or parameters as f in this definition . If Φin contains a single ( buggy ) linear region , we refer to this as single-region repair . If Φin contains multiple ( buggy ) linear regions , we refer to it as multi-region repair . Definition 2 ( Point-wise repair ) . Given a set of buggy inputs { x̃1 , . . . , x̃L } ⊂ Φin with their corresponding correct outputs { y1 , . . . , yL } and a ReLU DNN f , the point-wise repair problem is to find a modified ReLU DNN f̂ such that ∀i , f̂ ( x̃i ) = yi . We call the minimal variants of area repair and point-wise repair minimal area repair and minimal point-wise repair respectively . Minimality here is defined with respect to the maximum distance between f and f̂ over the whole input domain X . A point-wise repair can be generalized to an area repair through the following result . 2.5 FROM BUGGY INPUTS TO BUGGY LINEAR REGIONS . The linear region where an input x resides can be computed as follows . Lemma 1 . Lee et al . ( 2019 ) Consider a ReLU DNN f and an input x ∈ X . For every neuron zij , it induces a feasible set Aij ( x ) = { { x̄ ∈ X| ( Oxzij ) T x̄+ zij − ( Oxzij ) Tx ≥ 0 } if zij ≥ 0 { x̄ ∈ X| ( Oxzij ) T x̄+ zij − ( Oxzij ) Tx ≤ 0 } if zij < 0 ( 1 ) The set A ( x ) = ∩i , jAij ( x ) is the linear region that includes x . Note that A ( x ) is essentially the H-representation of the corresponding convex polytope . 2.6 REPAIR DESIDERATA . We argue that an effective repair algorithm for ReLU DNN should satisfy the following criteria . 1 . Preservation of CPWL : Given that the original network f models a CPWL function , the repaired network f̂ should still model a CPWL function . 2 . Soundness : A sound repair should completely remove the known buggy behaviors , i.e . it is a solution to the point-wise repair problem defined in Definition 2 . 3 . Completeness : Ideally , the algorithm should always be able find a repair for any given buggy input if it exists . 4 . Generalization : If there exists another buggy input x̃′ in the neighborhood of x̃ ( e.g . the same linear region ) , then the repair should also fix it . For example , suppose we have an x̃ that violates a specification which requires the output to be within some range . It is almost guaranteed that there exists another ( and infinitely many ) x̃′ in the same linear region that also violates the specification . 5 . Locality : We argue that a good repair should only induce a localized change to f in the function space . For example , in the context of ReLU DNN , if a linear region B does not border the repair regionA , i.e . B∩A = ∅ , then f̂ |B ( x ) = f |B ( x ) . 6 . Minimality : Some notion of distance between f and f̂ such as max |f− f̂ | should be minimized . Note that this is a significant departure from existing methods that focus on minimizing the change in weights which has no guarantee on the amount of change in the function space . 7 . Limited side effect : Repairing a buggy point should not adversely affect points that were originally correct . For example , repairing a buggy input x̃ in region A should not change another region from correct to incorrect . Formally , for any linear region C who is a neighbor of A , i.e . C ∩ A 6= ∅ , if f |C |= Φ , then f̂ |C |= Φ . 8 . Efficiency : The repair algorithm should terminate in polynomial time with respect to the size of the neural network and the number of buggy inputs .

Summary: This paper proposed an approach for repairing neural networks that use ReLU activation functions. Different with existing techniques that retrain the model or change the weights, the proposed approach generates a patch in the original model, i.e., add sub-models. The LP solver is used to solve the patch function for a linear region. The experimental  results demonstrated that the approach could outperform existing methods.