Sound and Complete Neural Network Repair with Minimality and Locality Guarantees

1 INTRODUCTION . Deep neural networks ( DNNs ) have demonstrated impressive performances on a wide variety of applications ranging from transportation Bojarski et al . ( 2016 ) to health care Shahid et al . ( 2019 ) . However , DNNs are not perfect . In many cases , especially when the DNNs are used in safety-critical contexts , it is important to correct erroneous outputs of a DNN as they are discovered after training . For instance , a neural network in charge of giving control advisories to the pilots in an aircraft collision avoidance system , such as the ACAS Xu network from Julian et al . ( 2019 ) , may produce an incorrect advisory for certain situations and cause the aircraft to turn towards the incoming aircraft , thereby jeopardizing the safety of both airplanes . In this paper , we consider the problem of neural network repair , i.e . given a trained neural network and a set of buggy inputs ( inputs on which the neural network produces incorrect predictions ) , repair the network so that the resulting network on those buggy inputs behave according to some given correctness specification . Ideally , the changes to the neural network function should be small so that the outputs on other inputs are either unchanged or altered in a small way . Existing efforts on neural network repair roughly fall into the following three categories . 1 . Retraining/fine-tuning . The idea is to retrain or fine-tune the network with the newly identified buggy inputs and the corresponding corrected outputs . Methods include counterexample-guided data augmentation Dreossi et al . ( 2018 ) ; Ren et al . ( 2020 ) , editable training Sinitsin et al . ( 2020 ) and training input selection Ma et al . ( 2018 ) . One major weakness of these approaches is the lack of formal guarantees – at the end of retraining/fine-tuning , there is no guarantee that the given buggy inputs are fixed and no new bugs are introduced . In addition , retraining can be very expensive and requires access to the original training data which is impractical in cases where the neural network is obtained from a third party or the training data is private . Fine-tuning , on the other hand , often faces the issue of catastrophic forgetting Kirkpatrick et al . ( 2017 ) . 2 . Direct weight modification . These approaches directly manipulate the weights in a neural network to fix the buggy inputs . The repair problem is typically cast into an optimization problem or a verification problem . For example , Dong et al . ( 2020 ) proposes to minimize a loss defined on the buggy inputs . Goldberger et al . ( 2020 ) uses an SMT solver to identify minimal weight changes to the output layer of the network so that the undesirable behaviors are removed . In general , the optimization-based approach can not guarantee removal of the buggy behaviors , and the verificationbased approach does not scale beyond networks of a few hundred neurons . In addition , both approaches suffer from substantial accuracy drops on normal inputs since weight changes may be a poor proxy for changes in the function space . 3 . Architecture extension . The third category of approaches extends the given NN architecture , such as by introducing more weight parameters , to facilitate more efficient repairs . The so-called Decoupled DNN architecture Sotoudeh & Thakur ( 2021 ) is the only work we know that falls into this category . Their idea is to decouple the activations of the network from values of the network by augmenting the original network . Their construction allows the formulation of any single-layer repair as an linear programming ( LP ) problem . The decoupling , however , causes the repaired network to become discontinuous ( in the functional sense ) . In addition , it still can not isolate the output change to a single buggy input from the rest of the inputs . In addition to the aforementioned limitations , a common weakness that is shared amongst these methods is that the induced changes , as a result of either retraining or direct weight modification , are global . This means that a correct behavior on another input , regardless of how far it is away from the buggy input , may not be preserved by the repair . Worse still , the repair on a new buggy input can end up invalidating the repair on a previous buggy input . The fundamental issue here is that limiting the changes to a few weights or a single layer only poses a structural constraint ( often for ease of computation ) ; it does not limit the changes on the input-output mapping of the neural network . It is known that even a single weight change can have a global effect on the output of a neural network . In this paper , we propose REASSURE , a novel methodology for neural network repair with locality , minimality , soundness and completeness guarantees . Our methodology targets continuous piecewise linear ( CPWL ) neural networks , specifically those that use the ReLU activation functions . The key idea of our approach is to leverage the CPWL property of ReLU networks to synthesize a patch network tailored to the linear region where the buggy input resides , which when combined with the original network , provably corrects the behavior on the buggy input . Our approach is both sound and complete – the repaired network is guaranteed to fix the buggy input , and a patch is guaranteed to be found for any buggy input . Moreover , our approach preserves the CPWL nature of ReLU networks , automatically generalizes the repair to all the points including other undetected buggy inputs inside the repair region , is minimal in terms of changes in the function space , and guarantees that outputs on inputs away from the repair region are unaltered . Figure 1 provides an illustrative comparison of our approach with other methods . Table 1 compares our approach with representative related works in terms of theoretical guarantees . We summarize our contributions below . • We present REASSURE , the first sound and complete repair methodology for ReLU networks with strong theoretical guarantees . • Our technique synthesizes a patch network , which when combined with the original neural network , provably corrects the behavior on the buggy input . This approach is a significant departure from existing methods that rely on retraining or direct weight manipulation . • Our technique is both effective and efficient – across a set of benchmarks , REASSURE can efficiently correct a set of buggy inputs or buggy areas with little or no change to the accuracy and overall functionality of the network . 2 BACKGROUND . 2.1 DEEP NEURAL NETWORKS . An R-layer feed-forward DNN f = κR ◦ σ ◦ κR−1 ◦ ... ◦ σ ◦ κ1 : X → Y is a composition of linear functions κr , r = 1 , 2 , ... , R and activation function σ , where X ⊆ Rm is a bounded input domain and Y ⊆ Rn is the output domain . Weights and biases of linear function { κr } r=1,2 , ... , R are parameters of the DNN . We call the first R− 1 layers hidden layers and the R-th layer the output layer . We use zij to denote the i-th neuron ( before activation ) in the j-th hidden layer . In this paper , we focus on ReLU DNNs , i.e . DNNs that use only the ReLU activation functions . It is known that an Rm → R function is representable by a ReLU DNN if and only if it is a continuous piecewise linear ( CPWL ) function Arora et al . ( 2016 ) . The ReLU function is defined as σ ( x ) = max ( x , 0 ) . We say that σ ( x ) is activated if σ ( x ) = x . 2.2 LINEAR REGIONS . A linear region A is the set of inputs that correspond to the same activation pattern in a ReLU DNN f Serra et al . ( 2017 ) . Geometrically , this corresponds to a convex polytope , which is an intersection of half spaces , in the input space X on which f is linear . We use f |A to denote the part of f on A . 2.3 CORRECTNESS SPECIFICATION . A correctness specification Φ = ( Φin , Φout ) is a tuple of two polytopes , where Φin is the union of some linear regions and Φout is a convex polytope . A DNN f is said to meet a specification Φ = ( Φin , Φout ) , denoted as f |= Φ , if and only if ∀x ∈ Φin , f ( x ) ∈ Φout . Example 1 . For a classification problem , we can formally write the specification that “ the prediction of any point in an area A is class k ” as Φ = ( Φin , Φout ) , where Φin = A and Φout = { y ∈ Rn | yk ≥ yi , ∀i 6= k } 1 . 2.4 PROBLEM DEFINITION . In this paper , we consider the following two repair problems . Definition 1 ( Area repair ) . Given a correctness specification Φ = ( Φin , Φout ) and a ReLU DNN f 6|= Φ , the area repair problem is to find a modified ReLU DNN f̂ such that f̂ |= Φ . 1Note that here y is the output of the layer right before the softmax layer in a classification network . Note that we do not require f̂ to have the same structure or parameters as f in this definition . If Φin contains a single ( buggy ) linear region , we refer to this as single-region repair . If Φin contains multiple ( buggy ) linear regions , we refer to it as multi-region repair . Definition 2 ( Point-wise repair ) . Given a set of buggy inputs { x̃1 , . . . , x̃L } ⊂ Φin with their corresponding correct outputs { y1 , . . . , yL } and a ReLU DNN f , the point-wise repair problem is to find a modified ReLU DNN f̂ such that ∀i , f̂ ( x̃i ) = yi . We call the minimal variants of area repair and point-wise repair minimal area repair and minimal point-wise repair respectively . Minimality here is defined with respect to the maximum distance between f and f̂ over the whole input domain X . A point-wise repair can be generalized to an area repair through the following result . 2.5 FROM BUGGY INPUTS TO BUGGY LINEAR REGIONS . The linear region where an input x resides can be computed as follows . Lemma 1 . Lee et al . ( 2019 ) Consider a ReLU DNN f and an input x ∈ X . For every neuron zij , it induces a feasible set Aij ( x ) = { { x̄ ∈ X| ( Oxzij ) T x̄+ zij − ( Oxzij ) Tx ≥ 0 } if zij ≥ 0 { x̄ ∈ X| ( Oxzij ) T x̄+ zij − ( Oxzij ) Tx ≤ 0 } if zij < 0 ( 1 ) The set A ( x ) = ∩i , jAij ( x ) is the linear region that includes x . Note that A ( x ) is essentially the H-representation of the corresponding convex polytope . 2.6 REPAIR DESIDERATA . We argue that an effective repair algorithm for ReLU DNN should satisfy the following criteria . 1 . Preservation of CPWL : Given that the original network f models a CPWL function , the repaired network f̂ should still model a CPWL function . 2 . Soundness : A sound repair should completely remove the known buggy behaviors , i.e . it is a solution to the point-wise repair problem defined in Definition 2 . 3 . Completeness : Ideally , the algorithm should always be able find a repair for any given buggy input if it exists . 4 . Generalization : If there exists another buggy input x̃′ in the neighborhood of x̃ ( e.g . the same linear region ) , then the repair should also fix it . For example , suppose we have an x̃ that violates a specification which requires the output to be within some range . It is almost guaranteed that there exists another ( and infinitely many ) x̃′ in the same linear region that also violates the specification . 5 . Locality : We argue that a good repair should only induce a localized change to f in the function space . For example , in the context of ReLU DNN , if a linear region B does not border the repair regionA , i.e . B∩A = ∅ , then f̂ |B ( x ) = f |B ( x ) . 6 . Minimality : Some notion of distance between f and f̂ such as max |f− f̂ | should be minimized . Note that this is a significant departure from existing methods that focus on minimizing the change in weights which has no guarantee on the amount of change in the function space . 7 . Limited side effect : Repairing a buggy point should not adversely affect points that were originally correct . For example , repairing a buggy input x̃ in region A should not change another region from correct to incorrect . Formally , for any linear region C who is a neighbor of A , i.e . C ∩ A 6= ∅ , if f |C |= Φ , then f̂ |C |= Φ . 8 . Efficiency : The repair algorithm should terminate in polynomial time with respect to the size of the neural network and the number of buggy inputs .

Summary: The paper presents a new method for the provable repair of neural networks. Both point-wise and polytope-based repairs are considered. The main idea is to synthesize a patch network and combine it with the buggy network. The patch network consists of two subcomponents: a support network that is only activated for the inputs in the linear region containing buggy inputs and an affine patch network that repairs the network in the buggy region. The experimental evaluation is performed on similar datasets and networks as prior work and shows the effectiveness of the new methodology.