Laplacian Networks: Bounding Indicator Function Smoothness for Neural Networks Robustness

1 Introduction . Deep Neural Networks ( DNNs ) provide state-of-the-art performance in many challenges in machine learning ( He et al. , 2016 ; Wu et al. , 2016 ) . Their ability to achieve good generalization is often explained by the fact they use very few priors about data ( LeCun et al. , 2015 ) . On the other hand , their strong dependency on data may lead to focus on biased features of the training dataset , resulting in a nonrobust classification performance . In the literature , authors have been interested in studying the robustness of DNNs in various conditions . These conditions include : • Robustness to isotropic noise , i.e. , small isotropic variations of the input ( Mallat , 2016 ) , typically meaning that the network function leads to a small Lipschitz constant . • Robustness to adversarial attacks , which can exploit knowledge about the network parameters or the training dataset ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ) . • Robustness to implementation defects , which can result in only approximately correct computations ( Hubara et al. , 2017 ) . To improve DNN robustness , three main families of solutions have been proposed in the literature . The first one involves enforcing smoothness , as measured by a Lipschitz constant , in the operators and having a minimum separation margin ( Mallat , 2016 ) . A similar approach has been proposed in ( Cisse et al. , 2017 ) , where the authors restrict the function of the network to be contractive . A second class of methods use intermediate representations obtained at various layers to perform the prediction ( Papernot and McDaniel , 2018 ) . Finally , in ( Kurakin et al. , 2016 ; Pezeshki et al. , 2016 ; Madry et al. , 2018 ) , the authors propose to train the network using noisy inputs so that it better generalizes to this type of noise . This has been shown to improve the robustness of the network to the specific type of noise used during training , but it is not guaranteed that this robustness would be extended to other types of deformations . In this work , we introduce a new regularizer that does not focus on a specific type of deformation , but aims at increasing robustness in general . As such , the proposed regularizer can be combined with other existing methods . It is inspired by recent developments in Graph Signal Processing ( GSP ) ( Shuman et al. , 2013 ) . GSP is a mathematical framework that extends classical Fourier analysis to complex topologies described by graphs , by introducing notions of frequency for signals defined on graphs . Thus , signals that are smooth on the graph ( i.e. , change slowly from one node to its neighbors ) will have most of their energy concentrated in the low frequencies . The proposed regularizer is based on constructing a series of graphs , one for each layer of the DNN architecture , where each graph captures the similarity between all training examples given their intermediate representation at that layer . Our proposed regularizer penalizes large changes in the smoothness of class indicator vectors ( viewed here as graph signals ) from one layer to the next . As a consequence , the distances between pairs of examples in different classes are only allowed to change slowly from one layer to the next . Note that because we use deep architectures , the regularizer does not prevent the smoothness from achieving its maximum value , but constraining the size of changes from layer to layer increases the robustness of the network function by controlling the distance to the boundary region , as supported by experiments in Section 4 . The outline of the paper is as follows . In Section 2 we present related work . In Section 3 we introduce the proposed regularizer . In Section 4 we evaluate the performance of our proposed method in various conditions and on vision benchmarks . Section 5 summarizes our conclusions . 2 Related work . DNN robustness may refer to many different problems . In this work we are mostly interested in the stability to deformations ( Mallat , 2016 ) , or noise , which can be due to multiple factors mentioned in the introduction . The most studied stability to deformations is in the context of adversarial attacks . It has been shown that very small imperceptible changes on the input of a trained DNN can result in missclassification of the input ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ) . These works have been primordial to show that DNNs may not be as robust to deformations as the test accuracy benchmarks would have lead one to believe . Other works , such as ( Recht et al. , 2018 ) , have shown that DNNs may also suffer from drops in performance when facing deformations that are not originated from adversarial attacks , but simply by re-sampling the test images . Multiple ways to improve robustness have been proposed in the literature . They range from the use of a model ensemble composed of k-nearest neighbors classifiers for each layer ( Papernot and McDaniel , 2018 ) , to the use of distillation as a mean to protect the network ( Papernot et al. , 2016a ) . Other methods introduce regularizers ( Gu and Rigazio , 2014 ) , control the Lipschitz constant of the network function ( Cisse et al. , 2017 ) or implement multiple strategies revolving around using deformations as a data augmentation procedure during the training phase ( Goodfellow et al. , 2014 ; Kurakin et al. , 2016 ; Moosavi Dezfooli et al. , 2016 ) . Compared to these works , our proposed method can be viewed as a regularizer that penalizes large deformations of the class boundaries throughout the network architecture , instead of focusing on a specific deformation of the input . As such , it can be combined with other mentioned strategies . Indeed , we demonstrate that the proposed method can be implemented in combination with ( Cisse et al. , 2017 ) , resulting in a network function such that small variations to the input lead to small variations in the decision , as in ( Cisse et al. , 2017 ) , while limiting the amount of change to the class boundaries . Note that our approach does not require using training data affected by a specific deformation , and our results could be further improved if such data were available for training , as shown in the Appendix . As for combining GSP and machine learning , this area has sparked interest recently . For example , the authors of ( Gripon et al. , 2018 ) show that it is possible to detect overfitting by tracking the evolution of the smoothness of a graph containing only training set examples . Another example is in ( Anirudh et al. , 2017 ) where the authors introduce different quantities related to GSP that can be used to extract interpretable results from DNNs . In ( Svoboda et al. , 2018 ) the authors exploit graph convolutional layers ( Bronstein et al. , 2017 ) to increase the robustness of the network . To the best of our knowledge , this is the first use of graph signal smoothness as a regularizer for deep neural network design . 3 Methodology . 3.1 Similarity preset and postset graphs . Consider a deep neural network architecture . Such a network is obtained by assembling layers of various types . Of particular interest are layers of the form x ` 7→ x ` +1 = h ` ( W ` x ` + b ` ) , where h ` is a nonlinear function , typically a ReLU , W ` is the weight tensor at layer ` , x ` is the intermediate representation of the input at layer ` and b ` is the corresponding bias tensor . Note that strides or pooling may be used . Assembling can be achieved in various ways : composition , concatenation , sums . . . so that we obtain a global function f that associates an input tensor x0 to an output tensor y = f ( x0 ) . When computing the output y associated with the input x0 , each layer ` of the architecture processes some input x ` and computes the corresponding output y ` = h ` ( W ` x ` + b ` ) . For a given layer ` and a batch of b inputs X = { x1 , . . . , xb } , we can obtain two sets X ` = { x ` 1 , . . . , x ` b } , called the preset , and Y ` = { y ` 1 , . . . , y ` b } , called the postset . Given a similarity measure s on tensors , from a preset we can build the similarity preset matrix : M ` pre [ i , j ] = s ( x ` i , x ` j ) , ∀1 ≤ i , j ≤ b , where M [ i , j ] denotes the element at line i and column j in M. The postset matrix is defined similarly . Consider a similarity ( either preset or postset ) matrix M ` . This matrix can be used to build a k-nearest neighbor similarity weighted graph G ` = 〈V , A ` 〉 , where V = { 1 , . . . , b } is the set of vertices and A ` is the weighted adjacency matrix defined as : A ` [ i , j ] = M ` [ i , j ] if M ` [ i , j ] ∈ arg maxi′ 6=j ( M ` [ i′ , j ] , k ) ⋃ arg maxj′ 6=i ( M ` [ i , j′ ] , k ) 0 otherwise , ∀i , j ∈ V , ( 1 ) where arg maxi ( ai , k ) denotes the indices of the k largest elements in { a1 , . . . , ab } . Note that by construction A ` is symmetric . 3.2 Smoothness of label signals . Given a weighted graph G ` = 〈V , A ` 〉 , we call Laplacian of G ` the matrix L ` = D ` −A ` , where D ` is the diagonal matrix such that : D ` [ i , i ] = ∑ j A ` [ i , j ] , ∀i ∈ V . Because L ` is symmetric and real-valued , it can be written : L ` = F ` Λ ` F ` > , ( 2 ) where F is orthonormal and contains eigenvectors of L ` as columns , F > denotes the transpose of F , and Λ is diagonal and contains eigenvalues of L ` is ascending order . Note that the constant vector 1 ∈ Rb is an eigenvector of L ` corresponding to eigenvalue 0 . Moreover , all eigenvalues of L ` are nonnegative . Consequently , 1/ √ n can be chosen as the first column in F. Consider a vector s ∈ Rb , we define ŝ the Graph Fourier Transform ( GFT ) of s on G ` as ( Shuman et al. , 2013 ) : ŝ = F > s . ( 3 ) Because the order of the eigenvectors is chosen so that the corresponding eigenvalues are in ascending order , if only the first few entries of ŝ are nonzero that indicates that s is low frequency ( smooth ) . In the extreme case where only the first entry of ŝ is nonzero we have that s is constant ( maximum smoothness ) . More generally , smoothness σ ` ( s ) of a signal s can be measured using the quadratic form of the Laplacian : σ ` ( s ) = s > L ` s = b∑ i , j=1 A ` [ i , j ] ( s [ i ] − s [ j ] ) 2 = b∑ i=1 Λ ` [ i , i ] ŝ [ i ] 2 , ( 4 ) where we note that s is smoother when σ ` ( s ) is smaller . In this paper we are particularly interested in smoothness of the label signals . We call label signal sc associated with class c a binary ( { 0 , 1 } ) vector whose nonzero coordinates are the ones corresponding to input vectors of class c. In other words , sc [ i ] = 1 ⇔ ( xi is in class c ) , ∀1 ≤ i ≤ b . Using Equation ( 4 ) , we obtain that the smoothness of the label signal sc is the sum of similarities between examples in distinct classes . Thus a smoothness of 0 means that examples in distinct classes have 0 similarity . Denote u the last layer of the architecture : yui = yi , ∀i . Note that in typical settings , where outputs of the networks are one-hot-bit encoded and no regularizer is used , at the end of the learning process it is expected that y > i yj ≈ 1 if i and j belong to the same class , and y > i yj ≈ 0 otherwise . Thus , assuming that cosine similarity is used to build the graph , the last layer smoothness for all c would be σupost ( sc ) ≈ 0 , since edge weights between nodes having different labels will be close to zero given Equation ( 4 ) . More generally , smoothness of sc at the preset or postset of a given layer measures the average similarity between examples in class c and examples in other classes ( σ ( sc ) decreases as the weights of edges connecting nodes in different classes decrease ) . Because the last layer can achieve σ ( sc ) ≈ 0 , we expect the smoothness metric σ at each layer to decrease as we go deeper in the network . Next we introduce a regularization strategy that limits how much σ can decrease from one layer to the next and can even prevent the last layer from achieving σ ( sc ) = 0 . This will be shown to improve generalization and robustness . The theoretical motivation for this choice is discussed in Section 3.4 .

Summary: The paper proposes to use a regularization which preserves nearest-neighbor smoothness from layer to layer. The approach is based on controlling the extent to which examples from different classes are separated from one layer to the next, in deep neural networks. The criterion computes the smoothness of the label vectors (one-hot encodings of class labels) along the nearest-neighbor graph constructed from the euclidian distances on a given layer's activations. From an algorithmic perspective, the regularization is applied by considering distances graphs on minibatches. Experiments on CIFAR-10 show that the method improves the robustness of the neural networks to different types of perturbations (perturbations of the input, aka adversarial examples, and quantization of the network weights/dropout0.