Patch-level Neighborhood Interpolation:  A General and Effective Graph-based Regularization Strategy

Regularization plays a crucial role in machine learning models , especially for deep neural networks . The existing regularization techniques mainly rely on the i.i.d . assumption and only consider the knowledge from the current sample , without the leverage of neighboring relationship between samples . In this work , we propose a general regularizer called Patch-level Neighborhood Interpolation ( Pani ) that conducts a non-local representation in the computation of network . Our proposal explicitly constructs patch-level graphs in different network layers and then linearly interpolates neighborhood patch features , serving as a general and effective regularization strategy . Further , we customize our approach into two kinds of popular regularization methods , namely Virtual Adversarial Training ( VAT ) and MixUp as well as its variants . The first derived Pani VAT presents a novel way to construct non-local adversarial smoothness by employing patch-level interpolated perturbations . In addition , the second derived Pani MixUp method extends the original MixUp regularization and its variant to the Pani version , achieving a significant improvement in the performance . Finally , extensive experiments are conducted to verify the effectiveness of our Patch-level Neighborhood Interpolation approach in both supervised and semi-supervised settings . 1 INTRODUCTION . In the statistical learning theory , regularization techniques are typically leveraged to achieve the trade-off between empirical error minimization and the control of model complexity ( Vapnik & Chervonenkis , 2015 ) . In contrast to the classical convex empirical risk minimization where regularization can rule out trivial solutions , regularization plays a rather different role in deep learning due to its highly non-convex optimization nature ( Zhang et al. , 2016 ) . Among all the explicit and implicit regularization , regularization with stochastic transformation , perturbations and randomness , such as adversarial training ( Goodfellow et al. , 2014 ) , dropout and MixUp ( Zhang et al. , 2017 ) , play a key role in the deep learning models due to their superiority in the performance ( Berthelot et al. , 2019b ; Zhang et al. , 2017 ; Miyato et al. , 2018 ; Berthelot et al. , 2019a ) . In this section , we firstly review two kinds of effective and prestigious regularization branches for deep neural networks , which can elegantly generalize from supervised learning to semi-supervised setting . Adversarial Training ( Goodfellow et al. , 2014 ; Madry et al. , 2017 ) can provide an additional regularization beyond that provided by other generic regularization strategies , such as dropout , pretraining and model averaging . However , recent works ( Zhang et al. , 2019 ; Tsipras et al. , 2018 ) demonstrated that this kind of training method holds a trade-off between the robustness and accuracy , limiting the efficacy of the adversarial regularization . Besides , Virtual Adversarial Training ( VAT ) ( Miyato et al. , 2018 ) can be regarded as a natural extension of adversarial training to semi-supervised setting through adversarially smoothing the posterior output distribution with the leverage of unlabeled data . This strategy has achieved great success in image classification ( Miyato et al. , 2018 ) , text classification ( Miyato et al. , 2016 ) and node classification ( Sun et al. , 2019 ) . Tangent-Normal Adversarial Regularization ( TNAR ) ( Yu et al. , 2019 ) extended VAT by taking the data manifold into consideration and applied VAT along the tangent space and the orthogonal normal space of the data manifold , outperforming previous semi-supervised approaches . MixUp ( Zhang et al. , 2017 ) augmented the training data by incorporating the prior knowledge that linear interpolation of input vectors should lead to linear interpolation of the associated targets , accomplishing consistent improvement of generalization on image , speech and tabular data . MixMatch ( Berthelot et al. , 2019b ) extended MixUp to semi-supervised tasks by guessing low-entropy labels for data-augmented unlabeled examples and mixing labeled and unlabeled data using MixUp . In contrast with VAT , MixMatch ( Berthelot et al. , 2019b ) utilizes one specific form of consistency regularization , i.e. , using the standard data augmentation for images , such as random horizontal flips , rather than computing adversarial perturbations to smooth the posterior distribution of the classifier . Nevertheless , the vast majority of regularization methods , including the aforementioned approaches , assume that the training samples are drawn independently and identically from an unknown data generating distribution . For instance , Support Vector Machine ( SVM ) , Back-Propagation ( BP ) for Neural Networks , and many other common algorithms implicitly make this assumption as part of their derivation . However , this i.i.d . assumption is commonly violated in realistic scenarios where batches or sub-groups of training samples are likely to have internal correlations . In particular , Dundar et al . ( 2007 ) demonstrated that accounting for the correlations in real-world training data leads to statistically significant improvements in accuracy . Similarly , Peer-Regularized Networks ( PeerNet ) ( Svoboda et al. , 2018 ) applied graph convolutions ( Velickovic et al. , 2017 ; Kipf & Welling , 2016 ) to harness information of peer samples , and verified its effectiveness on defending adversarial attacks . Motivated by these facts , we aim to design a general regularization strategy that can fully utilize the internal relationship between samples by explicitly constructing a graph within a minibatch in order to consistently improve the generalization of deep neural networks in both semi- and supervised settings . In this paper , we propose the Patch-level Neighborhood Interpolation ( Pani ) for deep neural networks , serving as a simple yet effective non-local regularization . We firstly construct a patch-level graph in each mini-batch during the stochastic gradient decent training process . Then we apply linear interpolation on the neighboring patch features and the resulting non-local representation additionally captures the relationship of neighboring patch features in different layers , serving as a general and effective regularization . Furthermore , to prove the generality and superiority of our Pani method , we explicitly customize our approach into two kinds of popular and general regularization strategies , i.e. , Virtual Adversarial Regularization and MixUp , resulting in Pani VAT and Pani MixUp . For the Pani VAT , we reformulate the construction of adversarial perturbations , transforming from solely depending on the current sample to the linear interpolation of neighboring patch features . This non-local adversarial perturbations can leverage the information of neighboring correlation from all samples within a batch , providing more informative adversarial smoothness in semisupervised setting . Besides , in the Pani MixUp , we extend MixUp and its variant MixMatch from image to patch level by mixing fine-grained patch features and corresponding supervised signals . Finally , we conduct extensive experiments to demonstrate that both of the two derived regularization strategies can outperform other state-of-the-art approaches in both supervised and semi-supervised tasks . More importantly , these successful cases verify the generality and superiority of our Patchlevel Neighborhood Interpolation method . Our contributions can be summarized as follow : • We propose a general interpolation strategy either in input or feature space , i.e. , Patch-level Neighborhood Interpolation , helping to improve the generalization of deep neural networks on both semi- and supervised scenarios . This strategy can serve as an effective graph-based representation method and has much potential to be leveraged in a wider range of tasks . • Based on our method , the customized approaches Pani VAT and Pani MixUP as well as Pani MixMatch can boost the generalization performance significantly , and thus provide a guidance to the deployment of our Pani strategy into more regularization methods . 2 OUR METHOD : PATCH-LEVEL NEIGHBORHOOD INTERPOLATION . Before introducing our approach , we highly recommend readers to go through some preliminary knowledge about VAT ( Miyato et al. , 2017 ) , MixUP ( Zhang et al. , 2017 ) and PeerNet ( Svoboda et al. , 2018 ) in Appendix A . For our method , one related work is PeerNet ( Svoboda et al. , 2018 ) that designed graph-based layers to defend against adversarial attacks , but unfortunately the construction of pixel-level K-NN graphs in PeerNet ( Svoboda et al. , 2018 ) is costly in computation . By contrast , our motivation is to develop a general regularization that can consistently boost the performance of deep neural networks in both semi- and supervised settings rather than the adversarial scenario . Besides , the construction way of a non-local layer in our method is more flexible and can be determined by the specific objective function , as elaborated in Section 2.1 and 2.2 . Moreover , our patch-level method can achieve computational advantage over pixel-level regularization , and incorporates more meaningful semantic correlations in different layers . Particularly , a flexible patch size can be chosen according to the size of receptive field in different layers , yielding more informative graph-based representation and better regularization performance . Concretely , as our Patch-level Neighborhood Interpolation ( Pani ) shown in Figure 1 , in the first step we determine the candidate peer images set Si for each image i . This can be achieved by random matching or computing the semantically nearest image neighbors using e.g . the cosine distance . Next , we construct the whole patches set Pi on the candidate peer images set Si for each image i by clipping the corresponding patches in the different locations on an input or a feature map . Following the establishment of patch set Pi , we construct K nearest neighbor patch graphs based on the distance of patch features in order to find the neighbors of each patch in patch set Pi for ∀i = 1 , .. , N . Mathematically , following the definition in the PeerNet , let zip be the p-th patch on the input or feature map Zi for the i-th image within one batch . Then denote the k-th nearest patch neighbor for zip as z jk qk taken from the patch qk of the peer image jk in the candidate set Si . Next , in order to leverage the knowledge from neighbors , different from graph attention mechanism in PeerNet , we apply a more straightforward linear interpolation on the neighboring patches for the current patch zip . Then , the general formulation of our Patch-level Neighborhood Interpolation can be presented as follows : z̃ip = z i p + K∑ k=1 ηipk ( z jk qk − zip ) , ( 1 ) where ηipk is the combination coefficient for the p-th patch of i-th image w.r.t its k-th patch neighbor , which can be computed through the power iteration similar to the manner of VAT , or through random sampling from a specific distribution in randomness-based regularization , e.g. , Mixup and its variants . Moreover , the choice of linear interpolation in Eq . 1 enjoys great computational advantage over the nonlinear GAT form in PeerNet in the computation of networks . Finally , after the patchlevel linear interpolation on patch features , we can obtain the refined graph-based representation Z̃i for i-th image , ∀i = 1 , ... , N . Note that our proposed method can explicitly combine the advantages of manifold regularization and non-local filtering in a flexible way , which we have a more detailed discussion about in Appendix B . Besides , to further demonstrate the generality and effectiveness of our Pani method , we provide Pani-version of two typical regularization strategies , i.e. , Virtual adversarial Training and Mixup as well its variant MixMatch , and verify the superiority of our Pani strategy on the significant boosting of accuracy . 2.1 PANI VAT . Based on our Patch-level Neighborhood Interpolation framework , we can construct a novel Pani VAT that utilizes the combination or interpolation of patch neighbors for each sample to manipulate the non-local perturbations , thus providing more informative adversarial smoothness in semi-supervised setting . Consider a more general composite function form of the classifier f , i.e. , f ( x ) = g ( z ) and z = h ( x ) where z denotes the hidden feature of input x or the input itself when the reduced form happens . Combining VAT formulation , i.e. , Eq . 7 in Appendix A , and Pani formulation , i.e. , Eq . 1 , we reformulate our Pani VAT with perturbations on L layers in a deep neural network as follows : max η D [ g ( z ) , g ( z̃ ( η ) ) ] s.t . L∑ l=1 w2l ‖η ( l ) ‖2 ≤ 2 , ( 2 ) where D measures the divergence between two distributions . η = { ηipk } denotes the generic perturbations from our Pani method and η ( l ) indicates the perturbations in l-th layer of network . z̃ ( η ) = { z̃ip } represents the smoothed feature map imposed by perturbation η considering all patches in the way shown in Eq . 1 . In particular , when L = 1 , adversarial perturbations are only imposed on the input feature , which is similar to the traditional virtual adversarial perturbations . Additionally , wl is the hyper-parameter , adjusting the weight of perturbation η ( l ) in different layers with the overall perturbations restrained in an -ball . Next , we still utilize the similar power iteration and finite difference proposed in VAT ( Miyato et al. , 2017 ) to compute the desired perturbation η∗ . Then the resulting full loss function is defined as : min θ L0 + βEx∼DRvadv ( x , η∗ ; θ ) , ( 3 ) where L0 is the original supervised loss and β controls the degree of adversarial smoothness . Rvadv ( x , y , η∗ ) = D [ g ( z ) , g ( z̃ ( η∗ ) ) ] can be attained after solving the optimization problem in Eq . 2 . For the implementation details , we describe them in Algorithm 1 . Algorithm 1 : Pani VAT within a Batch 1 : Input : neighbors K1 and K2 , classifier f , batch size B , perturbed layers L 2 : Initialization : combination coefficient η 3 : ComputeK1 nearest image neighbors based on the distance of the second last layer output from f and obtain K1 ( K1 ≤ B ) peer images set Si for each image i . 4 : for l = 1 to L do : 5 : Compute the patch set Pi for all K1 peer images on layer l for each image i . 6 : Construct a K2 nearest patch neighbors graph for each patch in each image i . 7 : Conduct Patch-level Neighborhood Interpolation via Eq . 1 for each patch . 8 : end for 9 : Conduct power iteration and finite difference in VAT to compute η∗ constrained by Eq . 2 . 10 : ReturnRvadv ( x , η∗ ; θ ) Remark . As shown in the adversarial part of Figure 1 , the rationality of our Pani VAT method lies in the fact that the constructed perturbations can entail more non-local information coming from the neighbors of current sample . Through the delicate patch-level interpolation among neighbors of each patch , the resulting non-local virtual adversarial perturbations are expected to provide more informative smoothness , thus enhancing the performance of classifier in the semi-supervised setting .

Summary: This paper proposed a new regularization method via patch level interpolation.  During the training,  images within a batch will be used to construct an image graph. For example, for a certain image, its nearest neighbors in the feature spaces will be used.  Then patches from its neighbors will be used to interpolate to each patch in that given image.  Thus a straightforward application for such regularization is semi-supervised training.  Moreover, in this paper it has demonstrated such regularization can be extended with virtual adversarial training and mixup training.