Stop memorizing: A data-dependent regularization framework for intrinsic pattern learning

1 INTRODUCTION . It remains an open question why DNNs , typically with far more model parameters than training samples , can achieve such small generalization error . Previous work used various complexity measures from statistical learning theory , such as VC dimension ( Vapnik , 1998 ) , Radamacher complexity ( Bartlett & Mendelson , 2002 ) , and uniform stability ( Bousquet & Elisseeff , 2002 ; Poggio et al. , 2004 ) , to provide an upper bound for the generalization error , suggesting that the effective capacity of DNNs , possibly with some regularization techniques , is usually limited . However , the experiments by Zhang et al . ( 2017 ) showed that , even with data-independent regularization , DNNs can perfectly fit the training data when the true labels are replaced by random labels , or when the training data are replaced by Gaussian noise . This suggests that DNNs with data-independent regularization have enough capacity to “ memorize ” the training data . This poses an interesting question for network regularization design : is there a way for DNNs to refuse to ( over ) fit training samples with random labels , while exhibiting better generalization power than conventional DNNs when trained with true labels ? Such networks are very important because they will extract only intrinsic patterns from the training data instead of memorizing miscellaneous details . One would expect that data-dependent regularizations should be a better choice for reducing the memorizing capacity of DNNs . Such regularizations are typically enforced by penalizing the standard softmax cross entropy loss with an extra geometric loss which regularizes the feature geometry ( Lezama et al. , 2018 ; Zhu et al. , 2018 ; Wen et al. , 2016 ) . However , regularizing DNNs with an extra geometric loss has two disadvantages : First , the output of the softmax layer , usually viewed as a probability distribution , is typically inconsistent with the feature geometry enforced by the geometric loss . Therefore , the geometric loss typically has a small weight to avoid jeopardizing the minimization of the softmax loss . Second , we find that DNNs with such regularization can still perfectly ( over ) fit random training samples or random labels . The reason is that the geometric loss ( because of its small weight ) is ignored and only the softmax loss is minimized . This suggests that simply penalizing the softmax loss with a geometric loss is not sufficient to regularize DNNs . Instead , the softmax loss should be replaced by a validation loss that is consistent with the enforced geometry . More specifically , every training batch B is split into two sub-batches , the geometry batch Bg and the validation batch Bv . The geometric loss lg is imposed on the features of Bg for them to exhibit a desired geometric structure . A semi-supervised learning algorithm based on the proposed feature geometry is then used to generate a predicted label distribution for the validation batch , which combined with the true labels defines a validation loss on Bv . The total loss on the training batch B is then defined as the weighted sum l = lg + λlv . Because the predicted label distribution on Bv is based on the enforced geometry , the geometric loss lg can no longer be neglected . Therefore , lg and lv will be minimized simultaneously , i.e. , the geometry is correctly enforced ( small lg ) and it can be used to predict validation samples ( small lv ) . We call such DNNs Geometrically-Regularized-Self-Validating neural Networks ( GRSVNets ) . See Figure 1a for a visual illustration of the network architecture . GRSVNet is a general architecture because every consistent geometry/validation pair can fit into this framework as long as the loss functions are differentiable . In this paper , we focus on a particular type of GRSVNet , the Orthogonal-Low-rank-Embedding-GRSVNet ( OLE-GRSVNet ) . More specifically , we impose the OLE loss ( Qiu & Sapiro , 2015 ) on the geometry batch to produce features residing in orthogonal subspaces , and we use the principal angles between the validation features and those subspaces to define a predicted label distribution on the validation batch . We prove that the loss function obtains its minimum if and only if the subspaces of different classes spanned by the features in the geometry batch are orthogonal , and the features in the validation batch reside perfectly in the subspaces corresponding to their labels ( see Figure 1e ) . We show in our experiments that OLE-GRSVNet has better generalization performance when trained on real data , but it refuses to memorize the training samples when given random training data or random labels , which suggests that OLE-GRSVNet effectively learns intrinsic patterns . Our contributions can be summarized as follows : • We proposed a general framework , GRSVNet , to effectively impose data-dependent DNN regularization . The core idea is the self-validation of the enforced geometry with a consistent validation loss on a separate batch of features . • We study a particular case of GRSVNet , OLE-GRSVNet , that can produce highly discriminative features : samples from the same class belong to a low-rank subspace , and the subspaces for different classes are orthogonal . • OLE-GRSVNet achieves better generalization performance when compared to DNNs with conventional regularizers . And more importantly , unlike conventional DNNs , OLEGRSVNet refuses to fit the training data ( i.e. , with a training error close to random guess ) when the training data or the training labels are randomly generated . This implies that OLE-GRSVNet never memorizes the training samples , only learns intrinsic patterns . 2 RELATED WORK . Many data-dependent regularizations focusing on feature geometry have been proposed for deep learning ( Lezama et al. , 2018 ; Zhu et al. , 2018 ; Wen et al. , 2016 ) . The center loss ( Wen et al. , 2016 ) produces compact clusters by minimizing the Euclidean distance between features and their class centers . LDMNet ( Zhu et al. , 2018 ) extracts features sampling a collection of low dimensional manifolds . The OLE loss ( Lezama et al. , 2018 ; Qiu & Sapiro , 2015 ) increases inter-class separation and intra-class similarity by embedding inputs into orthogonal low-rank subspaces . However , as mentioned in Section 1 , these regularizations are imposed by adding the geometric loss to the softmax loss , which , when viewed as a probability distribution , is typically not consistent with the desired geometry . Our proposed GRSVNet instead uses a validation loss based on the regularized geometry so that the predicted label distribution has a meaningful geometric interpretation . The way in which GRSVNets impose geometric loss and validation loss on two separate batches of features extracted with two identical baseline DNNs bears a certain resemblance to the siamese network architecture ( Chopra et al. , 2005 ) used extensively in metric learning ( Cheng et al. , 2016 ; Hadsell et al. , 2006 ; Hu et al. , 2014 ; Schroff et al. , 2015 ; Sun et al. , 2014 ) . The difference is , unlike contrastive loss ( Hadsell et al. , 2006 ) and triplet loss ( Schroff et al. , 2015 ) in metric learning , the feature geometry is explicitly regularized in GRSVNets , and a representation of the geometry , e.g. , basis of the low-rank subspace , can be later used directly for the classification of test data . Our work is also related to two recent papers ( Zhang et al. , 2017 ; Arpit et al. , 2017 ) addressing the memorization of DNNs . Zhang et al . ( 2017 ) empirically showed that conventional DNNs , even with data-independent regularization , are fully capable of memorizing random labels or random data . Arpit et al . ( 2017 ) argued that DNNs trained with stochastic gradient descent ( SGD ) tend to fit patterns first before memorizing miscellaneous details , suggesting that memorization of DNNs depends also on the data itself , and SGD with early stopping is a valid strategy in conventional DNN training . We demonstrate in our paper that when data-dependent regularization is imposed in accordance with the validation , GRSVNets will never memorize random labels or random data , and only extracts intrinsic patterns . An explanation of this phenomenon is provided in Section 4 . 3 GRSVNET AND ITS SPECIAL CASE : OLE-GRSVNET . As pointed out in Section 1 , the core idea of GRSVNet is to self-validate the geometry using a consistent validation loss . To contextualize this idea , we study a particular case , OLE-GRSVNet , where the regularized feature geometry is orthogonal low-rank subspaces , and the validation loss is defined by the principal angles between the validation features and the subspaces . 3.1 OLE LOSS . The OLE loss was originally proposed by Qiu & Sapiro ( 2015 ) . Consider a K-way classification problem . Let X = [ x1 , . . . , xN ] ∈ Rd×N be a collection of data points { xi } Ni=1 ⊂ Rd . Let Xc denote the submatrix of X formed by inputs of the c-th class . Qiu & Sapiro ( 2015 ) proposed to learn a linear transformation T : Rd → Rd that maps data from the same class Xc into a low-rank subspace , while mapping the entire data X into a high-rank linear space . This is achieved by solving : min T : Rd→Rd K∑ c=1 ‖TXc‖∗ − ‖TX‖∗ , s.t . ‖T‖2 = 1 , ( 1 ) where ‖ · ‖∗ is the matrix nuclear norm , which is a convex lower bound of the rank function on the unit ball in the operator norm ( Recht et al. , 2010 ) . The norm constraint ‖T‖2 = 1 is imposed to avoid the trivial solution T = 0 . It is proved by Qiu & Sapiro ( 2015 ) that the OLE loss ( 1 ) is always nonnegative , and the global optimum value 0 is obtained if TXc⊥TXc′ , ∀c 6= c′ . Lezama et al . ( 2018 ) later used OLE loss as a data-dependent regularization for deep learning . Given a baseline DNN that maps a batch of inputs X into the features Z = Φ ( X ; θ ) , the OLE loss on Z is lg ( Z ) = K∑ c=1 ‖Zc‖∗ − ‖Z‖∗ = K∑ c=1 ‖Φ ( Xc ; θ ) ‖∗ − ‖Φ ( X ; θ ) ‖∗ . ( 2 ) The OLE loss is later combined with the standard softmax loss for training , and we will henceforth call such network “ softmax+OLE. ” Softmax+OLE significantly improves the generalization performance , but it suffers from two problems because of the inconsistency between the softmax loss and the OLE loss : First , the learned features no longer exhibit the desired geometry of orthogonal low-rank subspaces . Second , as will be shown in Section 4 , softmax+OLE is still capable of memorizing random data or random labels , i.e. , it has not reduced the memorizing capacity of DNNs . 3.2 OLE-GRSVNET . We will now explain how to incorporate OLE loss into the GRSVNet framework . First , let us better understand the geometry enforced by the OLE loss by stating the following theorem . Theorem 1 . Let Z = [ Z1 , . . . , Zc ] be a horizontal concatenation of matrices { Zc } Kc=1 . The OLE loss lg ( Z ) defined in ( 2 ) is always nonnegative . Moreover , lg ( Z ) = 0 if and only if Z∗cZc′ = 0 , ∀c 6= c′ , i.e. , the column spaces of Zc and Zc′ are orthogonal . The proof of Theorem 1 , as well as those of the remaining theorems , is detailed in the Appendix . Note that Theorem 1 , which ensures that the OLE loss is minimized if and only if features of different classes are orthogonal , is a much stronger result than that by Qiu & Sapiro ( 2015 ) . We then need to define a validation loss lv that is consistent with the geometry enforced by lg . A natural choice would be the principal angles between the validation features and the subspaces spanned by { Zc } Kc=1 . Now we detail the architecture for OLE-GRSVNet . Given a baseline DNN , we split every training batch X ∈ Rd×|B| into two sub-batches , the geometry batch Xg ∈ Rd×|Bg| and the validation batch Xv ∈ Rd×|Bv| , which are mapped by the same baseline DNN into features Zg = Φ ( Xg ; θ ) and Zv = Φ ( Xv ; θ ) . The OLE loss lg ( Zg ) is imposed on the geometry batch to ensure span ( Zgc ) are orthogonal low-rank subspaces , where span ( Zgc ) is the column space of Z g c . Let Z g c = UcΣcV ∗ c be the ( compact ) singular value decomposition ( SVD ) of Zgc , then the columns of Uc form an orthonormal basis of span ( Zgc ) . For any feature z = Φ ( x ; θ ) ∈ Zv in the validation batch , its projection onto the subspace span ( Zgc ) is projc ( z ) = UcU ∗ cz . The cosine similarity between z and projc ( z ) is then defined as the ( unnormalized ) probability of x belonging to class c , i.e. , ŷc = P ( x ∈ c ) , 〈 z , projc ( z ) max ( ‖projc ( z ) ‖ , ε ) 〉/ K∑ c′=1 〈 z , projc′ ( z ) max ( ‖projc′ ( z ) ‖ , ε ) 〉 , ( 3 ) where a small ε is chosen for numerical stability . The validation loss for x is then defined as the cross entropy between the predicted distribution ŷ = ( ŷ1 , . . . , ŷK ) T ∈ RK and the true label y ∈ { 1 , . . . , K } . More specifically , let Yv ∈ R1×|Bv| and Ŷv ∈ RK×|Bv| be the collection of true labels and predicted label distributions on the validation batch , then the validation loss is defined as lv ( Y v , Ŷv ) = 1 |Bv| ∑ x∈Xv H ( δy , ŷ ) = − 1 |Bv| ∑ x∈Xv log ŷy , ( 4 ) where δy is the Dirac distribution at label y , and H ( · , · ) is the cross entropy between two distributions . The empirical loss l on the training batch X is then defined as l ( X , Y ) = l ( [ Xg , Xv ] , [ Yg , Yv ] ) = lg ( Z g ) + λlv ( Y v , Ŷv ) . ( 5 ) See Figure 1a for a visual illustration of the OLE-GRSVNet architecture . Because of the consistency between lg and lv , we have the following theorem : Theorem 2 . For any λ > 0 , and any geometry/validation splitting of X = [ Xg , Xv ] satisfying Xv contains at least one sample for each class , the empirical loss function defined in ( 5 ) is always nonnegative . l ( X , Y ) = 0 if and only if both of the following conditions hold true : • The features of the geometry batch belonging to different classes are orthogonal , i.e. , span ( Zgc ) ⊥ span ( Zgc′ ) , ∀c 6= c′ • For every datum x ∈ Xvc , i.e. , x belongs to class c in the validation batch , its feature z = Φ ( x ; θ ) belongs to span ( Zgc ) . Moreover , if l < ∞ , then rank ( span ( Zgc ) ) ≥ 1 , ∀c , i.e. , Φ ( · ; θ ) does not trivially map data into 0 . Remark : The requirement that λ > 0 is crucial in Theorem 2 , because otherwise the network can map every input into 0 and achieve the minimum . This is validated in our numerical experiments . After the training process has finished , we can then map the entire training data Xall = [ Xall1 , . . . , X all K ] ( or a random portion of Xall ) into their features Zall = Φ ( Xall ; θ∗ ) , where θ∗ is the learned parameter . The low-rank subspace span ( Zallc ) for class c can be obtained via the SVD of Z all c . The label of a test datum x is then determined by the principal angles between z = Φ ( x ; θ∗ ) and { span ( Zallc ) } Kc=1 .

Summary: The paper proposes a data-dependent regularization method which is coupled with softmax loss to train deep neural networks for classification. The paper turns to Orthogonal Low-rank Embedding (OLE) loss for the geometric constraint that one class of data/feature are assumed to reside on a low-rank subspace that subspaces of different classes are orthogonal ideally. The probability in the softmax is then modeled as cosine similarity between data feature and the class-specific subspaces. In this way, geometric loss and softmax loss have the common goal for optimization. Moreover, during training, the geometry enforced on one batch of features is simultaneously validated on a separate batch using a validation loss. The experiments seem to suggest such a model helps avoid overfitting/memorizing noisy training data. The paper reads well and is easy to follow.