Detecting Memorization in ReLU Networks

1 INTRODUCTION . A fundamental challenge in machine learning is balancing the bias-variance tradeoff , where overly simple learning models underfit the data ( suboptimal performance on the training data ) and overly complex models are expected to overfit or memorize the data ( perfect training set performance , but suboptimal test set performance ) . The latter direction of this tradeoff has come into question with the observation that deep neural networks do not memorize their training data despite having sufficient capacity to do so ( Zhang et al. , 2016 ) , the explanation of which is a matter of much interest . Due to their convenient gradient properties and excellent performance in practice , rectified-linear units ( ReLU ) have been widely adopted and are now ubiquitous in the field of deep learning . In addition , the relative simplicity of this function ( max ( · , 0 ) ) makes the analysis of ReLU networks more straight-forward than networks with other activation functions . We propose a new notion of ‘ non-linearity ’ of a ReLU layer with respect to an input batch . We show that networks that generalize well have deep layers that are approximately linear with respect to batches of similar inputs . In contrast , networks that memorize their training data are highly nonlinear with respect to similar inputs , even in deep layers . Our method is based on the fact that the main source of non-linearity in ReLU networks is the threshold at zero . This thresholding determines the support of the resulting activation matrix , which plays an important role in the analysis of non-negative matrices . As we discuss in Section 3 , the non-negative rank of a matrix is constrained by the shape of the support , and is therefore indicative of the degree of non-linearity in a ReLU activation matrix with respect to the input . Although computing the non-negative rank is NP-hard ( Vavasis , 2009 ) , we can restrict it with approximate non-negative matrix factorization ( NMF ) ( Lee & Seung , 1999 ) . Consequently , we propose to estimate the ‘ non-linearity ’ of a ReLU layer with respect to an input batch by performing NMF on a grid over the approximation rank k , and measuring the impact on network performance . This procedure can be seen as measuring the robustness of a neural network to increasing compression of its activations . We therefore compare our NMF-based approach to two additional dimensionality reduction techniques , namely principal component analysis ( PCA ) and random ablations . We informally define memorization as the implicit learning of a rule that associates a specific sample ( i.e. , with index i ) to a particular label ( e.g. , with index j ) . Such a rule does not benefit the network in terms of improving its performance on new data . We show that our NMF-based approach is extremely sensitive to memorization in neural networks . We report results for a variety of neural network architectures trained on several image and audio datasets . We conduct a layer-by-layer analysis and our results reveal interesting details on the internal mechanism of memorization in neural networks . Finally , as an indicator for memorization , we use our proposed measure to perform early stopping . 2 RELATED WORK . The study of factors involved in the bias-variance tradeoff in learning models goes back several decades . Classical results in statistical learning consider properties of learning models such as the VC-dimension ( Vapnik , 1998 ) and Rademacher complexity ( Bartlett & Mendelson , 2002 ) . These properties give generalization bounds in terms of the capacity model to ( over ) fit data . When considering the vast capacity of deep neural networks , such bounds become irrelevant and fail to explain their ability to generalize well in practice ( Zhang et al. , 2016 ; Bartlett et al. , 2017 ) . More direct analyses have been done with respect to a specific setting of model parameters . For instance , Bartlett ( 1998 ) showed that the number of weights in a network is less important compared to their scalar value ( e.g . ` 2-norm ) , and more recently Bartlett et al . ( 2017 ) presented a bound for deep neural networks based on the product of spectral norms of the network ’ s weight matrices . Achille & Soatto ( 2017 ) showed that memorizing networks contain more information in their weights . Methods to explain generalization have been proposed that examine a network ’ s robustness to perturbations ( Hochreiter & Schmidhuber , 1997 ; Chaudhari et al. , 2016 ; Keskar et al. , 2016 ; Neyshabur et al. , 2017 ; Li et al. , 2017 ) . These methods propose the notion of flatness of minima on the loss surface , assuming that perturbing the parameters without dramatically changing performance is an indicator of the generalization of a network . However , any reversible transformation , such as simple scaling , can arbitrarily manipulate the local flatness without affecting generalization ( Dinh et al. , 2017 ) . The procedure we propose can be viewed as applying perturbations , albeit to activations and not parameters , and must address this concern . The perturbations we apply to activations account for magnitude , since they depend on a change of rank or non-negative rank of the activation matrix , a property which is robust to rescaling and similar reversible transformations . In contrast to the methods described thus far , which deal exclusively with the parameters of the model , methods have been developed that account for the role of the data distribution . Liang et al . ( 2017 ) proposed to use the Fisher-Rao norm , which uses the geometry of the data distribution to weigh the contribution of different model parameters . The empirical studies of Morcos et al . ( 2018 ) and Novak et al . ( 2018 ) explore robustness to specific types of noise . The former uses Gaussian noise and masking noise injected into hidden activations , while the latter interpolates between input samples to study network behavior on and off the data manifold . In both cases , robustness to noise proved a reliable indicator for good generalization . Additionally , Arora et al . ( 2018 ) derive generalization bounds in terms of robustness to noise . Our experimental setup is reminiscent of Morcos et al . ( 2018 ) in that both methods apply a form of compression to hidden activations and test for robustness to this type of noise . Specifically , they set random axis-aligned directions in feature space to zero which can be viewed as a crude form of dimensionality reduction , i.e. , by simply removing canonical dimensions . In our experiments we refer to this method as random ablations . Our results show that robustness to NMF compression is much more correlated with low memorization/high generalization than robustness to random ablations . Arpit et al . ( 2017 ) have also studied various empirical aspect of memorization . As a dimensionality reduction technique , NMF has gained popularity due to its producing meaningful factorizations that lend themselves to qualitative interpretation across various domains such as document clustering ( Xu et al. , 2003 ) , audio source separation ( Grais & Erdogan , 2011 ) , and face recognition ( Guillamet & Vitria , 2002 ) . In the context of deep convolutional neural networks , Collins et al . ( 2018 ) applied NMF to the activations of an image classifier and showed that the result gives a decomposition into semantic parts , which benefits from the transformation invariance learned by the neural network . 3 METHOD . Consider a ReLU layer of a neural network , parameterized by a weight matrix W ∈ Rm×q . For a batch of n inputs X ∈ Rn×m , we compute the layer activation matrix A as follows : A = max ( XW , 0 ) ∈ Rn×q+ , ( 1 ) where R+ are the non-negative reals . We omit the bias term for notational convenience . 3.1 RELU-INDUCED SUPPORT . The processing of a single input x by a ReLU network is equivalent to sampling a sub-network that is linear with respect to the sample ( Wang et al. , 2016 ) . This could be accomplished by simply setting to zero the columns of each W whose dot product with the input is negative ( and would thus be set to zero by ReLU ) , and then removing the thresholding.1 Extending this notion to a batch of several input samples to a ReLU layer , suppose the samples are sufficiently close to each other such that they all share the same ReLU mask m ∈ { 0 , 1 } q . In this case , we may say that the layer is linear with respect to its input batch . This is because , for the entire batch , instead of using ReLU , we could zero out a subset of columns and obtain a linear system , i.e. , A = XW diag ( m ) . For an activation matrix A ( Equation 1 ) , we consider the support M = supp ( A ) , which we describe as a binary 0/1 matrix where Mi , j = 1 where Ai , j > 0 . Because A is a ReLU activation matrix , the structure of M is mainly determined by the thresholding at zero.2 Because thresholding is the main source of non-linearity in a ReLU network , the support takes on a special meaning in this case . 3.2 RECTANGLE COVER NUMBER AND NON-NEGATIVE RANK . We want to characterize how close to being linear a layer is with respect to its input X by examining the support of the resulting activations M . If all the rows of M are identical to a unique vector m , we can say the layer is completely linear with respect to X . In general , the ‘ simpler ’ the support M , the closer to linearity the layer . One measure that captures this idea is the rectangle cover number of a matrix , rc ( M ) , an important quantity in the study of communication complexity ( Klauck , 2003 ) . Also known as the Boolean rank , rc ( M ) is the smallest number r for which there exist binary matrices UB ∈ { 0 , 1 } n×r , VB ∈ 1This is a similar intuition to that of viewing dropout as an approximation to a model ensemble , where the dropout mask is seen to sample a sub-network ( Srivastava et al. , 2014 ) . 2The probability of an activation value being exactly zero prior to thresholding is negligible . { 0 , 1 } r×q such that their Boolean matrix multiplication satisfies M = UBVB . As a complexity measure for ReLU activations , rc ( M ) = 1 means the layer is linear with respect to its input , and higher values rc ( M ) imply increasing non-linearity . This is visualized in Figure 1 . Intuitively , imagine having to fit a layer with ‘ ReLU switches ’ , each of which controls a subset of weight matrix columns . In the linear case , one switch would suffice to describe the data . In the most non-linear case , we would require a switch for every column , which is also the maximal value of rc ( M ) . Because computing the rectangle cover number rc ( M ) is complex , several approximations and bounds to it have been studied ( Fiorini et al. , 2013 ) . For the support of a non-negative matrix , a well-known upper-bound is : rc ( supp ( A ) ) ≤ rank+ ( A ) , ( 2 ) where rank+ ( A ) is the non-negative rank of A ( Gillis & Glineur , 2012 ) that is defined as the smallest number k such that there exist non-negative matrices U+ ∈ Rn×k+ , V+ ∈ R k×q + such that A = U+V+ . Similar to the rectangle cover number , the non-negative rank is hard-constrained by the combinatorial arrangement of supp ( A ) , but additionally accounts for the actual value in the non-zero entries of A . While computing rank+ ( A ) is not easier than computing rc ( supp ( A ) ) , we can restrict it by performing approximate non-negative matrix factorization ( NMF ) .

Summary: The contribution of the paper is in proposing a quantitative measure of memorization based on the assumption that the activations at the deeper layers of a *generalizing* deep network should be invariant to intra-class variations. The measure corresponds to how well can the activation matrix of a batch be approximated by a low-rank decomposition. The paper proposes to use approximate non-negative matrix factorization and compares it to PCA. As for “wellness” it uses the final accuracy of the network after the activation is approximated in some layer(s).