SVMax: A Feature Embedding Regularizer

1 INTRODUCTION . A neural network ’ s knowledge is embodied in both its weights and activations . This difference manifests in how network pruning and knowledge distillation tackle the model compression problem . While pruning literature Li et al . ( 2016 ) ; Luo et al . ( 2017 ) ; Yu et al . ( 2018 ) compresses models by removing less significant weights , knowledge distillation Hinton et al . ( 2015 ) reduces computational complexity by matching a cumbersome network ’ s last layer activations ( logits ) . This perspective , of weight-knowledge versus activation-knowledge , emphasizes how neural network literature is dominated by explicit weight regularizers . In contrast , this paper leverages singular value decomposition ( SVD ) to regularize a network through its last layer activations – its feature embedding . Our formulation is inspired by principal component analysis ( PCA ) . Given a set of points and their covariance , PCA yields the set of orthogonal eigenvectors sorted by their eigenvalues . The principal component ( first eigenvector ) is the axis with the highest variation ( largest eigenvalue ) as shown in Figure 1c . The eigenvalues from PCA , and similarly the singular values from SVD , provide insights about the embedding space structure . As such , by regularizing the singular values , we reshape the feature embedding . The main contribution of this paper is to leverage the singular value decomposition of a network ’ s activations to regularize the embedding space . We achieve this objective through singular value maximization ( SVMax ) . The SVMax regularizer is oblivious to both the input-class ( labels ) and the sampling strategy . Thus it promotes a uniform embedding space in both supervised and unsupervised learning . Furthermore , we present a mathematical analysis of the mean singular value ’ s lower and upper bounds . This analysis makes tuning the SVMax ’ s balancing-hyperparameter easier , when the feature embedding is normalized to the unit circle . The SVMax regularizer promotes a uniform embedding space . During training , SVMax speeds up convergence by enabling large learning rates . The SVMax regularizer integrates seamlessly with various ranking losses . We apply the SVMax regularizer to the last feature embedding layer , but the same formulation can be applied to intermediate layers . The SVMax regularizer mitigates model collapse in both retrieval networks and generative adversarial networks ( GANs ) Goodfellow et al . ( 2014 ) ; Srivastava et al . ( 2017 ) ; Metz et al . ( 2017 ) . Furthermore , the SVMax regularizer is useful when training unsupervised feature embedding networks with a contrastive loss ( e.g. , CPC ) Noroozi et al . ( 2017 ) ; Oord et al . ( 2018 ) ; He et al . ( 2019 ) ; Tian et al . ( 2019 ) . In summary , we propose singular value maximization to regularize the feature embedding . In addition , we present a mathematical analysis of the mean singular value ’ s lower and upper bounds to reduce hyperparameter tuning ( Sec . 3 ) . We quantitatively evaluate how the SVMax regularizer significantly boosts the performance of ranking losses ( Sec . 4.1 ) . And we provide a qualitative evaluation of using SVMax in the unsupervised learning setting via GAN training ( Sec . 4.2 ) . 2 RELATED WORK . Network weight regularizers dominate the deep learning regularizer literature , because they support a large spectrum of tasks and architectures . Singular value decomposition ( SVD ) has been applied as a weight regularizer in several recent works Zhang et al . ( 2018 ) ; Sedghi et al . ( 2018 ) ; Guo & Ye ( 2019 ) . Zhang et al . ( 2018 ) employ SVD to avoid vanishing and exploding gradients in recurrent neural networks . Similarly , Guo & Ye ( 2019 ) bound the singular values of the convolutional layer around 1 to preserve the layer ’ s input and output norms . A bounded output norm mitigates the exploding/vanishing gradient problem . Weight regularizers share the common limitation that they do not enforce an explicit feature embedding objective and are thus ineffective against model collapse . Feature embedding regularizers have also been extensively studied , especially for classification networks Rippel et al . ( 2015 ) ; Wen et al . ( 2016 ) ; He et al . ( 2018 ) ; Hoffman et al . ( 2019 ) ; Taha et al . ( 2020 ) . These regularizers aim to maximize class margins , class compactness , or both simultaneously . For instance , Wen et al . ( 2016 ) propose center loss to explicitly learn class representatives and thus promote class compactness . In classification tasks , test samples are assumed to lie within the same classes of the training set , i.e. , closed-set identification . However , retrieval tasks , such as product re-identification , assume an open-set setting . Because of this , a retrieval network regularizer should aim to spread features across many dimensions to fully utilize the expressive power of the embedding space . Recent literature Sablayrolles et al . ( 2018 ) ; Zhang et al . ( 2017 ) has recognized the importance of a spread-out feature embedding . However , this literature is tailored to triplet loss and therefore assumes a particular sampling procedure . In this paper , we leverage SVD as a regularizer because it is simple , differentiable Ionescu et al . ( 2015 ) , and class oblivious . SVD has been used to promote low rank models to learn compact intermediate layer representations Kliegl et al . ( 2017 ) ; Sanyal et al . ( 2019 ) . This helps compress the network and speed up matrix multiplications on embedded devices ( iPhone and Raspberry Pi ) . In contrast , we regularize the embedding space through a high rank objective . By maximizing the mean singular value , we promote a higher rank representation – a spread-out embedding . 3 SINGULAR VALUE MAXIMIZATION ( SVMAX ) . We first introduce our mathematical notation . Let I denote the image space andEI ∈ Rd denote the feature embeddings space , where d is the dimension of the features . A feature embedding network is a function Fθ : I → EI , parameterized by the network ’ s weights θ . We quantify similarity between an image pair ( I1 , I2 ) via the Euclidean distance in feature space , i.e. , ‖EI1 − EI2‖2 . During training , a 2D matrix E ∈ Rb×d stores b samples ’ embeddings , where b is the mini-batch size . Assuming b ≥ d , the singular value decomposition ( SVD ) of E provides the singular values S = [ s1 , . , si , . , sd ] , where s1 and sd are the largest and smallest singular values , respectively . We maximize the mean singular value , sµ = 1d ∑d i=1 si , to regularize the network ’ s last layer activations – the feature embedding . By maximizing the mean singular value , the deep network spreads out its embeddings . This has the added benefit of implicitly regularizing the network ’ s weights θ . The proposed SVMax regularizer integrates with both supervised and unsupervised feature embedding networks as follows LNN = Lr − λ 1 d d∑ i=1 si = Lr − λsµ , ( 1 ) where Lr is the original network loss and λ is a balancing hyperparameter . Lower and Upper Bounds of the Mean Singular Value : One caveat to equation 1 is the hyperparameter λ . It is difficult to tune because the mean singular value sµ depends on the range of values inside E and its dimensions ( b , d ) . Thus , changing the batch size or embedding dimension requires a different λ . To address this , we utilize a common assumption in metric learning – the unit circle ( L2-normalized ) embedding assumption . This assumption provides both lower and upper bounds on ranking losses . This will allow us to impose lower and upper bounds on sµ . For an L2-normalized embeddingE , the largest singular value s1 is maximum when the matrix-rank of E equals one , i.e. , rank ( E ) = 1 , and si = 0 for i ∈ [ 2 , d ] . Horn & Johnson ( 1991 ) provide an upper bound on this largest singular value s1 as s∗ ( E ) ≤ √ ||E||1||E||∞ . This holds in equality for all L2-normalized E ∈ Rb×d with rank ( E ) = 1 . For an L2-normalized matrix E with ||E||1 = b , and ||E||∞ = 1 , this gives : s∗ ( E ) = √ ||E||1||E||∞ = √ b . ( 2 ) Thus , the lower bound L on sµ is L = s∗ ( E ) d = √ b d . Similarly , an upper bound is defined on the sum of the singular values Turkmen & Civciv ( 2007 ) ; Kong et al . ( 2018 ) ; Friedland & Lim ( 2016 ) . This summation is formally known as the nuclear norm of a matrix ||E||∗ . Hu ( 2015 ) established an upper bound on this summation using the Frobenius Norm ||E||F as follows ||E||∗ ≤ √ b× d max ( b , d ) ||E||F , ( 3 ) where ||E||F = ( ∑rows i=1 ∑cols j=1 |Eij | 2 ) 1 2 = √ b because of the L2-normalization assumption . Accordingly , the lower and upper bounds of sµ are [ L , U ] = [ s∗ ( E ) d , ||E||∗ d ] . With these bounds , we rewrite our final loss function as follows LNN = Lr + λ exp ( U − sµ U − L ) . ( 4 ) The SVMax regularizer grows exponentially ∈ [ 1 , e ] . We employ this loss function in all our retrieval experiments . It is important to note that the L2-normalized assumption makes λ tuning easier , but it is not required . Equation 4 makes the hyperparameter λ only dependent on the range of Lr which is also bounded for ranking losses . Lower and Upper Bounds of Ranking Losses : We briefly show that ranking losses are bounded when assuming an L2-normalized embedding . Equations 5 and 6 show triplet and contrastive losses , respectively , and their corresponding bounds [ L , U ] . TL ( a , p , n ) ∈T = [ ( D ( bac , bpc ) −D ( bac , bnc ) +m ) ] + [ L , U ] −−−→ [ 0 , 2 +m ] , ( 5 ) CL ( x , y ) ∈P = ( 1− δx , y ) D ( bxc , byc ) ) + δx , y [ m−D ( bxc , byc ) ) ] + [ L , U ] −−−→ [ 0 , 2 ] , ( 6 ) where [ • ] + = max ( 0 , • ) , m < 2 is the margin between classes , since 2 is the maximum distance on the unit circle . b•c and D ( , ) are the embedding and Euclidean distance functions , respectively . In equation 5 , a , p , and n are the anchor , positive , and negative images in a single triplet ( a , p , n ) from the triplets set T . In equation 6 , x and y form a single pair of images from the pairs set P . δx , y = 1 when x and y belong to different classes ; zero otherwise . In the supplementary material , we ( 1 ) show similar analysis for N-pair and angular losses , ( 2 ) provide an SVMax evaluation on small training batches , i.e. , b < d , and ( 3 ) evaluate the computational complexity of SVMax .

Summary: This paper proposes a regularization technique called SVMax (singular value maximization) that can mitigate model collapse and enable large learning rates to reduce training computation costs. The singular value decomposition of network activation is used to regularize the embedding space with unit circle embedding assumptions. In addition, a mathematical analysis of the mean singular value boundary is provided to reduce hyperparameter tuning. The authors evaluate the proposed method for the retrieval tasks and generative adversarial networks.