SVMax: A Feature Embedding Regularizer
Authors: Ahmed Taha
Alex Hanson
Abhinav Shrivastava
Larry S. Davis
Venue:  (2021)
Keywords: metric learning
model collapse
feature embedding
neural network regularizer

Abstract: A neural network regularizer (eg, weight decay) boosts performance by explicitly penalizing the complexity of a network. In this paper, we penalize inferior network activations -- feature embeddings -- which in turn regularize the network's weights implicitly. We propose singular value maximization (SVMax) to learn a uniform feature embedding. The SVMax regularizer integrates seamlessly with both supervised and unsupervised learning. During training, our formulation mitigates model collapse and enables larger learning rates. Thus, our formulation converges in fewer epochs, which reduces the training computational cost. We evaluate the SVMax regularizer using both retrieval and generative adversarial networks. We leverage a synthetic mixture of Gaussians dataset to evaluate SVMax in an unsupervised setting. For retrieval networks, SVMax achieves significant improvement margins across various ranking losses.

Under review as a conference paper at ICLR 2021
SVMAX: A FEATURE EMBEDDING REGULARIZER
Anonymous authors Paper under double-blind review
ABSTRACT
A neural network regularizer (e.g.. weight decay) boosts performance by explicitly penalizing the complexity of a network. In this paper, we penalize inferior network activations feature embeddings- which in turn regularize the network's weights implicitly. We propose singular value maximization (SVMax) to learn a uniform feature embedding. The SVMax regularizer integrates seamlessly with both su- pervised and unsupervised learning. During training, our formulation mitigates model collapse and enables larger learning rates. Thus, our formulation converges in fewer epochs, which reduces the training computational cost. We evaluate the SVMax regularizer using both retrieval and generative adversarial networks. We leverage synthetic mixture of Gaussians dataset to evaluate SVMax in an unsu- pervised setting. For retrieval networks, SVMax achieves significant improvement margins across various ranking losses.
INTRODUCTION
A neural network's knowledge embodied in both its weights and activations. This difference man- ifests in how network pruning and knowledge distillation tackle the model compression problem. While pruning literature Li al.] (2 Luo al. (2017); Yu al. (2018) compresses models by removing less significant weights, knowledge distillation Hinton et al. (2015) reduces computational complexity by matching a cumbersome network's last layer activations (logits). This perspective, of weight-knowledge versus siiivation-nowleede emphasizes how neural network literature is domi- nated by explicit weight regularizers. contrast, this paper leverages singular value decomposition (SVD) to regularize network through its last layer activations its feature embedding.
Our formulation inspired by principal component analysis (PCA). Given a set of points and their covariance, PCA yields the set of orthogonal eigenvectors sorted by their eigenvalues. The principal component (first eigenvector) is the axis with the highest variation (largest cigenvalue) as shown in Figure Tc The eigenvalues from PCA, and similarly the singular values from SVD, provide insights about the embedding space structure. As such, by regularizing the singular values, we reshape the feature embedding.
The main contribution of this paper is to leverage the singular value decomposition of a network's activations regularize the embedding space. We achieve this objective through singular value maximization (SVM.x). The SVMax regularizer is oblivious to both the input-class (labels) and the sampling strategy. Thus promotes uniform embedding space both supervised and unsupervised learning. Furthermore, we present a mathematical analysis of the mean singular value's 's lower and upper bounds. This analysis makes tuning the SVMax's balancing -hrerinampyprrrrrmmeteerr easier, when the feature embedding is normalized to the unit circle.
The SVMax regularizer promotes a uniform embedding space. During training, SVMax speeds up convergence by enabling large learning rates. The SVMax regularizer integrates seamlessly with various ranking losses. We apply the SVMax regularizer to the last feature embedding layer, but the same formulation be applicd to intermediate layers. The SVMax regularizer mitigates model collapse in both retrieval networks and generative adversarial networks (GANs) Goodfellow et al. (014)); (Srivastava et al. (2017); Metz al. (2017). Furthermore, the SVMax regularizer useful when training unsupervised feature embedding networks with a contrastive loss (e.g., Noroozi et al. (017); Oord et al. (2018) He et al. (2019) Tian et al. (2019).
In summary, we propose singular value maximization to regularize the feature embedding. In ad- dition, we present mathematical analysis of the mean singular value's lower and upper bounds

---

Under review as a conference paper at ICLR 2021
o
-1
0
(a) Singular values: (11.25, 1.14) (b) Singular values: (8.99, 6.86)
(c) PCA of a toy dataset
Figure 1: Feature embeddings scattered over the 2D unit circle. (ap. the features are polarized across a single axis; the singular value of the principal (horizontal) axis large while singular value of the secondary (vertical) axis small, respectively. In (b), the features are spread uniformly across both dimensions; both singular values are comparably large. (c) depicts the PCA analysis of toy 2D Gaussian dataset to demonstrate our intuition. The principal component (green) has the highest cigenvalue, i.e., the axis with the highest variation, while the second component (red) has smaller eigenvalue. Maximizing all eigenvalues promotes data dispersion across all dimensions. In this paper, we maximize the mean singular value regularize the feature embedding and avoid model collapse.
to reduce hyperparameter tuning (Sec. 3). We quantitatively evaluate how the SVMax regularizer significantly boosts the performance of ranking losses (Sec. 4.Ip. And we provide a qualitative evaluation of using SVMax in the unsupervised learning setting via GAN training (Sec. 4.2
2 RELATED WORK
Network weight regularizers dominate the deep learning regularizer literature, because they support a large spectrum of tasks and architectures. Singular value decomposition (SVD) has been applied as weight regularizer in several recent works Zhang et al. (2018) 8); [Sedghi et al. (2018); Guo & Ye 2219 Zhang et al. (208) employ SVD to avoid vanishing and exploding gradients recurrent neural networks. Similarly, Guo & Ye (2019) bound the singular values of the convolutional layer around to preserve the layer's s input and output norms. bounded output norm mitigates the exploding/vanishing gradient problem. Weight regularizers share the common limitation that they do not enforce an explicit feature embedding objective and are thus ineffective against model collapse.
Feature embedding regularizers have also been extensively studied, especially for classification net- works Rippel et al. (2015); Wen et al. (2016); He et al. (2018): Hoffman et al. (2019); Taha et al.) 2020 These regularizers aim maximize class margins, class compactness, ously. For instance, Wen et al. (2166) propose center loss to explicitly learn class representatives and thus promote class compactness. In classification tasks, test samples are assumed to lie within the same classes of the training set, i.e., closed-set identification. However, retrieval tasks, such as product re-identification, assume open-set setting. Because of this, retrieval network regularizer should aim to spread features across many dimensions to fully utilize the expressive power of the embedding space.
both simultane-
Recent literature Sablayrolles et al. 2018); Zhang et al. (011)) has recognized the importance of a spread-out feature embedding. However, this literature tailored to triplet loss and therefore assumes a particular sampling procedure. In this paper, we leverage SVD as regularizer because is simple, differentiable [onescu et al. (2015), and class oblivious. SVD has been used to promote low rank models to learn compact intermediate layer representations Kliegl al. (2017): Sanyal et al. (0199. This helps compress the network and speed up matrix multiplications on embedded devices (iPhone and Raspberry Pi). In contrast, regularize the embedding space through a high rank objective. By maximizing the mean singular value, we promote higher rank representation spread-out embedding.
2

---

Under review as a conference paper at ICLR 2021
3 SINGULAR VALUE MAXIMIZATION (SVMAX)
We first introduce our mathematical notation. Zet denote the image space and Ex Rºd denote the feature embeddings space, where is the dimension of the features. A feature embedding network is a function Fg. I > Ez, parameterized by the network's 's weights A. We quantify similarity between an image pair (I1, I2) via the Euclidean distance in feature space, i.e., Ez, EI, 1l2-
During training, a 2D matrix E Rxxd stores b samples' embeddings, where b the mini-bated size. Assuming 2 d, the singular value decomposition (SVD) of E provides the singular values = [s,,,, Sis sa). where 81 and 8y are the largest and smallest singular values, respectively. We maximize the mean singular value, Sy = Di=1 8,, to regularize the network's 's last layer activations the feature embedding. By maximizing the mean singular value, the deep network spreads out its embeddings. This has the added benefit of implicitly regularizing the network's 's weights 0. The proposed SVMax regularizer integrates with both supervised and unsupervised feature embedding networks as follows
NNN L, 1
8
Ly
Asp
where L, the original network loss and is a balancing hyperparameter.
Lower and Upper Bounds of the Mean Singular Value: One caveat to equation i the hyperpa- rameter It is difficult to tune because the mean singular value 8μ depends on the range of values inside E and its dimensions (b, d). Thus, changing the batch size embedding dimension requires a different X. To address this, we utilize a common assumption in metric learning the unit circle (L2-normalized) embedding assumption. This assumption provides both lower and upper bounds on ranking losses. This will allow us to impose lower and upper bounds on Sy.
For an L2-normalized embedding E, the largest singular value 81 is maximum when the matrix-rank of equals one, i.e., rank(E) = 1. and 8; = for E [2, d]. Horn & Johnson ( (191]) provide an upper bound on this largest singular value 81 as s (F) I /I lall B This holds in equality for all L2-normalized E E Raxd with rank (E) = 1. For an L2-normalized matrix E with [[]]] = b,
and ||] 1, this gives:
sE(E) = VI = Vb.
(2)
Thus, the lower bound L on sp is L (F) 4b 4
Similarly, upper bound defined on the sum of the singular values Turkmen & Civciv (2007);
Kong et al. (2018); FFiedland & Lim (2016). This summation is formally known as the nuclear norm of a matrix E 2. Hu (2015) established an upper bound on this summation using the Frobenius
Norm || Ff as follows
b x d mar(b, (b, d)
E S V
|
(3)
where ||]|| (E
Eig
Vb because of the L2-normalization assumption.
Accordingly, the lower and upper bounds of sy are [I, rewrite our final loss function as follows
8" (B)
d
With these bounds,
U s U - I
INN = Iap + xexp
(4)
The SVMax regularizer grows exponentially E [1, e]. We employ this loss function in all our re- trieval experiments. It is important note that the 1.2-normalized assumption makes À tuning easier, but is not required. Equation 4 makes the hyperparameter only dependent on the range of L, which is also bounded for ranking losses.
Lower and Upper Bounds of Ranking Losses: We briefly show that ranking losses are bounded when assuming 1.2-normalized embedding. Equations JanuatioossSnnd show triplet and contrastive losses,
3

---

Under review as a conference paper at ICLR 2021
Table 1: Quantitative evaluation on CUB-200-2011 with batch size 144, embedding dimension d 128 and multiple learning rates Ir = {0.01, 0.001, 0.0001). A Aru column indicates the R @ improvement margin relative the vanilla ranking loss. large learning rate increases the chance of model collapse, while small Ir slows convergence. À is dependent the ranking loss.
0.001
0.0001
Mothod
R@1
R@8
Apai
NMI
R@1
R@8
NMI
R@1
R@8
Ano
Contrastive
Vanilla
0.435
25.73
58.88
0.443
28.68
64.70
0.413
24.49
59.54
Spread-out
24.54
57.16
32.12
66.83
SVMax (Ours)
0.527
41.26
75.24
15.53
0.547
43.11
77.26
14.43
0,449
29.56
65.50
5.06
Triplet Loss
Vanilla
29.34
67.96
Spread-out
0.545
43.60
76.98
14.26
0.557
44.02
78.54
15.14
0.435
28.33
64.33
3.46
SVMax (Ours)
0.556
43.21
77.43
13.88
0.527
39.
74.17
10.25
0.401
25.07
60.01
0.20
SVMax À 0.1 (Ours)
0.547
43.80
77.97
14.17
0.557
43.89
78.11
15.01
0.436
28. 22
64.40
3.36
N-pair
Vanilla
50.32
27.
Spread-out
0.416
20.64
52.80
1.69
0,483
32 46
66.41
4.81
0.474
33 39
68.80
1.98
SVMax (Ours)
0.483
34.62
68.11
15.67
0.547
43.79
77.31
16.14
0.488
34.13
69.92
2.72
Angular
Vanilla Spread
0.470
28.51
60.03
0,508
38 9-1
72.88
0.538
11.
76.18
28.29
-0.25
72.86
76.23
SVMax (Ours)
0.487
32.88
66.27
4.31
0.523
41.29
74.71
2.35
0.531
42.00
76.30
0.20
respectively, and their corresponding bounds [L, U}.
[1.0]]
[(D(La), [x]) (aspn)))==((((([]]])) CL( = (1 )2 D(1=), (v))) ))))) + 5x., [m - D([=], lyD)]
Th(ap,n)er
D(La], [n]) + m)]+
[0, + m],
(5)
[L,U]
[0,2]]
(6)
where [+] = max(0,))) (0.,). +), m < is the margin between classes, since 2 is the maximum distance the unit circle. [+] and D2,) are the embedding and Fuclidean distance functions, respectively. In equation 5 a, p, and n are the anchor, positive, and negative images in a single triplet (a, n) from the triplets
T. In equation 6 and y form single pair of images from the pairs set P. 5x.y = when x and y belong to different classes; zero otherwise. In the supplementary material, (1) show similar analysis for N-pair and angular losses, (2) provide an SVMax evaluation on small training batches, i.e., b < d, and (3) evaluate the computational complexity of SVMax.
4 EXPERIMENTS
In this section, we evaluate SVMax using both supervised and unsupervised learning. We leverage retrieval and generative adversarial networks for quantitative and qualitative evaluations.
4.1 RETRIEVAL NETWORKS
We evaluate the SVMaa regularizer quantitatively using three datasets: CUB- 200-2011 Wah et al. (201T). Stanford CARS196 Krause al. (2013), and Stanford Online Prod- ucts Oh Song et al. (016)). We use GoogLeNet Szegedy et al. (2015) and ResNet50 He et al.) (2016); both pretrained on ImageNet Deng et al. (2009) and fine-tuned for K iterations. These are standard retrieval datasets and architectures. By default, the embedding E Rd=128 is normalized to the unit circle. In all experiments, a batch size
Technical Details:
144 is employed, the learning rate is fixed for K/2 iterations then decayed polynomially to le - 7 at iteration K. We use the SGD optimizer with 0.9 momentum. Each batch contains different classes and I different samples per class. For example, triplet loss employs
24 different classes and / = 6 instances per class. The mini-batch of N-pair loss contains 72 classes and a single positive pair per class, i.e.p = 72 and = 2. This same mini-batch setting is used for angular loss. For contrastive loss, p 36 and 4 are divided into 72 positive and 72 negative pairs. For CUB-200 and CARS196. K = 5, 000 iterations; for Stanford Online Products, K 20, 000.
4

---

Under review as a conference paper at ICLR 2021
Table 2: Quantitative evaluation on Stanford Online Products.
Ir
0.01
0.001 Ir
Ir = 0.0001
Method
NMI
R@1
R@8
Age
NMI
R
R@8
A Agai
NMT
R@1
R@8
ARai
Contrastive
Vanilla
0.816
18.23
34.07
0.820
25.70
43.27
0.813
34.30
48.49
Spread-out
0.811
18.87
35.74
0.64
0.822
29.97
46.69
1.27
0.824
36.15
51.22
1.85
SVMax (Ours
0.875
78.90
43.59
53.94
70.92
0.832
41.96
57.44
7.66
Triplel Loss
Vanilla
0.891
71.96
86.24
0.873
64.09
80.07
0.840
46.29
62.57
Spread-out
0.890
71.60
85.73
-0.36
0.872
64.23
80.10
0.840
46.68
63.04
0.39
SVMax A (Ours)
0.868
63.82
80.95
-8.15
0.857
58.04
75.14
-6.04
0.836
44.62
60.76
-1.67
SVMax (Ours)
0.889
71.48
85.97
-0.49
0.872
64.23
80.14
0.840
62.95
0.35
N puir
Vanilla
0.798
12.86
24.53
0.815
23.83
38.97
0.818
33.98
48.56
Spread-out
0.803
16.58
3.72
32.88
50.31
0.825
52.55
3.40
SVMax (Ouns)
0.871
57.76
76.05
44.90
0.858
54.70
71.S7
30.87
0.835
43.04
58.78
9.06
Angular
Vanilla
0.883
62.83
80.13
0.885
66.93
82.
0.856
54.29
71.14
Spread-out
0.883
62.73
79.96
-0.10
0.885
66.91
82. 09
-0.02
0.856
54.30
71.10
0.02
SMMax (Ours)
27
71.47
0.59
Baselines: We evaluate the SVMax regularizer using contrastive Hadsell et al. (2006), hard triplet Hoffer & Ailon (2015): Hermans al. (2017), N-pair (Sohn (2016) and angular Wang et al. 2017 losses. We use the margin m = 1 for contrastive loss, m7 0.2 for triplet loss, and the angle bound
45 for angular loss. Similar SVMax, multiple regularizers Kumar et al. (2016); Zhang et al. (2017);; (Snyyal et al.](220 (2019): Chen & Deng (2099) promote a uniform embedding space. Unlike SVMax, these regularizers require a supervised setting to push anchor- negatvve pairs apart. We employ the spread-out regularizer ZZang et al. (2017) baseline for its simplicity, with default hyperparameter a To enable the spread-out regularizer on non-triplet ranking losses, we pair every anchor with random negative sample from the training mini-batch.
Evaluation Metrics: For quantitative evaluation, we use the Recall metric and Normalized Mutual Info (NMI) on the test split.
The hyperparameter: = for both contrastive and N-pair losses, = 0.1 for triplet loss, and A for angular loss. We fix À across datasets, architectures, and other hyperparameters (b, d).
Results: Tables D and present quantitative retrieval evaluation on CUB-200 and Stanford Online Products datasets both using GoogLeNet. These tables provide in depth analysis and emphasize our improvement margins on a small and large dataset. Figure 2 provides quantitative evaluation on Stanford CARS196. We report the qualitative retrieval evaluation and quantitative evaluation on ResNet50 in the supplementary material. Our training hyperparameters. - learning rate Ir and number of iterations K do not favor particular ranking loss.
We evaluate SVMax on various learning rates. A large learning rate, e.g., Ir = 0.01, speeds up convergence, but increases the chance of model collapse. In contrast, a small rate, e.g., Ir 0.0001, is likely to avoid model collapse but is slow to converge. This undesirable effect is tolerable for small datasets where increasing the number of training iterations K does drastically increase the overall training time but it is infeasible for large datasets. For contrastive and N N-pair losses, SVMax is significantly superior both the vanilla and spread-out baselines, especially with large learning rate. A small Ir slows convergence and all approaches become equivalent. The spread-out regularizer /Zhan et al., (2017) and its hyperparameters are tuned for triplet loss. Thus, for this particular ranking loss, the SVMax and spread-out regularizers are par.
In our experiments, we employ a large learning rate because it is the simplest factor to introduce model collapse. However, the learning rate is not the only factor. Another factor is the training dataset size and its intra-class variations. small dataset with large intra-class variations increases the chances of model collapse. For example, pair of dissimilar birds from the same class jus- tifies a model collapse when coupled with a large learning rate. The hard triplet loss experiments emphasize this point because every anchor is paired with the hardest positive and negative samples.
5

---

Under review as a conference paper at ICLR 2021
Angular
N N-pair
Triplet
Contrastive
50 -
60
R@1
50
40
40
10
40
30
20
20
30
0.01
0.001
0.0001
Vanilla
Spread-our
SVMax
Figure 2: Quantitative evaluation on Stanford CARS196. X and Y-axis denote the learning rate Ir and recall @ performance, respectively.
On small fine-grained datasets like CUB-200 CARS196, the vanilla hard triplet loss suffers sig- nificantly. Yet, the same implementation is superior on a big dataset like Stanford Online Products. By carefully tuning the training hyperparameter on CUB-200, is possible avoid a degenerate solution. However, this tedious tuning process is unnecessary when using either the spread-out or the SVMax regularizer.
The vanilla N- N-pair loss underperforms because it does not support feature embedding on the unit circle. Both spread-out and SVMax mitigate this limitation. For angular loss, bigger = is employed to cope with the angular loss range. SVMax is a class oblivious regularizer. Thus, À should be significant enough to contribute the loss function without dominating the ranking loss.
Wu et al. (017)) show that the distance between any anchor- pair, which is randomly sam- pled from an n--iimnsiionl unit sphere, follows the normal distribution ((v2, ). This mean distance /2 is large relative the triplet loss margin
0.2, but comparable the contrastive loss margin m = 1. Accordingly, triplet loss converges to zero after a few iterations, because most triplets satisfy the margin m 0.2 constraint. When triplet loss equals zero, the SVMax regularizer with À = becomes the dominant term. However, the SVMax regularizer should not dominate because oblivious to data annotations; it equally pushes anchor- cichorrpositive and anchor-negatioe pairs apart. Reducing À to 0.1 solves this problem.
less aggressive triplet loss Schroff et al. (2015); Xuan al. (2020) is another way avoid model collapse. For instance, [Schroff et al. 2015) have proposed a triplet loss variant that employs semi- hard negatives. The semi-hard triplet loss is more stable than the aggressive hard triplet and lifted structured losses Song et al. (2016). Unfortunately, the semi-hard triplet loss assumes very large mini-batch (b = 1, 800 in Schroff et al. (2015),, which is impractical. Furthermore, when model collapse is avoided, aggressive triplet loss variants achieve superior performance Hermmnn et al. 2017). In contrast, the SVMax regularizer only requires a larger mini-batch than the embedding dimension, i.e., b 2 d, natural constraint for retrieval networks which favor compact embedding dimensions. Additionally, SVMax does not make any assumption about the sampling procedure. Thus, unlike (abllarolless et al. (2018); Zhang et al. (2017), SVMax supports various supervised ranking losses.
4.2 GENERATIVE ADVERSARIAL NETWORKS
Model collapse one of the main challenges of training generative adversarial networks (GANs) Metz et al. (2017): Srivastava et al. (2017): Mao et al. (209);; Salimans et al. (2016). To tackle this challenge, Metz et al. (2017) propose an unrolled GAN to prevent the generator from overfitting to the discriminator. In an unrN the generator observes the discriminator for / steps before updating the generator's parameters using the gradient from the final step. Alternatively, we leverage the simpler SVMax regularizer to avoid model collapse. We evaluate
regularizer using a simple GAN on a 2D mixture of 8 Gaussians arranged circle. This 2D baseline Metz et al. 2017); Srivastava et al. 2017) Bang & Shim (2018) provides simple qualitative evaluation and demonstrates SVMax's 's potential in unsupervised learning. We leverage this simple baseline because assume b 2 d, which does not hold for images.

---

Under review as a conference paper at ICLR 2021
Method
Step
Step 5k
Step 10k
Step 15k
Step 20k
Step 25k
Target
Vanilla GAN
Vanilla GAN+ SVMax
RD
:0
0
0
0
0
Unrolled GAN (5 steps)
Unrolled (5 steps)
SVMax
Figure 3: The SVMax regularizer mitigates model collapse in GAN trained on a toy 2D mixture of Gaussians dataset. Columns show heatmaps of the generator distributions at different training steps (iterations). The final column shows the groundtruth distribution. The first row shows the distributions generated by training vanilla GAN suffering model collapse. The second row shows the generated distribution when penalizing the generator's fake embedding with the SVMax regularizer. The third and fourth rows show two distributions generated using an unrolled-GAN with and without the SVMax regularizer, respectively. This high resolution figure is best viewed on screen with zoom capabilitiis.
Figure shows the dynamics of the GAN generator through time. We use a public PyTorch imple- mentation of Metz et al. (201)). We made a single modification to the code to use a relatively large learning rate, i.e., Ir = 0.025 for both the generator and discriminator. This single modification is simple and fast way to induce model collapse. The mixture of Gaussians circle has radius 2, i.e., the generated fake embedding neither L2- 1.2-nommaized nor strictly bounded by net- work layer. We kept the radius parameter unchanged emphasize that neither I.2-normalization nor strict-bounds are required. To mitigate the impact of lurking variables (e-g., random network initial- ization and mini-batch sampling),
fix the random generator's 's seed for all experiments. We apply SVMax to a vanilla and an unrolled GAN for five steps. We apply the vanilla SVMax regularizer (Eq. i.e., LnN LeAN Asp, where 0.01 and Sp is mean singular value of the generator fake embedding.
GANs typically used to generate high resolution images. This high-resolution output is the main limitation of the SVMax regularizer. The current formulation assumes the batch size is bigger than the embedding dimension, i.e., 2 d. This constraint is trivial for the Gaussians mixture 2D dataset and retrieval networks with a compact embedding dimensionality (e.g., d = (128, 256}). However, this constraint hinders high resolution image generators because the mini-batch size constraint be- comes 6 > W x H x C, where W, Ht and C' are the generated image's width, height, and number of channels, respectively. Nevertheless, this GAN experiment emphasizes the potential of the SVMax regularizer in unsupervised learning.
4.3 ABLATION STUDY
In this section, we evaluate two hypotheses: (I) the SVMax regularizer boosts retrieval performance because it learns uniform feature embedding, (2) the same SVMax hyperparameter À supports different embedding dimensions and batch sizes the main objective of the mean singular value's bounds analysis.
To evaluate the SVMax regularizer's impact on feature embeddings, we embed the MNIST dataset onto the 2D unit circle. In this experiment, we used a tiny CNN (one convolutional layer and one hidden layer). Figure 4 shows the embedding space after training for epochs. When using the SVMax regularizer, the feature embeddings spread out more uniformly and rapidly than the vanilla contrastive loss.
hhtp:///gtthhh.ccm/madeewiiaaa wtt.://aaaaoiniinnnoeoooao l/unrolled-gans

---

Under review as a conference paper at ICLR 2021
Method
Epoch
Epoch
Epoch
Epoch
Epoch 16
Epoch 32
Epoch 64
Contrastive
Contrastive SVMax
Figure 4: Qualitative feature embedding evaluation using the MNIST dataset projected onto the 2D unit circle. The first row shows the feature embedding learned using vanilla contrastive loss and the second row applies the SVMax-regularized. A random subset of the test split is projected for visualization purpose. Different colors denote different classes. The regularized feature embed- ding spreads out uniformly and rapidly. The supplementary material shows the feature embedding evolves vividly up to 200 epochs. This high resolution figure best seen on a screen.
Triples plt ba288, d=256
Triplet be288, da64
Triplet b=?2, d=64
40
40
R@1
30
0.0001
0.001
0.01
0.0001
0.001
0.01
0.0001
0.001
0.01
Contraetive b-286, d=25 wentttttttttti b-288, d-64 Contrastive b-72, d-56
50
40
40
30
30
0.0001
0.001
0.01
0.0001
0.001
0.01
0.0001
0.001
0.01
Vanilla
Spread-out
SVMax
Figure 5: Quantitative evaluation on CUB-200-2011 with various batch sizes b {288, 72) and embedding dimensions d = {256, 64) to demonstrate the stability of our hyperparameter. À = 1 for contrastive loss and 1 for triplet loss.
The mean singular value bound analysis makes tuning the hyperparameter À easier. This hyperpa- rameter becomes only dependent on the ranking loss and independent of both the batch size and the embedding dimension. Figure presents a quantitative evaluation using the CUB-200 dataset. We explore various batch sizes MobileNetV2[Sannde et al. (2018) to the big batch 288 on 24GB GPU. The supplementary
288, 72} and embedding dimensions d = {256. 64). We employ
material contains a similar evaluation on the Stanford Online Products and CARS196 datasets.
5 CONCLUSION
We have proposed singular value maximization (SVMax) feature embedding regularizer. SV- Max promotes a uniform embedding, mitigates model collapse, and enables large learning rates. Unlike other embedding regularizers, the SVMax regularizer supports large spectrum of rank- ing losses. Moreover,
oblivious to data annotation and, as such, supports both supervised and unsupervised learning. Qualitative evaluation using generative adversarial network demonstrates SVMax's 's potential in unsupervised learning. Quantitative retrieval evaluation highlight significant performance improvements due the SVMax regularizer.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, arXiv preprint 1rXiv:1610.01644, 2016.
Duhyeon Bang and Hyunjung Shim. Mggan: Solving mode collapse using manifold guided training. arXiv preprini arviv: 1804.04391, 2018.
Binghui Chen and Weihong Deng. Energy confused adversarial metric learning for zero-shot image retrieval and clustering. AAAI, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Shmuel Friedland and Lek-Heng Lim. The computational complexity of duality. SIAM Journal on Optimization, 26(4):2378-239) 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Peichang Guo and Qiang Ye. On regularization for convolutional kernel in neural networks. arXiv preprint arXiv: (ttp::996..04800 2019.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In CVPR, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv: 2019.
Xinwei He, Yang Zhou, Zhichao Zhou, Song Bai, and Xiang Bai. Triplet-center loss for multi-view 3d object retrieval. arXiv preprint arXiv:/803.0m 0888000088,,. 2018.
Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person identification. arXiv preprint arXiv: /tt:::/7000777,,, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint /5033v 02550.3..53,,,. 2015.
Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In International Workshop on Similarity-Basee Pattern Recognition, 2015.
Judy Hoffman, Daniel A Roberts, and Sho Yaida. Robust learning with jacobian regularization. arXiv preprint arXiv: 1908.01900.02200 2019.
Roger A Hor and Charles R Johnson. Topics in matrix analysis cambridge university press. Cam- bridge, UK, 1991.
Shenglong Hu. Relations of the nuclear norm tensor and its matrix flattenings. Linear Algebra and its Applications, 478: 188-199, 2015.
Catalin Ionescu, Orestis Vantzos, and Cristian Sminchisescu. Training deep networks with structured layers by matrix backpropagation. arXiv preprint Xiv.:1509.0081. 2015.
Markus Kliegl, Siddharth Goyal, Kexin Zhao, Kavya Srinet, and Mohammad Shoeybi. Trace norm regularization and faster inference for embedded speech recognition mnss. arXiv preprint 1rXiv:1770..09026. 2017.
Xu Kong, Jicheng Li, and Xiaolong Wang. New estimations on the upper bounds for the nuclear norm of tensor. Journal of inequalities and applications, 2018(1):282, 2018
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, 2013.

---

Under review as a conference paper at ICLR 2021
BG Kumar, Gustavo Carneiro, lan Reid, et al. Learning local image descriptors with deep siamese and triplet convolutional networks by minimising global loss functions. In Proceedings of the IEEE Conference Computer Vision and Pattemn Recognition, pp. 5385-5394, 2016.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint urXiv:1608.08710 2016.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In ICCV, 2017.
Qi Mao, Hsin- Hsin-Ying Lee, Hung-Yo Yu Tseng, Siwei Ma, and Ming-Hsuan Yang. Mode seeking genera- tive adversarial networks for diverse image synthesis. In CVPR, 2019.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. 2017.
Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learning count. In ICCV, 2017.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In CVPR, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- 2018.
tive coding, arXiv preprint 1rXiv:1807.08::
Oren Rippel, Manohar Paluri, Piotr Dollar, and Lubomir Bourdev. Metric learning with adaptive density discrimination. arXiv preprint arXiv: 1.009999 2015.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Spreading vectors for similarity search. arXiv preprint arXiv:18006 03198, 2018.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NIPS, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Amartya Sanyal, Varun Kanade, and Philip H. S. Torr Learning low-rank representations. arXiv preprint 180i 07090, 2019.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. CVPR, 2015.
Hanie Sedghi, Vineet Gupta, and Philip M Long. The singular values of convolutional layers. arXiv preprint arXiv: 10408, 2018.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NIPS, 2016.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In NSIP. 2017.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Ahmed Taha, Yi-Ting Chen, Teruhisa Misu, Abhinav Shrivastava, and Larry Davis. Boosting stan- dard classification architectures through ranking regularizer. In WACV, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.00) 1906.05899 2019.
Ramazan Turkmen and Haci Civciv. Some bounds for the singular values of matrices. Applied Mathematical Sciences, (49):2443-244)) 2007.
10

---

Under review as a conference paper at ICLR 2021
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, and Yuanqing Lin. Deep metric learning with angular loss. In ICCV, 2017.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In ECCV, 2016.
Chao-Yuan Wu. R Manmatha, Alexander J Smola, and Philipp Krahenbuhl. Sampling matters in deep embedding learning. ICCV, 2017.
Hong Xuan, Abby Stylianou, and Robert Pless. Improved embeddings with easy positive triplet mining. In WACV, 2020.
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In CVPR, 2018.
Jiong Zhang, Qi Lei, and Inderjit S Dhillon. Stabilizing gradients for deep neural networks via efficient svd parameterization. arXiv preprint arXiv:1803.09327, 2018.
Xu Zhang, Felix X Yu, Sanjiv Kumar, and Shih-Fu Chang. Learning spread-out local feature de- scriptors, In ICCV, 2017.
Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, Xi Peng, and Ahmed Elgammal. A generative adversarial approach for zhro-shot learning from noisy texts. In CVPR, 2018


Reviews:
[{"review_id": "vEQugCSV-Wv", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I have reviewed this paper in this year's Neurips. At that time, reviewers and AC have some good points but are not yet addressed (I have compared the two submissions carefully). \n\nThis paper proposes a Feature Embedding Regularizer (named SVMax) to regularize feature embedding during the CNN learning process. The idea is to push the distribution of high-dim feature embeddings to be a uniform distribution across the embedding space. The idea is implemented by adding a regularize to maximize the averaged singular value computed from a mini-batch. Experiments show that many deep metric learning methods can benefit from the proposed SVMax regularizer.\n\nPros:\n\n1) A simple but effective regularizer.\n2) This paper is well-written and easy to follow.\n\nCons:\nI re-emphasize some major points pointed by reviewers and AC:\n1) [From other reviewers] The experimental results seem to be worse than SOTA and the baseline method seems to be not well trained. For example, on CUB datasets, the baseline Trip method can reach above 50% R@1, but your result is only 47.7%. The same phenomenon happens on other baseline methods and backbone network ResNet50 (To my best knowledge, ResNet50 baseline can achieve 60%+ R@1.) So the experimental results make me confused about the effectiveness of the proposed method.\n\n2) Though the effectiveness of the proposed SVMax Regularizer is validated on the fine-grain image retrieval datasets. I would also want to see its performance on a broader image retrieval task. For example:\n\nRevisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking\n\nThe reason is that for this broader image retrieval task, the original feature embeddings may have been well-distributed across the embedding space, rather than shrunk to a limit embedding space for the fine-grain datasets.\n\nI think the above experiment needs to be done if we want to draw the conclusion that SVMax regularizer really works for deep metric learning methods.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ln0Rwo7GaEZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a new approach to regularize the feature embedding of neural networks. The proposed regularizer, maximizes the mean singular value of the feature matrix per batch, leading to a uniform spread of features. This enables learning with larger learning rates without the risk of model collapse. Authors derive lower and upper bounds for the proposed singular values loss, as well as popular ranking losses used in recent studies, eg. triplet loss and pairwise loss. These bounds help to tune the mixing parameter of network’s loss and the singular value loss.\n\nPros:\n\n1- The paper considers a crucial problem of learning with neural networks, ie. model collapse. The approach is systematic and shows promising results for learning with large learning rates, while a model without regularization fails to perform well.\n\n2- Authors derive lower and upper bounds for the proposed singular value loss, as well as well-known ranking losses in the literature. \n\n\nCons or comments:\n\n1- Even though there is a guide to choose parameter $\\lambda$ using the bounds on losses, authors use a fixed value in the experimental setup. In case that this value is according to the bounds for the particular setting, it would be nice that authors mention it.\n\n2- The proposed regularizer is particularly useful with a larger learning rate. Then, one expects that less number of epochs would be enough to converge to a good state of network. It seems that the number of epochs in the retrieval experiments is the same across methods. If SVMax with lr=0.01 and spread-out with lr=0.0001 use the same number of epochs, there will be not much gain using SVMax.\n\n3- Model collapse may appear in different modes, eg. unrolled GAN paper discusses these modes for GANs. It would be nice if authors investigate how the model collapse happens. For example in Table 1, lr=0.01, contrastive loss, vanilla and spread-out show very poor performance. One can investigate the principal components of the batch embeddings.\n\n4- Authors did not mention how they exactly incorporate the mean singular value into the loss. Is the exact value computed for each mini-batch? Or an estimation would be considered? How much computation overhead does the mean singular value loss add?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5qTnzBAW32", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\nThis paper is well written and easy to follow. It is also well organized from intuition to method and then to experiments. The background of the proposed method is interesting and useful. By regularizing the distribution of learned features is more likely to be benefit for robust embedding learning.The proposed method SVMax is novel to me. It has rigorous mathematical guarantees and analysis.Extensive experiments are conducted to demonstrate what the authors claimed. And corresponding analysis are also provided.\n\nCons:\nIt seems that the proposed method cannot consistently improve the performances over the baseline method, such as Angular-Loss et.al. What is the reason in fact? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "2mY7yHv9Po1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a regularization technique called SVMax (singular value maximization) that can mitigate model collapse and enable large learning rates to reduce training computation costs. The singular value decomposition of network activation is used to regularize the embedding space with unit circle embedding assumptions. In addition, a mathematical analysis of the mean singular value boundary is provided to reduce hyperparameter tuning. The authors evaluate the proposed method for the retrieval tasks and generative adversarial networks.\n\nThe idea looks promising in that it is simple and easy to use. However, the paper is a bit confusing and lacks clarity. I hope that the motivation for the introduction of the algorithm, analysis, and comparison experiments will be revealed well with care. A major revision of the work is needed.\n\nPros:\n+ By introducing a simple method, it leads to good performance within the experimental work.\n+ The flow of converting the former regularizer expression by mathematically analyzing the lower and upper bounds of the mean singular value is clear and well.\n+ It is impressive that the corner cases of the proposed algorithm are written in the supplementary material. I agree that the corner case will not really happen.\n\nCons:\n- Insufficient motivation. It seems not enough to simply apply singular value decomposition to the regularizer. In addition to the good results of the experiment, it is necessary to add more mathematical analysis, proof, etc. to see what this regularizer makes good.\n- Is it best for model collapse mitigation or large learning rate activation to be seen only as experimental results? Hopefully, there is something in this part that can analyze why SVMax is possible.\n- Why maximize mean singular value? Is maximizing the mean singular value the only way to spread out deep network embeddings? Can't maximize the min or max of singular values? It would be great if any comparison or analysis of this was supplemented.\n- I'm confused about how the lower and upper bounds of loss functions are used and why they are in the paper.\n- I would like to have various experimental results. In addition to the retrieval task, it would be nice if there was an experiment result from a classification task. Or, for example, it is better to experiment with local patch descriptors evaluated in your baseline algorithm paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new regularizer based on singular value decomposition in embedding space to avoid model collapse. The reviewes liked the simplicity of the idea, but there were some remaining concerns regarding the experiments. Moreover, two reviewers mentionned some concerns with respect to the clarity of the paper. While some concerns have been addressed by the rebuttal, in particular regarding the clarity of the paper, the concerns regarding the experiments remained, and the reviewers agreed that the paper needs a revision before publication. 

The main directions of improvement are to make the comparison with previous published results clearer, in particular comparing different methods with better hyperparameter tuning, and test on larger datasets. 
Decision: Reject