Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a regularization technique called SVMax (singular value maximization) that can mitigate model collapse and enable large learning rates to reduce training computation costs. The singular value decomposition of network activation is used to regularize the embedding space with unit circle embedding assumptions. In addition, a mathematical analysis of the mean singular value boundary is provided to reduce hyperparameter tuning. The authors evaluate the proposed method for the retrieval tasks and generative adversarial networks.

The idea looks promising in that it is simple and easy to use. However, the paper is a bit confusing and lacks clarity. I hope that the motivation for the introduction of the algorithm, analysis, and comparison experiments will be revealed well with care. A major revision of the work is needed.

Pros:
+ By introducing a simple method, it leads to good performance within the experimental work.
+ The flow of converting the former regularizer expression by mathematically analyzing the lower and upper bounds of the mean singular value is clear and well.
+ It is impressive that the corner cases of the proposed algorithm are written in the supplementary material. I agree that the corner case will not really happen.

Cons:
- Insufficient motivation. It seems not enough to simply apply singular value decomposition to the regularizer. In addition to the good results of the experiment, it is necessary to add more mathematical analysis, proof, etc. to see what this regularizer makes good.
- Is it best for model collapse mitigation or large learning rate activation to be seen only as experimental results? Hopefully, there is something in this part that can analyze why SVMax is possible.
- Why maximize mean singular value? Is maximizing the mean singular value the only way to spread out deep network embeddings? Can't maximize the min or max of singular values? It would be great if any comparison or analysis of this was supplemented.
- I'm confused about how the lower and upper bounds of loss functions are used and why they are in the paper.
- I would like to have various experimental results. In addition to the retrieval task, it would be nice if there was an experiment result from a classification task. Or, for example, it is better to experiment with local patch descriptors evaluated in your baseline algorithm paper.
review_title: Promising ideas, but insufficient motivation, analysis, and comparison

final_score: rating: 6: Marginally above acceptance threshold
review_content: Pros:
This paper is well written and easy to follow. It is also well organized from intuition to method and then to experiments. The background of the proposed method is interesting and useful. By regularizing the distribution of learned features is more likely to be benefit for robust embedding learning.The proposed method SVMax is novel to me. It has rigorous mathematical guarantees and analysis.Extensive experiments are conducted to demonstrate what the authors claimed. And corresponding analysis are also provided.

Cons:
It seems that the proposed method cannot consistently improve the performances over the baseline method, such as Angular-Loss et.al. What is the reason in fact? 
review_title: Borderline Accept

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

This paper proposes a new approach to regularize the feature embedding of neural networks. The proposed regularizer, maximizes the mean singular value of the feature matrix per batch, leading to a uniform spread of features. This enables learning with larger learning rates without the risk of model collapse. Authors derive lower and upper bounds for the proposed singular values loss, as well as popular ranking losses used in recent studies, eg. triplet loss and pairwise loss. These bounds help to tune the mixing parameter of network’s loss and the singular value loss.

Pros:

1- The paper considers a crucial problem of learning with neural networks, ie. model collapse. The approach is systematic and shows promising results for learning with large learning rates, while a model without regularization fails to perform well.

2- Authors derive lower and upper bounds for the proposed singular value loss, as well as well-known ranking losses in the literature. 


Cons or comments:

1- Even though there is a guide to choose parameter $\lambda$ using the bounds on losses, authors use a fixed value in the experimental setup. In case that this value is according to the bounds for the particular setting, it would be nice that authors mention it.

2- The proposed regularizer is particularly useful with a larger learning rate. Then, one expects that less number of epochs would be enough to converge to a good state of network. It seems that the number of epochs in the retrieval experiments is the same across methods. If SVMax with lr=0.01 and spread-out with lr=0.0001 use the same number of epochs, there will be not much gain using SVMax.

3- Model collapse may appear in different modes, eg. unrolled GAN paper discusses these modes for GANs. It would be nice if authors investigate how the model collapse happens. For example in Table 1, lr=0.01, contrastive loss, vanilla and spread-out show very poor performance. One can investigate the principal components of the batch embeddings.

4- Authors did not mention how they exactly incorporate the mean singular value into the loss. Is the exact value computed for each mini-batch? Or an estimation would be considered? How much computation overhead does the mean singular value loss add?

review_title: Good paper, novel idea, few missing points

final_score: rating: 5: Marginally below acceptance threshold
review_content: I have reviewed this paper in this year's Neurips. At that time, reviewers and AC have some good points but are not yet addressed (I have compared the two submissions carefully). 

This paper proposes a Feature Embedding Regularizer (named SVMax) to regularize feature embedding during the CNN learning process. The idea is to push the distribution of high-dim feature embeddings to be a uniform distribution across the embedding space. The idea is implemented by adding a regularize to maximize the averaged singular value computed from a mini-batch. Experiments show that many deep metric learning methods can benefit from the proposed SVMax regularizer.

Pros:

1) A simple but effective regularizer.
2) This paper is well-written and easy to follow.

Cons:
I re-emphasize some major points pointed by reviewers and AC:
1) [From other reviewers] The experimental results seem to be worse than SOTA and the baseline method seems to be not well trained. For example, on CUB datasets, the baseline Trip method can reach above 50% R@1, but your result is only 47.7%. The same phenomenon happens on other baseline methods and backbone network ResNet50 (To my best knowledge, ResNet50 baseline can achieve 60%+ R@1.) So the experimental results make me confused about the effectiveness of the proposed method.

2) Though the effectiveness of the proposed SVMax Regularizer is validated on the fine-grain image retrieval datasets. I would also want to see its performance on a broader image retrieval task. For example:

Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking

The reason is that for this broader image retrieval task, the original feature embeddings may have been well-distributed across the embedding space, rather than shrunk to a limit embedding space for the fine-grain datasets.

I think the above experiment needs to be done if we want to draw the conclusion that SVMax regularizer really works for deep metric learning methods.

review_title: A simple regularizer, but the performance is yet to be validated

Meta-review: The paper presents a new regularizer based on singular value decomposition in embedding space to avoid model collapse. The reviewes liked the simplicity of the idea, but there were some remaining concerns regarding the experiments. Moreover, two reviewers mentionned some concerns with respect to the clarity of the paper. While some concerns have been addressed by the rebuttal, in particular regarding the clarity of the paper, the concerns regarding the experiments remained, and the reviewers agreed that the paper needs a revision before publication. 

The main directions of improvement are to make the comparison with previous published results clearer, in particular comparing different methods with better hyperparameter tuning, and test on larger datasets. 
Decision: Reject