Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors provided elegant similarities between contrastive learning and deep metric learning. They decomposed contrastive loss objectives into two quantities evaluating the geometry of the learned representation space:
1. Alignment, and,
2. Uniformity. Where Alignment is the closeness that defines how close two positive pairs are on the embedding space. While the uniformity ensures the scattered. The authors found the representation learned by optimizing contrastive objective indeed has these two properties compared to representation learned using supervised objective. They also showed that both alignment and uniformity are required for learning better examples for adversarial example tasks.
main_review: Strengths
--------------------------------------------
1. Fairly well written, though a few sections, such as Sec 2 and 4 can be improved
2. Theoretical proves are given however they are largely motivate by [1]

Weakness
-------------------------------------------
1. Why the unit hypersphere is a nice feature space is not very well addressed in this paper. Is there any link between connected
sets with smooth boundaries are nearly linearly separable in the hyperspherical geometry and such linear separability is used here to disentangle relation?
2. "... We derive the theoretical analysis for the triplet loss to prove that the triplet loss..." Page 1, Para 2. IMHO this fact is already well studied and theoretically proved in [1]. A citation to [1] is missing on Page 1, Para 2. 
3. Ablation missing? For example, a study considering only positive, only contrastive, etc., is missing?
4. Negative samples are hard negative or soft negatives? How are we getting negatives is not very clear.

Ref:
-----------------------------------------------------
1. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere, ICML 2020
summary_of_the_review: The paper shows a closeness between contrastive learning and deep metric learning. They decomposed contrastive loss objectives into two quantities evaluating the geometry of the learned representation space:
1. Alignment, and,
2. Uniformity. However, the paper misses producing few ablations such as considering only positive, only contrastive, etc. How we mine negative samples is not very clear. Overall the paper seems like an extension of [1].


Ref:
-----------------------------------------------------
1. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere, ICML 2020
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper first analyzes the tuple-based metric losses on the hyperspherical space and studies two objective functions: intra-class alignment and hyperspherical uniformity. Then based on these two objective functions, the paper develops two ways of generating adversarial examples for improving deep metric learning. The empirical results show some advantages of the proposed method.
main_review: I like the general idea of the paper and the hyperspherical perspective on deep metric learning. It has been a common wisdom that deep metric learning should be performed on a hyperspherical space, but an rigorous and principled understanding is yet to be found. I believe this is an interesting and extremely important direction. In such sense, this paper is tackling an significant problem.

The proposed algorithm to improve deep metric learning method is rather straightforward. Instead of generating the adversarial examples using the standard metric learning loss (e.g. triplet loss), the paper takes another approach by attacking either intra-class alignment loss or hyperspherical uniformity loss. I think the intuition behind makes senses, and the method can be viewed as attacking the deep metric learning objective from a dual space (which is equivalent to tuple-based loss space under some regularity).

The empirical results seem good to me. The proposed ADML by attacking intra-class alignment loss / hyperspherical uniformity loss shows non-trivial gain compared to the other existing methods.

My concerns and suggestions are mostly in the following aspects:

(1) I understand that the decompostion to intra-class alignment and hyperspherical uniformity is from tuple-based losses, but will the generated adversarial examples also help contrastive loss / triplet loss? It will be interesting to see whether these adversarial examples are generally useful for deep metric learning methods.

(2) Since attacking either intra-class alignment loss or hyperspherical uniformity loss yields a consistent performance gain, what if attack both objective at the same time? It will be more interesting to formulate the attack to both intra-class alignment loss and hyperspherical uniformity loss in a soft manner. That is to say, you have a parameter to control the balance between attacking two losses. Although you end up with an additioanl hyperparameter, it will make the method more flexible and I will be curious to see how the performance varies with this hyperparameter.

(3) It will strengthen the experiments if there could be comparison among different adversarial attack methods. If every adversarial attack method works well for improving tuple-based losses, it will make the current conclusion more general.

(4) The presentation could be significantly improved and I think the current paper structure may be suboptimal. I think some of the analyses on decomposing tuple-based loss to intra-class aligment and hyperspherical uniformity can be put to appendix. Only the main result is put in the main paper. Then the authors can be sufficient space to comprehensively evaluate how and why these generated adversarial examples help deep metric learning.

(5) It will be helpful to visualize these adversarial examples on a 2-dim embedding space. Specifically, you can train the same model by setting the output feature dimension as 2. Then these features can be naturally visualized without any other visualization tools (such as T-SNE). Of course, using T-SNE may also be fine and interesting to see where these examples lie geometrically.

I am sitting between weak accept and accept. If my concerns are properly addressed, I will be happy to increase my score.
summary_of_the_review: This paper studies an interesting problem. Although the proposed method seems naive and incremental, its effectiveness and the significance of the topic really make up for that. Overall, I vote for weak accept, but may increase my rating based on the rebuttal.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper connects metric losses with the intra-class alignment and hyperspherical uniformity from theoretical analysis and proposes to boost metric learning augmented with adversarial samples generated by attacking alignment or uniformity objective. Experiments compare the proposed method with various baselines and the proposed method achieves the best performance.
main_review: 
The paper is well written and easy to follow. Though the proposed idea is interesting and the theoretical results are solid, the claimed contributions are marginal given the related works [1,2]. To be specific,

1. I think the theoretical analysis that analyzes the intra-class alignment and hyperspherical uniformity for the tuple-based metric is a simple extension of [1] given the similarity between contrastive learning and deep metric learning (DML). In particular, the proof of main theoretical results in Section 3.2 in this paper inherits from the proof of Theorem 1 in [1] Appendix. Therefore, the theoretical contribution is marginal.

2. The idea of using adversarial samples for DML has been well explored by [2], though with a different attacking objective (triplet loss). As proved in this paper that the triplet loss also optimizes the intra-class alignment and hyperspherical uniformity goals, so the proposed method has a marginal contribution.

I have some other questions:
1. Proposition 1 is unclearly motivated. I cannot connect it with other analysis.

2. The claim that Page 3 “because the embedding space of DML is a unit hypersphere, where the maximum distance between two points is 2, it’s not possible to separate all negative embeddings with a large margin. Actually on $S^{k−1}$, the number of points with pairwise distance larger or equal than $\sqrt{2}$ is at most k and the embedding dimension k is always smaller than the number of feature vectors, thus it’s impossible to make all distances between negative pairs exceed $\sqrt{2}$” seems problematic. Assume k=2, then the number of points with a pairwise distance larger or equal than $\sqrt{2}$ is supposed to be 4, imaging four points of a square in the circle. (If I am wrong, please correct me.)

3. Why not consider a method ADML+A+U that includes two kinds of adversarial samples simultaneously?

4. What is the lambda for experiment section 5.5?

5. When increasing lambda, will the robustness of the model be better?

6. In equation (7), $L_{uniformity}$ should connect to equation (2) (3).
 


Reference 

[1] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pp.9929–9939. PMLR, 2020

[2] Yueqi Duan, Wenzhao Zheng, Xudong Lin, Jiwen Lu, and Jie Zhou. Deep adversarial metric learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2780–2789, 2018.

summary_of_the_review: The paper has marginal contributions and limited novelty.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work introduces novel adversarial deep metric learning algorithm based on the fact that contrastive losses promote intra-class alignment and uniformity on the unit sphere. The authors provide a theoretical proof that this is also true for the triplet loss under mild assumptions. 
The adversarial examples used to regularize the training are generated by targeting the alignment and uniformity terms separately. This regularization strategy results in a significant robustness/performance boost on deep metric learning benchmarks.
main_review: Strengths:
- A thorough proof of the link between alignment/uniformity on the sphere is provided for the case of the triplet loss;
- The three variants of the proposed ADML provide a consist performance boost across various benchmarks;
- The relevance of attacking the alignment/uniformity terms is justified empirically (Table 1);
- The effect the hyperparameter $\lambda$ introduces by their training scheme is studied across multiple datasets.

Weaknesses:
- As stated in this paper, separating the triplet loss into the intra-class alignment and the hypersphere uniformity terms is intractable. Therefore it is unclear whether the performance gain is a result of the different loss choice or the robustness to intra-class alignment and the hypersphere uniformity perturbations. In other words:
    - Is the setting ADML+U+A, where both terms are adversarially attack, comparable to ADML+T? If that's the case then this would an argument in favor of attacking each one of the two terms separately. However, if that's not the case, then performance gain might be attributed to the InfoNCE loss being a better attack target;
    - The combination of the intra-class alignment and the hypersphere uniformity terms used to generate adversarial examples is the InfoNCE loss. Would training the ADML model using the InfoNCE loss result in comparable results?
I'm aware the gradients of the intra-class alignment and the hypersphere uniformity terms are only used to generate adversarial samples which are then used to train the model with the triplet loss. 

- Table 4 shows that the three ADML variants result in more robust models. However one needs to state that ADML+A has an unfair advantage compared to the rest of the models evaluated since it was explicitly trained for this scenario.
summary_of_the_review: This work extends known properties of contrastive losses to the triplet loss. Namely, the loss can be rewritten as the sum of two terms: a intra-class alignment term and a hypersphere uniformity term. Based on this, a novel regularization term was devised for training deep metric learning models based on adversarial attacks on each one of the two terms. The three proposed variants of the ADML model acheive convincing performance gains across different image retrieval benchmarks and are more robust to subsequent adversarial attacks. However, it is unclear whether gain in performance is due to adversarially perturbing the intra-class alignment and the hypersphere uniformity terms separately or due to InfoNCE having more informative gradients compared to the triplet loss.
review_title: null

Meta-review: While the paper has merits, I generally agree with negative reviewers. Among other issues, there were concerns about the theoretical contribution overlaps with prior work. While the authors argued the current work is not an extension, but rather designing ADML is. If this is the case, the paper should be rewritten to deemphasize the less novel contribution and focus more on what the authors believe to be the novel contribution. I don't believe in the practice of putting different messages (some novel and some not) into a paper with the hope that this makes the overall result "more novel". I'd suggest the author rewrite the paper and more clear about the message.
Decision: Reject