Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Authors research the problem of robust metric learning in this work. They propose a min-max formulation to learn the adversarial example and robust representations, simultaneously. The empirical study confirms the effectiveness of the proposed method. My concerns are as follows.

1.	In this work, authors adopt the positive example rather than anchor in a triplet for perturbation. However, as they illustrated in Section 3.2, the attack only can be applied on anchors while the reference points are fixed. The reason for current choice is that having perturbation on positive example achieves best performance as shown in appendix. But it may be due to the problem in the algorithm or implementation since the behavior of optimizing anchors is weird as reported. Moreover, perturbing anchors work well as reported in other work [2]. The current setting for perturbation is inconsistent with the practical applications.
2.	It is not clear how $\rho(x_i, x_j)$ is obtained. Is the optimization problem solved to be optimum or just an approximated solution?
3.	There is no baseline from robust optimization for comparison. At least robust triplet loss in [2] can be included. Besides, classification is a strong baseline for metric learning [3]. Therefore, a robust classification model can be involved in the empirical study.


[1] A. Sinha, et al. Certifying Some Distributional Robustness with Principled Adversarial Training.
[2] C. Mao, et al. Metric Learning for Adversarial Robustness.
[3] A. Zhai, et al. Classification is a Strong Baseline for Deep Metric Learning.

review_title: Learn robust representations with perturbation on reference points

final_score: rating: 5: Marginally below acceptance threshold
review_content: edit after rebuttal:

My opinion about the paper has not changed. Although the general idea is interesting, my main concern is that the approach aims at performing defense against a specific attack. The robustness of the approach w.r.t. other attacks (such as L_2 and L_0) needs to be evaluated.

====

The paper proposes a robust deep metric learning approach to adversarial attacks. Unlike previous Deep Metric Learning (DML) approaches, the approach focuses on robust optimization-based training that uses a saddle-point formulation. The approach considers the dependence of two classic metric losses (constrastive and triplet-loss) on the samples of a mini-batch to produce adversarial attacks. 
Given a pair of samples in a mini-batch i and j, a perturbed variant of i wrt j is created as formulated in the first equation of Section 3.4. Depending on whether i and j are similar or not, a perturbation delta in some epsilon-ball increases or decreases the squared Euclidean distance between the representations of i and j. 

The paper then evaluates how classic deep metric learning approaches are robust to Projected Gradient Descent (PGD) attacks. The results on some standard datasets show that classic metric learning approaches are very weak to PGD attacks (see Table 1), and the proposed approach is more robust as illustrated in Table 2.

The paper is well written in general although the experimental section is sometimes hard to follow. It took me some time to understand what the difference between the tables was. 
My main concern is that only one kind of adversarial attack is considered in the paper. In the second contribution, the authors state that classic DML approaches "do not have any robustness — their accuracy drops to close to zero when subjected to PGD attacks that we formulate." Classic DML approach are weak to the evaluated adversarial attack, but what about other kinds of adversarial attacks? 
It is not surprising that a specific method optimized for this attack is more robust. Is the proposed approach robust to other kinds of adversarial attacks?

If I understand correctly, according to the definition of the distance in Section 3.1, the perturbation delta is performed in the input space of the neural network. How is the argmax/argmin problem solved in Section 3.4? I tried to check the code but only saw inputs perturbed by -epsilon or +epsilon and then clipped. I do not see where the argmax/argmin problem is solved, if the neural network is highly nonconvex, the problem might be hard to solve. This needs a discussion, or at least a reference.
For instance, if (x_j, x_i) are dissimilar and x_j is in the epsilon-ball centered at x_i, then rho(x_i, x_j) should be equal to x_j. How can the proposed approach be robust to such a case?

How does the proposed approach have an impact on the norm of the learned representations?
Can the authors perform an analysis on the difference of representations between classic DML and the proposed approach?
What about the robustness for a different value of epsilon or type of norm used during training?




Minor comment: The equation in Formulation 1 is confusing because i is used as index twice (once in the sum, and once in the max). Also please keep equation indices for most equations, it makes reviewing easier.
review_title: The idea is interesting but the paper lacks some analysis

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

This paper analyzes the robustness to adversarial attacks of deep metric learning (DML) models for image similarity. A robust training framework is proposed. Experiments are performed on standard DML datasets, showing that existing deep metric learning models are vulnerable to adversarial attacks and that the proposed training protocol improves their robustness.

Positive points:
- The topic of rendering deep metric learning robust to adversarial attacks has received little attention in the past, and the results presented in the paper seem indeed novel.
- The proposed attack and robust training procedure show results consistent with the original PGD attack and the adversarial training based on it.
- The experiments show clear robustness improvements with the proposed training strategy (although, the performance of the models seems too low for practical use).
- The code for the submission is provided.

Concerns:
- The derivation of the proposed robust training framework for DML is a bit unclear. The final optimization objective seems to be stated without much justification as to why it enforces robustness.
- It would be great to be able to compare the proposed strategy with other results. Maybe R@1 could be compared to [Mao et al.] (cited by the paper)?.

Reasons for score:

Overall, I lean towards accepting the paper, as it seems to propose the first results around robust deep metric learning. The topic does seem somewhat relevant to the community. Good practices from the community around strong attacks and adversarial training seem to have been followed.

Questions / suggestions:
- How come robust performance on clean data is almost as good as baseline performance (Appendix Tab. 3)?

Minor comments:
- A few typos remain throughout the paper.
- Different values of the attack strength $\epsilon$ do not represent different threat models.
- Appendix Tab. 3: the bold value for CUB200-2011 with contrastive loss does not seem to reflect the best performance.

References:
[Mao et al.] Metric learning for adversarial robustness, 2019.
review_title: First results for robust deep metric learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
The authors propose a novel robust training approach for deep metric learning (DML), accounting for the dependencies of metric losses within mini-batches. The proposed approach is evaluated on several popular metric learning datasets, demonstrating that the method works as intended, and achieves a certain level of robustness, unlike the baseline non-robust model which achieves a very poor performance when exposed to an adversarial attack.

Comments:

In the opening paragraphs of the paper, the authors state ‘Our key insight is that during an inference-time attack, the positive point in the triplet will be modified by the attacker, and thus, during training, we must perturb positive points in the triplets, instead of anchors or negative points.’, yet towards the end (and despite having motivated it in between), they show that the results on this are inconclusive and that it is not clear that perturbing only the positive points in the triplets is the right / best thing to do. Unless I am misunderstanding something, this feels out of sync - either it is a key insight, or inconclusive? It can’t be both.

Opening paragraphs / under Q1 and Q2 in Section 4 (Experiments) - this feels both redundant (both were mentioned before under Contributions) and also out of place, as it mentions results before even introducing the experimental setup.

The authors evaluate their proposed robustness approach under a projected gradient descent (PGD) attack. While this is certainly sufficient to establish that the method works and that it provides some level of robustness to adversarial attacks, it feels really limiting to only assess a single attack type, of various options that are available - as it would have been potentially valuable to establish the degree of provided robustness under these different cases. As is, it is unclear whether the proposed approach will be universally helpful, or merely helpful with a particular attack type. While there is no reason to believe in the latter, the former hasn’t been substantiated nor argumented.

Apart from the general metrics shown in the tables, there isn’t much additional analysis that would aim to reveal whether there were any patterns in these datasets on where the method worked vs didn’t. For example, was the performance uniform across (pairs of) classes? If not, why? What about contrasting class pairs that are more/less similar? Or are there issues with rare classes?

Tables 1, 2 and 3 should include confidence intervals.

The authors use the phrase significant to qualify differences in several parts of the paper, yet there is no mention of which statistical test has been used to claim statistical significance of the differences? The authors should conduct proper statistical testing and highlight the exact test in the text.

There is no discussion of the limitations of the current approach / areas for potential improvement and future work in the main paper - instead, some open questions are mentioned in the Appendix. It would be good to include key discussion points in the main body of the paper.

Nit: Page 3, when giving lb(A,z) = c_k(A,z) - what if there is more than a single index returned by the argmin - what if there is a tie? The authors should specify if they are doing random tie breaking or taking the majority label (if a multi-way tie)

Nit: Figure 4, please update with the finalized results.
review_title: Interesting idea, promising results, though it seems a bit rushed

Meta-review: This paper proposed a novel Adversarial Deep Metric Learning approaches. The reviews pointed out the paper proposes an interesting idea and it is among the rare works that address directly robust metric learning which an important topic for efficient metric learning. 
Some concerns were raised about the analysis and the lack of comparisons notably with other types of adversarial attacks. 
The authors provide a rebuttal where they addressed some concerns raised by reviewers with some precisions on the work, its positioning with respect to other related papers and additional comparisons notably with other types of attacks. 
A minor remark: there is a typo in Eq(13), where the $z$ in the loss function is actually not defined and should be included in the max function.
That being said, the contribution is still limited in considering only the infinite norm, analysis and comparisons to prior work remain weak. The paper does not meet the requirements for acceptance to ICLR in its current form.
I have then to propose rejection.

Decision: Reject