Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a robust Bayesian deep metric learning framework against noise label inspired the BLMNN (Wang & Tan, 2018), deep metric learning (Hoffer & Ailon, 2015; Hu et al., 2015; Wang et al., 2017; Lu et al., 2017; Do et al., 2019), and Bayes by Backprop (Blundell et al., 2015). Directly applying the variational Bayes learning (Wang & Tan, 2018) in deep learning is challenging since it requires sampling from a distribution of the neural network parameters. Instead, this paper adapts the variational inference by Blundell et al. (Blundell et al., 2015), which allows to efficiently sample the parameters of a Bayes neural networks by using a backpropagation-compatible algorithm. The experimental results on several noisy data sets show that our novel proposed method can generalize better compared to the linear BLMNN (Wang & Tan, 2018) and the point estimation-based deep metric learning (Hoffer & Ailon,
2015; Lu et al., 2017), especially when the noise level increases.

Pros:
1.  This paper is well-organized and well-written.
2.  Adapting the variational inference by Blundell et al. (Blundell et al., 2015) for Bayesian DML sounds good.
3.  The theoretical analysis is completed (though meaningless)

Cons:
1.	The novelty is limited (just using sliding window to instead growing window). It is more like a combination of variational inference (Blundell et al., 2015) and DML.
2.	The results of Theorem 1 are meaningless. Some papers [1] Robustness and Generalization for Metric Learning. and [2] Deep Metric Learning: The Generalization Analysis and an Adaptive Algorithm. may help you to understand this point. 
review_title: This paper should not be accepted by ICLR2021

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper introduces a Bayesian deep metric learning framework that is robust against noise labels. The proposed method is inspired by the BLMNN (Wang & Tan, 2018), deep metric learning (Hoffer & Ailon, 2015; Hu et al., 2015; Wang et al., 2017; Lu et al., 2017; Do et al., 2019), and Bayes by Backprop (Blundell et al., 2015).  Different from BLMNN that only considers a linear metric learning, the authors’ framework can handle non-linear deep metric learning, which is useful for many real-life applications. Moreover, directly applying the variational Bayes learning (Wang & Tan, 2018) in deep learning is challenging since it requires sampling from a distribution of the neural network parameters. Instead, The author adapt the variational inference by Blundell et al. (Blundell et al., 2015), which allows to efficiently sample the parameters of a Bayes neural networks by using a backpropagation-compatible algorithm. They also theoretically show the robustness of the proposed method when working with label noise. The experimental results on several noisy data sets show that their novel proposed method can generalize better compared to the linear BLMNN (Wang & Tan, 2018) and the point estimation-based deep metric learning (Hoffer & Ailon, 2015; Lu et al., 2017), especially when the noise level increases. In my opinion, the main novelty of this

+ves: 
+ The idea of using the variational inference by Blundell et al. (Blundell et al., 2015) to derive Bayesian version of deep metric learning is interesting. 

+ Overall, the paper is well-written and well-organized. In particular, the Related Work section has a nice flow and puts the proposed method into context. Despite the method having limited novelty (sliding window instead of a growing window), the method has been well motivated in Sections 1 and 3. 

+ The theoretical analysis section is completed. And the results section is well structured. It's nice to see the advantages of the proposed method over other compared methods under different datasets with label noise
 
Concerns: 
- The key concern about the paper is the contributions of this paper. In my opinion, the novelty of this paper is limited. The main contribution is to propose a Bayesian version of deep metric learning. In fact, the main strategies and motivation from (Blundell et al., 2015) and (Wang & Tan, 2018). I cannot find any original contributions such as sampling methods and novel loss function in this paper. In a word, this paper only combines variational inference by Blundell et al. (Blundell et al., 2015) with deep metric learning without any original techniques.
 
- Lack of sufficient analysis to show that your method is robust to label noise. In modeling, the authors did not consider any label information, but the model is robust to label noise. This is questionable. In addition, the result of Theorem 1 is meaningless. 
I don’t want to waste my time explaining why your theoretical results are meaningless, but I recommend some papers as follows:
[1] Deep Metric Learning: The Generalization Analysis and an Adaptive Algorithm
[2] Robustness and Generalization for Metric Learning

-According to my experience, the Bayesian version of a model should be much higher than the original model. But the author's experimental results are not satisfied. 

I can't find any contribution that can conquer me, thus, this paper should not be accepted by ICLR2021

 




review_title: I can't find any contributions that can conquer me, thus, this paper should not be accepted by ICLR2021   

final_score: rating: 3: Clear rejection
review_content: Briefing:

This paper proposes a new Bayesian metric learning method, robust to noisy labels. The paper introduces a variational formulation for incorporating the Bayesian framework to triplet-loss training, with supporting experiments.

Strong points

The paper proposes fancy variational derivation for the Bayesian frameworks for triplet-loss training.

Weak points

(1) An essential reference [1] and other triplet ablations [2] seems to be missing.

[1] Lin, Xudong, et al. "Deep variational metric learning." Proceedings of the European Conference on Computer Vision (ECCV). 2018.

[2] Duan, Yueqi, et al. "Deep adversarial metric learning." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.


(2) Connected to (1), Experiments do not seem to be enough for supporting the superiority of the proposed method.


Comments:

(1) The author should clarify the difference between [1]. [1] also uses variational formulation for the triplet loss.

(2) Experimental setup seems to be not enough. Along with [1], typical retrieval dataset such as CUB-200- 2011, Stanford Online Products dataset, Cars196 dataset should be concerned.

(3) Along with [1] and [2], the author should compare recent triplet-loss-based embeddings with the mentioned typical retrieval datasets.

Note:

The paper seems to miss the related works that must compare. The paper also does not seem to include typical retrieval tasks to compare with other metric learning methods, which has been standard experiments in this field. Unless a clear explanation for this issue are provided, the reviewer cannot avoid rejecting this paper. 
review_title: Reviewer1 Comments

final_score: rating: 7: Good paper, accept
review_content: # Summary

The authors present a method to employ Bayesian deep learning with a metric learning (triplet loss) objective. This is a significant extension to existing Bayesian deep learning approaches, which have focused on regression and classification approaches with vanilla L2 and cross entropy losses.

# pros:

- The proposed method appears to be first to soundly combine deep metric learning with a Bayesian approach (Bayes by Backprop) to estimate parameter uncertainties. 

- In an extensive evaluation, the method is shown to deal well with highly noisy labels (up to 50%) in classification tasks like MNIST and CIFAR-10. In high noise setups, the method is shown to outperform linear Bayesian metric learning, as well as deep point estimate metric learning.

- The paper is clearly written

# cons:

- The method is not evaluated against the state of the art on the evaluated datasets, albeit the authors give a convincing reason for this, namely that the methodological novelty of the proposed approach warrants proof-of-concept results with a relatively simple neural network architecture

# comments:

- It would be interesting to hear in more detail what motivated the authors' choice of Blundell's approach to uncertainty estimation as compared to e.g. the Dropout- / Deep Gaussian Process-based works by Yarin Gal et al. 

- The abstract mentions "simulated label noise" as something the authors have considered, whereas the experiments mention "synthetic" and "realistic label noise", and conclusion refers to "simulated label noise" in the context of future work. Could the authors please clarify, and add a more thorough description of the respective types of noise?

- I wasn't able to fully follow the theoretical derivation of the authors on how the approach of Blundell et al. (2015), proposed for vanilla regression problems, transfers to a metric learning setup. Hence I have low confidence in my review. 

review_title: First theoretically sound approach to Bayesian deep metric learning. Improved performance for highly noisy labels compared to linear Bayesian model as well as deep point estimate model

Meta-review: Reviewers have commented on the lack of novelty of the paper as it reads only as applying the variational inference framework of Blundell et al. (2015) to deep metric learning (R2 and R4). Furthermore, the paper has not properly positioned itself when compared to previous works on "Deep variational metric learning" and "Deep adversarial metric learning" (R1) and other previous literature that have studied robustness for metric learning. The argument on robustness to noisy labels needs to be expanded and better fleshed out in a future version of the paper. 
Decision: Reject