Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper provides a method for learning Bergman divergences using input convex neural networks as the convex generating function. This eliminates the shortcomings of previous work on the topic including non convexity of the neural network for deep divergence learning and high computational complexity of linear Bregman divergence learning. They further use CNN's as feature extractors and implement joint training. They experiment with various tasks such as regression, ranking and clustering and outperform the competing methods. Further they extend the Bregman divergence learning in two directions, one where the divergence satisfies both triangle inequality and symmetry and another where only triangle inequality is satisfied. 
strength_and_weaknesses: The paper has many contributions and experiments. The idea of using input convex neural networks for learning Bergman divergences is new. The extensions to satisfy the triangle inequality and symmetry are also new. They have done extensive experimentation with different tasks and have compared to a significant number of competing methods. Further they have defined interesting divergence regression tasks such as BregMNIST and BregCifar.
clarity,_quality,_novelty_and_reproducibility: The overall quality of the writing is good. There are many sections in the paper which looks misleading at first but has a good flow in general.
summary_of_the_review: I think the paper has done enough work and has good quality for publication in the conference. Accept.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper addresses the limitation of previous methods for Bregman divergence learning and proposes a solution named Neural Bregman Divergences (NBD) using Input Convex Neural Network (ICNN), which can be computed efficiently and gives finer resolution to the generating function. Experimental results verify the performance of the proposed method on many metric learning tasks.
strength_and_weaknesses: Strength:

This paper developed a new Neural Bregman Divergence, which jointly learns a  Bregman measure and a feature extracting neural network. It also demonstrates that the proposed method more faithfully learns divergences over a set of both new and previously studied tasks, including asymmetric regression, ranking, and clustering.

Weakness:

1. The novelty of this paper is limited. The proposed method basically follows the existing Bregman learning methods (Siahkamari et al., 2020; Cilingir et al.,2020), and only jointly learns a Bregman measure and a feature extracting neural network in the proposed method. 

2. The experimental results of non-Bregman learning do not outperform SOTA methods. Does it mean the proposed method NBD only achieves better result on specific tasks where the underlying ground truth is from Bregman divergence? Will it restrict the application of NBD?

clarity,_quality,_novelty_and_reproducibility: The writing can significantly be improved. 
The novelty of this paper is limited. 
summary_of_the_review: This paper developed a new Neural Bregman Divergence, which jointly learns a Bregman measure and a feature extracting neural network.  It also demonstrates that the proposed method more faithfully learns divergences over a set of both new and previously studied tasks, including asymmetric regression, ranking, and clustering. Some experimental results verify the effectiveness of the proposed method. However, the novelty of this paper is limited.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method for learning a Bregman divergence using neural networks. A Bregman divergence is the divergence defined using a convex function, and any convex function has its corresponding Bregman divergence. The proposed method learns the divergence by representing the convex function by an Input Convex Neural Network (ICNN) (Amos et al., 2017). The paper provides a series of experiments to confirm the superiority of the proposed method over the previous methods.
strength_and_weaknesses: # Strengths
- The approach of directly learning the convex function by an ICNN seems interesting. Related papers on ICNNs support the representation power of the model.
- The paper presents quite extensive experiments, and the proposed method shows excellent performance.

# Weaknesses
- I have a concern with Eq. (2). The paper says "by construction, the resulting neural network satisfies convexity," but I don't think this is the case. If, additionally, $g$ were a non-decreasing function as assumed in Amos et al. (2017), I would understand the claim. I understand that the proposed method uses $g(x) = \log(1 + \exp(x))$ which is nondecreasing, so this may not be a big problem. However, I wonder if theories by Chen et al. (2019) and Pitis et al. (2020) apply to this activation function.
- If I understand well, some experiments assume that divergence values are given as labels, which I believe is not very realistic in practice.
clarity,_quality,_novelty_and_reproducibility: - The paper is well-written and easy to follow.
- It cites and discuss relevant previous work properly, and the motivation of the work is clear.
- Introducing ICNNs to learning Bregman divergences is novel as far as I know.
- I think the paper and the supplementary material provide sufficient details for reproducing their experiments.
- One thing that is not very clear to me is what labels are given in each experiment.

Questions:
- In Eq. (3), is $\tilde{y}$ fixed or varied with $\theta$ when we take the gradient?
- In Section 5.1, is the triplet loss the same as that for PBDL described in Section 2.1?
- In Algorithm 1, line 8, what loss does it use?
summary_of_the_review: Overall, I like the paper and the authors make great contributions with many empirical results. I have a few concerns that I mentioned above, but I tend to accept the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new method called NBD to learn Bregman divergences between data points using a deep neural network. The proposed method is tested on some synthetic and small machine learning benchmark data sets. The results show that NBD outperforms several other metric learning approaches for some tasks.
strength_and_weaknesses: Strength:

The authors adopted a more generic method to parameterize the phi function, which seems better than several other existing Bregman divergence learning approaches in various settings.


Weakness:

I doubt the usefulness of non-Euclidean distance learning. The authors enumerate the use in retrieval, clustering, and ranking in the introduction. However, there is little evidence in the paper to show significant improvement in these applications compared to Euclidean approaches. Instead, the authors focused on various artificial settings of the pairwise distances. Winning in such artificial scenarios may not justify the new method.

Section 5.1 seems to be a realistic application. But the MAP@10 values are low compared with the state-of-the-art. Suppose we just use an image classifier (e.g., ResNet or Vision Transformer) and calculate the MAP@10 using the softmax outputs and their Euclidean distances. What is the benefit of using the proposed method?

It is infeasible to show that NBD works well for arbitrary non-Bregman distance learning. In Section 5.4, the authors have to modify the distance function as a remedy, which makes the claims looser.

The experiment setting is not clear. How many supervised pairs were used in training (compared to the number of data points)? Which pairs did you sample?

Table 4: most SVHN results are 96.9, which looks unnatural. Why?

clarity,_quality,_novelty_and_reproducibility: The writing is not very good.
- The end of Section 1 mixes the contributions and paper outline. Consequently, the major highlight of the work is unclear.
- The description of the main method is too short (Sections 2.2 and 2.3). The choice of ICNN is not well-motivated. Was it used for Bregman divergence learning before? Why is the ICNN parameterization better? A theoretical analysis is missing.

summary_of_the_review: The paper seems to have a step forward in non-Euclidean distance learning. However, the main contribution is unclear, and the impact of the proposed method is questionable. I tend to give borderline rejection.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Thanks for your submission to ICLR.

This was a borderline paper, with two quite positive reviewers and two more negative reviewers.  On the positive side, the reviewers liked the use of input convex neural networks to the Bregman divergence learning problem, and appreciated the experimental results.  On the negative side, some reviewers questioned the novelty of the approach as well as the overall usefulness of learning non-Euclidean distances.

After reading through this paper myself, I tend to agree with the more positive reviewers on this paper.  In particular, I do think there is novelty in using ICNNs for the BD learning problem, and find that it yields a much more scalable and practical solution as compared to the methods described in Siahkamari et al. and Cilingir et al.  I am also not particularly concerned with the issue of usefulness---non-Euclidean distances have been shown to be useful in a variety of contexts---this is well established---and the authors of this paper need not further establish this.
justification_for_why_not_higher_score: Though the paper meets the bar for publication, its novelty is somewhat low.  Further, the method may not be broadly of interest.
justification_for_why_not_lower_score: See above.  The positives outweigh the negatives in my opinion, and I have taken a close look at the reviews and the paper.
Decision: Accept: poster