Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new class of discrepancies between two continuous probability distributions. The proposed class, which is called the H-divergence, contains an extended class of Jensen Shannon divergences called H-Jensen Shannon divergences and another class (2) called the H-Min divergences as special cases. The conditions that the two probability distributions have non-negative H-divergence are given. It is seen that the set of H-Jensen Shannon divergences includes the set of squared Maximum Mean Discrepancies as a subset. Estimation and convergence of the H-divergence are discussed. The H-divergence is applied to propose two-sample tests and experiments suggest that the proposed tests outperform some existing tests in terms of power. The proposed methods are applied to climate data for decision making in agriculture and energy production. 
main_review: **Strengths:**

(a) The proposed class of divergences, H-divergences, includes well-known divergences such as Jensen Shannon divergence and squared Maximum Mean Discrepancy as special cases. The H-divergence also contains new classes of divergences, including the ones defined in Equations (1) and (2), which appear to be potentially useful in practice.

(b) Some results are given to prove the convergence of the empirical H-divergence to its theoretical one; see Theorem 2 and Corollary 1. 

(c) The conditions on the non-negativity of the H-divergence are obtained; see Section 3.3. These theoretical results partly justify the proposed divergence as a reasonable discrepancy between two probability distributions.

(d) A sufficient number of experiments are given to demonstrate the usefulness of the presented methods; see Sections 4 and 5. 

(e) The paper is clearly written. Appendix provides helpful information including detailed proofs of the theoretical results of the main article.

&nbsp;

**Weaknesses:**

(f) Applications of divergences include not only two sample tests but also other methods such as robust estimation and independence tests. However the paper does not discuss possible applications other than two sample tests and plots in Figure 3. A brief discussion about other applications would be helpful for readers.

(g) The variance of the proposed estimator $\hat{D}^{\phi}_{\ell}  (\hat{p}_m || \hat{q}_m) $ is not discussed in the paper. Is it possible to discuss any result about the variance?

&nbsp;

**Minor Comments:**

(h) p.1, Section 1, 1st paragraph, l.13: Add a full stop after 'point'.

(i) p.1, Section 1, 2nd paragraph, l.1, etc.: There are at least three different expressions, namely, *H*-divergence, H-divergence and H-Divergence, in the paper to denote the same divergence. The expression should be standardized throughout the paper.

(j) p.4, Section 3.3, l.6 up: This property allow us to  ===>  This property allows us to 
summary_of_the_review: The paper is generally well-written. Properties of the proposed class of divergences are well-investigated. A sufficient number of experiments are provided to demonstrate the usefulness of the proposed method.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new category of divergences, called H-divergence, for measuring the discrepancy  between two probability distributions. H-divergence is based on the optimal loss with respect to a chosen decision task and is therefore decision dependent. Further, it generalizes a few well known divergences such as the Jensen-Shannon divergence and the maximum mean discrepancy family. The paper proves a few properties of H-divergence including a convergence result when H-divergence is estimated using a finite set of samples. Further, three examples are used to illustrate the applications of H-divergence  including two sample tests, assessing climate change, and feature selection, which demonstrate the advantage of H-divergence compared with other commonly used discrepancies.
main_review: Strengths

1. Measuring the difference between two probability distributions is a fundamental problem in machine learning. Although many different types of divergencies have been proposed in the literature, they are typically decision independent. H-divergence seems a simple yet effective way of incorporating decision related domain knowledge into the discrepancy measure. 
2. The theoretical results are non-trivial are provide insightful connections between H-divergence and other commonly used discrepancies. 
3. The paper is well written. The theoretical results are solid and clearly explained. The three examples  clearly illustrate the broad applicability of H-divergence. 

Questions:

1. Theorem 2 assumes that the same number of data points are sampled from p and q, respectively. I wonder if the result can be generalized to the case when the number of samples is different for p and q. 
2. Can H-divergence still be useful when the decision task is initially unknown (as in reward-free reinforcement learning) or uncertain (as in multi-task learning)? 
3. In Section 4.2, only the results for alpha = 0.05 are given. I wonder if similar patterns hold for other values of alpha. 

summary_of_the_review: Overall I think this is a very interesting paper. The new family of discrepancies is likely to find broad applications in machine learning and data science and can potentially inspire the development of other decision dependent discrepancy measures.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes H-divergence, a new type of divergence based on H-entropy, to compare two probability distributions. This new divergence includes some of the commonly used integral probability metrics and f-divergences, such as Jensen Shannon divergence and MMD as special cases. A crucial property of H-divergence is that it takes into account the decision loss; namely, it compares two distributions in a way that distinguishes them based on the optimal decision loss, i.e., "two distributions are different if the optimal decision loss is higher on their mixture than on each individual distribution." It also provides an empirical estimator for H-divergence and studies its convergence properties. The paper studies several use cases of H-divergence, including two-sample tests. Experiments demonstrate that H-divergence achieves higher test power than tests based on MMD under the same type I error rate. Another important use case of H-divergence is understanding whether differences between distributions are significant enough to affect decision making (from the viewpoint of minimizing loss) in different experiments, including how climate change affects various economic activities.

main_review: Strengths: 

Proposes a new type of divergence that compares two probability distributions from the lens of optimal decision-making. By setting an action set and loss function, one can identify "how far two distributions are" in terms of leading to the same optimal action. Moreover, the paper clearly explains the connection of H-divergence with f-divergence and integral probability metrics. Overall, H-divergence can be seen as a valuable contribution to the family of divergences used in machine learning tasks. 

Experiments are satisfactory. Especially, the flexibility achieved by using H-divergence by choosing application tailored actions and losses highlight the merit of H-divergence in performing two samples test. 

Weaknesses:

The convergence properties of the empirical estimator of H-divergence deserve more discussion in the main paper. In particular, the role of Rademacher complexity in the bounds in Theorem 2 and Corollary 1 is not clearly explained. It will be good to add a discussion in line with proof of Corollary 1 (on how fast Rademacher complexity vanishes) into the main paper. 

The paper does not give enough detail about computational aspects. This clearly depends on the structure of the action set and loss function. The paper would benefit from the characterization of a set of (general) sufficient conditions under which computation of H-divergence is feasible. 

summary_of_the_review: This paper makes a fundamental contribution to the family of divergences used for machine learning tasks and provides convincing evidence on the usefulness of H-divergence. However, there is still room for improvement related to the computation of H-divergence. Please describe the feasibility of computation of H-divergence for the experimental examples studied in the paper in your response.
review_title: null

Meta-review: Although by now there are several approaches for comparing probability distribution, the paper innovates by making their measure take into account the decision space and loss functions directly. The paper also frames its contribution within the literature at large. Reviewers were unanimous that the result is of major interest to the ICLR audience.
Decision: Accept (Oral)