Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 1: Trivial or wrong
review_content: The works proposes a generalization of MMD-squared distance. However, the submission seems to be an incomplete one.

Major comments:
1. Definition 2 seems to be the key definition in the work. However, there are multiple issues:
       a. It is not clear why it is called a kernel? Should not it be called distance? After all, it generalizes MMD-squared!
       b. "$K$" seems to be mixed up with "$k$". "$K$" seems to be the gram matrix and not the kernel.
       c. $k(y_i, y_j)$ is from $Y \times Y\rightarrow R$, and not from $R^n \times R^n \rightarrow R$.
       d. why should "\phi" belong to RKHS of k? Recall that RKHS of k will contain functions from R^n\rightarrow R.
2. I agree with the write-up which states  that results in section 4.2 are trivial.
3. section 4.3.1 are known results and need to be skipped.
4. Proof of theorem 5 seems to be completely missing. How is sup over f removed?
5. In Definition 6, what is $f$? Is a sup over f missing? Because of this and the previous issue, section 5.1 seems very incomplete.
6. Simulation section seems to be completely missing.
7. Connection with Bernstien polynomials highlighted in intro etc. seems to be missing.
review_title: Seems to be an incomplete submission with missing details

final_score: rating: 2: Strong rejection
review_content: Summary.
The authors describe a family of kernel functions on discrete probability measures. The kernel generalizes existing discrepancies such as the MMD and the KSD. The authors further provide plugin estimates based on empirical frequencies and some arguments for unbiased-ness.

While it is interesting to think about alternative estimators for comparing probability distributions, this paper falls short on the execution. I would recommend a major work-over before considering a submission again. There are many missing points in theory, experiments (there are none), and presentation. See below.

It is unclear to me why we would care about the proposed estimators
* There is no analysis showing that the presented kernels are useful in any way.
* I appreciate that the authors show that existing discrepancies are special cases of the proposed one, but I wonder again what that is useful for?
* This is in particular as the authors do not provide any sort of asymptotic analysis of the presented estimators. How can we use them for two-sample testing without that? Answering this question is one of the major parts of the kernel two-sample testing literature.

Theory.
* The presented theory consists of elementary manipulations that mostly follow existing literature, so there is very little actual innovation. For example of of page 5.

Experimental evaluation
* There is *no* experimental evaluation of the proposed estimators.
* It would have been interesting to compare the variances as a function of dataset characteristics.

Presentation
* There many grammar glitches, spelling mistages, missing articles, etc, to a point that it is hard to follow.
* There is no overview of the series of arguments in the later part of the paper.
* There is a lot of re-cited derivations from existing papers.


review_title: Interesting topic, but poorly executed

final_score: rating: 3: Clear rejection
review_content: The paper under review proposes to generalize MMD for discrete random variables whose labels take value in $\mathbb{R}^k$. They propose to estimate these generalized probability kernel distance using empirical estimators. Their properties are studied for two particular examples, namely a kernelized Stein discrenpancy and polynomials versions. Consistency and bias of both estimators are studied and bias corrected.

The paper has numerous typos, with approximate English and lacks of rigorousness when introducing mathematical concepts; multiple notations are never defined. On the theoretical side, the setting on which the contribution relies on is quite strange: in general, labels of discrete variables does not relate to any notion of distance/ordering as in a classical RKHS setting making the relevance of the methodology quite questionable. These point with other technical issues are summarized in following remarks:

## Major points
* As mentioned above, it is quite rare in a discrete setting that the labels lies in $\mathbb{R}$ and satisfies a notion of distance. The authors should better motivate this setting by giving at least on relevant example, either theoretical or practical, where such structure is relevant.
* What does a Stein operator in a discrete setting means? There is a diffenretial operator in Definition 5 that is difficult to generalize and apply in a discrete context.
* The symmetric KDSD introduced in Definition 6 is claimed to be a probability kernel, but the proof that is satisfies Definition 2 is not given.
* The so-called polynomial probability kernel seems to obviously require $l=k$ to satisfy conditions of Definition 3, i.e., that $|\phi(q,p)\| = 0$ implies that $p=q$. It can be called a probability kernel only in such condition.
\end{enumerate}



## Minor points
The paper has numerous typos and imprecisions; a subset of them are listed here.
* p.1: 'underline' should be 'underlying'.
* p.1 when using 'i.e.', always write ',e.i.,'.
* p.1 and onward: there is always a space missing before each parenthesis.
* p.1: Yi & Along (2020) should be a citep and not citet.
* p.1: 'remain futher study' should be for instance 'is left for future work'.
* p.1: KSD is not defined yet.
* p.2: 'the introducting' is `the introduction'.
* p.2 'in representing; is not right.
* p.2 Is $[k]$ the sample space? If yes what is $\{x_1,\dots,x_n\}$? A sample? What is the probability measure $v_i$? Do you mean the probability that $X$ falls in $v_i$?
* p.2 Definition 1: 'Given that distributionS p and q belong ... distributionS with...'. Also 'map' is singular. What is the 'function space' that you refer to? Also where does this definition comes from? Please give proper referencing.
* p.2: Why is there a line break right at the start of 4.1?
* p.2: what is an 'instance of integral probability metric'?
* p.2: last equation $\mu_p$ is not defined, the product opertor $<.,.>_{\mathcal{H}}$ is not defined. $\mathcal{H}$ is not defined.
* p.3: what these 'embedding functions'?
* p.3 the RBH kernel is not defined.
* p.3 second equation: what is $\phi$?
* p.3 Definition 2: the index the sample space should be k, i.e., $y_1,\dots,y_k$ if it refers to the distributions and $n$ for a sample. Here it should be $k$ as it is written distribution.
* p.3: 'examINE', 'members'.
* p.4: the 'brief' proof provided here is only working for discrete variables, while the proof in Gretton et. al deals with continuous variables.
* p.4: what is the 'term above'?
* p.5 'illustrate'
* p.5: there should not be such a thing as an 'art' in science. If you raise that question, then you should formally discuss this topic (choice of optimal $\phi$).
* p.5 Second equation: what are $x_s$ and $x_t$? Notations between this equation and the next are not consistent ($n$ is paired with $x$ and then with $y$ in the next equation).
* p.6: what is this so-called 'same property'?
* p.6: pmfs is never defined.
* p.6 Definition 5: notation $\mathcal{A}$ is never used. $s_p$ is not defined. $\Delta^*$ is not defined. If the latter is a differential operator, what does it means in the context of discrete random variables?
* p.6: what is 'form 5'?
* p.7: what forms 5 and 6?
* p.7: Theorem 5: operator $L$ is not defined. A dot is missing. Are $p$ and$q$ density functions of pmfs?
* p.7: 'preliminary results'.
* p.7: what does 'justing' mean?
* p.8: what is requirement $2$?
review_title: The paper has numerous typos, with approximate English and lacks of rigorousness when introducing mathematical concepts; multiple notations are never defined.

final_score: rating: 2: Strong rejection
review_content: This paper tries to propose a kernel-based discrepancy measure called generalised probability kernel that can unify MMD and KSD which is an interesting topic of discussion. The paper applies the new discrepancy to perform two-sample tests. 
The new kernel proposed, unlike the previous RKHS kernels that only depend on data-points, incorporate the notion of probability. e.g. kernel K_{p,q} depends on density p and q. also a symmetric version on discrete KSD is discussed.
Despite the idea is interesting, there are several flaws which can be reviewed.

Firstly, I think the paper is not clearly presented, with some confusing notation.
--in Definition 1, you defined a kernel, on distributions p and q, that is a k x k matrix; while in definition 2, the notion of K, are on samples and is a scalar output.
it is unclear of how \phi is defined in general; only examples are given later for specific cases so that we got an conjuncture.
--in Definition 5, why is it different from stein operator of KDSD? or it is supposed to say difference operator?

In addition I have several confusions:
1. why is MMD_E^2 an unbiased estimator? what happened to k(x_i, x_i)? it is not clear from the Bernstein polynomial introduced in appendix. 
2. in abstract, it claims that the kernels are between distributions instead of samples, but in the main text it is still evaluation at p_i=p(x_i) on samples; I m confused of the difference and novelty claimed.
3. The above concern brings up the question while applying on two sample test. 
--When the MMD is used to perform two-sample test, it is assumed that both p and q are unknown. however, to my understanding, we need to know p and q to define k_{prob}; how is this going to be applied to two-sample test? 
--for KSD setting, when the symmetric KDSD is introduced, it also seems to require p and q to known for two-sample testing. In the Liu2016 setting, where goodness-of-fit test is proposed with KSD, q is known (up to normalization) while p is unknown with samples; that is a key point why KSD is useful for goodness-of-fit test.
In addition, is there any argument on why the symmetric-KDSD might be better than KDSD Yang et.al 2018?

An additional point is regarding literature review, which is yet throughout  to check; e.g. as
Chwialkowski, et. al  "A kernel test of goodness of fit." proposed independently as Liu et.al for KSD goodness-of-fit test, that might be useful to cite.

In my point of view, ICLR may not be a venue of fit either. More reviews and clarifications may be required, for both kernel construction and application. 
review_title: new kernels incorporating probability notion are proposed to perform two-sample test; but not yet clearly defined/explained.

Meta-review: The focus of the submission is to define divergences on discrete probability measures. Particularly, the authors propose a common generalization of the well-known concept of maximum mean discrepancy and kernel Stein discrepancy.

As summarized by the reviewers the submission is in a rather preliminary form:
1)The work lacks motivation.
2)Literature review (there are 4 references in total) and numerical illustrations are missing.
3)The submission lacks proper mathematical formulation/rigor.
I highly recommend the authors to not submit similar draft manuscripts in the future.
Decision: Reject