Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes a new divergence, called spread divergence, to distinguish probability models. The approach is motivated from the concern that traditional divergence such as f-divergence or KL divergence may not always exist, in which the spread divergence may be a substitute. Some empirical supports are provided for the proposed method. Below I will summarize my concerns.

1. The spread divergence is proven no larger than the traditional divergence, so the paper claims this as an advantage of using spread divergence. My question is that if KL or f-divergence of two probability models is infinity, which means they distinguish the models very well, whether a new method is necessary (though it may provide a finite value). 

2. As a new method, it would be useful to thoroughly compare it with the traditional ones. There is a lack theoretical comparison with the KL or f-divergence. Some numerical examples are provided but seem not enough. For instance, the applications focus on the situations where likelihood is not defined, i.e., data are deterministic w/o observation noise. It is interesting to see other examples where likelihood is defined and how traditional methods perform.

3. Kernel based spread divergence has been a major focus of this paper. It is interesting to see which kernel maximizes the spread divergence. Section 3.2 considers Gaussian kernel. Is this an optimal option?

4. Section 5 compares EM and spread EM based on one experiment and claims the latter has smaller error.   Does the same conclusion holds true in other examples?

I believe the motivation of this paper is interesting. This would be a stronger paper if more theoretical and empirical analysis can be added.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: I think the paper must be Desk-rejected as the identity of the authors was revealed. 
This thing aside, the paper is an interesting contribution. The concept of spread divergence can be valuable in many context. The presentation is thorough and the theoretical part is correct. On the other hand, the examples are quite diverse and include a standard model (ICA) as well as modern deep generative models. Thus, it represents a valuable contribution worth of publication, if we ignore the identity revelation aspect. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper introduced a way to modify densities such that their support agrees and that the Kullback-Leibler divergence can be computed without diverging. Proof of concept of using the spread KL divergence to ICA and Deep Generative Models ($\delta$-VAE) are reported based on the study of spread MLE.

Comments:

In Sec 1, mention that f should be strictly convex at 1. Also mention 
Jensen-Shannon divergence, a KL symmetrization, which is always finite 
and used in GAN analysis.

In Sec 2, you can also choose to dilute the densities with a mixture: 
(1-\epsilon)p+\epsilon noise.
Explain why spread is better than that? Does spreading introduce 
spurious modes?, does it change distribution sufficiency? 
(Fisher-Neymann thm)

In Formula 4, there is an error: missing denominator of \sigma^2. See 
Appendix D too.

In footnote 4, page 8, missing a 1/2 factor in from of TV (that is upper 
bounded by 1 and not 2)

KL is relative entropy= cross-entropy minus entropy. What about spread KL?
In general, what statistical properties are kept by using the spread? 
(or its convolution subcase?)

Is spreading a trick that introduces a hyperparameter that can then be 
optimized for retaining discriminatory power, or is there
some deeper statistical theory to motivate it. I think spread MLE should 
be further explored and detailed to other scenarii.

Spreading can be done with convolution and in general by Eq.3:

Then what is the theoretical interpretation of doing non-convolutional 
spreading?


A drawback is that optimization on the spread noise hyperparameter is 
necessary (Fig 3b is indeed much better than Fig 3a).
Is there any first principles that can guide this optimization rather 
than black-box optimization?

Overall, it is a nice work but further statistical guiding principles 
or/and new ML applications of spread divergences/MLE will strengthen the 
work.
The connection, if any, with Jensen-Shannon divergence shall be stated 
and explored.

Minor comments:

In the abstract, state KL divergence instead of divergence because 
Jensen-Shannon divergence exists always.


Typos:
p. 6 boumd->bound
Bibliography : Cramir->Cramer, and various upper cases missing (eg. 
wasserstein ->Wasserstein)

review_title: Official Blind Review #2

Meta-review: This paper studies spread divergence between distributions, which may exist in settings where the divergence between said distributions does not. The reviewers feel this work does not have sufficient technical novelty to merit acceptance at this time.
Decision: Reject