Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes an importance-weighted estimator of the MMD, in order to estimate the MMD between distributions based on samples biased according to a known scheme. It then discusses how to estimate the scheme when it is unknown, and further proposes using it in either the MMD-based generative models of Y. Li et al. (2015) / Dziugaite et al. (2015), or in the MMD GAN of C.-L. Li et al. (2017).

The estimator itself is natural (and relatively obvious), though it has some drawbacks that aren't fully discussed (below).

The application to GAN-type learning is reasonable, and topical. The first, univariate, experiment shows that the scheme is at least plausible. But the second experiment, involving a simple T ratio based on whether an MNIST digit is a 0 or a 1, doesn't even really work! (The best model only gets the underrepresented class from 20% up to less than 40%, rather than the desired 50%, and the "more realistic" setting only to 33%.)

It would be helpful to debug whether this is due to the classifier being incorrect, estimator inaccuracies, or what. In particular, I would try using T based on a pretrained convnet independent of the autoencoder representation in the MMD GAN, to help diagnose where the failure mode comes from.

Without at least a working should-be-easy example like this, and with the rest of the paper's technical contribution so small, I just don't think this paper is ready for ICLR.

It's also worth noting that the equivalent algorithm for either vanilla GANs or Wasserstein GANs would be equally obvious.

Estimator:

In the discussion about (2): where does the 1/m bias come from? This doesn't seem to be in Robert and Casella section 3.3.2, which is the part of the book that I assume you're referring to (incidentally, you should specify that rather than just citing a 600-page textbook).

Moreover, it is worth noting that Robert and Cassela emphasize that if E[1 / \tilde T] is infinite, the importance sampling estimator can be quite bad (for example, the estimator may have infinite variance). This happens when \tilde T puts mass in a neighborhood around 0, i.e. when the thinned distribution doesn't have support at any place that P does. In the biased-observations case, this is in some sense unsurprising: if we don't see *any* data in a particular class of inputs, then our estimates can be quite bad (since we know nothing about a group of inputs that might strongly affect the results). In the modulating case, the equivalent situation is when F(x) lacks a mean, which seems less likely. Thus although this is probably not a huge problem for your case, it's worth at least mentioning. (See also the following relevant blog posts:  https://radfordneal.wordpress.com/2008/08/17/the-harmonic-mean-of-the-likelihood-worst-monte-carlo-method-ever/ and https://xianblog.wordpress.com/2012/03/12/is-vs-self-normalised-is/ .)

The paper might be improved by stating (and proving) a theorem with expressions for the rate of convergence of the estimator, and how they depend on T.


Minor:

Another piece of somewhat-related work is Xiong and Schneider, Learning from Point Sets with Observational Bias, UAI 2014.

Sutherland et al. 2016 and 2017, often referenced in the same block of citations, are the same paper.

On page 3, above (1): "Since we have projected the distributons into an infinite-dimensional space, the distance between the two distributions is zero if and only if all their moments are the same." An infinite-dimensional space isn't enough; the kernel must further be characteristic, as you mention. See e.g. Sriperumbuder et al. (AISTATS 2010) for more details.

Figure 1(b) seems to be plotting only the first term of \tilde T, without the + 0.5.
review_title: Interesting idea, not enough evaluation

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a modification of the objective used to train generative networks with an MMD adversary (i.e. as in Dziugaite et al or Li et al 2015), where importance weighting is used to evaluate the MMD against a target distribution which differs from the data distribution. The goal is that this could be used to correct for known bias in the training data — the example considered here is for class imbalance for known, fixed classes.

Using importance sampling to estimate the MMD is straightforward only if the relationship between the data-generating distribution and the desired target distribution is somehow known and computable. Unfortunately the treatment of how this can be learned in general in section 4 is rather thin, and the only actual example here is on class imbalance. It would be good to see a comparison with other approaches for handling class imbalance. A straightforward one would be to use a stratified sampling scheme in selecting minibatches — i.e. rather than drawing minibatches uniformly from labeled data, select each minibatch by sampling an equal number of representatives from each class from the data. (Fundamentally, this requires explicit labels for whatever sort of bias we wish to correct for, for every entry in the dataset.) I don't think the demonstration of how to compute the MMD with an importance sampling estimate is a sufficient contribution on its own.

Also, I am afraid I do not understand the description of subfigures a through c in figure 1. The target distribution p(x) is given in 1(a), a thinning function in 1(b), and an observed distribution in 1(c). As described, the observed data distribution in 1(c) should be found by multiplying the density in 1(a) by the function in 1(b) and then normalizing. However, the function \tilde T(x) in 1(b) takes values near zero when x < 0, meaning the product \tilde T(x)p(x) should also be near zero. But in figure 1(c), the mode of p(x) near x=0 actually has higher probability than the mode near x=2, despite the fact that there \tilde T(x) \approx 0.5. I think this might simply be a mistake in the definition of \tilde T(x), and that rather it should be 1.0 - \tilde T(x), but in any case this is quite confusing.

I also am confused by the results in figure 2. I would have thought that the right column, where the thinning function is used to correct for the class imbalance, would then have approximately equal numbers of zeros and ones in the generative samples. But, there are still more zeros by a factor of around 2.

Minor note: please double-check references, there seem to be some issues; for example, Sutherland et al is cited twice, once as appearing at ICML 2016 and once as appearing at ICML 2017.


review_title: Importance sampling correction to MMD for handling class imbalance

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper addresses the problem of sample selection bias in MMD-GANs. Instead of having access to an i.i.d. sample from the  distribution of interest, it is assumed that the dataset is subject to sample selection bias or the data has been gathered via a biased sample selection mechanism. Specifically, the observed data are drawn from the modified distribution T(x)P(x) where P(x) is the true distribution we aim to estimate and T(x) is an appropriately scaled "thinning function". Then, the authors proposed an estimate of the MMD between two distributions using weighted maximum mean discrepancy (MMD). The idea is in fact similar to an inverse probability weighting (IPW). They considered both when T(x) is known and when T(x) is unknown and must be estimated from the data. The proposed method was evaluated using both synthetic and real MNIST dataset. 

In brief, sample selection bias is generally a challenging problem in science, statistics, and machine learning, so the topic of this paper is interesting. Nevertheless, the motivation for investigating this problem specifically in MMD-GANs is not clear. What motivated you to study this problem specifically for GAN in the first place? How does solving this problem help us understand or solve the sample selection bias in general? Will it shed light on how to improve the stability of GAN? Also, the experiment results are too weak to make any justified conclusion.

Some comments and questions:

- How is sample selection bias related to the stability issue of training GAN? Does it worsen the stability?
- Have estimators in Eq. (2) and Eq. (3) been studied before? Are there any theoretical guarantees that this estimate will convergence to the true MMD? 
- On page 5, why T(men) = 1 and T(women) equals to the sample ratio of men to women in labeled subset?
- Can we use clustering to estimate the thinning function?
review_title: Unclear motivation and weak contribution!

Meta-review: The reviewers agree that the problem being addressed is interesting, however there are concerns with novelty and with the experimental results. An experiment beyond dealing with class imbalance would help strengthen this paper, as would experiments with other kinds of GANs.
Decision: Reject