Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
Given data from a canonical distribution P and data from distributions that are
independent transformations (mechanisms) applied on P, this paper aims to learn
1) those independent transformations; and 2) inverse transformations that map
data from transformed distributions to their corresponding canonical
distribution.

This is achieved by training a mixture of experts, where each expert is assumed to
model a single inverse transformation. Each expert can be seen as the generator
of a conditional GAN. The discriminator is trained to distinguish samples from
the canonical distribution P and those transformed distributions.

Experiments on MNIST data shows that in the end of training, each expert wins
almost all samples from one transformation and no other, which confirms that
each expert model a single inverse transformation.

Comments:
1) Besides samples from distributions that are results of applying independent
mechanisms, samples from the canonical distribution are also required to learn
the model. Are the samples from the canonical distribution always available in
practice? Since the canonical samples are needed for training, this problem 
setup seems not to be totally "unsupervised".

2) The authors only run experiments on the MNIST data, where 1) the mechanisms are
simulated and relatively simple, and 2) samples from the canonical distribution
are also available. Did the authors run experiments on other datasets?

3) This work seems to be related to the work on 1) disentangling factors of
variation; and 2) non-linear independent component analysis. Could the authors
add discussions to illustrate the difference between the proposed work and
those topics?

4) This work is motivated from the objective of causal inference, therefore it
might be helpful to add empirical results to show how the proposed method can
be used for causal inference.
review_title: Interesting problem setup, more empirical results and discussions on related works would improve the paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a framework to recover a set of independent mechanisms. In order to do so it uses a set of experts each one made out of a GAN.

My main concern with this work is that I don't see any mechanism in the framework that prevents an expert  (or few of them) to win all examples except its own learning capacities. p7 authors have also noticed that several experts fail to specialize and I bet that is the reason why.
Thus, authors should analyze how well we can have all/most experts specialize in a pool vs expert capacity/architecture.
It would also be great to integrate a direct regularization mechanism in the cost  in order to do so. Like for example a penalty in how many examples a expert has catched.

Moreover, the discrimator D  (which is trained to discriminate between real or fake examples) seems to be directly used to tell if an example is throw from the targeted distribution. It is not the same task. How D will handle an example far from fake or real ones ? Why will D answer negatively (or positively) on this example ? 



review_title: Review for Learning Independent Causal Mechanisms

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper describes a setting in which a system learns collections of inverse-mapping functions that transform altered inputs to their unaltered "canonical" counterparts, while only needing unassociated and separate sets of examples of each at training time.  Each inverse map is an "expert" E akin to a MoE expert, but instead of using a feed-forward gating on the input, an expert is selected (for training or inference) based on the value of a distribution-modeling function c applied to the output of all experts:  The expert with maximum value c(E(x)) is selected.  When c is an adversarially trained discriminator network, the experts learn to model the different transformations that map altered images back to unaltered ones.  This is demonstrated using MNIST with a small set of synthetic translations and noise.

The fact that these different inverse maps arise under these conditions is interesting --- and Figure 5 is quite convincing in showing how each expert generalizes.  However, I think the experimental conditions are very limited:  Only one collection of transformations is studied, and on MNIST digits only.  In particular, I found the fact that only one of ten transformations can be applied at a time (as opposed to a series of multiple transforms) to be restrictive.  This is touched on in the conclusion, but to me it seems fundamental, as any real-world new example will undergo significantly more complex processes with many different variables all applied at once.

Another direction I think would be interesting, is how few examples are needed in the canonical distribution?  For example, in MNIST, could the canonical distribution P be limited to just one example per digit (or just one example per mode / style of digit, e.g. "2" with loop, and without loop)?  The different handwriters of the digits, and sampling and scanning process, may themselves constitute in-the-wild transformations that might be inverted to single (or few) canonical examples --- Is this possible with this mechanism?

Overall, it is nice to see the different inverse maps arise naturally in this setting.  But I find the single setting limiting, and think the investigation could be pushed further into less restricted settings, a couple of which I mention above.



Other comments:

- c is first described to be any distribution model, e.g. the autoencoder described on p.5.  But it seems that using such a fixed, predefined c like the autoencoder may lead to collapse:  What is preventing an expert from learning a single constant mode that has high c value?  The adversarially trained c doesn't suffer from this, because presumably the discriminator will be able to learn the difference between a single constant mode output and the distribution P.  But if this is the case, it seems a critical part of the system, not a simple implementation choice as the text seems to say.

- The single-net baseline is good, but I'd like to get a clearer picture of its results.  p.8 says this didn't manage to "learn more than one inverse mechanism" --- Does that mean it learns to invert a single mechanism (that is always translates up, for example, when presented an image)?  Or that it learned some mix of transforms that didn't seem to generalize as well?  Or does it have some other behavior?  Also, I'm not entirely clear on how it was trained wrt c --- is argmax(c(E(x)) always just the single expert?  Is c also trained adversarially?  And if so, is the approximate identity initialization used?

review_title: limited setting, but exhibits interesting behavior

Meta-review: PROS:
1. All the reviewers thought that the work was interesting and showed promise
2. The paper is relatively well written

CONS:
1. Limited experimental evaluation (just MNIST)

The reviewers were all really on the fence about this but in the end felt that while the idea was a good one and the authors were responsive in their rebuttal, the experimental evaluation needed more work. 
Decision: Reject