Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies the challenges of disentangling independent factors of variation under weakly labeled data. 

A term "reference ambiguity" is introduced, which refers to the fact that there is no guarantee that two data points with same factor of variation will be mapped to the same point if there is only weakly labeled data to that extend. 

I am having a hard time understanding the message of the paper. The proof in section 3.1, although elementary, is nice. But then the authors train fairly standard networks in experiments (section 4) for datasets studied with these methods in the literature, and they fail to draw any connection to the introduced reference ambiguity concept. 

As written, the paper to me looks like two separate white papers: 
{beginning - to -end of section 3}: as a theoretical white paper that lacks experiments, and 
{section 4}: experiments with some recent methods / datasets (this part is almost like a cute course project). 

Either the paper lacks to harmonically present the over arching goal, or I have missed if there was any such message that was implicit in between the lines. A rewrite with strong connection between the theory and the experiments is required.
review_title: What is the overarching goal? There is lack of clarity between the theory parts and the experiments.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper considers the challenges of disentangling factors of variation in images: for example disentangling viewpoint from vehicle type in an image of a car. They identify a well-known problem, which they call "reference ambiguity", and show that in general without further assumptions one cannot tell apart two different factors of variation. 

They then go on to suggest an interesting AE+GAN architecture where the main novelty is the idea of taking triplets such that the first two instances vary in only one factor of variation, while the third instance varies in both from the pair. This is clever and allows them to try and disentangle the variation factors using a joint encoder-decoder architecture working on the triplet.

Pros:
1. Interesting use of constructed triplets.
2. Interesting use of GAN on the artificial instance named x_{3 \oplus 1}

Cons:
1. Lack of clarity: the paper is hard to follow at times. It's not entirely obvious how the theoretical part informs the practical part. See detailed comments below.
2. The theory addresses two widely recognized problems as if they're novel:  "reference ambiguity" and "shortcut problem". The second merely refers to the fact that unconstrained autoencoders will merely memorize the instance. 
3. Some of the architectural choices (the one derived from "shortcut problem") are barely explained or looked into.

Specific comments:

1. An important point regarding the reference ambiguity problem and eq. (2): a general bijective function mixing v and c would not have the two components as independent. The authors could have used this extremely important aspect of the generative process they posit in order to circumvent the problem of ambiguity. In fact, I suspect that this is what allows their method to succeed.

2. I think the intro could be made better if more concrete examples be made earlier on. Specifically the car-type/viewpoint example, along with noting what weak labels mean in that context.

3. In presenting autoencoders it is crucial to note that they are all built around the idea of compression. Otherwise, the perfect latent representation is z=x.

4. I would consider switching the order of sections 2 and 3, so the reader will be better grounded in what this paper is about before reading the related work.

5. In discussing attributes and "valid" features, I found the paper rather vague. An image has many attributes: the glint in the corner of a window, the hue of a leaf. The authors should be much more specific in this discussion and definite explicitly and clearly what they mean when they use these terms.

6. In equation (5), should it be p(v_1,v_2)? Or are v_1 and v_2 assumed to be independent? 

7. Under equation (5), the paper mentions an "autoencoder constraint". Such a constraint is not mentioned up to this point in the paper if I'm not mistaken. 

8. Also under equation (5): is this where the encoder requirements are defined? If so, please be more explicit about it. Also note that you should require c_1 \neq c_2. 

9. In proof of Proposition 1, there is discussion of N_c. N_c was mentioned before but never properly defined; same for R_c and C^-1. These should be part of the proposition statement or defined formally. Currently they are only discussed ad-hoc after equation (5). 

10 .In the proof of Proposition 1, what is f_c^-1 ? It's only defined later in the paper.

11. In general, what promises that f_c^-1 and f_v^-1 are well defined? Are f_c and f_v injective? Why? 

12. Before explaining the training of the model, the task should be defined properly. What is the goal of the training? 

13. In eq. (15) I am missing a term which addresses "the shortcut problem" as defined in the previous page.

14. The weak labels are never properly defined and are discussed in a vague manner. Please define what does that term mean in your context and what were the weak labels in each experiment.

15. In the conclusion, I would edit to say the "our trained model works well on *several* datasets". 


Minor comments:
Please use \citep when appropriate. Instead of "Generative Adversarial Nets Goodfellow et al. (2014)", you should have "Generative Adversarial Nets (Goodfellow et al., 2014)"
review_title: The paper offers some interesting ideas. However, the presentation is somewhat confusing and the resulting architecture does not seem justified by the theory

final_score: rating: 6: Marginally above acceptance threshold
review_content: Quality
The method description, particularly about reference ambiguity, I found difficult to follow. The experiments and analysis look solid, although it would be nice to see experiments on more challenging natural image datasets. 

Clarity
“In general this is not possible… “ - you are saying it is not possible to learn an encoder that recovers disentangled factors of variation? But that seems to be one of the main goals of the paper. It is not clear at all what is meant here or what the key problem is, which detracts from the paper’s motivation.

What is the purpose of R_v and R_c in eq 2? Why can these not be collapsed into the encoders N_v and N_c?

What does “different common factor” mean?

What is f_c in proof of proposition 1? Previously f (no subscript) was referred to as a rendering engine.

T(v,c) ~ p_v and c ~ p_c are said to be independent. But T(v,c) is explicitly defined in terms of c (equation 6). So which is correct?

Overall the argument seems plausible - pairs of images in which a single factor of variation changes have a reference ambiguity - but the details are unclear.

Originality

The model is very similar to Mathieu et al, although using image pairs rather than category labels directly. The idea of weakly-supervised disentangling has also been explored in many other papers, e.g. “Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis”, Yang et al. The description of reference ambiguity seems new and potentially valuable, but I did not find it easy to follow.

Significance

Disentangling factors of variation with weak supervision is an important problem, and this paper makes a modest advance in terms of the model and potentially in terms of the theory. The analysis in figure 3 I found particularly interesting - illustrating that the encoder embedding dimension can have a drastic effect on the shortcut problem. Overall I think this can be a significant contribution if the exposition can be improved.

Pros
- Proposed method allows disentangling two factors of variation given a training set of image pairs with one factor of variation matching and the other non-matching.
- A challenge inherent to weakly supervised disentangling called reference ambiguity is described.

Cons
- Only two factors of variation are studied, and the datasets are fairly simple.
- The method description and the description of reference ambiguity are unclear.
review_title: Progress on disentangling; could be written more clearly.

Meta-review: The paper proposes a method to disentangle style from content (two factor disentanglement) using weak labels (information about the common factor for a pair of images). It is similar to an earlier work by Mathieu et al (2016) with main novelty being in the use of the discriminator which operates with pairs of images in the proposed method. Authors also have some theoretical statements about two challenges in disentangling the factors but reviewers have complained about missing connection b/w theory and experiments, and about exposition in general.

The idea has novelty, although somewhat limited in the light of earlier work by Mathieu et al (2016)), and theoretical statements are also of interest but reviewers still feel the paper needs improvement in writing and presentation of results. I would recommend an invitation to the workshop track. 
Decision: Invite to Workshop Track