Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: 
The paper tries to bring some theoretical foundation to the weakly supervised disentanglement. Overall it is a good contribution, but the message of the paper is not clear.  The authors propose two notions: consistency and restrictiveness, which they don't imply each other. However, the experiment on real data shows that they are highly correlated.   Up until the experiment section, the paper is well written (although a bit verbose). It seems that it is great but unfinished work.

The paper is well written, but in my opinion, there is too much verbosity on page 4-5 on rather trivial definitions consistency and restrictiveness and a big box in the calculus of disentanglement that steals space from the main results.  In my opinion, those sections can be reduced so that other theorem can be covered. In my opinion, the theorem nine should be part of the main text. 

I understand the definition of "Sufficiency for Disentanglement " but it is not clear why it is important. Sure, it is a strong definition that says for any  $\mathcal{H}$ (and not a subset) the algorithm ($\mathcal{A}$) should be able to match the distribution of the observation but why is it a big deal according to the next paragraph?

I don't see any proof that Eq.11 should be between [0,1]. Yes, g is optimal, and if you enter suboptimal values to it, one expects the nominator to be less than dominator. However, g a function that is optimal in expectation, which does not mean for every s value it nominator is less than the denominator. In fact, some of the values in fig 3 are small negatives.

Fig 3 is not explained well: you are showing normalized consistency and restiveness. First of all, what is the dataset you tried this on? Second, why some values are negative?! These are supposed to be between [0,1]. Third, what is the take-home-message of this figure? the first two matrices from left show that the factors are consistent b/c they are almost diagonal. The third one from left shows that the algorithm you used is not restrictive? Then are you suggesting this as a metric of evaluation? I am not sure I understand the first figure from the right.
Overall, the authors perform a significant amount of experiments, but they did a poor job in summarizing the results.
Finally, the authors claim 
                  "...We believe this correlation between consistency and restrictiveness to have been a general source of confusion in the disentanglement literature, causing many to either observe or believe that restricted labeling or share pairing on $S_i$ (which only guarantees consistency) is sufficient for disentangling Si ..." 

Each of those methods should be analyzed separately to ensure that their algorithms do not induce restiveness. I just don't see the natural connection between your figure 4 and this conclusion that you made.

Minor:
Where is the proof for Theorem 1? In the Supp, it starts with Theorem 8, I guess you meant Lemma 8? You need to clean up the Supp so that one can find the proof easily. I suggest restructuring the Supp to less and finally proof of Thorem 1.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper first discusses some concepts related to disentanglement. The authors propose to decompose disentanglement into two distinct concepts: consistency and restrictiveness. Then, a calculus of disentanglement is introduced to reveal the relationship between restrictiveness and consistency. The proposed concepts are applied to analyze weak supervision methods. 

This paper is well structured. The presentation is easy to follow. The problem discussed is important to the machine learning community. The concepts discussed are supported by a large number of experiments. 

The assumption that disentanglement can be decomposed into consistency and restrictiveness might be flawed. Let us consider a generator $g(Z)$  that always generates the same image for all $Z \sim p(Z)$. Note that $g(Z)$ gives perfect consistency and perfect restrictiveness as defined in Equation (3) and (6). However, we consider $g(Z)$ is a bad generator, and we do not think the corresponding latent representation $Z$ achieves perfect disentanglement. Note that such $Z$, in general, gives low values in the existing disentanglement metrics.

This implies that we might need to introduce a third component to disentanglement, which I call it relevance. We should additionally assume that different $z_{ \setminus I}$ leads to different generated images. It might be challenging to measure relevance quantitatively under the probabilistic framework, but I believe this is necessary. 

In summary, I think the idea presented is interesting and useful. I believe this paper is promising and impactful after proper revision.  However, I do not recommend acceptance because it looks technically flawed. 

Minor:
In Figure 5, the illustration is clear to me, but I am not sure how the vertical axis simultaneously represents two variables $z_2, z_3$.
In the table on page 5, $n$ represents the number of dimensions, right?


review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary

The paper tries to construct a theoretical framework to rigorously analyze the disentanglement guarantees of weak supervision algorithms. In particular, it focuses on two concepts, consistency and restrictiveness which provides a formalism that precisely distinguishes when disentanglement arises from supervision versus model inductive bias. 

Strengths

The framework uses two simple concepts, consistency and restrictiveness for both generator and decoder. It also gives rise to a calculus. It is very useful to demonstrate the conditions under which various supervision strategies guarantee disentanglement. 

The paper also did a good job clarifying how consistency and restrictiveness differ from other disentanglement concepts used in the literature. 

Weaknesses

The paper does not propose effective methods for disentanglement in the weak supervision setting. 

The experimental section uses very toy datasets. It is not clear how the weak supervision signal can come from in realistic applications.
review_title: Official Blind Review #3

Meta-review: This paper first discusses some concepts related to disentanglement. The authors propose to decompose disentanglement into two distinct concepts: consistency and restrictiveness. Then, a calculus of disentanglement is introduced to reveal the relationship between restrictiveness and consistency. The proposed concepts are applied to analyze weak supervision methods. 

The reviewers ultimately decided this paper is well-written and has content which is of general interest to the ICLR community.
Decision: Accept (Poster)