Weakly Supervised Disentanglement with Guarantees
Authors: Rui Shu
Yining Chen
Abhishek Kumar
Stefano Ermon
Ben Poole
Venue:  (2020)
Keywords: disentanglement
theory of disentanglement
representation learning
generative models

Abstract: Learning disentangled representations that correspond to factors of variation in real-world data is critical to interpretable and human-controllable machine learning. Recently, concerns about the viability of learning disentangled representations in a purely unsupervised manner has spurred a shift toward the incorporation of weak supervision. However, there is currently no formalism that identifies when and how weak supervision will guarantee disentanglement. To address this issue, we provide a theoretical framework to assist in analyzing the disentanglement guarantees (or lack thereof) conferred by weak supervision when coupled with learning algorithms based on distribution matching. We empirically verify the guarantees and limitations of several weak supervision methods (restricted labeling, match-pairing, and rank-pairing), demonstrating the predictive power and usefulness of our theoretical framework.

Published as a conference paper at ICLR 2020
WEAKLY SUPERVISED DISENTANGLEMENT WITH GUARANTEES
Rui Shu'; Yining Chen', Abhishek Kumari, Stefano Ermon & Ben Poole Stanford University, *Google Brain "fruishu, cynnjjs ermon) ) e @stanford.... edu "aabiishk poole gooleb]@gh begogleeeeel......ooooo com
ABSTRACT
Learning disentangled representations that correspond to factors of variation in real-world data critical interpretable and human-controllable machine learn- ing. Recently, concerns about the viability of learning disentangled representa- tions in a purely unsupervised manner has spurred shift toward the incorporation of weak supervision. However, there is currently no formalism that identifies when and how weak supervision will guarantee disentanglement. To address this issue, we provide theoretical framework to assist in analyzing the disentangle- ment guarantees (or lack thereof) conferred by weak supervision when coupled with learning algorithms based on distribution matching. We empirically verify the guarantees and limitations of several weak supervision methods (restricted la- beling, match-pairing, and rank maank-airing), demonstrating the predictive power and usefulness of our theoretical framework.
INTRODUCTION
Many real-world datasets can be intuitively described via a data-generating process that first sam- ples an underlying set of interpretable factors, and chen-cooditional on those factors- generates an observed data point. For example, in image generation, one might first sample the object identity and pose, and then render an image with the object the correct pose. The goal of disentan- gled representation learning is to learn representation where each dimension of the representation corresponds to a distinct factor of variation in the dataset (Bengio et al., 2013). Learning such rep- resentations that align with the underlying factors of variation may be critical to the development of machine learning models that are explainable or human-controllable (Gilpin et al., 2018; Lee et al., 2019; Klys et al., 2018).
In recent years, disentanglement research has focused on the learning of such representations in an unsupervised fashion, using only independent samples from the data distribution without access to the true factors of variation (Higgins et al., 2017; Chen et al., 201 Kim & Mnih, 2018; 80;;; Esmaeili al., 2018). However, Locatello et al. (2019) demonstrated that many existing methods for the unsupervised learning of disentangled representations are brittle, requiring careful supervision-basee hyperparameter tuning. To build robust disentangled representation learning methods that do not require large amounts of supervised data, recent work has turned to forms of weak supervision (Chen& & Batmanghelich, 2019; Gabbay Hoshen, 2019). Weak supervision can allow one to build models that have interpretable representations even when human labeling is challenging (e.g., hair style in face generation, or style in music generation). While existing methods based on weakly- supervised learning demonstrate empirical gains, there is existing formalism for describing the theoretical guarantees conferred by different forms of weak supervision (Kulkarni et al., 2015; Reed et al., 2015; Bouchacourt et al., 2018).
In this paper, we present a comprchensive theoretical framework for weakly supervised disentangle- ment, and evaluate our framework on several datasets. Our contributions several-fold.
1. We formalize weakly-supervise learning as distribution matching in an extended space.
Work done during an internship Google Brain.

---

Published as a conference paper at ICLR 2020
2. We propose a set of definitions for disentanglement that can handle correlated factors and are inspired by many existing definitions in the literature (Higgins et al., 2018; Suter et al,, 2018; Ridgeway & Mozer, 2018).
3. Using these definitions, we provide a conceptually useful and theoretically rigorous calcu- lus of disentanglement.
4. We apply our theoretical framework of disentanglement to analyze three notable classes weak supervision methods (restricted labeling, match pairing, and rank pairing). We show that although certain weak supervision methods (e.g., style-labeling in style-content disentanglement) do not guarantee disentanglement, our calculus can determine whether disentanglement is guaranteed when multiple sources of weak supervision are combined.
5. Finally, we perform extensive experiments to systematically and empirically verify our predicted guarantees."
FROM UNSUPERVISED TO WEAKLY SUPERVISED DISTRIBUTION MATCHING
Our goal in disentangled representation learning to identify latent-variable generative model whose latent variables correspond to ground truth factors of variation in the data. To identify the role that weak supervision plays in providing guarantees on disentanglement, we first formalize the model families we are considering, the forms of weak supervision, and finally the metrics we will use to evaluate and prove components disentanglement.
We consider data-generating, processes where S E IR" are the factors of variation, with distribution (s)),, and X I is the observed data point which deterministic function of S, i.e., X (S)). Many existing algorithms in unsupervised learning of disentangled representations aim to
learn a latent-variable model with prior p(2) and generator where 9(2) g*().. However, simply matching the marginal distribution over data not enough: the learned latent variables Z and the true generating factors S could still be entangled with each other (Locatello et al., 2019).
To address the failures of unsupervised learning of disentangled representations, we leverage weak supervision, where information about the data-generating process is conveyed through additional observations. By performing distribution matching on an augmented space (instead of just on the observation X), provide guarantees learned representations.
Unsupervised p(x)
Restricted Labeling p(x,5y)
Match Pairing x(x,x')
Rank Pairing x(x,x.>)
Figure 1: Augmented data distributions derived from weak supervision. Shaded nodes denote ob- served quantities, and unshaded nodes represent unobserved (latent) variables.
We consider three practical forms of weak supervision: restricted labeling, match pairing, and rank pairing. All of these forms of supervision be thought of augmented forms of the original joint distribution, where we partition the latent variables in two 5 = (S1,Si, and either observe a subset of the latent variables or share latents between multiple samples. visualization of these augmented distributions is presented in Figure 1, and below we detail each form of weak supervision.
In restricted labeling, we observe subset of the ground truth factors, S1 in addition X. This allows us to perform distribution matching on p" x), the joint distribution over data and observed factors, instead of just the data, p" (1x, as in unsupervised learning. This form of supervision is often leveraged in style-conten disentanglement, where labels are available for content but not style (Kingma et al., 2014; Narayanaswamy et al., 2017; Chen et al., 2018b; Gabbay & Hoshen, 2019).
"ooee available ottp://siiccnoooogeccooooooeoees disentangle
2

---

Published as a conference paper at ICLR 2020
Match Pairing uses paired data, (x,z²) that share values for known subset of factors, I. For many data modalities, factors variation may be difficult to explicitly label. Instead, it may be easier to collect pairs of samples that share the same underlying factor (e.g., collecting pairs of images of different people wearing the same glasses is easier than defining labels for style of glasses). Match longer
pairing is weaker form of supervision than restricted labeling, the learning algorithm depends on the underlying value 3I, and only on the indices of shared factors L. Several variants of match pairing have appeared in the literature (Kulkarni al., 2015; Bouchacourt et al., 2018; Ridgeway & Mozer, 2018), but typically focus on groups of observations in contrast to the paired setting we consider in this paper.
Rank Pairing is another form of paired data generation where the pairs (z,z") are generated in an i..dd fashion, and an additional indicator variable y is observed that determines whether the cor- responding latent is greater than s/: when is easier to compare two samples with respect to an underlying factor than directly col- lect labels (e.g., comparing two object sizes versus providing ruler measurement of an object). Although supervision via ranking features prominently in the metric learning literature (McFee & Lanckriet, 2010; Wang et al., 2014), our focus in this paper will be rank pairing in the context of disentanglement guarantees.
{s: 2 s/}- Such form of supervision is effective
For each form of weak supervision,
train generative models with the structure in Figure 1, using data sampled from the ground truth model and a distribution matching objective. For example, for match pairing, train generative model (p(=). g) such that the paired random variable (g(Z,, Zu,), (41),g(Z, (Zr, Z())) from the generator matches the distribution of the corresponding paired random variable g")
from the augmented data distribution.
3 DEFINING DISENTANGLEMENT
To identify the role that weak supervision plays in providing guarantees on disentanglement, we introduce set of definitions that consistent with intuitions about what constitutes "disen- tanglement" and amenable to theoretical analysis. Our new definitions decompose disentanglement into two distinct concepts: consistency and restrictiveness. Different forms of weak supervision can enable consistency or restrictiveness on subsets of factors, and Section 4 we build up a cal- culus of disentanglement from these primitives. We discuss the relationship prior definitions of disentangiement in Appendix A.
3.1
DECOMPOSING DISENTANGLEMENT INTO CONSISTENCY AND RESTRICTIVENESS
A
O
O
O
!
Disentanglement
(b) Consistency
(c) Restrictiveness
Figure 2: Illustration of disentanglement, consistency, and restrictiveness of 21 with respect to the factor of variation size. Each image of shape represents the decoding g(21.) by the generative model. Each column denotes a fixed choice of 21. Each row denotes a fixed choice of (22, #3). A demonstration of consistency versus restrictiveness models from disent www.nnangglmeme....mmm. is available in Appendix B.
To ground our discussion of disentanglement, we consider an oracle that generates shapes with factors of variation for size (S1), shape (S2), and color (Sg). How can we determine whether Z, of our generative model "disentangles" the concept of size? Intuitively, one way to check whether Z, of the generative model disentangles size (S₁) is to visually inspect what happens as vary Zı, Zz, and Zg, and see whether the resulting visualizations are consistent with Figure 2a. In doing so, visual inspection checks for two properties:
3

---

iid Published as a conference paper at ICLR 2020
1. When Z1 is fixed, the size (S1) of the generated object never changes.
2. When only 21 is changed, the change is restricted to the size (Si) of the generated object, meaning that there is no change in S for + 1.
We argue that disentanglement decomposes into these two properties, which we refer to as generator consistency and generator restrictiveness. Next, we formalize these two properties.
Let H. be a hypothesis class of generative models from which we assume the true data gateraennr function drawn. Each element of the hypothesis class H tuple (p(s), 9, g)e, where p(s) de- scribes the distribution over factors of variation, the generator g is function that maps from the factor space S IR. to the observation space X' E IR"., and the encoder e is function that maps from X' + S. S and X can consist of both discrete and continuous random variables. We impose a few mild assumptions on H. (see Appendix I.1), Notably, we assume every factor of variation is exactly recoverable from the observation X, i.e. e(g(S)) S.
Given an oracle model h" = "p"
,ee) E H. we would like to learn a model h = (p. e H whose latent variables disentangle the latent variables in h². We refer to the latent-variables in the oracle ht as S and the alternative model h's latent variables as Z. If we further restrict h to only those models where g(Z) g5)) are equal in distribution, it is natural to align Z and S via S e og(Z), Under this relation between and S, our goal construct definitions that describe whether the latent code Z, disentangles the corresponding factor S,.
Generator Consistency. Let denote set of indices and P1 denote the generating process
21 ~ p(21)
Ar 2 ~ p(a,s 21).
(2)
This generating process samples Z₁ once and then conditionally samples Z, twice in i.i.d. fash- ion, We say that Z is consistent with Sy if
Ep, lle g(21, 2x)) et o x 2
where e7 is the oracle encoder restricted to the indices I.
Intuitively, Equation (3) states that, for any fixed choice of Z,, resampling of Z will not influence the oracle's measurement of the factors Sy. In other words, S₁ is invariant to changes in Z,, An illustration of a generative model where Z₁ is consistent with size (S₁) is provided in Figure 2b. A notable property of our definition is that the prescribed sampling process Ps does not require the underlying factors of variation to be statistically independent. We characterize this property in contrast to previous definitions of disentanglement in Appendix A.
Generator Restrictiveness. Let p/z denote the generating process
21 p(2,))
(4)
21. 2/ p(2l zx1)-
(5)
We say that 7, is restricted to Sy if
Ep, |le o g(2)) v1) ei, o( 2²)
(6)
Equation (6) states that, for any fixed choice of Z resampling of Z, will not influence the oracle's measurement of the factors S. In other words, S. is invariant to changes in Z,. Thus, changing Z₁ is restricted to modifying only Sj. An illustration of a generative model where Z1 is restricted to size (S1) is provided in Figure 2c.
Generator Disentanglement. We now say that Z, disentangles S₁ if Z, is consistent with and restricted to S,- If we denote consistency and restrictiveness via Boolean functions C(T) and R(T), can now concisely state that
= ==)==(II) A R(T).
(7)
where D(I) denotes whether Z, disentangles S₁- An illustration of a generative model where Z disentangles size (S1) is provided in Figure 2a. Note that while size increases monotonically with Z, the schematic figure, we wish to clarify that monotonicity is unrelated to the concepts of consistency and restrictiveness.
4

---

Published as a conference paper at ICLR 2020
3.2 RELATION TO BUECTIVITY-BASE DEFINITION OF DISENTANGLEMENT
Under mild assumptions H. distribution matching on g = g(S) combined with generator disentangiement on factor I implies the existence of two invertible functions fi and f such that the alignment via 5 = e" (Z) decomposes into
Si fi(Z)) (1(2,1)
(8)
5.
This expression highlights the connection between disentanglement and invariance, whereby S₁ only influenced by Z,, and Sn is only influenced by Zi,- However, such bijectivity -east definition of disentanglement does not naturally expose the underlying primitives of consistency and restrictiveness, which shall demonstrate in theory and experiments to be valuable concepts for describing disentanglement guarantees under weak supervision.
3.3 SECODER-BASHI DEFINITIONS FOR DISENTANGLEMENT
Our proposed definitions are Asymmetrie-mmcsurring the behavior of a generative model against an oracle encoder. So far, we have chosen to present the definitions from the perspective of learned generator (p. g) measured against an oracle encoder e". In this sense, they are generator- bbased definitions. We can also develop a parallel set of definitions for encoder-based consistency, restric- tiveness, and disentanglement within our framework simply by using an oracle generator (p",g" measured against learned encoder Below, present the encoder-based perspective consis- tency.
Encoder Consistency. Let pj denote the generating process
81 p" (s1)
(9)
8,7, 6/
ud p" (s))
(10)
This generating process samples S) once and then conditionally samples Sy twice in an i.i.d fashion. We say that S, is consistent with Z, if
Exp llle o g (""(sr, - er o" (())) 0.
(11)
We now make two important observations. First, valuable trait of our encoder-basee definitions that one can check for encoder consistency restrictiveness disentanglement as long as one has access to match pairing data from the oracle generator. This is in contrast to the existing disentanglement definitions and metrics, which require access to the ground truth factors (Higgins al., 2017; Kumar et al., 2018; Kim & Mnih, 2018; Chen et al., 2018a; Suter et al., 2018; Ridgeway & Mozer, 2018; Eastwood & Williams, 2018). The ability to check for our definitions supervised fashion is the key why we can develop theoretical framework using the language of consistency and restrictiveness. Second, encoder-basee based definitions are tractable to measure when testing on synthetic data, since the synthetic data directly serves the role of the oracle generator. As such, while we develop our theory to guarantee both generator-based and the encoder-based disentanglement, all of our measurements the experiments will be conducted with respect leamndd encoder.
weakly
We make three remarks on notations. First, D(1) D(fi)). Second, D(2) evaluates to true.
Finally, D(T) is implicitly dependent either (p.g, e²) (generator-based) or (p" .9", ,e) (encoder- based). Where important, we shall make this dependency explicit (e-g., let D(T ; P. e.,") denote generator-based disentanglement). We apply these conventions C and R analogously.
4 A CALCULUS OF DISENTANGLEMENT
There are several interesting relationships between restrictiveness and consistency. First, by defini- tion, C(J) is equivalent to R). Second, we can see from Figures 2b and 2c that C(I) and R(T) do not imply each other. Based on these observations and given that consistency and restrictiveness operate over subsets the random variables, a natural question that arises is whether consistency or restrictiveness certain sets of variables imply additional properties over other sets of variables.
5

---

Published as a conference paper at ICLR 2020
We develop calculus for discovering implied relationships between learned latent variables Z and ground truth factors of variation given known relationships as follows.
Calculus of Disentanglement
Consistency and Restrictiveness
C(T) + R(T)
R(T) + C(T)
C(T) - - R(V)
Union Rules
C(T) A C(J) = c(UU)
R(T) A R(J) = R(TUJ)
Intersection Rules C(T) A C(J) C(In))
R(I) A R(J) = B(NN))
Full Disentanglement Al=1 C(1) - A D(1)
AL R D(1)
N
Our calculus provides theoretically rigorous procedure for reasoning about disentanglement. In particular, it is no longer necessary to prove whether the supervision method interest satisfies con- sistency and restrictiveness for each and every factor. Instead, it suffices to show that a supervision method guarantees consistency or restrictiveness for subset of factors, and then combine multiple supervision methods via the calculus guarantee full disentanglement, We can additionally use the calculus to uncover consistency or restrictiveness on individual factors when weak supervision is available only for subset of variables. For example, achieving consistency plies consistency on the intersection Sz. Furthermore, we note that these rules are agnostic to using generator or encoder- based definitions. We defer the complete proofs Appendix 1.2.
S1.2 and S im-
FORMALIZING WEAK SUPERVISION WITH GUARANTEES
In this section, address the question of whether disentanglement arises from the supervision method or model inductive bias. This challenge was first put forth by Locatello et al. (2019), who noted that unsupervised disentanglement heavily reliant on model inductive bias. As we transition toward supervised approaches, it is crucial that we formalize what it means for disentanglement to be guaranteed by weak supervision.
Sufficiency for Disentanglement. Let P denote family of augmented distributions. We say that a weak supervision method S : H. is sufficient for learning a generator whose latent codes Z disentangle the factors S, if there exists learning algorithm A P + H. such that for any choice of (p"(s) ,e") E H. the procedure
S(p" (s). ee*) returns a model (p(z), 9, e) for which both D(T:P,9,e") and D(I; pp, ,,", hold, and (Z) d (S).
The key insight of this definition is that we force the strategy and learning algorithm pair (S, A) handle all possible oracles drawn from the hypothesis class H. This prevents the exploitation of model inductive bias, since any bias from the learning algorithm A toward reduced hypothesis class F. c H will result in failure to handle oracles in the complementary hypothesis class H. \ H.
The distribution matching requirement ( (S) ensures latent code informativeness. i.e., preventing trivial solutions where the latent code is uninformative (see Proposition 6 for formal statement). Intuitively, distribution matching paired with a deterministic generator guarantees in- vertibility the learned generator and encoder, enforcing that Z, cannot encode less information than S, (e.g., only encoding age group instead of numerical age) and vice versa.
ANALYSIS OF WEAK SUPERVISION METHODS
We now apply our theoretical framework to three practical weak supervision methods: restricted labeling, match pairing, and rank pairing. Our main theoretical findings that: (1) these methods can be applied in a targeted manner to provide single factor consistency or restrictiveness guarantees; (2) by enforcing consistency (or restrictiveness) all factors, can learn models with strong

---

Published as a conference paper at ICLR 2020
disentanglement performance. Correspondingly, Figure and Figure 5 are our main experimental results, demonstrating that these theoretical guarantees have predictive power in practice.
6.1 THEORETICAL GUARANTEES FROM WEAK SUPERVISION
We prove that if a training algorithm successfully matches the generated distribution data dis- tribution generated via restricted labeling, match pairing, or rank pairing of factors S₁, then Z₁ is guaranteed be consistent with Sy:
Theorem 1. Given any oracle (p" (s).g", ,e)) E H. consider the distribation-matching algorithm A that selects a model (p(z).g, e) E H. such that:
(Restricte 1. (0* (g(2) 21) (Restricted Labeling): or
_ (9(2), (1),, (Match Pairing);
2.
3. (g* (S'), 1 (S,
S S11) (g(Z), ((')),((z)) 1(2 2 Z7}) (Rank Pairing).
Then (p. g) satisfies C(T: P. e*) and e satisfies C( ;"""""""""""" ,g,")."
Theorem states that distribution-matchee of S, guarantees both generator and encoder consistency for the learned generator and encoder respectively. We note that while the complement rule C(T) = (()7) further guarantees that Z restricted to S. we can prove that the same supervision does not guarantee that Zr is restricted to Sy (Theorem 2). However, if we additionally have restricted labeling for St. or match pairing for S then can see from the calculus that we will have guaranteed R((T) A C()), thus implying disentanglement of factor T. We also note that while restricted labeling and match pairing can be applied on set of factors at once (i.e. [7] 2 1), rank pairing is restricted to one-dimensional factors for which an ordering exists. In the experiments below, we empirically verify the theoretical guarantees provided in Theorem 1.
under restricted labeling, match pairing, or rank pairing
6.2 EXPERIMENTS
We conducted experiments on five prominent datasets the disentanglement literature: Shapes3D (Kim & Mnih, 2018), dSprites (Higgins al., 2017), Scream-dSprites (Locatello al., 2019), Small- NORB (LeCun et al., 2004), and Cars3D (Reed et al., 2015). Since some of the underlying factors are treated as nuisance variables in SmallNORB and Scream-dSprites, we show in Appendix C that our theoretical framework can be easily adapted accordingly to handle such situations. We use gener- ative adversarial networks (GANs, Goodfellow et al. (2014)) for learning (p, g) but any distribution matching algorithm (e.g.. maximum likelihood training in tractable models, or VI in latent-variable models) could be applied. Our results are collected over a broad range of hyperparameter configu- rations (see Appendix H for details).
Since existing quantitative metrics of disentanglement all measure the performance of an encoder with respect to the true data generator, we trained an encoder post-hoc to approximately invert the learned generator, and measured all quantitative metrics (e.g., mutual information gap) on the encoder. Our theory assumes that the learned generator must be invertible. While this is not true for conventional GANs, our empirical results show that this is not an issue in practice (see Appendix G).
We present three sets of experimental results: (1) Single-factor experiments, where we show that our theory can be applied in targeted fashion to guarantee consistency or restrictiveness of a sin- gle factor. (2) Consistency versus restrictiveness experiments, where we show the extent which single-factor consistency and restrictiveness are correlated even when the models are only trained maximize one the other. (3) Full disentanglement experiments, where we apply our theory to fully disentangle all factors. A more extensive set of experiments can be found in the Appendix.
6.2.1 SINGLE-FACTOR CONSISTENCY AND RESTRICTIVENESS
We empirically verify that single- factor consistency or restrictiveness can be achieved with the su- pervision methods of interest. Note there two special cases of match pairing: one where S is
7

---

Published as a conference paper at ICLR 2020
AMC
Figure 3: Heatmap visualization of ablation studies that measure either single-factor consistency or single-factor restrictiveness as a function of various supervision methods, conducted on Shapes3D. Our theory predicts the diagonal components to achieve the highest scores. Note that share pairing, change pairing, and change pair intersection are special cases of match pairing.
the only factor that is shared between and x' and one where S; is the only factor that is changed. We distinguish these two conditions as share pairing and change pairing, respectively. Theorem shows that restricted labeling, share pairing, and rank pairing of the 7th factor are each sufficient su- pervision strategies for guaranteeing consistency on St. Change pairing at St is equivalent to share pairing at S the complement rule C(1) + R(\1) allows us to conclude that change pairing guarantees restrictiveness. The first four heatmaps in Figure 3 show the results for restricted label- ing, share pairing, change pairing, and rank pairing. The numbers shown in the heatmap the normalized consistency and restrictiveness scores. We define the normalized consistency score as
ट(I : :p".pp"' e) = Ep; ller o 9" (5I, s.)) - er g (s1, x
(12)
2, 2 "'' le, o 9" (s) es o (s')ll
This score is bounded on the interval [o, 1] (a consequence of Lemma 1) and is maximal when C(I; pp: p".""""""""" .g,,e) )) is satisfied. This normalization procedure is similar in spirit to the Interventional Robustness score in Suter et al. (2018). The normalized restrictiveness score F can be analogously defined. In practice, we estimate this score via Monte Carlo estimation.
The final heatmap in Figure demonstrates the calculus of intersection. In practice, it may be easier to acquire paired data where multiple factors change simultancously. If we have access to two kinds of datasets, one where S₁ are changed and one where S, are changed, our calculus predicts that training on both datasets will guarantee restrictiveness on SInJ. The final heatmap shows six such intersection settings and measures the normalized restrictiveness score; in all but one setting. the results are consistent with our theory. We show in Figure 7 that this inconsistency is attributable to due to sensitivity to a specific hyperparameter.
the failure of the GAN to distribution-
6.2.2 CONSISTENCY VERSUS RESTRICTIVENESS
Figure& 4: Correlation plot and scatterplots demonstrating the empirical relationship between e(i) and Fi) across all 864 models trained on Shapes3D.
We now determine the extent to which consistency and restrictiveness are correlated in practice. In Figure 4, we collected all 864 Shapes3D models that we trained in Section 6.2.1 and measured the consistency and restrictiveness of each model on each factor, providing both the correlation plot and scatterplots of č(i) versus F(i). Since the models trained in Section 6.2.1 only ever targeted the consistency or restrictiveness of a single factor, and since our calculus demonstrates that consistency and restrictiveness do not imply each other, might priori expect find correlation in Figure 4. Our results show that the correlation is actually quite strong. Since this correlation is not guaranteed by our choice of weak supervision, it is necessarily consequence of model inductive
8

---

Published as a conference paper at ICLR 2020
bias. We believe this correlation between consistency and restrictiveness have been general source of confusion in the disentanglement literature, causing many to either observe or believe that restricted labeling or share pairing on S; (which only guarantees consistency) is sufficient for disentangling S; (Kingma et al., 2014; Chen & Batmanghelich, 2019; Gabbay & Hoshen, 2019; Narayanaswamy et al., 2017). It remains an open question why consistency and restrictiveness so strongly correlated when training existing models on real-world data.
6.2.3 FULL DISENTANGLEMENT
Sramb
Figure 5: Disentanglement performance of a vanilla GAN, share pairing GAN, change pairing GAN, rank pairing GAN, and fully-labeled GAN, measured by the mutual information gap across sev- eral datasets. comprehensive set of performance evaluations on existing disentanglement metrics is available in Figure 13.
If we have access to share / change / lank-pairing data for each factor, our calculus states that is possible to guarantee full disentanglement. We trained our generative model on either complete share pairing, complete change pairing, or complete rank pairing, and measured disentanglemen performance via the discretized mutual information gap (Chen et al., 201 20;8;; Locatello et al., 2019). As negative and positive controls, we also show the performance fully- superied GAN where the latents fixed to the ground truth factors of variation. Our results in Figure 5 empirically verify that combining single-factor weak supervision datasets leads to consistently high disentanglement scores.
an unsupervised GAN and
CONCLUSION
In this work, construct theoretical framework to rigorously analyze the disentanglement guar- antees of weak supervision algorithms. Our paper clarifies several important concepts, such as con- sistency and restrictiveness, that have been hitherto confused or overlooked in the existing literature, and provides a formalism that precisely distinguishes when disentanglement arises from supervi- sion versus model inductive bias. Through our theory and comprehensive set of experiments, we demonstrated the conditions under which various supervision strategies guarantee disentanglement. Our work establishes several promising directions for future research. First, we hope that our for- malism and experiments inspire greater theoretical and scientific scrutiny of the inductive biases present in existing models. Second, we encourage the search for other learning algorithms (besides distribution-matchingg that may have theoretical guarantees when paired with the right form of su- pervision. Finally, we hope that our framework enables the theoretical analysis of other promising weak supervision methods.
ACKNOWLEDGMENTS
We would like to thank James Brofos and Honglin Yuan for their insightful discussions on the theoretical analysis in this paper, and Aditya Grover and Hung H. Bui for their helpful feedback during the course of this project.
9

---

Published as a conference paper at ICLR 2020
REFERENCES
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 3788)11788-1828, 2013.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. Thirty-Second AAAI Con- ference Artificial Intelligence, 2018.
Junxiang Chen and Kayhan Batmanghelich. Weakly supervised disentanglement by pairwise simi- larities. arXiv preprint arXiv: 190600000,, 2019.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. Advances in Neural Information Processing Systems, pp. 2610-2620, 2018a.
Yutian Chen, Yannis Assael, Brendan Shillingford, David Budden, Scott Reed, Heiga Zen, Quan Wang, Luis Cobo, Andrew Trask, Ben Laurie, et al. Sample efficient adaptive text-to-speech. arXiv preprint
2018b.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disen- tangled representations. ICLR, 2018.
Babak Esmacili, Hao Wu, Sarthak Jain, Alican Bozkurt, Narayanaswamy Siddharth, Brooks Paige, Dana Brooks, Jennifer Dy, and Jan-Wa Willem van de Meent. Structured disentangled representa- tions. arXiv preprint arXiv: 1804 880.00006, 2018.
Aviv Gabbay and Yedid Hoshen. Latent optimization for non-adversarial representation disentan- glement. arXiv preprint arXiv: 1796, 2019.
Leilani H Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pp. 80-89. IEEE, 2018.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Luigi Gresele, Paul K Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Schölkopf.
The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. arXiv 06642, 2019.
preprint
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR, 2(5):6, 2017.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.002230. 2018.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. ICML, 2018.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. Advances in neural information processing systems, pp. 3581-3589, 2014.
Jack Klys, Jake Snell, and Richard Zemel. Learning latent subspaces variational autoencoders. In Advances in Neural Information Processing Systems, pp. 6444-6454, 2018.
Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. In Advances in neural information processing systems, pp. 2539-2547, 2015.
10

---

Published as a conference paper at ICLR 2020
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentan- gled latent concepts from unlabeled observations. In ICLR, 2018.
Yann LeCun, Fu Jie Huang, Leon Bottou, et al. Learning methods for generic object recognition with invariance pose and lighting. In CVPR (2), pp. 97-104. Citeseer, 2004.
Hsin- Hsin-Ying Lee, Hung- Yu Tseng, Qi Mao, Jia-Bin Huang, Yu-Ding Lu, Maneesh Singh, and Ming- Hsuan Yang. Drit++: Diverse image-to-imane translation via disentangled representations. arXiv preprint arXiv:/905.01277 2019.
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled repre- sentations. /CML, 2019.
Brian McFee and Gert R Lanckriet. Metric learning to rank. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 775-782, 2010.
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. arXiv preprint
arXiv:1802.00663,, 2018.
Siddharth Narayanaswamy, T Brooks Paige, Jan-Willem Van de Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr. Learning disentangled representations with semi-supervised deep generative models. In Advances in Neural Information Processing Systems, pp. 5925-5935, 2017.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee, Deep visual analogy-making. In Advances in neural information processing systems, pp. 1262-1260, 2015.
Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddings with the f-statistic loss. Advances in Neural Information Processing Systems, pp. 185-194, 2018.
Raphael Suter, Dorde Miladinovic, Stefan Bauer, and Bernhard Schölkopf. Interventional robustness of deep latent variable models. JCML,, 2018.
Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James Philbin, Bo Chen, and Ying Wu. Learning fine-grained image similarity with deep ranking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1386-1393, 2014.

---

Published as a conference paper at ICLR 2020
APPENDIX
Our appendix consists of nine sections. We provide brief summary of each section below.
Appendix A: We elaborate on the connections between existing definitions of disentanglement and our definitions of consistency / restrictiveness / disentanglement. In particular, we highlight three notable properties of our definitions not present in many existing definitions.
Appendix B: We evaluate our consistency and restrictiveness metrics on the 10800 models in the disentanglemest .iib, and identify models where consistency and restrictiveness are not cor- related.
Appendix C: We adapt our definitions to be able to handle nuisance variables. We do so through a simple modification of the definition of restrictiveness.
Appendix D: We show several additional single-factor experiments. We first address one of the results in the main text that is not consistent with theory, and explain why it can be attributed to hyperparameter sensitivity. We next unwrap the heatmaps into more informative boxplots.
We provide an additional suite of consistency versus restrictiveness experiments by comparing the effects of training with share pairing (which guarantees consistency), change pairing (which guarantees restrictiveness), and both.
Appendix E:
Appendix F: We provide full disentanglement results on all five datasets as measured according to six different metrics of disentanglement found in the literature.
Appendix G: We show visualizations of a weakly supervised generative model trained to achieve full disentanglement.
Appendix H: We describe the set of hyperparameter configurations used in all our experiments.
Appendix I: We provide the complete of assumptions and proofs for theoretical framework.
A CONNECTIONS TO EXISTING DEFINITIONS
Numerous definitions of disentanglement are present in the literature (Higgins et al., 2017: 2018; Kim & Mnih. 2018; Suter et al., 2018; Ridgeway & Mozer, 2018; Eastwood & Williams, 2018; Chen et al., 2018a) 8a). We mostly defer to the terminology suggested by Ridgeway & Mozer (2018), which decomposes disentanglement into modularity, compactness, and explicitness. Modularity means a latent code Z, is predictive of at most one factor of variation Sy. Compactness means factor of variation S, is predicted by at most one latent code Zj. And explicitness means a factor of variation Sj is predicted by the latent codes via simple transformation (e.g. linear). Similar to Eastwood & Williams (2018); Higgins et al. (2018), we suggest a further decomposition of Ridgeway & Mozer (2018)'s explicitness into latent code informativeness and latent code simplicity. In this paper, omit latent code simplicity from consideration. Since informativeness of the latent code is already enforced by requirement that g(Z) is equal in distribution to g gSS)) (see Proposition 6), focus on comparing our proposed concepts of consistency and restrictiveness to modularity and compactness. We make note of three important distinctions.
Restrictiveness is not synonymous with either modularity or compactness. In Figure 2c, it is evident the factor of variation size is predictable any individual 2, (conversely, Z1 is not pre- dictable from any individual factor Sj). As such, Z1 is neither modular nor compact representation of size, despite being restricted to size. To our knowledge, no existing quantitative definition of disentanglement (or its decomposition) specifically measures restrictiveness.
Consistency and restrictiveness are invariant to statistically dependent factors of variation. Many existing definitions of disentanglement are instantiated by measuring the mutual information between Z and S. For example, Ridgeway & Mozer (2018) defines that a latent code Z, to be "ide- ally modular" if it has high mutual information with single factor Sj and zero mutual information with all other factors S.j- This presents a issue when the true factors variation themselves are statistically dependent; even if 21 S1, the latent code Zı would violate modularity if S₁ itself has positive mutual information with Sz. Consistency and restrictiveness circumvent this issue by relying on conditional resampling. Consistency, for example, only measures the extent to which S
12

---

Published as a conference paper at ICLR 2020
invariant resampling of Z when conditioned Z₁ and is thus achieved as long si is function of only 21 ii-irrespeetive of whether 8I and 8, are statistically dependent. this regard, definitions draw inspiration from Suter et al. (2018)'s intervention-base based definition but replaces the need for counterfactual reasoning with the simpler conditional sampling. Because we do not assume the factors of variation statistically independent, our theoretical analysis is also distinct from the closely-related match pairing analysis in Gresele et al. (2019).
Consistency and restrictiveness arise in weak supervision guarantees. One of goals is propose definitions that are amenable to theoretical analysis. As we can see in Section 4, consistency and restrictiveness serve as the core primitive concepts that we use to describe disentanglement guarantees conferred by various forms of weak supervision.
B EVALUATING CONSISTENCY AND RESTRICTIVENESS ON DISENTANGLEMENT-LE MODELS
To better understand the empirical relationship between consistency and restrictiveness, we cal- culated the normalized consistency and restrictiveness scores on the suite of 12800 models from disentang ement_1 _ib for each ground-truth factor. By using the normalized consistency and restrictiveness scores as probes, we were able to identify models that achieve high consistency but low restrictiveness (and vice versa). In Fig. 6, we highlight two models that are either consistent or restrictive for object color on the Shapes3D dataset.
(a) Consistent but not Restrictive
(b) Restrictive but not Consistent
Figure 6: Visualization of two models from di aigeetanglmeet llbb (model ids 11964 and 12307), matching the schematic in Fig. 2. For each panel, we visualize an interpolation along single latent across rows, with each row corresponding to fixed set of values for all other factors. In Fig. 6a, we can see that this factor consistenly represents object color, i.c. each column of images has the same object color, but as we move along rows we that other factors change as well, e.g. object type, thus this factor is not restricted to object color. In Fig. 6b, we see that varying the factor along each row results in changes across columns, we see that the representation color changes depending on the setting other factors, thus this factor is not consistent for object color.
object color but to no other attributes. However if we look
C HANDLING NUISANCE VARIABLES
Our theoretical framework handle nuisance variables, i.e., variables cannot or per- form weak supervision on. It may be impossible to label, or provide match- on that factor of variation. For example, while many features of an image measurable (such brightness and
13

---

Published as a conference paper at ICLR 2020
coloration), may not be able measure certain factors of variation generate data pairs where these factors are kept constant. In this case, we can let one additional variable act as nuisance variable that captures all additional sources of variation / stochasticity.
Formally, suppose the full set true factors is S U {m} R=+1, We define 77-onnistencyy C,( =
intuition that, with nuisance variable, for consistency, we still want changes to Z U {n} to not modify Sy; for restrictiveness, we want changes to Zi U [n} to only modify S {n). We define y-disentanglement as D2 = (T) (I).
C(T) and g-restrictiveness (T) R(T U {n))- This captures
All of our calculus still holds where we substitute (J), prove of the new full disentanglement rule illustration:
for R()), D(I); we
Proposition 1. A (i - Ni= (i).
Proof. On the one hand, N R
- c,
C(i) = c( = R(r)- On the other
hand, A C(i) R(1) (1). The reverse direction trivial.
A = N== R(i). Therefore LHS = Vi E [n]. R(i) A
In (Locatello al., 2019), the "instance" factor in SmalINORB and the background image factor in Scream-dSprites are treated as nuisance variables. By Proposition 1, as long as we perform weak supervision all of the non-nuisance variables (via tharing-pairing say) guarantee their consistency with respect to the corresponding true factor of variation, we still have guaranteed full disentanglement despite the existence of nuisance variable and the fact that cannot measure or perform weak supervision on nuisance variable.
14

---

Published as a conference paper at ICLR 2020
D SINGLE-FACTOR EXPERIMENTS
Col
3oth
1001
340
2008
DM
G41
AMC
0400
Figure 7: This is the same plot as Figure 7, but where we restrict our hyperparameter sweep to always set extra dense False. See Appendix H for details about hyperparameter sweep.
Figure 8: Restricted pairing guarantees consistency. Each plot shows the normalized consistency score each model for each factor variation. Our theory predicts each boxplot highlighted in red to achieve the highest consistency. Due the prevalence of restricted pairing in the existing literature, we chose to only conduct the single-factor restricted labeling experiment on Shapes3D.
15

---

Published as a conference paper at ICLR 2020
Doren
Luerina nore
Figure 9: Change pairing guarantees restrictiveness. Each plot shows normalized restrictiveness score of each model for each factor of variation (row) across different datasets (columns). Dif- ferent colors indicate models trained with change pairing on different factors. The appropriately- supervised model for each factor marked in red.
16

---

Published as a conference paper at ICLR 2020
Sepert3
Grid
Spprse whise tone
Figure 10: Share pairing guarantees consistency. Each plot shows normalized consistency of each model for each factor of variation (row) across different datasets (columns). Different colors indicate models trained with share pairing on different factors. The appropriately-supene model for each factor is marked in red.
17

---

Published as a conference paper at ICLR 2020
Sepert3
Grdh
Figure 11: Rank pairing guarantees consistency. Each plot shows normalized consistency of each model for each factor of variation (row) across different datasets (columns). Different colors indicate models trained with rank pairing on different factors. The appropriately-supent model for each factor is marked in red.
18

---

Published as a conference paper at ICLR 2020
E CONSISTENCY VERSUS RESTRICTIVENESS
Sumer dordes
WPPPy
Figure 12: Normalized consistency vs. restrictiveness score of different models on each factor (row) across different datasets (columns). many of the plots, we see that models trained via change- sharing (blue) achieve higher restrictiveness; models trained via share-sharing (orange) achieve higher consistency; models trained via both techniques (green) simultaneously achieve restrictive- ness and consistency in most cases.
19

---

Published as a conference paper at ICLR 2020
F FULL DISENTANGLEMENT EXPERIMENTS
Anensterie
340
Oee
Disentanglement performance of a vanilla GAN, share pairing GAN, change pairing GAN, rank pairing GAN, and lalle-labeled GAN, as measured by multiple disentanglement metrics in existing literature (rows) across multiple datasets (columns). According to almost all metries, weakly supervised models surpass the baseline, and in some cases, even outperform the fully-labeled model.
Figure 13:
20

---

Published as a conference paper at ICLR 2020
Mus
Figure 14: Performance of a vanilla GAN (blue), share pairing GAN (orange), change pairing GAN (green), rank pairing GAN (red), and fully-labeled GAN (purple), as measured by normalized con- sistency score of each factor (rows) across multiple datasets (columns). Factors {3, 4, 5} in the first column shows that distribution matching to all six change / share pairing datasets particularly challenging for the models when trained on certain hyperparameter choices. However, since con- sistency and restrictiveness can measured in weakly supervised settings, suffices to use these metrics for hyperparameter selection. We in Figure 16 and Appendix G that using consistency and restrictiveness for hyperparameter selection serves as viable weakly- suuelvss surrogate for existing aally-supervised disentanglement metries.
21

---

Published as a conference paper at ICLR 2020
Cexb
Mude
300
Det
Dee
Mude
Dee
Figure 15: Performance of a vanilla GAN (blue), share pairing GAN (orange), change pairing GAN (green), rank pairing GAN (red), and fuble-labeled GAN (purple), measured by normalized strictiveness score of each factor (rows) across multiple datasets (columns). Since restrictiveness and consistency complementary, complementary factors in this figure.
that the anomalies in Figure 14 are reflected in the
22

---

Published as a conference paper at ICLR 2020
(a) S
(b) dSprites
300
206,"
(c) Scream-dSprites
(d) SmalINORB
(c) Cars3l
Figure 16: Scatterplot of existing disentanglement metrics versus average normalized consistency and restrictiveness. Whereas existing disentanglement metrics are fully-supervised, to measure average normalized consistency and restrictiveness with weakly supervised data (share- pairing and match-pairing respectively). making it viable to perform hyperparameter tuning under weakly supervised conditions.
possible
23

---

Published as a conference paper at ICLR 2020
G FULL DISENTANGLEMENT VISUALIZATIONS
As a demonstration of the weakly-supervised generative models, we visualize our best-performing match-pairing generative models (as selected according the normalized consistency score aver- aged across all the factors). Recall from Figures 2a to 2c that, to visually check for consistency and restrictiveness, important that we not only ablate single factor (across the column), but also show that the factor stays consistent (down the row). Each block of x 12 images in Figures 17 checks for disentanglement of the corresponding factor. Each row constructed by random sampling of Zu and then ablating Zg.
Figure 17: Cars3D. Ground truth factors: elevation, azimuth, object type.
24

---

Published as a conference paper at ICLR 2020
Figure 18: Shapes3D. Ground truth factors: floor color, wall color, object color, object size, object type, and azimuth.
25

---

Published as a conference paper at ICLR 2020
Figure 19: dSprites. Ground truth factors: shape, scale, orientation, X-position, Y-position.
26

---

Published as a conference paper at ICLR 2020
Figure 20: Scream-dSprites. Ground truth factors: shape, scale, orientation, X-position, Y-position.
27

---

Published as a conference paper at ICLR 2020
Figure 21: SmaIINORB. Ground truth factors: category, elevation, azimuth, lighting condition.
28

---

Published as a conference paper at ICLR 2020
H HYPERPARAMETERS
Table 1: We trained a probablistic Gaussian encoder to approximately invert the generative model. The encoder is not trained jointly with the generator, but instead trained separately from the genera- tive model (i.e. encoder gradient does not backpropagate to generative model). During training, the encoder only exposed data generated by the learned generative model.
Encoder
X 4 spectral norm conv. 32. RReLU
4 X 4 spectral norm conv. 32. IReLU
X spectral norm conv. 64, IReLU
4 x 4 spectral norm conv. 64. IReLU flatten
128 spectral norm dense. IReLU
x z-dim spectral norm dense
Table 2: Generative model architecture.
Generator
128 dense. ReLU,, batchnorm.
1024 dense. ReLU. batchnorm.
x 4 x 64 reshape.
4 X 4 conv. 64. IReLU. batchnorm.
4 X 4 conv. 32. IReLU. batchnorm.
4 conv. 32. IReLU, batchnorm.
x 4 conv. 3. sigmoid
Table 3: Discriminator used for restricted labeling. Parts in red are part of hyperparameter search.
Discriminator Body
4 X 4 spectral norm conv. 32 x width. IReLU
spectral norm conv.
width. IReLU
4 4 spectral norm conv. 64 x width. IReLU
4 spectral norm conv. x width. IReLU flatten
extra dense: 128 x width spectral norm dense. IReLU
Discriminator Auxiliary Channel for Label
128 x width spectral norm dense. IReLU extra dense: 128 x width spectral norm dense. IReLU
Discriminator head
concatenate body and auxiliary. 128 width spectral norm dense. IReLU 128 x width spectral norm dense. IReLU spectral norm dense with bias.
29

---

Published as a conference paper at ICLR 2020
Table 4: Discriminator used for match pairing. We use a projection discriminator (Miyato & Koyama, 2018) and thus have an unconditional and conditional head, Parts in red are part of hyper- parameter search.
Discriminator Body Applied Separately to z and x'
4 4 spectral norm conv. 32 x width. IReLU
4 4 spectral norm conv. 32 x width. IReLU
4 4 spectral norm conv. 64 x width. IReLU
4 X 4 spectral norm conv. 64 x width. IReLU flatten
If extra dense: 128 x width spectral norm dense. IReLU
concatenate the pair.
128 x width spectral norm dense. IReLU 128 width spectral norm dense. IReLU
Unconditional Head
spectral norm dense with bias
Conditional Head
128 X width spectral norm dense
Table 5: Discriminator used for rank pairing. For rank-pairing, we use special variant of the projection discriminator, where the conditional logit is computed via taking the difference between the two pairs and multiplying by y +1]. The discriminator is thus implicitly taking on the role of an adversarially trained encoder that checks for violations of the ranking rule in embedding space. Parts in red are part of hyperparameter search.
Discriminator Body Applied Separately to x and x'
4 4 spectral norm conv. 32 x width. IReLU
4 4 spectral norm conv. 32 x width. IReLU
4 4 spectral norm conv. 64 x width. IReLU
4 x 4 spectral norm conv. 64 x width. IReLU flatten
If extra dense: 128 x width spectral norm dense. IReLU concatenate the pair.
Unconditional Head Applied Separately to a and x'
spectral norm dense with bias.
Conditional Head Applied Separately to and x'
y-dim spectral norm dense.
30

---

Published as a conference paper at ICLR 2020
For all models, use the Adam optimizer with 31 0.5, 32 0.999 and set the generator learning rate to 1 x 10-3 We use a batch size of 64 and set the leaky ReLU negative slope to 0.2.
To demonstrate some degree of robustness to hyperparameter choices, considered five different ablations:
1. Width multiplier on the discriminator network ((1,2)
2. Whether to add extra fully-connected layer the discriminator ((True, False)).
3. Whether to add bias term to the head ((True, False)).
4. Whether to use two-time scale learning rate by setting encoder-diserrmmnattor learning rate multipler to ((1,2)).
5. Whether
the default PyTorch Keras initialization scheme in all models.
As such, each of our experimental setting trains total of 32 dixtincl models. The only exception is the intersection experiments where we fixed the width multiplier to 1.
To give sense of the scale of our experimental setup, note that the 864 models in Figure 4 originate as follows:
1. 32 hyperparameter conditions restricted labeling conditions.
2. 32 hyperparameter conditions match pairing conditions.
3. 32 hyperparameter conditions 6 share pairing conditions.
4. 32 hyperparameter conditions rank pairing conditions.
5. 16 hyperparameter conditions intersection conditions.
31

---

Published as a conference paper at ICLR 2020
1 PROOFS
I.1 ASSUMPTIONS ON H
Assumption 1. Let D C [n]] indexes discrete random variables SD. Assume that the remaining
random variables Sc = SiD have probability density function p(sclsD) for any set of values sp
where p(Sn = 8p) > O.
Assumption 2. Without loss of generality, suppose S1.m [Sc. Spl ordered by concatenating the continuous variables with the discrete variables. Let ((sD) fint(supp((be (s))), sp) denote the interior of the support of the continuous conditional distribution of Se concatenated with its conditioning variable
drawn from Sp. With slight abuse of notation, let B(5)
U (>D:()>D>0) ((sD). We assume B(S) zig-zag connected, i.e., for any J, S [n], for any two points $1m, $1... E (()) that only differ in coordinates in I U J, there exists path {sf h=vr contained in ((S) such that
$1 81mm
(13)
2¹ $1m0
(14)
1+1 OA St t< T, either s = 8, or s = s++1
(15)
Intuitively, this assumption allows transition from $1mn to $1.n via a series of modifications that are only in 7 or only J. Note that that zig-zag connectedness is necessary for restrictiveness union (Propo- sition 3) and consistency intersection (Proposition 4). Fig. 22 gives examples where restrictiveness union not satisfied when zig-zag connectedness violated.
Assumption 3. For arbitrary coordinate j E [m] of g that maps continuous variable Xj. assume that gy(s) is continuous at s, Vs E (SS;; For arbitrary coordinate j E [m] of g that maps to discrete variable Xj. Van where p(sp) > O, we assume that gy(s) is constant over each connected component of int (supp(((((sc (n)).
Define ( analogously B(S). Symmetrically, for arbitrary coordinate i E [n] of e that maps to a continuous variable S,, we assume that e,(x)) is continuous at x, Vo E ({X); For arbitrary coordinate E [n] of that maps discrete S,, VID where p(zD) >> we assume that ex(x) is
constant over each connected component of int (supp((((æc xD)).
Assumption 4. Assume that every factor of variation is recoverable from the observation X. For- mally, (p, 9, e) satisfies the following property
E Ep(sna) g(s1:)) $1.nl1 0.
(16)
1.2 CALCULUS OF DISENTANGLEMENT
I.2.1 EXPECTED-NORM REDUCTION LEMMA
Lemma 1. Let v, be two random variables with distribution f(x. y) be arbitrary function. Then
E Beroate/Pu..---ttelllllfffff - f(x, 1112 <
y) ((x22//10
Proof. Assume w.l.o.g that (oo.:))-(o
0.
LHS = 2T (x [I r) 2 B. ( (x,
(17)
(x vx) 2 By "
(18)
2T( w)
[xx y)]l 2 By-nol)) v)
(19)
2E ff v)
(20)
2E (x. r)
f
(2))
-2( fx, 2 - 100--22222 2e (x f(x'
(22)
RHS.
(23)
32

---

Published as a conference paper at ICLR 2020
1.2.2 CONSISTENCY UNION
Let L = INJ.K -I-
Proposition 2. C(1) A CJ) = C(UJ)..
Proof.
C(T) = BE Een El 11, G(zas, 2L, 2k) - rs o G(2M, 2L, z = = 0.
(24)
For any fixed value of 2M, Z1,
22, E Ez ||r o G(zs, 2L, 2N; 2k) ri G( 2L, z/y z =
(25)
2 Ezy Fxx & ||r, o G(ZM, Fr, 2N, =x) - rI o G(z 21, 2N = 1|2
(26)
by plugging in x = nN. y = o into Lemma 1. Therefore
C(I) EEM E Ex ||r G(zM, (L, 2k) - ry G(zas 21, 2N,
2..22222k x]1 (27)
Similarly we have
C(J) =
BEMERL E. Ilr G(ZM. 2N, zK) TJ G(ZM: 21, 2N, (.,,,22kk = (28)
Eesree aN a Il G( #L, 2N, =k) IN G(
( M. žい 2N 22 0. (29)
As ISNN-= = JUNN- = IUJ, adding the above two equations gives us (UU))..
1.2.3
RESTRICTIVENESS UNION
Zz
Z2
22
1
1
1
1
2
2
Z1
Z:
Z
2₁
Figure 22: Zig-zag connectedness is necessary for restriveness union. Here n m 3. Colored areas indicate the support of p(æ1, 22); the marked numbers indicate the measurement of $3 given (21, 22). Left two panels satisfy zig zag connectedness (the paths are marked in gray) while the right two do not (indeed R(1) R(2) # R R[ 2))). In the right-most panel, any zig-zag path connecting two points from blue and orange areas has to pass through boundary of the support (disallowed).
Similarly define index sets L, K, M.N.
Proposition 3. Under assumptions specified in Appendix L11 R() A R(J) = R(JUJ).
Proof. Denote f exx o g. We claim that
R(T) = TE, E 1I -(2), 0.
(30)
- ((2,,=)) 2,) f(2), 211) ( (</),
(31)
We first prove the backward direction: When we draw zy p(a,), 21, 21, p(z (()) let E, denote the event that (ir) & B()), and F. denote the event that (2/: 2v) $ ((Z). Rearder the indices of (z), as (=c,2D). The probability that (=1, zu) & B(2) (i.e..z) is on the boundary of B(2p)) is O. Therefore Pr[E,] = Pr[Eg] = O. Therefore Pr[E, Eg] < Pr[E,] + Pr[E,] = O,
i.e. with probability 1, [11 2 f(=),=,)) 0.
33

---

Published as a conference paper at ICLR 2020
Now prove the forward direction: Assume for the sake of contradiction that E (() such that ((2,2)) < f Denote U I Innn: W == =V C.C. We have f(zu, æy; , =Q) < f zy, =w, =Q)- Since f is contin- in the interior of B([EU, zwl), and is also continuous (or constant) at (z zv, 2w, 2Q) in the interior of B[ =w]), we can draw open balls of radius r around each point, l.e, (2v, zq) C B([20,20) zw]) and (z C CB( zw]). where
uous (or constant) at (2U, 2V, 2W, =Q) ,
V(zi, xQ) E Br(zv, zQ), V(2D. =Q) E B, (,(,, f(zu, sw, 22) < fl 2V aw, =Q)
(32)
When we draw aV p(a,), ~ p(zz)),)) let C denote the event that (zI, = (E 2u, ²Q zw), (z)
(z), f "..: where (2)) E (Q) and (=A,=0) E (x(=) Since both balls have positive volume, Pr(C] > 0. However, - Av) - > O whenever event C' happens, which contradicts R(T). Therefore ((21,1)). 2,) E B(Z), =,))
We have shown that
R(I) = V(ZM; 21,, AN, 2k) (Z =2, AN, ,N, E ((M,, 2N, = s( #L, RN; =K) (33)
Similarly
R(J) 4(2M 21, 2x), (R), 2/, xx) B(Z), 2w f(zM: 2'v 2x).
(34)
R(IUJ) + V(ZM EL, VL, an. 2R) (KM xx) E B(Z), f(zM EL, NN. = f(z's: 2/2, p'y. 2x)
(35)
Let the zig-zag path between (EM zL, 2k) and (z z/,,
B(Z) be Repeatedly applying the equivalent conditions of R(T) and R(J) gives
{(=he, =ly =k)) us
f(zM, 21, NN, zk) = [( #t, 2/: =k) =.. 1( ,
T-1 T-1 .NN ,2k) = [( =/, z/v, 2x)
(36)
1.3 CONSISTENCY AND RESTIVENESS INTERSECTION
Proposition 4. Under the same assumptions as restrictiveness union, C(I)AC(J) = C(IJJ).
Proof.
C(T) CJ)
A R(LUV)
(37)
(38)
CONTUN) C(INJ).
(39)
(40)
Proposition 5. R(I) R(J) = R(INJ).
Proof analogous Proposition 4.
I.4 DISTRIBUTION MATCHING GUARANTEES LATENT CODE INFORMATIVENESS
Proposition 6. If (p', e")) E H. and (p.g.e E H. and (S) & g(Z), then there exists continuous function such that
=|1 0.
(41)
34

---

Published as a conference paper at ICLR 2020
Proof. We show that r e* g satisfies Proposition 6. By Assumption 4,
E, g" (s) - =||²= O.
(42)
E g(z) =||² O.
(43)
By the same reasoning as in the proof of Proposition 3,
E.lle" o g (s) - a||² = = Vs E (g")) = 8.
(44)
Eg Eglloo g(2) - =||2 =1 = Vz E B(Z), eo g() =
(45)
Let s ~ p(s). We claim that Pr[Ei] = 1, where En denote the event that 32 E B(Z) such that (s()) = g(2). Suppose the contrary that there is mensere- set S C supp(p(s)) such
that Vs E S. no E B)) satisfies g*))) g(z). Let X (0(8) S}. As y a g (),
(r, (s) E X = Pr, [g(2) E X > O. Therefore 32 S supp(p(z)) B(Z) such that X' C (g(2):
2 E 2)}. But supp(p(=)) B(Z) has measure O. Contradiction.
When we draw s, let Eg denote the event that & E B(S). Pr[Ea] = 1. so Pr[E, A Ez] =1
1. When E1 A Eg happens, ex ogo ogocog (s) e" agoeo g(2) e" g(() ex (s) =s. Therefore
E ogoeog (s) -|1 O.
(46)
1.5 WEAK SUPERVISION GUARANTEE
Theorem 1. Given any oracle (p' (s).g"eed ,ee) E H. consider the distribution-matching algorithm A that selects model (p(2). 9, e) E H such that:
1. ()(S), - (o(Z), Z1) (Restricted Labeling); or
2. S.)
_ (9(2), 2,1), (.)) (Match Pairing); or 9" (5") 1 1 S $11) - (g(Z)) 9 (2') 12 S Z7}) (Rank Pairing).
3. (S)
Then (p. g) satisfies C(I : p. 9, e²) and e satisfies C(I ; g")
Proof. We prove the three cases separately:
1. Since (Id, d (zg: 21), consider the measurable function
f(a. = (a) B11..
(47)
We have
(zle)(za) - 2
= Elle (x,)) 2/ = =0.
(48)
By the same reasoning as in the proof of Proposition 3,
Ezlle o g(z)= 21 = 0 - Vz E B(2), 0 = 21-
(49)
Therefore
E,, E ele ,( - , 22) O. i.e., g satisfies C(I e") By symmetry, satisfies C(1 ;p',g",e).
(50)
2.
(S) - (g(Z), (1) Z(1) (51)
le o S,, - S |e (2 ej o Zr, 21131
(52)
E,, |1, (()))a)) oj g(اا) 11)1 = 0.
(53)
So g satisfies C(T: P. g; e*). By symmetry, e satisfies C(T p", .e).).
35

---

Published as a conference paper at ICLR 2020
3. Let = {i),, f= og. Distribution matching implies that, with probability 1 over random draws of 2', the following event P happens:
Z1 <=<= = f(2) <=
(54)
i.e.,
Eg 1 [-P] = 0.
(55)
Let W -WW-=VD-=VVV-- We showed in the proof of Proposition 3 that
zw, zQ) = f(z,, zw; =Q). (56) C(1) = (wz), (2z, z/w, =²4) E B(2), f(z, aw,
We prove by contradiction. Suppose F(21, =Q), (21, =(2) E B(2) such that f(zI, zw, =Q) < f(=1, :
(a) Case 1: Zy is discrete.
Since f is constant both at (2z, zw, z0) in the interior of B([2),2w]), and at (21, "w, z/n) in the interior of B([=), w we can draw open balls of radius r > around each point, ie., (zQ) c B((z), and (x(2 C. B(2z,,Ww)) where
V=O E (=Q)), V2 E (=(2):: [(11, sw: 2Q) < f(z Aw,, =Q)--
(57)
When we draw 2, p(2). let denote the event that this specific value of 2 is picked for both x', and we picked zu E (z z,, E (=Q)). Since both balls have positive volume, Pr[C] 0. However, P does not happen whenever event C happens, since 27 = 2 but f()) > f(z'), which contradicts Pr[P] =.
(b) Case 2: is continuous.
Similar to case 1, we can draw open balls of radius r > 0 around each point, i.e., (21, C B(zw) and (x(21=z(2) C C((W))
where
2(27,2)) (2(2 = Br(21, =2) f(2), zw, =2) f(z Ww: =U)-
(58)
Let H = ((=+,+²) E (,1,, (Q) : 2 >= 21). H²= ( E B, = of <= 21}. When we draw 2, z p(z), let C' denote the event that we picked H faw],2 E H²
]. Since H' H² have positive volume, Pr[C| > 0.
However, P does not happen whenever event C' happens, since 21 <= z but f(z) > f(='), which contradicts Pr[P] Ir
Therefore we showed
V(=1, zw, =Q), (21, sw. =6) E B()), aw,z)) = f(21, zw, #6),
(59)
i.e., satisfies C(I :.:.ge.) By symmetry, e satisfies C(I CI::p"''' ;p"""",),,
1.6 WEAK SUPERVISION IMPOSSIBILITY RESULT
Theorem 2. Weak supervision via restricted labeling, match pairing, or ranking on 81 is not suffi- cient for learning generative model whose latent code Z₁ restricted to Sj.
Proof. We construct the following counterexample Let
= 3 and 1 = {1}. The data gener-
= 31 82, sy]- 11), 9(2) [21, cos 22 sin (2) 2g sin (2, 22 + cos (z) Then (I., $T) d (z), but R(I:p.g.e") and R(I ; pp,",," .g" , e) does not hold. The same counterexample applicable for match pairing and rank pairing.
ation process is 81 ~ uni 2T)). (82, sy) ~ (nif([(z;)) y) x² + 1² < 11),
Consider a generator 21 (0, 2T)), (sg, S3)
unif(1 y) x² +
36


Reviews:
[{"review_id": "rkez3PlAtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper first discusses some concepts related to disentanglement. The authors propose to decompose disentanglement into two distinct concepts: consistency and restrictiveness. Then, a calculus of disentanglement is introduced to reveal the relationship between restrictiveness and consistency. The proposed concepts are applied to analyze weak supervision methods. \n\nThis paper is well structured. The presentation is easy to follow. The problem discussed is important to the machine learning community. The concepts discussed are supported by a large number of experiments. \n\nThe assumption that disentanglement can be decomposed into consistency and restrictiveness might be flawed. Let us consider a generator $g(Z)$  that always generates the same image for all $Z \\sim p(Z)$. Note that $g(Z)$ gives perfect consistency and perfect restrictiveness as defined in Equation (3) and (6). However, we consider $g(Z)$ is a bad generator, and we do not think the corresponding latent representation $Z$ achieves perfect disentanglement. Note that such $Z$, in general, gives low values in the existing disentanglement metrics.\n\nThis implies that we might need to introduce a third component to disentanglement, which I call it relevance. We should additionally assume that different $z_{ \\setminus I}$ leads to different generated images. It might be challenging to measure relevance quantitatively under the probabilistic framework, but I believe this is necessary. \n\nIn summary, I think the idea presented is interesting and useful. I believe this paper is promising and impactful after proper revision.  However, I do not recommend acceptance because it looks technically flawed. \n\nMinor:\nIn Figure 5, the illustration is clear to me, but I am not sure how the vertical axis simultaneously represents two variables $z_2, z_3$.\nIn the table on page 5, $n$ represents the number of dimensions, right?\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJgQ9TMl5r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper tries to bring some theoretical foundation to the weakly supervised disentanglement. Overall it is a good contribution, but the message of the paper is not clear.  The authors propose two notions: consistency and restrictiveness, which they don't imply each other. However, the experiment on real data shows that they are highly correlated.   Up until the experiment section, the paper is well written (although a bit verbose). It seems that it is great but unfinished work.\n\nThe paper is well written, but in my opinion, there is too much verbosity on page 4-5 on rather trivial definitions consistency and restrictiveness and a big box in the calculus of disentanglement that steals space from the main results.  In my opinion, those sections can be reduced so that other theorem can be covered. In my opinion, the theorem nine should be part of the main text. \n\nI understand the definition of \"Sufficiency for Disentanglement \" but it is not clear why it is important. Sure, it is a strong definition that says for any  $\\mathcal{H}$ (and not a subset) the algorithm ($\\mathcal{A}$) should be able to match the distribution of the observation but why is it a big deal according to the next paragraph?\n\nI don't see any proof that Eq.11 should be between [0,1]. Yes, g is optimal, and if you enter suboptimal values to it, one expects the nominator to be less than dominator. However, g a function that is optimal in expectation, which does not mean for every s value it nominator is less than the denominator. In fact, some of the values in fig 3 are small negatives.\n\nFig 3 is not explained well: you are showing normalized consistency and restiveness. First of all, what is the dataset you tried this on? Second, why some values are negative?! These are supposed to be between [0,1]. Third, what is the take-home-message of this figure? the first two matrices from left show that the factors are consistent b/c they are almost diagonal. The third one from left shows that the algorithm you used is not restrictive? Then are you suggesting this as a metric of evaluation? I am not sure I understand the first figure from the right.\nOverall, the authors perform a significant amount of experiments, but they did a poor job in summarizing the results.\nFinally, the authors claim \n                  \"...We believe this correlation between consistency and restrictiveness to have been a general source of confusion in the disentanglement literature, causing many to either observe or believe that restricted labeling or share pairing on $S_i$ (which only guarantees consistency) is sufficient for disentangling Si ...\" \n\nEach of those methods should be analyzed separately to ensure that their algorithms do not induce restiveness. I just don't see the natural connection between your figure 4 and this conclusion that you made.\n\nMinor:\nWhere is the proof for Theorem 1? In the Supp, it starts with Theorem 8, I guess you meant Lemma 8? You need to clean up the Supp so that one can find the proof easily. I suggest restructuring the Supp to less and finally proof of Thorem 1.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HyeZnJF6KH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper tries to construct a theoretical framework to rigorously analyze the disentanglement guarantees of weak supervision algorithms. In particular, it focuses on two concepts, consistency and restrictiveness which provides a formalism that precisely distinguishes when disentanglement arises from supervision versus model inductive bias. \n\nStrengths\n\nThe framework uses two simple concepts, consistency and restrictiveness for both generator and decoder. It also gives rise to a calculus. It is very useful to demonstrate the conditions under which various supervision strategies guarantee disentanglement. \n\nThe paper also did a good job clarifying how consistency and restrictiveness differ from other disentanglement concepts used in the literature. \n\nWeaknesses\n\nThe paper does not propose effective methods for disentanglement in the weak supervision setting. \n\nThe experimental section uses very toy datasets. It is not clear how the weak supervision signal can come from in realistic applications.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper first discusses some concepts related to disentanglement. The authors propose to decompose disentanglement into two distinct concepts: consistency and restrictiveness. Then, a calculus of disentanglement is introduced to reveal the relationship between restrictiveness and consistency. The proposed concepts are applied to analyze weak supervision methods. 

The reviewers ultimately decided this paper is well-written and has content which is of general interest to the ICLR community.
Decision: Accept (Poster)