Synergies Between Disentanglement and Sparsity: a Multi-Task Learning Perspective
Authors: Sebastien Lachapelle
Tristan Deleu
Divyat Mahajan
Ioannis Mitliagkas
Yoshua Bengio
Simon Lacoste-Julien
Quentin Bertrand
Venue: Submitted to ICLR 2023 (2023)
Keywords: Disentanglement
identifiability
multi-task learning
sparsity
transfer learning
meta-learing

Abstract: Although disentangled representations are often said to be beneficial for downstream tasks, current empirical and theoretical understanding is limited. In this work, we provide evidence that disentangled representations coupled with sparse base-predictors improve generalization. In the context of multi-task learning, we prove a new identifiability result that provides conditions under which maximally sparse base-predictors yield disentangled representations. Motivated by this theoretical result, we propose a practical approach to learn disentangled representations based on a sparsity-promoting bi-level optimization problem. Finally, we explore a meta-learning version of this algorithm based on group Lasso multiclass SVM base-predictors, for which we derive a tractable dual formulation. It obtains competitive results on standard few-shot classification benchmarks, while each task is using only a fraction of the learned representations. 

Under review as a conference paper at ICLR 2023
SYNERGIES BETWEEN DISENTANGLEMENT AND
SPARSITY: A MULTI-TASK LEARNING PERSPECTIVE
Anonymous authors Paper under double-blind review
ABSTRACT
Although disentangled representations are often said to be beneficial for down- stream tasks, current empirical and theoretical understanding limited. In this work, we provide evidence that disentangled representations coupled with sparse base-predictors improve generalization. In the context of multi-task learning, we prove a new identifiability result that provides conditions under which maximally sparse base-predictors yield disentangled representations. Motivated by this the- oretical result, propose a practical approach to learn disentangled representa- bi-level optimization problem. Finally, we tions based on a version of this algorithm based on group Lasso multiclass explore meta-learning SVM base-predictors, for which we derive a tractable dual formulation. It obtains competitive results standard few-shot classification benchmarks, while each task is using only a fraction of the learned representations.
INTRODUCTION
The recent literature on self-supervised learning has provided evidence that learning a representation on large corpuses of data yield strong performances wide variety of downstream tasks (De- vlin et al., 2018; Chen et al., 2020), especially in few-shot learning scenarios where the training data for these tasks is limited (Brown et al., 2020b; Dosovitskiy al., 2021; Radford et al., 2021). Beyond transferring across multiple tasks, these learned representations also lead to improved ro- bustness against distribution shifts (Wortsman et al., 2022) well as stunning text-conditioned image generation (Ramesh et al., 2022). However, preliminary assessments of the latter has high- lighted shortcomings related to compositionality (Marcus et al., 2022), suggesting new algorithmic innovations are needed to make further progress.
Another line of work has argued for the integration of ideas from causality to make progress to- wards more robust and transferable machine learning systems (Pearl, 2019; Schölkopf, 2019; Goyal & Bengio, 2022). Causal representation learning has emerged recently as a field aiming to define and learn representations suited for causal reasoning (Schölkopf al., 2021). This set of ideas is strongly related to learning disentangled representations (Bengio et al., 2013). Informally, a rep- resentation is considered disentangled when its components natural and interpretable factors of variations, such as object positions, colors or shape. A Although a plethora of works have investigated theoretically under which conditions disentanglement is possi- ble (Hyvärinen & Morioka, 2016; 2017: Hyvärinen et al., 2019; Khemakhem et al., 2020a; Locatello al., 2020a; Klindt al., 2021; Von Kügelgen et al., 2021: Gresele et al., 2021; Lachapelle al., 2022: Lippe et al., 2022b; Ahuja et al., 2022c), fewer works have tackled how disentangled repre- sentation could be beneficial for downstream tasks. Those who did mainly provide empirical rather than theoretical evidence for or against its usefulness (Locatello et al., 2019; van Steenkiste et al., 2019; Miladinović et al., 2019; Dittadi et al., 2021: Montero et al., 2021).
in one-to-one correspondence with
In this work, we explore synergies between disentanglement and sparse base- prseepredsstors in the con- text of multi-task learning. At the heart of our contributions is the assumption that only small subset of all factors of variations are useful for each downstream task, and this subset might change from one task to another. We will refer to such tasks as sparse tasks, and their corresponding sets of useful factors as their supports. This assumption was initially suggested by Bengio et al. (2013, Section 3.5): "the feature set being trained may be destined to be used in multiple tasks that may have distinct [and unknown] subsets of relevant features. Considerations such these lead to

---

Under review as a conference paper at ICLR 2023
the conclusion that the most robust approach to feature learning is to disentangle as many factors as possible, discarding as little information about the data as is practical". This strategy is very much in line with the current self-supervised learning trend (Radford et al., 2021), except for its focus on disentanglement.
Our main contributions are the following: (i) We formalize this "sparse task assumption" and gue theoretically and empirically how, in this context, disentangled representations coupled with sparsity-regularized base-predictors can obtain better generalization than their entangled counter- parts (Section 2.1). (ii) We introduce a novel identifiability result (Theorem 1) which shows how one can leverage multiple sparse tasks to learn a shared disentangled representation by regulariz- ing the task-specific predictors to be maximally sparse (Section 2.2.1). Crucially, Assumption 7 formalizes how diverse the task supports have to be in order to guarantee disentanglement (iii) Mo- tivated by this result, we propose a tractable bi-level optimization (Problem (4)) to learn the shared representation while regularizing the task-specific base-predictone to sparse (Section 2.2.2). We validate our theory by showing our approach can indeed disentangle latent factors on tasks con- structed from the 3D Shapes dataset (Burgess & Kim, 2018). (iv) Finally, we draw a connection between this li-le optimization problem and some formulations from the meta- beean-ee litera- ture (Section 2.3). Inspired by our identifiability result, we enhance an existing method (Lee et al., 2019), where the base-learners are now group-sparse SVMs. We show that this new meta-learning algorithm achieves competitive performance on the minilmageNett benchmark (Vinyals et al., 2016), 6), while only using fraction of the learned representation.
SYNERGIES BETWEEN DISENTANGLEMENT AND SPARSITY
In this section, formally introduce the notion of entangled and disentangled representations. First, we assume the existence of some ground-trutt encoder function fo : IR + IR" that maps observations X' _ Rd e.g.. images, to its corresponding interpretable and usually lower dimen- sional representation fx E RTm m I d. The exact form of this ground-trutt encoder depends on the task hand, but also on what the machine learning practitioner considers as interpretable. The learned encoder function is denoted by fo IRd + RRT and should not be conflated with the ground- truth representation fe- For example, fa can be parametrized by a neural network. Throughout, we are going to use the following definition of disentanglement.
Definition 1 (Disentangled Representation, Khemakhem et al. 2020a; Lachapelle et al. 2022). A learned encoder function f& IRd - RTh is said to be disentangled w.r.t. the ground-truth represen- tation fo when there exists an invertible diagonal matrix D and a permutation matrix P such that, for all x E X. fz( fe()) Otherwise the encoder fō is said be entangled.
Intuitively, a representation is disentangled when there is a one-to-one correspondence between its components and the components of the ground-truie trrtt representation, up to rescaling. Note that there exist less stringent notions of disentanglement which allow for component-wist nonlinear invertible transformations of the factors (Hyvärinen & Morioka, 2017: Hyvärinen al., 2019).
Notation. Capital bold letters denote matrices and lower case bold letters denote vectors. The set of integers from 1 to is denoted by [n). We write || for the Euclidean norm on vectors and the Frobenius norm on matrices. For a matrix A E |R*xm, ||Al|2, = m A.gl and ンド= 115,, where 1 is the indicator function. The ground-truth parameter of the
||All 2,0
m
encoder function is 0, while that of the learned representation is Ô. We follow this convention for all the parameters throughout. Table in Appendix summarizes all the notation.
2.1 DISENTANGLEMENT AND SPARSE BASE-PREDICTORS FOR IMPROVED GENERALIZATION
In this section, we compare the generalization performance of entangled and disentangled repre- sentations on sparse downstream tasks. We show that the maximum likelihood estimator (defined in Problem (1)) computed on linearly equivalent representations (entangled disentangled) yield the same model (Proposition 1). However, disentangled representations have better generalization properties when combined with sparse base case-prreictor (Proposition 2 and Figure 1).
First, the learned representation fg is assumed to be linearly equivalent to the ground ground-truth repre- sentation fo, i.e. there exists invertible matrix L such that, for all E X. fz(x) Lfo(
2

---

Under review as a conference paper at ICLR 2023
Note that despite being assumed linearly equivalent, the learned representation fa might not be dis- entangled (Definition 1); in that case, we say the representation is linearly entangled. When we refer to disentangled representation. we write L DP. Roeder et al. (2021) have shown that many common methods learn representations identifiable up to linear equivalence, such as deep neural networks for classification, contrastive learning (Oord et al., 2018; Radford al., 2021) and autoregressive language models (Mikolov et al., 2010; Brown et al., 2020a).
Consider the following maximum likelihood estimator (MLE):²
( = arg max
Σ (2.,))D
logp(y; fa()),
(I)
W
where y denotes the label, D = {(x(11 y (O) is the dataset, p(y; 17) is distribution over labels parameterized by 17 E R and W E Rkxm is the task-specific predictor'. The following result shows that the maximum likelhood estimator defined in Problem (1) is invariant to invertible linear transformations of the features. Note that it is an almost direct consequence of the invariance of MLE to reparametrization (Casella & Berger, 2001, Thm. 10). See Appendix A for proof.
Proposition 1 (MLE Invariance to Invertible Linear Transformations of the Features). Let W! and w be the solutions to Problem (1) with the representations fg and fa. respectively (which assume are unique). If there exists an invertible matrix L such that, Yx E X, fo (x) Lfe(x);
then we have Vx E X. w!!)) fg(æ)) = w fo(x).
Proposition 1 shows that the model ply: W:! fa)))) learned by Problem (1) is independent of L, i.e., the model is the same for disentangled and linearly entangled representations. We thus expect both disentangled and linearly entangled representations to perform identically on downstream tasks.
In what follows, we assume the data is generated according to the following process.
Assumption 1 (Data generation process). The input-label pairs are i.i.d. samples from the dis- tribution p(æ, y) == ply x) x))((=)) with ply x)) = ply; W fa())), where W RRaxx is the ground-truth coefficient matrix.
To formalize the hypothesis that only a subset the the features fo(x) are actually useful to predict the target y. we assume that the ground-truth coefficient matrix W is column sparse, i.e., W| I|2,0 = é < m. Under this assumption, it is natural to constrain the MLE as such:
W (+))) == arg max
Σ (m.
poy: W ff (x)
s.t. | 12,00 Se.
W
The following proposition will help us understand how this additional constraint interacts with rep- resentations that are disentangled or linearly entangled. See Appendix A for a proof.
Proposition 2 (Population MLE for Linearly Entangled Representations), Let w(() be the solu- tion of the population-based MLE, arg max Ep(a.)) log p(y; W (a)) (assumed to be unique).
Suppose fg linearly equivalent fe. and Assumption I holds, then, Wi() WL-1
From Proposition 2, one can see that if the representation Sg is disentangled, then W 1|2.0 = [[ (WP))~11l2. = W ||.00 = l. Thus, in that case, the sparsity constraint in Problem (2) does not exclude the population MLE estimator from its hypothesis class, and yields a decrease in the generalization gap (Bickel et al., 2009; Lounici et al., 2a; Mohri et al., 2018) without biasing the estimator. Contrarily, when fg is linearly entangled, the population MLE might have more nonzero columns than the gruund-truth, and thus would be excluded from the hypothesis space of Problem (2), which, in turn, would bias the estimator.
Empirical validation. We now present a simple simulated experiment to validate the above claim that disentangled representations coupled with sparsity regularization can have better generaliza- tion. Figure compares the generalization performance of the convex relaxation of Problem (2)
We assume the solution is unique.
"(y: 17) could be Gaussian density (regression) or categorical distribution (classification).
3

---

Under review as a conference paper at ICLR 2023
Entangled-Lasso Disentangled-Lasso
Entangled Ridge
Disentangled-Riin Ridge
E m
5%
t/m = 20%
m 80%
1.0
0.5
0.0
50
100
150
50
100
150
50
100
150
# samples
# samples
# samples
Figure 1: Test performance for the entangled and disentangled representation using Lasso and Ridge regression. All the results are averaged over 10 seeds, with standard error shown in error bars.
(Lasso regression, Tibshirani 1996) and Ridge regression (Hoerl & Kennard, 1970) on both dis- entangled and linearly entangled representations. Lasso regression coupled with the disentangled representation obtains better generalization than the other alternatives when 8/m- = 5% and when the number of samples is very small. We can also see that, disentanglement, sparsity regularization and sufficient sparsity in the ground-trutt truth data generating process are necessary to see a significant improvement, in line with discussion. Lastly, the performance of all methods converge to the same value when the number of samples grows. See Appendix D.1 for more details and discussion on the results.
2.2 DISENTANGLEMEN VIA SPARSE MULTITASK LEARNING
In Section 2.1, we argued that disentangled representations can improve generalization when com- bined with sparse preedireds,,ors, but we did provide an approach to learn them. We first provide a new identification result (Theorem 1, Section 2.2.1), 1), which states that in the context of sparse mul- titask learning, sparse base-predictors yield disentangled representations. Then, in Section 2.2.2, we provide a practical way to learn disentangled representations motivated by our identifiability result.
Throughout this section, we assume the learner is given a set of T datasets (DI....: Dr} where each dataset D; := {(x((.)), ) consists of n couples of input æ Rd and label y D. The set of labels ) might contain either class indices or real values, depending on whether we are concerned with classification or regression tasks.
2.2.1 IDENTIFIABILITY ANALYSIS
We present the main theoretical result of our work which shows how learning shared rep- resentation across tasks while penalizing the task-specific -eee-predittor to be sparse can induce disentanglement. Our theory relies the following ground-truth data generating process:
Assumption 2 (Ground-truth data generating process). For each task t, the dataset DD is made of i..dd samples from the distribution p(x, y (())= := ply I x, pp(x w(t)) with ply ar W()) == p(y; W fa(x)), where W(O E RRxm is the task-specific ground-truth coefficient matrix. Moreover, the matrices ((t) are i..d. samples from some probability measure Pw with support W. Also, for all W W, the support of p(æ W) is X C Rd (fixed across tasks).
The above assumption states that (i) the ground-truth coefficient matrices ('t) are task-specifie while the representation Se is shared across all the tasks, (ii) the task-specific ((1) are sampled i..dd. from some distribution Fw, and (iii) the support of is shared across tasks.
Assumption 3 (Identifiability of v)). The parameter v is identifiable from ply; n), i.e. Vy; p(ynn) p(y; 7)
1 -)-
This property holds, e.g., when p(y: 17) is a Gaussian in the usual μ,, a² parameterization. Generally, it also holds for minimal parameterizations of exponential families (Wainwright & Jordan, 2008).
The following assumption requires the ground truth representation fo( to vary enough such that its image cannot be trapped inside proper subspace.
4

---

Under review as a conference paper at ICLR 2023
Twrs
Illustration of Assumption 6 showing three examples of distri- bution Pwis- The red distribution satisfies the assumption, but the blue and orange distributions do not. The red lines are level sets of a Gaussian distribution with full rank covariance. The blue line represents the support of Gaussian distribution with a low rank covariance. The orange dots rep- resents distribution with finite support. The green vector shows that the condition violated for both the blue and the orange distribution, since, in both cases, W1,5a = 0 (orthogonal) with probability greater than zero.
Figure 2:
W..1
- (.90
A-1
Figure 3: The leftmost figure represents S, the support of some p(S). The other figures form a verification that As- sumption holds for S.
2
3
3
j
j
j
Assumption (Sufffcient representation variability). There exists (m))) E X such that the matrix F := [fo((( )..... fo fo (m(m) invertible.
The following assumption requires that the support of the distribution Pw is sufficiently rich.
Assumption (Sufficient task variability). There exists .1,.... fm E [k] such that the rows w( $1.:1... (m(m) are linearly independent.
W(m) W and row indices
Under Assumptions 2 to 5 the representation So is identifiable up to linear equivalence (see Theo- rem 2 in Appendix B). Similar results where shown by Roeder al. (2021); Ahuja al. (2022c). The next assumptions will guarantee disentanglement.
In order to formalize the intuitive idea that most tasks do not require all features, we will denote by (*() the support of the matrix ((t), i.e. g(t) := {j E [m]] w + 0}. In other words, St()) the set of features which useful to predict y in the 1-th task; note that it is unknown to the learner. For our analysis, we decompose Pw as Pw = Σ S(p(()))) P(S) wis where P([m] is the collection of all subsets of [ı7)), p(S) is the probability that the support of W is S and (PW] is the conditional distribution of W given that its support is S. Let S be the support of the distribution p(S), i.c. S : {S E P((m) p(S) > 0}. The set S will have an important role in Assumption & Theorem 1.
The following assumption requires that P Pwi does not concentrate on certain proper subspaces. Assumption 6 (Intra-support sufficient task variability). For all S E S and all e RlSI
Fwis(W E [Rkxm W.sa = =0 =0
We illustrate the above assumption in the simpler case where h = 1. For instance, Assumption 6 holds when the distribution of W1.s S has a density w.r.t. the Lebesgue measure on RISI, which true for example when Wi.s S ~ (O, Σ) and the covariance matrix Σ full rank (red distribution in Figure 2). However, if Σ is not full rank, the probability distribution of Wi.s S concentrates its mass
proper linear subspace V Ç. R!S which violates Assumption 6 (blue distribution in Figure 2). Another important counter-example is when Pwis concentrates some of its mass point ( i.e. P Pwis( (w)) > 0 (orange distribution in Figure 2). Interestingly, there are distributions over W1.s S that do not have density w.r.t. the Lebesgue measure. but still satisfy Assumption This the case, e.g., when W1,s S puts uniform mass over (1S) - 1)- dimensional sphere embedded in (RIS] and centered at zero. See Appendix B.2 for a justification.
The following assumption requires that the support S of p(S) is "rich enough". Assumption (Sufficient support variability). For all j [m.) UsEsijes S [m] (i)-
Intuitively, Assumption 7 requires that, for every feature supports cover all features except j itself. Figure shows an example of satisfying Assumption 7. Removing the latter would only yield partial disentanglement (Lachapelle & Lacoste-Julien. 2022).
find set of tasks such that their
We are now ready to show the main theoretical result of this work, which provides a bi-level op- timization problem for which the optimal representations are guaranteed to be disentangled. It as-
5

---

Under review as a conference paper at ICLR 2023
sumes infinitely many tasks observed, with task- ground-truth matrices W sampled from
Pw- We denote by W(W) the task-specific estimator of W. See Appendix B.1 for proof. Note that we suggest a tractable relaxation in Section 2.2.2.
Theorem (Sparse multi-task learning for disentanglement), Let ê be a minimizer of log ply: ((W) fg(x))
min F pp)
i
s.t. WWEW, W(W) arg min W s.t. |
Ep(n.s) log W fo(())
(3)
Then, under Assumptions 2 to 7, fø is disentangled w.t. fe (Definition 1).
Intuitively, this optimization problem effectively selects representation Sg that (i) allows a perfect
fit the data distribution, and (ii) allows the task-specific estimators VV(W) to be as sparse as the ground-truth truth W. With the same disentanglement guarantees, Theorem 4 in Appendix B presents variation of Problem (3) which enforces the weaker constraint F [W/(W))2)) ||2,, I FF [| |2,00 instead of (WW S
| |2,0 for each task W individually.
2.2.2 TRACTABLE BILEVEL OPTIMIZATION PROBLEMS FOR SPARSE MULTITASK LEARNING
Problem (3) shown to yield disentangled representation (Theorem 1), but is intractable due to the 1.20-seminorm. Thus we use the 2 convex relaxation of the T2.0-seminorm, which is also known to promote group sparsity (Obozinski et 2006; Argyriou et. 2008; Lounici al., 2009):
min
logp(y; W fς(2:))
Tn
(a.y)ED,
(4)
s.t. VtE [T], (0) E arg min
Σ
log W (z)))) + Xel W
W
(m.y)ED
Following Bengio (2000); Pedregosa (2016), one can compute the (hyper)gradient of the outer func- tion using implicit differentiation, even if the inner optimization problem is non-smooth (Bertrand et al., 2020; Bolte et al., 2021; Malézieux et al., 2022: Bolte et al., 2022). Once the hypergradient computed, one optimize Problem (4) using usual first-order methods (Wright & Nocedal, 1999).
Note that the quantity (() fo (x) is invariant to simultaneous rescaling of W by scalar and of (xx) by its inverse. Thus, without constraints on fa()),, || can be made arbitrarily small. This is usual problem in sparse dictionary learning KKreutz-eelgado et al., 2003; Mairal et al., 2008; 2009; 2011), where unit-norm constraints are usually imposed on the column of the dictionary. Here, since fa is parametrized by neural network, normalization (loffe & Szegedy, 2015; Ba et al., 2016) to control its norm. Since the number of relevant features might be task-dependent, Problem (4) has one regularization hyperparameter X per task. To limit the number of hyperparameters in practice, we select X == for all E [T].
suggest apply batch layer
2.3 LINK WITH META-LEARNIH
In the setting known as meta-learning (Finn et al., 2017), for large number of tasks T, we are given training datasets Dirain which usually contains a small number of samples n. Unlike in the multi-task setting though (i.e., unlike in Section 2.2), we are also given separated test datasets Diest to evaluate how well the learned model generalizes to new test samples. In meta-learning, the goal is to learn a training procedure which will generalize well on out-of-distributtin lasks. The bi-level formulation Problem (4) is closely related to metric-based meta-learning (Snell et al., 2017: Bertinetto et al., 2019), where a shared representation fø is learned across all tasks. The representation jointly learned with simple task-specific classifiers, which are usually optimization- based classifiers, such as support-vector machines. Formally, metric-based meta-learning can be formulated as follows
min EDiess
Lont (W(*),
( fz(().
s.t. ((( E arg min
E
Lin (of(()))))
W
(D.

---

Under review as a conference paper at ICLR 2023
Inspired by Lee et al. (2019), where the base-classifiers were multiclass support- machines (SVMs, Crammer & Singer 2001), we propose to use group Lasso penalized multiclass SVMs, in order to introduce sparsity in the base-learners, with Y E Raxk the one-hot encoding of y R":
Lin (Wn; ( v) = max (WW,: u.: WE) fg(x) Y.) [
+ 2n W (5)
2,1
72
In few-shot learning settings, the number of features m is usually much larger than the number of samples n (in Lee et al. 2019, m = 1.6-10* and n < 25). In such scenarios, SVMs-like problems are usually solved through their dual (Boyd et al., 2004, Chap. 5) problems, for computational (Hsieh et al., 2008) and theoretical
& Zhang, 2012) benefits.
Proposition 3. (Dual Group Lasso Soft-Margin Multiclass SVM.) The dual of the inner problem with Lin defined in (5) writes
n
min AEROXR A2
Σ |BST ((Y
-AY-A)TA F F. 2.) (Y
Σ
1 A==1
1A.20:
(6)
with BST the block soft-thresholding operator: BST (a, +
(1-7/|lal), a, F E R
the concatenation of (zf(z)) (a.)) In addition, the primal-dual link writes, for all j E [m].
W BST ((Y-
>1) 122-
Proof of Proposition 3 can be found in Appendix C.I. The objective of Problem (6) is composed of smooth term and block separable non-smooth term, hence it can be solved efficiently using proximal block coordinate descent (Tseng, 2001). As stated in Section 2.2, argmin differentiation of the solution of Problem (6) can be done using implicit differentiation (Bertrand et al., 2022). Although Theorem 1 is not directly applicable to the meta-learning formulation proposed in this section, conjecture that similar techniques could be reused to prove an identifiability result in this setting.
3 RELATED WORK
Disentanglement. Since the work of Bengio et al. (2013), many methods have been proposed to learn disentangled representations based on various heuristics (Higgins et al., 2017: Chen et al., 2018; Kim & Mnih, 2018; Kumar et al., 2018; 8; Bouchacourt et al., 2018). Following the work of Locatello al. (2019). which highlighted the lack of identifiability in modern deep generative models, many works have proposed more or less weak forms of supervision motivated by identifia- bility analyses (Locatello et al., 2020a; Klindt et al., 2021: Von Kügelgen al., 2021; Ahuja al., 2022a;e; Zheng et al., 2022). A similar line of work have adopted the causal representation learn- ing perspective (Lachapelle et al., 2022; Lachapelle & Lacoste-Julien 2022; Lippe et al., 2022b;a; Ahuja et al., 2022b; Yao et al., 2022; Brehmer et al., 2022). The problem of identifiability was well known among the independent component analysis (ICA) community (Hyvärinen et al., 2001: Hyvärinen & Pajunen, 1999) which came up with solutions for general nonlinear mixing functions by leveraging auxiliary information (Hyvärinen & Morioka, 2016; 2017; Hyvärinen et al., 2019; Khemakhem et al., 2020a;b). Another approach is consider restricted hypothesis classes of mix- ing functions (Taleb & Jutten, 1999; Gresele et al., 2021). Contrarily to most of the ahove works, we do not assume that the inputs æ are generated by transforming latent random variable through a bijective decoder g. Instead, we assume the existence of a not necessarily bijective ground-truth feature extractor (a) from which the labels can be predicted using only a subset of its compo- nents in every tasks (Assumption 2). Many of these works make assumptions about the distribution of latent factors, e.g., (conditional) independence, exponential family or other parametric assump- tions. In contrast, we make comparatively weaker assumptions the support of the ground-trudd features (Assumption 4), which are allowed to present dependencies (Section 4). Locatello et al. (2020b) proposed semi-supervised learning approach to disentangle in cases where a few samples are labelled with the factors of variations themselves. This is different from our approach as the la- bels that we consider can be sampled from some p(y; W f (xx)) which is more general. Ahuja al. (2022c) consider a setting similar to ours, but they rely on the independence and non non-gaussianity of the latent factors for disentanglement using linear ICA.
Multi-task, transfer & invariant learning. The statistical advantages of multi-task representation learning is well understood (Lounici et al., 201 la;b; Maurer et al., 2016). However, apart from Zhang
7

---

Under review as a conference paper at ICLR 2023
inner-Lasso
inner-Ridge
inner-Ridge ICA
Correlation = 0.0
Correlation 0.9
Multiple correlation values
1.0
MCC
0.5
0.0
0.01
0.03
0.1
0.3
1.0
0.0
0.01
0.03
0.1
0.3
1.0
0.6
0.8
1.0
X/ A/Aaax
A/ A/Aaxxx
Correlation
Noise scale
0
Noise scale
1
Multiple noise values
1.0
MCC
I
0.0
0.01
0.03
0.1
0.3
1.0
0.0
0.01
0.03
0.1
0.3
1.0
0.25
0.5
0.75
A/2
A/Amasc
A/ A
Noise scale
Figure 4: Disentanglement performance (MCC) for inner-Lasso, Riner-Ridge and Diner-Ridge com- bined with ICA function of the regularization parameter (left and middle). Varying level of correlation between latents (top) and of noise on the latents (bottom). The right columns shows performance of the best hyperparameter for different values of correlation and noise.
et al. (2022), theoretical benefits of disentanglement for transfer learning are not clearly established. Some works have investigated this question empirically and obtained both positive (van Steenkiste et al., 2019; Miladinović et al., 2019; Dittadi et al., 2021) and negative results (Locatello et al., 2019; Montero et al., 2021). Invariant risk minimization (Arjovsky et al., 2020; Ahuja et al., 2020; Krueger et al., 2021; Lu al., 2021) aims learning a representation that elicits inva-invariant base-predictor. This differs from our approach which learns base- preedirresssors that are task-specific.
Dictionary learning and sparse coding. We contrast our approach, which jointly learns dense representation and sparse base- base-predictors (Problem (4)), with the line of work which consists in learning sparse representations (Chen al., 1998; Gribonval & Lesage, 2006). For instance, sparse dictionary learning (Mairal et al., 2009; 2011; Maurer et al., 2013) is an unsupervised technique which aims learning sparse representations that refer to atoms of learned dictionary. Contrar- ily to our method which computes the representation of single imput by evaluating a function approximator fø. in sparse dictionary learning, the representation of single input computed by minimizing a reconstruction loss. In the case of supervised dictionary learning (Mairal et al., 2008), an additional (potentially expressive) classifier learned. This large literature has lead to wide variety of estimators: for instance, Mairal et al. (2008, Eq. 4), which minimizes the sum of the classification error and the approximation error of the code, or Mairal et al. (201 I); Malézieux et al. (2022), which introduce bi-level formulations which shares similarities with our formulations.
EXPERIMENTS
Semi-real experiments on 3D Shapes. We illustrate Theorem 1 by applying Problem (4) to tasks generated using the 3D Shapes dataset (Burgess & Kim, 2018).
Data generation. For all tasks f, the labelled dataset D, = (2x((t))) } is generated by first sampling the ground-truth latent variables (t..) == fo i.i.d. according to some distribution p(z), while the corresponding input is obtained doing (t.0)) = fo (2((0)) (fo is invertible in 3D Shapes). Then, a sparse weight vector w((t) is sampled randomly to compute the labels of each example as (t..) := w((( (+(1.)) + e(t.), where (f.,)) is independent Gaussian noise. Figure explores various choice of p(z), i.c. by varying the level of correlation between the latent variables and by varying the level of noise on the ground-truth latents. See Appendix D.2 for more details about the data generating process.
Algorithms. In this setting where p(y: 17) is a Gaussian with fixed variance, the inner problem of Problem (4) amounts to Lasso regression, we thus refer to this approach inner-Lasso. We also evaluate a simple variation of Problem (4) in which the L, norm is replaced by an Lg norm, and refer to it as inder-Ridge. In addition evaluate the representation obtained by performing linear
8

---

Under review as a conference paper at ICLR 2023
A0max
0.005
0.01
0.02
0.03
0.05
0.07
0.1
1.0-
Accuracy
0.6
0.8
0.4
0.7
Aa
0.2
Meta-train Meta-tast
0.6-
0.0
2500
5000
7500 10000 12500
15000
0.0
0.2
0.4
0.6
0.8
1.0
Features
Average sparsity level
Figure 5: Effect of sparsity on the percentage of tasks using specific features, with our meta-learning objective, on minilmageNet (left). The accuracy of the meta-learning algorithm and the average level of sparsity in the base-learners, as À varies (right).
ICA (Comon, 1992) on the representation learned by inner-Ridge: the case X corresponds to the approach of Ahuja et al. (2022c).
Discussion. Figure 4 reports disentanglement performance of the three methods, as measured by the mean correlation coefficient, or MCC (Hyvärinen & Morioka, 2016; Khemakhem et al., 2020a) (Ap- pendix D.2). In all settings, inner-Lasso obtains high MCC for some values of X, being on par or surpassing the baselines. As the theory suggests, it is robust to high levels of correlations between the latents, as opposed to unnr-Riidge with ICA which is very much affected by strong correlations (since ICA assumes independence). We also see how additional noise on the latent variables hurts inner-Ridge with ICA while leaving inner-Lasso unaffected. Figure in Appendix D.2 shows that all methods find representation which is linearly equivalent the ground-truth representa- tion, except for very large values of A. Refer to Appendix D.2 for more details. Appendix D.2.4 presents experiments showing to what extent inner-Lasso robust to violations of Assumption 7. Appendix D.2.5 presents a visual evaluation of disentanglement. Appendix D.2.6 shows results for the same experiments with the DCI metric (Eastwood Williams, 2018).
Few-shot learning experiments. Despite the lack of ground truth latent factors in standard few-shot learning benchmarks, we also evaluate our meaa-lee objective introduced in Section 2.3, using the dual formulation of the group Lasso penalized SVM as our base-learner, on the minilmageNet dataset (Vinyals et al., 2016). The objective of this experiment is to show that the sparse formulation of the meta-learning objective is capable of reaching similar levels of performance, while using fraction of the features. Details about the experimental settings are provided Appendix D.3.
Discussion. In Figure 5 (left), we report how frequently the learned features are used by the base- learner on meta-training tasks; the gradual decrease in usage suggests that the features are reused in different contexts, across different tasks. We also observe (Figure 5, right) that adding sparsity the base learner may also improve performance on meta-training tasks, while only using a fraction of all the features available in the learned representation, supporting our observations in Section 2.1 the effect of sparsity on generalization on natural images (see Appendix D.3 for further discussion about how this still tests generalization). We also observe that some level of sparsity improves the performance on novel meta-test tasks, albeit to a smaller extent.
CONCLUSION
In this work, we investigated the synergies between sparsity, disentanglement and generalization. We showed that when the downstream task can be solved using only a fraction of the factors of variations, disentangled representations combined with sparse base-predictors can improve general- ization (Section 2.... Our novel identifiability result (Theorem 1) sheds light how, in multi-task setting, sparsity regularization on the task-specific predictors can induce disentanglement This led practical bi-level optimization problem that was shown yield disentangled representations on regression tasks based on the 3D Shapes dataset. Finally, we explored a meta-learning formulation extending this approach, and showed how sparse base-learners can help with generalization, while only using small fraction of the features.

---

Under review as a conference paper at ICLR 2023
REFERENCES
K. Ahuja, K. Shanmugam, K. R. Varshney, and A. Dhurandhar. Invariant risk minimization games.
In Proceedings of the 37th International Conference on Machine Learning. 2020.
K. Ahuja, J. Hartford, and Bengio. Properties from mechanisms: an equivariance perspective on identifiable representation learning. In International Conference on Learning Representations, 2022a.
K. Ahuja, J. Hartford, and Y. Bengio. Weakly supervised representation learning with sparse pertur- bations, 2022b.
K. Ahuja, D. Mahajan, V. Syrgkanis, and Mitliagkas. Towards efficient representation identifica- tion in supervised learning. In First Conference Causal Learning and Reasoning, 2022c.
A. Argyriou, T. Evgeniou, and M. Pontil. Convex multi-task feature learning. Machine learning, 73 (3):243-272) 2008.
M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization, 2020.
L. Ba, J. Kiros, and Hinton. Layer normalization. 2016.
arXiv preprint arXiv:1607.06450
Y. Bengio. Gradient-based optimization of hyperparameters. Neural computation, 12(8);1889-1900 2000.
Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 2013.
L. Bertinetto, J. F. Henriques, P. HS Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. 2019.
Q. Bertrand, Q Klopfenstein, M. Blondel, Vaiter, Gramfort, and Salmon. Implicit differ- entiation of lasso-type models for hyperparameter optimization. In International Conference on Machine Learning, pp. 810-821. PMLR, 2020.
Q. Bertrand, Q. Klopfenstein, M. Massias, M. Blondel, S. Vaiter, A. Gramfort, and J. Salmon. Implicit differentiation for fast hyperparameter selection in non-smooth convex learning. JMLR, 2022.
P. J. Bickel, Y. Ritov, and A. B. Tsybakov. Simultaneous analysis of lasso and Dantzig selector. The Annals of statistics, 1755 2009.
J. Bolte, T. Le, E., Pauwels, and T. Silveti-Falls. Nonsmooth implicit differentiation for machine- learning and optimization. Advances in neural information processing systems, 34: 14:33337-,,,49 2021.
J. Bolte, E. Pauwels, and S. Vaiter. Automatic differentiation of nonsmooth iterative algorithms. NeurIPS, 2022.
D. Bouchacourt, R. Tomioka, and S. Nowozin. Multi-level variational autoencoder: Learning dis- entangled representations from grouped observations. Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
S. P. Boyd, and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.
Brehmer, P. De Haan, P. Lippe, and T. Cohen. Weakly supervised causal representation learning. In Advances in Neural Information Processing Systems, 2022.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam,
G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu. C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few- shot learners. In Advances in Neural Information Processing Systems, 2020a.
10

---

Under review as a conference paper at ICLR 2023
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in Neural Informa- tion Processing Systems, 33:1877-1901, 2020b.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http://gghhuu.oomm/emmmm/////dhd 2018.
G. Casella and R. Berger. Statistical Inference. Duxbury Resource Center, 2001.
R. T. Q. Chen, X. Li, R. G., and D. Duvenaud. Isolating sources of disentanglement in vaes. In Advances in Neural Information Processing Systems, 2018.
S. S. Chen, D. L. Donoho, and M. A. Saunders. Atomic decomposition by basis pursuit. SIAM Journal on Scientific Computing, 1998.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning, pp. 1597-1607. PMLR, 2020.
P. Comon. Independent component analysis. Higher-Order Statistics, 1992.
K. Crammer and Y. Singer. On the algorithmic implementation of multiclass kemel--baed vector machines. Journal of machine learning research, 2(Dec):265-292, 2001.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv:1810.04805, 2018
A. Dittadi, F. Träuble, F. Locatello, M. Wuthrich, V. Agrawal, O. Winther, S. BBuue, and B. Schölkopf. On the transfer of disentangled representations in realistic settings. In Interna- tional Conference on Learning Representations, 2021.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D) Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. International Conference Learning Representations, 2021.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disen- tangled representations. In International Conference on Learning Representations, 2018.
C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep net- works. In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
A. Goyal and Y. Bengio. Inductive biases for deep learning of higher-level cognition. Proc. R. Soc. 478: 20210066 2022.
L. Gresele, Von Kügelgen, V. Stimper, Schölkopf, and M. Besserve. Independent mechanism analysis, new concept? In Advances in Neural Information Processing Systems, 2021.
R. Gribonval and S. Lesage. A survey of sparse component analysis for blind source separation: principles, perspectives, and new challenges. In ESANN' '06 proceedings 14th European Sympo- sium on Artificial Neural Networks, 2006.
I. Higgins, L. Matthey, A. Pal, C. P. Burgess, X. Glorot, M. Botvinick, Mohamed, and A. Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
A. E. Hoerl and R. W. Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55-67, 1970.
Hospedales, Antoniou, P. Micaelli, and A. Storkey. Meta-leariing in neural networks: A survey. IEEE transactions pattern analysis and machine intelligence, 44(9):5149-511 2021.
C.J. Hsich, K.-W. Chang, C.-J. Lin, S. S. Keerthi, and S. Sundararajan. A dual coordinate de- scent method for large-scale linear svm. Proceedings of the 25th international conference on Machine learning, pp. 408 415, 2008

---

Under review as a conference paper at ICLR 2023
A. Hyvärinen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In Advances in Neural Information Processing Systems, 2016.
A. Hyvärinen and H. Morioka. Nonlinear ICA of Temporally Dependent Stationary Sources. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 2017.
A. Hyvärinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks, 1999.
A. Hyvärinen, Karhumen, and E. Oja. Independent Component Analysis. Wiley, 2001.
A. Hyvärinen, H. Sasaki, and R. E. Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In AISTATS. PMLR, 2019.
S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing in- ternal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, 2015.
I. Khemakhem, D. Kingma, R. Monti, and A. Hyvärinen. Variational autoencoders and nonlinear ica: A unifying framework. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, 2020a.
I. Khemakhem, R. Monti, D. Kingma, and A. Hyvärinen. Ice-beem: Identifiable conditional energy- based deep models based on nonlinear ica. In Advances in Neural Information Processing Sys- tems, 2020b.
H. Kim and A. Mnih. Disentangling by factorising. In Proceedings of the 35th International Con- ference on Machine Learning, 2018.
D. A. Klindt, L. Schott, Y Sharma, Ustyuzhaninov, W. Brendel, M. Bethge, and D. M. Paiton. To- wards nonlinear disentanglement in natural data with temporal sparse coding. 9th International Conference Learning Representations, 2021.
K. Kreutz-Delgado, J. F. Murray, Rao, Engan, T.-W. Lee, and T. J. Sejnowski. Dictionary learning algorithms for sparse representation. Neural computation, 15(2):349-396, 2003.
D. Krueger, Caballero, J.-H. Jacobsen, A. Zhang, J. Binas, R. Le Priol, D. Zhang, and A. Courville. Out-of-distribution generalization via risk extrapolation ([re)x), 2021.
A. Kumar, P. Sattigeri, and A. Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. In International Conference on Learning Representations, 2018.
S. Lachapelle and S. Lacoste-Julic Partial disentanglement via mechanism sparsity. In UAI 2022 Workshop on Causal Representation Learning, 2022.
S. Lachapelle, P. Rodriguez Lopez, Y. Sharma, K. E. Everett, R. Le Priol, A. Lacoste, and S. Lacoste- Julien. Disentanglement via mechanism sparsity regularization: new principle for nonlinear ICA. In First Conference Causal Learning and Reasoning, 2022.
K. Lee, S.Maji, Ravichandran, and S. Soatto. Meta-learning with differentiable convex optimiza- tion. In Proceedings of the IEFF/CVF conference on computer vision and patlern recognition, pp. 10657-10665, 2019.
P. Lippe, S. Magliacane, S. Löwe, Y. M Asano, T. Cohen, and E. Gavves. iCITRIS: Causal repre- sentation learning for instantancous temporal effects. In UAI 2022 Workshop on Causal Repre- sentation Learning, 2022a.
Lippe, S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and E. Gavves. CITRIS: Causal identifi- ability from temporal intervened sequences, 2022b.
F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. Proceed- ings of the 36th International Conference Machine Learning, 2019.
12

---

Under review as a conference paper at ICLR 2023
F. Locatello, B. Poole, G. Raetsch, B. Schölkopf. O. Bachem, and M. Tschannen. Seakkl-supervisee disentanglement without compromises. Proceedings of the 37th International Conference on Machine Learning. 2020a.
Locatello, M. Tschannen, S. Bauer, G. Rätsch, B. Schölkopf, and O. Bachem. Disentangling factors of variations using few labels. In International Conference Learning Representations, 2020b. URL https:/ openreview net forumm Sarummid=Syyaueeee SygaggEEK
K. Lounici, M. Pontil, A. B. Tsybakov, and S. Van De Geer. Taking advantage of sparsity in multi- task learning. arXiv preprint arXiv:0903.1468 2009.
K. Lounici, M. Pontil, and A. B. Tsybakov. Oracle inequalities and optimal inference under group sparsity. The Annals of statistics, 201 la.
K. Lounici, M. Pontil, S. under group sparsity. The annals of statistics, 201
Van De Geer, and A. B Tsybakov. Oracle inequalities and optimal inference
Chaochao Lu, Yuhuai Wu, Jose Miguel Hernández-Loban Lobato, and Bernhard Schölkopf. Nonlinear invariant risk minimization: causal approach, 2021.
J. Mairal, J. Ponce, G. Sapiro, A. Zisserman, and F. Bach. Supervised dictionary learning, Advances in neural information processing systems, 21, 2008.
J. Mairal, F. Bach, J. Ponce, and G. Sapiro. Online dictionary learning for sparse coding. In Pro- ceedings of the 26th annual international conference on machine learning, pp. 689-6966 2009.
J. Mairal, F. Bach, and J. Ponce. Task-driven dictionary learning. IEEE transactions on pattern analysis and machine intelligence, 34(4):791-804, 2011.
B. Malézieux, T. Moreau, and M. Kowalski. Dictionary and prior learning with unrolled algorithms for unsupervised inverse problems. ICLR, 2022.
G. Marcus, E. Davis, and S. Aaronson. A very preliminary analysis of dall-e 2. arXiv preprint 20::22204.3887, 2022.
A. Maurer, M. Pontil, and B. Romera-Paredes. Sparse coding for multitask and transfer learning. ICML' 13, 2013.
A. Maurer, M. Pontil, and B. Romera-Paredes. The benefit of multitask representation learning. J. Mach. Learn. Res., 2016.
T. Mikolov, M. Karafiát, L. Burget, J. Cernocky, and Khudanpur. Recurrent neural network based language model. ISCA, 2010.
D. Miladinović, M. W. Gondal, Schölkopf, J. M. Buhmann, and Bauer. Disentangled state space representations. arXiv preprint (rXivv1906.03255 2019.
M. Mohri, A. Rostamizadeh, and Talwalkar. MIT Press, 2018.
M. L. Montero, C. JH Ludwig, R. P. Costa, G. Malhotra, and J. Bowers. The role of disentanglement in generalisation. International Conference on Learning Representations, 2021.
G. Obozinski, B. Taskar, and M. Jordan. Multi-task feature selection. Statistics Department, UC Berkeley, Tech. Rep, 2(2.2):2, 2006.
A. Oord, Li,, and O. Vinyals. Representation learning with contrastive predictiiee Advances in Neural Information Processing Systems, 2018.
J. Pearl. The seven tools of causal inference, with reflections on machine learning. Commun. ACM, 2019.
Pedregosa. Hyperparameter optimization with approximate gradient. International conference machine learning, pp. 737-746. PMLR, 2016.
13

---

Under review as a conference paper at ICLR 2023
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, et al. Learning transferable visual models from natural language supervision. In Inter- national Conference Machine Learning, pp.
PMLR, 2021.
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprini arXiv:2204.06122 2022.
P. Richtárik and M. Takáč. Iteration complexity of randomized block-coordinate descent methods for minimizing composite function. Mathematical Programming, 144(1):1-38, 2014.
G. Roeder, L. Metz, and D. P. Kingma. On linear identifiability of learned representations. In Proceedings of the 38th International Conference Machine Learning. 2021.
B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE Advances in Machine Learning and Deep Neural Networks, 2021.
B. Schölkopf. Causality for machine learning, 2019.
Shalev-Shwartz and Zhang. Stochastic dual coordinate ascent methods for regularized loss minimization. The Journal of Machine Learning Research, 2012.
J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems, 30, 2017.
A. Taleb and C. Jutten. Source separation in post-nonlinear mixtures. IEEE Transactions on Signal Processing, 1999.
R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Sociery: Series B (Methodological). 58(1) 1996.
P. Tseng. Convergence of block coordinate descent method for nondifferentiable minimization. Journal of optimization theory and applications, 109(3):475-494, 2001.
S. van Steenkiste, F. Locatello, J. Schmidhuber, and O. Bachem. Are disentangled representations helpful for abstract visual reasoning? In Advances Neural Information Processing Systems, 2019.
O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Von Kügelgen, Y. Sharma, I.. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F, Locatello. Supe-supervised learning with data augmentations provably isolates content from style. In Thirty- Fifth Conference on Neural Information Processing Systems, 2021.
M. J. Wainwright and M. I. Jordan. Graphical models, exponential families, and variational infer- ence. Found. Trends Mach. Learn., 2008
M. Wortsman, Ilharco, J. W. Kim, M. Li, S. Kornblith, R. Roelofs, Lopes, H. Hajishirzi, A. Farhadi, H. Namkoong, and L. Schmidt. Robust fine-tuning of zero-shot models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 7959-7977 June 2022.
S. Wright and J. Nocedal. Numerical optimization. Springer Science, 35(67-68):7, 1999.
W. Yao, Y. Sun, A. Ho, C. Sun, and Zhang. Learning temporally causal latent processes from general temporal data. In International Conference on Learning Representations, 2022.
H. Zhang, Y.F. Zhang, W. Liu, A. Weller, B. Schölkopf, and E. Xing. Towards principled disentan- glement for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Y. Zheng, I. Ng. and K. Zhang. On the identifiability of nonlinear ICA with unconditional priors. In ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022.
14

---

Under review as a conference paper at ICLR 2023
CONTENTS
Introduction
Synergies between Disentanglement and Sparsity
2.1
Disentanglement and sparse base-predictors for improved generalization
2.2
Disentanglement via sparse multitask learning
2.2.1
Identifiability analysis
2.2.2
Tractable bilevel optimization problems for sparse multitask learning
2.3
Link with meta-learning
Related Work
4 Experiments
5 Conclusion
Proofs of Section 2.1
16
Identifiability Theory
17
B.1 Proof of Theorem
20
B.2 A distribution without density satisfying Assumption 6
21
C Optimization details
21
C.1 Group Lasso SVM Dual
D Experimental details
23
D.I Disentangled representation coupled with sparsity regularization improves general- ization
23
D.2 Semi-Real Experiments on 3D Shapes
24
D.2.1
Dataset generation
24
D.2.2
Metries
25
D.2.3
Architecture, inner solver & hyperparameters
25
D.2.4
Experiments violating assumptions
25
D.2.5
Visual evaluation
26
D.2.6
Additional metrics for disentanglement
27
D.3
Meta-learning experiments
28
15

---

Under review as a conference paper at ICLR 2023
Table 1: Table of Notation.
Norms & pseudonorms
||-I|
Euclidean norm on vectors and Frobenius norm on matrices
|A| 2.1 [
m
|Agll
2,0
2==1
1 111 where 1 is the indicator function.
Data
2 E Rd
Observations
X
Rd
Support of observations
E R y
Target
Support of targets
Learne//ground truth model
W E Rexm
Ground-truth coefficients
W Rexm
Learned coefficients
Ground-truth parameters of the representation
Learned parameters of the representation Ground -ruth representation Learned representation
fo Rd R"
fo Rd RM
7 R
Parameter of the distribution p(y: m)
Pw
Distribution over ground-truth coefficient matrices W
(j E [m]] Wi 7 0} (support of W)
Pwis
Conditional distribution of W given S.
P(S)
Ground -ruth distribution over possible supports $
S
Support of the distribution p(S)
Optimization
W
Primal variable
A
Dual variable
h" : a H
supberd (a, b) lb(b), Fenchel conjugate the function h. Rd - IR
[[g:
min; s(a b) + g(b) inf-convolution of the functions, f and g
BST: : (a,7) +
(1 7/ - [), a. block soft-thresholding operator
A PROOFS OF SECTION 2.1
Proposition 1 (MLE Invariance Invertible Linear Transformations of the Features). Let W!!!) and (0 be the solutions to Problem (1) with the representations fa and fo. respectively (which we assume are unique). If there exists an invertible matrix L such that, We E X, fo( Lfo(x);
then we have, Vo E X, ((() fg(Ø) (1 fo(
Proof. By definition of (V( we have that, for all W e Rkxmm
Σ logp(;: ( fo(x)) >
Σ
log (f))))
(7)
(w.
Σ
logp(v: W 0 Lfe(z))
Σ
logp(y: W Lf)
(8)
(E
(x.
16

---

Under review as a conference paper at ICLR 2023
Because R L Rexm, we have that, for all W Rkxm.
Σ
logply: ( Lfo(z)) 2
Σ
logply; W fo(z)),
(9)
(m.
which is to say that ( W L, or put differently, W W L-1, implies
( fo(z) L w fø(x),
(10)
which is what we wanted to show.
Proposition 2 (Population MLE for Linearly Entangled Representations). Let W be the solu- tion of the population-basee MLE, arg maxw Ep(may) log W f (x)) (assumed to be unique).
Suppose fg is linearly equivalent fe, and Assumption 1 holds, then, w WW
Proof. By definition of W!l). we have that, for all W E IR
F p(())) log loy ( Sg (x)) F ((a.)) logply: W fg()))
(11)
E)(aa) logp W Lfe (x)) Ep(a,) log W Lf)
(12)
In particular, the inequality holds for W WL which yields E E((a))) logp(y; w Lfo(æ))
2 Ep(x,)) logp(;; W (o(())
(13)
0 pp(e,)) log W fo())) logp(y: w Lfo(z))
(14)
0 2 Max KL(p(: W (=)) || plu: W Lfe(x))).
(15)
Since the KL is always non-negative, we have that.
W ff(()) ll ply: W Lfo(x))) =
(16)
which in turn implies
Ex(e.)) logp(y; W Lfo(w)) Eple.)) ply W fo()))
(17)
F (((a)) log W 10) Lfo(m)) P( log ply; W L Lff(z))
(18)
Em(nn) log pv: W 6 fg (x)) Ep(a,)) log p(y W L (z))
(19)
(20)
Since the solution to the population MIE from Proposition 2 is assumed to be unique, this equality holds if and only if vi-(0) WL-!
B IDENTIFIABILITY THEORY
The following lemma will be important for proving Theorem 3. The argument is taken from Lachapelle al. (2022).
Lemma (Sparsity pattern of an invertible matrix contains a permutation). Let L Rmxm be invertible matrix. Then, there exists permutation a such that Li & for all i.
Proof. Since the matrix L is invertible, its determinant is non-zero, i.e.
det(L)=
Σ so (o) II L., + 0,
(21)
BEG
where G,, is the set of m- pe-peumutations This equation implies that at least one term of the sum is non-zero, meaning there exists &. such that for all i E [m]. Ls,((s) 0.
17

---

Under review as a conference paper at ICLR 2023
For all W W, going to denote by W some estimator of W. The following result provides conditions under which if W(W) allows perfect fit of the ground-truth distribution p(y æ, W), then the representation fo and the parameter W are identified up to an invertible linear transformation. Many works have showed similar results in various context (Hyvärinen & Morioka, 2016; Khemakhem et al., 2020a; Roeder el al., 2021; Ahuja et al., 2022c). We reuse some of their proof techniques.
Theorem 2 (Linear identifiability). Let W : W + Rkxm, Suppose Assumptions' 2 to 5 hold and that, for all W E W. E X' and y E V, the following holds
ply; W (y fa(x)) p(u p(u
(22)
Then, there exists an invertible matrix E |R such that, for all x E X, fo(x) = Lfe(w) and such that, for all W W. W(W) WL
Proof. By Assumption 3, Equation (23) implies that W fa())
W(W)
Assumption
w
ensures that can construct invertible matrix U ==
Construct analogously ú
W ( faz
w(wa)
This allows us write ff(2) f (z). Left-multiplingg by U-1 on both sides
(Wils) iage
yields fo(x) = (av), where L U- Using the invertible matrix F from Assumption 4, can thus write F LF where we defined F == [f) ( (x((d))) Since F invertible, so are L and F.
By substituting F = LF WF W(W) F, we obtain WLF WW) F. By miglt-mu both sides by F-1, we obtain WL = WWW)
The following theorem is where most of the theoretical contribution of this work lies. Note that The- orem 1, from the main text, is a straightforward application of this result.
Theorem 3. (Disentanglement via task sparsity) Let W W. Rkxm, Suppose Assumptions to 7 hold and that, for all W E W. x E X and y E V, the following holds
ply: (W) fo(x)) plu: W fe(x))
(23)
Morevover, assume that E|] W (W) 11.00 I E W
2.00 where both expectations are taken w.rl. Pw and [||]2 := E (W, # 0) with 1 (.) the indicator function. Then, fg is disentangled w.r.f. fo (Definition l).
Proof. First all, by Assumptions to 5, we can apply Theorem to conclude that fo(æ) Lfg (x) and W L = (W) for some invertible matrix L.
We thus write E W[WL|20 F|
18

---

Under review as a conference paper at ICLR 2023
We can write
E|
W 2.0
(p()
( of # 0) |S]
(24)
j=1
m
pp(s)
E[ 3 + 0) |S]
(25)
5=1
m p())
Pwis| W.y + 0
(26)
j=1
m
= ((S)
Σ 1(j E S)
(27)
j=1
where the last step follows from the definition of S. We now perform similar steps for E| W L L1l2.0:
m
F| T1220 = (s) F
1WW L.j #0) |s]
(28)
j=1
m
= (S)
E[1 L, + 0)
(29)
j=1
= pp())
P Pwis W b.j + #0
(30)
J=1
= F (S)
P wis W.sLs #0).
(31)
Notice that
Pwis W.s is Lsj +0] 1 Pwis W.sLs. 0
(32)
Let N, be the support of L,g, ie. :=
== ei [m] L, # 0}. When Sn 0, Ls.j
and thus P L.... 0j = 1. When N + 0, Lsj- + 0, by Assumption we have that
ss = 0] 0. 0. Thus
Pwis W. Ls # 0] 1(8 n N, 0) 1(8n,, # #0),
(33)
(34)
which allows us to write
E W L 12,0 Ep(s)
I(SNN, * 0).
(35)
We thus have that
El|WD S W 1220
(36)
m
Ep(s) Σ N, + < (s) Σ ((jES).
(37)
Since L is invertible, by Lemma 1, there exists a permutation a [m] - [m] such that, for all
j E [m]. Lj(()) # 0. In other words, for all [m]. E Noo)- Of course we can permute the terms of the hh..s. of eq. (37), which yields
m
Ep(S)
(SSNNo)) #0) S Ep(s) Σ ()
(38)
j=1
E Ex(S)
(1(S () #0) - 1(j E ()) 00.
(39)
1=1
19

---

Under review as a conference paper at ICLR 2023
We notice that each term 1(5 0 Noj #0] 1(j E S) since whenever j S, also have that j E S No(s (recall E Nz)))) Thus, the Lhss. of eq. (39) is a sum of non-negative terms which is itself non-positive. This means that every term in the sum is zero:
AA S, Vj [m], 1(S (GN)) # 0) 1(j S).
(40)
Importantly.
Vj E [m], AS S. j = 8n SON))) 0,
(41)
and since Sn S(No)) + NoU) 3 S we have that
Vj E [m], AS E S, j = Now1 S
(42)
Aj E [m], No(1) _ N S
(43)
SES sES
By Assumption 7, we have that USEsieeS = [m] \ (i)- By taking the complement on both
sides and using De Morgan's law, we get AsEsljes S = ()}. which implies that No(y) = (i) by Equation (43), Thus, L DP where D is invertible diagonal matrix and P is permutation matrix.
B. PROOF OF THEOREM
Before presenting Theorem I from the main text, first present variation of it where
strain El) l|2.0 be smaller than El W 12,0- We note that this is weaker than imposing
[ 11120 S ||ll2 for W E W, as the case in Problem (3) of Theorem 1.
Theorem 4 (Sparse multitask learning for disentanglement). Let Ô be a minimizer of fo fxx))
min F Ep( logp(y; (W))
S.I. AWEW W((W) E arg min p(a))w)) - logply; W fa(a))
(44)
w
E (W 112.0 - Ep 11.0.
Then, under Assumptions 2 to 7, fg is disentangled wat. fo (Definition 1).
Proof. First, notice that
0 S EFw Eptai KL(p W fe())) || ply; ( fo())))
(45)
Erw Ep(a.//)) logp(y; W fo())) Er w logp(y; W fo(x))) W((W))
(46)
For fixed value of and W. is well known that (L(p(y; W fo())) p(y: ( fg (x)))
and only if, for all E V, ply; W fa())) = p(y: ( (x)). ( By Assumption 3, this is equivalent to W fe())
W(W) fz(z). Thus, for the equality to hold in eq. (45), we need fe()) = ( fx) everywhere. Of course, the global minimum can be achieved by re- specting E || 1|2.0 S FF ||2,,,, simply by setting 0 = O and (W) == :.
The above implies that if Ô is some minimizer of Problem (44), we must have that W fo()) (WW ( (x) everywhere and F WW() S [Fw [/o Thus, Theorem 3 implies the desired conclusion.
L
Based on Theorem 4, we can slightly adjust the argument to prove Theorem from the main text.
Theorem (Sparse multi-rask learning for disentanglement). Let Ô be a minimizer of
min F E Ep(ea/)) log p(y: (W) fx)))
s.t. WEWW, W(W arg min W a.c.
Epta.y/w) - log W Wfz(z))
(3)
Then, under Assumptions 2 7, Sa is disentangled w.r.. So (Definition 1).
20

---

Under review as a conference paper at ICLR 2023
Proof. The first part of the argument in the proof of Theorem 4 applies here as well, meaning: for the equality to hold in eq. (45), need W fa -W( f (a:) everywhere. This global minimum can be achieved by respecting [[WWW) 12.0 Σ and W(W) = W.
W 12,00 for all W E W simply by setting O ==
This means that if 0 some minimizer of Problem (3), we must have that W fo(x) W fz( holds everywhere and that, for all W E W, [[W(W) 12.0 I [| Of course, this means EFw || I|o I EFw |FwllWWl WWloo which allows to apply Theorem to obtain the desired conclu- sion.
B.2 A DISTRIBUTION WITHOUT DENSITY SATISFYING ASSUMPTION 6
Interestingly, there distributions W1.s S that do not have density w.r.t the Lebesgue measure, but still satisfy Assumption 6. This is the case, e.g., when Wi,s S puts uniform mass over a (IS) - -)- -idiennsioa sphere embedded in IRIS) and centered at zero. In that case, for all E IR (R!SiN(O the intersection of span(a)+ which is (IS) - -)ddimensional, with the (JS) 1)- dimensional sphere is (IS) 2)-dimensional and thus has probability zero of occurring. One can certainly construct more exotic examples of measures satisfying Assumption that concentrate mass on lower dimensional manifold.
C OPTIMIZATION DETAILS
C.I GROUP LASSO SVM DUAL
Notation. The Fenchel conjugate function h Rd + is written h" and defined for any y E Rd, by ht hyyy) supzeR= (z, y) - h(z).
Definition 2. (Primal Group Lasso Soft-Margin Multiclass SVM.) The primal problem of the group Lasso soft-margin multiclass SVM is defined as
"
min WeRexm
(W; Σ
max (1+ (W y. W. Ya) + W1.. 2/2 1|12 KE
(47)
Proposition 3. (Dual Group Lasso Soft-Margin Multiclass SVM.) The dual of the inner problem with Lin as defined in (5) writes
1 m
n
min AERUEA
|BST ((Y AA) - A)) (
Σ
1A.20.
(6)
A2 j=1
A==1
with BST the block soft-thresholding operator: BST (a,T) + (1-7/llal) F Raxm
(e the concatenation of (a. In addition, the primal-dual link writes, for all j E [m],
W.y BST ((Y - AA) F.j.A1) /A2.
The primal objective 47 can be hard to minimize with modern solvers. Moreover in few-shot learn- ing applications, the number of features m is usually much larger than the number of samples n (in Lee et al. 2019, m = 1.6 10a and n < 25), hence we solve the dual of Problem (47).
Proof of Proposition 3. Let g: : + A1 b |/ Proof of Proposition 3 is composed of the following lemmas.
Lemma 2.
i) The dual of Problem (47) is
min Eg - F,)
AERwXk 1=1
(48)
s.t. ViE [m).
A===1, W Aa 20,
where 9" is the Fenchel conjugate of the function g-
21

---

Under review as a conference paper at ICLR 2023
ii) The Fenchel conjugate of the function g writes
Vu VOER g"(v) -
1 |BB)
(49)
A2
Lemmas 4 i) and 4 (i) yields Proposition 3.
Proof of Lemma 4 i).
The Lagrangian Problem (47) writes:
m
L(W, (.W. E 9(W Σ &
&:
W F. Wt F.: Ya) (50)
0gL(W, M. A, 0 yields vi
[n]] Σ |=|
Au 1. Then the Lagrangian rewrites
min min L(W, A) min W
Σ
W
E W Di: F. Wt. F. Ya)Aa
&
i=1
i=1
min g W
F Y W. ΣΣ Y.A
W
==1
1=1
t=
(IY-A)TF,WW.
=-9 (A)
Then the dual problem writes:
min Σ
g" ((Y - A) F.) + (Y,A)
(51)
A
==1
S. [
Σ Aa = vie In [k], Aa
(52)
Proof of Lemma 4 ii). Let h ||u||2 x uu The proof of Lemma 4 i) is done using the following steps.
Lemma 3. i) (T) = + 1101 (11111101 11112) (0/K)).
ii) (0111))))) (w) 51 zz BBST( (Ke, 1)1l?
Proof of Lemma 4 i). With n = /
A1 the Fenchel transform of w + ||00|22 + x||uo||?
h(u) ||u/|2 1/
h" (v) = sup (u T w || - (
2n || + sup ( [ - w/> ||l||2)
L ||/1 inf ( w/>13 + [|00|12)
Zi
zu || (1111301111)0///>)
22

---

Under review as a conference paper at ICLR 2023
Proof of Lemma 4 ii).
(-11))))1)))))
(31111201-112)
(2N 11-112 + LB, (0)
(v w 2x 1
sup
11w||>11
$ ||o/|1
+ sup
AO
w
2
| 1=11
IBBT 1)112
2x
g(()) /
A1 |BST(u/A1, 1)ll
2K
X
|BBS(uu/ /1, 1)
2\2
1
|BST 11)
A2
D EXPERIMENTAL DETAILS
D.1 DISENTANGLED REPRESENTATION COUPLED WITH SPARSITY REGULARIZATION IMPROVES GENERALIZATION
We consider the following data generating process: We sample the ground-truth features fa(x) from a Gaussian distribution (0, Σ) where Σ E IRmx and Σ,; 0.9 6ー01, Moreover, the labels are given by y = w fo (a:) + where un E 1Rm 6 (N00, 0.04) and m = 100. The ground-truth weight vector w is sampled once from (0, [mxm) and mask some its components to zero: we vary the fraction of meaningful features (f/m) from very sparse (f/m 5%) to less sparse (L/m 80%) settings. For each case, we study the sample complexity by varying the number of training samples from 25 to 150, but evaluating the generalization performance a larger test dataset (1000 sam- ples). To generate the entangled representations, we multiply the true latent variables fa()) by a randomly sampled orthogonal matrix L, i.e., fœ( [f)( For the disentangled representa- tion. we simply consider the true latents, i.c. fØ(x) == fe(x). Note that in principle we could have considered invertible matrix L that a component-wiss rescaling for the disentangled representation. The advantage of not doing so and opting for our approach is that the conditioning number of the covariance matrix of f is the same for both the entangled and the disentangled, hence offering a fairer comparison.
orthogonal for the linearly entangled representation and
For both the case of entangled and disentangled representation, we solve the regression problem with Lasso and Ridge regression, where the associated hyperparameters (regularization strength) were inferred using 5-fold cross validation on the input training dataset. Using both lasso and ridge regression would help us show the effect of encouraging sparsity.
In Figure for the sparsest case (f/m- = 5%), we observe that that Disentangled-Lase approach has the best performance when we have less training samples, while the Entangled-Lasso approach performs the worst. As we increase the number of training samples, the performance of Entangled- Lasso approaches that of Disentangled Lasso, Lasso, however, learning under the Disentangled-Lasse ap- proach is sample efficient. Disentangled-Lon Lasso obtains R² greater than 0.5 with only 25 training samples, while other approaches obtain R² close zero. Also, Disentagled-Lasse converges the
23

---

Under review as a conference paper at ICLR 2023
optimal R using only 50 training samples, while Entangled-Lasse does the same with 150 samples samples.
Note that the improvement due disentanglement does not happen for the of ridge regression as expected and there no of difference between the methods Disentangled Ridge and Entangled- Ridge because the L.2 norm is invariant orthogonal transformation. Also, having sparsity in the underlying task important. Disentangled-L shows the max improvement for the case of f/m 5%, with the gains reducing as we decrease the sparsity in the underlying task (l/m 80%).
D.2 SEMI-REAL EXPERIMENTS ON 3D SHAPES
*- inner-Lasso
inner-Ridge - inner-Ridge + ICA
Correlation 0.0
Correlation = 0.9
Multiple correlation values
0.0
0.01
0.03
0.1
0.3
1.0
0.0
0.01
0.03
0.1
0.3
1.0
0.0
0.5 Correlation
1.0
X/ A/max
A/Amxx
Noise scale
0
Noise scale
Multiple noise values
0.8
0.4
0.0
0.01
0.03
0.1
0.3
1.0
0.0
0.01
0.03
0.1
0.3
1.0
0.25
0.5
0.75
X/Auax
A/Auax
Noise scale
Figure 6: Prediction performance (R Score) for inner-Lasso, Diner-Ridge and inner-Ridge combined with ICA function of the regularization parameter (left and middle). Varying level of correlation between latents (top) and noise on the latents (bottom). The right columns shows performance of the best hyperparameter for different values of correlation and noise levels.
D.2.1 DATASET GENERATION
Details 3D Shapes. The 3D Shapes dataset (Burgess & Kim, 2018) contains synthetic images of colored shapes resting in a simple 3D scene. These images vary across 6 factors: Floor hue (10 values linearly spaced in [0, 1]); Wall hue (10 values linearly spaced in [O, 1)); Object hue (10 values linearly spaced in [O, I]); Scale (8 values linearly spaced in [O, I]); Shape (4 values in [0, 1, 2. 3)); and Orientation (15 values linearly spaced in [-30, 30)). These are the factors we aim to disentangle. We standardize them to have mean 0 and variance 1. We denote by Z C R6, the set of all possible latent factor combinations. In our framework, this corresponds the support of the ground-truth features (ar).. We note that the points in Z are arranged in grid-like fashion in R
Task generation. For all tasks the labelled dataset D, ((ar(ti)), (())))) is generated by first sampling the ground-truth latent variables (*,,)) (e((()))) iidd. according to some dis- tribution p(z) over Z, while the corresponding input is obtained doing (t.1) fo (2((0)) (fo invertible 3D Shapes). Then, sparse weight vector ( sampled randomly by doing (t) (t()) @ (t),, were is the Hadamard (component-wise) product, (()) ./(0, /) and s E [0, 13 is a binary vector with independent components sampled from a Bernoulli distribution with (p = 0.5). Then, the labels are computedfor each example as ((..)) == (t)) (8.6) + c(t,))
where (t.i) is independent Gaussian noise. In every tasks, the dataset has size n = 50. New tasks generated continuously
train. Figures 4 and 6 explores various choices of p(z). i.e. by varying the level of correlation between the latent variables and by varying the level of noise on the ground-truth latents.
24

---

Under review as a conference paper at ICLR 2023
Noise latents. To make the dataset slightly more realistic, get rid of the artificial grid-like structure of the latents by adding noise to it. This procedure transforms Z into a new support Z where is the noise level. Formally, Za Uzez(z uz} where the u₂ are iid samples from the uniform over the hypercube
A3
Az1 2
A a
A a
A 2
A66
x
a
0
2
2
2
where Az; denotes the gap between contiguous values of the factor 2;. When a 0. no noise is added and the support 2 is unchanged, i.e., 31- Z. As long as a E [0, 1], contiguous points in Z cannot be interchanged in a.. We also clarify that the ground-truth mapping fo is modified to fo.. consequently: for æ E X. fe.a := fo (+) + uz We emphasize that the u are sampled only once such that (z) is actually deterministic mapping.
Varying correlations. To verify that our approach is robust to correlations the latents, we con- struct p(z) follows: We consider a Gaussian density centered at 0 with covariance Σ := + i(i = 5) - p). Then, we evaluate this density on the points of Z,, and renormalize to have well-defined probability distribution over Z,,. We denote by pz, the distribution obtain by this construction.
In the top rows of Figures 4 and 6, the latents are sampled from pa=1,p(z) and p varies between 0 and 0,99. the bottom rows of Figures and 6, the latents are sampled from (0)) and a varies from 0 to 1.
D.2.2 METRICS
We evaluate disentanglement via the mean correlation coefficient (Hyvärinen & Morioka, 2016: Khemakhem et al., 2020a) which is computed as follows: The Pearson correlation ma- trix C' between the ground-truth features and learned ones is computed. Then, MCC max Epemuuatttsiiions in - (C,s()))- We also evaluate linear equivalence by performing linear re- gression to predict the ground-truth factors from the learned ones, and report the mean of the Pear- son correlations between the ground-trute truth latents and the learned ones. This metric is known as the coefficient of multiple correlation, R, and turns out to be the square-root known coefficient of determination, R The advantage of using R over R2 is that we always have MCC < R.
the more widely
D.2.3 ARCHITECTURE. INNER SOLVER & HYPERPARAMETERS
We use the four-layer convolutional neural network typically used in the disentanglement litera- ture (Locatello et al., 2019). As mentioned in Section 2.2.2, the norm of the representation fæ) must be controlled make sure the regularization remains effective. To do so, we apply batch nor- malization (loffe & Szegedy, 2015) the very last layer of the neural network and do not learn its scale and shift parameters. Empirically, we do see the expected behavior that, without any normal- ization, the norm of fa()) explodes as we train, leading to instabilities and low sparsity.
In these experiments, the distribution p(y; 17) used for learning is a Gaussian with fixed variance. In that case, the inner problem of Section 2.2.2 reduces Lasso regression. Computing the hy- pergradient w.r.t requires solving this inner problem. To do so, we use Proximal Coordinate Descent (Tseng, 2001; Richtárik & Takáč, 2014).
In Figures and 6, we explore various levels of regularization x. In our implementation inner- Lasso, Amax F ylloo where F E R**m is the design matrix of the features of the samples of task, while in the inner-Ridge implementation, Amax == 1 F1
D.2.4 EXPERIMENTS VIOLATING ASSUMPTIONS
In this section, explore variations of the experiments of Section 4, but this time the assumptions of Theorem are violated.
Figure 7 shows different degrees of violation of Assumption 7. We consider the cases where == {[1,2), (3, 4), (5. (5.6)} (block size = 2). S == {{1,2,3) 3), (4, 5, 6}} (block size = 3) and S ((1,,,,3, 4,5, 6)) (block size 6). Note that the latter corresponds to having no sparsity
25

---

Under review as a conference paper at ICLR 2023
at all in the ground-truth model, iee. all tasks requires all features. The reader verify that these three cases indeed violate Assumption 7. In all cases, the distribution p(S) puts uniform mass over NO, I)
its support S. Similarly to the experiments from the main text, and ~ p(S) (s the binary representation of the set S). Overall, we can see that inner-L does perform well when Assumption is violated. For example, when there is sparsity at all (block size 6), inner-Lasso performs poorly and is even surpassed by inner-Ridge. Nevertheless, for mild violations (block size 2), disentanglement (as measured by MCC) remains reasonably high. We further notice that all methods obtain very good R score in all settings. This is expected in light of Theorem which guarantees identifiability up to linear transformation without requiring Assumption 7.
@ where
inser-Lasso
inner-Ridge
inner-Ridge + ICA
Block size
Block size
Block size
1.0
0.8
MCC
0.6
0.4
0.0
0.01
0.03
0.1
0.3
1.0
0.0
0.01
0.03
0.1
0.3
1.0
0.0
0.01
0.03
0.1
0.3
1.0
A/A AAausx
A/ Amaas
A/ Amaax
Block size =
Block size =
Block size =
1.0
2
0.5
0.0
0.0
0.01
0.03
0.1 A/A Amaxx
0.3
1.0
0.0
0.01
0.03
0.1
0.3
1.0
0.0
0.01
0.03
0.1
0.3
1.0
A/Aras
A/ Amass
Figure 7: Disentanglement (MCC, top) and prediction (R Score, bottom) performances for inner- Lasso, inner-Ridge and inner-Ridge combined with ICA function of the regularization param- eter. The metrics are plotted for multiple value of block size for the support. Block size = corresponds to no sparsity in the ground truth coefficients.
Figure 8 presents experiments that are identitical to those of Figure 4 in the main text, except for how is generated. Here, the components of are sampled independently according w, Laplace(μ = 0, b = 1). We note that, under this process, the probability that This means all features are useful and Assumption is violated. That being said, due to the fat tail behavior of the Laplacian distribution, many components of w will be close to zero (relatively to its variance). Thus, this can be thought of as a weaker form of sparsity where many features are relatively unimportant. Figure 8 shows that inner-Lasso can still disentangle very well. In fact, the performance very similar to the experiments that presented actual sparsity (Figure 4).
0 is zero.
D.2.5 VISUAL EVALUATION
Figures 9 to 12 show how various learned representations respond to changing single factor of variation in the image (Higgins al., 2017, Figure 7.A.B). We what was expected: the higher the MCC, the more disentangled the learned features appear, thus validating MCC as a good metric for disentanglement. See captions for details.
26

---

Under review as a conference paper at ICLR 2023
inner-Lasso
inner-Ridge
inner-Ridge + ICA
Correlation
Correlation 0.95
Multiple correlation values FEREERS
1.0
0.8
MCC
0.6
0.4
0.0
.03
0.3
0.0
0.3
0.6
A/Amax
N/Amax
Correlation
Correlation 0.0
Correlation 0.95
Multiple correlation values
1.0
0.5
0.0
0.0
01
.03
0.3
1.0
0.0
01
.03
0.1
0.3
1.0
0.6
0.8
1.0
A/Amax
A/Amax
Correlation
Noise scale
0
Noise scale
Multiple noise values
1.0
0.8
MCC
0.6
0.4
0.0
.01
.03
0.1
0.3
1.0
0.0
011
.03
0.1
0.3
1.0
0.25
0.5
0.75
A/Amax
A/Amax
Noise scale
Noise scale = 0
Noise scale
Multiple noise values
1
0.8
0.6
0.4
0.0
.01
0.1 A/Amax
.03
0.3
1.0
0.0
.01
.03
0.1 A/Amax
0.3
1.0
0
0.25 0.5 0.75
Noise scale
Figure 8: Same experiment as Figure but the task coefficient vectors are sampled from Laplacian distribution (instead of what was described in Appendix D.2.1). Performance is barely affected, showing some amount of robustness violations of Assumption 7.
D.2.6 ADDITIONAL METRICS FOR DISENTANGLEEMENT
We implemented metrics from the DCI framework (Eastwood & Williams, 2018) to evaluate dis- entanglement. 1) DCI-Disentanglemmet: How many ground truth latent components are related to particular component of the learned latent representation; 2) DCI-Completeness:: How many learned latent components are related to particular component of the ground truth latent represen- tation. Note that for the definition of disentangiement used in the present work Definition 1, we want both DCI-disentanglemen and DCI-completeness to high.
The DCI framework requires matrix of relative importance. In implementation, this matrix is the coefficient matrix resulting from performing linear regression with inputs as the learned latent representation fa(x) and targets the ground truth latent representation fø(æ), and denote the solution as the matrix W. Further, denote by / = |W] as the importance matrix, as I denotes the relevance of inferred latent fg (m) for predicting the true latent fo fx))
27

---

Under review as a conference paper at ICLR 2023
Now, for computing DCI-diseetanglemmnt, we normalize each row of the importance matrix f/.. by its sum so that it represents a probability distribution. Then disentanglement is given by x Σx 1 :[),;))) where H denotes the entropy of a distribution. Note that for the desired case of each ground truth latent component being explained by single inferred latent component, would have :[) = as we have a one-hot vector for the probability distribution. Similarly, for the case of each ground truth latent component being explained uniformly by all the inferred latents, (+, :]) would be maximized and hence the DCI score would be minimized. To compute the DCI-complessness, we first normalize each column of the importance matrix [[: by its sum so that it represents a probability distribution and then compute 1 X >; t 1 ((...)))
Figure 13 shows the results for the 3D Shapes experiments (Section 4) with the DCI metric to evaluate disentanglement. Notice that we find the same trend as we had with the MCC metric 4, that inner-Lasso more robust to correlation between the latent variables, and inner-Ridge + ICA performance drops down significantly with increasing correlation.
D.3 META-LEARNING EXPERIMENTS
Experimental settings. We evaluate the performance of our meta-learning algorithm based on a group-sparse SVM base-learner on the minilmageNet (Vinyals et al., 2016) dataset. Following the standard nomenclature in few-shot classification (Hospedales et al., 2021) with k-shot N-way, where N is the number of classes in each classification task, and & is the number of samples per class in the training dataset Dirain, we consider 2 settings: 1-shot 5-way, and 5-shot 5-way. Note that the results presented in Figure only show the performance on 5-shot classification. We use the same residual network architecture as in (Lee et al., 2019), with 12 layers and representation of size p = 1.6 10a
Even though we consider similar base-learner as MetaOptNet (Lee et al., 2019) (namely, a SVM), our control experiment with À 0 cannot be directly compared to the performance of the model reported in that prior work. The reason is that in order to control for any other sources of "effective regularization" (e.g., data augmentation, label smoothing), we do not include the modifications made in MetaOptNet improve performance. Moreover, used different solver (proximal block- coordinate descent, as opposed to a QP solver) to solve the inner problem Problem (6).
Generalization meta-training tasks. In Section 2.3, argued that evaluating the performance of the learned representations on meta-training tasks (i.e., tasks similar to the ones seen during meta- training) still shows the generalization capacity to new tasks. Indeed, those new tasks which evaluate performance were created using the same classes as the tasks used during meta-training, but using combination of classes that may have not been seen in any tasks used for optimizing Problem (5). However, evaluation in meta-learning is typically done on meta-test tasks, i.c. tasks based on concepts that were never seen by any task during meta- vetaattaiiinng This evaluation requires stronger notion of generalization, closer to diittii-iitiiioo generalization.
Base-learner
5-way 1-shot
5-way 5-shot
SVM (A = 0)
53.29 = 0.60%
69.26 0.51%
Group-sparse SVM (A = 0.01)
54.22 = 0.61%
70.01 + 0.51 %
MetaOptNet (Lee al., 2019)
64.09 = 0.62%
80.00 + 0.45%
Table 2: Performance of our meta-learning algorithm on the minilmageNet benchmark. The perfor-
mance is reported as the mean accuracy and 95% confidence interval on 1000 meta-test tasks. We also report the performance of MctOptNet (Lee et al., 2019) as reference, even though the perfor- mance not directly comparable to our SVM baseline (see text for details).
Nonetheless, we observe in Table 2 that the performance of the meta-learning method improves as the base-leamners are group-sparse.
28

---

Under review as a conference paper at ICLR 2023
2
Figure 9: Varying one factor at a time in the image and showing how the learned representation varies response. This representation was learned by inner-Lasso (best hyperparameter) on dataset with 0 correlation between latents and noise scale of 1. The corresponding MCC is 0.99. We can see that varying a single factor the image always result in changing a single factor in the leamned representation.
29

---

Under review as a conference paper at ICLR 2023
Varying one factor at a time the image and showing how the learned representation varies in response. This representation Rnner-Ridge without regularization coefficient equal to zero) on a dataset with O correlation between and noise scale of 1. The corresponding MCC is 0.63. We the image result in changing multiple factors in the learned representation, i.e. the representation not disentangled.
Figure 10:
learned without regularization of any kind (i.c. with
that varying single factor in
30

---

Under review as a conference paper at ICLR 2023
Figure 11: Varying one factor at a time in the image and showing how the learned representation varies response. This representation was learned with inner-Lasso (best hyperparameter) on dataset with correlation 0.9 between latents and noise scale of 1. The corresponding MCC is 0.96. Qualitatively, the representation appears to be well disentangled, but not as well as Figure (reflected by drop in MCC of 0.03).
31

---

Under review as a conference paper at ICLR 2023
2-
Varying one factor at time in the image and showing how the learned representation varies in response. This representation was learned with inner-Ridge (best hyperparameter) on dataset with correlation 0.9 between latents and noise scale of The corresponding MCC is 0.79. For most latent factors, we cannot identify a dominant feature, except maybe for background and object colors. The representation appears more disentangled than Figure 10, but less disentan- gled than Figure 11, as reflected by their corresponding MCC values.
Figure 12:
32

---

Under review as a conference paper at ICLR 2023
inner-Lasso
inner-Ridge
inner-Ridge ICA
DtlDDisettnglemmen
Correlation
0.0
Correlation
0.9
Multiple correlation values
0.5
0.0
0.0
0.01
0.03
1.0
0.0
0.01
0.03
0.1
1.0
A/max
Correlation
VCl-Completeness
Correlation
0.0
Correlation
0.9
Multiple correlation values
0.5
F1
0.0
0.0
0.01
0.03
0.1
1.0
0.0
0.01
0.1
1.0
A/ Amax
A/ Arnox
Correlation
Figure 13:
Disentanglement performance (DCI) for inner-Lasso, inner-Ridge and inner-Ridge com- bined with ICA as a function of the regularization parameter (left and middle). The right columns shows performance of the best hyperparameter for different values of correlation and noise. The top row shows the results for the disentanglement metric of DCI and the bottom row shows the results for the completeness metric of DCI.
33


Reviews:
[{"review_id": "aoJJehZ0Gt", "reviewer": "Reviewer_iXc7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors prove a new identifiability result that provides conditions under which maximally sparse base-predictors yield disentangled representations. Based on this theoretical result, they propose a practical approach to learn disentangled representations based on a sparsity-promoting bi-level optimization problem. The authors also explore a meta-learning version of the algorithm based on group Lasso multi-class SVM base predictors. ", "summary_of_the_review": "The theory developed in the paper is relatively novel. The paper could be stronger with additional experimental results on disentanglement. ", "strength_and_weaknesses": "Strength:\nThe authors developed theoretical studies and results with extensive mathematical proof. The theory in the paper is relatively novel. \n\nWeakness:\n1. The experiments on disentanglement are insufficient to support the claims.  The authors need to visually show the disentanglement effects regarding different latent factors.  Otherwise, we cannot say the learned representations are meaningful. \n\n2. Only one disentanglement metric was used in the paper, the author could use multiple disentanglement scores to validate the results. \n\n3. The author could present additional experimental results on how disentanglement benefits prediction or other tasks, and it could make the paper stronger. \n\n4. The notations of different variables are not easy to follow. The author could list them in a table to improve readability. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KkoE_EhiK1", "reviewer": "Reviewer_wTbQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the disentangled representation learning. In particular, authors provide the evidence on the improved generalization when the disentangled representations  coupled with sparse base-predictors are learned. The paper then presents a theoretical result on identifiability condition wonder with maximally sparse base-predictors that lead to the desired disentangled representations. Relying on the presented theory, representation that maximizes sparsity is sought, in a hope of eventual disentanglement, in the context of the multi-task learning. Such sparsity maximization however not being trivial, authors propose a sparsity-promoting bi-level optimization paradigm for learning. Later, a connection between bi-lavel optimization and the meta-learning problems is established.  \n", "summary_of_the_review": "Overall, the paper makes a good case for the addressed problems and provides convincing and theoretical insights. The paper can benefit from more intuitive explanations (considering a broader audience) and the experiments on more realistic datasets.", "strength_and_weaknesses": "++The paper is well written and motivated. The addressed problem is a problem of sufficient interest. \n\n++The theoretical consideration of the paper is praiseworthy. Specifically, identifiability result  of Theorem 1 is particularly interesting, intuitive, and convincing. \n\n++The paper also proposes bi-level optimization and meta-learning based approaches, which are empirically shown to be effective in Figure 1,  with fewer samples for the Disentangled-Lasso method.\n\n++Supplementary material with additional theoretical and experimental analysis is helpful.\n\n--The reported experiments are rather toy-like settings. It is not clear when the proposed method is expected to be useful. For example, results reported in Table.1 (supplementary material) shows that the proposed method does not compete well against MetaOptNet (although not being directly comparable).\n\n--The meta-learning formulation of Section 3.2 is not easy to follow. The section lacks intuitive explanation and the jump form equation (7) to Proposition (3) is not well explained.\n\n--The paper relies a lot on the supplementary material. This in itself is not a big problem, given the nature of the addressed problem. However, I believe that the paper can be made more intuitive with the help of graphic illustrations and by bringing the B.2 Discussion of Assumption in the main paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FeV78ti1Lny", "reviewer": "Reviewer_GZww", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study how disentangled representations can be beneficial for downstream tasks under certain assumptions. The authors come up with Theorem 1, which states that leveraging multi-task learning can naturally learn disentangled representation when regularizing the base-predictors to achieve maximal-level sparsity. The authors propose to solve the problem of spare multi-task learning via solving its relaxed problem. The authors show empirical evidence on few-shot learning to support their theorem and to validate the effectiveness of their method. ", "summary_of_the_review": "Overall, I think this is an interesting paper which aims to thoroughly understand the relationship between disentanglement, sparsity and generalization. The authors tried to formalize a few intuitions, propose a few theoretical results and prove them under the dome of formalized assumptions. I think the paper shed some light on better understanding disentangled representations. It’s unclear to me how strong the theoretical results are, how good the methods would be on large realistic datasets.", "strength_and_weaknesses": "Strength:\n\n-- The paper formalizes the sparse task assumption (I.e., most tasks do not require all features), and introduces a novel identifiability result. The authors validate that disentanglement can be achieved by combining with sparse base-predictors, and can improve generalization, via both empirical evidence and the theories (under certain assumptions).\n\n-- The theoretical analysis shed light on better understanding disentangled representations.\n\n\nOverall, The authors could consider conducting experiments on more tasks and more datasets. I did realize some potential weaknesses on motivation and experiments:\n\n-- Do the authors want to show disentanglement helps with generalization, or sparsity regularization helps with generalization? The authors can consider comparing with the disentanglement methods that do not combine with sparse base-predictors. \n\n-- The authors propose Theorem 1 (and other theoretical results) under certain assumptions. I’m not sure how realistic those assumptions are. According to Figure one, the disentangled methods would work well under two assumptions — The ground-truth coefficient matrix (See equation 2) should be very sparse, and there should be very few training samples. Without the two conditions, combining sparsity and disentanglement do not show clear benefits.\n\n-- It seems most theorems and propositions are based on MLE, while the meta learning part switches to SVM. \nI don’t know how to understand the significance of the benefits of the group-sparse SVM method (when compared with SVM). For example, lambda/lambda_max = 0.01 yields the best performance (according to Figure 3 and Table 1 in the Appendix). In most configurations, the group-sparse solutions do not seem to outperform the vanilla SVM. In the configuration of 0.01, the benefit in terms of Meta-test is also not so big.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2d7cKrZ1aq", "reviewer": "Reviewer_5SbL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present technical results showing that disentangled representations improve generalization if base predictors are sparse (they only use a limited number of features from the representation). Building on this, the authors propose an algorithm that learns disentangled representations (by encouraging base-predictor sparsity) and show its effectiveness on semi-synthetic data and on meta-learning. ", "summary_of_the_review": "The authors present a series of technical and empirical results that explain the practical benefit of disentangled representations if the base predictors are sparse. I am not an expert of the research area, but this appears to be an interesting and insightful paper. It would be nice to see the usefulness of the proposed approach on additional benchmarks.", "strength_and_weaknesses": "The series of technical results is definitely interesting: the authors start from the assumption that only few features of the representation are useful for prediction. Building on the assumption, they provide insight on why learning disentangled representations is beneficial to downstream performance. Finally, an algorithm is proposed to capitalize on the gained insight (and is made practical, by operating in the dual space, for the meta-learning case).\nThe results would appear to be adequately backed by relevant semi-synthetic experiments and on meta-learning.\n\nIt would be nice, as an add-on, if the authors could provide evidence that the proposed approach is beneficial in a larger variety of contexts. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject