Leveraging sparse and shared feature activations for disentangled representation learning
Authors: Marco Fumero
Florian Wenzel
Luca Zancato
Alessandro Achille
Emanuele Rodolà
Stefano Soatto
Bernhard Schölkopf
Francesco Locatello
Venue: NeurIPS 2023 spotlight (2023)
Keywords: disentanglement
OOD generalization
multitask learning

Abstract: Recovering the latent factors of variation of high dimensional data has so far focused on simple synthetic settings. Mostly building on unsupervised and weakly-supervised objectives, prior work missed out on the positive implications for representation learning on real world data. In this work, we propose to leverage knowledge extracted from a diversified set of supervised tasks to learn a common disentangled representation. Assuming each supervised task only depends on an unknown subset of the factors of variation, we disentangle the feature space of a supervised multi-task model, with features activating sparsely across different tasks and information being shared as appropriate. Importantly, we never directly observe the factors of variations but establish that access to multiple tasks is sufficient for identifiability under sufficiency and minimality assumptions.
We validate our approach on six real world distribution shift benchmarks, and different data modalities (images, text), demonstrating how disentangled representations can be transferred to real settings.

Leveraging sparse and shared feature activations for disentangled representation learning
Marco Fumero
Florian Wenzel Amazon AWS
Luca Zancato Amazon AWS
Alessandro Achille Amazon AWS
Sapienza, University of Rome
Emanuele Rodolà Sapienza, University of Rome
Stefano Soatto
Bernhard Schölkopf
Francesco Locatello IST Austria
Amazon AWS
Amazon AWS
Abstract
Research on recovering the latent factors of variation of high dimensional data has so far focused on simple synthetic settings. Mostly building on unsupervised and weakly-supervised objectives, prior work missed out the positive implications for representation learning on real world data. In this work, we propose to leverage knowledge extracted from diversified set of supervised tasks to learn a common disentangled representation. Assuming that each supervised task only depends on unknown subset of the factors of variation, we disentangle the feature space of supervised multi-task model, with features activating sparsely across different tasks and information being shared as appropriate. Importantly, we never directly observe the factors of variations, but establish that access to multiple tasks is sufficient for identifiability under sufficiency and minimality assumptions. We validate our approach six real world distribution shift benchmarks, and different data modalities (images, text), demonstrating how disentangled representations can be transferred to real settings.
Introduction
A fundamental question in deep learning is how to learn meaningful and reusable representation from
high dimensional data observations [8] 75 78 77). core area research pursuing centered on where the aim is to learn a representation which recovers the factors of variations (FOVs) underlying the data distribution. Disentangled representations expected to contain all the information present in the data in compact and interpretable structure [46] [16] while being independent from a particular task [29]. has been argued that separating information into interventionally independent factors (78) can enable robust
disentangled representation learning (DRL) [6
3 33
downstream predictions, which was partially validated in synthetic settings 19, 58 Unfortunately,
these benefits did not materialize in real world representations learning problems, largely limited by lack of scalability existing approaches.
In this work we focus on leveraging knowledge from different task objectives to learn better represen- tations of high dimensional data, and explore the link with disentanglement and outti--distributtin (OOD) generalization tasks are indeed expected to be richer and generalize better to new, possibly antoof-distributinn tasks. However, this is not always the weaker models. This phenomenon, known as negative transfer [6] 9T in the context of transfer learning task competition used to learn two different tasks that require expressing high feature variability and/or coverage. Aiming the same features for different objectives makes them noisy and often increases the sensitivity to spurious correlations [35 27 7. as features can be both predictive and detrimental for
real data distributions. Representations learned from large diversity of
different tasks compete with each other and lead to
(83 in multitask learning, happens when limited capacity model is
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

different tasks. Instead, we leverage diverse set of tasks and assume that each task only depends on an unknown subset of the factors variation. We show that disentangled representations naturally emerge without any annotation of the factors of variations under the following two representation constraints:
Sparse sufficiency: Features should activate sparsely with respect to tasks. The representation is sparsely sufficient in the sense that any given task can be solved using few features.
Minimality: Features are maximally shared across tasks whenever possible. The representa- tion is minimal in the sense that features are encouraged to be reused, i.e., duplicated or split features are avoided.
These properties are intuitively desirable to obtain features that (i) are disentangled w.r.t. to the factors of variations underlying the task data distribution (which we also theoretically argue in Proposition 2.1. (ii) generalize better in settings where test data undergo distribution shifts with respect to the training distributions, and (iii) suffer less from problems related to negative transfer phenomena. To learn such representations in practice, we implement a meta learning approach, enforcing feature sufficiency and sharing with a sparsity regularizer and an entropy based feature sharing regularizer, respectively, incorporated in the base learner. Experimentally, show that model learns meaningful disentangled representations that enable strong generalization on real world data sets. Our contributions can be summarized as follows:
We demonstrate that is possible to learn disentangled representations leveraging knowledge from distribution of tasks. For this, we propose a meta learning approach learn a feature space from a collection of tasks while incorporating our sparse sufficiency and minimality principles favoring task specific features to coexist with general features.
Following previous literature, test approach synthetic data, validating in ideal- ized controlled setting that our sufficiency and minimality principles lead to disentangled features w.r.t. the ground truth factors of variation, as expected from our identifiability result in Proposition 2.1
We extend our empirical evaluation to non-synthetic data where factors variations are not known, and show that approach generalizes well out-of-distribution different domain generalization and distribution shift benchmarks.
2 Method
Given a distribution of tasks T and data yt) P. for each task t. we aim to learn disentangled representation gxx) E 2 S RMM,, which generalizes well to unseen tasks. We learn this representation g by imposing the sparse sufficiency and minimality inductive biases.
2.1 Learning sparse and shared features
Our architecture (see Figure M is composed of a backbone module go that is shared across tasks and a separate linear classification head fø., which is specific to each task l. The backbone is responsible to compute and learn a general feature representation for all classification tasks. The linear head solves a specific classification problem for the task-specific data (xt, Vr) P; in the feature space 2 while enforcing the feature sufficiency and minimality principles. Adopting the typical meta-learning setting (34), the backbone module go can be viewed as the meia learner while the task-specific classification heads Se, be viewed as the base learners. In the meta-learning setting we assume to have access to samples for a new task given by a support set U, with elements (x"" y
U. These samples are used to fit the linear head fa- leading to the optimal feature weights for the given task. For query x Q, the prediction is obtained by computing the forward pass ; = fe- (go ())).
Enforcing feature minimality and sufficiency. To solve task in the feature space 2 of the backbone module we impose the following regularizer Reg(ø) on the classification heads fo with parameter E X'xMxX where T is the number tasks, M the number of features, and C the number of classes. The regularizer is responsible for enforcing the feature minimality and sufficiency

---

Limner
Leunter
x
ge
2
x²
22
Sa
Figure 1: Model scheme: Illustrations of the (Top) the inner loop stage and outer loop following the steps of the algorithmic procedure described Section B.1 in the Appendix.
properties. composed of the weighted sum of sparsity penalty RegLi and an entropy-based feature sharing penalty: Regsharing
Reg(Ø) aRgg,, (0) + &Regsharingg (0),
with scalar weights a and B. The penalty terms are defined by:
RegL, (0)
1
Σ lrrm.el
TC
t,c,m
Regsharing (0) H(Ōm))
Σ @mlog(@m)
(3)
where 5m 1o Ei. mm TC Erc.. ldc.e.m are the normalized classifier parameters. Sufficiency enforced by sparsity regularizer given by the Lı-norm, which constrains classification head to only sparse subset of the features. Minimality is enforced by the feature sharing term: minimizing the entropy of the distribution of feature importances (i.e. normalized |or)) averaged across mini batch of T tasks, leads to a more peaked distribution of activations across tasks. This forces features to cluster across tasks and therefore be reused by different tasks, when useful. We remark that different choices for the regularizers coming from the linear multitask learning literature (e.g. [59] 39 38) to enforce sparse sufficiency and minimality are indeed possibile. We leave their exploration as future direction.
2.2 Training method
We train the model in meta-learning fashion by minimizing the test eπrоr over the expectation of the task distribution t ~ T. This be formalized bi-level optimization problem. The optimal backbone model ga- is given by the outer optimization problem:
min Ex Lootee (Jor (go(x))) ))},
where Jer are the optimal classifiers obtained from solving the inner optimization problem, and (x, 21)) E Qe are the test (or query) datum from the query set Q2 for task t. Let U, be the support set with samples (xy w/()) E for task i, where typically the support set is distinct from the query set, i.e., U n Q = (.. The optimal classifiers for are given by the inner optimization problem:
min T Cimer (D! v Reg(o),
(5)
where 8.'² (x( For both the inner loss Linner and outer loss Louter we use the cross entropy loss.
Task generation. Our method can be applied in a standard supervised classification setting where we construct the tasks on the fly as follows. We define task / as C-way classification problem. We first select a random subset of classes from a training domain Derasn which contains K Kerain classes. For each class consider the corresponding data points and select random support set Ue with elements (xV' / E U and disjoint random query set Q2 with elements (x² .y Qt.
Algorithm. In practice we solve the bi-level optimization problem (4 and (5) as follows. In each iteration we sample a batch of 7 tasks with the associated support and query set as described above. First, we use the samples from the support set S, to fit the linear heads fo by solving the inner optimization problem (5) using stochastic gradient descent for fixed number of steps. Second,

---

use the samples from the query set Q₁ to update the backbone ge by solving the outer optimization problem 4 using implicit differentiation [T]. Since the optimal solution of the linear heads ø* depend on the backbone go, straightforward differentiation w.r.t. 0 ís not possible. We remedy this issue by using the approximation strategy of [88] to compute the implicit gradients. The algorithm
is summarized in section B.T of the Appendix.
2.3 Theoretical analysis
We analyze the implications of the proposed minimality and sparse sufficiency principles and show in a controlled setting that they indeed lead to identifiability. As outlined in Figure we assume that there exists set of independent latent factors 2 ~ II p(2;) that generate the observations via an unknown mixing function x = g (z). Additionally, we assume that the labels y/s for a task & only depend on a subset of the factors indexed by S. P(S), where S is an index set on z E Z, via some unknown mixing function yt- - fr (z) (potentially different for different tasks). We formalize the two principles that imposed on f* by:
1. sufficiency: ft- = fi" s. for St ~ ppS)
2. minimality: AS' + St C S s.t. fils = f2.
where fls, denotes that the input to a function f is restricted to the index set given by S, (all remaining entries are set to zero). (I) states that ft only uses a subset of features, and (2) states that there are not be duplicate features.
Proposition 2.1. Assume that g" is diffeomorphism (smooth with smooth inverse), f satisfies the sufficiency and minimality properties stated above, and p(S) satisfies: p(S D (i)) > 0 or p(i (S) - (S' n S)) >
Observing unlimited data from p(X, Y), it is possible to recover a representation 2 that is an axis aligned, component wise transformation of
Overall, we see this proposition validation that in idealized setting inductive biases are sufficient to recover the factors of variation. Note that the proof is non-constructive and does not entail specific method. In practice, we rely on the same constraints inductive biases that lead to this theoretical identifiability and experimentally show that disentangled representations emerge in controlled synthetic settings. On real data, (1) we cannot directly measure disentanglement, (2) notion of global ground-truth factors may even be ill-posed, and (3) the assumptions of Proposition [2. are likely violated. Still, sparse sufficiency and minimality yield some meaningful factorization of the representation for the considered tasks.
Remarks:
Relation to 47 and (58): Our theoretical result can be reconnected with concurrent work 47 and can be seen as a corollary with a different proof technique and slightly relaxed assumptions. The main difference is that our feature minimality allows of factors of variations is unknown, which we found critical in real world data sets (the main focus of our paper). Instead, they only assume sparse sufficiency, which is enough for identifiability if the ground-truth number of factors is known, but is not enough to recover high disentaglement when this is not the case (see Figure 3 and does not translate well to real data, see Table 16 with the empirical comparison in Appendix D.8. Interestingly, their analysis also hints at the fact that our approach also benefits in terms of sample complexity on transfer learning downstream tasks. Our proof technique follows the general construction developed for multi-view data in (58), adapted to our different setting. Instead of observing multiple views with shared factors of variation, we observe single task that only depend on a subset of the factors.
also cover the case where the number
3 Related work
Learning from multiple tasks and domains. Our method addresses the problem of learning
general representation across multiple and possibly unseen tasks [15] 103) and environments [105]
32
44
97
63
94
64 that may be competing with each other during training [6 91 83). Prior
research tackled task competition by introducing task specific modules that do not interact during
training [67 101
80
While successfully learning specialized modules, these approaches can not
leverage synergistic information between tasks, when present. On the other hand, our approach is closer to multi-task methods that aim at learning generalist model, leveraging multi-task interactions [106]] 5]. Other approaches that leverage meta-learning objective for multi-task learning have been

---

Z
g*
f* St
S
T
Figure 2: Assumed causal generative model: the gray variables are unobserved. Observations x are generated by some unknown mixing of set of factors of variations z. Additionally, we observe distribution of supervised tasks, only depending on subset of factors of variations indexed by S.
particular, [50] proposes learn a generalist model in few-shot learning setting without explicitly favoring feature sharing, nor sparsity. Instead, we rephrase the multi-task objective function encoding both feature sharing and sparsity to avoid task competition.
formulated [8
50 g
Similar to prior work in domain generalization, we assume the existence of stable features for a given task [64] 86 40, [0] and amortize the learning over the multiple environments. Differently than prior work, we do not aim to learn an invariant representation priori. Instead, we learn sufficient and minimal features for each task, which are selected at test time fitting the linear head on them. In light of [22] one can interpret our approach as learning the final classifier using empirical risk minimization but over features learned with information from the multiple domains.
Disentangled representations. Disentanglement representation learning g 331 aims at recovering the factors of variations underlying a given data distribution. (56) proved that without any form of supervision (whether direct or indirect) on the Factors of Variation (FOV) is not possible to
recover them. Much work has then focused on identifiable settings [58 25 from non-i.i.d. data,
even allowing for latent causal relations between the factors. Different approaches can be largely grouped in two categories. First, data may be non-independently sampled, for example assuming sparse interventions or a sparse latent dynamics (30) [55] [3 100 2 79 48). Second, data may be
-enidenticaaly distributed, for example being clustered in annotated groups [37
95 60). Our
method follows the latter, but we do not make assumptions on the factor distribution across tasks (only their relevance terms of sufficiency and minimality). This also reflected in our method, as we train for supervised classification opposed to contrastive or unsupervised learning as common the disentanglement literature. The only exception the work of [47 discussed in Section
Experiments
We start by highlighting here the experimental setup of this paper along with its motivation.
Synthetic experiments. We first evaluate our method on benchmarks from the disentanglement where we have access to ground-truth annotations and we can assess quantitatively how well we can learn disentangled representations. We further investigate how minimality and feature sharing are correlated with disentanglement measures (Section[4.1) and how well our representations, which are learned from a limited of tasks, generalize their composition. The purpose of these experiments is to validate our theoretical statement, showing that if the assumptions of Proposition 2.1 hold, our methods quantitatively recover the factors of variation.
literature 62 14 7I, 49
Domain generalization. On real data sets, we can neither quantitatively measure disentanglement nor are we guaranteed identifiability (as assumptions may be violated). Ultimately, the goal of disentangled representations is to learn features that lend themselves to be easily and robustly transferred downstream tasks. Therefore, with respect to downstream tasks subject to distribution shifts, where isolating spurious features was found improve generalization in synthetic settings [19 58 To assess how robust our representations are to distribution shifts, we evaluate our method on domain generalization and domain shift tasks on six different benchmarks (Section 4.2).. In a domain generalization setting, we not have access to samples coming from the testing domain, which is considered to be OOD w.r.t. to the training domains. However, in order to solve a new task, our method relies on set labeled data test time to fit the linear head on top of the feature space. Our strategy is to sample data points from the training
first evaluate the usefulness of our representations
5

---

distribution, balanced by class, assuming that the label set Y does not change in the testing domain, although its distribution may undergo subpopulation shifts.
Few-shot transfer learning. Lastly, we test the adaptability of the feature space to new domains with limited labeled samples. For transfer learning tasks, we fit linear head using the available limited supervised data. The sparsity penalty a is set to the value used in training; the feature sharing parameter / is defaulted to zero unless specified.
Experimental setting. To have fair comparison with other methods in the literature, we adopt the standard experimental setting of prior work [32 44 Hyperparameters and 3 are tuned performing model selection on validation set, unless specified otherwise. For comparison with baselines, we substitute our backbone with that of the baseline (e.g. for ERM models, we detach the classification head) and then fit a new linear head on the same data. The linear head module trained at test time on top of the features is the same both for our and compared methods. Despite its simplicity, we report the ERM baseline for comparison in our experiments in the main paper, since it has been shown to perform best in average on domain generalization benchmarks [32 44]. We further compare with
other consolidated approaches in the literature such as IRM 44. COR AL [85 and GroupDRO [73
and include a large and comprehensive comparison with 199 10
53 26
in
Appendix D.4 Experimental details are fully described in Appendix
4.1 Synthetic experiments
We start by demonstrating that our approach is able to recover the factors of variation underlying a synthetic data distribution like [62] For these experiments, we assume to have partial information on subset of factors of variation Z, and we aim to learn a representation 2 that aligns with them while ignoring any spurious factors that may be present. We sample random tasks from a distribution T (see Appendix C.3 for details) 5and focus on binary tasks, with Y = {0, 1). For the DSpri i tes dataset an example of valid task is "There is a big objecl on the left of the image". this case, the partially observed factors (quantized to only two values) are the x position and size. In Table we show how the feature sufficiency and minimality properties enable disentanglement in the learned representations. We train two identical models FOVs, showing that, for different datasets [62 14 49 [T],, the same model without regularizers achieves a similar in-distribution (ID) accuracy, but much lower disentanglement.
random distribution of sparse tasks defined on
We then randomly draw and fix 2 groups of tasks with supports S1, S2 (18 in total), which all have support on two FOVs, |S|| = =S21| = 2. The groups share one factor of variation and differ in the other one, i.e. S n S2 = (i} for some {i} Z. The data in these tasks are subject to spurious correlations, i.c. FOVs not in the task support may be spuriously correlated with the task label. We start from an overestimate of the dimension of ž of 6, trying recover z of size 3. We train our network to solve these tasks, enforcing sufficiency and minimality on the rep- resentation with different regularization degrees. In Figure 3 show how the alignment of the learned features with the ground truth factors of variations depend on the choice of a, a,3. going from no disentanglement (DC' = 27.8). to good alignment as we enforce more sufficiency and minimality. The model that attains the best aligmment (DCI = 98.8) uses both sparsity and feature sharing. Sufficiency alone (akin to the method of [7 is able to select the right support for each task, but features are split or duplicated, attaining lower disentanglement (DCT = 71.9). The feature sharing penalty ensures clustering
100
9 dE
80
20
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
B
Figure 3: Role of minimality: We plot the DCI metric of a set of models (red dots) trained on fixed tasks from DSprites: Training without regularizers leads to no disentangiement (green). Enforcing sparsity alone (yellow, akin to (47) achieves good disentanglement (DCI 71.9),
but some features may be split or duplicated. Enforeing both minimality and sparse sufficiency (magenta) attains the best DCT (98.8). When ,3 is too high (> 0.25) activated features collapses into few clusters with respect to tasks. For complete results and experiments on additional datasets see Table 8 and Figures 6. 7 in Appendix.

---

in the feature space w.r.t. tasks, ensuring to reach high disentanglement, although it may result in the failure cases, when 3 too high (5 > 0.25).
Table 1: Enforcing disentanglement: DCI [2] disentanglement scores and ID accuracy on test samples for model trained without enforcing sufficiency and minimality (top row), and model with the regularizers activated (bottom row). While attaining similar performance on accuracy, the model with the activated regularizer always show higher disentanglement. See Table for additional scores.
Dsprites
Dshapps
SmallNorb
Cars
No reg (DCI,Acc) (16.6,94.4)
(44.4,96.2)
(16.5,96.1)
(60.5,99.8)
a (DCL,Acc) (69.9.95.8)
(87.7, 95.8)
(55.8,95.6)
(92.3,99.8)
Disentanglement and minimality are corre- lated. In the synthetic setting, we also show
100
the role of the feature sharing penalty. Mini-
20
mizing the entropy of feature activations across mini-batches of tasks results in clusters in the feature space. We investigate how the strength of this penalty correlates well with disentangle- ment metrics [22] training different models on Dsprites which differ by the value of B3. For 15 models trained increasing 3 from to 0.2 linearly, we observe a correlation coefficient with the DCI metric associated to representa- tions compute by each model of 94.7, showing that the feature sharing property strongly encour- ages disentanglement. This confirms again that sufficiency alone (i.e. enforcing sparsity) is not enough to attain good disentanglement.
70
Accurscy
60
40
20
NoReg
55/2
[5|-3
[S|-4
[5|-5
Tack support
Figure 4: Task compositional generalization: Mean accuracy over 100 random test tasks reported for group of tasks of growing support (second, third, fourth column) for a model trained without inductive biases (blue, attaining DCI = 29.4) and enforcing them (orange, DCI 59.4). The latter show better compositional generalization resulting from the properties enforced on the representation. Exact values are reported in Table 9 in Appendix.
Task compositional generalization. Finally, evaluate the generalization capabilities of the features learned by our method by testing our model on set of unseen tasks obtained by combining tasks seen during training. To do this, we first train two models on the AbstractDspritte tasks, where we limit the support of each task to be within 2 (i.e. ||| = 2). The models differ in activating/dcuctivaiing the regularizers on the linear heads. Then, we test on 100 tasks drawn from distribution with increasing support on the factors of variation (|S) 3, S = 4, |S] 5), which correspond to composition of tasks in the training distribution; see Figure with the accompaning Table in Appendix D
dataset using a random distribution of
4.2 Domain Generalization
In this section evaluate our method benchmarks coming from the domain generalization field (32 93 70) and subpopulation distribution shifts [73 to show that a feature space learned with our inductive biases performs well out of real world data distribution.
Subpopulation shifts. Subpopulation shifts occur when the distribution of minority groups changes across domains. Our claim is that a feature space that satisfies sparse sufficiency and minimality is more robust to spurious correlations which may affect minority groups, and should transfer better to new distributions. To validate this, we test on two benchmarks Waterbirds [73] and divilComment s (see [CP.
For both, we use the train and test split of the original dataset. In Table last row, we report the results on the test set of Naterbi rás for the different groups the dataset (landbirds on land, landbirds water, waterbirds land, and waterbirds on water, respectively). We fit the linear head
7

---

Table Quantitative results for few-shot transfer learning, with our method consistently outperform- ing ERM across all sample sizes and data sets.
N-shot/Algorithm
OOD accuracy (averaged by domains)
1-shot
PACS
VLCS
OfficeHome
Waterbirds
ERM
80.5
59.7
79.8 56.4
Ours
81.5
68.2
58.4
88.4 4
5-shot ERM
87.1
71.7
75.7
79.8
Ours
88.3
74.5
77.0
87.6
10-shot ERM
87.9
74.0
81.0
84.2
Ours
90.4
77.3
82.0
89.2
Table 3: Quantitative evaluation on Camelyon17. we report aceuracy both on ID and OOD splits. Our approach achieves significantly higher validation and test OOD accuracy.
Validation(ID)
Validation (OOD)
Test (OOD)
ERM
93.2
84
70.3
CORAL
95.4
86.2
59.5
IRM
91.6
86.2
64.2
Ours
93.2 #0.3
89.9+0.6
74.1+0.2
on a random subset of the training domain, balanced by class, repeat 10 times and report accuracy and standard deviation on test. For EivilComents we report the average and worst accuracy in Figure where we compare with ERM and groupDRO [73]. While performing almost on par w.r.t. ERM, method is more robust spurious correlation in the dataset, showing the higher worst group accuracy. Importantly, we outperform GroupDRO, which uses information on the subdomain statistics, while we do not assume any prior knowledge about them. Results per group are reported in the Appendix (Table IT].
100
DomainBed. We evaluate the domain generalization performance on the PACS, VLCS and OfficeHome
20
datasets from the DomainBed [32] test suite (see Ap-
pendix I for more details). On these datasets, we
train N and leave out for testing. Reg- ularization parameters and | are tuned according validation sets of PACS, and used accordingly on the other dataset. For these experiments we use a Re ResNet pretrained bone, as done in (32) To fit the linear head we sam- ple 10 times with different samples sizes from the training domains and we report the mean score and standard deviation. Results are reported in Table 4 showing how enforcing sparse sufficiency and mini- mality leads consistently to better OOD performance. Comparisons with 13 additional baselines is in Ap- pendix D.4
ERM
30-
DRO
20
10
Imagenet [17] as back-
Avg graup
Figure 5: Quantitative results on CivilCom- ments: we report the accuracy on test av- eraged across all demographic groups (left group), and the worst group accuracy, on the right. Our method (green) performs similarly in terms of average accuracy and outperforms in terms of worst group accuracy, without us- ing any knowledge on the group composition in the training data. For exact values and error estimates, see Table 10 in the Appendix.
Camelyon17. The model is trained according to the original splits in the dataset. Table we report the accuracy of our model on in-distribution and OOD
splits, compared with different baselines 884 44. Our method shows the best performance on the OOD test domains. The intuition of why this happens is that, due to minimality, we retain more features which are shared across the three training domains, giving less importance to the ones that are domain-speciffe (which contain the spurious correlations with the hospital environmental informations). This can be further enforced at test time, as we show in the ablation in Appendix D.9 trading off in distribution performance for OOD accuracy.

---

Table 4:
Results for domain generalization on DomainBed. Our approach achieves consistently higher average OOD generalization, outperforming ERM in all cases except one.
Dataset/Algorithm
OOD accuracy (by domain)
PACS S
A
P
C
Average 85.7
ERM
77.9 + 0.4
88.1 0.1
97.8
= 0.0
79.1 = 0.9
Ours
83.1 0.1
86.7# 0.8
97.8 0.1
83.5 0.1
87.5
VLCS
C
V
S
Average 77.4
ERM
97.61 1.0
63.3 - 0.9
76.4 + 1.5
72.2 1 0.5
Ours
98.11 0.2
63.4 0.5
78.2 0.7
73.9 0.8
78.4
OfficeHome
P
R
Average 67.5
ERM
53,41 0.6
62.7 1.1
76.5 = 0.4
77.3 =
Ours
56.31 0.1
66.7 + 0.7
79. 0.5
81.3 0.4
70.9
Waterbirds ERM
LL 98.6 + 0.3
LW 52.05 +
WL
WW
Average 81.3
68.5
93 + 0.3
Ours
99.5 0.1
73.0 + 2.5
85.0
95.5 0.4
90.5
4.3 Few-shot transfer learning.
We finally show the ability of features learned with method to adapt new domain with a small number of samples in few-shot setting. We compare the results with ERM in Table 2 averaged by domains in each benchmark dataset. The full scores for each domain are Appendix D.5 for 1-shot, 5-shot, and 10-shot setting, reporting the mean accuracy and standard deviations over 100 draws. Our approach achieves consistently higher accuracy than ERM, showing the better adaptation capabilities of our minimal and sufficently sparse feature space.
4.4 Additional results
In Appendix we report large collection of additional results, including comparison with 14 baseline methods on the domain shift benchmarks (D.4). a qualitative and quantitative analysis on the minimality and sparse sufficiency properties in the real setting (D.2). a favorable additional comparison on meta learning benchmarks, with 6 other baselines including [47] an ablation
study on the effect of clustering features test time (D.9p. and demonstration the possibility to obtain task similarity measure as consequence of our approach DD7
5 Conclusions
In this paper, we demonstrated how to learn disentangled representations from distribution of tasks by enforcing feature sparsity and sharing. We have shown this setting is identifiable and have validated experimentally in a synthetic and controlled setting. Additionally, we have empirically shown that these representations beneficial for generalizing isolating spurious and domain specific factors that should not used under distribution shift.
in real-world settings,
Limitations and future work: The main limitation of our work is the global assumption on the strength of the sparsity and feature sharing regularizers a and across all tasks. In real settings these properties of the representations might need to change for different tasks. We have already observed this the synthetic setting Figure 3] where when > 0.25 features cluster excessively and are unable to achieve clear disentanglement and do not generalize well. Future work may exploit some level knowledge on the task distribution (e.g. some measure of distance on tasks) order to tune a, | adaptively during training, or train conditioning
a distribution of regularization parameters as (21, enabling more generalization at test time. Another limitation is in the sampling procedure to fit the linear head at test time: sampling randomly from the training set (balanced by class) may not be enough to achieve the best performance under distributions shifts. Alternative sampling procedures, e.g. ones that incorporate knowledge the distribution shift if available (as in 43 may lead to better performance at test time.

---

Acknowledgments and Disclosure of Funding
Marco Fumero and Emanuele Rodolà were supported by the ERC grant 80280255 (SPECGEO), PRIN 2020 project no. no.2020TA3K9N (LEGO.AT), and PNRR MUR project PE0000013-FAIR, Marco Fumero and Francesco Locatello were partially Amazon while working at this project. We thank Julius von Kügelgen, Sebastian Lachapelle and the anonymous reviewers for their feedback and suggestions.
References
[1] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian J. Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 9525-9536, 2018.
[2] Kartik Ahuja, Karthikeyan Shanmugam, Kush R. Varshney, and Amit Dhurandhar. Invariant risk minimization games. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 145-155 PMLR, 2020.
[3] Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H Falk, and Ioannis Mitliagkas. Generalizing to unseen domains via distribution matching. ArXiv preprint, abs/1911 1000044 2019.
[4] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. ArXiv preprint, abs/1907.02893, 2019.
[5] Jinze Bai, Rui Men, Hao Yang. Xuancheng Ren, Kai Dang, Yichang Zhang, Xiaohuan Zhou, Peng Wang. Sinan Tan, An Yang, et al. Ofasys: multi-modal multi-task learning system for building generalist models. ArXiv preprint, abs/2212.04400 2022.
[6] Peter Bandi. Camelyon17 17 dataset.
[7] Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pages 456-473, 2018
[8] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-1888 2013.
[9] Luca Bertinetto, João F. Henriques, Philip H. S. Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In 7th International Conference Learning Representa- tions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019,
[10] Gilles Blanchard, Aniket Anand Deshmukh, Ürun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. The Journal of Machine Learning Research, 22(1))46-100, 2021.
[11] Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig, Stephan Hoyer, Felipe Llinares- López, Fabian Pedregosa, and Jean- chillppp Vert. Efficient and modular implicit differentiation. ArXiv preprint, fobs/2105.1511 2021.
[12] Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. ArXiv preprint, abs/1903.04561, 2019.
[13] Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco Cohen. Weakly supervised causal representation learning. ArXiv preprint, abs/2203. 16437, 2022.
[14] Chris Burgess and Hyunjik Kim. 3d shapes dataset. https//gsihhhhcom/ddppmnn///llls dataset/, 2018
10

---

[15] Rich Caruana. Multitask learning. Machine learning, 28(1):41-755 1997.
[16]
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. In Daniel D. Lee, Masashi Sugiyama. Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 2172-2180, 2016.
[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248-255. IEEE Computer Society, 2009.
[18] Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[19] Andrea Dittadi, Frederik Träuble, Francesco Locatello, Manuel Wuthrich, Vaibhav Agrawal, Ole Winther, Stefan Bauer, and Bernhard Schölkopf. On the transfer of disentangled repre- sentations in realistic settings. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[20] Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and mitigating unintended bias text classification. 2018
[21] Alexey Dosovitskiy and Josip Djolonga. You only train once: Loss-conditional training of deep networks. In International conference on learning representations, 2020.
[22] Cian Eastwood and Christopher K. I. Williams, A framework for the quantitative evaluation of disentangled representations. 6th International Conference on Learning Representations, ICLR 20/8, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
[23] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal html.
[24] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178-178. IEEE, 2004.
[25] Marco Fumero, Luca Cosmo, Simone Melzi, and Emanuele Rodolà. Learning disentangled representations via product manifold projection. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 3530-3540. PMLR, 2021.
[26] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François training of neural
Laviolette, Mario Marchand, and Victor Lempitsky. networks. The journal of machine learning research, 17(1):2096-203 2016.
[27] Robert Geirhos, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 (1111)66656677 2020.
Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel,
[28] Zhengyang Geng, Xin- Yu Zhang, Shaojie Bai, Yisen Wang, and Zhouchen Lin. On training implicit models. In Mare Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, I, NeurIPS 2021, December 6-14, 2021, virtual, pages 24247-24266 2021.
11

---

[29] Ian J. Goodfellow, Quoe V. Le, Andrew M. Saxe, Honglak Lee, and Andrew Y. Ng. Measuring invariances in deep networks. In Yoshua Bengio. Dale Schuurmans, John D. Lafferty, Christo- pher K. I. Williams, and Aron Culotta, editors, Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems 2009. Pro- ceedings of a meeting held 7-10 December 2009, Vancouver, British Columbia, Canada, pages 646-654, Curran Associates, Inc., 2009.
[30] Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. In International Conference on Learning Representations, 2020.
[31] Andreas Griewank and Andrea Walther. Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM, 2008.
[32] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[33] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In 5th International Conference on Learning Repre- sentations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[34] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. arXiv preprint arXiv:2004.054333 2020.
[35] Ziniu Hu, Zhe Zhao, Xinyang Yi, Tiansheng Yao, Lichan Hong, Yizhou Sun, and Ed H Chi Improving multi-task generalization via regularizing spurious correlation. ArXiv preprint, 2022.
[36] Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross- domain generalization. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 124-140. Springer, 2020.
[37] Aapo Hyvärinen, Hiroaki Sasaki, and Richard E. Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In Kamalika Chaudhuri and Masashi Sugiyama, editors, The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pages 859-868. PMLR, 2019.
[38] Ali Jalali, Sujay Sanghavi. Chao Ruan, and Pradeep Ravikumar. dirty model for tusk learning. In J. Lafferty, C. Williams, J. Shawe-Taylee R. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010.
[39] Hicham Janati, Marco Cuturi, and Alexandre Gramfort. Wasserstein regularization for sparse multi-task regression. In Kamalika Chaudhuri and Masashi Sugiyama, editors, The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 18 April 2019. Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pages 1407-1416. PMLR, 2019.
[40] Yibo Jiang and Victor Veitch. Invariant and transportable representations for anti-causal domain shifts, 2022.
[41] Ilyes Khemakhem, Diederik P. Kingma, Ricardo Pio Monti, and Aapo Hyvärinen. Variational autoencoders and nonlincar ICA: A unifying framework. In Silvia Chiappa and Roberto Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy], volume 108 of Proceedings of Machine Learning Research, pages 2207-2217. PMLR, 2020.
[42] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
12

---

[43] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re- re-training is sufficient for robustmess to spurious correlations. ArXiv preprint, 00s/22044002977, 2022.
[44] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran S. Haque, Sara M. Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 5637-5664. PMLR. 2021.
[45] David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pages 5815-
5826. PMLR, 2021.
[46] Tejas D. Kulkarni, William F. Whitney, Pushmeet Kohli, and Joshua B. Tenenbaum. Deep convolutional inverse graphics network. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Gamett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2539-2547, 2015.
[47] Sébastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagkas, Yoshua Bengio, Simon Lacoste-Julies and Quentin Bertrand. Synergies between disentanglement and sparsity: a multi-task learning perspective. ArXiv preprint, abs/2211 1.16666, 2022.
[48] Sébastien Lachapelle, Pau Rodriguez. Yash Sharma, Katie E Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. Conference on Causal Learning and Reasoning, pages
428 484. PMLR, 2022.
[49] Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pages 11-104. IEEE, 2004.
[50] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In JEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 10657-10665. Computer Vision Foundation / IEEE, 2019.
[51] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta- learning for domain generalization. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[52] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. In IEEE International Conference on Computer Vision, ICCV 2017, Venice. Italy, October 22-29, 2017, pages
IEEE Computer Society, 2017.
[53] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 54000-509, 2018
[54] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European conference on computer vision (ECCV), pages 6246639, 2018.
[55] Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M. Asano, Taco Cohen, and Stratis Gavves. CITRIS: causal identifiability from temporal intervened sequences. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, Interna- tional Conference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 Proceedings of Machine Learning Research, pages 13557-13603 PMLR, 2022.
13

---

[56] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 41 14-44.2. PMLR, 2019.
[57] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. A sober look at the unsupervised learning of disentangled representations and their evaluation. J. Mach. Learn. Res., 21:209:1 1229::22, 2020.
[58] Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In Proceed- ings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 6348 6359. PMLR, 2020.
[59] Aurelie C. Lozano and Grzegorz. Swirsze7. Multi-level lasso for sparse multi-task regression. In Proceedings of the 29th International Conference on Machine Learning, ICMI. 2012, Edinburgh, Scotland, UK, June 26 July 1, 2012. ieml.ce/ Omnipress, 2012.
[60] Chaochao Lu, Yuhuai Wu, José Miguel Hernández-Lobato, and Bernhard Schölkopf. In- variant causal representation learning for out-of-distribution generalization. In International Conference on Learning Representations, 2022.
[61] Zvika Marx, Michael T Rosenstein, Leslie Pack Kaelbling, and Thomas G Dietterich. Transfer learning with an ensemble of background tasks. Inductive Transfer, 10, 2005.
[62] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentangle- 2017.
ment testing sprites dataset. https://github.comm/// http:/hhwwh..oomddememo//////rdd
[63] John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 7721 7721-7735. PMLR, 2021.
[64] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In Proceedings of the 30th International Conference on Machine Learning. ICML 2013, Atlanta, GA, USA, 16-21 June 2013, volume 28 of JMLR Workshop and Conference Proceedings, pages 10-18. JMLR.org, 2013.
[65] Hyeonseob Nam, HyunJae Lee, Jongehan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8690-8699, 8699, 2021.
[66] Boris N. Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. TADAM: task dependent adaptive metric for improved few-shot learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchh and Roman Garett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 719-729, 2018
[67] G. Parascandolo, N. Kilbertus, M. Mojas-Carulla,, and Schölkopf. Learning independent causal mechanisms. In Proceedings the PMLR 80:4036-4044, 2018
the 35th International Conference on Machine Learning,
[68] Ji Ho Park, Jamin Shin, and Pascale Fung. Reducing gender bias in abusive language detection. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pages 2799-2804 Brussels, Belgium, 2018. Association for Computational Linguistics.
14

---

[69] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. 7achary Vito,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[70] Jielin Qiu, Yi Zhu, Xingjann Shi, Florian Wenzel, Zhiqiang Tang, Ding Zhao, Bo Li, and Mu Li. Are multimodal models robust to image and text perturbations? ArXiv preprint, abs/2212.080444, 2022.
[71] Scott E. Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 1252-1260, 2015.
[72] Bryan C Russell, Antonio Torralba, Kevin P Murphy, and William T Freeman. Labelme: a database and web-based tool for image annotation. International journal of computer vision, 77(1):157-173, 2008.
[73] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. ArXiv preprint, abs/1911 1.08731, 2019.
[74] Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterizztion exacerbates spurious correlations. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, PMLR, 2020.
volume 119 of Proceedings of Machine Learning Research, pages
[75] Ruslan Salakhutdinov. Deep learning. Sofus A. Macskassy, Claudia Perlich, Jure Leskovec, Wei Wang, and Rayid Ghani, editors, The 20th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '14 New York, NY, USA - August 24 27, 2014, page 1973. ACM, 2014.
[76] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv preprint, 1ts/1990.01108, 2019.
[77] Jürgen Schmidhuber. Learning factorial codes by predictability minimization. Neural compu- tation, (66::863-87, 1992.
[78] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 10965)x612-63, 2021.
[79] Anna Seigal, Chandler Squires, and Caroline Uhler. Linear causal disentanglement via interventions. ArXiv preprint, abs/2211.164665 2022.
[80] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. FLAVA: A foundational language and vision alignment model. ArXiv preprint, abs/21 12.04482, 2021.
[81] Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4077-4087, 2017.
[82] Peter Sorrenson, Carsten Rother, and Ullrich Köthe. Disentanglement by nonlinear ICA with general incompressible-ftow networks (GIN). In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
15

---

[83] Trevor Standley, Amir Roshan Zamir, Dawn Chen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 9120-9132. PMLR, 2020.
[84] Baochen Sun, Jiashi Feng. and Kate Saenko. Correlation aligmment for unsupervised domain adaptation. In Domain Adaptation in Computer Vision Applications, pages 153-1714 Springer, 2017.
[85] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pages 443-450. Springer, 2016.
[86] Victor Veitch, Alexander D' Amour, Steve Yadlowsky, and Jacob Eisenstein. Counterfactual invariance to spurious correlations: Why and how to pass stress tests, 2021.
[87] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 5385-5394, IEEE Computer Society, 2017.
[88] Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 3630-3638, 2016.
[89] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[90] Zihao Wang and Victor Veitch. A unified causal view of domain invariant representation learning. ArXiv preprint, abs/2208.0666 0008000068777880 2022.
[91] Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime G. Carbonell. Characterizing and avoiding negative transfer. In JEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 11293-11300 Computer Vision Foundation / IEFE, 2019.
[92] Martin Wattenberg, Fernanda Viégas, and Ian Johnson. How to use t-sne effectively. Distill, 1(10):e2, 2016.
[93] Florian Wenzel, Andrea Dittadi, Peter V. Gehler, Carl-Johann Simon-Gabriel, Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schicle, Bernhard Schölkopf, and Francesco Locatello. Assaying out-of-distributio generalization in transfer learning. In Neural Information Processing Systems, 2022.
[94] Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alviss Rebuffi, Ira Ktena, Krishna- murthy Dvijotham, and Ali Taylan Cemgil. A fine-grained analysis on distribution shift. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[95] Matthew Willetts and Brooks Paige. I don't ' need u: Identifiable non-linear ica without side information. ArXiv preprint, abs/2106.05238, 2021.
[96] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's trans- formers:
State-of-the-arr natural language processing. ArXiv preprint, abs/1910.03771, 2019,
[97] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korblith and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models im- proves accuracy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICMI. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 23965-23992 PMLR, 2022.
16

---

[98] Jianxiong Xiao, James Hays, Krista A. Ehinger. Aude Oliva, and Antonio Torraba. SUN database: Large-scale scene recognition from abbey to zoo. In The Twenty- Third IEEE Conference Computer Vision and Puttern Recognition, CVPR 2010, San Francisco, CA, USA, 13-18 June 2010, pages 3485-3492. IEEE Computer Society, 2010.
[99] Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001 00000.0077, 2020.
[100] Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[101] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision. ArXiv preprint, abs/2111. 1432, 2021.
[102] Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning to adapt to domain shift. Advances in Neural Information Processing Systems, 34:23664-2366 2021.
[103] Yu Zhang and Qiang Yang. An overview of multi-task learning. National Science Review, 5(1):30-43, 2018.
[104] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6))1452 1464, 2017.
[105] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. 2021.
[106] Jinguo Zhu, Xizhou Zhu, Wenhai Wang. Xiaohua Wang, Hongsheng Li, Xiaogang Wang, and Jifeng Dai. Uni-perceiver-moot moe: Learnnn sparse generalist models with conditional moes. ArXiv preprint, 0006/2206.06674, 2022.
17


Reviews:
[{"review_id": "fG9Bl3WTaQ", "reviewer": "Reviewer_Aoq2", "summary": "The paper proposes to learn disentangled representations by leveraging knowledge from a diverse set of supervised task objectives. Specifically, the paper shows that disentangled representations can emerge without ground-truth annotations of the factors of variation under two assumptions: 1) sparse sufficiency, and 2) minimality. Practically, the paper implements a meta-learning objective for representation learning with the sparse sufficiency and minimality constraints. The authors also evaluate the learnt representations for out-of-distribution (OOD) generalization. ", "questions": "I think the paper is already in a good and sufficient state to be accepted as it is. I do not have any further questions for the authors.", "limitations": "The authors have adequately addressed the limitations of the proposed method.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is very well written and easy to follow.\n- The paper appropriately situates the proposed method among all the related works, including highly related concurrent work.  \n- The paper has multiple significant contributions that would benefit the research community — it presents the theory as well shows experimentally that appropriate inductive biases can sufficiently recover the factors of variation from the data.\n- The disentanglement results are promising. The paper contains ablation studies to show the importance of individual components such as the feature sharing property.\n- The proposed method is evaluated for domain generalization and domain shift tasks on six different benchmarks, achieving convincing results.", "weaknesses": "In my opinion, the paper does not have any major weaknesses. To me, barring a few grammatical mistakes, it already reads better than a lot of accepted papers. The only thing I could suggest is to make the caption of Figure 1 a bit more descriptive so that it conveys a full picture by itself and doesn’t rely on the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "LHJIqCnQJu", "reviewer": "Reviewer_BjS9", "summary": "The foundation of the proposed work is a form of regularization motivated by the problem of disentanglement.  A supervised task is deconstructed into randomly generated sub-tasks, and the same trained representations are used for all sub-tasks (followed by a linear head specific to each sub-task).  Two regularizers are then introduced on the feature activations: one which encourages sparsity (L1) and one which encourages minimality by penalizing the entropy of the distribution of feature activations.  \n\nThe disentanglement motivation is such: If we assume each sub-task only requires information from a subset of some ground truth disentangled factors of variation of the data, it is argued that the training procedure and regularization will eventually disentangle the factors of variation completely.  Disentangled factors confer various advantages in representation learning, one of which being increased robustness to shifts in the data distribution because spurious factors are able to be separated from the rest. This advantage forms a large part of the proposed work.  After establishing the intended effect of the regularization on synthetic data where generative factors are known, the focus shifts to real data where domain shift is a challenge.", "questions": "Can the authors please elaborate on the sub-task generation process for the real datasets?  Take Camelyon17 as an example -- how were sub-tasks generated out of this binary classification problem?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method is sensible while being nontrivial.  It is presented well, and the experimental results, tackling either idealized disentanglement or practical domain shift, cover a lot of ground.  It is interesting to leverage the ability to decompose label information as a way to condition representations and encourage structural properties of extracted factors of variation.", "weaknesses": "There are two perspectives with which to view the introduced regularization and training procedure.  The first is in terms of what it accomplishes with respect to disentanglement in synthetic scenarios, and the second in terms of its utility to address domain shift in real datasets (disregarding disentanglement as unrealistic).  The support for both perspectives could be strengthened.  \n\nFrom the disentanglement perspective, it’s not clear how much everything depends on Y containing full information about the generative factors.  If Y depends on fewer factors than X (as is more often the case), how can Proposition 2.1 hold?  For example, if the original labels (before any task deconstruction) have no “angle” information from DSprites, no sub-task should extract angle information, and even with unlimited samples from p(x,y), the “angle” generative factor will be ignored as spurious.  If Y depends on the factors in an entangled way, e.g. I(xpos;Y)=I(ypos;Y)=0, but I(xpos,ypos;Y)>0, then there is no way to create sub-tasks out of Y that depend on xpos but not ypos.  Even with unlimited samples there should be no disentanglement of the xpos and ypos factors.  Thus it seems that the disentanglement is predicated on having I(Y;Z)=H(Z), ie all factor information in Y; if true, this is quite a severe constraint.  More clarity around this point would be helpful, either through a constructed example on the synthetic dataset or elaboration about what’s expected of Y when discussing Proposition 2.1.\n\nFrom the real world, pragmatism perspective, the domain shift results are numerous yet somewhat insubstantial.  The comparisons in the main text are predominantly to ERM, which does not seem particularly convincing.  In the research cited to support using ERM as a baseline (Gulrajani and Lopez-Paz 2020), the competitiveness seemed to only come from re-tuning for ERM anew everything pertaining to training and model selection -- but I gather all conditions match for the ERM as the proposed method.  Given two methods, identical except that one has two additional hyperparameters to tune, the one with additional hyperparameters should prevail.  There are many more domain shift results in the Supp, though they are thrown together without much in the way of a unifying perspective.  A significant number of baselines are compared to in App. D4,5, yet the results are presented without exposition.  Curiously, the proposed method is a significant outlier, performing several standard deviations above all fourteen baselines, on a single category of each experiment of D4 (S in CLSV, C in PACS, C in ACPR), yet the performance on other categories is always middling or even the worst.  Can the authors speak to this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for all their suggestions, comments and questions. We will address them in the following. We remain available during the discussion phase to further clarify any question or concern.  \n\n### **Disentanglement**\n\nThe reviewer is correct in that each task does not depend on all the factors of variation at the same time. Proposition 2.1 has two key assumptions on the task distribution. These are $p(S \\cap S' = \\{i\\})> 0$ or  $p( \\{i\\} \\in (S \\cup S') -(S'\\cap S))>0$. These states that a factor of variation is identifiable when it is in the intersection of two tasks (more precisely, in the intersection between the two sets indexing the factors of variation relevant for each task) or in the union minus the intersection. Intuitively, it means that there are two tasks that have that factor of variation in common or have several factors in common but only one different. Full identifiability of all factors of variations happen only if the union of all tasks span all the factors. If a factor does not influence any task, it is treated as a nuisance variable that is not learned as in [a]. Future works could explore identifiability of blocks of factors, following the weaker identifiability definition of [b]. \n\n_[a] Emergence of Invariance and Disentanglement in Deep Representations, Achille and Soatto, JMLR 2018_\n\n_[b] Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style, Von Kügelgen et al., NeurIPS 2021_\n\n\n\n\n### **Domain shift**\n\n- **Setup** : We performed the experiments on real datasets putting ourselves in the same exact settings, concerning architecture, augmentations, model selection strategy and evaluation of [32,44]. This was to ensure the fairest possible comparison. We compare with the 13 baselines  in Appendix D4,5 (see point below) other than ERM, which follow the same setting, and their results are reported directly from [32,44]. As highlighted in the main paper, we chose to report mainly ERM  for the sake of conciseness as it is the method that performs best on average, which was confirmed by several results [44,32,c]. \nWe stress that we don’t claim that our method performs best in all settings (see limitations section 5) , but we observe convincing evidence  that the sparse sufficiency and minimality are useful properties for OOD generalization, in most cases.\n\n_[c] Wiles, O., Gowal, S., Stimberg, F., Alvise-Rebuffi, S., Ktena, I., Dvijotham, K., & Cemgil, T. (2021). A fine-grained analysis on distribution shift_\n\n- **Hyperparameters** : we argue that our method does not correspond to ERM with two additional hyperparameters: In particular we differ in that (i) we learn from a distribution of tasks (ii) we leverage a meta learning approach, and, most importantly, (iii) we incorporate two inductive biases in the learning process, namely sparse sufficiency and minimality. What is shared with the settings of [32,44] is the architectures, augmentations, model selection strategy and evaluation to ensure the fairest comparison.\n\n- **Baselines results in Appendix D4** : In the Tables of the appendix D.4 we accidentally made some mistakes in reporting our results, due to a permutation of the column order with respect to Table 4 in the main paper. In Table D.4.1 we accidentally swapped columns “V” and “S” for our method. In Table D.4.2, we swapped columns “A” and “C”. Similarly, In Table D.4.3 we swapped our results for columns A and C. Please refer to the results in Table 4 of the main paper for the correct order and results for our method.\nWe apologize for the confusion, we will correct this in the revision. In light of this, our method performs consistently better or on par with others.In general we hypothesize that we have the best performance when the target domain shares a good amount of information with the intersection of the training domains.This leads to very good gaps in performance when the domain is farther from the training distribution and the transfer is more difficult (e.g. we do 83.5 on “S” in PACS). As remarked in the limitations section, more principled ways to tune sparsity and feature sharing regularization parameters could bring performance even higher. We will incorporate all these considerations into Appendix D4,5.\n\n### **Task distribution on real datasets**\nPlease see the response to this point in the General Response section."}, {"review_id": "yxLxgIL3nn", "reviewer": "Reviewer_vgpp", "summary": "The paper studies the problem of learning disentangled representations in a setting with access to multiple supervised tasks, each of which is assumed to depend only on a subset of the underlying factors of variation. While the factors of variation are never directly observed, two principles are incorporated to allow their identifiability; i) sparse sufficiency (“sparsity”) ii) minimality - encouraging features to be shared / reused across tasks. The proposed approach is demonstrated to achieve a higher degree of disentanglement on synthetic data compared to previous methods, as well as better out-of-distribution generalization in real image domains. ", "questions": "1. There are several generative approaches to learning disentangled representations which are worth mentioning e.g. [1, 2]. However, these methods rely on weak supervision e.g. observing some of the factors of variation for some of the images. Given the binary labels constructed in this paper for some of the dataset, I wonder if there is a way to “quantify” the amount of supervision provided to the proposed model. This is not trivial as no direct labels are used but as long as many of “binary” labels are constructed the effective level of supervision is increased. Could the authors elaborate on this perspective?\n\n[1] Locatello et al. “Disentangling factors of variations using few labels”. In ICLR, 2020.\n\n[2] Gabbay et al. “An Image is Worth More Than a Thousand Words: Towards Disentanglement in the Wild”. In NeurIPS 2021.\n\n2. DCI metric is a combination of three different metrics: Disentanglement, Completeness and Informativeness. However, the results in this paper are presented as a single number. Is this the average of the three? or just the Disentanglement? I request the authors to present the metrics separately in the quantitative evaluation as they may shed more light on the performance of each of the considered methods. \n\n3. Minor typos:\n* Line 45 - w.r.t to\n* Line 57 - “cohexist” → “coexist” \n", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of studying disentanglement in the context of generalization to downstream tasks is well motivated, especially in real image domains.\n\n2. While the “sparsity” regularization already introduced for disentanglement in previous works, including the concurrent work of [47], the second “minimality” term seems novel. Including this term is well justified for real world problems when the number of FoV is unknown, and the results in the paper highlight its effectiveness.\n\n3. The experimental section contains both quantitative disentanglement evaluation on synthetic datasets, as well as appropriate assessment for domain generalization in real image data, where the ground truth for the factors of variation is unknown.\n", "weaknesses": "1. The idea of multitask meta-learning still looks a bit “over” complicated to me. I think the paper would benefit from including the following simple baseline for comparison: combining all the supervision from all tasks into a single multilabel classification task. The feature layer before the classification head should still include the sparsity term. Can the authors elaborate on the ERM baseline? Is this one the “simple” baseline I just mentioned?\n\n2. The dataset considered in [47] is 3Dshapes. However, I do not see a comparison on this benchmark against [47]. Could the authors report the DCI evaluation of the proposed method and [47] on 3Dshapes?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for all their suggestions, comments and questions. We will address them in the following. We remain available during the discussion phase to further clarify any question or concern. \n\n### **Metalearning and baselines**\n\nFor comparison with the ERM baseline we detach the original head (last layer) from the architecture and attach the same sparse linear head that we use in our method to be trained on the support tasks at test time.\nThis corresponds to comparing the feature space obtained from our method with the feature space obtained by concatenating all tasks/domains together and training with ERM. For further comments on the choice of employing meta learning in our approach, please see the related point in the General Response section. \n\n### **Comparison with [47]**\n\nThe dataset employed in [47] is indeed 3D shapes but a direct comparison with [47] is not feasible as we don’t have access to the task distribution with which the experiment was performed. \nNevertheless [47] is akin to our method without the feature sharing regularizer and we show direct comparison with it in Figure 5 (purple curve vs yellow curve). Here we show that when the number of factors of variations is unknown, sparsity alone (yellow dashed line) cannot achieve full disentanglement, in confirmation of our theoretical result in Proposition 2.1.\n**In Figure A of the attached PDF to the General Response, we report the same experiment performed on the 3DShapes dataset, which demonstrates that sparsity alone [akin to 47] is not sufficient to achieve maximal disentanglement**, in this setting.\n\n### **Supervision on the FOVs**\n\nWe consider our method in a similar setting as the weakly supervised method like [58,37,25,a,b]. The supervision in our case comes indirectly from the task label and is necessary for obtaining identifiability results (without any supervision this was proven not possible, e.g. in  [56]). \nQuantifying the amount of supervision given by the task labels, is indeed an interesting direction. Overall we agree with the reviewer that this is not easy to estimate quantitatively this amount due to large variation of unobserved quantities. \nA possible direction would be to inspect the task distribution (assuming full knowledge of it) and estimate the probabilities needed for identifiability in Proposition 2.1, i.e. the assumptions on the task distribution $p(S \\cap S' = \\{i\\})> 0$ or  $p( \\{i\\} \\in (S \\cup S') -(S'\\cap S))>0$. Namely the probability of two tasks to share a single factor and the probability that two arbitrary tasks share most factors but differ in one.\nA related source of complexity comes from the number of factors of variations (FoVs) in the dataset and the number of possible values (assuming discrete variables) that they can assume.  As this grows, one can expect that more observations from p(x,y) are needed in order to learn disentangled representations, but also that tit becomes harder to estimate the probabilities stated above. Assuming a good estimate of the these probabilities, the number of FoVs, the number of possible values and the probability distribution used to sample them (in  most cases this is uniform), one could draw a loose estimate of the number of observation from p(x,y) needed in practice to learn a representations disentangled w.r.t. to the factors of variation.\n\n\n\n_[a] Locatello et al. “Disentangling factors of variations using few labels”. In ICLR, 2020._\n\n_[b] Gabbay et al. “An Image is Worth More Than a Thousand Words: Towards Disentanglement in the Wild”. In NeurIPS 2021._\n\n\n### **DCI Metric**\n\nPlease see the response to this point in the General Response section. \n\n\n"}, {"review_id": "qpl6eROdrT", "reviewer": "Reviewer_S5sy", "summary": "The paper proposes a novel regularization technique for disentangled representation learning. This regularizer is based on two principles formalized in the paper: 1) sparsity and 2) minimality. It assumes access to multiple tasks during training where each task uses only a subset of features, and different tasks share some of the features. The paper also provides theoretical results showing that these two principles are enough for the identification of disentangled factors. The proposed approach is validated, showing the significance of both regularization terms. Empirical results show the benefits of the proposed feature learning technique over standard ERM training.\n", "questions": "1. How the DCI metric is defined exactly? The paper refers to [22], which has three separate metrics. Is DCI a combination of those? If yes, how are they combined?\n2. The Waterbirds dataset has only two classes; how is task distribution defined?\n3. Is regularization used or not when learning a novel test task? If yes, where do multiple tasks need for it come from?\n4. For the experiments on DomainBed, it is stated that pretrained RN50 is used. How the proposed method influences the features learned in this case? Do you finetune it?\n5. In the definition of the two principles, what is the difference between $f_t^*/f^*/f^*_{S_t}$? They are not properly introduced.\n6. See 3 from weaknesses.\n    - How does the sharing term \"forces features to cluster across tasks and therefore be reused by different tasks, when useful\" (L92-93)? For example, consider 3 FoVs $f_i$ and 3 binary classification tasks $t_i$ (with logits for simplification, i.e., M=1) corresponding to each combination of only 2 features with weights +1/-1, i.e., $y = \\sigma(f_{S_t[1]} - f_{S_t[2]})$, $S_t \\in \\{(1, 2), (1, 3), (2, 3)\\}$ In this case, $\\tilde{\\phi}_m = 2/6 = 1/3$ (removing the extra $1/TC$ term, see minor comments), resulting in a uniform distribution that results in high entropy (maximum possible). On the other hand, the sparsity and minimality are satisfied. It seems that the distribution should be over $t,c$ instead of $m$.\n    - How does it correspond to the minimality principle introduced in Sec 2.3? The latter seems to suggest that no (smaller? see 9 below) subset of features can solve the task. Why having an L1 loss is not enough to satisfy this? If using fewer features is possible, the corresponding weights will be set to 0 following the L1 regularizer.\n7. While the overall idea is sound, it is unclear why the meta-learning framework is necessary. Can the same technique be applied to the standard multi-task setting where different linear heads are learned simultaneously with a shared feature extractor?\n\n\nMinor:\n8. Why scaling by $1/TC$ in L88? $\\tilde{\\phi}_m$ is not a distribution in this case.\n9. in L129 do you mean $S' \\subset S_t$? Otherwise can include $S_t$ and be sufficient for classification.\n", "limitations": "Limitations are well-addressed.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "-  The proposed two formalized principles (sparse sufficiency and minimality) and their sufficiency for disentangled representation learning are valuable contributions to the field.\n-  The empirical study assesses the quality of the learned representations beyond disentanglement, showing the practical benefits of the proposed approach.\n-  The overall idea and motivation are sound.\n", "weaknesses": "1. The paper lacks clarity; certain details and symbols' introductions are missing for better understanding (more in the question section). The writing can be improved.\n    - For example, proposition 2.1 does not clearly state what S' and i are. Also, the symbol $f_t^*$ is not introduced. The proposition also misses subsequent discussion for better understanding. \n    - The text has a number of mistakes. For example, L50: \"a**n** entropy\", L57: \"co~~h~~exist\", L77: \"give**n**\", L88: the formula has an extra parenthesis, L110: subscript $t$ is missing for $y$s.\n2. The empirical evaluation is insufficient and lacks comparisons with other disentangled representation learning methods as well as few-shot/meta-learning methods.\n    - Tab. 1 lacks other repr. learning methods.\n    - Tab. 2 and 4 lack other few-shot/meta-learning methods. The main paper refers to Appendix for the latter mentioning a 10% improvement on MiniImageNet. However, Tab.15 in Appendix does not support the claim, with MetaOptNet being the best.\n    - Fig. 5 lacks other domain adaptation methods.\n3. It is not entirely clear how the sharing regularizer in (3) corresponds to the minimality principle introduced in Sec. 2.3. Also, it is unclear how exactly it favors clustering. Further clarifications are needed. (see questions for a more concrete question)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for all their suggestions, questions and comments. We will address them in the following. We remain available  during the discussion period for any further question or concern.  \n\n### **Clarity**\n\nWe thank the reviewer for pointing out unclear sections and typos.\nConcerning Proposition 2.1, $S,S'$ correspond to two arbitrary instances of the index set $\\mathcal{S}$, representing the support of two arbitrary tasks, and ${i}$ is a singleton. The assumption $p(S \\cap S' = \\{i\\})> 0$ or  $p( \\{i\\} \\in (S \\cup S') -(S'\\cap S))>0$  in Proposition 2.1 states that:\n- The probability of two tasks having a singleton intersection of support on the factor of variations is non zero \n- The probability that their difference of supports is a singleton is non zero.\n\nIntuitively, it means that there are two tasks that have either one factor of variation in common or have several factors in common but only one different. We will add a visualization example in the paper to help intuition.\nFinally, $f_t^*$ is defined as the unknown mixing function that defines uniquely a task $t$. $f_t^*$ and $f_{S_t}^*$ corresponds to the same object, thank you for spotting this and apologize for the confusion.\nWe will incorporate these considerations in the theoretical section, to make it clearer and more intuitive. Nevertheless, we respectfully remark that clarity was a strength pointed out by R1 and R2.\n\n\n\n### **Empirical evaluation**\n\n\n- **Synthetic**: The experiments in section 4.1 (Table 1) serve to validate our theory and demonstrate that sufficiency and minimality lead to disentangled representations. It would be non trivial to compare to other methods which are not in the multi-task learning setting (e.g. $\\beta$-VAE), as their assumptions are different. Nevertheless, we compare with [47] in Figure 3, which shares a similar setting, showing that sparsity alone (yellow dashed line) cannot achieve the best DCI score when the number of factors of variation is unknown. **We included  a similar experiment to further compare with [47] on the 3DShapes dataset, in Figure A, in the attached PDF.**\n\n- **Domain shift** : For the experiments on domain shifts we compare with other 13 baselines in Appendix D.4. We reported ERM in the main paper for the sake of brevity as is the one that in average performs better, as observed by [32,44].\nWe report comparison with other metalearning baseline in Appendix D.8. The 10% improvement on MiniImagenet is with respect to the method of [47], we will clarify this misunderstanding. \nWe respectfully remark that the extensive experiments and evaluation were pointed out by reviewers (BjS9, Aoq2,v gpp) and that we agree that our main claim is to show how sparse sufficiency and minimality are useful properties to achieve disentanglement and OOD robustness.\n\n### **Minimality and feature sharing**\n\nThe minimality principle states that for a task $t$ there does not exist a different assignment of its support (S’) that is able to solve $t$. This means that there are no duplicated features. This is needed when the dimensionality of the representations is higher than the number of factors of variation (which is typically our setting).\nIn this case sufficiency is not enough to attain disentanglement as features are still allowed to be duplicated or splitted across the representation entries (see Figure 3 and 6).\n\n**Clustering**: The  regularization minimizes the entropy of the task-averaged distribution $\\tilde{\\phi}_m$ of feature importances (i.e. absolute value of the coefficients of the linear predictor).  By doing this we enforce  $\\tilde{\\phi}_m$ to be more  peaked. This means that for a batch of tasks more weight will be given to the more shared features.\nWe demonstrate this phenomenon in the paper with the experiment in Figure 6 in Appendix D.1, it can be observed how imposing growing values of regularization strength $\\beta$ promotes clustering in the task/feature space, i.e. many tasks will share the same features. When enforced too much this clustering effect collapses everything into one or few features shared among tasks and becomes harmful for performance (see last row of Figure 6).\n\nWe believe that this comment should also address the counter example (for which M=1 is an odd assumption as no disentangled representation is possible with continuous factors as per our theory). We’re happy to further discuss these points if needed. Concerning the the scaling factor $\\frac{1}{TC}$ under Equation 3, it should appear both at the numerator and denominator of  $\\tilde{\\phi}_m$. We will fix this, thanks.\n\n\n### **Waterbirds dataset task generation**\n\nPlease see the response to this point in the General Response section.\n\n### **Regularization at test time**\n\nFor a fair comparison we adopt a linear head with the sparsity regularization for both our method and the ERM. We’ll clarify this in Appendix C.4\nThe feature sharing is turned off at test time. The only exception are the interesting ablation experiments in D.9. There, we show that using feature sharing at test time we can trade off better performance OOD with slightly lower performances closer to the training distribution (Tab 16, last row). As confirmed by these preliminary experiments, tuning feature sharing at test time on unsupervised test data is an interesting direction for future work.\n\n### **Finetuning on domain bed**\nFor all the experiments on domain shifts, we started with a pretrained network on Imagenet and we finetune it on the target dataset using our method. The networks in the synthetic experiments are instead trained from scratch. In this regard, it would be interesting as future work to train a network from scratch on a large scale dataset like Imagenet with our method, and then testing the transferability of the resulting model compared with ERM pretrained models.\n\n### **On the necessity of metalearning**\nPlease see the response to this point in the General Response section."}]

Author rebuttal:
# General Response

We thank all reviewers for their valuable insights, constructive feedback, and inquiries. In particular, we're glad that all reviewers agree on the value of the contribution, the soundness of the idea, the usefulness of the proposed properties, clarity (reviewers Aoq2,BjS9) and extensive experimental evaluation (reviewers Aoq2,BjS9,vgpp).

In this comprehensive response, we aim to address common concerns and questions raised by multiple reviewers. Subsequently, we will provide direct responses to the specific questions posed by each reviewer. We remain available during the discussion period to clarify any further concern or question





## **DCI metric** ( Reviewers **S5sy** and **vgpp**)

The DCI metric reported refers to the disentanglement component of the DCI metric as standardly adopted in many disentanglement method literature see e.g. [56,58,25]. We measured the completeness and informativeness scores of [22] for the models in Table 1, and we report them in the Table below.  We will incorporate this Table in the Appendix of the manuscript.


|                   		 | _DSprites_ | _3DShapes_ | _SmallNorb_ | _Cars_ |
|----------------------------|------------|------------|-------------|--------|
| **Without regularization**    |   		 |   		 |    		 |   	 |
| _DCI Disentanglement_ 	 | 16.6  	 | 44.4  	 | 16.5   	 | 60.5   |
| _DCI Completeness_		 | 17.5  	 | 39.1  	 | 12.9   	 | 50.8   |
| _DCI Informativeness_ 	 | 88.0  	 | 87.6  	 | 90.5   	 | 95.5   |
|                   		 |   		 |   		 |    		 |   	 |
| **With regularization** |   		 |   		 |    		 |   	 |
| _DCI Disentanglement_ 	 | 69.9  	 | 87.7  	 | 60.5   	 | 92.3   |
| _DCI Completeness_		 | 72.3  	 | 88.4  	 | 63.2   	 | 57.1   |
| _DCI Informativeness_ 	 | 96.0  	 | 95.7  	 | 95.4   	 | 99.7   |


## **Task distribution on real datasets** (Reviewers **BjS9** and **S5sy**)

The procedure is described in Appendix C.4 and we summarize it here:
 
Each task is a  $C$-class classification problem. To generate it, we sample randomly  $C$ classes from the set of available $K_{train}$ classes in the dataset. For each of the $C$ classes we then sample  $| U |$ support samples to fit the linear head, solving Equation 5, and $| Q |$ query sample to update the backbone weights, optimizing Equation 4. In our experiments, we set $C=5$, when $K_{train}$> $C$, while for the datasets where $K_{train} <5$, we set $C=k_{train}$.
For binary datasets such as Camelyon17 or Waterbirds the possible classes to be predicted are always the same across tasks: what is changing is the composition of $U$ and $S$.
Keeping their cardinality low, we ensure that some tasks will not contain spurious correlation that may be present in the dataset, while other ones will still retain it, and the regularizers will satisfy solutions which discards the spurious information. We can observe evidence of this in the experimental results in Table 4 and in Figure 7 in the Appendix.


## **On the necessity of metalearning**  (Reviewers **S5sy** and **vgpp**)

Meta learning is adopted for two key reasons:
- *Scaling* :  as we’re working with thousands of tasks it would be impractical to keep in memory the classification heads for each task. For example, in the domain shift a lower bound of the estimate of the possible tasks is $\binom{D*K_{train}}{C}$ where $D$ domains, $K_{train}$ classes in each domain, and $C$ classes composing a task.
- *Generalization* :  we employ a meta learning objective to generalize better to unseen tasks. The objective in equation 4 is indeed formulated  w.r.t. to query samples  $Q$, while the linear head is fitted on the support samples $S$ for each task. This is not possible in standard multitask learning formulations.


Meta-review: All the reviewers unanimous agree to accept this paper.

 The authors are encouraged to improve this work by addressing the concerns of reviewers in the camera ready, such as

1.	Add experiments on 3Dshapes

2.	Add comparisons with other disentangled representation learning methods

3.	Improve the clarity of the paper
Decision: Accept (spotlight)