Self-Supervised Learning Disentangled Group Representation as Feature
Authors: Tan Wang
Zhongqi Yue
Jianqiang Huang
Qianru Sun
Hanwang Zhang
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Self-Supervised Learning
Feature Disentanglement
Invariant Risk Minimization
Group Representation

Abstract: A good visual representation is an inference map from observations (images) to features (vectors) that faithfully reflects the hidden modularized generative factors (semantics). In this paper, we formulate the notion of "good" representation from a group-theoretic view using Higgins' definition of disentangled representation, and show that existing Self-Supervised Learning (SSL) only disentangles simple augmentation features such as rotation and colorization, thus unable to modularize the remaining semantics. To break the limitation, we propose an iterative SSL algorithm: Iterative Partition-based Invariant Risk Minimization (IP-IRM), which successfully grounds the abstract semantics and the group acting on them into concrete contrastive learning. At each iteration, IP-IRM first partitions the training samples into two subsets that correspond to an entangled group element. Then, it minimizes a subset-invariant contrastive loss, where the invariance guarantees to disentangle the group element. We prove that IP-IRM converges to a fully disentangled representation and show its effectiveness on various benchmarks. Codes are available at https://github.com/Wangt-CN/IP-IRM.

Self-Supervised Learning Disentangled Group Representation as Feature
Tan Wang'
Zhongqi Yue! Jianqiang Huang
Qianru Sun
Hanwang Zhang'
Nanyang Technological University "Singapore Management University DDamo Academy, Alibaba Group
{tan317, yuez0003, hanwangzhang edu.
j lanqiang jqh@gmail.cmm com
qi gianrusundsmu. edu..gd
Abstract
A good visual representation is inference map from observations (images) features (vectors) that faithfully reflects the hidden modularized generative factors (semantics). In this paper, formulate the notion of "good" representation from group-theoretic view using Higgins' definition of disentangled representation [40), and show that existing Self-Supervised Learning (SSL.) only disentangles simple augmentation features such as rotation and colorization, thus unable to modularize the remaining semantics. To break the limitation, we propose an iterative SSL algorithm: Iterative Partition-basee successfully grounds the abstract semantics and the group acting on them into concrete contrastive learning. At each iteration, IP-IRM first partitions the training samples into two subsets that correspond an entangled group element. Then, it minimizes a mubset-invarian contrastive loss, where the invariance guarantees disentangle the group element. We prove that IP-IRM converges a fully disentangled representation and show its effectiveness on various benchmarks. Codes are available at httttp:://giii //githb. com/ CHIIPIIICIIIIIN
Invariant Risk Minimization (IP-IRM), which
Introduction
"reed"
=
g "green"
Deep learning is all about learning feature representa- tions [5). Compared to the conventional end-to-end super- vised learning, Supervised Learning (SSL) first learns generic feature representation (e.g., network backbone) by training with unsupervised pretext tasks such as the pre- vailing contrastive objective [36] [6], and then the above stage-1 I feature expected to serve various stage-2 appli- cations with proper fine-tuning. SSL for visual represen- tation is so fascinating that it is the first time that we can obtain "good" visual features for free, just like the trending pre-training in NLP community [26 S). However, most SSL works only care how much stage-2 performance SSL feature can improve, but overlook what feature SSL learning, why can be learned, what cannot be learned, what the gap between SSL and Supervised Learning (SL) is, and when SSL can surpass SL.?
U color digit
g u
U
color x digit
+
X' color digit
X
g
color
digit
red
digit
green
digit
Figure I: Disentangled representation is an equivariant map between the semantic space if and the vector space X, which is decom- posed into "color" and "digit".
The crux of answering those questions is to formally un- derstand what feature representation is and what good one is. We postulate the classic world model of visual generation and feature representation [[ 69 as in Figure I Let 2/ be a set of (unseen) semantics, e.g., attributes such as "digit" and "color". There
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Aug. Related
Aug. Unrelated
Layer 29
Layor
29
18
Layer 29
Simaai IP-1M
80
60
40
SimCLR IP-IRM
S0MCLR IP-IRM
(a)
(b)
Figure 2: (a) The heat map visualizes feature dimensions related to augmentations (aug. related) and unrelated augmentations (aug. unrelated), whose respective classification accuracy is shown in the bar chart below. Dashed bar denotes the accuracy using full feature dimensions. Experiment was performed on STLIO 2e] with representation learnt with SimCLR [16] and our IP-IRM. (b) Visualization of CNN activations 7ir of filters on layer 29 and 18 of VGG (75) trained on ImageNet100 [8]. The filters were chosen by first clustering the aug.
unrelated filters with k-means (k- 4) and then selecting the filters corresponding to the cluster centers.
is a set of independent and causal mechanisms [66 :: I. generating images from semantics, e.g., writing a digit "0" when thinking of "0" [74]. A visual representation the inference process I + X' that maps image pixels to vector space features, e.g., neural network. We define semantic representation as the functional composition f : l + I - X. In this paper, we are only interested in the parameterization of the inference process for feature extraction, but not the generation process, Le., we assume 47 E I, 3u E U, such that = =(u)) is fixed as the observation of each image sample. Therefore, representation, or simply representation, and we slightly abuse ((1)= := f ()) i.e., é and f share the same trainable parameters. We call the vector x = 6(I) as feature, where x E X.
consider semantic and visual representations the same feature
We propose to use Higgins' definition of disentangled representation 40 to define what is "good".
Definition 1. (Disentangled Representation) Let G be the group acting Ut ie., g.u. U x U transforms u E U, e.g., a "turn green" " group element changing the semantic "red" to "green". Suppose there is direct product decomposition g 91 X ... 9m and =lh x... Xmm where acts on Ud respectively. feauure representation is disentangled if there exists a group G acting on X such that:
1. Equivariant: 4g E G, Vu EU, (( e.g., the feature of the changed semantic: "red" to "green" in U, equivalent to directly change the color vector in X from "red" to "green".
2. Decomposable: there a decomposition X X1 ....X Xm, such that each X, is fixed by the action of all gj-j + i and affected only by gs. e.g., changing the "color" semantic in l does not affect the " "digit" vector in X.
Compared to the previous definition of feature representation which is a static mapping, the disentan- gled representation in Definition 1 is dynamic as explicitly incorporate group representation [35), which is homomorphism from group to group actions common to use the feature space X' as a shorthand this is where our title stands.
space, e.g., G - X' x X. and it is
Definition 1 defines "good" features in the common views: 1) Robustness: good feature should be invariant the change of environmental semantics, such as external interventions [45] 87 or domain shifts (32). By the above definition, change is always retained in subspace X. while others are not affected. Hence, the subsequent classifier will focus on the invariant features and ignore the ever-changing X, 2) Zero-shot Generalization: even if a new combination of semantics is unseen in training, each semantic has been learned as features. So, the metrics of each X, trained by seen samples remain valid for unseen samples [95].
Are the existing SSL methods learning disentangled representations? No. We show in Section 4 that they can only disentangle representations according to the hand-crafted augmentations, e.g., color jitter and rotation. For example, in Figure (a), even if we only use the augmentation-
feature,
Note that g: can also denote cyclic subgroup G: such as rotation [0 1 360"). or countable one but treated as cyclic such as translation [(0, (1, (width, height)] and color [0 1 255).

---

the classification accuracy of a standard SSL (SimCLR [16] does not lose much as compared to the full feature use. Figure 2 (b) visualizes that the CNN features in each layer are indeed entangled (e.g., tyre, motor, and background in the motorcycle image). In contrast, our approach IP-IRM, to be introduced below, disentangles more useful features beyond augmentations.
In this paper, we propose Iterative Partition-based Invariant Risk Minimization (IP-IRM |ai pa:m]) that guarantees learn disentangled representations in an SSL fashion. We present the algorithm in Section 3 followed by the theoretical justifications in Section 4 In a nutshell, at each iteration, IP-IRM first partitions the training data into two disjoint subsets, each of which is an orbit of the already disentangled group, and the cross- orbit group corresponds to an entangled group element 9:- Then, we adopt the Invariant Risk Minimization (IRM) (2 implement a partition-based SSL, which disentangles the representation X₂ w.r.f. gt. Iterating the above two steps eventually converges we show promising experimental results on various feature disentanglement and SSL benchmarks.
to fully disentangled representation w.r.f. [ gs- In Section 5
Related Work
Self-Supervised Learning. SSL aims to learn representations from unlabeled data with hand-crafted pretext tasks [28 63 [3 Recently, Contrastive learning [65]| 38 80 [6 prevails in most state-of- the-art methods. The key is to map positive samples closer, while pushing apart negative ones in the feature space. Specifically, the positive samples are from the augmented views 82 94 42 of each instance and the negative ones are other instances. Along this dircction, follow-up methods are mainly four-fold: 1) Memory-bant
61
storing the prototypes of all the instances computed previously into a memory bank to benefit from a large number of negative samples. 2) Using siamese network 77 to avoid representation collapse 34 19. 83.. 3) Assigning clusters samples integrate
190 61 36 18]
56]_ 4) Seeking hard negative samples with adversarial training or better sampling strategies 77 20 44 48). In contrast, our proposed IP-IRM jumps out of the above frame and introduces the disentangled representation into SSL with group theory to show the limitations of existing SSL and how to break through them.
inter-instance similarity into contrastive learning I 12 13
88
Disentangled Representation. This notion dates back to 44. and henceforward becomes a high- level goal of separating the factors of variations in the data [84] 79 86 [58] Several works aim to view [1 27 and measuring the properties of a disentangled representation explicitly (2 722. We adopt the recent group-theoretic definition from Higgins et al. (40), which not only unifies the existing, but also resolves the previous controversial points
provide a more precise description (27 29 72] by adopting an information-theoreet
591. Although supervised learning of disentangled (4)] unsupervised disentanglement based on is still believed to be theoretically challenging (59). Thanks to the Higgins' definition, we prove that the proposed IP-IRM converges with full-semantic disentanglement using group representation theory. Notably, IP-IRM learns disentangled represen- tation with an inference process, without using generative models as in all the existing unsupervised methods, making IP-IRM applicable even on large-scale datasets.
78
representation is well-studied field 100
43
10
77,
GAN 17 64
57 7 or VAE (39
15
99
50
1) a homo- morphism (e.g., a mapping function) from the group to its group action acting a vector space, and 2) the vector space. Usually, when there is no ambiguity, we can use either element as the definition. Most existing works focus on learning the first element. They first define the group of interest, such as spherical rotations
Group Representation Learning. A group representation has two elements [7] 35]:
[24] or image scaling 89 76). and then learn the parameters of the group actions [23] 46 68). In contrast, we focus the second element; more specifically, we
interested in learning a map between two vector spaces: image pixel space and feature vector space. Our representation learning is flexible because it delays the group action lcarning to downstream tasks on demand. For example, in a classification task, a classifier can be seen as a group action that invariant to class-agnostic groups but equivariant to class-specific groups (see Section
3 IP-IRM Algorithm
Notations. Our goal is to learn the feature extractor o in a self-supervised fashion. We define partition matrix P E (0, 1] that partitions N training images into disjoint subsets. P.,+ = if the i-th image belongs to the k-th subset and 0 otherwise. Suppose we have pretext task loss
3

---

function [ 8 = k. P) defined on the samples in the k-th subset, where A = 1 is a "dummy" parameter used to evaluate the invariance of the SSL loss across the subsets (later discussed Step 1). For example, & can be defined as:
exp (x' x²
L(0,0= 1,
Σ XEd
log
E EX'MUx exp (x" 0)
where X2 O(LL[P., = 1}). =
and x' E x is the augmented view feature of Xz.
Input. N training images. Randomly initialized ø. partition matrix P initialized such that the first column of P is 1, i.e., all samples belong to the first subset. Set P (P).
Output. Disentangled feature extractor Φb
Step [Update ΦI. We update e by:
min Σ Σ
1, P)12
(2)
PEP R=1
where \ is hyper-parameter. The second term delineates how far the contrast in one subset is from a constant baseline = The minimization of both them encourages é in different subsets close to the same baseline, i.e. invariance across the subsets. See IRM (2) for more details. In particular, the first iteration corresponds to the standard SSL with X; in Eq. containing all training images.
Step 2 [Update PJ. We fix o and find a new partition P by
P
arg max P
Σ
- do -,,0_,,,,
where λ2 is a
In practice, we use a continuous partition matrix Nx2 during optimization and then threshold it to {0, 1]
We update P PUP and iterate the above two steps until convergence.
Justification
Recall that IP-IRM uses training sample partitions to learn the disentangled representations w.r.f. IT 9s- As we have G-equivariant feature map between the sample space I and feature space X' (the equivariance is later guaranteed by Lemmal] [T. we slightly abuse the notation by using X to denote both spaces. Also, we assume that X' is a homogeneous space of G, i.e., any sample x e X can be transited from another sample x by a group action g x. Intuitively, G is all you need to describe the diversity of the training set. It is worth noting that is any group element in g while d: a Cartesian "building block" of G. e.g., 9 can decomposed by (01, Pt..9......... 9m.)
We show that partition and group are tightly connected by the concept of orbit. Given a sample X. its group orbit w.r.t. is a sample G(x) {g- x g G). As shown in Figure 3 (a), G is a set of attributes shared by classes, e.g., "color" and "pose ", the orbit is the sample set of the class of x; in Figure 3 if G denotes augmentations, the orbit is the of augmented images. In particular, we can see that the disjoint orbits in Figure naturally form a partition. Formally, we have the following definition:
47 Given a subgroup 'D C G, it partitions X into the disjoins subsets: (D(ci x).... ..., D(ck x)}, where le is the number of cosets (c. D), and the cosets form a factor G {ci) 1 j=r In particular, q; can be considered as sample of the i-th class, transited from any sample x E X.
Definition 2. (Orbit & Partition
Interestingly, the partition offers new perspective for the training data format in Supervised Learning (SL) and Self-Supervised Learning (SSL). SL, shown in Figure (a), (a), the data labeled with l classes, each of which is orbit with D(c; x) training samples, whose variations are depicted
'Given K with K X ... ... x cx. then D ((d, e) E D) a normal subgroup G, and g/Di is isomorphic 47 We write G/D f with slight abuse of notation.

---

Portition
de
IP-RM
(a) Supervised Learning
(b) Self-Supervised Learning
(c) Our IP-IRM
Figure 3: Each orbit only illustrates with 5 samples. (a) Orbit: the training samples of class; D in-orbit actions: intra-cla class variations (1-2:standingg 2-13:blacken, 3-Mjjuminng, 4-5::whten. 5-+L:running): g/D cross-orhit actions: inter-class variations. (b) Orbit: a sample and its augmented samples; D in-orbit actions: augmentations (1->2:clock-wise rotation, 2-33:color jitter, 3-+4gray scale, 1-#5xcouuterclookwiss rotation, 5->1:color); G/D cross-orbit actions: inter-sample variations. (c) Step 2 in IP-IRM discowers 2 orhits, where the cross-orbit action corresponds group action "green to red" or "red green", which is yet disentangled.
by the class-sharing attribute group D. The cross-orbit group action, e.g., Cdog X, can be read as "turn x into a dog" and such "turn" is always valid due to the assumption that X' is a homogeneous space of G. In SSL, as shown in Figure (b), each training sample is augmented by the group D. So, ((G x) consists of all the augmentations of the i-th sample, where the cross-orbis group action C . x can be read as "turn x into the i-th sample".
Thanks to the orbit and partition view of training data, we are ready to revisit model generalization in group-theoretic view by using invariance and equivariance name is disentanglement. For SI., we expect that a good feature is disentangled into a class-agnostic part and class-specific part: the former (latter) is invariant (equivariant) to /DD-cross--e traverse, but equivariant (invariant) to D-in-orbit traverse. By using such feature, a model can generalize to diverse testing samples (limited to (D] variations) by only keeping the class-specific feature. Formally, we prove that we can achieve such disentanglement by contrastive learning:
sides of the coin, whose
Lemma 1. (Disentanglement by Contrastive Learning) Training loss log exp(x x,) gles X w.r.t (G/D) x D, where x, and xj are from the same orbit.
Evex exp(X] x) disentan-
We can draw the following interesting corollaries from Lemma T (details in Appendix):
1. If we use all the samples in the denominator of the loss, can approximate to G-equivariant features given limited training samples. This because the loss minimization guarantees v(x,,x) x,) E X x Xi * x, & Xj, i.e., any pair corresponds to group action.
2. Conventional cross-entropy loss in SL is a special case, if we define x E X = (xi,..., x classifier weights. So, SL does not guarantee the disentanglement of G/D, which causes generalization error if the class domain of downstream task is different from SL pre-training, e.g., subset of G/D.
3. In contrastive learning based SSL, D = "augmentations" (recall Figure D. and the number of augmentations Daugg is generally much smaller compared to the class-wise sample diversity [Dst.] in SL. This enables the SL model to generalize to more diverse testing samples ( ((Dst.) by filtering out the class-agnostic features (e.g., background) and focusing on the class-specific ones (e.g., foreground), which explains why SSL is worse than SL. in downstream classification.
4. In SL, if the number of training samples per orbit is not enough, Le., smaller than [D(c, x)]. the disentanglement between D and G/D cannot be guaranteed, such as the challenges in few-shot learning (96). Fortunately, in SSL., the number is enough as we always include all the augmented samples in training. Moreover, we conjecture that Daug only contains simple cyclic group elements such as rotation and colorization, which are easier for representation learning.
I Lemma 1 does not guarantee the decomposability of each d E DD Nonetheless, the downstream model can still generalize by keeping the class-specific features affected by G/D. Therefore, the key to fill the gap even let SSL surpass SL is to achieve the full disentanglement of G G/Daug.
5

---

Method
DCI
IRS
MOD
EXP
LR
GBT
Average
VAE [1
0.948+0.04
0.664=0.121
0.968+0.07
0.824+0019
0.948=0004
0.849+0007
(--AE 41
0.945_00022
2.005
0.96310.006
0.809
0.945
0.874100:15
CMNIST
B-AnnealVAE
0,911=0002
0.790:00075
0.965
0.821
0.91 0.0.002
0.880=00066
B-TCVAE
0.914_00008
0. 0.864.00.085
0,96210.010
0.801
0,91410.008
0.89110004
Factor VAE 500
0.916-0004
0.893+0355
0.947+6.011
0.770+a.05
0.916+0.005
0.888+0014
SimCLR [6]
0.882-00019
0.767::0.025
0.97610.011
0.863100006
0.87610.015
0.873100066
IP-IRM (Ours)
0.9
0.785=00031
0.990=00002
0,921=0:00
0.916+0.007
0.906=0.011
VAE 15
0.351-0026
0.284+0.009
0.820+n.015
0.802+0.054
0,421
0.352+0.027
0.505+00.08
(-VAE E
0.369=00021
0.283
0.782
0,807
0.427
0.368
0.50620011
Shapes33
B-AnnealVAA 9
0.327=00069
0.412=0:049
0,743=00070
0.643=00.013
0.259=0.011
0,328
0.452=0002
8-TCVAE 15
0.470_00.035
0,291 10,025
0.707100051
0,821 10,054
0.409.0.004
0.46910.034
0.54510:002
Factor VAE [50)
0.340-0.02
0.316+0.016
0.815+0041
0.738+0.043
0.319+0045
0.339+0.021
0.478++0.00
SimCLR [6]
0.43910.000 0.005_00016
0.678
0.733 10,00
0,53610.015
0.645.10.026
IP-RRM (Ours)
0.565+0.m23
0.420+0.014
0.766+00036
0.959-0007
0.757=0022
0,565=0002
0.672+0.017
Table I: Results on disentanglement metries of existing unsupervised disentanglement methods, standard SSL (SimCLR (T6) and IP-IRM using CMNIST [2] and Shapes3D [50]. Note that IRS is based on intervening the
semantics which requires access to the labels of all the semanties, and hence not applicable for CMNIST dataset. Results are averaged over trails (mean + std).
Theorem 1. The representation is fully disentangled w.r.1. g/Dang if and only if E 9/D mug the contrastive loss in Eq. (1 invariant to the orbits of partition (g' x). g'( x)}, where g' = g/c = Daug x X Ci-1 X C++1 X .... X Ck.
The maximization in Step is based on the contra- cnntra-position of the sufficient condition of Theorem Denote the currently disentangled group as D (initially Dang). If we can find partition P to maximize the loss in Eq. 33] i.e., SSL loss is variant across the orbits, then 3h E G/D such that the representation of h entangled, i.e., P (D(h- x), D(h-1 x)). Figure 3 (c) illustrates
discovered partition about color. The minimization in Step 1 is based on the necessary condition of Theorem Based on the discovered P if we minimize Eq. (2). we can further disentangle and update D + 'D x h. Overall, IP-IRM converges as G G/Daug is finite. Note that an improved contrastive objective 92 can further disentangle each Daug and achieve full disentanglement
w.r.t. G.
Experiments
5.1 Unsupervised Disentanglement
Datasets. We used two datasets. CMNIST (2 has 60,000 digit images with semantic labels of digits (0-9) and colors (red and green). These images differ in other semantics (e.g., slant and font) that are not labeled. Morcover, there is strong correlation between digits and colors (most 0-4 in red and 5-9 in green), increasing the difficulty to disentangle them. Shapes3D (50) contains 480,000 images with 6 labelled semantics, i.e., size, type, azimuth, as well as floor, wall and object color. Note that we only considered the first three semantics for evaluation, as the standard augmentations in SSL will contaminate any color-related semantics.
Settings. We adopted 6 representative disentanglement metries: Disentangle Metric for Informa-
tiveness (DCI) [29], Interventional Robusiness Score (IRS) (79], Explicitness Score (EXP) [72),
Modularity Score (MOD) [72 and the accuracy of predicting the ground-truth semantic labels by two classification models called logistic regression (LR) and gradient boosted frees (GBT) (59]. Specifically, DCI and EXP measure the explicitness, i.e., the values of semantics can be decoded from the feature using linear transformation. MOD and IRS measure the modularity, i.e., whether each feature dimension is equivariant to the shift of a single semantic. See Appendix for more detailed formula of the metrics. In evaluation, we trained CNN-based feature extractor backbones with comparable number of parameters for all the baselines and our IP-IRM. The full implementation details are in Appendix.
Results. Table [ we compared the proposed IP-IRM to the standard SSL method SimCLR [16 as well as several generative disentanglement methods [1 15 50 On both CMNIST and
Shapes3D dataset, IP-IRM outperforms SimCLR regarding all metries except for only IRS where the most relative gain is 8.8% for MOD. For this MOD, we notice that VAE performs better than

---

CMNIST
STL10
"aipllaee"
Thome's
Ship
"airsione"
Shhg
horre
"ildd"
"trucl
"dea"
"dea
SimCLR
IP-IRM
SimCLR
IP-IRM
(a)
(b)
Figure 4: The t-SNE
visualizations of learned feature spaces using SimCLR 16] and IP-IRM on CM-
NIST [2] and STL.0 22 For CMIST in (a), annotate the digit and color near each cluster. We annotate
only half of the feature points for SimCLR to avoid clutter. For STL10 in (b), show the labels of the classes.
[Color]
SimCLR
IP-IRM
0
Subber
Color
0
10 00 O
0
CINNIST
Partition #2
[Right
Digit
0
22
Size
Shapes3D
8
Partition
[Slant]
Azimuthh Type
Subset
12
0
0
O
22
(a)
(b)
Figure 5: Visualization of the obtained partitions P" during training. Each partition has subset and the displayed images are randomly sampled from each subset. (b) Visualization the variance of each feature dimension when perturbing the semantic indicated on the left. The most equivariant dimensions are indicated by triangles and their corresponding indices.
IP- IP-IRM by 6 points, i.e., 0.82 0.76 for Shapes3D. This is because VAE explicitly pursues high modularity score through regularizing the dimension-wise independence in the feature space. However, this regularization is adversarial to discriminative objectives [14 95). Indeed, we can observe from the column of LR (i.e., the performance of downstream linear classification) that VAE methods have clearly poor performance especially on the more challenging dataset Shapes3D. We can draw the same conclusion from the results of GBT. Different from VAE methods, our IP-IRM is optimized towards disentanglement without such regularization, and is thus able to outperform the others in downstream tasks while obtaining a competitive value of modularity.
What do IP-IRM features look like? Figure 4 visualizes the features learned by SimCLR and our IP-IRM on two datasets: CMNIST in Figure 4 (a) and STL10 dataset in Figure (b). In the following, we use Figure 4 (a) as the example, and can easily draw the similar conclusions from Figure(((b). On the left-hand side of Figure 4 (a), it is obvious that there is no clear boundary to distinguish the semantic of color in the SimCLR feature space. Besides, the features of the same digit semantic are scattered two regions. On the right-hand side of (a), we have 3 observations for IP-IRM. 1) The features are well clustered and each cluster corresponds either digit or color. This validates the equivariant property of IP-IRM representation that it responds to any changes of the existing semantics, e.g., digit and color on this dataset. 2) The feature space has the symmetrical structure for each individual semantic, validating the decomposable property of IP-IRM representation. More specifically, i) mirroring a feature (W.E. "** in the figure center) indicates the change on the only semantic of color, regardless of the other semantic (digit); and ii) counterclockwise rotation (denoted by black arrows from same-colored to 7) indicates change on the only semantic of digit. 3) IP-IRM reveals the true distribution (similarity) of different classes. For example, digits 3, 5, 8 sharing sub-parts (curved bottoms and turnings) have closer feature points in the IP-IRM feature space.
specific semantic of
7

---

How does IP-IRM disentangle features? 1) Discovered : To visualize the discovered partitions P each maximization step, we performed an experieen on a binary CMNIST (digit 0 and in color red and green), and show the results in Figure 5 (a). Please kindly refer to Appendix for the full results on CMNIST. First, each partition tells apart a specific semantic into two subsets, egg. in Partition #1, red and green digits are separated. Second, besides the obvious semantics digit and color (labelled on the dataset), we can discover new semanties, e.g., the digit slant shown in Partition #3. 2) Disentangled Representation: In Figure (b), we aim to visualize how equivariant each feature dimension is to the change of each semantic, le., a darker color shows that a dimension is more equivariant w.E.f. the semantic indicated on the left. We can see that SimCLR fails to learn the decomposable representation, e.g., the 8-th dimension captures azimuth, type and size in Shapes3D. In contrast, our IP- IRM achieves disentanglement by representing the semantics into interpretable dimensions, e.g., the 6-th and 7-th dimensions captures the size, the 4-th for type and the 2-nd and 9-th for azimuth on the Shapes3D. Overall, the results support the justification in Section 4 Le., we discover a new semantic (affected by h) through the partition P at each iteration and IP-IRM eventually converges with disentangled representation.
5.2 Self-Supervised Learning
Datasets and Settings. We conducted the SSL evaluations on 2 standard benchmarks following [8
20 48). Cifar100 [54] contains 60,000 images in 100 classes and STL10 [22] has 113,000 images
baselines, and learned the representations for 400 and 1000 epochs. We evaluated both linear and k-NN (k- = 200) accuracies
in 10 classes. We used SimCLR [16]. DCL (20) and HCL 488]
for the downstream classification task. Implementation details are in appendix.
Results. We demonstrate our results and compare with baselines in Table 2 Incorporating IP-IRM to the 3 base- lines brings consistent performance boosts to downstream classification models in all settings, e.g., improv- ing the linear models by 5.55% on STLI0 and 2.92% on Cifar 100. In particular, we observe that IP-IRM brings huge performance gain with k-NN classifiers, e.g. 4.23% using HCL+IP-IRA
STL10
Cifar100
Method
k-NN
Linear
k-NN
Linear
400 epoch training
SimCLR
16
73.60
78.89
54.94
66.63
DCL
20
78.82
82.56
57.29
68.59
HCL
48
80.06
87.60
59.61
69.22
SimCLR-IP-IRM
79.66
84,44
59.10
69.55
DCL+IP-IRM
81.51
85.36
58.37
68.76
HCL+IP-IRM
84.29
87.81
60.05
69.95
1,000 epoch training
STL10, i.e. the distance metrics in the IP-IRM fea- ture space more faithfully reflects the class semantic differences. This validates that our algorithm further disentangles compared to the stan- dard SSL Moreover, by extending the training process to 1,000 epochs with MixUp (55). SimCLR+IP-IIE achieves further performance boost on both datasets, e.g., 5.28% for N-NN and 4.35% for linear classifier over SimCLR baseline on STL dataset. Notably, our LimCLR+IP-IRE surpasses vanilla supervised learning Cifar under the same evaluation setting. Still, the quality of disentanglement cannot be fully evaluated when the training and test samples are identically distributed- representation is more equivariant to class semantics, does not reveal if the representation is decomposable. Hence we present out-of -ut-of-distribution (OOD) setting in Section 5.3 to further show this property.
SimCLR
16
78.60
84.24
59.45
68.73
SimCLR 55
79.80
85.56
63.67
72.18
SimCLR SI +IP-IRM
85.08
89.91
65.82
73.99
Supervised
73.72
Supervised -MixU 97
74.19
Table 2: Accuracy (%) of k-NN and linear classifiers on STL10 (2
and Cifar100 [S4] using the representations of SimCLR [16].
DCL (20). HCL
488 and those after incorporating our IP-
IRM. SimCLR denotes SimCLR with MixUp regularization. Supervised" represents the supervised learning that keeps the same codehase, optimizer and parameters with SSL. stage-2 fine-tuning while only adds the learning rate decay at 60 and 80 epoch.
the improved accuracy demonstrates that IP-IRM
Is IP-IRM sensitive to the values of hyper parametertt 1) X and X2 in Eq. 2 and Eq. (3) In Figure (a), we observe that the best performance is achieved with X and X2 taking values from 0.2 to 0.5 on both datasets. All accuracies drop sharply if using \ 1.0. The reason is that a higher X forces the model to push the d-induced similarity to fixed baseline 0 = 1, rather than decrease the loss L on the pretext task, leading to poor convergence. 2) The number of epochs. In Figure (b),
8

---

84
STL10
01
STL10
80.53
02 24
83.00
80:45
03
80.27
78.76
80.73
79.50
K-NN Top-1 Acc (%)
IP-IRM
1.0
SimCLR
X
-35
Cifar100
83:87
Cfar100
67.17
IPJRM
0.5
85.90
66.77 76
65
70
65.90
-50
06.07
Epochs
(a)
(b)
Figure 6: Our ablation study on the STL10 and Cifar100 datasets. (a) The Top-1 accuracy (%) of linear classifiers using different values of λı and λ2 (in Eq. 2 and Eq. (3. by training for 200 epochs on two datasets. (b) The Top-1 accuracy (%) of k-NN classifiers updated every 50 epochs.
datasets, for which we trained the models for 700 epochs and
plot the 1 accuracies of using k-NN classifiers along the 700-epoch training of two kinds of SSL representations- -SNssS-SiI.RRR and IP-IRM. obvious that IP-IRM converges faster and achieves higher accuracy than SimCLR. It is worth to highlight that the STL10, the accuracy of SimCLR starts oscillate and grow slowly after the 150-th epoch, while ours keeps on improving. This an empirical evidence that IP-IRM keeps on disentangling more and more semantics in the feature space, and has the potential of improvement through long-term training.
5.3 Potential on Large-Scale Data
Datasets. We evaluated on the standard benchmark of supervised learning ImageNet ILSVRC- 2012 (25) which has in total 1,331,10 1,67,667 images in 1,000 classes. To further reveal if representation decomposable,
used NICO (37, which is real-worlde image dataset designed for OOD eval- uations. It contains 25,00 5,,000 images in 19 classes, with strong correlation between the foreground dogs on grass). We also studied the transferabil-
and background in the train split (e.g.,
ity of the learned representation following [30 52]: FGVC Aircraft (Aircraft) [60], Caltech-101
(Caltech) (3I] Stanford Cars (Cars) [93] Cifar10 (53. Cifar100
[53] DTD 21]. Oxford 102
Flowers (Flowers) 62), Food-101 (Food) (6. Oxford- Pets (Pets) 67 and SUN397 (SUN) (91).
These datasets include coarse- to fine-grained classification tasks, and vary the amount of train-
ing data (2,000-75,00 75,000 images) and classes (10-397 classes), representing wide range of transfer learning settings.
Settings. For the ImageNet, all the representations were trained for 200 epochs due to limited computing resourees. We followed the common setting (80 36), using linear classifier, and report Top-1 classification accuracies. For NICO, we fixed the ImageNet pre-trained ResNet-50 backbone and fine-tuned the classifier. See appendix for more training details. For the transfer learning, we followed 30 52 to report the classification accuracies on Cars, Cifar- 10, Cifar-100 DTD, Food, SUN and the average per-class accuracies on Aircraft, Caltech, Flowers, Pets. We call them uniformly as Accuracy. We used the few-shot n-way-k-shot setting for model evaluation. Specifically, we randomly sampled 2,000 episodes from the test splits of above datasets. An episode contains n classes, each with & training samples and 15 testing samples, where we fine-tuned the linear classifier (backbone weights frozen) for 100 epochs on the training samples, and evaluated the classifier on the testing samples. We evaluated with k (results of n 5, k 20 in Appendix).
ImageNet and NICO. In Table ImageNet accuracy, our IP-IRM achieves the best performance over all baseline models. Yet we believe that this does not show the full potential of IP-IRM, because ImageNet a larger-scale dataset with many semantics, and it hard achieve full disentanglement of all semantics within the limited 200 epochs. To evaluate the feature decomposability of IP- IRM, we compared the performance on NICO with various SSL baselines in Table 3 where our approach significantly outperforms the baselines by 1.5-4.2% This validates IP-IRM feature is more decomposable-if each semantic feature (e.g., background) decomposed some fixed dimensions and some classes vary with such semantic, then the classifier will recognize this as

---

Blephan
Shoap
Method
ImageNet
NICO
Moño-v2 Input
InsDis [90
56.5
65.6
PCL. [56
61.5
72.6
PIRL [6]
63.6
69.1
MoCo-vl 36
60.6
69.3
SimCLR (repro.)
16
63.1
64.5
MoCo-v2 (repro.)
18
67.3
78.0
SimSiam (repro.)
19
68.8
66.7
McCo-Ng
SiMCLR=IP-IRM
64.8
66.7
+IP
MoCo-v2+IP-IRM
67.6
79.5
SimSiam=IP-IRN
69.1
70.9
Figure 7: Visualization of CAM [8] im- ages from NICO 37 dataset using represen- tations of the baseline MoCo-v2 our IP-IRM.
Table 3: ImageNet and NICO Top-1 Accuracy (%) of linear classifiers trained on the representations learnt with different SSL methods.
Method
Aircraft
Caltech
Cars
Cifarl0
Cifar 100
DTD
Flowers
Food
Pets
SUN
Average
InsDis 90
35.07
75.97
37.49
51.49
57.61
69.38
77.35
50.01
66.38
74.97
59.57
PCL 56
36,86
90.72
39.68
59.26
60.78
69.53
67.50
57.06
88.31
84.51
65.42
PIRL 61
36.70
78.63
39.21
49.85
55.23
70.43
78.37
51.61
69.40
76.64
60.61
McCo-vl 36
35.31
79.60
36.35
46.96
51.62
68.76
75.42
49.77
68.32
74.77
58.69
MoCo-v2
18
31,98
92.32
41.47
56.50
63.33
78.00
80.05
57.25
83.23
88.10
67.22
IP-IRM (Ours)
32.98
93.16
42.87
60.73
68.54
79.30
82.68
59.61
85.23
89,38
69.44
Table 4: Accuracy (%) of 5-way-5- few-shot evaluation using the image representation learned on Ima-
geNet 25
More detailed results are given in Appendix.
non-disrrmminative variant feature and hence focus other more discriminative features (i.e., foreground). In this way, even though some classes are confounded by those non-discriminative features (e.g., most of the "dog" images with "grass" background), the fixed dimensions still help classifiers neglect those non-discriminative ones. We further visualized the CAM (98] on NICO in Figure which indeed shows that IP-IRM helps the classifier focus on the foreground regions.
Few-Shot Tasks. As shown in Table 4 our IP-IRM significantly improves the performance of 5-way-5-shot setting, e.g., outperform the baseline MoCo-v2 by 2.2%. This is because IP-IRM can further disentangled (PDang over SSL, which is essential for representations to generalize to different downstream class domains (recall Corollary 2 of Lemma 1). This is also in line with recent works [66 showing that a disentangled representation is especially beneficial in low-shot scenarios, and further demonstrates the importance of disentanglement in downstream tasks.
Conclusion
We presented an unsupervised disentangled representation learning method called Iterative Partition- based Invariant Risk Minimization (IP-IRM), based Self-Supervised Learning (SSL). IP-IRM iteratively partitions the dataset into semantic-related subsets, and learns a representation invariant across the subsets using SSL with IRM loss. We show that with theoretical guarantee, IP-IRM converges with a disentangled representation under the group- -heoortttreetaall view, which fundamentally surpasses the capabilities of existing SSL and fully-supervised learning. Our proposed theory is backed by strong empirical results in disentanglement metrics, SSL classification accuracy and transfer performance. IP-IRM achieves disentanglement without using generative models, making widely applicable on large-scale visual tasks. As future directions, we will continue to explore the application of group theory in representation learning and seek additional forms of inductive bias for faster convergence.
Acknowledgments and Disclosure of Funding
The authors would like to thank all reviewers for their constructive suggestions. This research partly supported by the Alibaba-NTU Joint Research Institute, the S under its AME YIRG Grant (Project No. A20E6c0101), and the Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 2 grant.
10

---

References
[1] Philip W Anderson. More different. Science, 1972.
[2] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani. and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint arXiv: 1907.02877,9 2019.
[3] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint 19066 00900, 2019.
[4] Yoshua Bengio. Learning deep architectures for Al. Now Publishers Inc. 2009.
[5] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 1888)1179--1888, 2013.
[6] Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 discriminative components with random forests. In European conference on computer vision, 2014.
[7] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. Advances in neural information processing systems, 6:737-744, 1993.
[8]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ily Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
[9] Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in B-vae. arXiv preprint 1.X.v:/804.03599 2018.
[10] Ruichu Cai, Zijian Li, Pengfei Wei, Jie Qiao, Kun Zhang, and Zhifeng Hao. Learning disentangled semantic representation for domain adaptation. In IJCAI: proceedings of the conference, 2019.
[11] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference Computer Vision (ECCV), pages 132-149, 2018.
[12] Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre- training of image features non-curated data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2959-2968, 2019.
[13] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.088 09882, 2020.
[14] Long Chen, Hanwang Zhang, Jun Xiao, Wei Liu, and Shih-Fu Chang. Zero-shot visual recognition using semanties presttivin adversarial embedding networks. In CVPR, 2018.
[15] Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentanglement variational autoencoders. In Advances in neural information processing systems, 2018.
[16] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[17] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, 2016.

---

[18] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04223 042977 2020.
[19] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint arXiv:201 1. 10566, 2020.
[20] Ching- Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. arXiv preprint arXiv:2007.00224, 2020.
[21] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos. Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2014.
[22] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
[23] Taco Cohen and Max Welling. Learning the irreducible representations of commutative lie groups. In International Conference on Machine Learning, 2014.
[24] Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. In ICLR, 2018.
[25] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-22-22 255. leee, 2009.
[26] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), 2019.
[27] Kien Do and Truyen Tran. Theory and evaluation metrics for learning disentangled representa- tions. In International conference on learning representations, 2020.
[28] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pages 1422-1430, 2015.
[29] Cian Eastwood and Christopher Williams. A framework for the quantitative evaluation of disentangled representations. In International conference on learning representations, 2018.
[30] Linus Ericsson, Henry Gouk, and Timothy M. Hospedales. How Well Do Self-Supervised Models Transfer? In CVPR, 2021.
[31] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop. 2004.
[32] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Doomain-avversriia training of neural networks. The Journal of Machine Learning Research, 2016.
[33] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.055 1803.07728, 2018.
[34] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent: A new approach self-supervised learning. arXiv preprint 2rXiv:2006.07733, 2020.
[35] W.F.J Harris, W. Fulton, and J. Harris. Representation Theory: A First Course. 1991.
[36] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv: 2019.
12

---

[37] Yue He, Zheyan Shen, and Peng Cui. Towards non-iid image classification: A dataset and baselines. Pattern Recognition, 110:107383, 2021.
[38] Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International Conference on Machine Leurning, pages 4182-4192. PMLR, 2020.
[39] I. Higgins, Loic Matthey, A. Pal, C. Burgess, Xavier Glorot, M. Botvinick, S. Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
[40] Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.022:: 2018.
[41] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. International conference on learning representations,
2017.
[42] R Devon Hjelm, Alex Fedorov, Samuel Marchi-Mar,,,,onn, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1808. 06670, 2018.
[43] Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Learning decompose and disentangle representations for video prediction. In Advances in neural information processing systems, 2018.
[44] Qianjiang Hu, Xiao Wang, Wei Hu, and Guo-Jun Qi. Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. arXiv preprint #rXiv:2011.08435 2020.
[45] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples not bugs, they features. In Advances in Neural Information Processing Systems, 2019.
[46] Andrew Jaegle, Stephen Phillips, Daphne Ippolito, and Kostas Danillidis. Understanding image motion with group representations. 2018
[47] Thomas W. Judson. Abstract Algebra: Theory and Applicutions (The Prindle, Weber & Schmidi Series in Advanced Mathematics). Prindle Weber Schmidt, 1994.
[48] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. arXiv preprint arXiv:2010.01028 2020.
[49] Theofanis Karaletsos, Serge Belongie, and Gunnar Rätsch. Bayesian representation learning with oracle constraints. International conference on learning representations, 2015.
[50] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, 2018.
[51] Diederik Kingma and Max Welling. Auto-encoding variational bayes. ICLR, 2014.
[52] Simon Korbblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[53] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2012.
[54] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[55] Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee. i-mix: domain-agnostic strategy for contrastive representation learning. In ICLR, 2021.
13

---

[56] Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven CH Hoi. Prototypical contrastive learning unsupervised representations. arXiv preprint arXiv:2005.04900 0.0000 2020.
[57] Zinan Lin, Kiran Thekumparampil, Giulia Fanti, and Sewoong Oh. Infogan-cr and modelcen- trality: Self-supervised model training and selection for disentangling gans. In International Conference on Machine Learning, 2020.
[58] F. Locatello, M. Tschannen, S. Bauer, G. Rätsch, B. Schölkopf, and O. Bachem. Disentan- gling factors of variations using few labels. In 8th International Conference Learning Representations (ICLR), 2020.
[59] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Ractsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference machine learning. 2019.
[60] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv: 130.,555,,,,,, 2013.
[61] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707-6717, 2020.
[62] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, 2008.
[63] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision, pages 69 84. Springer, 2016.
[64] Utkarsh Ojha, Krishna Kumar Singh, Cho-Jui Hsieh, and Yong Jae Lee. Elastic-infogan: Unsupervised disentangled representation learning in class-imbalanced data. In Advances in neural information processing systems, 2020.
[65] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint urXiv:1807.0377 807,03748, 2018
[66] Giambattista Parascandolo, Niki Kilbertus, Mateo Rojas-Carulla, Carulla, and Bernhard Schölkopf. Learning independent causal mechanisms. In Proceedings of the 35th International Conference Machine Learning, pages
2018.
[67] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, 2012.
[68] Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled represen- tations and group structure of dynamical environments. Advances in Neural Information Processing Systems, 2020.
[69] Rajesh PN Rao and Daniel I. Ruderman. Learning lie groups for invariant visual perception. Advances in neural information processing systems, 1999.
[70] Scott Reed, Kihyuk Sohn, Yuting Zhang, and Honglak Lee. Learning to disentangle factors of variation with manifold interaction. In International conference on machine learning, 2014.
[71] Xuanchi Ren, Tao Yang, Yuwang Wang, and Wenjun Zeng. Do generative models know disentanglement? contrastive learning is all you need. arXiv preprint arXiv:2102.10544. 2021.
[72] Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddings with the f-statistic loss. In Advances in neural information processing systems, 2018
[73] Joshua Robinson, Ching-Yar Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint 2xX1v:2010.04522.22 2020.
14

---

[74] Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa. Kun Zhang, and Joris Mooij. On causal and anticausal learning. In Proceedings of the 29th International Conference Machine Learning, 2012.
[75] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations, 2015.
[76] Ivan Sosnovik, Michat Szmaja, and Arnold Smeulders. Scale-equivariant steerable networks. 2020.
[77] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 2014.
[78] Raphael Suter, Djordje Miladinovic, Stefan Bauer, and Bernhard Schölkopf. Interventional robustness of deep latent variable models. arXiv, 2018.
[79] Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disen- tangled causal mechanisms: Validating deep representations for interventional robustness. In International Conference on Machine Learning, 2019.
[80] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, arXiv preprint arXiv:1906.05844 2019.
[81] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, European conference on computer vision, 2020.
[82] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint arXiv:2005. 10243, 2020.
[83] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. arXiv preprint arXiv:2102.068111 2021.
[84] Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled representation learning gan for pose- invariant face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ( (CVPR), 2017.
[85] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
[86] Sjoerd Van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disentangled representations helpful for abstract visual reasoning? In Advances in neural information processing systems, 2019.
[87] Tan Wang, Chang Zhou, Qianru Sun, and Hanwang Zhang. Causal attention for unbiased visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[88] Xudong Wang. Ziwei Liu, and Stella X Yu. Unsupervised feature learning by cross-level discrimination between instances and groups. arXiv preprint arXiv:2008.0335 2020.
[89] Daniel Worrall and Max Welling. Deep spaces: Equivariance over scale. In NeurIPS, 2019.
[90] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733-3742. 2018.
[91] Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, 2010.
[92] Tete Xiao, Xiaolong Wang, Alexei Efros, and Trevor Darrell. What should not contrastive in contrastive learning. In International Conference on Learning Representations, 2021.
15

---

[93] Linjie Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. A large-scale car dataset for fine- aneeggrinedd categorization and verification. In 2015 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2015.
[94] Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6210-6219 2019.
[95] Zhongqi Yue, Tan Wang, Hanwang Zhang, Qianru Sun, and Xian-Sheng Hua. Counterfactual zero-shot and open-set visual recognition. In CVPR, 2021.
[96] Zhongqi Yue, Hanwang Zhang, Qianru Sun, and Xian-Sheng Hua. Interventional few-shot learning. In NeurIPS, 2020.
[97] Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
[98] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921-2929, 2016.
[99] Yizhe Zhu, Martin Renqiang Min, Asim Kadav, and Hans Peter Graf. S3vae: Self-supervised sequential vae for representation disentanglement and data generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, 2020.
[100] Zhenyao Zhu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Multi-view pereeptron: deep model for learning face identity and view representations. Advances in Neural Information Processing Systems 27 (NIPS 2014), 2014.
16


Reviews:
[{"review_id": "p06URvKpozA", "reviewer": "Reviewer_4t1m", "summary": "This paper proposes a scalable approach to disentangled representation learning using the self supervised learning framework, rather than the typically used generative model framework. Their approach is principled, being grounded in the Higgins et al definition of disentangling from the symmetry perspective. It also appears to be scalable, which is a very exciting step in the subfield of unsupervised disentangled representation learning.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written and makes a very exciting step in unsupervised disentangled representation learning. I am happy to argue for its acceptance. Saying this, I do have a few concerns.\n\n1) Is K a hyperparameter? How should one choose it? What were the choices for CMNIST and 3D shapes? Why does increasing it result in reduced performance? How do the authors propose to scale their algorithm to disentangling many different subspaces without the ability to increase K?\n\n2) What are the augmentations used, and how are the negative examples chosen? If the authors just augment the existing SSL algorithms with their native augmentations/negative examples with the proposed disentangling framework, then it should be stated in text, as otherwise the reader is left wondering. Is there a way to choose augmentations that would not interfere with disentangling, as the authors suggested e.g. that colour information was not disentangled because it was subsumed by the augmentations.\n\nMinor: Line 147 \"In particular, SSL and fully-supervised learning are two special cases of Gi-disentanglement\" - I think it would be better to move it out of the lemma, as this is not really a part of it, and instead add this to the following more in depth discussion of this point later in the section.\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "bwwcdM9Eb6U", "reviewer": "Reviewer_2DAu", "summary": "This paper proposes a self-supervised learning model which is a variant of the Invariant Risk Minimization with an iterative partition mechanism. The method can encourage the self-supervised learning of a disentangled representation. Empirical results validate the effectiveness of the proposed model on self-supervised learning and disentangled representation learning.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe proposed method depends on the Invariant Risk Minimization model. The disentanglement validation depends on the symmetry-based (SB) disentangled representation definition. The overall learning setting is the same as the general self-supervised learning. The core technical originality I personally believe is the iterative partition mechanism, but I would also consider the combination of IRM, SSL, and SB-disentanglement is novel and original as well as they are not directly related.\nQuality:\n1. The experimental results clearly validate the disentanglement and self-supervised learning quality as they improve the existing SSL baselines by large margins. \n2. However, my main concern is about the justification part (Section 4) of the paper, where I am not fully convinced by the proofs of Lemma 1 (and therefore Theorem 1). In Line 155, the paper states there are two possible cases if Lemma 1 doesn't hold. But why are there only these two cases? It looks like the line 155-156 and the appendix line 61-62 try to describe the same thing: (1) main paper line 155-156: 'f is a collapsed representation that maps all inputs to a fixed vector, which is impossible in today's deep model training [88]'; (2) appendix line 61-62: 'The representation is not equivariant w.r.t. the action of \\Pi G_i. As discussed in Lemma 1, this is impossible in today's deep model training [26].' However, nothing in the Lemma 1 discussion is related to the appendix statement about the equivariance with the product of subgroups, neither does the cited paper ([26] of the appendix). This makes me very confused about the correctness of the proof B.1 in the appendix. \nIn main paper lines 156-159, the paper cites [35] to say there is always a linear projection. But why is it related to the group G_{t+1}? And why is it related to the proof of Lemma 1?\nIf the SB-disentanglement is actually realized as claimed in section 4, can the authors show the specific learned groups (or subgroups) empirically? As long as I can see in the disentanglement experiments, what the paper tries to emphasize is the separation of latent subspaces, which corresponds to traditional disentangled representation learning without considering the groups that can actually act on them.\n3. The unsupervised disentanglement experiment only reports the means of the metrics and no standard deviations. Typically models should be reported with runs of multiple random seeds to get an evaluation on disentanglement performance.\nClarity:\nThere could be some improvements.\n1. There should be some preliminaries about IRM. At least a general introduction about what problem IRM is trying to solve, why it works, and why the SSL task falls into its setting.\n2. The justification part (Section 4) should be improved on its proof of Lemma 1 and Theorem 1 (see my concerns in the Quality part).\nSignificance:\nThe direction of improving the disentanglement property of self-supervised learning representations is promising and important. The results are significant and I believe it will benefit the representation learning community.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "Wz39fk2BMR", "reviewer": "Reviewer_Z5Cr", "summary": "In this paper, the authors focus on the problem of learning unsupervised representations that are also disentangled. To this end, they build on top of the idea of invariant risk minimization and propose Iterative Partition-based Invariant Risk Minimization (IP-IRM) which iteratively assigns examples to partitions and constrains invariance in the loss among these partitions. They justify their approach mathematically according to a formal definition of disentanglement. The then perform multiple experiments qualitatively and quantitatively demonstrating the learned representations are disentangled. Furthermore, they show promising results on Out-of-distribution classification tasks. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The paper is well written, flows well, and is easy to understand. \n\n2. The problem is very relevant. There has been a great deal of work on learning self-supervised representations in computer vision. However, much work has focused on improving image classification numbers and less on how well the representation is disentangled. Disentanglement is an essential foundation for abstract reasoning.\n\n3. The paper is very well motivated mathematically and conceptually. They show, using a formal definition of disentanglement, why IP-IRM is a sensible approach.\n\n4. Quantitative and qualitative experimentation is very thorough and convincing. \n\nWeaknesses:\n\n1. For the OOD experiments on Imagenet, NICO, and related, the improvements in performance over the state of the art is smaller.\n\nOverall:\n\nThe paper addresses an important but underexplored topic in self-supervised learning: disentangled representations. The authors firmly justify IP-IRM mathematically with Higgins's definition of disentanglement. Evaluation of the approach is thorough, and the quantitative and qualitative results convincing. I believe this would be a good contribution to the conference. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they have addressed limitations and social impacts.", "review_text": "", "extra_scores": {"ethical_concerns": "I see no ethical concerns with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 Hour"}, "extra_text": {}}, {"review_id": "DCv7U8NATJC", "reviewer": "Reviewer_nijM", "summary": "This paper presents a new self-supervised learning technique that learns to map image inputs to \"disentangled\" vectors iteratively by alternating two steps. First, all of the data is assumed to be in the same subset and they learn a representation by minimizing a SimCLR like contrastive loss (here the positive pairs are augmented versions of the same image, while negative samples are other images in the batch). In the second step, they find the partition of the dataset into 2 subsets that maximize the contrastive loss. This new partition is added to the set of all partitions considered, and we repeat the two steps (again learning the representation that minimizes contrastive loss over all partitions and then finding the next partition and so on). They use a regularized form of the contrastive loss, which adds a term taken from earlier work (\"Invariation Risk Minimization\"). This terms measures the magnitude of the gradient of the contrastive loss wrt to the temperature parameter (at temperature=1). They evaluate their technique against alternatives on a wide range of disentanglement metrics and downstream classification accuracy, and show their technique performs well.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think the paper is quite interesting and should be of interest to the community. I especially like the idea of finding the worst partition in terms of the representation learned so far and re-learning the representation based on this partition. I also found the evaluations in the paper quite comprehensive.\n\nI think the main issue with the paper is that it makes rather strong claims but fails to support some of these. For example, the authors argue that their technique can disentangle the full semantic space (i.e., learn a fully disentangled representation) in a fully unsupervised manner. (in contrast to other SSL techniques, which can only learn to disentangle the semantic information wrt hand-crafted augmentations.) However, to me it is not clear how this squares with the Locatello et al. result that shows that fully unsupervised learning of disentangled representations is not possible. They mention in passing that Higgins et al.'s group theoretic definition of disentanglement solves the problem with Locatello et al.'s result but this is not clear to me. Locatello et al. in essence shows that there are multiple equivalent representations and it is impossible to pick one over another as more disentangled without some supervision. And Higgins et al. also accept this fact as they point out in their paper that their disentanglement definition assumes a \"natural\" decomposition is already given (i.e., what it means to be disentangled). I should say that I found it difficult to fully grasp what the authors' claim is, so please let me know if I misunderstood anything.\n\nThis brings me to my other point. I think the paper right now is unfortunately hard to read. The paper gets rather technical at times, and it is hard to follow the argument (for example, discussion after Lemma 1 in Section 4 is hard to follow). While I appreciate the authors' presenting theoretical results, I think the paper would benefit greatly if the technique could be better motivated on an intuitive level, and the technical arguments similarly could be presented on an intuitive level.\n\nPlease also see my further comments below:\n- Figure 2 is hard to understand. What is being classified on the left panel? Also, it is not clear what I should see on the heatmaps on the right.\n- line 105-106, the authors say \"Higgins et al., ..., resolves the previous controversial points\" wrt to Locatello et al. for example. As I mentioned above, this is not clear to me. It'd be worth expanding this a little bit further.\n- What is the effect of regularizer term from \"Invariant Risk Maximization\"? The authors say it \"regularizes phi to be invariant across subset in a partition\". What does this mean exactly? Can you give an intuitive example of what this term does? What would happen if you didn't have this term?\n- line 155, the authors say \"f is a collapsed representation that maps all inputs to a fixed vector, which is impossible in today’s deep model training\". What does this mean? It is impossible because of the regularization or tricks like EMA that people use. I don't see how the referenced work [88] is relevant to this point either.\n- To me it seemed like finding the partition that maximizes loss is akin to finding hard negatives. Is this intuition correct? If so, would it be useful to mention sth along these lines in the paper?\n- For the results in Table 2, how do you combine IP-IRM with SimCLR etc.?\n- In the checklist, the authors mention limitations are discussed in section 6, but I couldn't see this.\n- I understand that the authors may not have had enough time to get standard deviations on all of the results, but it'd be nice to add these. (I have seen the results in appendix).\n- While reading this paper, I was reminded of the paper \"What Should Not Be Contrastive in Contrastive Learning\" ()https://arxiv.org/abs/2008.05659). This paper learns essentially a separate subspace for each augmentation (which encourages the representations not to throw away information like SimCLR etc. might do). It'd be interesting to see how well their method compares to this. (To be clear, this is totally optional. I understand the authors might not have time to work on this.)\n- typos:\n  - line 74, \"attributed first part\" -> \"attributed to first part\"\n  - line 218, \"regrading\"\n  - line 228, \"What does IP-IRM feature look like?\" -> \"What do IP-IRM features look like?\"\n  - line 239, \"regardless the other semantic\" -> \"regardless of the other semantic\"\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel approach to an important and popular problem, and in doing so provides a natural but novel application of *another* important and popular line of work (IRM). The work is more theoretically justified than most in the area, and has reasonably satisfying experimental results. Overall, it will make a fine contribution to the conference. To maximize that, please make sure to incorporate the outcome of the reviewer discussions – some of which seemed fairly illuminating – into the final version of the paper.
Decision: Accept (Spotlight)