Transferring disentangled representations: bridging the gap between synthetic and real images
Authors: Jacopo Dapueto
Nicoletta Noceti
Francesca Odone
Venue: NeurIPS 2024 poster (2024)
Keywords: Disentangled Representations
Transfer Learning
Syn2Real
Primary area: machine_vision

Abstract: Developing meaningful and efficient representations that separate the fundamental structure of the data generation mechanism is crucial in representation learning. However, Disentangled Representation Learning has not fully shown its potential on real images, because of correlated generative factors, their resolution and limited access to ground truth labels. Specifically on the latter, we investigate the possibility of leveraging synthetic data to learn general-purpose disentangled representations applicable to real data, discussing the effect of fine-tuning and what properties of disentanglement are preserved after the transfer. We provide an extensive empirical study to address these issues. In addition, we propose a new interpretable intervention-based metric, to measure the quality of factors encoding in the representation. Our results indicate that some level of disentanglement, transferring a representation from synthetic to real data, is possible and effective.

Transferring disentangled representations: bridging the gap between synthetic and real images
Francesca Odone jacopo. dapueto@edu. unige. .iige.it it cacollettanncettii. noceti, francesca. odcaae odnneeennnge......... it Mal.Ga-DIBRIS, Università degli studi di Genova, Genova, Italy
Jacopo Dapueto
Nicoletta Noceti
(nicoletta.
Abstract
Developing meaningful and efficient representations that separate the fundamental structure of the data generation mechanism is crucial in representation learning. However, Disentangled Representation Learning has not fully shown its potential on real images, because of correlated generative factors, their resolution and limited access to ground truth labels. Specifically on the latter, we investigate the possibility of leveraging synthetic data to learn ceneral-purpont disentangled representations applicable to real data, discussing the effect of fine-tuning and what properties of disentanglement are preserved after the transfer. We provide an extensive empirical study to address these issues. In addition, we propose a new interpretable intervention-base based metric, measure the quality of factors encoding in the representation. Our results indicate that some level of disentanglement. transferring a representation from synthetic to real data, possible and effective.
Introduction
Developing meaningful, reusable and efficient representations is a critical step in representation learn- ing 51 57 67). Disentangled Representation Learning (DRL) []] 40 24 67 aims to learn models that can identify and disentangle underlying Factors of Variation (FoVs), hidden in the observable data. These models encode them in an interpretable and compact shape BI 9 773,. independently
from the task at hand 22 39 66 67]. Moreover, DRL enhances explainability, robustness, and generalization capacity across various applications [67]. Disentangled representations have been shown useful for various downstream tasks, such as FoVs prediction 41 40], image generation
72 48 43
59] and translation 19 38 fair classification [56] 39] abstract reasoning [66] 63)
domain adaptation (5 and out-of-distribun
(OOD) generalization 11 [1]
While all the abovementioned methods may rely different definitions of disentanglement (see just as examples 23] 64)), and in this sense comprehensive comparison is hard, they usually share the observation that some level of supervision on the FoVs is beneficial for disentangiement. However, labelling every single factor to achieve fully supervised disentanglement is costly or even unfeasible For this reason, DRL has been mostly validated on synthetic simulated data, usually acquired on purpose [1 41 60), and there is limited understanding of the potential of DRL to address general- ceneral-purpose purpose representation tasks. as well as the specific challenges of the real world (e.g. the presence of clutter and occlusion, correlation between factors [II], etc.). Such challenges may prevent the model from learning perfectly disentangled representations (55]
70 52
In this work, we propose the adoption of Disentangled Representation (DR) transfer to deal with complex realistic/real dataset. DL. transferring was explored in [20). where Source models learnt in an unsupervised manner were transferred to a Target dataset, by transferring hyperparameters. The authors observed limited effectiveness in the direct transfer representations. Instead, Dittadi et al. | found out that disentangled representation can help in OOD generalization from simulated to smaller real dataset. both cases, the study involved very specific types of dataset, built to emulate
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the real one in every detail. Recently, Fumero et al. [18] addressed disentanglement in real data without the need for Fo annotation. leveraging the knowledge extracted from diversified set of supervised tasks to learn a common disentangled representation to be transferred to real settings. We follow a different direction, setting up very straightforward and generalizable procedure: we resort to weakly supervised approach 41 [5] 26 to learn DRs on Source datasets where the FoVs are known and annotated, to then transfer (with no supervision) such representation to a Target dataset where the Fo are not known available. Our final aim is to consider real datasets as a Target, while synthetic data (where Fo annotation is easy to obtain) can be employed as a Source.
The paper presents three main contributions: (1) a novel metric to assess the quality of disen- tanglement, which is interpretable, classifier-free and informative on the structure the latent representation; (2) DR transfer methodology to Target datasets without FoV annotation; (3) an extensive experimental analysis that considers different (Source, Target) pairs and quantitatively assesses the expressiveness of the learnt DR on Target of different nature (including the case where the gap between Source and Target is large), taking into consideration the main expected properties of disentangled representation. We discuss the role of fine-tuning and the need to reason on the distance between Source and Target datasets.
The paper is organized follows. In Section 2 we propose and discuss our new intervention-basse
metric, OMES. In Section 3.3] we introduce our transfer approach to DRL, and provide a thorough analysis of different types of transfer scenarios (synthetic to synthetic, synthetic to real, real to real). Section is left to the conclusions.
2 Evaluating the quality of disentanglement
2.1 Background
While there is no universally accepted definition of disentanglement, there is common agreement on the properties that DR should have 12 54
66 57
Modularity [55] A factor influences only a portion of the representation space, and only this factor influences this subspace. This is achievable if the FoV are independent, meaning that variation in one FoV does not affect others.
Compactness [55]
The subset of the representation space affected by a FoV should be as small as possible (ideally, only one dimension). This property is also called completeness in [13
Explicitness 54 DR should explicitly describe the factors, thus it should favour Fo classification.
The taxonomy presented in 12 groups all metrics in three families (see summary in Table 6 in App.): Intervention-based metrics compare codes by intervention, either creating subsets of data in which one or more factors are kept constant (BetaVAE 244 and FactorVAE (27). or in which only one factor is varying (RF-VAE (28),, and predicting which factors were involved in the intervention; Predictor-basee based metrics use regressors or classifiers to predict factors from DR (DCI Disentanglement [13] and SAP (2]) or intervened subsets (BetaVAE, FactorVAE and RF- VAE); information-basedd metries leverage information theory principles. such as mutual information, to
quantify factor-DR relationships (Mutual Information Gap (MIG) [ 12), MED (6), Modularity (55) and InfoMEC (25)).
Intervention- based metrics have the advantage of providing control over the factor and the corre- sponding representation. However, they are all based classifiers, thus they depend method, hyperparameter settings and model capacity. The latter consideration can be extended to all Predictor- based metrics. On the other hand, Information-basee of Mutual Information, which is dependent on an estimator and its parameters 7 Motivated by these limitations, we introduce in the next section new metric, to the best of our knowledge, the first classifier-fres intervention-base metric.
methods are mainly ground the computation
2.2 Our metric: OMES
OMES (Overlap Multiple Encoding Scores) is an intervention-based metric measuring the quality of factor encoding in the representation while providing information about its structure: we measure modularity, analyzing how the FoVs overlap, and compactness, detecting and quantifying how a factor is encoded in the dimensions of the representation.

---

Algorithm Compute association matrix S between dimensions and FoVs
Require: Dg R R2, k),
P n number Fo
Ensure: R² R22 E R k
RN
m== |A|, N number of pairs in D
1: S f- ZEROS( n)
2: for j= n do
3:
R) = == and R3 = R == j.:
4:
for
1 do
5:
PC = PearsonCors
6:
S[h. = abs(PC)
7: end for
8: end for
9: return S
m matrix
Algorithm 2 Overlap score of FoV j: OS
Algorithm 3 Encoding score of FoV j: MES
Require: matrix S, FoV index j
Require: matrix S, FoV index
Ensure: S E m
Ensure: S E Rmxx
scores +- ZEROS(m)
scores < ZEROS(m)
2: for h to m do > dimension h
2: for h = 1 to m do
dimension
3: is +- ZEROS(n); is
is ZEROS(m): is[h]= =
4: scores[h] -MAE(is. S(h,
4: scores[/] = -MAB(is. S(;
5: end for
end for
6: return POOLING(scores)
6: return POOLING((cores)
Given an image X, with + its mapping into a d dimensiooaal latent disentangled space, ((X) = r, r E IRd. We discard dimensions whose empirical standard deviation extremely small (< 0.05), meaning that the dimensions are inactive [68 10].This leaves us with subset of d active dimensions, to which we will refer in the following.
Let D be dataset formed by image pairs, D {(X) x2, &)) where X! X? are two images that differ for only the FoV k;
OMES requires computing weighted association matrix S between the dimensions of the representation and the FoVs, with higher association values if the factor is encoded in certain dimension (see Algorithm [ we consider the representations of the image pairs dataset D, obtaining Dz. In matrix notation we may write it as Dz [R R k), where the pair R' and R2 are N m-dimeensioall matrices with each row is the representation ((X)) r! and ((X?) r respectively. For each FoV & extract the rows of D such that the i-th entry of vector & is k; k. we call this set D Each entry S(h, of the association matrix relates a dimension of the estimated disentangled representation with FoV j. Its value in the range [0, 1] with elements close to 1 corresponding to dimension that effectively captures the variations of FoV. The association is based on a correlation analysis: since the samples from D are paired differ only for the FoV k, we expect good representation not to correlate where such FoV is encoded. To ease interpretability. we transform the obtained values (see Algorithm [ line 6) so that high values denote strong association between dimension and FoV.
the i-th image pair
When the model exhibits perfect disentanglement, each row and column of the association matrix S present just one element with a high association, corresponding to the only dimension where the factor is encoded. We thus measure the level of disentanglement through similarity with ideal array, where the association matrix shows all Os but the positions the correct associations, where there are 1s.
We rely on the above considerations to derive our metric as linear combination of two main contributions. The Overlup Score (OS) penalizes the overlap of different FoVs the same dimensions (Algorithm 2 in this case, each row of S, associated with a dimension, is compared with the ideal array) and hence measures Modularity, while the Multiple Encoding Score (MES) penalizes the encoding of the same factor into different dimensions (Algorithm in this case each column

---

Podkr
0.25
0.54
6:12
0.23
0.22
0:13
B/62
0:32
2.77
3,3
6:41
1:22
2400
8:00
0:02
0.22
506
G0T
0.55
0.21
0.29
6:12
0.88
0/9
3.4
0.24
0.70
0
0.33
0.22
0:76
0.20
0.41
0.31
04
0.22
0.2
0:8T
0:31
0.23
0.21
8:07
08
23
5.0
3.3
10.0
123
5:3
044
cdartderr
D:72
500
DED
0.57
PaUSerO
Disotargione Intonaayy
Figure 1: Dataset Noisy-dSprites: Left:: Scores of the proposed metrie for each FoV, a is fixed to 0.5. Center Left:: Association matrix of an unsupervised model (33 = 6). Center Right: Association matrix of a weakly-supervin model. Right: Scores of synthetic Association matrices simulating underfitting, partial disentanglement and almost perfect disentanglement.
of S corresponding to FoV is compared with the ideal array) measuring Compactness. In both algorithms, we derive a vector summarizing the contribution of all dimensions for the FoV. The final score in [0, 1] (higher values meaning higher disentanglement) can be obtained with pooling (either MAX or AVERAGE) on the vector. The OMES metric is computed as
OMES(S)- = n Σ OS(S, - a)) (S)
With a = 0, OMES only measures the Compactness of the representation (MES component); with = 1, instead, our metric measures the Modularity only (with OS). Values of a in the interval (0, 1) can be used balance the importance of both contributions.
Relation with existing metries. To the best of our knowledge, the only metrics capturing more than one property are DCI [13] and the very recent InfoMEC (25). Differently from DCI, our metric is intervention-based with no influence on the choice of the specific classifier that may inevitably impact the results, as observed in 77 With respect to InfoMEC, that must be applied to quantized latent codes, our metric is more general and accepts continuous latents.
OMES based on the intervention of the FoVs, thus we require the FoV to be (at least partially) known: in particular, samples are coupled so that they differ in one FoV only. In this, OMES differs from existing intervention-basen metrics [4 27 in which the intervention the opposite (samples have only one FoV in common). Our pairing requires less supervision, and it is usually easier to obtain during data acquisition (for instance, from videos 41 In addition, it has been shown that this type of pairing provides more guarantees on disentanglement properties 41.
methods, we exploit Correlation instead of Mutual Informa- tion, hence we do not need its estimation that can be sensitive to parameters choice (e.g. granularity of the discretization 7
Finally, compared to Information-
and choice estimator [51 7
2.3
OMES assessment
We now analyze OMES, extending previous studies on the unsupervised [40] and weakly supervised
41 setting. As for the unsupervised case, we exploit available 5400 trained models from 40
(3
datasets, 6 values for B, 50 random seeds, 6 unsupervised methods; VAE (4), FactorVAE [27].
3-TCVAE B VAE- (22, VIP-VAE-II 32 Annealed (5); in this section we report an
analysis on Noisy-dSprites, the remaining 2 benchmarks can be found in the Appendix [.. and B.2
Instead, for the weakly supervised case trained models are not available, so we reproduce the models as in 41 training them on Shapes3D and on other datasets that can be found in Appendix [B3
OMES interpretation. The metric, by construction, allows us to compute the overall score and score for each FoV separately: we can thus interpret the effect of hyperparameters on the single FoV, and evaluate the FoV separately in each dimension of the representation. Moreover, by inspecting the metric at a factor level, we may identify uneven behaviours (e.g. models performing similarly on average but for different contributions from the factors).
Fig. I (Left) shows the metric scores for different values of 33 keeping the different FoV separated: the FoVs less affected by reconstruction (e.g. PosX and PosY) exhibit an increasing disentanglement score as 3 grows. On the other hand, Shape and Orientation present maximum value around = and then decrease because they are more susceptible to the reconstruction quality, which degrades for larger values of 3. In Fig. (Center Left) an association matrix S is generated from one
4

---

Shapw
BeteMEE
00
80
41
47
37
11
06
E1.00
60
51
54
92
13
FoctoWE (.0)
80
100
52
26
39
28
MBG
49
100 76
43
50
08110000
DC1
100
1UL10000
64
29
19
42
50
Voduartty
NEDOTA
-46
-34
6N
(0)
(G)
Figure 2: Left: Rank-correlation between metrics of models trained on Noisy-dSprites. Center: Scores distribution of the metrics on Noisy-dSprites. Right: Rank correlations (Spearman) of ELBO, reconstruction loss, and the test accuracy of GBT and MLP classifier trained on 10,000 labelled data points with disentanglement metrics. In all plots OMES computed with a 0.5.
of the unsupervised models (8- (A-VAH) trained with 33 6: Shape and Orientation encoded in the same dimension (overlapping) and produce lower values because of the reconstruction, while PosX and PosY are encoded in multiple dimensions and mostly overlapping. Scale does not seem to be well represented. In Fig. (Center Right) we report the association matrix obtained by weakly-supervised model (Ada-GVAE): the factors PosX, PosY and Scale are disentangled while Shape and Orientatin in are encoded in the representation with high intensity in different dimensions, with overlaps. T (Right) shows OMES values (a = 0.5) computed over synthetic association matrices S, obtained by perturbing the ideal (diagonal) one. The perturbation aims to simulate scenarios: noisy matrices where the disentanglement can be more less strongly derived; models exhibiting partial disentanglement; models with no disentanglement. As it can be appreciated, the metric score nicely reflects the disentanglement intensity.
Agreement of OMES with other disentanglement metries. We extend the analysis in 440 The rank-correlation (e.g. Fig. 2 (Left)) between the previously proposed metrics and our OMES (for 0.5) shows that the latter has high level of correlation with MIG and DCI, but mild correlation with Beta VAE, VAc (OMES is based on the opposite intervention type), and Modularity. This consistent on all benchmarks. We show in Fig. (Center) the score distribution of the metries, computed on the whole set of models. We observe OMES produces wider range of values with respect to MIG and DCI: our metric looks more descriptive, similarly to Beta VAE and FactorVAR
Agreement with performance metrics. Similarly 41 supervised setting and discuss the rank correlation between our metric and performance evaluations (ELBO, reconstruction loss, and test error of FoVs classifier). Our analysis, reported in Fig. 2 (right) shows that OMES performs similarly to DCI, negatively correlated with the Reconstruction loss, and positively with the ELBO. also correlates with the performances of GBT10000 (the classifier we will use in the experiments) while it mildly does with MLP10000. This empirical evidence is in line with what was observed in [T]. The correlation with the classification score sign that OMES able to capture the property of expliciteness of the representation, although it is not directly measured by our metric. is worth mentioning the correlation of OMES with the performance metrics is more stable than what is obtained by other metrics across different datasets (see the Appendix B.3
consider the more informative weakly-
3 Transferring disentangled representations
Fully unsupervised disentangled representation learning has been shown unsatisfactory in many scenarios [40]. However, annotating the FoVs can be a very critical and uncertain process. this section, propose ceneral-purposs. methodology for transferring disentangled representations learned from supervised synthetic or simulated data to an unsupervised dataset (in terms of the FoVs). This approach allows evaluate the effectiveness of disentangled representations transfer, and its potential in real-world applications.
3.1 Our methodology and research questions
Most of the focus in learning disentangled representations has been on synthetic datasets whose ground truth factors exhibit perfect independence by design 44
53
[5 34 4. Instead, real-world
5

---

Table I: Summary of the datasets and their properties. in the # FoV refers to the possible presence of hidden factors.
Complete annotation
Dataset
Real
Occlusions
#FoV
Independence
Resolution
#Images
dSprites
X
64 x 64
737K
Noisy-dSprites
X
X
64 x 64
737K
Color-dSprites
X
X
64 x 64
4,4M
Noisy-Color-dhe Color-dSprites
X
64 x 64
4,4M
Stapes3D
/
64
480K
Isaac3D
/
128
737K
Coil100-Augmente
/
V
/
128 x 128
1.1M
RGB-D Objects
3 X X
256 x 256
35K
scenarios present several challenges that we want to investigate in our analysis.
We consider 3-VAE models with weakly supervised learning specifically we adopted Ada-GVAE 41. for its simplicity and its sampling strategy similar to our metric, on a Source Dataset, using pairs of images that differ in factors of variation. We set l 1 as it shown to lead to higher disentanglement [41] Following 11, we vary the parameter ,3 in (1,2), sufficient to achieve high disentanglement with weak supervision 41]
We evaluate the quality of the disentanglement in a transfer learning scenario, assessing the trans- ferability of the disentangled representation on Target Dataset, with the final aim of targeting real scenarios, The evaluation we report considers our metric OMES, as well as DCI and MIG, the most widely used metries in the literature [50] 18 17 14
25 45]. Moreover, in accordance to 440 41 I.I. we evaluate the quality of the disentanglement also in terms of accuracy w.r.t. downstream classification task, with a classifier per FoV. We evaluate the latter in two modalities: (1) Considering the entire representation and (2) Selecting with OMES the dimension of the representa- tion that best encodes the FoVs. Our analysis addresses three main research questions: Q1 How well does disentanglement transfer, and how much does it depend on the distance between Source and Target Dataset?
69
We will consider different transfer learning scenarios (syn2syn, syn2real, real2real) and pairs (Source, Target) datasets with different distances.
Q2 Which properties of a DR are preserved the Target Dataset?
We will discuss Explicimess of the representation (through FoV classification), Compactness (analysing the component MES of OMES, the MIG metric, well as the performances of the one-dimensional representation), Modularity (relying on the OS component of OMES and on DCT). Q3 How effective fine-tuming the disentanglement? We will consider the performances the FoVs classification, the compactness and the modularity on the Target dataset before and after fine-tuning.
3.2 Datasets
In our analysis, we consider both synthetic and real datasets offering different challenges, a summary of their properties is in Tab. [T] Some of the datasets are DRL-compliant meaning that there is full independence between the FoVs (this is reported in column Indepencence), and FoVs appear in all their possible combinations. This is easy to achieve if the dataset is specifically tailored for DRL, but can not be easily obtained general.
Shape, Scale, Rotation, and y Positions. Variants of the dataset have been proposed: in Noisy-dSprites the background is filled with uniform noise; Color-dSprites includes Color as an additional FoV; adds uniform noise to the latter. We refer to them as: N-dSprites, C-dSprites and N-C-dSprites.
dSprites 44 is a dataset of 2D shapes generated from 5 ground truth FoVs:
Doisy-Color-dSprie
Shapes3D 4
is a dataset of 3D shapes, generated from 6 ground truth FoVs: Floor colour, Wall colour, Object colour, Scale, Shape and Orientation. It characterized by the presence of Occlusions.
Isaac3D 47 is synthetic dataset of 3D scene of a kitchen where robot arm is holding objects variety configurations. characterized by real-world complex FoVs, including robot movements, camera height, environmental conditions (e.g. lighting).

---

There are few real datasets available specifically meant for DRL. (20 is collection of datasets covering the transitions from simulated to real data, which is, however, not fully available at the moment. [1] is not appropriate for our analysis since the real data section is very small compared to the complexity of the task. We consider instead real benchmarks proposed for classification tasks, chosen to reflect some of the real-world challenges but possessing some "semantic connection" with the synthetic dataset we refer to, e.g. in terms of the expected Vs This allows us to reason on the potential of transferability. Example images are in Appendix C.I
Coil is derived from Coil100 [46.. The original dataset contains 7200 real color images of 100 objects. The objects were placed on motorized turntable against black background. The turntable was rotated vary object pose w.r.t. a fixed camera. producing self-occlusions and 2D silhouette changes. We augment the original dataset with two additional FoV, a planar rotation (9 angles) and a scaling (18 values). Therefore, we identify 4 FoV (Objects, Pose, Rotation and Scale) that, by construction, are independent. To consider in our analysis a real dataset visually related to dSprite, we derived a binary version of Coil, called Coil(bin), by applying Otsu's thresholding [49
RGBD-Objec (33 is a dataset of 300 common houschold objects acquired by a RGB-D camera.
The objects are organized into 51 Categories and a varying number instances for each category. For each object, 3 video sequences have been acquired with different camera heights (Elevation so that the object is viewed from different angles while rotating (Pose). Then, images have been cropped so that the object is always in central position. For our experiments, instance per category to make it semantically similar to Coil100 but with the additional complexity of variability in the background, presence of occlusions and clutter. Hence, control 3 Fo (Category, Elevation, Pose), but other factors are hidden or not annotated (e.g. Background, Illumination, etc.) due a realistic acquisition protocol. We refer to RGBD-- as RGBD. We also use a variant of the dataset, including depth maps only, referred to as RGBD(depth).
used subset with one object
3.3 Experimental analysis
Implementation details. We trained 20 different models (10 random seeds values of (3 for each Source dataset. We adopted the same training strategy as in [ (see Appendix C.2 As for Fs classification, following [TT 41], we consider Gradient Boosted Trees (GBT) [6] and Multilayer Perceptron (MLP) (37] with 2 hidden layers of size 256. Since the specific choice of a classifier is not crucial for our analysis, here we report GBT, MLP can be found in Appendix C.7 Fine-tuning to the Target dataset of the VAE models is unsupervised and it is carried out for 50k steps.
Tables description. The tables group different experiments based on the Target dataset. For each FoV, we report under the name the number of values the factor can assume (i.e. its granularity). The tables report the average classification performance over the 20 models, before and after fine-tuning. The latter is reported parenthesis in terms of gain or loss w.r.t. the performance before the fine-tuning. All is the average performance of all FoVs.
The column Pruned highlights the two different representation modalities: if the classifier is trained on the whole representation (X), or using only one dimension, i.c. the one showing the strongest encoding of certain FoV according to the OMES metric (V). As already mentioned, a good performance of the former is an indication of explicitness, while the latter is a positive sign of compactmess. Tables also report metrics assessing Modularity (our MES and DCI) and Compactness (our OS and MIG).
Note that we exploit the interpretability of OMES in the transfer learning process to select the most representative dimension of the representation for the classification (the "Pruned" columns).
(1) Synthetic synthetic. As baseline, we consider the case in which both Source and Target datasets are synthetic and we have access to the annotation of the Fo they are DRL-compliant. If Source and Target have the same FoVs (S=dSprites with T=NoisyddSpries or S=Color-dSprites with Moloiss-Coloo-dSpritee see Table we observe that pruning the representation to just one dimension maintains, on average, stable performances. This shows that the compactesss of the representation is preserved for the Target dataset, both before and after fine-tuning.
Fine-tuning allows for improved performance in terms of explicitness preserving the remaining properties of the representation, also in the case of the pruned representation. The Orientation FoV is difficult in these datasets as it suffers from reconstruction errors. We increase complexity by adding new FoV to the Target dataset (S=dSprite with T=Color-dSprite, see Table 2)
7

---

Table 2: Quantitative evaluation of transferred disentangled models using the dSprites family of
datasets. We transfer from a Source (ST) to a Target Dataset (TD). We report the average classification accuracy obtained with GBT on the full and the pruned representations (see text). The last columns on the right report comparison between disentanglement metrics, including MES and OS.
Fo\s(S))
Modularity
Compactnesat
SD
Pruned
Culor
Shape
Scale
Orientation
PoxX
(32)
PosY
Our
DCT
Our
MIG
6.7
17.8
17.1
30,0
dSprites
(+27.9)
(+26.1)
(+17.3)
31.8
22.0
32.1
13,9
52.1
44.7
3.8
14.3
14.3
25.8
(+6.3)
(+4.0)
(+3.9)
(-6.9)
dSgries
(46.6)
(48.3)
(43.5)
(422.5)
(421.9)
(412.6)
30,8
94.2
86.9
44.8
76.3
75.7
68.1
C-dSpites
(+15.4)
(-1.)
(+0.5)
(-4.2)
(-511
(-5.3)
(+3.2)
26.2
74.9
74.6
(+1.6)
(410.4)
(43.5)
(40.3)
(+6.8)
(+1.0)
(4.1)
(4.3)
3330
930
9,20
C-Sppites N-C-dSprites
(+65.2)
(+7.3)
(+16.)
(+0.9)
(+19.7)
(+20.)
(+21.7)
28.9
5.4
29.0
188
29.1
38.3
25.3
2.6
6.0
533
17.9
(+0.3)
(+2.2)
(-1.)
(-1.0)
(+52.3)
(+4.5)
(+11.2
(60.4)
(67.1)
(67.4)
(+13.8)
Table Target dataset: Shapes3D (see Table
Moon
Modularity %6)
Compactnens's %)
SD
TD
Pruned
Floor Hue (10)
Wall Hoc (30)
Objecct Huc (10)
Scale (8)
Shape (4)
Oricntation
(15)
All
Our (O5)
DCT
Our (MES)
MIG
78.0
$0.3
41.6
25.4
55.5
35.3
53.0
dSgriics
(+14.1)
(+13.1)
(425.6)
(+2.5)
(+18.6)
(40.8)
(+13.6)
(40.6)
26,9
(49.0)
9.6
(+13)
25,0
(40.4)
5,3
C-HSprites
Shupes3D
60.6
30.2
(+10.4)
25.3
44.0
29.4
40.3
(+0.8
(+74)
(+2.0)
(-1.4)
(3.88
(-3.2)
(+12.1)
(e4.0)
(+18.)
(0.5)
(+5.3)
02.3
02.5
29.8
10.3
31.9
21.8
28.9
baa:3D
(+26.2)
44.5
(+28.2)
42.5
(+57.9)
26.1
(+10.3)
22.3
(+35.2)
41.6
(+28.
13.8
(+27.))
31.8
(46.1)
24.3
(416.7)
3.6
(+8.3)
21.2
(65.2)
1.5
(+8.3)
(+14.4)
(+16.5)
(66.6)
(+20.3)
(+14.5)
(+13.4)
All FoVs in common between Source and Target are effectively classified, again except Orientation As for the new FoV (Color), we report lower performances, but we can appreciate a significant improvement with fine-tuning if we exploit global representation. Instead, observe lower improvement with the pruned representation, suggesting that the new factor is not encoded one single dimension.
To further increase the distance between Source and Target, we consider pairs for which the semantics of the FoVs are the same, but they are different in appearance, granularity, and composition (S=Color-dSprite with T=Shapes3D, see T)ble we can observe that even without fine-tuning, the latent representation allows the classification of the dominant FoV's of the dataset, i.e. Floor Hue and Wall Hue, also when focusing on a single dimension. Fine-tuning positively affects the average classification accuracy, especially when using the whole representation.
We finally reason on the gap between Source and Target datasets in terms of complexity. When the Source is simpler than the Target but still they have some FoVs in common, possibly with different appearances, (e.g. S=Shapes3D, T=Isaac3D, Table Jable/aand Table I0) we can appreciate the effectiveness of transfer and fine-tuning for all metrics. Conversely, when the Source is much more complex than the Target (e.g. when S=Isaac3D, T=Shapes3D) one could expect the richness in the Source to be directly transferrable to the simpler Target. However, we observe that the finetuning is still beneficial for all the disentanglement metrics. This be explained by the "domain" dependence of VAE models.
Discussion. Disentanglement transfers well between synthetic datasets with the same FoVs, w.r.t. all the propertics. If the Target includes new FoVs, fine-tuning is necessary for the new FoV, but also for the entire representation, FoV. When the Source and Target become significantly different, fine-tuning is also beneficial. We conclude that when both Source and Target synthetic and DRL-compliant, the properties of disentangled representation are preserved before and after tine-tuning, especially when the datasets have Fo in common even though they have different appearance.
compactness and modularity partially degraded by the new
(2) Synthetic to Real. We now analyse the potential of transferring a disentangled representation from an appropriately generated Synthetic Source (DRL-compliant) to Real Target. We first consider Real
8

---

Table 4: Target dataset: Coil100 and variants (see Table 2).
Mean uccuracy un (oVs(
(%d
Compactnese ()
SD
TD
Pruned
Object
(100)
Ples
(72)
Orientation
(18)
Seale (9)
All
(OS)
DCI
(MES)
Our
MIG
dSprites
(bin)
Coil
(+3.9)
4,62002
1.5)0.1)
L.7(0.)
48.1 (+1.7)
(80
333
38.677.4
(+.2
(4.))77)
57.8 (+2.)
(1277
25.7(+2.11
(.1.8)
(4.0()
(-0.)
346(-1322
386147.6)
345(+1044)
C-SSprites
(+1(+)8)
27.2
(+2.)
(3.4++0.1)
L.600.)
(1,(
(322
(4.60.3)
StapedD
Coll
4
(661(9243)
(3
123(0124)
(1.2
3.((335)
(5.00))0)
(.7())))
(22644))
(.39
9.72 (44.)
(3.5
10.21 (43.)
Coiltinn
209(41266
(6200
(04(4128))
22:3(4885)
(6.
10.4 (-5.)
26.1 (-24)
(.6(-3.5
(.3
1.5(0)))
(-1.8
(-04)
(2..0)
Table 5: Target dataset: RGBD-Objects and variants (see Table
Mean accuracy FoVa(%)
Modularity
Compactnesst %+
TD
Pruned
Category (51)
Elevatation (4)
(263)
Pose
All
Our (0S)
DCI
Our (MES)
MIG
64.3 (14.3)
(3.0
0.3 (+0.0)
49.2 (+2.4)
USprites
34.1 (-4.3)
(-1.5
(1000
33.9 (-1.9)
34.3 (+0.)
(+0.5)
(-0.2)
3.4 (+0.5)
3tupssD
(depth)
RGBD
(+1544)
(+11.7)
(14.()
(+7.8)
(1000
(+000
(45.5)
(e7.7)
(40.8)
(2.0
(-0.))
2,0 (40.8)
17.3 (15.5)
0.3 (+6.7)
(5.41)
Coil(bin)
(-..()
(+0.3)
(-2.0)
(-2.4)
86,4 (13.2)
(-1.4)
(.000
22.1 (60.6)
C-dSprites
(3.7()
(40.0)
(0,1(=13))
35.7 (40.))
7.8 (-2.3)
(-1.5)
1.8 (-0-4)
36.7 (+1.6)
(-0.8)
0.2 (+0.1)
(405))
83.3 (96
(433))
0.3 (40.1)
48.3(63.3)
RGBD
35.0(+
(10.6)
(+0.3)
2.0 (-1.)
35.7 (40. 1)
(-0.7)
0.3 (+0.0)
2-.8)-0.2)
Coil(bin)
19.3 (P9.3)
(43.6)
0.3 (400)
46.309500)
(40.5)
5.1 (40.))
24.2
(-0.2)
36,3 (40.4)
48.6 (-0.0)
0.2 (+0.0)
(4.1))
Targets with FoVs independence. In Table we first analyse (S=Color-dSprite with T=Coil): the Target dataset shares some FoVs with the Source (such as Scale, inplane Orientation, and Object, the latter related to shape), but their variability and granularity may be very different (e.g. the shape/object can assume 3 values on Source and 100 on Target) ). Other FoV are new, such as Pose (encoding 3D rotations). The accuracy we achieve is very uneven on the different Fo in particular Pose, since the Source does not incorporate any 3D information. We notice improvement with fine-tuning but also a degradation in performances with the pruned representation (a sign the representation does not produce good disentanglement on the Target). An exception this last comment is the FoV Scale, whose accuracy does not degrade significantly with one dimension only (an indication this FoV is well represented in one dimension).
With the same Target, we assess a representation which is incorporating some level of 3D information ( S=Shapes3D, with T=Coil, Table 4. This choice does not bring any benefit since the synthetic Shapes3D includes a very simplified form of pose variation. Orientation accuracy degrades, as it not captured by the Source (indced. with fine tuning this performance improves). Considering the Target dataset, it clearly presents several new challenges w.r.t. Source ones, we produce a last experiment with simplified binary version of the Target, meant to be more similar to the binary images in dSprite. In this case, some Fo are very well represented (Orientatin on and Scale), while Object presents low performances due the decrease in the image descriptive power caused by binarization.
We now consider another real Target, RGBD-Object (see Table As for the former, we consider Color-dSprite as a synthetic source: here, the same considerations about the Pose we discussed before are valid, and on the other FoVs, we observe the global representation is effective (and marginally improved by fine-tuning). while the pruned representation leads to lower performances, as a sign the representation is not perfectly disentangled on the Target. Here again, we investigate the possibility of transferring to simplified versions of the dataset (in this case, we consider the Depth channel only). The FoV Elevation improves significantly as it is not directly affected by color information. Concerning Modularity and Table 5), fine-tuning seems to have small influence. In particular, we observe a small degradation for RGBD-Objects (Table 5). coherent with the challenges of real dataset with unknown/hidden factors perturbing the encoding of the FoV.
Discussion, If the Source is synthetic (and DRL-compliant), and the Target is Real, the quality of transferring seems to depend on the distance between datasets and is not even across different FoVs: Fo that are more similar between the datasets are more easily represented, and they exhibit better compactness properties. Fine-tuning is bringing significant benefit in Explicitness and some

---

notice that (limited) benefit in Modularity and Compactness. If the Target incorporates unknown hidden factors, as we may expect to happen the real world, Modularity and Compactness transfer worse, and the benefit of fine-tuning is limited.
(3) Real to Real. We conclude by discussing the possibility of transferring from DRL-compliant real dataset to another real one. As a first task, we consider as a Source a simplified version of the Target (specifically, S=Coil-binary, with T=Coil): the source should encode the factors not related RGB, while the finetuning should improve the disentanglement and the explicitness of the representation. However, this is not the case with Coil100, whose representations degrade the Modularity, and the finetuning only affects the entire representation.
We then consider larger variation between Real Source and Real Target (specifically. S=Coil with BRRGBD--bjeee see T5: we obtain similar results to those of Color-dSprites as Source Dataset (comparable Explicitness); with a reduction on the performances obtained by the pruned representation. Notice that adopting the binary Coil Source causes only a limited reduction in Explicitness, and this was somewhat unexpected as we have a large gap in complexity between Source and Target. Our experiments did not consider RGBD-Objects acting Source dataset, not being DRL-compliant.
Discussion. Using real DRL-compliant dataset as a Source, we do not appreciate any benefit. Fine- tuning not particularly effective. At the same time, we notice that some level of disentanglement transfer can be observed.
4 Limitations
limitation of our current work is the adoption of a specific family of approaches (VAE-based). The generalization of our finding to more recent vector- based approaches (e.g. [71 62 48 36 needs further investigation. However, each family of approaches for disentanglement learning follows specific paradigms that may require tailored designs for transfer learning. In other words, while the general transfer methodology is still applicable, it might need proper tuning to perform optimally depending on the particular learning approach.
5 Conclusions
In this paper, we discussed the potential of transferring a Disentangled Representation as strategy to address disentanglement in real data. We learned the representation from a Source Dataset in a weakly supervised manner and transfered it to Target Dataset, where supervision the FoVs was difficult or impossible to obtain. We identified three main scientific questions, summarised in Section 3.2 which recall draw conclusions our study. Starting from question Q2, the properties of disentangled representations that are preserved after transferring, we may conclude Explicitness is usually well maintained, while Modularity and Compactness are reduced we move from synthetic to real. More precisely, we appreciate a degradation in the global metrics (such as OS and ME), while on the compactness through the analysis of the 1-dimensional pruned representations, some FoV may transfer very well.
As for Q3, we may observe that fine-tuning is almost always beneficial, and never causes any harm. Q1, a much wider question discussing under what circumstances transfer is effective, leads us to conclude that some structural similarity between Source and Target datasets is necessary, including similar aanges/granularig of variations of related factors. A quantification of the similarity among datasets is still under investigation; the results of our study suggest one could design synthetic data to capture/disentangl/ specific factors of interest.
Future directions. Currently, we are exploring quantitative methods to assess the distance between Source and Target datasets. In the near future will target more specific applications, such as biomedical image classification or action recognition from videos, to discuss and relate the general results we are reporting in this paper to more specific and challenging domains.
10

---

Acknowledgement
We acknowledge the financial support from PNRR MUR Project PE0000013 "Future Artifi- cial Intelligence Research (FAIR)", funded by the European Union NextGenerationEU, CUP 133C24000430007
References
[1] Bengio, Y., Courville, A., Vincent, P.: Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence 35(8),
(2013)
[2] Bouchacourt, D., Tomioka, R., Nowozin, S.: Multi-level variational autoencoder: Learning dis- entangled representations from grouped observations. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 32 (2018)
[3] Bowman, S.R., Vilnis, I.., Vinyals, O., Dai, A.M., Jozefowicz, R., Bengio, S.: Generating sentences from a continuous space. arXiv preprint arXiv: (2015)
[4] Burgess, C., Kim, H.: 3d shapes dataset. http://gshhbbcommmeepmm////aaggg
(2018)
[5] Burgess, C.P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., Lerchner, A.: Understanding disentangling 3-vae. arXiv preprint 104.03399 (2018)
[6] Cao, J., Nai, R., Yang, Q., Huang, J., Gao, Y:: An empirical study on disentanglement of negative-free contrastive learning. Advances in Neural Information Processing Systems 35, 1210-1222 (2022)
[7] Carbonneau, M.A., Zaidi, J., Boilard, J., Gagnon, G.: Mcasuring disentanglement: A review of
metries. IEEE transactions on neural networks and learning systems (2022)
[8] Chen, R.T., Li, X., Grosse, R.B., Duvenaud, D.K.: Isolating sources of disentanglement in
variational autoencoders. Advances in neural information processing systems 31 (2018)
[9] Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I,, Abbeel, P:: Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems 29 (2016)
[10] Dang, H., Huu, T.T., Nguyen. T.M., Ho, N.: Beyond vanilla variational autoencoders: Detecting posterior collapse in conditional and hierarchical variational autoencoders. In: The Twelfth International Conference on Learning Representations (2024), https: /ttps://oppnrrevie net/ forum/id-4zZFG1i0
[11] Dittadi, A., Träuble, F., Locatello, F., Wüthrich, M., Agrawal, V., Winther, O., Bauer, S., Schölkopf, B.: On the transfer of disentangled representations in realistic settings. arXiv preprint 11ti:20000114000 (2020)
[12] Do, K., Tran, T.: Theory and evaluation metries for learning disentangled representations. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net (2020), httpp:///ppenrviee...oo
4d=HJgK0h4Ywr
[13] Eastwood, C., Williams, C.K.: A framework for the quantitative evaluation of disentangled representations. In: International conference on learning representations (2018)
[14] Eddahmani, I., Pham, C.H., Napoléon, T., Badoc, I., Fouefack, J.R., El-Bouz, M.: Unsupervised learning of disentangled representation via auto-encoding: A survey. Sensors 23(4), 2362 (2023)
[15] Fidler, S., Dickinson, S., Urtasun, R.: 3d object detection and viewpoint estimation with deformable 3d cuboid model. Advances in neural information processing systems 25 (2012)
[16] Friedman, J.H.: Greedy function approximation: gradient boosting machine. Annals of statistics pp. 1189-1232 (2001)

---

[17] Fumero, M., Cosmo, L., Melzi, S., Rodola, E.: Learning disentangled representations via product manifold projection. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th In- ternational Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 139, pp. 3530-3540. PMLR (18-24 Jul 2021), https://proceedingg wtap:/iiinnnnm....eeeeeeeee .ress v139/ (.339/umero2la.cc html
[18] Fumero, M., Wenzel, F., Zancato, L., Achille, A., Rodolà, E., Soatto, S., Schölkopf, B., Locatello, F:: Leveraging sparse and shared feature activations for disentangled representation learning. Advances in Neural Information Processing Systems 36 (2024)
[19] Gabbay, A., Hoshen, Y.: Scaling-up disentanglement for image translation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 6783-6792 (2021)
[20] Gondal, M.W. Wuthrich, M., Miladinovic, D., Locatello, F., Breidt, M., Volchkov, V., Akpo, J., Bachem, O., Schölkopf, B., Bauer, S.: On the transfer of inductive bias from simulation the real world: a new disentanglement dataset. Advances in Neural Information Processing Systems 32 (2019)
[21] Gonzalez-Garcia. A., Van De Weijer, J., Bengio, Y:: Image-to-image translation for cross- domain disentanglement. Advances in neural information processing systems 31 (2018)
[22] Goodfellow, I., Lee, H., Le, Q., Saxe, A., Ng, A.: Measuring invariances in deep networks. Advances in neural information processing systems 22 (2009)
[23] Higgins, I,, Amos, D., Pfau, D., Racaniere, S., Matthey, L., Rezende, D., Lerehner, A.: Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230 (2018)
[24] Higgins, I., Matthey, L., Pal, A., Burgess, C.P., Glorot, X., Botvinick, M.M., Mohamed, S., Lerchner, A.: beta-vae: Learning basic visual concepts with a constrained variational framework. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net (2017), https //openreview net net//forum/iddsyy2tt0ggg
[25] Hsu, K., Dorrell, w., Whittington, J., Wu, J., Finn, C.: Disentanglement via latent quantization. Advances in Neural Information Processing Systems 36 (2024)
[26] Kahana, J., Hoshen, Y.: contrastive objective for learning disentangled representations. In: European Conference on Computer Vision. pp. 579-595. Springer (2022)
[27] Kim, H., Mnih, A.: Disentangling by factorising. In: International Conference on Machine Learning. pp. 2649-2658. PMLR (2018)
[28] Kim, M., Wang. Y., Sahu, P., Pavlovic, V.: Relevance factor vae: Learning and identifying disentangled factors. arXiv preprint arXiv:1902.01568 (2019)
[29] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
[30] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint 1312.11312.6114 (2013)
[31] Kulkarni, T.D., Whitney, W.F., Kohli, P., Tenenbaum, J.: Deep convolutional inverse graphics network. Advances in neural information processing systems 28 (2015)
[32] Kumar, A., Sattigeri, P., Balakrishnan, A.: Variational inference of disentangled latent concepts from unlabeled observations. arXiv preprint arXiv: 11111111110000 1.08488 (2017)
[33] Lai, K., Bo, L., Ren, X., Fox, D.: A large-scale hierarchical multi-view rgb-d object dataset. In: 2011 IEEE international conference on robotics and automation. pp. 18 1817-1824, IEEE (2011)
[34] LeCun, Y., Huang, F.J., Bottou, L.: Learning methods for generic object recognition with invariance to pose and lighting. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. vol. 2, pp. II-104 IEFE (2004)
12

---

[35] Lee, S., Cho, S., Im, S.: Dranet: Disentangling representation and adaptation networks for unsu- pervised cross-domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15252-15261 (2021)
[36] Lin, Z., Thekumparampil, K., Fanti, G., Oh, S.: Infogan-cr and modelcentrality: Self Self-supervised model training and selection for disentangling gans. In: international conference on machine learning. Pp. 6127 6139. PMLR (2020)
[37] Lippmann, R.: Book review: Neural networks, a comprehensive foundation, by simon haykin. Int. Neural Syst. 5(4), 363-364 (1994). htsss///ooogg/010112222202767700 https / //doi..og/10. /144//00990657940000000000
[38]
Liu, Y., Sangineto, E.,, Chen, Y., Bao, I.., Zhang, H., Sebe, N., Lepri, B., Wang. W., De Nadai, M.: Smoothing the disentangled latent style space for unsupervised image-to-image translation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10785-10794 (2021)
[39] Locatello, F., Abbati, G., Rainforth, T., Bauer, S., Schölkopf, B., Bachem, O.: On the fairness of disentangled representations. Advances neural information processing systems 32 (2019)
[40] Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., Bachem, O.: Chal- lenging common assumptions in the unsupervised learning of disentangled representations. In: international conference on machine learning. pp. 41 11144122, PMLR (2019)
[41] Locatello, F., Poole, B., Rätsch, G., Schölkopf, B., Bachem, O., Tschannen, M.: Weakly- supervised disentanglement without compromises. In: International Conference Machine Learning. pp. 6348 6359. PMLR (2020)
[42] Lu, Z., Wu, C., Chen, X., Wang, Y., Bai, L.. Qiao, Y., Liu, X.: Hierarchical diffusion au- toencoders and disentangled image manipulation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 5374-5383 (2024)
[43] Ma, L., Sun, Q.. Georgoulis, S., Van Gool, L., Schiele, B., Fritz, M.: Disentangled person image generation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 99-108 (2018)
[44] Matthey, L., Higgins, I., Hassabis, D., Lerchner, A.: dsprites: Disentanglement testing sprites dataset. wttpg/hwww.oooe/eeeemmm//rrsssss
(2017)
[45] Mo, S., Sun, Z., Li, C.: Representation disentanglement in generative models with contrastive learning. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 1531-1540 (2023)
[46] Nayar, Murase, H.: Columbia object image library: Coil-100. Tech. Rep. CUCS-006-96. Department of Computer Science, Columbia University (February 1996)
[47] Nie, W.: High resolution disentanglement datasets. hutp::/giiab.ccom///bbbb//hhhhhh disentanglemennt-daaassss (2019)
[48] Nie, W., Karras, T., Garg, A., Debnath, S., Patney, A., Patel, A.B., Anandkumar, A.: Semi- supervised stylegan for disentanglement learning. In: Proceedings of the 37th International Conference on Machine Learning. pp. 7360-7369 (2020)
[49] Otsu, N.: A threshold selection method from gray-level histograms. IEFE transactions on systems, man, and cybernetics 9(1), 62-66 (1979)
[50] Pandey, A.,
Fanuel, M., Schreurs, J., Suykens, J.A.: Disentangled representation learning and generation with manifold optimization. Neural Computation 34(10), 2009-2034 (2022)
[51] Paninski, L.: Estimation of entropy and mutual information. Neural computation 15(6), 1191- 1253 (2003)
[52] Pham, C.H., Ladjal, S., Newson, A.: Pca-ae: Principal component analysis autoencoder for organising the latent space of generative networks. Journal of Mathematical Imaging and Vision 64(5), 569-585 (2022)
13

---

[53] Reed, S., Sohn, K., Zhang, Y., Lee, H.: Learning to disentangle factors of variation with manifold interaction. In: International conference on machine learning. pp. 1431-1439. PMLR (2014)
[54] Ridgeway, K: A survey of inductive biases for factorial representation-leamingg CoRR abs/1612.05299 (2016),
05299
[55] Ridgeway, K., Mozer, M.C.: Learning deep disentangled embeddings with the f-statistic loss. Advances in neural information processing systems 31 (2018)
[56] Sarhan, M.H., Navab, N., Eslami, A., Albarqouni, S.: Fairness by learning orthogonal disentan- gled representations. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16. pp. 746-761. Springer (2020)
[57] Schmidhuber, J.: Learning factorial codes by predictability minimization. Neural computation 4(6), 863- 879 (1992)
[58] Schölkopf. B., Locatello, F., Bauer. S., Ke, N.R., Kalchbrenner, N., Goyal, A., Bengio, Y.: Toward causal representation learning. Proceedings of the IEEE 109(5), 612-634 (2021)
[59] Shi, Y., Yang, X., Wan, Y., Shen, X.: Semanticstylegan: Learning compositional generative priors for controllable image synthesis and editing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1125--12264 (2022)
[60] Shu, R., Chen, Y., Kumar, A., Ermon. S., Poole, B.: Weakly supervised disentanglement with guarantees. arXiv preprint
(2019)
[61] Sønderby, C.K., Raiko, T., Maaløe, L., Sønderby, S.K., Winther, O.: Ladder variational autoencoders. Advances neural information processing systems 29 (2016)
[62] Song, Y., Keller, A., Sebe, N., Welling, M.: Flow factorized representation learning. Advances
in Neural Information Processing Systems 36 (2024)
[63] Steenbrugge, X., Leroux, S., Verbelen, T., Dhoedt, B.: Improving generalization for abstract
reasoning tasks using disentangled feature representations. arXiv preprint 1.XXv:1811.0444 (2018)
[64] Suter, R., Miladinovic, D., Schölkopf, B., Bauer, S.: Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In: International Conference Machine Learning. pp. 6056-6065. PMLR (2019)
[65] Träuble, F., Creager, E., Kilbertus, N., Locatello, F., Dittadi, A., Goyal, A., Schölkopf, B., Bauer, S.: On disentangled representations learned from correlated data, In: International Conference on Machine Learning. pp. 10401-10412. PMLR (2021)
[66] Van Steenkiste, S., Locatello, F,, Schmidhuber, J., Bachem, O.: Are disentangled representations helpful for abstract visual reasoning? Advances in neural information processing systems 32 (2019)
[67] Wang, X., Chen, H., Tang, S., Wu, Z.,, Zhu, W.: Disentangled representation learning (2023)
[68] Wang, Y., Blei, D., Cunningham, J.P.: Posterior collapse and latent variable non-identifiability.
Advances in Neural Information Processing Systems 34, 5443 5455 (2021)
[69] Watters, N., Matthey, L., Burgess, C.P., Lerchner, A.: Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint McXiv:1901.0700) (2019)
[70] Xiang. S., Gu, Y., Xiang, P., Chai, M., Li, H., Zhao, Y., He, M.: Disunknown: Distilling unknown factors for disentanglement learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14810-14819 (October 2021)
[71] Yang, T., Wang, Y., Lu, Y., Zheng, N.: Disdiff: Unsupervised disentanglement diffusion probabilistic models. Advances in Neural Information Processing Systems 36 (2024)
14

---

[72] Zhu, J.Y., Zhang, Z., Zhang, C., Wu, J., Torralba, A., Tenenbaum, J., Freeman, B.: Visual object networks: Image generation with disentangled 3d representations. Advances in neural information processing systems 31 (2018)
[73] Zhu, X., Xu, C., Tao, D.: Where and what? examining interpretable disentangled representations. In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5861-5870 (2021)
15

---

Table 6: Summary of different metrics for disentanglement learning. L and K are the numbers of latent variables and ground truth factors, respectively.
Intervention based
Information based
Predictor based
Measured Property
Metric
Classifier
#Classifiers
BetaVAE
x
Linear/majority-vote
Modularity
Fac
X
Linear/majonty-von
Modularity
SAP
X
Threshold value
X K
Compactness
MIG
None
Compactness
LASSO
Modularity
DCI
X
K
Compactness
Random forest
Explicimesss
Modularity
X
X
None
0
Modularity
Modularity
OMES(Our)
None
Compactness
1.0
0.9
OMES
0.8
0.7
0.6
0.5
(1)
(II)
(III)
(IV)
(V)
(VI)
(VII)
Figure 3: Boxplot of the distribution OMES comparing association matrices S, for different a
values: (I), (II) and (III) are the results of simulated scenarios where only Overlap (I) and Multiple Encoding (II) or both (II) are represented, (IV). (V), (VT) are obtained with reak-superviiion (respec- tively, on datasets Shapes3D,, Color-dSprites, Noisy-dSprites; (VII) Noisy-dSprite with unsupervised model.
A Evaluating the quality of disentanglement
Table reports the main characteristics of the well-ostablished and most used disentanglement metrics. Note that OMES is the only one both Interventional-basee and information-basedd measuring Modularity and Compactness.
B OMES assessment
In this section, we report the evaluation of the 1800 models trained on Noisy-dSprites, 1800 models trained on SmallNORB and 1800 models trained on Cars3D, all from [4]. Here we report the extensions of the results in Section 2.3
B.1 OMES interpretation
Fig. shows our metric OMES scores for different values of 3 keeping the different FoV separated, for Noisy-dSprites (Left), SmalINORB (Center) and Cars3D (Right). is fixed to 0.5.
In addition, Figure shows the range of values with different a for a selection S. We include synthetic cases ((I, (III), (IV)) producing high scores, and (V) generated from Shapes3D, is very similar. The scores of the Noisy-dSprites models ((VI), (VII) are lower, as it is more challenging dataset. Color-dSprites (V) is easier to disentangle and output values in between Shapes3D and Noisy-dSprites. Note how the boxplots generated from real models produce smaller range of values w.r.t the simulated cases; the choice of does not appear critical in the real case.
16

---

B.2 Agreement of OMES with other disentanglement metrics
Fig- Ishows the metric scores for different values of 3 keeping the different FoV with a 0.5, the benchmark datasets (Noisy-dsprine (Left), SmalINORB (Center), Cars3D (Right).. In general the greater the 3 the higher the disentanglement but the factors strictly related to reconstruction quality fail to be encoded in the representation.
Fig. [ shows the distribution of the disentanglement metrics, extending the plot from 40 with metric OMES computed with different values of a E {0.0, 0.3, 0.5, 0.8, 1.0). We observe the higher the a the less variable the distributions of our metrics, meaning that the models are more similar in terms of Multiple Encoding than they are in terms of Overlap.
Fig. shows the rank correlations of the disentanglement metrics of the models trained on Noisy- dSprites, extending the plot from 40) with our metric OMES computed with different values of E {0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0). We observe the higher the (Multiple Encoding) the higher the correlations with BetaVAE Score and FactorVAE Score and negative correlations with Modularity.
Analogously, Fig- shows the rank correlations of the disentanglement metrics of the models trained on SmallNORB, extending the plot from [40] with our metric OMES computed with different values of a E {0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0].
Finally, Fig. g shows the rank correlations of the disentanglement metrics of the models trained on Cars3D, extending the plot from 40 with our metric OMES computed with different values of E {0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}.
B.3 Agreeement with performance metrics
Fig. shows Rank correlation with ELBO, reconstruction loss, and test error of Fo classifier for the models trained on the weak-supervised setting that was shown to be more interesting for this analysis. We consider all the different Source datasets used in the transfer experiments.
Pactr
CRESS
Figure 4: Scores of the proposed metric for each FoV is fixed to 0.5) of the 5400 models 40 1800 models trained on Noisy-dsprites (Left); 1800 models trained on SmalINORB (Center); 1800 models trained on Cars3D (Right).
Figure 5: Distribution of different metrics the 5400 models in [40]: 1800 models trained on Noisy- dsprites (Left); 1800 models trained SmalINORB (Center): 1800 models trained on Cars3D (Right). This is an extension of the plots in 40), we added our metrics with different values of (0.0, 0.3, 0.5, 0.8, 1.0].
17

---

Q.O
0.1
0.2
Beta/AE Score
100
BD
45
47
37
BataVAE Score
100
90
45
41
47
37
BatalAE Scoe
103
45
47
37
FactriNE
Score
M4G
46
49
100
76
7
43
4B
MN8
46
49
100
76
43
19
MGG
45
49
100
76
7
43
19
Disentangemeet
76
Dicentriglmmot
100
Diserannloment
76
-8
Mockarity
47
10C
28
Nodi Earty
26
100
Moduarity
47
100
127
37
39
36
100
37
43
OMEB
100
O145
25
72
100
ONES
18
100
Besidt Soow
MB
Noddkr
S00
Facar Booe
Moodant
OME3
DesiKE Dooe
Facavilr Soos
M4
DC: Drannanggeer
Montartt
EAP
QMES
Beakii
100
101
90
Facorinn Score
80
100
49
28
39
26
FadorWWE Seore
100
S2
39
27
FactorkKE Soore
80
100
48
25
39
28
45
100
43
49
49
76
43
45
100
76
43
DO: Diseizangommnt
76
72
DCI Disenlongeemont
100
DC: Diserrangloment
Modksrryy
47
26
100
14
.26
47
25
100
14
.25
47
26
100
-44
OME8
20
100
CMEB
10
27
73
20
100
OWESS
11
28
50
75
21
100
Facto beadkl Soon
Moddinty
SAP
FaArrNE Soons
MB
Mocdont
CMES
BeoWWE Soone
Facaodd Soow
MWG
Dowminngarer
Mocuarty
RAP
OMBS
BeatkE
Soore
100
FactoVAE Soore
80
100
4/
26
30
28
FactovVRE Score
80
100
82
26
39
20
FactorVWE Soore
80
100
48
62
26
30
30
100
45
100
DC: Deentargaement
41
76
100
30
73
DCI Dhsentorghemret
100
38
DC: Deserrangooment
41
76
10U
8
73
100
-22
47
100
BAP
8AP
100
8AP
OMES
12
51
21
100
OMES
13
20
78
100
OWES
14
30
52
-0
100
DAAVAE Store
MB2
SAU
FassrNNE Soore
MG
BCt Danda mendeneeeee
ONES
RANNWE Store
MIS
DCL Dantargkeee
SAP
OVES
Fassiin
DCt
0.9
1.0
BouuVEE Score
100
45
16
BoLAWE Score
100
80
37
17
FacteriVE
80
100
4G
26
39
31
FactoriAE Score
80
100
82
39
MG
100
MIG
DGI Deertarglemet
41
52
76
100
36
73
CCI D'aentanghemmnt
52
76
100
30
72
Noculanty
Modularty
8AP
37
38
4.3
14
100
22
SAP
37
30
36
100
23
100
DENAE Stae
190
SAV
FaseiNE Soore
103
DO Dowhhrgdeee
Moduistty
SXP
CVES
DO:
Figure 6: Rank correlation of different metrics the same dataset (Noisy-dSprites) computed on the 1800 models in 440.. This is an extension of the plots in [40], we added our metrics with different values of a E {0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0).
18

---

0.0
0.1
0.2
Beta/AE Score
100
3G
I60
39
+0
BataVAE Score
100
53
3G
4b
E00
39
BataAAE Scor
100
S6
45
-80
39
Score
M4G
66
100
79
30
76
28
MB8
56
34
100
70
-50
76
26
MIG
56
100
79
-50
76
26
Disentangemeet
79
Dicentriglmmot
100
Diserannlomert
Mockarity
88
:50
10C
Nodi Earty
600
.50
100
28
Moduarit
400
800
500
100
39
76
86
100
39
76
20
OMEB
100
O1ES
31
28
100
ONES
-1
28
100
Beaid Soor
MB
Noodke
CAEES
Facar Bome
Moodan
OMBB
Deaiki Sooe
Facavilr Sooe
MG
DC: Drantangimer
Nontarty
EAP
Beakii
100
100
63
Facoriin Score
63
100
34
409
-31
FadorWWE Seore
100
38
-31
FactorkKE Soore
63
100
34
39
-09
-31
56
100
-50
78
25
58
70
70
25
96
100
79
-0
76
DO: Diseizargommet
79
DCI: Disenlongeemont
100
DC: Diserranglement
45
Modkkstty
60
.68
.50
42
100
28
.60
.68
.50
42
100
28
-60
.69
.00
100
28
OMES
25
52
100
OMEB
-31
100
OWESS
-1
-31
26
100
beatkl Soon
MB
Moddinty
SAP
FaArrNE Soons
OMBS
BeoWWE Soove
Facaoid Sooee
MWG
Dowminggaeer
Moculrty
RAP
FactriME
BeatkE
56
Soore
FactoVAE Soore
63
100
34
400
-31
FactovVEE Score
83
100
30
460
-31
FactorVWE Soore
68
100
34
30
-60
100
50
DC: Desntarglmment
45
79
100
28
DCI Disentrrghmmeet
45
39
100
12
58
27
DC: Deverrangooment
45
78
T0U
42
600
-09
-50
100
-80
-99
-2
-60
-50
100
BAP
30
76
8AР
100
8AP
30
OMES
24
51
100
OMES
-2
-31
24
27
50
100
OWES
.2
28
27
100
DAAVAE Store
MG
DCt reeeenggeeen
SAU
FassrNNE Soore
MG
BCt Daert dedeeee
ONES
RANNWE Store
Dettebll Scom
MIS
DCL Dantargkeee
SAP
OVES
FassiinE
0.9
1.0
BouuVEE Score
100
-60
-2
BoLAWE Score
100
83
45
39
FacteriVE
63
100
34
400
-30
FactoriAE Score
100
39
-20
MG
100
MIG
DCI Deentarglemet
45
39
79
100
50
27
CCI D'aentanghemmet
45
39
79
100
42
50
27
Noculanty
Modularty
BAP
38
76
100
SAP
38
86
-2
100
47
DENAA Stae
IMG
DO. Diororggomnt
Modiiall
SAP
FasarNE Soore
103
DO Dowthrgeeee
Moduistt
SXP
CVES
Figure 7: Rank correlation of different metrics the same dataset (SmalINORB) computed on the 1800 models in 44).. This is an extension of the plots in [40], we added our metrics with different values of a E {0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0).
19

---

0.0
0.1
0.2
Beta/AE Score
10C
37
22
29
13
34
BetaVAEE Score
100
37
22
38
29
13
35
BetalAE Scoe
100
37
22
3B
29
13
26
FactriNE
FedorWAE Soore
100
Boord
MGG
22
66
100
67
S2
43
62
MN3
22
86
100
67
52
43
62
MIG
22
66
100
67
52
43
62
DC: Disentangommet
Dicentanglmmon
100
Diserannloment
Mockarity
29
38
52
10C
61
Nidi Eerty
29
39
57
100
18
02
Moduarity
100
622
13
16
25
40
100
25
13
27
15
OMES
34
100
OMES
51
76
100
ONES
100
Basilt Soore
MB
Noodke
S00
CAIES
Facar Bome
DC Deeetnnggeeee
Moodanty
OMB3
DeasiK Sooe
Facavilr Soose
MG
OCL Dranninggmer
Nontartt
EAP
OMES
Beakin
BetalAE
100
22
101
Berat/
22
Facoriin Score
37
100
66
39
40
51
FadorWWEE Seore
100
60
39
40
51
FactorkKE Soore
37
100
88
60
39
40
61
100
52
43
82
88
43
62
22
100
52
43
62
DC: Disoiiangoomen
36
67
57
80
DCI: Dicenlongeemen
100
DC: Diserrangloeeett
80
Modkkrtty
29
3B
32
100
18
62
Modi
29
39
52
57
100
18
63
Medulaiity
29
39
62
57
100
18
63
100
CMB8
36
82
80
24
100
OMEB
37
51
80
63
100
OWESS
37
51
62
80
63
24
100
FactriME beatkl Soon
MB
Moddinty
SAP
BeatkE Soons
FaArMEE Soons
Mocdont
CMBS
BeoWWE Soore
Eacaoie Soowe
MWG
Dowminngamer
Mocutrty
RAP
0.8
22
Soore
BealKKE
FactoVAE Soore
37
100
66
30
40
51
FactovVEE Score
37
100
60
36
40
S1
FactorWKE Boore
37
100
66
80
59
40
51
100
100
DC: Desntargommen
38
67
100
57
27
81
DCI Disentorghemee
30
80
67
100
57
61
DC: Desersangooment
38
67
10U
57
27
81
Wockdanty
29
52
100
63
39
100
18
63
29
100
63
BAP
BAP
100
8AP
13
OMES
37
82
81
63
23
100
OMES
38
51
61
100
OWES
38
51
100
DAAVAE Store
MGC
DCI Deenenggoner
SAP
FassiiNE Soore
MG
ONES
RANNWE Store
MIS
DCL Bantargkeeet
SAP
OVES
BCI Derdtnddeeee
0.9
4
1.0
BouuVEE Score
100
29
BoLAWE Score
100
38
13
FactoriVE
37
100
66
30
40
51
FactoriAE Score
100
60
40
51
MG
100
MIG
96
61
DCI Deentarglemet
30
60
67
100
57
27
31
LCI D'mentanghemmett
30
90
67
100
57
27
61
Noculanty
100
Modulariy
8AP
13
45
16
100
23
SAP
1.3
40
100
100
DENAEE Stae
18G
DO: Deororggomnt
SAV
FusriNE Soure
103
DO Dewhhrgdeee
Modsistty
SXP
CVES
FacariNE
Figure& 8: Rank correlation of different metrics the same dataset (Cars3D) computed on the 1800 models in [40] This is an extension of the plots in [40), we added our metrics with different values of a E {0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0. 0.8, 0.9, 1.0].
20

---

dSprites
Color-dSprites
ELBO
18 -21 36 -39
-49
ELBO
45
25
37
28
41
GBT10000
43
24
61 69
37
GBT10000
60
8
40
46
40
MLP10000
61
-2
85
37
25
MLP10000
75
14
40
26
29
Recons. loss
21
18
-22
35
46
Recons, loss
38-27
11--2744--1-33-
BetaVAE
FactorVAES:
OMES
DCI
MIG
BetaVAE
FactorVAE
OMES
DCI
MIG
Coil-100
Coil-100-Binary
ELBO
16 -55
20
11 -2
ELBO
56
52
46
61
58
GBT10000
60 78
80
85 67
GBT10000
63
57
61
84
84
MLP10000
15 20
34 53 60
MLP10000
47
55
59
71
79
Recons, loss
-20 13 -23
-11 2
Recons, loss
-33
-61
-27
-77-33-29
BetaVAE
FactorVAEs,
OMES
DCI
MIG
BetaVAE
FactorVAES.
OMES
DCI
MIG
Shape3D
ELBO
60
51
54
92
13
GBT10000
55
63
76
84
23
MLP10000
64
29
19
42
50
Recons. loss
-46-19-34
-79
-6
BetaVAE
FactorVAES,
OMES
DCI
MIG
Figure 9: Rank correlations (Spearman) of ELBO, reconstruction loss, and the accuracy of GBT and a MIP classifier trained on 10,000 labelled data points with disentanglement metrics. In all plots OMES is computed with 0.5.
21

---

Table 7: Datasets info and examples. Specifically, the Variant column shows the corresponding
variants of the original dataset
Dataset FoV
Original
Variant
Color-dSprites
FoV
values
Color
7
Shape
Scale
Orientation
PosX
PosY
32
Sh
values
Floor Huc
Wall Hue
10
Object
Scale
8
Shape
Orientation
15
Codl Ooggmmtteee
FoV
# values
Object
100
Pose
72
Orientation
Scale
RGBD Objects
FoV
# values
Category
Elevation
4
Pose
263
C Transfer experiment
Here provide additional information about the architecture of the models for the transfer experi- ments. Moreover, we include the tables reporting the average performances of the GBT and MLP classifiers.
C.1 Datasets
Table 7 reports the main information about the used dataset, e.g. FoV and number of classes, together with some examples of the original dataset, plus some samples of one variant of the given dataset, such Color-dSprites and Noisy- cusy-Color-dSpriess The only exception is Shapes3D which does not have any variant, so different samples drawn from the same dataset are shown.
C.2 Architecture
Sshows the architecture of the model used for all the transfer experiments. We trained multiple
models with 10 different random seeds each value 3. Hence, we obtain 20 models for each
Source dataset. We adopted the Adam optimizer [29] with default parameters, batch size=64 and over the first 50k training steps. We maintained the latent dimension fixed to 10 for all the experiments.
400k steps. We used linear deterministic warm-up [1] 61 3
22

---

Table 8: Encoder and Decoder architecture for the transfer experiments.
Encoder
Decoder
Input: 64 64 #channels
Input: R.
4 x 4 conv, 32 LeakyRelu(0.022) stride 2
FC 8192 LeakyRelu(O.02)
4 X 4 conv, 64 LcakyRelu(O.02),, stride 2
FC 8 x 8 x 128
4 x 4 conv, 128 Lea)y stride 2
4 x 4 upconw, 64 LeakyRelu(00022) stride 2
Flatten
4 x 4 upconv, 32 LeakyRelu(0.02)) stride 2
2 x FC 10
x 4 upconw, #channels Sigmoid, stride
Coloe dSprites Noisy. dSprites
dSprites
Shapes3D
Shapa33 Aollm00-Aummetedd
Shapes3D RGBD Objets
Figure 10: Some reconstructions generated from the fine-tuned models of different Source (SD)
Target (TD) couples.
Figure 11: (Left) Reconstruction of samples of Coil100 of fine-tuned model trained originally on Color-dSprites, same as Fig. 10 (Right) Reconstruction samples of 1oi model trained from scratch on it.
C.3
Transfer reconstruction
Some reconstructions, generated by the VAE models after they are fine-tuned, are depicted in Fig. The quality of the reconstruction good even the encoding is obtained by training the model first completely different source dataset and then fine-tuning the model for few iterations.
10
Fig. I1 (Right) shows the reconstruction of the same samples of Coil Augmmetedd from the model trained from scratch on it. Comparing the latter with the reconstructions of Fig. Left) it can be observed that the quality comparable (with some exceptions), and so with the fine-tuned models, are not losing much information from the data.
23

---

Table 9: Transfer from Shapes33 (Souree) to Isaac3D (Target), Average classification accuracy over the 20 models of the GBT classifier, before and after fine-tuning (see Table 2
Maan accurscy S)
Pruned
Object shupe (3)
Objeect scale (4)
Camera beight (4)
X moovement
(8)
Y Y-movenenn
(5)
Light intensity
(4)
Light y-direction
(6)
Object color
(4)
Wall color (4)
All
140
$4,0
99.2
3:.9
25.0
$3.5
85.4
29.5
78.1
$1.3
(+5.1)
(+54.6)
(+17.4)
(-29.6)
(+4.3)
(+14.0)
(+12.7)
(+13.0)
(+18.)
(+16.7)
35.8
33.5
26.7
21.6
69.4
67.5
22.0
61.5
42.1
(+3.6)
(+24.1)
(+11.)
(-153)
(+2.)
(+17.3
(+14,4)
(+7.3)
(+16.0)
(+12.)
Table 10: Transfer from S (Source) to Isaac3D (Target). The Compactness and Modularity scores of the same models of Table
Modularity' %)
Compactness($ %)
Our (OS)
Our (MES)
Pruned
DCI
MIG
X
25.1
6,3
21.2
2.2 (+5.9)
(+9.7)
(+16.1)
(+10.2)
C.4
Transfer protocol
Each GBT and MLP classifiers are trained the latent representation extracted from the Encoder of the Ada-GVAB models so that the train split comprises 10000 samples and the test split is of 5000 samples.
With unsupervised fine-tuning on the Target dataset, means that the model is trained as a simple
VAE 30
C.5 GBT & MLP performance distribution
In this section, we report the performance distribution of the GBT classifiers on the target FoV (see Fig. 12.. the figures on the left depict the performances on the representation before fine-tuning and on the right are depicted the results after the fine-tuning of the representation. Fig. 13 shows the performance distribution of the MLP classifiers on the target FoV, the figures on the left depict the performances on the representation before fine-tuning and on the right depict the results after the fine-tuning of the representation.
C.6 Transfer from Shapes3D to Isaac3D
In this section, report the results of the transfer from Shapes3D to Isaac3D, Table 9 for the Explicitness and Table for Modularity and Compactness. We observe that the transfer is overall effective according to all the disentanglement scores even though the Target dataset is much more complex than the Source. If we compare the results of the transfer from Shapes3D other real datasets such as Coil100 (Table 4 and RGBD-Objects (Table 5). we notice that in the case of Isaac3D the boost of finetuning is more noticeable.
This suggests that, if source and target have common Vs with similar appearance reasonable performances on real Target dataset even if the source synthetic dataset is much simpler.
obtain
C.7 MI.Ps performances
In this section, we report the performances of the MLP classifiers on the FoVs, for the sake of comparison we report the scores of the disentanglement metrics on the representation input.
We associate the tables regarding the same Target dataset: Table IT corresponds to the same representa-
tions of Table 2 in the main document; Table 12 to Table Table to Table/4and Table 14 to Table
Overall we can observe higher performances obtained with MLP, especially on classifiers trained on the entire representation. This happens because MLP can easily disentangle an entangled represen- tation by observing different dimensions a time while GBTs partition the input space into regions aligned with the axes, making it harder to observe multiple dimensions at time and so the Fo
24

---

SD: Color dSprites TD: Noisy dSpiid
SD: ColondSSprits TD: Noisy Caloy-Colondppriiee
1.0
pruned
pruned
Downtream peformanoo
0.8
Tree
Downstren task performanes
0.8
True
04
0.4
0.2
0.2
3.0
Color
Shape
Scale
Orientation
PosX
PosY
Color
Shape
Scaln
Orientation
PoxX
PosY
fov
SD: Color- dSprites
TD: Shapes3D
SD: Color dSprites
TD: Shapes3D
1.0
pruned
pruned
peffomanoe
0.8
Thue
Downtream task perforannes
0.8
True
0.4
0.4
0.2
0.2
3.0
Floor buo
Wall Hue
Object Hue
Scale
Shape
Orientation
Floce hoc
Wall Hue
Object Hul
Scale
Shape
Crientation
fow
SD: Coko-dSSrites-
TD: Coi110- Augmented
SD: Colar-dSprites
TD: CoiI:00AAumentted
1.0
1.0
pruned
pruned
Downstream peformances
2UR
True
Downstream task performance
0.8
Truc
0.4
0.4
0.2
0.2
3.0
Object
Pose
Orientation
Scale
Object
Pose
Oriercation
Scale
fov
SD: Color-dSprites
TD: RGBD Objects
SD: Colar-dSpites
TD: RGBD Objects
1.0
1,0
pruned
pruned
2U8
True
0.5
True
Doundtrsam bokk
Downstream
0.4
0.4
3.2
0.2
0.0
0.0
Casegory
Azimuth
Pose
Category
Azimuth
Pose
lov
for
Figure 12: Some examples of the performance distribution of the GBT classifiers before (Left) and after (Right) fine-tuning of different Source (SD) Target (TD) couples.
25

---

SD: Color-dSprites TD: Noisy-Color-eee Color-dSprites
SD: Color-dSprites TD: Noisy-Colon Colon dSprites
Pune
praved
08
16
Cate
Shape
Scale
not
Scle
Oree Liber
Poct
May
SD: Color-oSprin dSprites TD: 3D
SD: dSpritdSprites
TD: Shapes3D
Punec
0.5
Dicamres cal arterspoo
144
Fale
Plie
Wiul
Orgict Hay
Scale
Shane
Orantaiin
Rlat
Vall
Okgint
Scale
Shape
Denitriee
SD: Cola--1Sprites TD; Coll10-AAggean
SD: Colar-dSprites
TD: Coil:00-Augmented
1.8
1.0
pruned
68
Downstream task performance
0.8
Truc
06
0.4
0.2
0/
Objict
Pase
Orenziioo
Scla
Object
Pose
fov
Oriencation
Scale
SD: Coke-eSprites- TD: RGBD Objects
SD: Color-dSprite ies
TD: RGBD Objects
1.8
1G
09
14
Dowatiee paforvec
066
Cregrr
Acrihh
Post
Carger
Aders
Ploe
Figure 13: Some examples of the performance distribution of the MLP classifiers before (Left) and after (Right) fine-tuning of different Source (SD) Target (TD) couples.
26

---

Table 11: Target dataset: dSprites and variant. The averaged performances of the MLP classifiers and disentanglement metrics, together with the average improvement with the finetuning (in brackets).
Mean Accuracy Fo)a(()
Modularityy (%)
Compactnesal
SD
Pruned
Color
Shape
Scale
(6)
Orientation
(40)
PoxX (32)
PosY (32)
All
(OS)
DCI
Our (MES)
MIG
64.3
5.7
20.1
19.4
31.3
N-dSppriss
(+45.9)
(453.7)
(454.0)
(454.0
(-44.2)
31.8
22.0
32,
13,9
25.8
(+6.3)
(+4.0)
(45.0)
(+6.)
dSprices
(+14.3)
(+22.3)
(e5.8)
(+27.7)
(+26.7)
(-20,0)
35.3
95.3
98.3
67.5
33.2
855
78.5
C-llsrites
(445.0)
(40.2)
(42.1
(45.9)
(-1.)
(-2.)
(-8.1)
19.3
7522
21.1
73.8
73.4
36.7
(-10.4)
(+13.1)
(+6.4)
(+3,))
(1.8)
(1.6)
(-1.)
3.1
10.3
11.1
21.1
C-pSprsss
diprites
(642.1)
(+35.2)
28.9
5,4
20.0
29.9
39.4
25.8
2.7
6.8
6.0
18.4
(+0.3)
(+2.2)
(119)
(+1.0)
(+55.4)
(45.3)
(+12.9)
(40.7)
(+1244)
(412.6)
(-16.6)
Table 12: Target dataset: Shapes3D. The averaged performances the MLP classifiers and disentan- glement metrics, together with the average improvement with the finetuning (in brackets).
Mean Accuracy on Fe
Modularity %)
Compactness %)
SD
ID
Pruned
Flour Hine (10)
Wall Huue (10)
Object Tlne
(10)
Scale (8)
Shupe (4)
Orientation (15)
All
Our (0S)
DCI
Our (MES)
MIG
34.7
86.8
5733
53.4
67.5
51.1
63.4
dSpcites
(+15.3)
(+13.2)
(-40.3
(+55.8)
(+27.9)
(+44.)
(+52.9)
26.9
23.9
53
64.1
61.3
31.9
23.1
45.2
27.0
42.1
(+0.6)
(v9.0)
(+1.)
(0.4)
(+14.6)
(+18.4)
(+33.))
(+16,7)
(-2344
(+13.4)
(+19.9)
Shapes3D
88.1
87.4
623
19.3
61.7
3600
66:3
C-35price
(+11.9)
(+12.))
(437.5)
(456.6)
(431.6)
(642.3
(432.11
(+0.)
30.6
(97.4)
13.7
(+2.))
28,2
(-1.4
84
This aligns with what observed in 41 11. Given this, is preferable to observe the performances of the GBT because they can give clearer about the disentanglement properties of the representation.
C.8 GBT & MLP performances standard deviation
In this section, we report the standard deviation of performances of the GBT and MLP classifiers on the FoVs, for the sake of comparison we also report the standard deviation of scores of the disentanglement metrics on the representation input.
MLP classifiers: Table 19 corresponds to the same representations of Table 11 in the Appendix;
Table 20 to Table 12 Table 21 to Table 13 and Table to Table 14
GBT classifiers: Table 13 corresponds to the same representations of Table 2 in the Appendix; Table 16 T Table [7 to Table[[and Table 18 Table
Note that the metrics scores in the MLPs and GBT's tables are the same because we refer to the same representations. In general, we can observe the models without fine-tuning have small standard deviation, while fine-tuned models perform with higher variation.
Table 13: Target dataset: Coill00-augmented and variants. The averaged performances of the MLP classifiers and disentanglement metries, together with the average improvement with the finetuning (in brackets).
Accuracy
Modularity
Compactncse
SD
Prumed
(100)
Oricatation
Scale
(MES)
Our
MIG
USprites
Coil (bin)
(4227)
(4055)
(-18.)
44.64221.6)
37.8(+2.)
(1.4
25.7 (+2.) (+1.8
(-266
1..))
19.51:5.0
(613)
162(-11D)
C-1Sprites
(63(4411)
19(408)
(C2(2))))
436(4254)
(4.7(2221))
33.9
(+.4
27.20
2-86-00.)
15(+02)
(2.9422.)
312(-22)
166((404)
Shapes3D
Coll
(133)3383)
1.71100
31.7002
3.2(43.5)
25.0
(403)
1.((4.3)
24.14433)
(1.35+33455
255(-02)
(-188)
(4316)
(4.0
Coilibin)
363(322)
(-.3)
26.1 2-214
5.6
12.5(+5.))
(..)
(4.7
(
14.3 (42.1)
27

---

Table 14: Target dataset: RGBD-Objects and variants. The averaged performances of the MLP classifiers and disentanglement metries, together with the average improvement with the finetuning (in brackets).
Mean Accuracy on Fo\a(
Medularity
Compactness %)
TD
Pruned
Category (51)
Elevatation (4)
(263)
Pose
All
Our (08)
DCI
Our (MES)
MIG
(+9.3)
(0.3))
(.3.)
54.8 (+4.6)
dSprites
34.3 (+0.6)
(1.0
22.1 (-0.2)
34 (40.5)
(-2.9)
(-0.)))))
35.1 (,30)
Stapes3D
RGBD (depth)
(.3.)
35.3 (-0.8)
(2.0
22,4 (-0.5)
(10.8)
28.4
66.7 (16.4)
31.8 (.3
(+6.7)
(+0.0)
Coilibin)
(-.6(-
(+0.3)
23.5 (-2.0)
(-244)
(x3.8)
(-2.3)
(.300
22.7 (e0.5)
C-USprites
94.3 (43.2)
83.9 (46.2)
0.3 (-0.1)
(43.1)
(-0.7)
(.8
24.7 (-1.5)
1.8 (-0.)
Coil
RGBD
(5.61927)
80 (927)111)
(1.2(=170) (4000
35.0
4.2 (+0.6)
23.5 (-0.3)
(-.
Coil(bin)
93.7 (44.1)
(43.)
(2000
(0.2(220)
35.3 (-0.5
(.0.2
241.
1,3 (-0.)
(2.5(+1.1)
(+0.3)
31.1
Table 15: Target dataset:
dSprites and variant. The standard deviation of the performances of the GBT classifiers and disentanglement metries, together with the standard deviation of the performances with the finetuning (in brackets).
Stundurd deviation Accuracy FoVs(S)
Modularity
Compactnessi
Shape
TD
Pruned
(7)
(3)
(6)
(40)
(32)
(32)
(OS)
DCI
(MES)
MIG
0,02
0.02
0.01
0.01
0.02
N-dSprites
(8.29)
(6.00)
(5.55)
(6.65)
(7.73)
0.01
0.01
0.01
0.01
0.06
0.02
0.01
0.01
0.01
(3.43)
(7.23)
(2.99)
(6.29)
dSprites
(8.55)
(6:98)
(4.67)
(12.33)
(1349)
0.01
C-dSprites
(9.44)
(1.31)
(1.89)
(3.02)
(1.52)
0.01
0.01
0.01
0.03
0.05
0.05
0.01
0.01
(0.67)
(1.93)
C C-dSprites
N C-d-piitses
x
(0.78)
0003
(1.53)
0.01
(1.53)
(0.33)
(1.72)
0.01
(1.41)
(2.09)
(2.99)
(2:44)
(2.00)
(7.33)
(1.18)
(3.99)
(0.31)
(2.60)
(2:15)
D Experiments Compute Resources
All the experiments have been executed with NVIDIA Quadro RTX 6000. On average, the training and evaluation of a single Source model take hours. Each fine-tuning and final evaluation takes 1.5 hours. Overall, the whole bunch of transfer experiments and our metric assessment take approximately 100 hours.
Table 16: Target dataset: Shapes3D. The standard deviation of the performances of the GBT classifiers and disentanglement metries, together with the standard deviation of the performances with the finetuning (in brackets).
Standard deviation, Accuracy on FoYs(
Modularity
Compactaessi %%
SD
TD
Pruned
Floor Hue (10)
Wall Hue
(10)
Object Hue
(10)
Scale
(8)
Shape (4)
Orientation (15)
Our (OS)
DC1
Our (MES)
MIG
0.05
0.05
0.05
0.05
0.06
(2.77)
dSprites
0.06
0.06
0.04
0.00
0.04
0.04
(5.00)
(4.93)
(4.52)
(2.13)
Shapes3D
(10.43)
0.19
(1.44)
0.07
(6.40)
0.06
(5.57)
0.01
(11.17)
1114
(6.71
0104
C-dSprites
(10.63)
0.02
0.08
0.07
0.06
0.03
0.04
0.04
(4.57)
(3,82)
(8.13)
(11.1)
(9.17)
(5:53)
(10.63)
(7.35)
28

---

Table 17: Target dataset: Coil Coil -00-aumeeete and variants. The standard deviation of the performances of the GBT classifiers and disentanglement metrics, together with the standard deviation of the performances with the finetuning (in brackets).
Standard deviation Accuracy on Fo)s(())
Modularity %6)
Compactnessi %%)
Pruned
Object (100)
Puse
(72)
Orientation
(18)
Scale
(9)
Our (08)
DCI
(MES)
Our
MIG
dSprites
Coil (bin)
x
0.01 (0.72)
0.00 (0:21)
0.01 (0.69)
0.01 (0.58
0.00((067)
(0.51
(0:48)
(0.51)
0.003 (0.51)
0.02 (0.54)
0.01 (1.15)
0.05 (1,85)
0.05 (1)4)
0.01 (1.56)
C-dSprites
0.00((048)
(0.51)
(0:48)
(0.51)
0.03 (3.56)
0.00 (0.15)
0.01 (0.81)
0.01 (1.87)
(3.123
0.01 0.000018)
0.01 (1.30)
Shupes3D
Coil
0.01 (1.35)
0.01 (1.41)
0.01 (1.07) 0.01 (1.19)
0.01 (3.94)
(0.1))
0.01 (2.02)
0.01 (2.98)
(2.242
(.00(0.14)
(1.45
0.02 (1.52)
Coil(hin)
0.01 (1.62)
0.02 (1.52)
0.01 (1.17) 0.01 (1.03)
0.01 (3.94) (0.18)
(2.57)
0.02 (3.51)
Table 18: Target dataset: RGBD- and variants. The standard deviation of the performances of the GBT classifiers and disentanglemen metrics, together with the standard deviation of the performances with the finetuning (in brackets).
Standurd deviation Accuracy FoVs(%)
Mudularity(%)
Compactnesal %)
Category (51)
Elevatation
Pose
Our (OS)
Our (MES)
SD
TD
Pruned
(4)
(263)
DCI
MIG
0.01 (1.79)
0.02 (0.66)
0.00 (0.11)
dSprites
0.01 (0.47) 0.02 (4.03)
0.01 (0.54) 0.01 (2.47)
0.05 (5.58)
Shapes33
RGBD (depth)
0.02 (0.96)
0.02 (0.59)
0.00 (0.11)
0.01 (0.54)
0.02 (4.03)
0.01 (0.54) 0.01 (2:47)
0.05 (6.07)
0.04 (5.60)
0.00 (0.07)
0.27 (32
(113(1
0:00 (0.10
Coil(hin)
0.01 (0.53)
0.05 (5.73)
0.01 (0.57) 0.02 (1.31)
/
0,14 (18.0))
0.11 (1 (11.46)
0.00 (0.08)
C-dSprites
0.01 (0.75)
(0.99)
(0.11)
0.00 (0.39)
0.02 (2.39)
0.01 (0.77) 0.01 (0.85)
0.09 (12.79)
0.02 (3.64)
0.00
0.01 (0.77)
0,01 (1.02)
0.00 (0.11)
Coil
RGBD
0.00 (0.36)
0.01 (2.92)
0.01 (0.67)
0.01 (0.62)
0.04 (11.12)
0,02 (2.88)
0.00 (0.11)
0.02 (0.70)
0.01 (0.83)
0.00 (0.11)
Coil(hin)
0.00 (0.42)
(2.43)
0.01 (0.90) 0.00 (0.64)
0.10 (12.38)
0,03 (3.76)
0.00 (0.08)
E
Future directions
We briefly summarise the future direction of our work.
On the data set side, we only considered the synthetic datasets applicable a wider number of tasks, other synthetic datasets exist but either they are limited in the number of FoV or very specific to a task. We limited our analysis two real datasets, plus their simplified variants, tackle a broad but limited number of challenges. We will extend our analysis to more complex real datasets with an increasing number of known and unknown factors, but this requires we also design more complex synthetic datasets able to tackle these complications. For the methods, we will explore the effect of the dimensions of the latent space and different kinds of supervision for training the Source model, as well as including (partial) supervision on the fine-tuning. On the applications side, we will analyse
Table 19: Target dataset: dSprites and variant. The standard deviation of the performances of the MLP classifiers and disentanglement metries, together with the standard deviation of the performances with the finetuning (in brackets).
STD Accuracy on Fo)
Modularity 96)
Compactneso (%)
SD
TD
Prumed
Color
Shape
(3)
Scale
(6)
Orientation
(40)
PocX
(32)
PasY (32)
(OS)
Our
DCI
Our (MES)
MIG
0.03
0.02
N-dSprites
(1.36)
(3r11)
(7.36)
(4.13)
(5:05)
0.01
0.01
0.01
0.01
0.06
0.03
0.01
0.02
0.01
(3.43)
(7.23)
(2.99)
(6.29)
dSprites
(10.38)
(12.7)
(7.66)
(13.12)
(14.77)
008
0.00
0.01
0106
0.05
0.03
C-dSprites
(5.14)
(0.27)
(0.60)
(3.49)
(5.27)
(4:01)
0.01
0.06
0.06
0.02
0.02
(0.67)
(1.93)
(1.02)
(2.99)
(21.72)
(3.62
(3.71)
(7.10)
(1.54)
(1.54)
074
0.01
0.01
0000
0.01
0.02
C- C-ipprites
N-C- -Sprites
(0.08)
(2.34)
(1.84)
(0.82)
(1.53)
(1.18)
0.01
0.01
0.01
0.01
0.00
0.01
0.01
0.00
0.02
0.02
(2.09)
(2.99)
(2.44)
(2.00)
(7.10)
(2.13)
(4.27)
(0.10)
(5.12)
(4.01)
29

---

Table 20: Target dataset: Shapes3D. The standard deviation of the performances of the MLP classifiers and disentanglement metries, together with the standard deviation of the performances with the finetuning (in brackets).
STD Accuracy on Fo
Modularity'ss (6)
Compactnesst (%
SD
TD
Prnned
Ploor Hue (10)
Wall Hue (10)
Object Hue
(10)
Scale
(8)
Shape (4)
Orientation
(15)
(OS)
Our
DC1
Our (MES)
MIG
0.06
0.04
0.07
0.02
0.04
0.06
dSprites
(0.03)
0.05
0.87
0.05
0.33
0.04
0.05
(4.93)
(4.52)
0.02
(7.53)
(9.73)
(12.26)
(12,59)
(13.96)
(14.91)
Shapes31)
0:315
0.34
0:06
0.06
0.04
8:05
C-USprites
(0.00)
(0.00)
(0.02)
(3.56)
(0.78)
(1.46)
0.02
0.09
0.08
0.08
0,02
0.05
0.04
(4.57)
(3.82)
(9.44)
(12.63)
(12.50)
(10.19)
(11.03)
(12.73)
Table 21: Target dataset: Coill -0-aummeeee and variants. The standard deviation of the performances of the MLP classifiers and disentanglement metrics, together with the standard deviation of the performances with the finetuning (in brackets).
STD Accuracy FoVe(S
Modularity %-)
Compactnessi %%
Object (100)
Pose
Orientation
Scale
Our
(72)
(18)
(9)
(OS)
(MES)
MIG
Coil
0.02 (0.90)
0.00 (0.25)
0.01 (0.67)
0.02 0.02 (1.21)
dSprites
(hin)
0.01 (0.69)
0.00 (0.17)
0.01 (0.70)
0.01 (0.94)
(0.67)
(0.51)
(0.51)
0.02
0.00 (0.25)
0.01 (0.67)
(1.21)))
C-dSprites
0.00((048)
(0.51)
(0:48)
0.00 (0.51)
0.01 (0:69)
(.017)
0.01 (0.70)
0.01 (0.94)
Shupes3D
Coil
(0:11
(00000)66)
(0.77)
0.02 (1.16)
0.01 (1.35)
0.01 (1.41)
0.01 (1.07)
0.01 (1.19)
0.01 (6.04)
(0.18)
0.01 (3.44)
0.01 (3.44)
(1.65)
(0.55)
(2.02)
Coiltbin)
0.01 (1.62)
0.02 (1.52)
26.1 (-2.4) 0.01 (1.03)
0.03 (6.86) (0.21)
(3.56)
the effect of transferring from different priors (e.g. video sequences, expert knowledge in biological data) and investigate if transferring a disentangled representation will help to increase the level of interpretability of the target representation.
Table 22: Target dataset: RGBD--Ohects and variants. The standard deviation of the performances of the MLP classifiers and disentanglement metries, together with the standard deviation of the performances with the finetuning (in brackets).
Aveuracy Fo
Modulurity(
Compactnesst %
SD
TD
Pruned
Categury (51)
Elevatation (4)
(263)
Pose
Our (0S)
DCI
Our (MES)
MIG
0.01 (1.07)
0.01 (0.89)
0.00 (0.07)
dSprites
0.01 (0.47)
(4.03)
(0.54)
(2.47)
0.07 (7.45)
0.05 (5.32)
0.00 (0.10)
0.01 (1.07)
0.01 (0.89)
0.00 (0.07)
Shapes3D
(depth)
0.07 (7.45)
0.05 (5.32)
0.00
0.01 (0.54)
0.02 (4.03)
0.01 (0.54)
(2.47)
0.35 (38.97)
0.17 (17.92)
0.00 (0.07)
Coil(bin)
0.01 (0.53)
0.05 (5.73)
0.01 (0.57) 0.02
0.16 (20.37)
0.12 (13.5)))
0.00 (0.09)
0.01 (0.38)
0.022
(0.06)
C-dSprites
0.00 (0.39)
0.02 (2.39)
0.01 (0.77)
(0.85)
0.08 (13.95)
0.03 (6.62)
0.00 (0.07)
0.01 (0.30)
0.01 (0.87)
0.00 (0.0)
Coil
RGBD
0.00 (0.36)
0.01 (2.92)
0.01 (0.67)
0.01 (0.62)
0.04 (12.19)
(5.64)
0.00 (0.08)
0.01 (0.32)
0.02 (0.75)
0:00 (0.09)
Coil(bin)
(0.42)
(2.43)
(0.90)
(0.64)
0.11 (13.39)
0.05 (6.53)
0.00 (0.09)
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claimed constributions is metric described Sec. 2 and disentangled representation trasferring methodology defined and experimentally assessed in Sec. 3.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Sec. 3.3 and in Sec. (Limitations), we discuss the potential and the limitations of the methodology.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
31

---

Answer:
[NA]
Justification:
We do not provide theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: In Section 3.3 and in Appendix C we provide all the information to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
32

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the code as supplementary material, while the datasets we used are all publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Sections 3.3 and Section 2 and in Appendix C we provide the details to understand the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: While we report in the main document the Tables of the averaged measure- ments, we report in Tables of the same structure the standard deviation of our transfer experiments in Appendix C.8
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
33

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix D we reporting approximation of the time execution and GPU hardware.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurrps. http://ps..ppppulic/ciiciceeeees /tt ctseeeeeeeeessccceeeeeeeeee
Answer: [Yes]
Justification: We read the NeurIPS Code of Ethics and the paper conforms with it (we do not involve human subjects or participants, we used already existing datasets respecting their terms of usage, etc.).
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is social impact.
Guidelines:
34

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is no risk of misuse because our work does not include generating tasks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The exploitation of already existing models have been properly cited with the original authors.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
35

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide the code with instructions about the usage, reproducibility of experiments and license.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The work does include experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The work does not include experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
36

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "dfDDAdVj7o", "reviewer": "Reviewer_GNcy", "summary": "This paper proposes a novel classifier-free metric for quantitatively measuring disentanglement and investigates transferring disentangled representations from a synthetic dataset with ground truth factors of variation (FoV) to a target real-world dataset. The authors introduce OMES, a novel intervention-based metric that evaluates the correlation between representations of two images sharing all FoVs. OMES computes the overlap between different FoVs (Overlap Score) and measures the encoding of each factor within single dimensions (Multiple Encoding Score). Using these metrics, the paper analyzes the properties of disentangled representations and source/target distributions to improve disentanglement transfer. The main contributions of the paper are:\n- Introduction of a novel classifier-free metric (OMES) for disentanglement evaluation, which reduces hyper-parameter sensitivity.\n- Extensive empirical study on transferring disentangled representations from source to target datasets, revealing the potential and properties of disentanglement transfer.", "questions": "- In transferring disentangled representation learning, how much of the improvement comes from transfer learning compared to training from scratch on the target dataset, in terms of disentanglement metrics? This comparison would provide a clearer understanding of the benefits and effectiveness of the transfer learning approach.\n- Can you provide more analysis on why OMES has relatively low correlation on FactorVAE score in Figure 2? \n- Most of the investigations are done on VAE-based models. Would it have the same implications with other recent disentangled representation learning approaches [1, 2, 3], which employ more powerful generative models? \n\n\n[1] Yang, et al. \"Disdiff: Unsupervised disentanglement of diffusion probabilistic models.\", in NeurIPS 23. \n\n[2] Lin et al., “Infogan-cr and modelcentrality: Self-supervised model training and selection for disentangling gans”, In ICML 20.\n\n[3] Ren et al., “Learning disentangled representation by exploiting pretrained generative models: A contrastive learning view”, in ICLR 21.", "limitations": "- Section 3.3 describes the limitation of transferring disentanglement learning.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is easy to follow. \n- A simple yet novel classifier-free metric removes the dependency on hyper-parameters and enables reasonable comparison between various configurations and benchmarks. Also, this metric maintains the reasonable assessment of disentanglement compared to the conventional metrics and provides an informative tool for analyzing the each factor of variations. \n- Thorough empirical analysis provides comprehensive understanding on a novel metric and disentanglement transfer learning.", "weaknesses": "- The implications on transferring disentanglement from synthetic datasets to complex real-world datasets are somewhat limited. Although the empirical study indicates that a smaller distance between the source and target datasets and shared FoVs between them are beneficial, these are expected properties of conventional transfer learning. The paper would be strengthened by discussing how to specifically select a proper source dataset for a given target dataset with unknown factors of variation. For example, it would be useful to define and measure the structural similarity between a target dataset and potential source datasets.\n- The presentation could be improved. At first glance, it feels like two independent topics (metrics and transfer learning) are being introduced, making it hard to understand why a novel metric is needed for transferring disentangled representations. It would help readers if the connections between these two components were more clearly explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their effort spent in providing valuable feedback on our work. In the response, we will follow the paragraph listing structure of the review.\n\n* > The implications on transferring disentanglement from synthetic datasets ... and potential source datasets. \n\n    Please refer to the common response.\n\n* > The presentation could be improved...were more clearly explained.\n\n     We agree with the reviewer that the paper is rather dense in some parts, and this does not help the understanding. We will revise the contributions of our work to better clarify the connections between them in the final version of the paper, should it be accepted.\n\n     The novel metric is not strictly required by the application to transfer learning, but rather by more general needs for disentanglement metrics, the first being interpretability. We also wanted our metric not to be based on a classifier and to be able to estimate different properties of disentangled representations (*i.e.* modularity, compactness) simultaneously.  \n\n     In this sense, the contributions of our work are indeed two, somehow independent: the new metric OMES and the transfer learning methodology. However, it is worth noticing that the interpretability of OMES allows us to directly exploit it in the transfer learning process, for instance, to select the most representative dimension of the representation for the classification experiments (we named “Pruned” in all our tables). \n\n* > In transferring disentangled representation learning, how much ... effectiveness of the transfer learning approach.\n\n     We thank the reviewer for the question that allows us to illustrate a different aspect of our approach. We start observing that training from scratch on the Target dataset is usually not possible in the application domain we have in mind, where the annotation of the latent factors is not available in most cases. This explains why we did not consider this scenario in our experimental analysis. Nevertheless, we performed a further evaluation to quantify the loss of transferring rather than training from scratch. \n\n     We used the source models trained directly on Shapes3D, Coil100 and Isaac3d (which we trained under the suggestion of Rev. R67v)  and we compared their performances with the models trained on different Sources and finetuned on the 3 datasets.  The results are reported in Table 4 of the attached pdf. It reports the average scores of the models trained from scratch (with weak supervision).  In brackets instead, we put the average differences between the scores obtained from transfer learning + finetuning, and the ones with training from scratch (*i.e.* a negative value means we are losing when applying the transfer methodology). \n\n     In general, we observe that the models trained from scratch perform almost always better than the transferred ones. This gap is expected because the models trained from scratch use the annotation of the dataset, while the finetuning is unsupervised. However, in this experiment, the gap is noticeable only when Shape3D is a Target, this could be explained by the fact that it has relatively simple transformations easier to disentangle when the annotation of factors is used for training. On the contrary, for datasets with real-world more complex (harder to learn and disentangle) transformations,the performances with finetuning is closer to the training from scratch. We also observe that, given the complexity, their overall performance is lower than the ones of Shape3D.\n\n\n* > Can you provide more analysis on why OMES ... FactorVAE score in Figure 2?\n\n    We first observe that even if they are all intervention-based metrics, the formulation of the input pairs of our metric and FactorVAE/BetaVAE are the opposite (we sample pairs of images where all factors are the same but one, while FactorVAE and BetaVAE fix one common factor while randomly sampling the values of the others). The correlation of OMES with FactorVAE is poor (28) but an even lower correlation can be observed with BetaVAE (11). The latter can be explained by observing that when computing OMES we consider contributions from (latent dimension, factor) associations accurately selected to maximize the correlation, while  BetaVAE keeps the information from the entire latent representations (despite some dimensions might be uninformative or even potentially harmful). Concerning FactorVAE instead, we notice that it essentially estimates the performance of a linear classifier on input-output pairs composed of the fixed latent factor (output) and the latent dimension which presumably mostly encodes the factor (input). This step is also a core for OMES, although implemented following a different strategy (*i.e.* we rely on high correlation rather than low variance to select the “best” encoding factor).\n\n    Among the properties that may explain the different behaviours of OMES and FactorVAE/BetaVAE in Fig. 2 there is also the fact they measure only the modularity of disentanglement, while our metric OMES also accounts for compactness since alpha=0.5 (this comparison is reported in Table 6 in the Appendix). (For more details on the role of alpha, we refer the reviewer to the answer to Rev. Fxij on the same topic).\n\n\n* > Most of the investigations are done on VAE-based models ... employ more powerful generative models?\n\n   Please refer to the common response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C5WMCsxUd8", "reviewer": "Reviewer_R67v", "summary": "This paper conducts an empirical investigation into the problem of transferring disentangled representations from synthetic data to real-world data (syn2syn, syn2real, real2real). They start with well-defined research questions and perform the investigation on the feasibility and effectiveness of transfer learning step by step. Besides the investigation, this paper also proposes an intervention-based metric measuring the quality of factor encoding in the representation while providing information about its structure.", "questions": "Please see the weaknesses.", "limitations": "Please refer to the weaknesses. There are no particular limitations of this paper -- there is a common limitation of disentangled representation learning that these approaches are not scalable to large datasets.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed metric OMES (Overlap Multiple Encoding Scores) is designed to measure two properties of disentangled representations: modularity and compactness. The proposed metric agrees well with other established metrics and also agrees well with performance metrics. \n\n2. The experiments are extensive and comprehensive, covering four application scenarios (syn2syn, syn2real, real2real) and supported by six datasets. Some empirical insights are also revealed from the experiments. For example, the authors suggest that Explicitness is usually well maintained, while Modularity and Compactness are reduced as we move from synthetic to real. Interestingly, the authors also found out that fine-tuning is always beneficial, which is not an expected behavior to me.", "weaknesses": "1. The presentation of the proposed metric has poor readability and is not structured well. The description from line85 to line132 is a bit messed up: for example, the input image pair and the motivation for choosing subsets of latent dimensions at line102 should be described earlier, while it might be more appropriate to move the high-level description to the introduction. I would suggest the authors add a few headlines to the metric introduction at least or rephrase the contents. \n\n2. Could you please consider synthetic datasets composed of more complex transformations, such as Falcol3D and Issac3D [1]? Though the datasets in the paper are very diverse, I feel a bit the transformations are relatively too simple. It is not sure to me if these insights transfer to real-world complex transformations. In particular, is fine-tuning still beneficial? \n\n3. How about vector-based disentanglement methods [2][3]? How do the metrics and experiments generalize to vector-based methods? Do the authors think the insights will also transfer to these vector-based approaches?\n\n\n> [1] Semi-supervised stylegan for disentanglement learning. ICML. 2020. \n> \n> [2] Flow Factorized Representation Learning. NeurIPS. 2023.\n> \n> [3] Unsupervised Learning of Disentangled Representations from Video. NeurIPS. 2017.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable insights provided with their comment. In the response, we will follow the structure of the bullet list in the review.\n* > The presentation of the proposed metric has poor readability... at least or rephrase the contents.\n\n   We agree with the reviewer that this section would benefit from a revision to improve the readability (which is probably not optimal also due to the space constraints). As suggested by the reviewer, we will anticipate the details on the setup and the main design choices, and we will move the high-level description and motivations to the introduction. We hope this will help the clarity and readability of this important section of our work.\n\n* > Could you please consider synthetic datasets composed of more complex transformations ... is fine-tuning still beneficial?\n\n     We thank the reviewer for the observation, which allows us to further elaborate on the strengths of our proposed methodologies. \n\n     Considering the limited time for the rebuttal, among the two datasets suggested by the reviewer, we opted for Isaac3 because we find it to be more in line with the scenarios already considered, while providing a considerably higher complexity, with its 9 latent factors.  \n\n     With Isaac3D, we performed some further experiments by using the new dataset both as a Source and as a Target, and following the procedure described in Sec. 3.1 and common to all the experiments already included in the paper. The results of these new experiments are reported in the pdf attached to the rebuttal. More specifically, in Tables 1 and 2 we report the results when Isaac3D is used as a Target. In this case, the models trained on Shape3D have been used as a Source.  \n\n     We observe that the transfer is overall effective according to all the disentanglement scores. If we focus in particular on the “All” column, we notice that the performance here is higher than the one we obtain when Shape3D is Source and Coil is Target (Tab. 4 in the main paper) or RGBD is Target (Tab. 5 in the main paper). This is in line with the “picture” provided by OMES (Isaac3D as Target: 23,15, which becomes 33.1 after finetuning; Coil as Target: 28.35, and 27.95; RGBD: 28.85 and 28.2). We notice that only for Isaac3D the boost of finetuning is appreciable, while the baseline result is lower than for Coil and RGBD (that are real datasets). Overall, these empirical evidences may suggest that the robustness of transferring disentangled representations is influenced by several factors including the Sim2Real challenge but also the number of factors, their granularity, and how many factors Source and Target have in common. \n\n     This is related to a more general concept of distance between datasets, we will consider in our future investigations and that we briefly discuss in the general response.\n\n     On the other hand, Table 3 of the attached pdf reports the transfer from Isaac3D as the Source to Shape3D as the Target. Although the richer transformations of Isaac3D, the model adaptation with finetuning is still beneficial for all the disentanglement metrics. We think this may be explained by the “domain” dependence of  VAE models, for which an adaptation step is in general beneficial.\n\n* > How about vector-based disentanglement methods ... insights will also transfer to these vector-based approaches?\n\n     Please refer to the common response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ee3bS9gW3G", "reviewer": "Reviewer_Fxij", "summary": "The work is situated in the area of Disentangled Representation Learning. The authors propose a new intervention-based metric (OMES) to assess the degree of disentanglement in different models. They perform extensive and comprehensive experiments validating and comparing OMES with other metrics. They thoroughly discuss the behavior of their metric with respect to other metrics and provide intuitions for the observed behavior. Finally, they use OMES to study transfer in disentangled representations from a source to a target dataset. The authors wish to understand which disentanglement properties remain after the transfer which they measure with OMES (and other metrics).", "questions": "OMES has one hyperparameter alpha in Eq. 1. As far as I have seen, it has always been set to 0.5. Does alpha need tuning / is 0.5 the optimal choice? What happens if alpha is set to a different value? Does this value depend on the dataset?\n\nThe authors assess the distance between the source and the target data in order to understand how this distance affects transfer performance. While the distance between two datasets is hard to measure, I felt the use of “distance” was a bit loose here, especially because the first research question explicitly involves the distance: “How well does disentanglement transfer, and how much does it depend on the distance between Source and Target Dataset?” In line 51, the authors write: “We discuss the role of fine-tuning and the need to reason on the distance between Source and Target datasets.” I have not really seen any reasoning on the distance between the Source and Target datasets in the paper and would be interested to hear how the authors think about it. The distance could be measured in FoV or in the pixel space or in the “domain” space, for example. We could have the datasets A, B and C where A has 5 FoVs and consists of real images. Then, B would have 4 FoVs (a subset of A) and consist of synthetic, but realistic looking images. Finally, C would have the same 5 FoVs as A, but would consist of sketches, that is, the image distribution would be very far away from A and B. Which dataset would then be closer to A in terms of “distance”: B or C? And how would we expect the disentanglement measures to transfer? I would think that the trained VAE heavily depends on the image domain and switching real images to sketches would destroy all performance in terms of disentanglement.", "limitations": "I think a paragraph which explicitly discusses the limitations of the approach would be beneficial for the paper.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is very well written and well structured. The OMES metric is explained and motivated well. The comparisons to the other metrics are comprehensive and sensible. I especially appreciate the formulation of the main research questions in Section 3.1 which guide the logic of the paper.\n\nThe conducted experiments are very extensive and I appreciate the additional results in Appendix C.", "weaknesses": "I do not see major weaknesses with this paper. I think this is a solid and a mature submission. I have some comments and questions which should be well addressable.\n\nI found the results section in 3.3. quite hard to read. Each of the transfer experiments consists of one large block of text which is a bit hard to parse. I encourage the authors to think about restructuring it and introducing more paragraphs to promote readability.\n\nFig.1 left: the ylabel should be renamed to “Disentanglement score” for clarity.\n\nTable 2: Please write a more descriptive caption\n\n\n### Typos:\nLine 33: tranferring\n\nLine 169: Agreeement\n\nLine 589: Table 7 not Fig 7", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for considering our work and for their valuable comments and insights.\n* > I found the results section in 3.3 quite hard to read ... promote readability.\n\n   We agree with the reviewer that, mainly due to space limitations, the experimental section is very compact. We will attempt to re-design the section to improve its readability (for instance with bulleted lists if more space will be allowed) and will try to refer more explicitly to the research questions to provide clearer storytelling in our discussion.\n\n* > Fig.1 left: the label should be renamed ...  for clarity.\n\n   The reviewer is right, we will correct the label.\n\n* > Table 2: Please write a more descriptive caption\n\n   We agree with the reviewer that a more descriptive caption is needed for readability. Unfortunately, we sacrificed the captions (the one in Table 2, but also some others in the following) for space reasons. We will revise it in compatibility with space constraints. A possible caption is the following: “Disentanglement metrics of transfer to XXX (Dataset Target). Average scores before and after finetuning; in brackets, is the difference between finetuning and direct transfer.”\n\n* > Typos:\n\n   We thank the reviewer for pointing this out. We will correct all the typos in the revised version of the paper.\n\n* > OMES has one hyperparameter alpha ... on the dataset?\n\n   We agree with the reviewer that the text lacks an appropriate discussion on alpha. We will add some comments on the main paper, and we will provide more details in the Appendix, where an empirical analysis on the role of alpha was actually already provided (Appendix B.3), but without an appropriate and focused discussion.  \t\n\n    Alpha does not need a specific tuning, since its role and behaviour are predictable by design. With reference to Eq. 1 we start observing that with alpha=0, OMES only measures the Compactness (MES) of the representation; with alpha=1, instead, our metric measures the Modularity (OS) only. Values of alpha in the interval (0,1) can be used to balance the importance of both contributions. In this sense, we can say it does not depend on the dataset, while it depends on which property(ies) we want to evaluate on the model. This is also implicitly shown in our experimental analysis (Sec. 2.3) where different datasets have been employed.\n\n    In Appendix B we report the results with different values of alpha, in which we show the behaviour of OMES when changing its value. In the main paper, we considered instead the general case of alpha=0.5 (*i.e.* compactness and modularity have the same importance) in the absence of valid reasons to favour one property or the other. We think this choice also allows us to have a fair comparison with the other disentanglement metrics.\n\n* > The authors assess the distance between the source and the target data... destroy all performance in terms of disentanglement.\n\n   Please refer to the common response.\n\n* > I think a paragraph which ... would be beneficial for the paper.\n\n   We agree with the reviewer that a section which more explicitly discusses the limitations of our approach would be beneficial for our work. In the present version of the paper, as observed by Rev. GNcy, we only discuss the limitations of transferring disentangled representations in Sec. 3.3, when commenting on the experimental analysis. Other important limitations of our current work are the use of a specific family of approaches (VAE-based, as noticed by Revs. GNcy and R67v) and the lack of a more considered strategy to reason on the distance between Source and Target datasets (this reviewer and Rev. GNcy), that may give insights on the best choice for the model to transfer (*i.e.* Source) depending on the specific task (*i.e.* Target) at hand.\n\n    We will add details on the main paper, if possible (given the space constraints), or alternatively in the Appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their efforts in reading our paper and also for providing valuable insights and new interpretations for our work. There is a general agreement on the effectiveness of the new metric and the extensive assessment with a thorough experimental analysis.

In this common response, we address comments shared by more than one reviewer. We are responding to each reviewer independently on the remaining points.
* Rev. **GNcy** and Rev. **Fxij** express concern about *how to assess the distance between the Source and the Target datasets*. While a discussion on datasets distance is out of the scope of this paper, the reviewers' comments lead us to see a potential application of our approach in future research. 

  It is true the word “distance” may be not appropriate, in a sense we are assessing *how two datasets are related mainly in terms of FoVs*: 
  1. If both Source and  Target are synthetic, we normally possess a prior on their FoVs and we may reason on their similarity in a structured manner (choosing Source and Target with a majority of identical factors, such as dSprite and color-dSprite, or with similar factors such as Coil100 and BinaryCoil100). 
   2. If the Target is real, we may not possess a proper FoVs annotation but we may qualitatively identify some dominant FoVs of interest, which we could use to find an appropriate Source where such factors are present.
   3. We notice that, even in the supervised case, FoVs are often similar but not identical, they may differ in internal variability/granularity (*e.g.* *Scale* factor in dSprites and Shapes3D, *Pose* in Coil100 and RGBD-Objects), or may have the same name but be different in nature (*e.g.* *Orientation* in dSprites and Shapes3D).

   We conducted the transfer experiments considering different scenarios (syn2syn, syn2real and real2real) to verify which disentanglement properties are preserved or degraded with the transfer. For this, we reasoned on scenarios of increasing complexity, including, for instance, scenarios where the Target has the same FoVs of the Source plus an additional one, or where the Source is a “simplified” version of the Target (*e.g.* from a binarized version of Coil100 to Coil100).  Interestingly, as reported in Sec. 3.3 using a real dataset as a Source does not provide particular improvement in performance with respect to the use of a synthetic Source if we apply the fine-tuning. 

   In the case of real data, this might suggest that *using a synthetic Source dataset* (well related to the Target one in terms of FoVs) and adapting the model with finetuning can (in principle) *provide similar performances to models trained on real images*, and hence closer in pixel and domain space. In the revised version of the paper, we will add a note on the choice of the Source depending on the Target.

***

* Rev. **GNcy** and Rev. **R67v** observe that our methodology mostly employs VAE-based methods and *ask about the implications of using more recent/powerful methods*. We agree with the reviewers that an analysis of the generalisation of our 
insights to different disentanglement learning approaches is needed, and it will be the object of our investigations in the near future.

   Our decision to consider VAE-based models in this work, and Ada-GVAE in particular, is due to their *simplicity*. Keeping the complexity of the disentanglement learning approach under control, we are relatively confident that the benefits we might observe are due to the transfer methodology of disentanglement rather than to the high expressive power of complex models for disentangled representation learning. It is also worth mentioning that the sampling strategy of Ada-GVAE methods is similar to the one we adopt for OMES, and this allows us to obtain an overall coherent procedure.

   To add some more observations on the approaches suggested by the reviewers, we observe that recent works based on generative mechanisms belong to the vector-based family of approaches. This means that they adopt more than one dimension in the form of a vector to encode a generative factor, while dimension-based methods (*e.g.* VAE) usually adopt a single dimension to represent the factor. For this reason, vector-based methods are more suitable for coarse-grained factors (capturing more information) while dimension-based methods are suitable for fine-grained factors. As observed in [A], the latter is more appropriate when investigating under-explored research directions (as in our case) being more precisely defined. 


   All the existing disentanglement metrics discussed in the work have been proposed for evaluating VAE-based methods, including ours. However, they can be easily used to evaluate vector-based methods, for example, performing PCA as post-processing on the representation before evaluation, as already done in the literature [B, C].
   We have no particular reasons to think our insights are not transferring to vector-based approaches. However, each family of approaches for disentanglement learning *follows specific paradigms that may require tailored designs for transfer learning*. In other words, while the general transfer methodology is still applicable, it may need proper tuning to perform optimally depending on the particular learning approach. 

   The generalization of our methodology to different disentanglement learning approaches will be object of our future investigation, since they represent a further step forward from the current state of our work. We will include the suggested works in the bibliography of this paper, referring to them in conclusions and future works.


   [A] Wang, et al. "*Disentangled representation learning.*" arXiv:2211.11695; PAMI 2024 (accepted)

   [B] Yang, et al. "*Disdiff: Unsupervised disentanglement of diffusion probabilistic models.*", in NeurIPS 23.

   [C] Du, et al. “*Unsupervised Learning of Compositional Energy Concepts*” in NeurIPS 21.

Meta-review: The authors make a solid contribution to the field of disentangled representation learning by introducing OMES, a novel metric for evaluating disentanglement, and by providing a comprehensive empirical study on disentanglement transfer. While the paper is technically sound and the proposed metric has the potential for significant impact, there are areas for improvement, particularly in the presentation and generalization of the findings. Please use the provided feedback to enhance clarity and discuss broader applicability of the results.
Decision: Accept (poster)