Transferring disentangled representations: bridging the gap between synthetic and real images

Authors: jacopo  dapueto
nicoletta  noceti
francesca  odone
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:21:59

Abstract: Developing meaningful and efficient representations that separate the fundamental structure of the data generation mechanism is crucial in representation learning. However, Disentangled Representation Learning has not fully shown its potential on real images, because of correlated generative factors, their resolution and limited access to ground truth labels. Specifically on the latter, we investigate the possibility of leveraging synthetic data to learn general-purpose disentangled representations applicable to real data, discussing the effect of fine-tuning and what properties of disentanglement are preserved after the transfer. We provide an extensive empirical study to address these issues. In addition, we propose a new interpretable intervention-based metric, to measure the quality of factors encoding in the representation. Our results indicate that some level of disentanglement, transferring a representation from synthetic to real data, is possible and effective.

Main paper: 
 introduction: 
 Developing meaningful, reusable and efficient representations is a critical step in representation learning [1, 58, 57, 67] . Disentangled Representation Learning (DRL) [1, 40, 24, 67] aims to learn models that can identify and disentangle underlying Factors of Variation (FoVs), hidden in the observable data. These models encode them in an interpretable and compact shape [31, 9, 1, 73] , independently from the task at hand [22, 39, 66, 67] . Moreover, DRL enhances explainability, robustness, and generalization capacity across various applications [67] . Disentangled representations have been shown useful for various downstream tasks, such as FoVs prediction [41, 40] , image generation [72, 48, 43, 42, 59] and translation [21, 19, 38] , fair classification [56, 39] , abstract reasoning [66, 63] , domain adaptation [35] , and out-of-distribution (OOD) generalization [11, 20] . While all the abovementioned methods may rely on different definitions of disentanglement (see just as examples [1, 23, 64] ), and in this sense a comprehensive comparison is hard, they usually share the observation that some level of supervision on the FoVs is beneficial for disentanglement. However, labelling every single factor to achieve fully supervised disentanglement is costly or even unfeasible [70, 52] . For this reason, DRL has been mostly validated on synthetic or simulated data, usually acquired on purpose [11, 41, 60] , and there is a limited understanding of the potential of DRL to address general-purpose representation tasks, as well as the specific challenges of the real world (e.g. the presence of clutter and occlusion, correlation between factors [11] , etc.). Such challenges may prevent the model from learning perfectly disentangled representations [65] . In this work, we propose the adoption of Disentangled Representation (DR) transfer to deal with complex realistic/real dataset. DL transferring was explored in [20] , where Source models learnt in an unsupervised manner were transferred to a Target dataset, by transferring hyperparameters. The authors observed a limited effectiveness in the direct transfer of representations. Instead, Dittadi et al. [11] found out that disentangled representation can help in OOD generalization from a simulated to a smaller real dataset. In both cases, the study involved very specific types of dataset, built to emulate the real one in every detail. Recently, Fumero et al. [18] addressed disentanglement in real data without the need for FoVs annotation, leveraging the knowledge extracted from a diversified set of supervised tasks to learn a common disentangled representation to be transferred to real settings. We follow a different direction, setting up a very straightforward and generalizable procedure: we resort to a weakly supervised approach [41, 25, 26] to learn DRs on Source datasets where the FoVs are known and annotated, to then transfer (with no supervision) such representation to a Target dataset where the FoVs are not known or available. Our final aim is to consider real datasets as a Target, while synthetic data (where FoVs annotation is easy to obtain) can be employed as a Source. The paper presents three main contributions: (1) a novel metric to assess the quality of disentanglement, which is interpretable, classifier-free and informative on the structure of the latent representation; (2) a DR transfer methodology to Target datasets without FoV annotation; (3) an extensive experimental analysis that considers different (Source, Target) pairs and quantitatively assesses the expressiveness of the learnt DR on Target of different nature (including the case where the gap between Source and Target is large), taking into consideration the main expected properties of disentangled representation. We discuss the role of fine-tuning and the need to reason on the distance between Source and Target datasets. The paper is organized as follows. In Section 2, we propose and discuss our new intervention-based metric, OMES. In Section 3.3, we introduce our transfer approach to DRL, and provide a thorough analysis of different types of transfer scenarios (synthetic to synthetic, synthetic to real, real to real). Section 5 is left to the conclusions.2 Evaluating the quality of disentanglement 
background: 
 While there is no universally accepted definition of disentanglement, there is common agreement on the properties that a DR should have [12, 54, 66, 1, 57, 2] :Modularity [55] : A factor influences only a portion of the representation space, and only this factor influences this subspace. This is achievable if the FoV are independent, meaning that a variation in one FoV does not affect others.Compactness [55] : The subset of the representation space affected by a FoV should be as small as possible (ideally, only one dimension). This property is also called completeness in [13] .Explicitness [54] : DR should explicitly describe the factors, thus it should favour FoVs classification.The taxonomy presented in [12] groups all metrics in three families (see a summary in Table 6 in App.): Intervention-based metrics compare codes by intervention, either creating subsets of data in which one or more factors are kept constant (BetaVAE [24] and FactorVAE [27] ), or in which only one factor is varying (RF-VAE [28] ), and predicting which factors were involved in the intervention; Predictor-based metrics use regressors or classifiers to predict factors from DR (DCI Disentanglement [13] and SAP [32] ) or intervened subsets (BetaVAE, FactorVAE and RF-VAE);Information-based metrics leverage information theory principles, such as mutual information, to quantify factor-DR relationships (Mutual Information Gap (MIG) [8, 12] , MED [6] , Modularity [55] and InfoMEC [25] ).Intervention-based metrics have the advantage of providing control over the factor and the corresponding representation. However, they are all based on classifiers, thus they depend on method, hyperparameter settings and model capacity. The latter consideration can be extended to all Predictorbased metrics. On the other hand, Information-based methods are mainly ground on the computation of Mutual Information, which is dependent on an estimator and its parameters [51, 7] . Motivated by these limitations, we introduce in the next section a new metric, to the best of our knowledge, the first classifier-free intervention-based metric.OMES (Overlap Multiple Encoding Scores) is an intervention-based metric measuring the quality of factor encoding in the representation while providing information about its structure: we measure modularity, analyzing how the FoVs overlap, and compactness, detecting and quantifying how a factor is encoded in the dimensions of the representation.Algorithm 1 Compute association matrix S between dimensions and FoVs Require:D Φ = [R R R 1 , R R R 2 , k k k], n ▷ n number of FoVs Ensure: R R R 1 , R R R 2 ∈ R N ×m , k k k ∈ R N ▷ m = |A|, N number of pairs in D 1: S ← ZEROS(m, n) 2: for j = 1 to n do 3: R R R 1 j = R R R 1 [k k k == j, :] and R R R 2 j = R R R 2 [k k k == j, :] 4:for h = 1 to m do 5:P C = P earsonCorr(R R R 1 j [:, h], R R R 2 j [:, h]) 6:S[h, j] = 1 -abs(P C) i S ← ZEROS(m); i S [h] = 1 4: scores[h] = 1 -MAE(i S , S[:, j]) 5: end for 6: return POOLING(scores)Given an image X, with Φ its mapping into a d-dimensional latent disentangled space, Φ(X) = r, r ∈ R d . We discard dimensions whose empirical standard deviation is extremely small (< 0.05), meaning that the dimensions are inactive [68, 10] .This leaves us with a subset of m ≤ d active dimensions, to which we will refer in the following. Let D be a dataset formed by image pairs,D = {(X 1 i , X 2 i , k i )} N i=1, where X 1 i , X 2 i are two images that differ for only the FoV k i .OMES requires computing a weighted association matrix S between the dimensions of the representation and the FoVs, with higher association values if the factor is encoded in a certain dimension (see Algorithm 1): we consider the representations of the image pairs in dataset D, obtaining D Φ . In matrix notation we may write it asD Φ = [R R R 1 , R R R 2 , k k k], where the pair R R R 1 and R R R 2 are N × m-dimensional matrices with each row i is the representation of the i-th image pair Φ(X 1 i ) = r 1 i and Φ(X 2 i ) = r 2 i respectively.For each FoV k we extract the rows of D Φ such that the i-th entry of vector k k k is k i = k, we call this set D k Φ . Each entry S[h, j] of the association matrix relates a dimension h of the estimated disentangled representation with a FoV j. Its value is in the range [0, 1] with elements close to 1 corresponding to a dimension that effectively captures the variations of a FoV. The association is based on a correlation analysis: since the samples from D k Φ are paired to differ only for the FoV k, we expect a good representation not to correlate where such FoV is encoded. To ease interpretability, we transform the obtained values (see Algorithm 1, line 6) so that high values denote a strong association between dimension and FoV. When the model exhibits perfect disentanglement, each row and column of the association matrix S present just one element with a high association, corresponding to the only dimension where the factor is encoded. We thus measure the level of disentanglement through similarity with an ideal array, where the association matrix shows all 0s but in the positions of the correct associations, where there are 1s.We rely on the above considerations to derive our metric as a linear combination of two main contributions. The Overlap Score (OS) penalizes the overlap of different FoVs in the same dimensions (Algorithm 2 -in this case, each row of S, associated with a dimension, is compared with the ideal array) and hence measures Modularity, while the Multiple Encoding Score (MES) penalizes the encoding of the same factor into different dimensions (Algorithm 3 -in this case each column of S corresponding to a FoV is compared with the ideal array) measuring Compactness. In both algorithms, we derive a vector summarizing the contribution of all dimensions for the FoV. The final score in [0, 1] (higher values meaning higher disentanglement) can be obtained with a pooling (either MAX or AVERAGE) on the vector. The OMES metric is computed asEQUATIONWith α = 0, OMES only measures the Compactness of the representation (MES component); with α = 1, instead, our metric measures the Modularity only (with OS). Values of α in the interval (0, 1) can be used to balance the importance of both contributions.Relation with existing metrics. To the best of our knowledge, the only metrics capturing more than one property are DCI [13] and the very recent InfoMEC [25] . Differently from DCI, our metric is intervention-based with no influence on the choice of the specific classifier that may inevitably impact the results, as observed in [7] . With respect to InfoMEC, that must be applied to quantized latent codes, our metric is more general and accepts continuous latents. OMES is based on the intervention of the FoVs, thus we require the FoV to be (at least partially) known: in particular, samples are coupled so that they differ in one FoV only. In this, OMES differs from existing intervention-based metrics [24, 27] in which the intervention is the opposite (samples have only one FoV in common). Our pairing requires less supervision, and it is usually easier to obtain during data acquisition (for instance, from videos [41] ). In addition, it has been shown that this type of pairing provides more guarantees on disentanglement properties [60, 41] .Finally, compared to Information-based methods, we exploit Correlation instead of Mutual Information, hence we do not need its estimation that can be sensitive to parameters choice (e.g. granularity of the discretization [7] ) and choice of estimator [51, 7] .We now analyze OMES, extending previous studies on the unsupervised [40] and weakly supervised [41] setting. As for the unsupervised case, we exploit available 5400 trained models from [40] (3 datasets, 6 values for β, 50 random seeds, 6 unsupervised methods:β-VAE [24] , FactorVAE [27] , β-TCVAE [8] , DIP-VAE-I [32] , DIP-VAE-II [32] , AnnealedVAE [5] ); in this section we report an analysis on Noisy-dSprites, the remaining 2 benchmarks can be found in the Appendix B.1 and B.2. Instead, for the weakly supervised case trained models are not available, so we reproduce the models as in [41] training them on Shapes3D and on other datasets that can be found in Appendix B.3.The metric, by construction, allows us to compute the overall score and a score for each FoV separately: we can thus interpret the effect of hyperparameters on the single FoV, and evaluate the FoV separately in each dimension of the representation. Moreover, by inspecting the metric at a factor level, we may identify uneven behaviours (e.g. models performing similarly on average but for different contributions from the factors). The perturbation aims to simulate 3 scenarios: noisy matrices where the disentanglement can be more or less strongly derived; models exhibiting partial disentanglement; models with no disentanglement. As it can be appreciated, the metric score nicely reflects the disentanglement intensity.Agreement of OMES with other disentanglement metrics. We extend the analysis in [40] . The rank-correlation (e.g. Fig. 2 (Left)) between the previously proposed metrics and our OMES (for α = 0.5) shows that the latter has a high level of correlation with MIG and DCI, but mild correlation with BetaVAE, FactorVAE (OMES is based on the opposite intervention type), and Modularity. This is consistent on all benchmarks. We show in Fig. 2 (Center) the score distribution of the metrics, computed on the whole set of models. We observe OMES produces a wider range of values with respect to MIG and DCI: our metric looks more descriptive, similarly to BetaVAE and FactorVAE.Agreement with performance metrics. Similarly to [41] , we consider the more informative weaklysupervised setting and discuss the rank correlation between our metric and performance evaluations (ELBO, reconstruction loss, and test error of FoVs classifier). Our analysis, reported in Fig. 2 (right) shows that OMES performs similarly to DCI, negatively correlated with the Reconstruction loss, and positively with the ELBO. It also correlates with the performances of GBT10000 (the classifier we will use in the experiments) while it mildly does with MLP10000. This empirical evidence is in line with what was observed in [11] . The correlation with the classification score is a sign that OMES is able to capture the property of expliciteness of the representation, although it is not directly measured by our metric. It is worth mentioning the correlation of OMES with the performance metrics is more stable than what is obtained by other metrics across different datasets (see the Appendix B.3).Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: In Sec. 3.3 and in Sec. 4 (Limitations), we discuss the potential and the limitations of the methodology.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
transferring disentangled representations: 
 Fully unsupervised disentangled representation learning has been shown unsatisfactory in many scenarios [40] . However, annotating the FoVs can be a very critical and uncertain process. In this section, we propose a general-purpose methodology for transferring disentangled representations learned from supervised synthetic or simulated data to an unsupervised dataset (in terms of the FoVs). This approach allows us to evaluate the effectiveness of disentangled representations transfer, and its potential in real-world applications.Most of the focus in learning disentangled representations has been on synthetic datasets whose ground truth factors exhibit perfect independence by design [44, 53, 15, 34, 4] . Instead, real-world scenarios present several challenges that we want to investigate in our analysis. We consider β-VAE models with weakly supervised learning specifically we adopted Ada-GVAE [41] , for its simplicity and its sampling strategy similar to our metric, on a Source Dataset, using pairs of images that differ in ℓ factors of variation. We set ℓ = 1 as it was shown to lead to higher disentanglement [41] . Following [11] , we vary the parameter β in {1, 2}, sufficient to achieve high disentanglement with weak supervision [11, 41] .We evaluate the quality of the disentanglement in a transfer learning scenario, assessing the transferability of the disentangled representation on a Target Dataset, with the final aim of targeting real scenarios. The evaluation we report considers our metric OMES, as well as DCI and MIG, the most widely used metrics in the literature [50, 18, 17, 14, 69, 25, 45] . Moreover, in accordance to [40, 41, 11] , we evaluate the quality of the disentanglement also in terms of accuracy w.r.t. a downstream classification task, with a classifier per FoV. We evaluate the latter in two modalities: (1) Considering the entire representation and (2) Selecting with OMES the dimension of the representation that best encodes the FoVs. Our analysis addresses three main research questions: Q1 -How well does disentanglement transfer, and how much does it depend on the distance between Source and Target Dataset?We will consider different transfer learning scenarios (syn2syn, syn2real, real2real) and pairs (Source, Target) datasets with different distances. Q2 -Which properties of a DR are preserved on the Target Dataset?We will discuss Explicitness of the representation (through FoV classification), Compactness (analysing the component MES of OMES, the MIG metric, as well as the performances of the one-dimensional representation), Modularity (relying on the OS component of OMES and on DCI). Q3 -How effective is fine-tuning on the disentanglement?We will consider the performances of the FoVs classification, the compactness and the modularity on the Target dataset before and after fine-tuning.In our analysis, we consider both synthetic and real datasets offering different challenges, a summary of their properties is in Tab. 1. Some of the datasets are DRL-compliant, meaning that there is full independence between the FoVs (this is reported in column Indepencence), and FoVs appear in all their possible combinations. This is easy to achieve if the dataset is specifically tailored for DRL, but it can not be easily obtained in general.dSprites [44] is a dataset of 2D shapes generated from 5 ground truth FoVs: Shape, Scale, Rotation, x and y Positions. Variants of the dataset have been proposed: in Noisy-dSprites the background is filled with uniform noise; Color-dSprites includes Color as an additional FoV; Noisy-Color-dSprites adds uniform noise to the latter. We refer to them as: N-dSprites, C-dSprites and N-C-dSprites. Shapes3D [4] is a dataset of 3D shapes, generated from 6 ground truth FoVs: Floor colour, Wall colour, Object colour, Scale, Shape and Orientation. It is characterized by the presence of Occlusions.Isaac3D [47] is a synthetic dataset of a 3D scene of a kitchen where a robot arm is holding objects in a variety of configurations. It is characterized by 9 real-world complex FoVs, including robot movements, camera height, environmental conditions (e.g. lighting).There are few real datasets available specifically meant for DRL. [20] is a collection of datasets covering the transitions from simulated to real data, which is, however, not fully available at the moment. [11] is not appropriate for our analysis since the real data section is very small compared to the complexity of the task. We consider instead real benchmarks proposed for classification tasks, chosen to reflect some of the real-world challenges but possessing some "semantic connection" with the synthetic dataset we refer to, e.g. in terms of the expected FoVs. This allows us to reason on the potential of transferability. Example images are in Appendix C.1.Coil is derived from Coil100 [46] . The original dataset contains 7200 real color images of 100 objects. The objects were placed on a motorized turntable against a black background. The turntable was rotated to vary object pose w.r.t. a fixed camera, producing self-occlusions and 2D silhouette changes. We augment the original dataset with two additional FoV, a planar rotation (9 angles) and a scaling (18 values). Therefore, we identify 4 FoV (Objects, Pose, Rotation and Scale) that, by construction, are independent. To consider in our analysis a real dataset visually related to dSprite, we derived a binary version of Coil, called Coil(bin), by applying Otsu's thresholding [49] . RGBD-Objects [33] is a dataset of 300 common household objects acquired by a RGB-D camera.The objects are organized into 51 Categories and a varying number of instances for each category.For each object, 3 video sequences have been acquired with different camera heights (Elevation) so that the object is viewed from different angles while rotating (Pose). Then, images have been cropped so that the object is always in a central position. For our experiments, we used a subset with one object instance per category to make it semantically similar to Coil100 but with the additional complexity of variability in the background, presence of occlusions and clutter. Hence, we control 3 FoVs (Category, Elevation, Pose), but other factors are hidden or not annotated (e.g. Background, Illumination, etc.) due to a realistic acquisition protocol. We refer to RGBD-Objects as RGBD. We also use a variant of the dataset, including depth maps only, referred to as RGBD(depth).Implementation details. We trained 20 different models (10 random seeds × 2 values of β) for each Source dataset. We adopted the same training strategy as in [11] (see Appendix C.2). As for FoVs classification, following [11, 41] , we consider Gradient Boosted Trees (GBT) [16] and a Multilayer Perceptron (MLP) [37] with 2 hidden layers of size 256. Since the specific choice of a classifier is not crucial for our analysis, here we report GBT, MLP can be found in Appendix C.7. Fine-tuning to the Target dataset of the VAE models is unsupervised and it is carried out for 50k steps.Tables description. The tables group different experiments based on the Target dataset. For each FoV, we report under the name the number of values the factor can assume (i.e. its granularity). The tables report the average classification performance over the 20 models, before and after fine-tuning. The latter is reported in parenthesis in terms of gain or loss w.r.t. the performance before the fine-tuning. All is the average performance of all FoVs. The column Pruned highlights the two different representation modalities: if the classifier is trained on the whole representation (✗), or using only one dimension, i.e. the one showing the strongest encoding of a certain FoV according to the OMES metric (✓). As already mentioned, a good performance of the former is an indication of explicitness, while the latter is a positive sign of compactness. Tables also report metrics assessing Modularity (our MES and DCI) and Compactness (our OS and MIG). Note that we exploit the interpretability of OMES in the transfer learning process to select the most representative dimension of the representation for the classification (the "Pruned" columns).(1) Synthetic to synthetic. As a baseline, we consider the case in which both Source and Target datasets are synthetic and we have access to the annotation of the FoVs, they are DRL-compliant. If Source and Target have the same FoVs (S=dSprites with T=Noisy-dSprites or S=Color-dSprites with T=Noisy-Color-dSprites, see Table 2 ) we observe that pruning the representation to just one dimension maintains, on average, stable performances. This shows that the compactness of the representation is preserved for the Target dataset, both before and after fine-tuning. Fine-tuning allows for improved performance in terms of explicitness preserving the remaining properties of the representation, also in the case of the pruned representation. The Orientation FoV is difficult in these datasets as it suffers from reconstruction errors. We increase complexity by adding a new FoV to the Target dataset (S=dSprite with T=Color-dSprite, see Table 2 ).Table 2 : Quantitative evaluation of transferred disentangled models using the dSprites family of datasets. We transfer from a Source (ST) to a Target Dataset (TD). We report the average classification accuracy obtained with GBT on the full and the pruned representations (see text). The last columns on the right report a comparison between disentanglement metrics, including MES and OS. All FoVs in common between Source and Target are effectively classified, again except Orientation. As for the new FoV (Color), we report lower performances, but we can appreciate a significant improvement with fine-tuning if we exploit a global representation. Instead, we observe a lower improvement with the pruned representation, suggesting that the new factor is not encoded in one single dimension. To further increase the distance between Source and Target, we consider pairs for which the semantics of the FoVs are the same, but they are different in appearance, granularity, and composition (S=Color-dSprite with T=Shapes3D, see Table 3 ): we can observe that even without fine-tuning, the latent representation allows the classification of the dominant FoVs of the dataset, i.e. Floor Hue and Wall Hue, also when focusing on a single dimension. Fine-tuning positively affects the average classification accuracy, especially when using the whole representation.We finally reason on the gap between Source and Target datasets in terms of complexity. When the Source is simpler than the Target but still they have some FoVs in common, possibly with different appearances, (e.g. S=Shapes3D, T=Isaac3D, Table 9 and Table 10 ) we can appreciate the effectiveness of transfer and fine-tuning for all metrics. Conversely, when the Source is much more complex than the Target (e.g. when S=Isaac3D, T=Shapes3D) one could expect the richness in the Source to be directly transferrable to the simpler Target. However, we observe that the finetuning is still beneficial for all the disentanglement metrics. This can be explained by the "domain" dependence of VAE models.Discussion. Disentanglement transfers well between synthetic datasets with the same FoVs, w.r.t. all the properties. If the Target includes new FoVs, fine-tuning is necessary for the new FoV, but also for the entire representation, as compactness and modularity are partially degraded by the new FoV. When the Source and Target become significantly different, fine-tuning is also beneficial. We can conclude that when both Source and Target are synthetic and DRL-compliant, the properties of disentangled representation are preserved before and after fine-tuning, especially when the datasets have FoVs in common even though they have different appearance.(2) Synthetic to Real. We now analyse the potential of transferring a disentangled representation from an appropriately generated Synthetic Source (DRL-compliant) to a Real Target. We first consider Real Targets with FoVs independence. In Table 4 , we first analyse (S=Color-dSprite with T=Coil): the Target dataset shares some FoVs with the Source (such as Scale, inplane Orientation, and Object, the latter related to shape), but their variability and granularity may be very different (e.g. the shape/object can assume 3 values on Source and 100 on Target ). Other FoV are new, such as Pose (encoding 3D rotations). The accuracy we achieve is very uneven on the different FoVs, in particular Pose, since the Source does not incorporate any 3D information. We notice an improvement with fine-tuning but also a degradation in performances with the pruned representation (a sign the representation does not produce a good disentanglement on the Target). An exception to this last comment is the FoV Scale, whose accuracy does not degrade significantly with one dimension only (an indication this FoV is well represented in one dimension).With the same Target, we assess a representation which is incorporating some level of 3D information ( S=Shapes3D, with T=Coil, Table 4 ). This choice does not bring any benefit since the synthetic Shapes3D includes a very simplified form of pose variation. Orientation accuracy degrades, as it is not captured by the Source (indeed, with fine tuning this performance improves). Considering the Target dataset, it clearly presents several new challenges w.r.t. Source ones, we produce a last experiment with a simplified binary version of the Target, meant to be more similar to the binary images in dSprite. In this case, some FoVs are very well represented (Orientation and Scale), while Object presents low performances due to the decrease in the image descriptive power caused by binarization.We now consider another real Target, RGBD-Object (see Table 5 ). As for the former, we consider Color-dSprite as a synthetic source: here, the same considerations about the Pose we discussed before are valid, and on the other FoVs, we observe the global representation is effective (and marginally improved by fine-tuning), while the pruned representation leads to lower performances, as a sign the representation is not perfectly disentangled on the Target. Here again, we investigate the possibility of transferring to simplified versions of the dataset (in this case, we consider the Depth channel only). The FoV Elevation improves significantly as it is not directly affected by color information. Concerning Modularity (Table 4 and Table 5 ), fine-tuning seems to have a small influence. In particular, we observe a small degradation for RGBD-Objects (Table 5 ), coherent with the challenges of a real dataset with unknown/hidden factors perturbing the encoding of the FoV.Discussion. If the Source is synthetic (and DRL-compliant), and the Target is Real, the quality of transferring seems to depend on the distance between datasets and is not even across different FoVs: FoVs that are more similar between the datasets are more easily represented, and they exhibit better compactness properties. Fine-tuning is bringing a significant benefit in Explicitness and some (limited) benefit in Modularity and Compactness. If the Target incorporates unknown hidden factors, as we may expect to happen in the real world, Modularity and Compactness transfer worse, and the benefit of fine-tuning is limited.(3) Real to Real. We conclude by discussing the possibility of transferring from a DRL-compliant real dataset to another real one. As a first task, we consider as a Source a simplified version of the Target (specifically, S=Coil-binary, with T=Coil): the source should encode the factors not related to RGB, while the finetuning should improve the disentanglement and the explicitness of the representation. However, this is not the case with Coil100, whose representations degrade the Modularity, and the finetuning only affects the entire representation.We then consider a larger variation between Real Source and Real Target (specifically, S=Coil with T=RGBD-Object, see Table 5 ): we obtain similar results to those of Color-dSprites as Source Dataset (comparable Explicitness), with a reduction on the performances obtained by the pruned representation. Notice that adopting the binary Coil as a Source causes only a limited reduction in Explicitness, and this was somewhat unexpected as we have a large gap in complexity between Source and Target. Our experiments did not consider RGBD-Objects acting as a Source dataset, not being DRL-compliant.Discussion. Using a real DRL-compliant dataset as a Source, we do not appreciate any benefit. Finetuning is not particularly effective. At the same time, we notice that some level of disentanglement transfer can be observed.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We provide the code as supplementary material, while the datasets we used are all publicly available.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
limitations: 
 A limitation of our current work is the adoption of a specific family of approaches (VAE-based). The generalization of our finding to more recent vector-based approaches (e.g. [71, 62, 48, 36] ) needs further investigation. However, each family of approaches for disentanglement learning follows specific paradigms that may require tailored designs for transfer learning. In other words, while the general transfer methodology is still applicable, it might need proper tuning to perform optimally depending on the particular learning approach. 
conclusions: 
 In this paper, we discussed the potential of transferring a Disentangled Representation as a strategy to address disentanglement in real data. We learned the representation from a Source Dataset in a weakly supervised manner and transfered it to a Target Dataset, where supervision on the FoVs was difficult or impossible to obtain. We identified three main scientific questions, summarised in Section 3.2, which we recall to draw conclusions on our study. Starting from question Q2, on the properties of disentangled representations that are preserved after transferring, we may conclude Explicitness is usually well maintained, while Modularity and Compactness are reduced as we move from synthetic to real. More precisely, we appreciate a degradation in the global metrics (such as OS and ME), while on the compactness through the analysis of the 1-dimensional pruned representations, we notice that some FoV may transfer very well.As for Q3, we may observe that fine-tuning is almost always beneficial, and it never causes any harm. Q1, a much wider question discussing under what circumstances transfer is effective, leads us to conclude that some structural similarity between Source and Target datasets is necessary, including similar ranges/granularity of variations of related factors. A quantification of the similarity among datasets is still under investigation; the results of our study suggest one could design synthetic data to capture/disentangle specific factors of interest. Future directions. Currently, we are exploring quantitative methods to assess the distance between Source and Target datasets. In the near future will target more specific applications, such as biomedical image classification or action recognition from videos, to discuss and relate the general results we are reporting in this paper to more specific and challenging domains. A Evaluating the quality of disentanglement  
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: In Sections 3.3 and Section 2 and in Appendix C we provide all the details to understand the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: While we report in the main document the Tables of the averaged measurements, we report in Tables of the same structure the standard deviation of our transfer experiments in Appendix C.8. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: We provide the code with instructions about the usage, reproducibility of experiments and license.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The work does not include experiments with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The work does not include experiments with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section, we report the evaluation of the 1800 models trained on Noisy-dSprites, 1800 models trained on SmallNORB and 1800 models trained on Cars3D, all from [40] . Here we report the extensions of the results in Section 2.3Fig. 4 shows our metric OMES scores for different values of β keeping the different FoV separated, for Noisy-dSprites (Left), SmallNORB (Center) and Cars3D (Right). α is fixed to 0.5.In addition, Figure 3 shows the range of values with different α for a selection of S. We include 3 synthetic cases ((I), (III), (IV)) producing high scores, and (V) generated from Shapes3D, is very similar. The scores of the Noisy-dSprites models ((VI), (VII)) are lower, as it is a more challenging dataset. Color-dSprites (V) is easier to disentangle and output values in between Shapes3D and Noisy-dSprites. Note how the boxplots generated from real models produce a smaller range of values w.r.t the simulated cases; the choice of α does not appear critical in the real case.Fig. 4 shows the metric scores for different values of β keeping the different FoV with α = 0.5, the 3 benchmark datasets (Noisy-dsprites (Left), SmallNORB (Center), Cars3D (Right)). In general the greater the β the higher the disentanglement but the factors strictly related to reconstruction quality fail to be encoded in the representation.Fig. 5 shows the distribution of the disentanglement metrics, extending the plot from [40] with our metric OMES computed with different values of α ∈ {0.0, 0.3, 0.5, 0.8, 1.0}. We observe the higher the α the less variable the distributions of our metrics, meaning that the models are more similar in terms of Multiple Encoding than they are in terms of Overlap.Fig. 6 shows the rank correlations of the disentanglement metrics of the models trained on Noisy-dSprites, extending the plot from [40] with our metric OMES computed with different values of α ∈ {0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}. We observe the higher the α (Multiple Encoding) the higher the correlations with BetaVAE Score and FactorVAE Score and negative correlations with Modularity.Analogously, Fig. 7 shows the rank correlations of the disentanglement metrics of the models trained on SmallNORB, extending the plot from [40] with our metric OMES computed with different values of α ∈ {0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}.Finally, Fig. 8 shows the rank correlations of the disentanglement metrics of the models trained on Cars3D, extending the plot from [40] with our metric OMES computed with different values of α ∈ {0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}.Fig. 9 shows Rank correlation with ELBO, reconstruction loss, and test error of FoVs classifier for the models trained on the weak-supervised setting that was shown to be more interesting for this analysis. We consider all the different Source datasets used in the transfer experiments. Here we provide additional information about the architecture of the models for the transfer experiments. Moreover, we include the tables reporting the average performances of the GBT and MLP classifiers.Table 7 reports the main information about the used dataset, e.g. FoV and number of classes, together with some examples of the original dataset, plus some samples of one variant of the given dataset, such as Color-dSprites and Noisy-Color-dSprites. The only exception is Shapes3D which does not have any variant, so different samples drawn from the same dataset are shown.Table 8 shows the architecture of the model used for all the transfer experiments. We trained multiple models with 10 different random seeds for each value of β. Hence, we obtain 20 models for each Source dataset. We adopted the Adam optimizer [29] with default parameters, batch size=64 and 400k steps. We used linear deterministic warm-up [11, 61, 3] over the first 50k training steps. We maintained the latent dimension fixed to 10 for all the experiments. Comparing the latter with the reconstructions of Fig. 11 (Left) it can be observed that the quality is comparable (with some exceptions), and so with the fine-tuned models, we are not losing much information from the data. Each GBT and MLP classifiers are trained on the latent representation r extracted from the Encoder of the Ada-GVAE models so that the train split comprises 10000 samples and the test split is of 5000 samples. With unsupervised fine-tuning on the Target dataset, it means that the model is trained as a simple VAE [30] .In this section, we report the performance distribution of the GBT classifiers on the target FoV (see Fig. 12 ), the figures on the left depict the performances on the representation before fine-tuning and on the right are depicted the results after the fine-tuning of the representation. Fig. 13 shows the performance distribution of the MLP classifiers on the target FoV, the figures on the left depict the performances on the representation before fine-tuning and on the right depict the results after the fine-tuning of the representation.In this section, we report the results of the transfer from Shapes3D to Isaac3D, see Table 9 for the Explicitness and Table 10 for Modularity and Compactness. We observe that the transfer is overall effective according to all the disentanglement scores even though the Target dataset is much more complex than the Source. If we compare the results of the transfer from Shapes3D to other real datasets such as Coil100 (Table 4 ) and RGBD-Objects (Table 5 ), we notice that in the case of Isaac3D the boost of finetuning is more noticeable. This suggests that, if source and target have common FoVs with similar appearance we can obtain reasonable performances on a real Target dataset even if the source synthetic dataset is much simpler.In this section, we report the performances of the MLP classifiers on the FoVs, for the sake of comparison we report the scores of the disentanglement metrics on the representation input.We associate the tables regarding the same Target dataset: Table 11 corresponds to the same representations of Table 2 in the main document; Table 12 to Table 3 ; Table 13 to Table 4 and Table 14 to Table 5 .Overall we can observe higher performances obtained with MLP, especially on the classifiers trained on the entire representation. This happens because MLP can easily disentangle an entangled representation by observing different dimensions at a time while GBTs partition the input space into regions aligned with the axes, making it harder to observe multiple dimensions at a time and so the FoVs. This aligns with what is observed in [41, 11] . Given this, it is preferable to observe the performances of the GBT because they can give a clearer about the disentanglement properties of the representation.In this section, we report the standard deviation of performances of the GBT and MLP classifiers on the FoVs, for the sake of comparison we also report the standard deviation of scores of the disentanglement metrics on the representation input.MLP classifiers: Table 19 corresponds to the same representations of Table 11 in the Appendix; Table 20 to Table 12 ; Table 21 to Table 13 and Table 22 to Table 14 .GBT classifiers: Table 15 corresponds to the same representations of Table 2 in the Appendix; Table 16 to Table 3 ; Table 17 to Table 4 and Table 18 to Table 5 . Note that the metrics scores in the MLPs and GBTs tables are the same because we refer to the same representations. In general, we can observe the models without fine-tuning have a small standard deviation, while fine-tuned models perform with higher variation. All the experiments have been executed with an NVIDIA Quadro RTX 6000. On average, the training and evaluation of a single Source model take 5 hours. Each fine-tuning and final evaluation takes 1.5 hours. Overall, the whole bunch of transfer experiments and our metric assessment take approximately 1100 hours.Table 16 : Target dataset: Shapes3D. The standard deviation of the performances of the GBT classifiers and disentanglement metrics, together with the standard deviation of the performances with the finetuning (in brackets).Standard deviation Accuracy on FoVs(%) Modularity(%) Compactness(%) SD TD Pruned Floor Hue (10) Wall Hue (10) Object Hue ( We briefly summarise the future direction of our work.On the data set side, we only considered the synthetic datasets applicable to a wider number of tasks, other synthetic datasets exist but either they are limited in the number of FoV or very specific to a task. We limited our analysis to two real datasets, plus their simplified variants, to tackle a broad but limited number of challenges. We will extend our analysis to more complex real datasets with an increasing number of known and unknown factors, but this requires we also design more complex synthetic datasets able to tackle these complications. For the methods, we will explore the effect of the dimensions of the latent space and different kinds of supervision for training the Source model, as well as including (partial) supervision on the fine-tuning. On the applications side, we will analyse the effect of transferring from different priors (e.g. video sequences, expert knowledge in biological data) and investigate if transferring a disentangled representation will help to increase the level of interpretability of the target representation. Justification: The claimed constributions is a metric described in Sec. 2 and a disentangled representation trasferring methodology defined and experimentally assessed in Sec. 3.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: In Appendix D we are reporting an approximation of the time execution and GPU hardware.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: We read the NeurIPS Code of Ethics and the paper conforms with it (we do not involve human subjects or participants, we used already existing datasets respecting their terms of usage, etc.).Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no social impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Justification: There is no risk of misuse because our work does not include generating tasks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The exploitation of already existing models have been properly cited with the original authors.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 


Decision: The authors make a solid contribution to the field of disentangled representation learning by introducing OMES, a novel metric for evaluating disentanglement, and by providing a comprehensive empirical study on disentanglement transfer. While the paper is technically sound and the proposed metric has the potential for significant impact, there are areas for improvement, particularly in the presentation and generalization of the findings. Please use the provided feedback to enhance clarity and discuss broader applicability of the results.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel classifier-free metric for quantitatively measuring disentanglement and investigates transferring disentangled representations from a synthetic dataset with ground truth factors of variation (FoV) to a target real-world dataset. The authors introduce OMES, a novel intervention-based metric that evaluates the correlation between representations of two images sharing all FoVs. OMES computes the overlap between different FoVs (Overlap Score) and measures the encoding of each factor within single dimensions (Multiple Encoding Score). Using these metrics, the paper analyzes the properties of disentangled representations and source/target distributions to improve disentanglement transfer. The main contributions of the paper are:
- Introduction of a novel classifier-free metric (OMES) for disentanglement evaluation, which reduces hyper-parameter sensitivity.
- Extensive empirical study on transferring disentangled representations from source to target datasets, revealing the potential and properties of disentanglement transfer.
strength_weakness: strength_weakness: - The implications on transferring disentanglement from synthetic datasets to complex real-world datasets are somewhat limited. Although the empirical study indicates that a smaller distance between the source and target datasets and shared FoVs between them are beneficial, these are expected properties of conventional transfer learning. The paper would be strengthened by discussing how to specifically select a proper source dataset for a given target dataset with unknown factors of variation. For example, it would be useful to define and measure the structural similarity between a target dataset and potential source datasets.
- The presentation could be improved. At first glance, it feels like two independent topics (metrics and transfer learning) are being introduced, making it hard to understand why a novel metric is needed for transferring disentangled representations. It would help readers if the connections between these two components were more clearly explained.
questions: questions: - In transferring disentangled representation learning, how much of the improvement comes from transfer learning compared to training from scratch on the target dataset, in terms of disentanglement metrics? This comparison would provide a clearer understanding of the benefits and effectiveness of the transfer learning approach.
- Can you provide more analysis on why OMES has relatively low correlation on FactorVAE score in Figure 2? 
- Most of the investigations are done on VAE-based models. Would it have the same implications with other recent disentangled representation learning approaches [1, 2, 3], which employ more powerful generative models? 


[1] Yang, et al. "Disdiff: Unsupervised disentanglement of diffusion probabilistic models.", in NeurIPS 23. 

[2] Lin et al., “Infogan-cr and modelcentrality: Self-supervised model training and selection for disentangling gans”, In ICML 20.

[3] Ren et al., “Learning disentangled representation by exploiting pretrained generative models: A contrastive learning view”, in ICLR 21.
limitations: limitations: - Section 3.3 describes the limitation of transferring disentanglement learning.
paper_summary: paper_summary: This paper conducts an empirical investigation into the problem of transferring disentangled representations from synthetic data to real-world data (syn2syn, syn2real, real2real). They start with well-defined research questions and perform the investigation on the feasibility and effectiveness of transfer learning step by step. Besides the investigation, this paper also proposes an intervention-based metric measuring the quality of factor encoding in the representation while providing information about its structure.
strength_weakness: strength_weakness: 1. The presentation of the proposed metric has poor readability and is not structured well. The description from line85 to line132 is a bit messed up: for example, the input image pair and the motivation for choosing subsets of latent dimensions at line102 should be described earlier, while it might be more appropriate to move the high-level description to the introduction. I would suggest the authors add a few headlines to the metric introduction at least or rephrase the contents. 

2. Could you please consider synthetic datasets composed of more complex transformations, such as Falcol3D and Issac3D [1]? Though the datasets in the paper are very diverse, I feel a bit the transformations are relatively too simple. It is not sure to me if these insights transfer to real-world complex transformations. In particular, is fine-tuning still beneficial? 

3. How about vector-based disentanglement methods [2][3]? How do the metrics and experiments generalize to vector-based methods? Do the authors think the insights will also transfer to these vector-based approaches?


> [1] Semi-supervised stylegan for disentanglement learning. ICML. 2020. 
> 
> [2] Flow Factorized Representation Learning. NeurIPS. 2023.
> 
> [3] Unsupervised Learning of Disentangled Representations from Video. NeurIPS. 2017.
questions: questions: Please see the weaknesses.
limitations: limitations: Please refer to the weaknesses. There are no particular limitations of this paper -- there is a common limitation of disentangled representation learning that these approaches are not scalable to large datasets.
paper_summary: paper_summary: The work is situated in the area of Disentangled Representation Learning. The authors propose a new intervention-based metric (OMES) to assess the degree of disentanglement in different models. They perform extensive and comprehensive experiments validating and comparing OMES with other metrics. They thoroughly discuss the behavior of their metric with respect to other metrics and provide intuitions for the observed behavior. Finally, they use OMES to study transfer in disentangled representations from a source to a target dataset. The authors wish to understand which disentanglement properties remain after the transfer which they measure with OMES (and other metrics).
strength_weakness: strength_weakness: I do not see major weaknesses with this paper. I think this is a solid and a mature submission. I have some comments and questions which should be well addressable.

I found the results section in 3.3. quite hard to read. Each of the transfer experiments consists of one large block of text which is a bit hard to parse. I encourage the authors to think about restructuring it and introducing more paragraphs to promote readability.

Fig.1 left: the ylabel should be renamed to “Disentanglement score” for clarity.

Table 2: Please write a more descriptive caption


### Typos:
Line 33: tranferring

Line 169: Agreeement

Line 589: Table 7 not Fig 7
questions: questions: OMES has one hyperparameter alpha in Eq. 1. As far as I have seen, it has always been set to 0.5. Does alpha need tuning / is 0.5 the optimal choice? What happens if alpha is set to a different value? Does this value depend on the dataset?

The authors assess the distance between the source and the target data in order to understand how this distance affects transfer performance. While the distance between two datasets is hard to measure, I felt the use of “distance” was a bit loose here, especially because the first research question explicitly involves the distance: “How well does disentanglement transfer, and how much does it depend on the distance between Source and Target Dataset?” In line 51, the authors write: “We discuss the role of fine-tuning and the need to reason on the distance between Source and Target datasets.” I have not really seen any reasoning on the distance between the Source and Target datasets in the paper and would be interested to hear how the authors think about it. The distance could be measured in FoV or in the pixel space or in the “domain” space, for example. We could have the datasets A, B and C where A has 5 FoVs and consists of real images. Then, B would have 4 FoVs (a subset of A) and consist of synthetic, but realistic looking images. Finally, C would have the same 5 FoVs as A, but would consist of sketches, that is, the image distribution would be very far away from A and B. Which dataset would then be closer to A in terms of “distance”: B or C? And how would we expect the disentanglement measures to transfer? I would think that the trained VAE heavily depends on the image domain and switching real images to sketches would destroy all performance in terms of disentanglement.
limitations: limitations: I think a paragraph which explicitly discusses the limitations of the approach would be beneficial for the paper.

Comments: title: Response to the rebuttal
Response to the Authors' comment
Response to Author Rebuttal
Thanks for the response
comment: I have read the rebuttal, the other reviews and the comments. I thank the authors for their responses and clarifications and I am keeping my original score.
Thank you for the detailed comment. I think the improvements seems significant (I guess the scales of DCI and MIG are just typo? e.g., DCI=0.14 means 14.0 I guess?) and the impact of transferring disentangled representation from the source dataset is now clear. Now I have no remaining concerns so I will raise the score to weak accept. I thank the authors for the detailed response.
We thank the reviewer for appreciating the rebuttal.

We now realise we misunderstood one request. Indeed, in Table 4 of the pdf we attached to the rebuttal we reported the performances of the weakly-supervised model trained on the Target dataset and the differences (in brackets) of the scores obtained on the Target in this way or with our full pipeline, while the reviewer was asking for a comparison with the unsupervised approach directly applied to the "target" dataset.

In the earliest stages of our work, we actually adopted the unsupervised approach on our reference datasets and we empirically observed its limitations emerging already on “simple” datasets. For this reason, we opted for some level of supervision.

 To give a quantification of what we gain with this change, we report here the score obtained in our original unsupervised experiments focused on the use of two synthetic datasets, namely Color-dSprites and Noisy-dSprites (for details on the dataset see the main paper). With unsupervised training from scratch on Color-dSprite we have DCI=0.14 and MIG=0.07, which become respectively 53.3 and 34.3 with our full pipeline based on transfer learning (Source dataset: dSprites). 

On Noisy-dSprites, with unsupervised learning from scratch, we obtained DCI=0.05 and MIG=0.02, which become 26.0 and 36.0 respectively with our full approach. 
As a reference, training from scratch on the two datasets using Ada-gVAE we have DCI=64.2 and MIG=49.5, showing the large expected gap between unsupervised and weakly-supervised training from scratch.
I appreciate the authors' effort in addressing the concerns. 
Most of the concerns are clarified. Although I still believe that including experiments on other disentangled representation learning methods would provide a more comprehensive understanding, I agree that the authors' transferring method is not limited to specific methods and this is not a crucial concern. 

However, my concern about the comparison to a model trained from scratch remains unresolved. 
If I understand correctly, their experiments involve two steps: (1) weakly supervised learning (Ada-GVAE) on the Source Dataset, and (2) fine-tuning the model in an unsupervised manner (which does not require GT FoVs). I was asking for the performance gap between step (1) -> (2) (the proposed method) and step (2) alone (training from scratch on the target dataset) to evaluate the impact of step (1). However, it seems that the authors reported the performance of weakly supervised learning (Ada-GVAE) on the Target Dataset, which does not clarify the effect of transferring from the Source Dataset. Please correct me if I have misunderstanding here.
Thanks for the response!

I like the new experiments of Isaac3D and I will keep my original score.