Graph-based Unsupervised Disentangled Representation Learning via Multimodal Large Language Models
Authors: Baao Xie
Qiuyu Chen
Yunnan Wang
Zequn Zhang
Xin Jin
Wenjun Zeng
Venue: NeurIPS 2024 poster (2024)
Keywords: Disentangled representation learning
Interpretable and explainable AI
Multimodal large language model
Computer Vision
Primary area: interpretability_and_explainability
TL;DR: We propose an unsupervised graph-based disentanglement framework to learn the independent factors and their interrelations within complex data, upon the intergration of beta-VAE and multimodal large language models.

Abstract: Disentangled representation learning (DRL) aims to identify and decompose underlying factors behind observations, thus facilitating data perception and generation. However, current DRL approaches often rely on the unrealistic assumption that semantic factors are statistically independent. In reality, these factors may exhibit correlations, which off-the-shelf solutions have yet to properly address. To tackle this challenge, we introduce a bidirectional weighted graph-based framework, to learn factorized attributes and their interrelations within complex data. Specifically, we propose a $\beta$-VAE based module to extract factors as the initial nodes of the graph, and leverage the multimodal large language model (MLLM) to discover and rank latent correlations, thereby updating the weighted edges. By integrating these complementary modules, our model successfully achieves fine-grained, practical and unsupervised disentanglement. Experiments demonstrate our method's superior performance in disentanglement and reconstruction. Furthermore, the model inherits enhanced interpretability and generalizability from MLLMs.

Graph-based Unsupervised Disentangled Representation Learning via Multimodal Large Language Models
Baao Xie Qiuyu Chen Yunnan Wang Zequn Zhang Xin Jin Wenjun Zeng'
Ningbo Institute of Digital Twin, Eastern Institute Technology, Ningbo, China Shanghai Jiao Tong University, Shanghai, China 3 University of Science and Technology of China, Hefei, China bxie@idt. eitech, edu. j jinxin@eitech, edu. cn
Abstract
Disentangled representation learning (DRL) aims to identify and decompose under- lying factors behind observations, thus facilitating data perception and generation. However, current DRL approaches often rely on the unrealistic assumption that semantic factors are statistically independent. In reality, these factors may exhibit correlations, which off-the-shelf solutions have yet to properly address. To tackle this challenge, we introduce a bidirectional weighted graph-based framework, to learn factorized attributes and their interrelations within complex data. Specifically, we propose a 3- V-V. based module to extract factors as the initial nodes the graph, and leverage the multimodal large language model (MLLM) to discover and rank latent correlations, thereby updating the weighted edges. By integrating these complementary modules, our model successfully achieves fine-grained, practical and unsupervised disentanglement, Experiments demonstrate our method's 's supe- rior performance in disentanglement and reconstruction. Furthermore, the model inherits enhanced interpretability and generalizability from MLLMs.
1 Introduction
Disentangled representation learning (DRL) is major goal of artificial intelligence (AI), acclaimed for its enhancement of model robustness, interpretabilityy and generalizability. Essentially, DRL methods imitate the understanding processes of biological intelligence, wherein comprehension of real-world is achieved by separating observations into distinct factors [1]. In this form, specific attributes (c.g., object color, shape, and size) exhibit exclusive sensitivity to the changes of specific factors. Learning of such disentangled representations is of great importance across various do- mains, c.g., computer vision [2. 3, 4, 5, 6]. natural language processing [7, 8, 9], and Al generated content [10, 11, 12, 13]. the current phase, unsupervised DRL methods primarily utilize the Varia- tional Autoencoder (VAE) framework [14], probabilistic model learning representations through a regularization term. This term involves the Kullback-Leibler divergence between the posterior distribution of latent factors and a standard multivariate Gaussian prior, thereby encouraging the factorized representations. To strengthen disentanglement, co-current research [15, 16, 17, 18] focus on the optimization and refinement of the original VAE regularizers, resulting in the family of VAE-based DRL approaches.
Despite the advanced results of the simple and synthetic datasets, VAE-based DRL methods still fall short in interpretability and robustness that are required for effective disentanglement in complex data [19]. This limitation mainly stems from the unrealistic assumption that underlying factors
*in Jin the corresponding author. Code is available at here.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

VAB bele-bseedde flat DRL
GEM: B-VAE and MLLM based DRL Framework with DisGraph
imprescaain
DisGraph
Encoder
@ Meg grece legice inlatere
Encoder
define
g-VAr
hasire
eyes
serill
npwr
VAE-based structural DR.
MLLM
define
mustache
Enooder
beard
br or vetenn
mouth
Imsgge
Eatien
Uhoupertieed
Logiical and praciical
& sedighted relations
directional
generol cubiliyy from MLUM
Ieneri interporsalillyy
Figure 1: The comparison of typical DRI. frameworks with our GEM. The limitations of conventional DRL methods are presented on the left. Conversely, the right- side illustrates the advantages of our framework, which benefited from the integration of the 3-VAE and MLLMs.
are countable, independent, and can be fully disentangled in an unsupervised manner (refer to the top-left of Figure 1). In contrast, the real-world variables are pervasively correlated: red apples are more common than yellow ones; elderly people are more frequent with white hair and receding hairline. Accordingly, an increasing number of recent studies [20, 21, 19, 22] showcases that purely unsupervised DRL is fundamentally impossible without extra priors and inductive biases.
The struggle of typical DRL methods on the complex data returns us back to the essential goal of DRL, i.e., understanding the world as biological intelligence does. This cognitive process can be naturally segmented into three phases: attribute extraction, interrelation perception, and knowledge combination [23, 24], where the latter two stages should not be neglected. From this perspective. several structured DRI. approaches, typically known as Hierarchical DRL [25, 26, 12] and Causal DRI. [21, 27, 28, 29I, have involved the correlations between attributes. However, these approaches usually require extra supervision, and their relations are invariably represented by binary and unidi- rectional fusion, thus limiting the model performance in practical scenarios (refer to the bottom-left of Figure 1). Inspired by the analysis above, we argue that effective and practical disentanglement framework should meet the following criteria: (i) the framework should be fully unsupervised; (ii) the framework should be able disentangle factors while concurrently discovering logical interrelations among them; (iii) the interrelations should be modeled as bidirectional, with corresponding impact scores assigned to each, thereby improving model performance in complex scenarios. On this basis, we propose novel Graph- based dis Entangleeenn framework with Multimodel large language mod- els, dubbed GEM. Specifically, our model employs two complementary branches: 3- 8-VAE based disentanglement branch for the attribute extraction, and a multimodal large language model (MLLM) based branch for the interrelation discovery. The relation-awan awaree representations into a disentangled bidirectional weighted graph (DisGraph). which presents distinct factors as nodes, interrelations as edges, and impact scores as weights. The parameters of the graph are dynamically updated and refined via a graph learner. The experimental results show that GEM achieves superior performance on ane-grained and relation-ave aware disentanglement, while preserving the reconstruction quality. Furthermore, the model is endowed with superior interpretability and generalizability that derived from MLLMs. All in all. our main contributions can be summarised as:
further embedded
To our best knowledge, are the first leverage the commonsense reasoning of MILMSs to discover and rank the semantic interrelations from the perspective of DRL.
We propose a novel and practical disentanglement framework built upon 8- 3-VAE and MLLMs to learn the independent factors and their interrelations in an unsupervised way.
We introduce bidirectional and self-driven graph architecture to encode the relation- aware representations, thus facilitating practical and controllable disentanglement.
2
Related Work
2.1 Standard Disentangled Representation Learning
The definition of DRL. is intuitively given by Bengio et al. [1] technique to separate semantic factors behind observational data. This approach assumes that individual data attributes are sensitive to changes in single latent factors, while not being affected by other factors. The disentanglement of

---

attributes is believed helpful for downstream tasks, e.g., generative models [3, 30, 3, 32], medical imaging [33, 34, 35], image editing [36, 37, 38, 39], and 3D reconstruction [40, 41, 42].
Traditional DRL, methods primarily utilizes the VAE framework, achieving a measure of disentan- glement static datasets. This framework has been further enhanced by extensive models such as 3-VAE [15], 3-TCVAE [16], DIP-VAE [4], factorVAE [43], RF-VAE [44], and a-TCVAE [18] through the optimizations of regularization terms. Despite the successes on simple and statie datasets, standard DRL approaches still encounter challenges in complex data. It is mainly due to the flat and unrealistic assumption: data properties are independent and can be factorized into distinct factors [1, 21, 45, 46]. Locatello et al. [20] have proven that unsupervised DRL is fundamentally im- possible without extra priors. Thus, subsequent studies have demonstrated that practical DRL model with appropriate inductive biases can enhance the disentanglement in real scenes [47, 48, 49, 50].
2.2 Structured Disentangled Representation Learning
In contrast the flat and VAE-based DRL methods, recent research gradually realize that latent factors might naturally involve semantic interrelations, deriving to the branch of structured disentangled representation learning [19]. Within this domain, Hierarchical DRL and Causal DRL are mostly relevant to our work. Hierarchical DRL. presumes that underlying factors have different levels of semantic abstraction, either dependent [51] or independent [25] across levels. While straightforward, Li et al. [25] propose hierarchical VAE-based model to learn semantic representations. Furthermore, Singh et al. [12] introduce FineGAN, a three-tier hierarchical framework for controllable object generation. Li et al. [52] also propose hierarchical DRL framework aimed at facilitating image-to- image translation. Differently, our framework aims to achieve fine- fine-grained disentanglement where the targeted attributes are always flat, e.g., the wrinkle, lipstick, and mustache of faces. Therefore, we rely on the flat representations, but place a strong emphasis on the mutual relations between attributes.
Similarly. Causal DRL methods endeavor to capture the causal relations between disentangled factors. As the first, Yang et al. [21] propose VAI to discover relations from the perspective of causality. Further, Shen et al. [29] propose a weakly supervised framework DEAR with the structured causal model (SCM) as prior. However in our view, current Causal DRL methods have at least three unpractical issues: (i) rely on various degrees of supervision; (ii) aim to model specific event rather than a common scenario; (iii) the causal relationship is often overly simplistic, being impractically binary and unidirectional, i.e., paired variables A and B only have two possible causal relations: either A + B or A + B (otherwise unrelated). In practical, it is common for paired variables to exhibit bidirectional influence, and the impact of such bidirectional relations should be properly ranked.
2.3 Multimodal Large Language Models
Recent years have witnessed the remarkable advancements in Multimodal Large Language Model (MLLM) [53, 54, 55, 56]. Since the release of Generative Pre-trained Transformer (GPT) [57], there has been a research trend over MLLMS regarding to its demonstrated potential in processing multimodal data [58, 59, 60]. As the variants of GPT-4, GPT-4 with Vision (GPT-4V) [61] and GPT-4 omni (GPT-40) [62] enhance to process textual and visual data, enabling richer, context-aware interactions across a range of multimodalities. Concurrently, following works such as Gemini [58], Claude [63], NEXT-GPT's [64] and GLM-4 [65] have strengthen the support to additional modalities.
The powerful capacities of MLLMs gradually make researchers aware of its latent perceptual knowl- edge embedded within networks. Gandelsman et al. [59] investigate the way that CLIP encoder understands visual data, by decomposing representations into individual components. In addition, Basu et al. [66] propose a mechanistic localization approach to explore how the visual properties are encoded in MLLMs. However, our best knowledge, there is limited exploration into leveraging the commonsense reasoning of MLLMs from the perspective of DRL. And we are the first to employ MILMS discover and rank interrelations between semantic factors in the DRL. framework.
3 Methodology
To achieve fine-grained and relation-aware disentanglement, we propose GEM, a novel and practical framework that synergizes the strengths of DRL, and MI.LMs by a bidirectional weighted DisGraph. As depicted in Figure 2, GEM is comprised of two complementary modules: 3-VAE based
3

---

lntent space z
uttributes deterningg
DisGraph(())
Encoder
Decoder
Landmark
wetr foriuuees inhialiattion
prompt engineering
internlet ors disoverry
Graph Learner
MLLM Pat
Egm
JMMAE branch
MLLM branch
Basic blocks
[..... Loss functions
Opooaions
Inner operations
Nodes
Edges
Figure 2: Pipeline of our GEM. The model consists of two complementary branches, termed as 3-VAE branch (blue) and a MILM branch (brown). The former utilizes 3-V based semantic encoder Feem to disentangle underlying factors, while the latter employs prompt engineering to discover and rank interrelations. The bidirectional weighted DisGragh further proposed to embed relation-aware representations, with its parameters optimized constantly by GNN network ggnn.
branch dedicated to extract attributes (Section 3.1), and a MLLM- M-ased branch to discover and rank interrelations (Section 3.2). The relation-aware representations are then embedded into the DisGragh (Section 3.3), which presents factors as nodes, interrelations as edges, and impact scores as weights.
3.1 VAE based Attribute Determining Branch
The fundamental objective of vanilla VAE is to approximate data distributions by employing a maximum likelihood estimation framework outlined in Eq. 1:
pog))(x DxL (gs po + C(0, (),
(1)
where the variational posterior distribution 4s (z) x) is utilized to represent the probability distribution
of the latent variable 2 given the observation æ. The key of Eq. is maximizing the approximation
log po (x) of the true posterior distribution (=(2 and (z(x).
Specifically, the first term of Eq. 2 corresponds to the Kullback-Leebbe (KL.) divergence measuring the distance between distribution qa (z(z)) x) and po [z)x). The second term is denoted as the variational evidence lower bound (ELBO). Empirically, the maximization of ELBO is employed to provide a stringent tight lower bound for the original log- ligeliill log (x(x). ELBO can be reformulated as:
[( Φ) Ex,,) [log [x(2)( BD (q.
(2)
where the initial term, i.e., conditional logarithmic likelihood (...() [log Pa (x [x|z)] is responsible for the reconstruction. Typically, the latent variable is assumed to follow standard Gaussian distribution (0, 1) for (2) so that the KL term actually imposes independent constraints on the representations. Furthermore, subsequent studies [15, 17, 67] highlight that a extra penalty coefficient prior to the KL term, denoted by 3. can significantly strengthen disentanglement. When 3 is set to 1, the 8- 3-VAE reverts to the standard VAE framework. And an increase in 33 encourages more disentangled representations but harms the performance of reconstruction as trade-off. As per the Information Bottleneck (IB) theory [17], constraining the information input to DRL models (e.g.. via 3 penalty coefficient) inherently enables them to identify and learn the most representative factors for successful reconstruction. For instance, when trained on the Shapes3D (a collection of synthetic objects) with a merely three-dimensional latent variable, the attribute determining branch tends to learn the most critical factors, observed to be "object color", "object shape", , and "background shape". These attributes organized in the three dimensions, ordered by their reconstruction contribution.
Specifically, within the processes of this branch, the input image is firstly subjected to processing step utilizing landmark detection functions as instructed by [68] and [69] (see Figure 2). It serves as a regularization phase, to remain the key features through targeted cropping. Additional derivations of this process are documented in the appendix. Then, the pre-processed To is fed into 3 3-VAE based branch, designed to disentangle factors associated with each dimension in the latent variable 2 E Z. However, the input of decoder Dree is the relation-aware variable Zre! = AT f from the DisGraph, rather than the z E Z. It means the prior assumption of po(2) E (0 1) is no longer hold. To address this issue, we reformulate the loss function in 3-VAE as follows:
Lyem (p.7.0) 0) PKL(90(tt (x, 2). (x,)
(3)
Vol. I.m (0, 7, ==(=() EEzg((s Ve log po, (x, 2)
V gx
(4)
(0 2)

---

L (0, 7.8) ==(a() En-p(s) V log
p0,, 2)
V0z
(5)
q0(I, 2)
V. Legm (0, 7, G (2) Ewmple) V2 log
(2,,
V
(6)
2 (T(
where the Φ, 0 and are the learnable parameters of Esems Drec and DisGraph G, respectively. Let's polz,s)
say (z v) = log (a.2) and the the gradients with respect to x and can be obtained during backpropagation by the cross-entropy:
No softplus (-D (=D ((i;=))) zi)) +
No
Ladv = LD((..)) Nbc.N.r.
Σ
soft plus (D (22, (a zi))
where Noc and Nm represent the number of samples and the posterior samples in batch, respectively, Obviously, this loss resembles the adversarial loss utilized in Generative Adversarial Networks (GAN) [70]. Therefore, we employ the adversarial training strategy to optimize (xx, y). Combined with the disentanglement term from the original B- 3-VAE indicated as Ldis the total loss for the attribute determining branch can be expressed as:
Ltotal Aadv + Adis + Agemmagem
(8)
where the λadv. Adis and Agem serve as hyperparameters to balance the disentanglement capability and reconstruction quality, with default values set to 0.8, 0.6 and 0.6, respectively. The detailed derivation process of the adversarial training strategy is provided in the supplementary material.
3.2 MLLM-based Interrelation Discovery Branch
Given a pre-processed image In with targeted attributes A = 21, initialized by the 3-VAE branch, our objective is to discover and rank the mutual relations for each pair within A. As represented by the brown blocks in Figure 2, we employ MILLMs as relation predictor Pret to discover and rank interrelations. Initially, the MLLM is required to score from 0 to 5 for each attribute, where 0 indicates the attribute's 's absence, and denotes its highest expression. As shown in Figure 3, the queries can be formulated as a question in natural language with the input image Lo-
Based on the attribute scores, we subsequently employ Somers' algorithm [71] to rank the bidirectional impact scores of interrelations. For attribute pair (A,.A,) we determine num-
<BOS> You are Al assistant to evaluate the specific attributes based given prompts and images. Please assign a score ranging from 0 to
each attribute, where 0 Indicates the absence of the attribute, and 5 signifies the highest expression of the attribute.
ber of concordant pairs Nc and discordant pairs ND, as delineated by Kendall's Tau [72] algo- rithm. Subsequently, the impact score Siy within = 31,2,3,... k} can be denoted as:
Input image: <image>
Targeted attribute: cattributez Desired degree
No Na
Sij N. No + T; For the reversed relation of (A,, A,), the impact score can be denoted as Syz or Sjz=
(9)
Please assign <score> that accurately reflects the expressive level to which the <attribute> exhibited the <image>, Ensure that you consider and each <attribute> independently and provide only the numerical <score> score as your final answer.
N. Na
S/y S5; N. Na +
(10)
where T; and T₁ is the number of ties only for the independent variable A; and Ay. respec- tively. The calculated and S' are used for initialization and refinement of DisGraph (see Section 3.3). As illustrated in Figure portant to clarify that the primary goal of the MLLM branch GEM lations, where the statistical relativity between two attributes is of primary concern, rather than the absolute scores for the individual attribute. For example, given collection of facial images, it is acceptable if the scores of "age" and "bald" exhibit a positive correlation, even if the specific score values are fluctuating. To ascertain the reliability of MLLMs for interrelation discovery, extra experiments performed as shown in Section 4.4.
Response:
<EOS>
Figure 3: A simplified example of the template for prompting MILMS to evaluate attributes. Specifi- cally, <text> is the interactive token, while <BOS> and <BOS> are tokens denoting the start and end of the input MLLMs, respectively.
is im-
discover interre-
5

---

0 young to
absolute soores
to score "age"
4
2
4or
3ord
2or
MLLMs
score from 0-5 for each atsikute to score "bold'
Our model
X
3or4
4or5
0
Dor
5
3
1or2
bolch in 5 thisk to baid
The scores from MLLMs are fluctuating, which is accaptable, as our model propared to learn the statistical relotivity between attributes
the relechotiy between amroutes
to 'age"
2or2
3r3
or
MLLMs
Our model
for wach alirksute
to Score 'bald"
000
1or2
3ord
Figure 4: Our aim is using the commonsense knowledge behind MLLMs to equip GEM with ability of interrelations discovery, where a certain degree of fluctuations absolute scores are acceptable.
3.3 Bidirectional Weighted DisGraph
Based on the extracted factors and interrelations, we then propose the bidirectional weighted DisGraph (.A, S) to integrate the semantic representations. Specifically, .A is the set of n |.A| nodes, embodying the disentangled attributes as factors. Besides, is the set of = |E| edges, and stands for the weights of these edges. An E and its corresponding impact score E S are embedded. Consequently, G can be presented as the learnable weighted adjacency matrix A E [0. 1|
According to the definitions above, the model firstly constructs a sketched adjacency matrix Ao R*** upon the factors and relations initialized by the 3-VAE branch and MLLM branch. Specifically, treat the averaged impact scores of the first 1,500 images processed by MLLMs, as initial weights of relations. We further employ an unsupervised graph learner Egen to dynamically refine the parameters of DisGraph by the structure bootstrapping mechanism [73] and multi-view graph contrastive learning [74]. The optimization function of Egnn can be formulated as:
T ( = 1 (T(--1) A) D A -
(11)
It converts the sketched adjacency matrix An into node embedding T via the GNN-based multilayer network, where 10 (-) is the embedding function with learnable parameters w of the 1-th layer and T()) is the output matrix. The augmented adjacency matrix A = A incorporates self-loops based on the initial matrix Ao. and D is the degree matrix of A. Further, w(() = Ω E R=** denotes the parameter matrix of the 1-th layer, with o(.) non-linear function that enhances training stability.
Figure 5 illustrates the compre- hensive training algorithm of
Algorithm Truining Algoritam GEM
Input: Image datnser X. Encoder Eure. DisGraph G. Decoder Dao. Discriminator D. panameters the encoder Prems DisGinph decoder Drce Discriminator D as 6, 7
our model. The encoder pro-
cesses input images and outputs the disentangled latent variable
Output: Disentangled latent variable : gene, Correlatin--mmch Istent variable Sya ger
reconstracted image &
while S do
2, which subsequently initializes
MLLM(())
I/ Gerting the attribute by NLLM
the embeddings of nodes in Dis-
A.g while d
SamersD(att); // Gotting adjacency zatrix by Soner's D algorithn
Graph. The adjacency matrix
E.
of DisGraph is calculated using
embeeklnnglii Maskiz.ly
conheddings G(A.s, G/A....cmheediinng; emhoddings):
Somer's D algorithm, which pro- cesses the attribute scores out-
// Refine the DisGrape by using ONE
Reet W.... E,entccidinngiii 2 D(ive):
putted by the MLLM. Following this initialization, a Graph Neu- ral Network (GNN) refines the structure of DisGraph. The aver- age of the feature matrix within DisGraph is then forwarded to the decoder to reconstruct im- ages. Concurrently, the discriminator is trained to approximate the gradient of the loss function. Assuming that the model's performance is upper bounded by the norm of its gradient, which satisfies (PL) condition, this configuration ensures the suboptimality of the model.
Calculase ligen Lats and
m VoLgimlh 9) V.
d + e- m Volgenl (1) 7,7,6) 22 Volanii
7 R. Layne (8. 9):
V. L.
i
Figure 5: Overall training algorithm of GEM.
the

---

4 Experiments
We evaluate the GEM on two datasets: 1) CelebA [75] contains 200,000 high-quality facial images. Each image is annotated with 40 binary attribute labels, making it a widely used benchmark for supervised DRL methodologies. Operating in unsupervised manner, we do not utilize ground-truth labels from this dataset, yet we still conduct comparisons against the supervised approaches: 2) LSUN [76] consists of about one million images across various object categories such as cars, buildings, animals, etc. We select a typical subset from both scene categories and object categories, bedroom and horse, respectively. We believe these two datasets are diverse enough to assess our method covering complex data of different object types.
Datasets.
Implementation details. We implement GEM with PyTorch [77]. The landmark pre-processing settings follow the instructions of [68] and [69]. In addition, we employ the latest GPT-40 [62] as the interrelation predictor. For every experiment, the latent dimension size is set to 6. Concentrating the disentanglement capacity of the framework, all experimental images are resized to a resolution of 64 x64 to minimize computational resources. For high-defintiion outcomes at 256x256, refer to Appendix A.7. All the experiments are processed using the Adam optimizer with a learning rate of le-4, and conducted on the Nvidia Tesla A100 GPUs, with batch size of 32.
Baselines for Comparison. We evaluate the GEM with state-of-the-art DRL methods on the disentanglement capacity, reconstruction quality, and computational efficiency. The comparison encompasses supervised and unsupervised models, including standard VAE [14], 3-VAE [15], 3- TCVAE [16], FactorVAE [43] and DEAR [29]. All baselines are trained using the complete CelebA dataset under the configurations previously specified.
4.1 Qualitative Results
To evaluate the GEM's effectiveness of relation-aware and fine-grained disentanglement, we perform qualitative analyses with FactorVAE [43] and DEAR [29]. The experiments are conducted on CelebA, a standard benchmark that has been previously validated as compatible with these methods. We select the six fine- gree-grriedd facial attributes from the database including Bangs, Bald, Gender, Beard, Blond, and Makeup. The disentanglement results are represented by traversals across various latent dimensions, where each dimension corresponds to distinct attributes.
Beand
FactorVAE
DEAR
Buld
Semantic interrelations determined by MLLMs
Boonge
Blond
off Gonder
Bold
Gender
Bold
AUUI
ASC
4.0
628
Beard
64
Gendery
E25
674
AG3
6:00
Ude
6/1
635
Mand G.ndeerGnndrrr
631
6.M
011
413
Mahsup inlueed salyender
Oener
Seat
Bied
GEM())rs)
Figure 6: Qualitative comparisons between GEM and typical DRL Methods. Each row in facial images corresponds to the traversal results on a specific attribute, as indicated adjacent to the images (i.e. Bangs, Bald, Gender, Beard, Blond, and Makeup). GEM exhibits superior ability in fine-gras grained disentanglement with discovered practical and bidirectional relations (illustrated by the heatmap).
7

---

As illustrated in Figure 6, GEM effectively achieves fine- theeggrined and relation-aware disentanglement, via the integration of DRL and MLLMs. The interrelations determined by MLLMs are depicted as a heatmap in the bottom-right of Figure 6, where deeper colors reflect stronger relations. Since DisGraph is bidirectional, the impact scores for bidirectional relations between a pair of attributes may vary, resulting in an asymmetric matrix. Specifically, in the first row of GEM's 's result, person with heavier Bangs is less likely to be Bald, and the hair tends to be Blond, which is considered logical by MLLMs. Furthermore, as shown in the second and third rows, males (Gender) are more likely to be Bald and less likely to have Blond hair. The attribute Makeup is considered as relatively independent, with lower impacts scores among other attributes.
In comparison, DEAR demonstrates limitations in learning specific attributes such as Bald (second row) and Gender (third row), while the relations between attributes appear to be tenuous. To our knowledge, this underperformance may stem from the stringent nature of causal relations, which are single-directional and heavily rely on the quality of prior. For FactorVAF, since it is flat DRL framework, we employ the same causal relations used in DEAR to make it relation-aware As shown in Figure 4, GEM surpasses FactorVAE in both attribute disentanglement and relation discovery, which indicates the importance of specially-designed modules within our framework.
4.2 Quantitative Results
Table reports the results of Frechet Inception Distance (FID) [70] and Kernal Inception Distance (KID) [70] scores to verify the quality of reconstructed images. To ensure statistical significance, each comparison model undergoes three rounds of evaluations in the same configuration. The results indicate that GEM outperforms both typical unsupervised (VAE, 3-VAE, 3-TCVAE, FactorVAE) and supervised approaches (DEAR) in terms of reconstruction quality. To our understanding, this superior performance attributed to the specialized training strategy implemented the framework.
Table 1: Quantitative comparison results with typical DRL approaches in FID and KID.
CelebA
LSUN- horse
LSUN- bedroom
Method
FID L
KID x10 4
FID
KID x10
FID 4
KID x x10 +
VAE [15]
53.3
$1.4 0.4
172.8 1.7
181.7 2.1
195.8 4.1
226.4 5.4
8-VAE [15]
1.6
2.7
3.2
2.3
288.1
5.7
225.7 6.0
3-TCVAE [16]
1.9.1
113.2
173.0
217.35
179.2 1.0
Vac [43]
1.3
9.5
2.5
235.7 1.5.3
172.8
DEAR [29]
0.3
52.6
1.6
1.3.7
177.6
157.8 1 2.3
GEM (Ours)
46.0
0.2
1.1
6.75
1254
76.1 + 1.1
As shown in Table 1, GEM surpasses baseline models in reconstruction quality on the datasets of CelebA, LSUN-horse, and LSUN-bedroom. However, the use of the disentanglement coefficient in the 3- 3-VAE branch leads to an inevitable trade-off in reconstruction quality, making the model less comparable the models focused on generation quality (e.g., GAN and Diffusion [78]). Therefore, the integration with leading generative models can be a direction for our future work. For additional comparison results, please refer to Appendix A.
Table 2: Computational efficiency report in parameters size, FLOPPs memory cost and training time.
Models
Params(M)
GFLOPs(B)
Mem(M)
TT(s)
FactorVAE
55.9
3.8
200.5
63.9
DEAR
53.4
3.5
267.2
91.8
GEM (Single)
44.7
2.8
173.6
51.5
GEM (Full)
49.6
3.2
222.8
78.9
Furthermore, we evaluate four relation-aware models: FactorVAF, DEAR, GEM (Single), and GEM (Full), on quantitative comparisons of computational resources. Notably, GEM (Single) is the variant GEM that incorporates single attribute determination branch (we only provide the initial relations make it relation-aware) Table 2 shows that GEM outperforms DEAR and is comparable to VAE on computational efficiency. This is mainly attributed to FactorVAE's utilization of simple convolutional encoder, whereas GEM employs 3-VAE based encoder strengthen disentanglement. In addition, the efficiency of full GEM is slightly inferior to GEM (Single), due to the extra modules for relation discovery and refinement.
8

---

4.3 Evaluations of Interpretability and Generalizability
As a by-product, GEM inherits the interpretability and generalizability MLLMs. Theoretically, ow- ing the commonsense reasoning faculties of MLLMs, our model can be generalized to discover any attributes and interrelations across various real- world objects and seenes. To demonstrate the robust- ness and generalizability of GEM, we perform extra experiments specifically targeting the typical object subset LSUN-horse and the scene subset LSUN-bedroom.. Furthermore, test the attributes beyond the 40 specified in CelebA, collectively showcasing the model's 's superiority. To highlight the characteristics of bidirectional weighted DisGragh, we intentionally select paired fine-grained attributes exhibiting inconsistent bidirectional relations.
complex scenes in LSUN,
Dark
Dark
gereio Cheeks
Cheeks
Cheeks
0.13
Dark *
bidierctional interrelations: Dark skin influences Cheeks, but Cheeks does not influence Dark skin.
Ground
Ground
0.04 geraen Running
Color
Color
Running
Running
0.55
Ground Color
bidierctional interrelations: Running influences Ground Culor. but Ground Cotor does not influence Running.
Window Size
Window
Size Brightess
0.16
Brightness
Brightnest
0.84
Window Slae
bidierctional interrelations: Brightness influences Window Size, hut Window Size does not infinence Brightness.
Figure 7: Relation-aware disentanglement results on LSUN and the attributes beyond CelebA. Paired fine-grained attributes with inconsistent bidirectional relations chosen to indicate effectiveness.
As depicted in Figure 7, GEM successfully achieve fine- mne-grained disentanglement on complex scenes, while identifying bidirectional and weighted relations among attributes. Furthermore, the artifacts observed in the results of LSUN datasets are mainly due the datasets' clutter (evidenced by the increase of FID and KID scores in Table 1). Nonetheless, despite the ambiguous and challenging nature the data, GEM still obtain commendable disentanglement outcomes, affirming robustness.
4.4 Evaluations of MILMs
Our model leverages the commonsense knowledge embedded MILMS to predict interrelations. This is predicated the assumption that MILMS, including their future iterations, are powerful and reliable enough to comprehend the physical rules of the real world (e.g., aging brings wrinkles, sunrise brings light, etc.). Therefore, before utilizing the interrelation discovery branch, is imperative to evaluate the reliability of MLLMs. This evaluation guarantees that the identified interrelations and their associated impact scores are dependable and can be effectively applied to downstream modules. To this purpose, we evaluate three latest MLLMs including GPT-4o, GPT-4y and GLM-4 against the ground truth attributes of the CelebA dataset. The horizontal axis presents the targeted attributes selected from the CelebA, where the vertical axis presents the percentage of scoring accuracy.
120
Avg Accuracy
GLM-4: 71 71.3%
80
GPT-Aw: 78.9%
19
GPT-4o: 88.4%
40
Zero-scoring
Young
Gender
Hailine
Makeup
Chubby
Eyebag
Mustache
Beard
Hat
Bald
Glasses
Blend
Lipstck
Smile
Sidebar
Bangs
Eyebroos
GPT-4v: 1.5%
GPT-4o: 0.025%
GLM-4
GPT-4v
GPT-4e
Figure 8: Evaluation experiments on the various MLLMs for attributes scoring.

---

As reported in Figure 8, GPT-4o outperforms other models on individual attribute scoring, achieving accuracy exceeding 90% for the majority of attributes. Specifically, it exhibits superior performance on attributes like Beard, Young, and Fyebrows, where other models yield significantly lower scores. In addition, GPT-4o achieves the highest average aceuracy of 88.4% and the lowest zero-scoring rate at 0.25%, indicating a minimal rate of the meaningless predictions where all attributes are scored as zero. We conducted further evaluations on individual attributes, where GPT-4o also demonstrated superior performance (see Appendix A.7). Based on the evaluations, employ GPT-4o as the interrelation predictor the model.
4.5 Ablation Study
To analyze the effecttivnesss of individual components in GEM, we perform an ablation study focusing on the importance of the 3-VAE based branch, GNN-based graph learner Egnn. and adversarial training strategy. The CelebA dataset served as the experimental platform for the investigations. It is worth noting that the complete removal of B- VAE branch is infeasible, as it would prevent the model from extracting attributes. Therefore to evaluate the importance of independent attribute extraction, we replace the 3- VAE with the vanilla VAE, which does not enforce the independence of factors.
Interrelations without the soll anydringin TeCGroph
unable dieestanglment (but reconetratton qualry bas increased
Bongs
Bald
Gender
w/o GNN based graph learner
Replace VAE with the vanilla VAE
decline in mecostraction quality (but dactors are effectively disentangled)
Bonge
Bald
Gender
w/o Adversarial training strategy
Full model
Figure 9: Ablation replacing B-VAE with VAE, w/o graph learner, and w/o adversarial strategy.
As depicted in Figure 9, replacing 3- VAE encoder results in a declined disentanglement capability, albeit with improvement in reconstruction quality. In addition, the removal of GNN-based graph learner prevents the parameter updating of DisGraph, leading to the inaccurate determination of relations (e.g., the relation between Bald and Gender weakens). It is worth noting that the removal of both graph learner and initialization process within the framework precludes the learning of interrelations. Furthermore, eliminating the adversarial training strategy in GEM and relying solely on the standard VAE loss function results in significant decline in reconstruction quality. The aforeementoned results highlight the effectiveness of each part of our framework.
Conclusion
In this paper, aim to explore the logical interrelations between semantic attributes within complex data, which is a critical challenge that existing DRL have yet to properly address. To this end, introduce GEM, a B-VAE and MLLMs-based framework, designed to achieve fine-grained and relation-aware disentanglement. In this framework, DRL and MLLMS are integrated via a bidirectional and self-dríven graph. Both qualitative and quantitative experiments demonstrate GEM's superior disentanglement and reconstruction capacities over typical DRL models. In addition, the model shows its enhanced interpretability and generalizability inherited from MLLMS.
6 Acknowledgments
This research is supported by the IDT Foundation of Youth Doctoral Innovation [Grant $203.2.01.32 ..3.00.0, the National Natural Science Foundation of China [Grant 62302246) and the Zhejiang Provincial Natural Science Foundation of China [Grant LQ23F010008).
10

---

References
[1] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence. 35(8):1798- 1828, 2013.
[2] Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled representation learning gan for pose-invariant face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1424,,14,2222 2017.
[3] Mingi Kwon, Jacscok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. arXiv preprint arXiv:2210. 220006600, 2022.
[4] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlaheled observations. arXiv preprint arXiv:171 2017.
[5] Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In International conference on machine learning, pages 9786-9796. PMLR, 2020.
[6] Dingkun Liu, Hongjie Zhou, Yilu Qu, Huimei Zhang, and Yongdong Xu. Umman: Unsupervised
multi multi-graph merge adversarial network for disease prediction based on intestinal flora. arXiv preprint axXiv:2407.27114 2024.
[7] Pengyu Cheng, Martin Renqiang Min, Dinghan Shen, Christopher Malon, Yizhe Zhang, Yitong Li, and Lawrence Carin. Improving disentangled text representation learning with infereition-theorrtiic guidance. arXiv preprint (xXx:2006.00693, 2020.
[8] Zhenya Huang, Xin Lin, Hao Wang, Qi Liu, Enhong Chen, Jianhui Ma, Yu Su, and Wei Tong. Disenqnet: Disentangled representation learning educational questions. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 696-704, 2021.
[9] Giangiacomo Mercatali and André Freitas. Disentangling generative factors in natural language with discrete variational autoencoders. arXiv preprint arXiv:2109.01 07690,0071600000001110.11 2021.
[10] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Picter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
[11] Hong Chen, Yipeng Zhang, Xin Wang. Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. Disenbooth: Disentangled parameter-efficeen tuning for subject-driven text-to-image generation. arXiv preprint arXiv:2305.03: 03333334. 2023.
[12] Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. Finegan: Unsupervised hierarchical disentangle- ment for fine-grained object generation and discovery. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6490-6499, 2019.
[13] Yutong Hu, Yang Tan, Andi Han, Lirong Zheng, Liang Hong, and Bingxin Zhou. Secondary structure- guided novel protein sequence generation with latent graph diffusion. arXiv preprint arXtv:2407.07443, 2024.
[14] Diederik Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 131.v11312.6114 2013.
[15] Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR (Poster), 3, 2017.
[16] Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
[17] Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in 3-vac. arXiv preprint arXiv: 804.03599, 2018.
[18] Cristian Meo, Louis Mahon, Anirudh Goyal, and Justin Dauwels. B-te-vae: On the relationship between disentanglement and diversity. In The Twelfth International Conference on Learning Representations, 2023.
[19] Xin Wang, Hong Chen, Si' ao Tang. Zihao Wu, and Wenwu Zhu. Disentangled representation learning. arXiv preprint arX1v:2211.1165,, 2022.

---

[20] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly. Bernhard Schölkopf. and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. international conference on machine learning, pages 4124. PMLR, 2019.
[21] Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Dis- entangled representation learning via neural structural causal models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 9593-9602, 2021.
[22] Xingyi Yang. Jingwen Yc, and Xinchao Wang. Factorizing knowledge in neural networks. In European Conference on Computer Vision, pages 73 91. Springer, 2022.
[23] Saeed Amizadch, Hamid Palangi, Alex Polozav, Yichen Huang, and Kazuhito Koishida. Neuro-symbolic visual reasoning: Disentangling. In International Conference on Machine Learning, pages 279-290. Pmir, 2020.
[24] Tameem Adel, Han Zhao, and Richard E Turner. Continual learning with adaptive weights (claw). arXiv preprint XXiv:1911.09511 2019.
[25] Zhiyuan Li, Jaideep Vitthal Murkute, Prashnna Kumar Gyawali, and Linwei Wang. Progressive learning and disentanglement of hierarchical representations. arXiv preprint arXiv:2002.10549 2020.
[26] Bin Tong, Chao Wang. Martin Klinkigt. Yoshiyuki Kobayashi, and Yuuichi Nonaka. Hierarchical dis- entanglement of discriminative latent features for zero-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogmition, pages 11467-11476, 2019.
[27] Pengzhou Wu and Kenji Fukumizu. B-intact-vae: Identifying and estimating causal effects under limited overlap. arXiv preprint 10.05225, 2021.
[28] Di Fan, Yannian Kou, and Chuanhou Gao. Causal disentangled representation learning with vae and causal flows.
[29] Xinwei Shen, Furui Liu, Hanze Dong, Qing Lian, Zhitang Chen, and Tong Zhang. Weakly supervised disentangled generative causal representation learning. Journal of Machine Learning Research, 23(241):1- 55, 2022.
[30] Insu Jeon, Wonkwang Lec, Myeongjang Pyeon, and Gunhce Kim. Ib-gan: Disentangled representa- tion learning with information bottleneck generative adversarial networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7926-7934, 2021.
[31] Tao Yang, Yuwang Wang. Cuiling Lan, Yan Lu, and Nanning Zheng. Vector-based representation is the key: A study on disentanglement and compositional generalization. arXiv preprint arXiv:2305.1806, 2023.
[32] Xin Jin, Bohan Li, Baao Xie, Wenyao Zhang, Jinming Liu, Ziqiang Li, Tao Yang, and Wenjun Zeng- Closed-loop unsupervised representation disentanglement with 3-vae distillation and diffusion probabilistic feedback. arXiv preprint 24i2:2220200234. 2024.
[33] Agisilaos Chartsias, Thomas Joyce, Giorgos Papanastasiou. Scott Semple. Michelle Williams, David Newby, Rohan Dharmakumar, and Sotirios A Tsaftaris. Disentangled representation learning in cardiac image analysis. Medical image analysis, 58: 10155,, 2019.
[34] Cosmin Bercea, Benedikt Wiestler, Daniel Rueckert, and Shadi Albarqouni. Federated disentangled representation learning for unsupervised brain anomaly detection. Nature Machine Intelligence, 4(8):685- 695, 2022.
[35] Lianrui Zuo, Yihao Liu, Jerry I. Prince, and Aaron Carass. An overview of disentangled representation lcarning for mr image harmonization. Deep Learning for Medical Image Analysis, pages 135-152, 2024.
[36] Abel Gonzalez-Garcia, Joost Van De Weijer, and Yoshua Bengio. Image-to-image translation for cross- domain disentanglement. Advances in neural information processing systems, 31, 2018.
[37] Hsin- Ying Lee, Hung-Yo Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming- Yang. Diverse image- to-image translation via disentangled representations. In Proceedings of the European conference on computer vision (ECCV), pages 35-51, 2018.
[38] Boqiang Zhang. Hongtao Xie, Zuan Gao, and Yuxin Wang. Choose what you need: Disentangled representation learning for scene text recognition, removal and editing, arXiv preprint arXiv:2405.0437. 2024.
12

---

[39] Piaopiao Yu, Jie Guo, Fan Huang, Cheng Zhou, Hongwei Che, Xiao Ling, and Yanwen Guo. Hierarchical disentangled representation learning for outdoor illumination estimation and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15313-15322, 2021.
[40] Baao Xie, Bohan Li, Zequn Zhang, Junting Dong, Xin Jin, Jingyu Yang. and Wenjun Zeng. Navinerf: Nerf- based 3d representation disentanglement by latent semantic navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17992-18002, 2023.
[41] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advances Neural Information Processing Systems, 33:20154-20166, 2020.
[42] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11453-1 1464, 2021.
[43] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International conference on machine learning, pages 2649-26588 PMLR, 2018.
[44] Minyoung Kim, Yuting Wang, Pritish Sahu, and Vladimir Pavlovic. Relevance factor vae: Learning and idemtifying disentangled factors. arXiv preprint 1rXiv:1902.01563 2019.
[45] Zihao Chen, Wenyong Wang, and Sai Zou. Break the spell of total correlation in betatovae, arXiv preprint a0Xiv:2110.08794, 2022.
[46] Karsten Roth, Mark Ibrahim, Zeynep Akata, Pascal Vincent, and Diane Bouchacourt. Disentanglement of correlated factors via hausdorff factorized support. arXiv preprint 2rX0:::2200007347, 2022.
[47] Yunhao Ge. Sami Abu-El-Haija, Gan Xin, and Laurent Itti. Zero-shot synthesis with group-supervisee learning. arXiv preprint arXiv:2009.00 06586, 2020.
[48] Matthew J Vowels, Necati Cihan Camgoz, and Richard Bowden. Gated variational autoencoders: Incorpo- rating weak supervision encourage disentanglement. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pages 125-132. IEEE, 2020.
[49] Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 32, 2018
[50] Attila Szabo, Qiyang Hu, Tiziano Portenier, Matthias Zwicker, and Paolo Favaro. Understanding degenera- cies and ambiguities in attribute transfer. In Proceedings of the European Conference on Computer Vision (ECCV), pages 700-714, 2018
[51] Andrew Ross and Finale Doshi- Velez. Benchmarks, algorithms, and metries for hicrarchical disentangle- ment. International Conference on Machine Learning, pages
PMLR, 2021.
[52] Xinyang Li. Shengchuan Zhang, Jie Hu, Liujuan Cao, Xiaopeng Hong, Xudong Mao, Feiyue Huang. Yongjian Wu, and Rongrong Ji. Image-to-image translation via hierarchical style disentanglement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8639-86666 8648, 2021.
[53] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang. Yupeng Hou, Yinggian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. 8763, PMLR, 2021.
[55] Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-Ilava: Mixture of experts for large vision- languaee models. arXiv preprint arXiv:2401. 2024.
[56] Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473. 2024.
[57] R OpenAl. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5), 2023.
[58] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean Baptistt Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai. Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
13

---

[59] Yossi Gandelsman, Alexei A Efros. and Jacob Steinhardt. Interpreting clip's image representation via text-based decomposition. arXiv preprint arXiv:2310.05916, 2023.
[60]
Zhengyuan Yang. Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: preliminary explorations with gpt-4v (ision). arxiv, arXiv preprint arXiv:2309.14221 2023.
[61]
Gpt-4v(ision) system card. 2023.
[62] Hello, gpt-4o. OpenAI, 2025.
[63] Maxim Enis and Mark Hopkins. From Ilm to nmt: Advancing low-resource machine translation with claude. arXiv preprint arXiv:2404. 13813, 2024.
[64] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal Ilm. arXiv preprint 2.39v:2300.055/17 2023.
[65] Angus Yang, Zehan Li, and Jie Li. Advancing genai assisted programming-a comparative study on prompt efficiency and code quality between gpt-4 and glm-4. arXiv preprint arXiv: 12782,, 2024.
[66] Samyadeep Basu, Keivan Rezaci, Ryan Rossi, Cherry Zhao, Vlad Morariu, Varun Manjunatha, and Soheil Feizi. On mechanistic knowledge localization text-to-image generative models. arXiv preprint arXiv:2405.01000 2024.
[67] Harshvardhan Sikka, Weishun Zhong, Jun Yin, and Cengiz Pehlevant. A closer look at disentangling in 8-vac. In 2019 53rd Asilomar Conference on Signals, Systems, and Computers, pages 888-895. IEEE, 2019.
[68] Deokyun Kim, Minseon Kim, Gihyun Kwon, and Dac-Shik Kim. Progressive face super-resolution via attention to facial landmark, arXiv preprint (rXiv:/908.00. 2019.
[69] Giuseppe Amato, Fabrizio Falchi, Claudio Gennaro, and Claudio Vairo. A comparison of face verification with facial landmarks and deep features. 10th International Conference on Advances in Multimedia (MMEDIA), pages 1-6, 2018.
[70] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[71] Robert Somers. new asymmetric measure of association for ordinal variables. American sociological review, pages 799-811, 1962.
[72] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2)881-93 1938.
[73] Yixin Liu, Zheng, Daokun Zhang, Hongxu Chen, Hao Peng, and Shirui Pan. Towards unsupervised deep graph structure learning. In Proceedings of the ACM Web Conference 2022, pages 1392-1403, 2022.
[74] Thomas N Kipf and Max Welling. supe-super classification with graph convolutional networks. arXiv preprint 1609,v1600,0000
2016.
[75] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[76] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song. Thomas Funkhouser, and Jianxiong Xiao. Lsun: Con- struction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1506.0336633 2015.
[77] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[78] Jonathan Ho, Ajay Jain, and Pieter Abheel, Denoising diffusion probabilistic models. Advances in neural information procesing systems, 33:6840-68511 2020.
[79] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling:
Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.02905. 2024.
14

---

A Appendix
A.1 Discussion
Disentanglement Ability, As shown in Table 3, evaluate the disentanglement ability of model (DRL branch only) against typical DRL models. Our model outperforms them across numerous metrics. However, it is really meaningful and interesting to discuss "what is better disentanglemen""
Table 3: Disentanglement metrics among VAE, 3-VAE and GEM.
Models
MIG
IRS
MI
Info
VAE
0.19
0.35
0.84
0.81
8- VAE
0.49
0.55
0.88
0.82
GEM
0.53
0.60
0.86
0.85
means the better performance on independently decomposing factors, then the inclusion of interrelations might not seem beneficial; however, if it refers to a better performance/practicality for real and complex scenarios, our disentanglement paradigm excels by statistically capturing the logical rules of real world. Specifically, the inclusion of interrelations can be beneficial in model generalizability, counterfactual reasoning and practical usages.
Trade-off between Quality and Interpretability. Even though our model achieved superior perfor- mance among DRL approaches, an inevitable trade-off between reconstruction and disentanglement remains, resulting in decreased reconstruction quality compared to the models focused on generation quality such as GANs and Diffusions (see Table 4).
Table Quantitative comparison results with leading image generation models FID and KID,
CelebA (64x64)
CelebA (256x256)
Method
FID
KID x103
FID
KID x103
GEM (Ours)
46.05
48.32
50.93
51.01
Vanilla VAE
53.39
51.48
56.82
61.26
StyleGAN2 (40k steps)
12.94
9.20
18.02
19.55
DDPM (Diffusion, T 1k)
8.56
6,56
15.93
10.01
DDIM (Implicit Diffusion, T 1k)
10.04
8.15
16.24
13.62
Stable Diffusion (finettningg)
7.72
7.22
10.63
9.17
Since our model is oriented towards interpretability, we consider this trade-off acceptable. However, it is insightful leverage the advantages of both DRL and non-DRL models within a mutually beneficial closed-loop architecture, and we will make efforts to improve our work this direction.
Current Limitations. Compared to existing DRL. approaches, GEM emphasizes discovering un- derlying interrelations between attributes. This logical and effective framework can benefit wide range of downstream tasks and practical applications such controllable generation, medical image analysis, and automatic driving. However, there are still some limitations to this method. Firstly, the trade-off between reconstruction quality and disentanglement capacity, the domain, is still not properly addressed in this work. To tackle it, we are currently investigating the integration of powerful generative models, e.g., diffusion models [78] and visual auto-regressive models [79], into our framework. Secondly, the current implementation GEM is not designed to work with 3D data, where 3D representations much complex. To understand our real world, would be necessary to enhance the model with specific improvements to handle 3D scenes.
common challenge in
A.2 Dataset details
CelebA. The Celebrity Faces Attributes (CelebA) dataset [75] is widely-used large-scale face attributes dataset that contains more than 200,000 celebrity images, each annotated with 40 attributes.
15

---

These annotations cover wide range of facial attributes such as 'smiling', "wearing Hat', "young'. "wavy Hair', 'male', and 'mustache', These attributes are labeled as present or absent each image. This dataset is designed for various DRI. and computer vision tasks, such as face recognition, face attribute disentanglement, and face editing. Commonly, we use employ the entirety of the CelebA dataset, which includes 162,770 images for training. 19,867 for validation, and 19,962 for testing.
LSUN. The Large-scale Scene Understanding (LSUN) dataset [76] a comprehensive collection for deep learning and computer vision, widely used in the domain of perceptual analysis and attribute disentanglement. This dataset consists of around one million labeled images for each of 10 scene categories and 20 object categories. The scene categories include diverse environments such as bedrooms, conference rooms, dining rooms, kitchens, living rooms. and etc. The object categories in LSUN include horse, car, church, etc. We select typical subset. LSUN-horse, to demonstrate the model's s generalizability. We believe these two datasets are diverse enough to verify our GEM.
A.3
Baseline details
A.3.1
3-VAE
The 3-VAE [15] is an extension of the standard VAE [14], introducing an adjustable hyperparameter prior the KL term in vanilla VAE:
[(0, () = (2)x)) [logp((x|z)) - 3D (4s (1s(2)x)lloo(3))
(12)
where the penalty coefficient balances the disentanglement capacity and reconstruction quality. When 3 >> I. this penalty increases the emphasis on learning disentangled representations in the latent space. However, increasing can cause trade-off on the reconstruction quality.
A.3.2 3-TCVAE
The Total Correlation beta-VAE (3-TCVAE) [16] builds upon the 3-VAE further enhance the disentanglement of latent representations. achieves this by the reduction the total correlation (TC) term, extracted from the KL divergence term:
( (((x)II[((ss) + (a [ (1,))
Bpta) L(a ()][)()))) LL =)]l(()))(())) SKL
(13)
where represents the dimension of latent code 2. The penalty coefficient / selectively applied to the second term, i.e. TC term, the right side of the loss function. It aims to make latent variables statistically independent of each other, therefore enhancing disentanglement, We include 3-TCVAE in the quantitative comparisons, following the official implementation.
A.3.3 Factor VAE
The FactorVAE [43] is a typical variant of VAE, which employs discriminator network in an adversarial manner to accurately estimate and minimize the total correlation term in the loss function. This adversarial strategy further enforces the factorization of the latent space, leading to improved disentanglement. We include 3-VAE, 3-TCVAE, FactorVAE in the quantitative comparisons follow- ing their official implementation. We evaluate the reconstruction quality by employing the Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) metrics for comparative assessment.
A.3.4 DEAR
DEAR [29] is the weakly supervised structural disengagement framework. It facilitates causal representation learning by adopting a structural causal model (SCM) as the prior distribution. This SCM prior is supervised by the information the ground-truth factors and their underlying causal structure from the database. We integrate DEAR into the quantitative and qualitative comparisons, training with the annotations provided by the CelebA. All in all, we evaluate our unsupervised framework against both unsupervised (8- VAE, 3-TCVAE, FactorVAE) and supervised (DEAR) DRL methods, thereby rigorously evaluating the capabilities in reconstruction and disentanglement.
A.4 Face landmark results
Landmark detection are algorithm and technique utilized to detect and track specific key points in the image or video, encompassing wide range of applications across various fields such as
16

---

computer vision, robotics, and geospatial analysis. In this work, we employ landmark detection as pre-processing method extract the key points of the main object, thus removing the redundant parts in the image through cropping and resizing.
OOD
OOD
D
Figure 10: 68-points landmark pee-prccessig for data from the CelebA.
We introduce the pre-pros pre-prcessiing phase for CelebA, where 68 landmark points are identified and extracted commonly. As shown Figure 10, points 17 present the jawline, points 18 27 for the eyebrows, points 28 to 36 for the nose, points 37-48 for the eyes, and points 49-68 identify the
lips.
NY NNNY
E G
AWA
LE
Landmare
Landmark
Landmark
Figure 11: Image normalization based on the landmark detection.
Figure 11 illustrates some results of the pre-processing block. The normalized images benefit
subsequent factor disentanglement and interrelation discovery.
A.5 Details of adversarial training strategy
As described in Section 3.1, the prior assumption that po(z) ~ ~N(0 1) no longer holds, we reformu- late the B-VAE loss as:
[(0,0) Φ) Eznge (x) (x) (BErgaa))))) [log (x]=2) xDx (go (2(x)))))
go(2(z)
(14)
Ex, (x BBe.eng(ts.))ogg
p=(z)
Subsequently, the gradient of L with respect & can be derived as:
17

---

VoL(0,0) ==Do(z)
-V ,oog (20 Vox
(0(2
Bg (=(log(po(()))) Blog(
1
po(2)
Yo 40
Byng(s) Vzllog(po =)) Blog
(0(x))
(15)
90 90(2, 2))
Bgng(() Vallog(po (2, 2)) Blog(
Vex
4á (z)
( z) (x 2)
(g Vzolog(
| BBa V Vaqa(z)Vozz
where the first term the right side needs be approximated by neural network due to the altered assumption. Therefore, we define:
p(r, = p(((,,) E Es)p(z, E Eg) + (x),2) E Do)p(1,=1(x,0)) E Do) = =p((x,2) Eo)gs( 2) + p((((((((1))) Do)po (2r,
(16)
then
p((x,2) E Eglx, p((x,2) E Es Es)p(a, 21 Ex)
p
1
as (x. 1
pe (a.=)
(((x,2) 2) E Dolar 2) p((,,=) E Da)p(a, (1x, Do)
(17)
p
+ 1 po(2.) (2.2))
(( E Dg)
a p((,, 2) E E&)
Given the controllable proportion of the input images, denoted by E [0, 1J, we define:
D(x,8) v) log( Pa (2(2,2) 2) BBg (xr, 2)
(18)
then
po 22) D(a,s)
(19)
(2,
where 3 E (1, +oo) is proposed to enhance the disentanglement ability of 3-VAE. We subsequently define the variable k:
suppose
B
p((x, 2) E 20) + ke D(2,2)
(20)
p((x, x) E Dolar,) 1 ((e,)
since : 1- p(0, 2) E Eglar + p((xx 2) Dola,
Given that the solution for & is determined to be 1, under the condition that = 3 = (otherwise v(x, 2) D(x, =) =0 which is obviously impossible),
logically imperative to require distribution p(a, (x, z) Es) and p(z, vl(x, z) E Dp to closely approximate p(z, 2). On this basis, we found:
18

---

p(a, - p(z, 2 2) E Ex)
- p((a, x) E E z) p((,22) E Ex)
(21)
p z) - p(z, =| 2) E D&)
- p((z, 2) E D2 2) p((,,2) E Da)
where if = 1, we attain the optimization objective (( y) E Da) - p((,, v) E Eτ) Nm
To
this end, we can finally use the cross-entropy algorithm to optimize D):
Leade = L(D) = cross crros__eettvee grros_entrrouu(pg..oo
(1 2:) E Fep) Roggp (x;; Fogl,, p((xi, zi) E Do)logp((<;, 24) Dola, =)]
Nbc 0=0
Nuc -ooplpuus(-DDar soot:-osofpppppooo
Nbo 1=0
Nbc
[softplus (itpluus(DD((,,))) 2,)) + softplus( (D 2;))
= Noc. im
Nbc
Σ softplus(-D(x; (og,))DD
softplus(((n,, 2;))],
Nb
(22)
where softplus(x)) In (1 + +*) of samples and posterior samples batch. The additional adversarial loss ensures the maintenance of reconstruction quality, tending to diminish the capacity for disentanglement increases.
) is smooth activation function. Nbc and N.. represent the number
A.6 Details of interrelation determining strategy
In the interrelation discovery branch, we propose the Somers' Delta (Somers' D) [71] algorithm to determine and rank the bidirectional relations among the attributes extracted by the 3-VAE branch. Somers' D is a statistical measure used assess the strength and direction of the relation between variables. It is nonparametric measure that can be considered measure of rank correlation, similar to Kendall's tau [72], but with a focus on asymmetric relations.
Specifically, the calculation of impact scores based on Somers' D is upon the number of concordant pairs (C') and discordant pairs (D). The formula of Somers' D variable Y and X can be given as:
C-D Dyx C+D+T,
C-D DXY = C+D+T,
(23)
where T. is the number ties only for the independent variable X, and T, is the number of ties only for the independent variable Y. The obtained attribute scores and interrelations from MLLMs are evaluated. For example, suppose we have the sample dataset S = (1,2), (3,1), (2.3):
Variable
Pairs
Value
N Na
(1,2) vs (2.3)
(1,2) (3.1) and (2.3) (3.1)
Ny
None
0
Then the Somers' D indicator can be calculated as follows:
N. Na 1-
D
(24)
N. + TT, 1-2+0 3
This obtains a value approximately -0.33, signifying negative correlation between variables X and Y. This calculation demonstrates that the Somers' D metric is straightforward to calculate and
19

---

is particularly applicable to ordinal variables. Furthermore, Somers' D is asymmetric and capable of distinguishing bidirectional relationships between variables. These characteristics make highly suitable for integration into our model.
A.7 Additional results
We perform additional evaluation results for individual attributes on different MLLMS with Ground Truth (GT) labels in CelebA. As shown in Figure 12, the results demonstrate the reliability of the GPT-4o employed in our work.
disenbutton of the ottribute for "ponder"
distibution of the attribute score for 'beand"
ditribution of the octnbute score for "makeup'
3156
2520
2280
1000
1520
1260
760
Score
GUN-4 DFT-4)
Figure 12: Reconstruction results of the CelebA
We also present additional results that demonstrate the capacity of GEM in fine-grained and relation- aware disentanglement. Figure 13, Figure 14, Figure demonstrates the reconstruction results on the CelebA, LSUN- bedroom and LSUN-horre, respectively. It is obvious that following the landmark pre- rre-processing. GEM effectively identify the main part of facial images while mitigating noise. This enhancement facilitates the downstream processes of the model. In addition, as depicted in Figure 16, GEM capable of processing high-definition images given sufficient computational resources.
20

---

Figure 13: Reconstruction results of the CelebA.
21

---

Figure 14: Reconstruction results of the LSUN-bedroom.
22

---

Figure 15: Reconstruction results of the LSUN-horse.
Bangs
Bald
Gender
2
Beard
Blond
Makeup
N
Figure 16: dighhdefinitin results at 256x256 on the CebebA.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly claim the contribution and scope in the abstract and introduction, substantiated by theroretical and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations and future work are discussed in Appendix A.1, which will be included in the camera-ready version.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
24

---

Answer: [Yes]
Justification: All the theoretical statements, theories and assumptions related to this work are referenced. The proofs and derivations of formulas in the paper clearly stated in Section 3 and Appendix A.5.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We are committed to ensuring the reproducibility of this study. The project will be open-sourced, and comprehensive details for reproducibility can be found in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
25

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: The codes will be released the camera-ready version, with detailed instruc- tions for user to reproduce.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nig) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips.
/wcc//ggi//////eeeeeeee /ooeeeeeoooeebbiiiicceee wwwisssssiiicyyyooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, pre-processed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental settings for our model, baseline, datasets and MLLMs can be found in Section 4 and Appendix A.2, A.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The results of quantitative comparisons presented this paper are accompanied by errors bar achieve statistical significance (see Section 4.2).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The computational efficiency of framework is reported and compared with typical DRL models (see Section 4.2).
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp://nnuuppppeeeeeeee ct/public/ Eereee/ttiiiseeeeeeeeeeee csGuidelines"
Answer: [Yes]
Justification:
To our best knowledge, this study does not involve any ethical issues.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The potential positive societal impacts discussed in the paper and Ap- pendix A.I, which will be included the camera-ready version. To our best knowledge. this work does not cause any negative societal impacts.
Justification:
Guidelines:
27

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: Our framework rely on the VAE model, MLLMs and GNN model, so our safeguards are the same as theirs. To the best of our knowledge, these models poses no risks on misuse or dual-use.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the public baselines and datasets used in this paper are properly credited (see Section 4).
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
28

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The assets of this project, primarily the codes, are well documented and will released the camera ready version.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The research presented in the paper exclusively utilizes public datasets and does not involve any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
29

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "0l5FPca2nK", "reviewer": "Reviewer_nREj", "summary": "Objective\nThe paper presents a novel framework for disentangled representation learning, which aims to identify and separate the underlying factors of variation in complex data. The primary goal is to enhance the interpretability and robustness of data perception and generation models.\n\nMethodology\nThe proposed method integrates a β-VAE (Variational Autoencoder) with a bidirectional weighted graph framework and utilizes Multimodal Large Language Models (MLLM) to discover and rank potential relationships between factors. This integrated approach ensures fine-grained, practical unsupervised disentanglement.\n\nContributions\n1. Innovative Framework: Combining β-VAE with MLLM and a bidirectional weighted graph for enhanced disentangled representation learning.\n2. Fine-Grained Disentanglement: Achieves more detailed and practical separation of data factors.\n3. Improved Interpretability: The model inherits the interpretability and generalization capabilities of MLLMs.\n4. Robust Evaluation: Shows superior performance in terms of disentanglement and reconstruction quality on benchmark datasets.", "questions": "1. I did not fully understand how the paper finds the correspondence between the latent representations output by the Encoder and the semantic attributes. For example, in Figure 1, how do the latent representations obtained through the Encoder correspond to attributes such as hat, eyes, etc.? Could you provide a detailed explanation of this process?\n\n2. In the step where MLLMs are used to evaluate attributes (Figure 3), is there a specific principle guiding the MLLM to score the attributes? For instance, what is the difference when the MLLM scores the same attribute as 2, 3, or 4? Additionally, what would the results be if other MLLMs mentioned in your related work section were used for scoring instead of relying solely on the GPT-4 series?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality.\nThe paper presents a unique combination of β-VAE and Multimodal Large Language Models (MLLM) within a bidirectional weighted graph framework. This novel integration allows for capturing and ranking complex relationships between factors, addressing limitations in previous methods.\n\nQuality.\nThe methodology is rigorous, with well-designed components and thorough evaluations on the CelebA and LSUN datasets. The use of Graph Neural Networks (GNNs) for optimizing the DisGraph demonstrates a high level of sophistication and effectiveness.\n\nClarity.\nThe paper is well-organized and clearly written, with detailed explanations and helpful diagrams. Each component of the framework is logically explained, making the complex methodology accessible to readers.\n\nSignificance.\nThe framework enhances interpretability and robustness in disentangled representation learning, with practical implications for image generation and AI explainability. It sets a new direction for future research.", "weaknesses": "1. Dependence on MLLM: The framework heavily relies on Multimodal Large Language Models (GPT-4) to discover and rank relationships between factors. This dependence can be problematic if the MLLM is not sufficiently trained on relevant data or if it introduces biases present in its training corpus. Additionally, relying on a single score from MLLM may not be sufficiently convincing, making this step overly dependent on the accuracy and reliability of MLLM. Despite the evaluations in section 4.4, which assess the accuracy of the scores, having specific guidelines or principles for MLLM scoring would be more convincing than relying solely on a single score.\n\n2. Lack of Detailed Attribution: While the paper introduces a novel framework, it lacks detailed explanations on how each latent representation specifically maps to distinct attributes. This can make it challenging to understand and interpret the exact role of each variable in practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We greatly appreciate your insightful comments and commit to refining our manuscript based on your suggestions. Below, we address all your concerns.*\n\n**W1: Dependence on MLLM**  \n\n**R1:** Thanks for the insightful comments. First, we would like to clarify that our model leverages the commonsense knowledge embedded in MLLMs to predict interrelations. This is predicated on the assumption that MLLMs, including their future iterations, are powerful and reliable enough to comprehend the physical rules of the real world (e.g., aging brings wrinkles, sunrise brings light, etc.). To substantiate the reliability, we evaluate the scores from SOTA MLLMs with ground truths, as detailed in **Section 4.4 and Figure A1 in the attachment**.\n\nNevertheless, we totally agree with your comments, that a solitary score is not convincing enough to represent MLLMs' knowledge. Towards it, two perspectives of considerations are undertaken: 1) enhancing the prompting guidelines to have more informative outputs (e.g., likelihood probabilities, multi-level categorical outputs, averaged iterative scores and etc.); 2) interpreting lightweight bias mitigation modules with certain prior knowledge (e.g., [1] and [2]).  \n\n*[1]Hauzenberger et al., Modular and on-demand bias mitigation with attribute-removal subnetworks, 2023.*\n\n*[2]Kumar et al., Parameter-efficient modularised bias mitigation via AdapterFusion, 2023.*     \n  \n**W2: Lack of Detailed Attribution**\n\n**R2:** Thanks. Actually, it is a frequent question why unsupervised DRL models possess the ability to align specific attributes. We provide a brief discussion here and will offer a comprehensive explanation in the revision. \n\nTo address your concern, we need to analyze it from the information-theoretic perspective. As per the Information Bottleneck (IB) theory [3], constraining the information input to the DRL model (e.g., by the penalty coefficient $\\beta$ in $\\beta$-VAE), inherently enables the model to **identify and learn the most representative factors for successful reconstruction.** For instance, when trained on the Shapes3D (a collection of simple, synthetic objects) with a merely three-dimensional latent variable, the model tends to learn the most critical factors which are observed to be *\"object colour\"*, *\"object shape\"*, and *\"background shape\"*. **These attributes are aligned in the three dimensions, organized in order of their contribution to reconstruction.** Similarly for facial images, the model spontaneously learns and organizes the most informative attributes (e.g., hair, gender and etc.) in the disentangled dimensions.\n\nIn addition, in our framework, we employ a landmark detection for pre-processing, extracting pivotal object features and discarding image redundancies via cropping and resizing. It further helps the mapping of each latent representation to unique attributes.\n\n*[3] Burgess et al., Understanding disentangling in $\\beta$-VAE, 2018.*\n\n**Q1: I did not fully understand how the paper finds the correspondence between the latent representations output by the Encoder and the semantic attributes**\n\n**R3:** In the response for W2, we have clarified that for a collection of data, the most informative attributes will be unsupervisedly learned by the $\\beta$-VAE and sequentially aligned across several dimensions. It highlights the role of $\\beta$-VAE branch to **disentangle and initialize the attributes**. Nevertheless, these initialized attributes need a further human observation to extract textual concepts for subsequent MLLM prompting.\n\n**Q2: Is there a specific principle guiding the MLLM to score the attributes? For instance, what is the difference when the MLLM scores the same attribute as 2, 3, or 4? What would the results be if other MLLMs were used for scoring?**\n\n**R4:** In the paper, Figure 3 illustrates the specific guiding principle, which prompts MLLMs to classify one attribute into multiple degrees on its own judgement. Specifically, it assigns scores ranging from 0 to 5 for each attribute, where 0 indicates the attribute’s absence, and 5 denotes its highest expression. For example, for scoring attribute *''smile\"*, MLLMs tend to assign a score of 0 for the absence of a smile and a score of 5 for a full laugh, **according to the statistical distribution they have learned from extensive data.**\n\nTowards your 2nd question, we wish to further clarify: since the goal of the MLLM branch in GEM is to discover the interrelations, **the statistical relativity between two attributes is of primary concern, rather than the absolute scores for the individual attribute (see Figure A2 in the attachment).** For example, given a collection of facial images, it is acceptable if the scores of *\"age\"* and *\"bald\"* exhibit a positive correlation, even if the specific score values are fluctuating.\n\nAddressing your final concern, we believe that a less capable MLLM would hinder the learning of attribute interrelations. We substantiated this by replacing GPT-4o with the inferior GLM and assessing the outcome. **Figure A3** in the attachment demonstrates that GLMs' limited perceptual capacity yielded counterfactual outcomes. Therefore, the MLLM bias mitigation methods mentioned in **R1** are valuable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DDqgd59VZK", "reviewer": "Reviewer_qyA4", "summary": "The paper presents a novel framework that integrates β-VAE with multimodal large language models within a graph structure, DisGraph, to enhance disentangled representation learning. This approach allows for effective handling of complex and interdependent data attributes in an unsupervised manner. The model dynamically updates relationships between attributes, improving disentanglement and interpretability compared to traditional methods. Extensive experiments demonstrate its superior performance in both disentanglement and reconstruction.", "questions": "1. How are the generated node embeddings T generated by DisGraph associated with the losses introduced in Section 3.1?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper successfully integrates β-VAE with multimodal large language models (MLLMs) within a graph-based framework, which is a novel approach.\n2. The use of a graph-based approach to model the relationships among attributes addresses a significant gap in existing DRL methods.\n3. The experiments are well-designed, covering various aspects of the model’s performance.", "weaknesses": "1. The paper would benefit significantly from clearer writing and better organization. The paper occasionally uses technical terms and concepts without adequate definitions or explanations.\n2. While the paper addresses the unrealistic assumption of statistical independence in many DRL methods, the solution proposed still relies heavily on the disentanglement abilities of β-VAE, which itself often presupposes some level of independence or weak dependence among latent variables.\n3. Also, the reliance on pre-trained multimodal large language models might introduce biases from these models into the disentangled representations.\n4. A deeper theoretical analysis of why and how the inclusion of interrelations leads to better disentanglement would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We greatly appreciate your insightful comments and commit to refining our manuscript based on your suggestions. Below, we address all your concerns.*\n\n**W1: The paper would benefit significantly from clearer writing, organization and explanations**  \n\n**R1:** Thanks for your suggestion, we promise to enhance the manuscript on the writing style and structural organization. In addition, we will ensure that all technical concepts presented in the paper are associated with clear and comprehensive definitions or derivations (e.g., Somers’ D algorithm).\n\n**W2:  The heavy reliance on the disentanglement abilities of β-VAE, which itself often presupposes some level of independence or weak dependence.**  \n\n**R2:** Thanks for your insightful comments. We wish to re-emphasize the key of our work: To move beyond the unrealistic independence assumption of traditional DRL, we propose a more **logical and practical DRL paradigm** that involves the bidirectional interrelations between attributes. In this framework, **$\\beta$-VAE branch is only employed to extract and initialize latent factors.** The weak dependence or partial independence among initial factors is acceptable, as their interrelations will be subsequently determined, overwritten, and updated by the proposed MLLM branch and DisGraph. In this way,  our method aligns more closely with real-world dynamics and offers broader applicability.\n\n**W3:  Reliance on pre-trained multimodal large language models might introduce biases**  \n\n**R3:** Our framework is based on the belief that MLLMs, including their future iterations, are sufficiently robust to comprehend logical rules of reality (e.g., aging brings wrinkles, sunrise brings light, etc.). These rules are represented as interrelations between entities. On this basis, we have corroborated the efficacy of MLLMs using ground truths to affirm their effectiveness in straightforward scenarios (see **Section 4.4 and Figure A1 in the attachment**). However, we totally agree with you that even the most powerful MLLMs (e.g., GPT-4o) can exhibit bias on limited pre-training data. To address this, we are exploring potential solutions from both within the MLLM (e.g., data pre-processing, bias-aware module, knowledge editing, and etc.) and from our end (e.g., weight redistribution, debiasing modelling, decoding strategy modification and etc.).\n\n**W4:  A deeper theoretical analysis of why and how the inclusion of interrelations leads to better disentanglement would be valuable**  \n\n**R4:** Thanks. It is really meaningful and interesting to discuss **\"what is a better disentanglement\"**. If it means the better performance on independently decomposing factors, then the inclusion of interrelations might not seem beneficial; however, if it refers to a better performance/practicality for real and complex scenarios, our disentanglement paradigm excels by statistically capturing the logical rules of real world.  Specifically, the inclusion of interrelations can be beneficial in model generalizability, counterfactual reasoning and practical usages. We will further conduct analysis and discussion on this point in the manuscript, through both theoretical analysis and experimental investigations.\n\n**Q1:  How are the generated node embeddings T generated by DisGraph associated with the losses?**  \n\n**R5:** Our apologies for the unclear descriptions. In brief, DisGraph generates the embedding matrix $T$ based on updated parameters to derive the latent variable $z$ through the aggregation of node embeddings. $z$ is subsequently decoded by $D_{rec}$ to reconstruct the image within the bounds of our loss functions. We welcome your reference to the **R1 for Reviewer d3xR** for a comprehensive discussion of the model's loss functions and workflow.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kxlpHsWHqx", "reviewer": "Reviewer_d3xR", "summary": "To achieve fine-grained, interpretable and unsupervised disentangled representation learning (DRL), this paper proposes a new framework by integrating $\\beta$-variational autoencoder ( $\\beta$-VAE), multimodal large language model (MLLM) and graph learning into a single pipeline. Experimental results show that the proposed framework can achieve a better trade-off in the capability of disentanglement and quality of reconstruction than the evaluated baselines under different datasets.", "questions": "The questions of the paper are listed as follows.   \n1. Is Equation (2) instead of Equation (1) the fundamental objective of vanilla VAE?   \n2. What are the differences among $p_{\\theta}(z|x)$, $p_{\\theta}(x|z)$ and $p_{\\theta}(z)$? Why can they share the same parameters $\\theta$?   \n3. Why can the MLLM have the capability to provide accurate attribute scores given the multimodal input provided by the framework?    \n4. If it has been proved that unsupervised DRL is impossible without extra prior, why does the proposed unsupervised framework work? What is the extra prior or inductive bias here?  \n5. Could you please give more explanation about why is the proposed algorithm to obtain the impact scores between the pair of attributes reasonable?   \n6. What are the potential use cases of the disentanglement capability from DRL?", "limitations": "The limitations are shown in the section of Weaknesses and Questions.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The strengths of the paper are listed as follows.    \n1. The paper is well-motivated. Figure 1 clearly illustrates the limitations of the existing works in DRL and the advantages of the proposed framework, which clearly shows the motivation of the paper.   \n2. This paper provides a thorough and insightful summarization of the related work in DRL, which highlights the contribution of the proposed framework.   \n3. It is great that the authors could provide detailed and diverse qualitative results in the section of experiments. It can give the readers a more clear view to understand the effect brought by the proposed framework.", "weaknesses": "The weaknesses of the paper are listed as follows.    \n1. It would be better if the authors could first formulate the problem as an optimization problem mathematically before introducing the details of the method to solve the problem. In the problem formulation, the input, output, constraints and objectives should be clearly defined.      \n2. In the proposed framework, the input to the decoder $D_{rec}$ is not from the normal distribution but from the variable extracted from DisGraph. It is a critical point that needs to be highlighted in Figure 2. However, it is missing in Figure 2.   \n3. Some technical details are not clear as listed below.   \nA. The input to the decoder $D_{rec}$ is not from the normal distribution but from the variable extracted from DisGraph. But how to make this process differentiable for end-to-end training is unclear.      \nB. It is not clear how to train the graph learner in an unsupervised way. What is the loss function? What is the dimension of the node feature? Moreover, how to update the adjacency matrix of the DisGraph given the updated weights of the GNN?    \nC. In Figure 2, it is not clear about the usage of a set of extra images input to the landmark function.   \n4. It would be better if the authors could use some metric to quantify the disentanglement capability of the DRL algorithms and show the quantitative results to evaluate the proposed framework and the baselines.   \n5. It would be better if the authors could provide a more intuitive explanation of the loss function they designed. What is the usage of each term within the loss function? Why can they address the limitations of the existing challenges of DRL?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We value your insightful feedback and will refine our manuscript accordingly. Here, we address each of your concerns.*\n\n**W1: Detailed definitions of the model**  \n\n**R1:** Thanks for your suggestions. We will detail the model's parameters, definitions, and optimization strategies in revision. Here, we present a brief overview of for clarity:\n\n| Component        | Input                                      | Output                                   |\n|------------------|--------------------------------------------|------------------------------------------|\n| Encoder $E_{sem}$ | Image $x \\in \\mathbb{R}^{n \\times n}$      | Disentangled latent variable $z_{dis} \\in \\mathbb{R}^{pre}$ |\n| DisGraph $G$      | Disentangled latent variable $z_{dis} \\in \\mathbb{R}^{pre}$ | Correlation-involved latent variable $z \\in \\mathbb{R}^{pre}$ |\n| Decoder $D_{rec}$ |Correlation-involved latent variable $z \\in \\mathbb{R}^{pre}$   | Reconstructed image $\\hat{x} \\in \\mathbb{R}^{n \\times n}$ |\n\nwhere $pre$ is the pre-defined dimensionality for all latent variables. During training, the optimizable parameters of the encoder $E_{sem}$, DisGraph $G$ and decoder $D_{rec}$ are denoted as $\\phi$, $\\gamma$ and $\\theta$, respectively. And the optimization objectives can be formulated as follows:\n\n$L_{gem}$ ($\\phi$, $\\gamma$, $\\theta$) = $D_{\\mathrm{KL}}(q_{\\phi}(x, z)$,$p_{\\gamma,\\theta}(x, z))$,  $L_{dis} = -D_{\\mathrm{KL}}\\left(q_{\\phi}(\\mathbf{z}|\\mathbf{x})  \\||  p_{\\theta}(\\mathbf{z})\\right) $\n\n$L_{adv} = L(D) = \\frac{1}{N} \\left[ \\sum\\limits_{i=0; (x_i, z_i) \\in E_\\phi}^{N} \\text{softplus}(-D(x_i, z_i)) + \\sum\\limits_{i=0; (x_i, z_i) \\in D_\\theta}^{N} \\text{softplus}(D(x_i, z_i)) \\right]$\n\n$L_{total} = \\lambda_{gem} L_{gem} + \\lambda_{dis} L_{dis} + \\lambda_{adv} L_{adv} $\n\nIn the revised total loss, as distinct from the manuscript's version, we partition the original $L_{dis}$ into two components: $L_{gem}$ for reconstruction and $L_{dis}$ for disentanglement.\n\n**W2: Missing details in Figure 2**  \n\n**R2:** Thanks for your thorough review. We have updated Figure 2 to highlight the input distribution of $D_{rec}$ accordingly.\n\n**W3_A: How to make it differentiable for end-to-end training in the model**  \n\n**R3:** Based on the latent variable $z_{dis}$ from encoder $E_{sem}$, we generate $n$ nodes, each representing one of the $n$ semantic attributes. In practice, each node mirrors $z_{dis}$'s dimensionality but isolates the $i$-th attribute by masking all dimensions except the $i$-th, thereby to learn the $i$-th attribute. These nodes, combined with interrelations from $P_{rel}$, form the DisGraph, which outputs the embedding matrix $T$. This matrix is utilized to calculate $z$ by averaging each volume, subsequently decoded by $D_{rec}$ to reconstruct the image with our loss functions. All aforementioned modules are differentiable. \n\n**W3_B: Unclearness of the Graph learner training**  \n\n**R4:** As described in Line 195, we follow the instructions by Liu et al. to unsupervisedly train the graph learner. Specifically, the adjacency matrix are updated within $E_{gnn}$, by the Structure Bootstrapping Mechanism and Multi-view Graph Contrastive Learning. \n\n**W3_C: Unclearness in Figure 2 about Landmark**  \n\n**R5:** In this work, we only apply off-the-shelf landmark models for data pre-processing, which does not require any extra data in the databases. We will accordingly clarify it in Figure 2.\n\n**W4: Disentanglement metrics**  \n\n**R6:** Our research emphasizes incorporating attribute interrelations, where enhancing disentanglement is not our primary objective (refer to **R4 to Reviewer qyA4**). However, to address your concern, we have conducted extra experiments as shown in **Figure A4 in the attachment**.\n\n**W5:  Explanations of the loss function**  \n\n**R7:** We have detailed our loss functions in **R1**. Regarding your final concern, our approach addresses DRL’s limitations by: 1) contributing to a more practical paradigm; 2) balancing reconstruction and disentanglement abilities by adjusting $\\lambda_{gem}$ and $\\lambda_{dis}$. \n\n**Q1:  Is Equation (2) the fundamental objective of vanilla VAE?**  \n\n**R8:** Equation (2) represents the objective function of the $\\beta$-VAE, while Equation (1) represents the likelihood estimation for vanilla VAE.\n\n**Q2:  Justifications of parameters $\\theta$**  \n\n**R9:** We adopt the original VAE notations: $p_{\\theta}(z)$ implies $z$ follows a standard normal distribution without parameters. In $p_{\\theta}(z|x)$ and $p_{\\theta}(x|z)$, $\\theta$ denotes decoder parameters. $p_{\\theta}(z|x)$ is used solely for theoretical derivation of the variational lower bound, with no parameter sharing in practice.\n\n**Q3:  Why can MLLMs provide accurate scores?**  \n\n**R10:** Theoretically, the impressive capabilities of MLLMs, especially in realistic AI generation, reveal that they can comprehend real-world concepts at a certain level, informed by in-depth studies (see Section 2.3); Experimentally, our evaluations of SOTA MLLMs confirm their reliability (see Section 4.4 and **Figure A1 in attachment**).\n\n**Q4:  Reliance on unsupervised DRL**  \n\n**R11:** Refer to the **R2 for Reviewer qyA4**, the DRL branch extracts and initializes latent factors. Although these factors exhibit partial independence, this limitation is acceptable as their interrelations will be refined and updated by the MLLM branch and DisGraph. Hence, our model does not require any external supervision or priors.\n\n**Q5: Explanations on interrelations determining**  \n\n**R12:** For determining the interrelationships between two attribute scores from MLLMs, we employ correlation analysis algorithms (e.g., Somers' D). Due to spatial constraints, please refer to **R2 for Reviewer SQvd** for the details of this part.\n\n**Q6: Potential applications**  \n\n**R13:** Our practical and relation-aware DRL framework can be applied in domains like AI-generated content, explainable AI, medical imaging, robotics, and autonomous vehicles.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wKvmz8c4kU", "reviewer": "Reviewer_SQvd", "summary": "Researchers introduced a bidirectional weighted graph-based framework to explore factorized attributes and their interrelations within complex data. They proposed a -VAE module for extracting initial factors and utilized a multimodal large language model (MLLM) to uncover latent correlations and update weighted edges. Integrating these modules enabled their model to achieve superior unsupervised disentanglement and reconstruction performance, inheriting interpretability and generalizability from MLLMs.", "questions": "1. How is the effectiveness of DisGraph ensured? What problems could arise if errors are introduced into DisGraph?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper introduces a graph-based approach to model interrelationships within complex data, aiming to integrate background knowledge into Deep Reinforcement Learning (DRL). I find this idea novel and intriguing.\n2. The paper presents a rigorous framework with clear exposition and straightforward methodology, making it accessible and easy to understand.", "weaknesses": "1. The paper lacks significant innovation at the neural network and algorithmic levels. While the introduction of DisGraph and its optimization methods is proposed to be effective, insufficient explanation is provided regarding why they work. Strengthening this aspect of the description would enhance the paper's persuasiveness.\n2. The paper could provide a brief explanation of some methods used, such as the Somers' D algorithm, even if included in an appendix. Currently, this aspect appears somewhat incomplete.\n3. An explanation of the update mechanism for the entire model at the end of Section 3 would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We greatly appreciate all of your valuable suggestions, which play a pivotal role in enhancing the quality of our paper. Below we address all your concerns.*\n\n**W1&W3: Detailed explanations of the optimization mechanism for the entire model**  \n\n**R1:** Thanks for your feedback. We promise to enhance our manuscript with more detailed explanations of the model's optimization mechanism. Below, we offer a concise explanation of the optimization mechanism for clarity:\n\nDuring the training, the optimizable parameters of the encoder $E_{sem}$, DisGraph $G$ and decoder $D_{rec}$ are denoted as $\\phi$, $\\gamma$ and $\\theta$, respectively. And the optimization objectives can be formulated as follows:\n\n$L_{gem}$ ($\\phi$, $\\gamma$, $\\theta$) = $D_{\\mathrm{KL}}(q_{\\phi}(x, z)$,$p_{\\gamma,\\theta}(x, z))$,  $L_{dis} = -D_{\\mathrm{KL}}\\left(q_{\\phi}(\\mathbf{z}|\\mathbf{x})  \\||  p_{\\theta}(\\mathbf{z})\\right) $\n\n$L_{total} = \\lambda_{gem} L_{gem} + \\lambda_{dis} L_{dis} + \\lambda_{adv} L_{adv} $\n\nwhere the specific optimization processes can be formulated as:\n\n$\\nabla_{\\theta} L_{gem}(\\phi,\\gamma,\\theta) \\overset{x=D_{\\theta}(z)}{=} -E_{z\\sim q(z)}\\nabla_{x}\\left[\\log\\left(\\frac{p_{\\theta,\\gamma}(x,z)}{ q_\\phi(x, z)}\\right)\\right]\\nabla_{\\theta}x$\n\n$\\nabla_{\\phi} L_{gem}(\\phi,\\gamma,\\theta) \\overset{z=E_{\\phi}(x)}{=} E_{x\\sim p(x)}\\nabla_{z}\\left[\\log\\left(\\frac{p_{\\theta,\\gamma}(x,z)}{ q_\\phi(x, z)}\\right)\\right]\\nabla_{\\phi}z$\n\n$\\nabla_{\\gamma} L_{gem}(\\phi,\\gamma,\\theta) \\overset{z=G_{\\gamma}(z_{dis})}{=} E_{x\\sim p(x)}\\nabla_{z}\\left[\\log\\left(\\frac{p_{\\theta,\\gamma}(x,z)}{ q_\\phi(x, z)}\\right)\\right]\\nabla_{\\gamma}z$\n\nFurthermore, the optimization objective of the discriminator can be expressed as:\n\n$L_{adv} = L(D) = \\frac{1}{N} \\left[ \\sum\\limits_{i=0; (x_i, z_i) \\in E_\\phi}^{N} \\text{softplus}(-D(x_i, z_i)) + \\sum\\limits_{i=0; (x_i, z_i) \\in D_\\theta}^{N} \\text{softplus}(D(x_i, z_i)) \\right]$\n\nwhere $\\(D^*(x,z)=\\log\\left(\\frac{p_{\\gamma, \\theta}(x,z)}{q_{\\phi}(x,z)}\\right)\\)$. The discriminator $D$ can be used to fit $D^*$ to achieve gradient estimation and complete the training. You may also check the **R1and R3 for Reviewer d3xR**, for insights into the framework's definitions and workflow.\n\n**W2: A brief explanation of some methods used, such as Somers' D algorithm, will be beneficial**  \n\n**R2:** Thanks for your suggestions. We will meticulously integrate comprehensive definitions and clarifications of all employed functions. Specifically, for the Somers' D algorithm you referenced, we provide the subsequent example to enhance your understanding. Suppose we have the sample dataset $S=\\{(1,2),(3,1),(2,3)\\}$:\n\n| **Variable** | Pairs                          | Value |\n|--------------|--------------------------------|-------|\n| $N_c$        | (1,2) vs (2,3)                 | 1     |\n| $N_d$        | (1,2) vs (3,1) and (2,3) vs (3,1) | 2     |\n| $T_y$        | None                           | 0     |\n\nSomers' D indicator can be calculated as follows:\n\n$ D =\\frac{N_c-N_d}{N_c+N_d+T_y} = \\frac{1 - 2}{1 + 2 + 0}  = -\\frac{1}{3} $\n\nThis obtains a value of approximately -0.33, signifying a negative correlation between variables $X$ and $Y$. This calculation demonstrates that the Somers' D metric is straightforward to calculate and is particularly applicable to ordinal variables. Furthermore, Somers' D is asymmetric and capable of distinguishing bidirectional relationships between variables. These characteristics make it highly suitable for integration into our model.\n\n**Q1: How is the effectiveness of DisGraph ensured? What problems could arise if errors are introduced into DisGraph?**  \n\n**R3:** To ensure the effectiveness of DisGraph, we perform an ablation experiment by disabling the DisGraph (see Figure 7 in Section 4.5). Since completely removing DisGraph is infeasible due to its role in embedding bidirectional weighted relations, we alternatively utilize an initial version of DisGraph without the updating module $E_{gnn}$. Refer to the top-left in Figure 7, where the model, using an initial Graph, exhibits weakened or inaccurate relational awareness (e.g., the relation between *\"Bald\"* and *\"Gender\"* weakens). This observation demonstrates the effectiveness of DisGraph. \n\nRegarding your second question, to our knowledge, the errors in interrelation identification from MLLMs may lead to counterfactual outcomes. For example, applied with a less capable MLLM that does not recognize the positive correlations between *\"banana\"* and *\"yellow\"* or *\"age\"* and *\"white hair\"*, the model may obtain *a red banana* or *a white hair baby* learned from the errors introduced into DisGraph (refer to the counterfactual results by a weaker MLLM in **Figure A3 of the attachment**). In our perspective, even advanced MLLMs (e.g., GPT-4o) can exhibit a certain bias. To enhance robustness of GEM, we are investigating potential improvements both within the MLLM (e.g., data pre-processing, bias-aware modules, and knowledge editing) and externally (e.g., weight redistribution, debiasing modelling, and modifications to decoding strategies).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lp96TKssBE", "reviewer": "Reviewer_Y3bu", "summary": "This paper introduces a bidirectional weighted graph-based framework, to learn factorized attributes and their interrelations within complex data. The authors proposed a β-VAE based module to extract factors as the initial nodes of the graph, and leverage the multimodal large language model (MLLM) to discover and rank latent correlations, thereby updating the weighted edges. Experiments demonstrate evidence of effective performance in disentanglement and reconstruction.", "questions": "* For the bidirectional weighted graph, it is unclear why this is a good choice. From my knowledge, a causal DAG would be a better choice to represent the interrelation of the entities, and there are many works (e.g., [1]) available deriving the closed-form distributions of the features under this assumption.\n\n[1] BayesDAG: Gradient-Based Posterior Inference for Causal Discovery. NIPS 2023.", "limitations": "The authors addressed some limitations of their works.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper integrates the VAE with multimodal large language model, which provides strong interpretability via the open-world knowledge of the large language models/multimodal models.", "weaknesses": "* As a generative model, β-VAE is extensively discussed with many useful variants. While this work heavily borrows the existing advances of  β-VAE, I would respect ground-breaking findings or improvements on this model (e.g., closed-form derivation of the feature distribution z after employing a graphical model). Otherwise speaking about absolute performance, I expect a standard diffusion model can easily outperform β-VAE and most of its variants (And authors should include comparison and discussion with diffusion models, even it is less likely that the proposed method can outperform, but insights on what β-VAE is lacking would be insightful, although it’s less meaningful in the era of diffusion model). \n* Related to above, the technical novelty and motivation of this work is limited since the current model can be easily understood as the combination of β-VAE and Large Multimodal models. There is less strong justification while this combination is optimal rather than other obvious choices (e.g., integrating a vision-language model, or diffusion+MLLM (since diffusion models are also trained with variational inference)). Author should either provide strong justification (better if theoretical) or more empirical validation showing the proposed framework is optimal than other obvious variants to enhance their motivation and novelty. \n* Line 148, it is obvious that z dos not follow multivariate normal, and author proposed Eq. 4 as the solution. Surprisingly, the author only showed the derivation of this equation (which is trivial in my view) in the appendix, with no justification why this is optimal. Why α=1? Or how strong is this assumption? Is the resulting objective still convex? What is the distribution of z under the new objective? Overall the authors imposed made many conditions in the ease of derivation with many justifications missing.\n* The authors only used GCN to learn the graph representation, which ruled out many other prominent choices (E.G., GAT, gatV2, GIN, GTN, GraphSAGE). authors should conduct thorough ablation study to select the optimal GNN architecture.\n* As an applied paper, it is less acceptable that codes are not released in the current phase. As many papers release their codes in anonymous GitHub repos and recently there are higher requirements of reproducibility and codeavilability in top venues. We also need to check the overall soundness of the empirical implementation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We greatly appreciate your insightful comments and commit to refining our manuscript based on your suggestions. Below, we address all your concerns.*\n\n**W1: Why not replace $\\beta$-VAE with advanced generative models**\n\n**R1:** Thanks for your insightful comments to help us improve our work. Firstly, we would like to discuss our standpoint on using other generative models, especially the Diffusion you mentioned. In our framework, the core of $\\beta$-VAE based encoder $E_{sem}$ is to extract and initialize disentangled latent factors. **This important perception block cannot be ideally replaced by Diffusion or other non-DRL generative models, as they do not produce an orthogonal and disentangled latent space**, and thus can not comprehend and extract disentangled factors (even though some of them are also trained with variational inference). \n\nContrary to replacing $\\beta$-VAE with the Diffusion (or other variants) as the semantic encoder, we tend to integrate it as the decoder $D_{rec}$.  This integration capitalizes on the superior generative abilities of the Diffusion model to improve the reconstruction fidelity of our framework. Concomitantly, the enhanced perceptual capacity of our model can, in turn, refine the generative process of the Diffusion, **thereby establishing a synergistic, mutually beneficial closed-loop**. We are currently implementing this feature. However, should time constraints arise, we promise to at least include an extra discussion section to cover these content.\n\n**W2: Lacking technical novelty and the justifications of technical combinations** \n\n**R2**: First, **we must emphasize that the technical novelty and motivation of this work** is: 1) We are first to leverage the commonsense reasoning of MLLMs to discover and rank the semantic interrelations for DRL; 2) We propose a novel and practical disentanglement framework built upon β-VAE and MLLMs, with a bidirectional graph architecture,  specifically designed to learn the interrelations between independent factors.\n\nThe naive combination of diffusion+MLLM could not achieve the same purpose of learning relation-aware representations, thereby facilitating practical and controllable disentanglement. This shortfall arises because Diffusion models are not designed for disentanglement, as they do not produce an orthogonal and disentangled latent space. Therefore, it is hard to directly compare our proposed method with the combinations you mentioned. \n\n**W3: Unclearness of equations and justifications in Section 3.1** \n\n**R3**: It's appreciated for your valuable comments. We apologize for not detailing the assumptions in the manuscript, and we promise to expand both the paper and the appendix to clarify these conditions and justifications explicitly.\n\nSpecifically, regarding the question \"Is the resulting objective still convex?\", we assumed that the model is upper bounded by the norm of its gradient, which satisfies the Polyak-Lojasiewicz (PL) condition, thus ensuring the suboptimality of the model. The PL condition is weaker than (strong) convexity, meaning it can be applied in a broader range of scenarios. Existing models such as L1-regularized linear regression and logistic regression have been proven to satisfy this condition, which supports our decision to use logistic regression for gradient fitting.\n\nThe assumption that $\\alpha$=1 can be achieved by controlling the ratio of real samples to latent space samples. In the preliminary stage, we have tested model performance across a spectrum of hyper-parameters $\\alpha$. Our empirical findings revealed a notable degradation in the quality of generated images when $\\alpha$ was below 0.2 or above 5.5. Conversely, the KID metric exhibited stable consistency at $\\alpha$=1 for most scenarios. \n\nAnd for your last concern: Given our utilization of the discriminator for gradient fitting, the distribution of $z$ under the new objective does not affect the final optimization procedure. Consequently, we did not delve into the new distribution properties of $z$. \n\n**W4: Authors should conduct thorough ablation study to select the optimal GNN architecture** \n\n**R4:** Thanks for your suggestions. We will consider potentially prominent graph choices with experimental proof, provided they can be optimized within our framework in an unsupervised manner.\n\n**W5: Authors should release their code**\n\n**R5**: Sure, we have released our project and sent the anonymous GitHub link to the AC as required. We highly welcome fellow researchers to follow and help improve our work.\n\n**Q1: For the bidirectional weighted graph, it is unclear why this is a good choice.**\n\n**R6**: Thank you for the feedback. As described in Section 2.2 and Figure 1, we analyzed the reasons for proposing DisGraph for embedding knowledge instead of using a causal graph: **1) the causal relationship is often overly simplistic, typically represented as binary and impractical.** However, in practice, paired variables commonly exhibit bidirectional influence, each impacting the other to varying degrees. For example, an increase in \"age\" can positively influence \"baldness\", whereas increased \"baldness\" does not significantly affect \"age\" in return;  2) Causal learning approaches are designed to model an event based on causal inference. **However, our setting, akin to most DRL approaches, is to model the scenario within the observed image. When considering the interrelations between observed vision attributes, the binary causal relation are inadequate.**  Furthermore, we conduct the comparison results with a causal model DEAR in Section 4.1, to show the experimental superiority of our choice. Regarding your comments, we recognize that the detailed explanations of our advantages over structured DRL (Hierarchical DRL, Causal DRL, etc.) were not sufficiently clarified. We commit to refine this aspect in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
*Dear reviewers,*

*First of all, we would like to thank all reviewers for their time and efforts in reviewing this paper. These insightful comments are really helpful in guiding to improve the manuscript.*

***We have made our efforts to meticulously address each concern raised by the reviewers. Please refer to separate responses for details. We have also attached a one-page PDF to support our responses.***

*We hope that the responses sufficiently address the reviewers' concerns. And we are open to further discussions if there be any unresolved issues.*

*Finally, we promise a careful revision of the manuscript according to these comments and discussions. We have released our code online (as attaching external links is not allowed, we have sent an anonymous GitHub link to the AC as required), and hope our work can provide valuable insights to the community.*

*Sincerely yours,*

*Authors*

Meta-review: Reviewers noted the novel approach combining β-VAE with MLLM and a bidirectional weighted graph for disentangled representation learning, as well as enhanced interpretability, as the paper's principal strengths. Initially, the reviewers raised a list of concerns, notably that the paper occasionally lacks adequate definitions or explanations for technical terms and concepts, lacks a deeper theoretical analysis of why and how the inclusion of the interrelations leads to better disentanglement, lacks detailed explanations,  clear definitions for constraints and objectives,  lacks metrics to quantify the disentanglement capability of the DRL algorithm,  an explanation for the designed loss function, contains unclearness in the graph learner training, and an unclear description of how the node embeddings are associated with the losses. Additionally, there is a lack of comparison with more advanced generative models, such as diffusion models.

The authors' rebuttal and the discussion that followed have elucidated many of these concerns, and the authors have promised to include the proposed improvements in the final version of the paper. Based on this, acceptance is recommended. The authors are asked to thoroughly follow up with the requested adjustments, with special attention to the following points: 1) explain why such generalized disentangled representations, despite not fulfilling the principal standard requirement of having independent factors of variations, may nevertheless be useful in practical situations; 2) add more evaluations with disentanglement metrics such as DCI, FactorVAE, and SAP;  3) it would be beneficial to provide more experimental evidence of the advantages of the paper's approach on another challenging dataset.
Decision: Accept (poster)