Graph-based Unsupervised Disentangled Representation Learning via Multimodal Large Language Models

Authors: baao  xie
qiuyu  chen
yunnan  wang
zequn  zhang
xin  jin
wenjun  zeng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 10:14:00

Abstract: Disentangled representation learning (DRL) aims to identify and decompose underlying factors behind observations, thus facilitating data perception and generation. However, current DRL approaches often rely on the unrealistic assumption that semantic factors are statistically independent. In reality, these factors may exhibit correlations, which off-the-shelf solutions have yet to properly address. To tackle this challenge, we introduce a bidirectional weighted graph-based framework, to learn factorized attributes and their interrelations within complex data. Specifically, we propose a $\beta$-VAE based module to extract factors as the initial nodes of the graph, and leverage the multimodal large language model (MLLM) to discover and rank latent correlations, thereby updating the weighted edges. By integrating these complementary modules, our model successfully achieves fine-grained, practical and unsupervised disentanglement. Experiments demonstrate our method's superior performance in disentanglement and reconstruction. Furthermore, the model inherits enhanced interpretability and generalizability from MLLMs.

Main paper: 
 introduction: 
 Disentangled representation learning (DRL) is a major goal of artificial intelligence (AI), acclaimed for its enhancement of model robustness, interpretability, and generalizability. Essentially, DRL methods imitate the understanding processes of biological intelligence, wherein comprehension of real-world is achieved by separating observations into distinct factors [1] . In this form, specific attributes (e.g., object color, shape, and size) exhibit exclusive sensitivity to the changes of specific factors. Learning of such disentangled representations is of great importance across various domains, e.g., computer vision [2, 3, 4, 5] , natural language processing [6, 7, 8] , and AI generated content [9, 10, 11] . In the current phase, unsupervised DRL methods primarily utilize the Variational Autoencoder (VAE) framework [12] , a probabilistic model learning representations through a regularization term. This term involves the Kullback-Leibler divergence between the posterior distribution of latent factors and a standard multivariate Gaussian prior, thereby encouraging the factorized representations. To strengthen disentanglement, co-current research [13, 14, 15, 16] focus on the optimization and refinement of the original VAE regularizers, resulting in the family of VAE-based DRL approaches.Despite the advanced results of the simple and synthetic datasets, VAE-based DRL methods still fall short in interpretability and robustness that are required for effective disentanglement in complex data [17] . This limitation mainly stems from the unrealistic assumption that underlying factors are countable, independent, and can be fully disentangled in an unsupervised manner (refer to the Fundamentally impossible to achieve [18] Rely on priors and supervisions top-left of Figure 1 ). In contrast, the real-world variables are pervasively correlated: red apples are more common than yellow ones; elderly people are more frequent with white hair and a receding hairline. Accordingly, an increasing number of recent studies [18, 19, 17, 20] showcases that purely unsupervised DRL is fundamentally impossible without extra priors and inductive biases.The struggle of typical DRL methods on the complex data returns us back to the essential goal of DRL, i.e., understanding the world as biological intelligence does. This cognitive process can be naturally segmented into three phases: attribute extraction, interrelation perception, and knowledge combination [21, 22] , where the latter two stages should not be neglected. From this perspective, several structured DRL approaches, typically known as Hierarchical DRL [23, 24, 11] and Causal DRL [19, 25, 26, 27] , have involved the correlations between attributes. However, these approaches usually require extra supervision, and their relations are invariably represented by binary and unidirectional fusion, thus limiting the model performance in practical scenarios (refer to the bottom-left of Figure 1 ). Inspired by the analysis above, we argue that an effective and practical disentanglement framework should meet the following criteria: (i) the framework should be fully unsupervised; (ii) the framework should be able to disentangle factors while concurrently discovering logical interrelations among them; (iii) the interrelations should be modeled as bidirectional, with corresponding impact scores assigned to each, thereby improving model performance in complex scenarios. On this basis, we propose a novel Graph-based disEntanglement framework with Multimodel large language models, dubbed GEM. Specifically, our model employs two complementary branches: a β-VAE based disentanglement branch for the attribute extraction, and a multimodal large language model (MLLM) based branch for the interrelation discovery. The relation-aware representations are further embedded into a disentangled bidirectional weighted graph (DisGraph), which presents distinct factors as nodes, interrelations as edges, and impact scores as weights. The parameters of the graph are dynamically updated and refined via a graph learner. The experimental results show that GEM achieves superior performance on fine-grained and relation-aware disentanglement, while preserving the reconstruction quality. Furthermore, the model is endowed with superior interpretability and generalizability that derived from MLLMs. All in all, our main contributions can be summarised as:• To our best knowledge, we are the first to leverage the commonsense reasoning of MLLMs to discover and rank the semantic interrelations from the perspective of DRL.• We propose a novel and practical disentanglement framework built upon β-VAE and MLLMs to learn the independent factors and their interrelations in an unsupervised way.• We introduce a bidirectional and self-driven graph architecture to encode the relation-aware representations, thus facilitating practical and controllable disentanglement.2 Related Work 
standard disentangled representation learning: 
 The definition of DRL is intuitively given by Bengio et al. [1] as a technique to separate semantic factors behind observational data. This approach assumes that individual data attributes are sensitive to changes in single latent factors, while not being affected by other factors. The disentanglement of attributes is believed helpful for downstream tasks, e.g., generative models [3, 28, 5, 29, 30] , medical imaging [31, 32, 33] , image editing [34, 35, 36, 37] , and 3D reconstruction [38, 39, 40] .Traditional DRL methods primarily utilizes the VAE framework, achieving a measure of disentanglement on static datasets. This framework has been further enhanced by extensive models such as β-VAE [13] , β-TCVAE [14] , DIP-VAE [4] , FactorVAE [41] , RF-VAE [42] , and α-TCVAE [16] through the optimizations of regularization terms. Despite the successes on simple and static datasets, standard DRL approaches still encounter challenges in complex data. It is mainly due to the flat and unrealistic assumption: data properties are independent and can be factorized into distinct factors [1, 19, 43, 44] . Locatello et al. [18] have proven that unsupervised DRL is fundamentally impossible without extra priors. Thus, subsequent studies have demonstrated that a practical DRL model with appropriate inductive biases can enhance the disentanglement in real scenes [45, 46, 47, 48] .In contrast to the flat and VAE-based DRL methods, recent research gradually realize that latent factors might naturally involve semantic interrelations, deriving to the branch of structured disentangled representation learning [17] . Within this domain, Hierarchical DRL and Causal DRL are mostly relevant to our work. Hierarchical DRL presumes that underlying factors have different levels of semantic abstraction, either dependent [49] or independent [23] across levels. While straightforward, Li et al. [23] propose a hierarchical VAE-based model to learn semantic representations. Furthermore, Singh et al. [11] introduce FineGAN, a three-tier hierarchical framework for controllable object generation. Li et al. [50] also propose a hierarchical DRL framework aimed at facilitating image-toimage translation. Differently, our framework aims to achieve fine-grained disentanglement, where the targeted attributes are always flat, e.g., the wrinkle, lipstick, and mustache of faces. Therefore, we rely on the flat representations, but place a strong emphasis on the mutual relations between attributes.Similarly, Causal DRL methods endeavor to capture the causal relations between disentangled factors. As the first, Yang et al. [19] propose CausalVAE to discover relations from the perspective of causality. Further, Shen et al. [27] propose a weakly supervised framework DEAR with the structured causal model (SCM) as prior. However in our view, current Causal DRL methods have at least three unpractical issues: (i) rely on various degrees of supervision; (ii) aim to model a specific event rather than a common scenario; (iii) the causal relationship is often overly simplistic, being impractically binary and unidirectional, i.e., paired variables A and B only have two possible causal relations: either A → B or A ← B (otherwise unrelated). In practical, it is common for paired variables to exhibit bidirectional influence, and the impact of such bidirectional relations should be properly ranked.Recent years have witnessed the remarkable advancements in Multimodal Large Language Model (MLLM) [51, 52, 53, 54] . Since the release of Generative Pre-trained Transformer (GPT) [55] , there has been a research trend over MLLMs regarding to its demonstrated potential in processing multimodal data [56, 57, 58] . As the variants of GPT-4, GPT-4 with Vision (GPT-4V) [59] and GPT-4 omni (GPT-4o) [60] enhance to process textual and visual data, enabling richer, context-aware interactions across a range of multimodalities. Concurrently, following works such as Gemini [56] , Claude [61] , NExT-GPTs [62] and GLM-4 [63] have strengthen the support to additional modalities.The powerful capacities of MLLMs gradually make researchers aware of its latent perceptual knowledge embedded within networks. Gandelsman et al. [57] investigate the way that CLIP encoder understands visual data, by decomposing representations into individual components. In addition, Basu et al. [64] propose a mechanistic localization approach to explore how the visual properties are encoded in MLLMs. However, to our best knowledge, there is limited exploration into leveraging the commonsense reasoning of MLLMs from the perspective of DRL. And we are the first to employ MLLMs to discover and rank interrelations between semantic factors in the DRL framework.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Limitations and future work are discussed in Appendix A.1, which will be included in the camera-ready version.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 To achieve fine-grained and relation-aware disentanglement, we propose GEM, a novel and practical framework that synergizes the strengths of DRL and MLLMs by a bidirectional weighted DisGraph. As depicted in Figure 2 , GEM is comprised of two complementary modules: a β-VAE based branch dedicated to extract attributes (Section 3.1), and a MLLM-based branch to discover and rank 3), which presents factors as nodes, interrelations as edges, and impact scores as weights.The fundamental objective of vanilla VAE is to approximate data distributions by employing a maximum likelihood estimation framework as outlined in Eq. 1:EQUATIONwhere the variational posterior distribution q ϕ (z|x) is utilized to represent the probability distribution of the latent variable z given the observation x. The key of Eq. 1 is maximizing the approximation log p θ (x) of the true posterior distribution p ϕ (z|x) and q ϕ (z|x).Specifically, the first term of Eq. 2 corresponds to the Kullback-Leibler (KL) divergence measuring the distance between distribution q ϕ (z|x) and p θ (z|x). The second term is denoted as the variational evidence lower bound (ELBO). Empirically, the maximization of ELBO is employed to provide a stringent tight lower bound for the original log-likelihood log p θ (x). ELBO can be reformulated as:EQUATIONwhere the initial term, i.e., conditional logarithmic likelihood E q ϕ (z|x) [log p θ (x|z)] is responsible for the reconstruction. Typically, the latent variable z is assumed to follow a standard Gaussian distribution N (0, 1) for p θ (z), so that the KL term actually imposes independent constraints on the representations. Furthermore, subsequent studies [13, 15, 65] highlight that a extra penalty coefficient prior to the KL term, denoted by β, can significantly strengthen disentanglement. When β is set to 1, the β-VAE reverts to the standard VAE framework. And an increase in β encourages more disentangled representations but harms the performance of reconstruction as a trade-off. As per the Information Bottleneck (IB) theory [15] , constraining the information input to DRL models (e.g., via β penalty coefficient) inherently enables them to identify and learn the most representative factors for successful reconstruction. For instance, when trained on the Shapes3D (a collection of synthetic objects) with a merely three-dimensional latent variable, the attribute determining branch tends to learn the most critical factors, observed to be "object color", "object shape", and "background shape". These attributes are organized in the three dimensions, ordered by their reconstruction contribution.Specifically, within the processes of this branch, the input image is firstly subjected to a pre-processing step utilizing landmark detection functions as instructed by [66] and [67] (see Figure 2 ). It serves as a regularization phase, to remain the key features through targeted cropping. Additional derivations of this process are documented in the appendix. Then, the pre-processed I 0 is fed into a β-VAE based branch, designed to disentangle factors associated with each dimension in the latent variable z ∈ Z. However, the input of decoder D rec is the relation-aware variable z rel = A T z from the DisGraph, rather than the z ∈ Z. It means the prior assumption of p θ (z) ∈ N (0, 1) is no longer hold. To address this issue, we reformulate the loss function in β-VAE as follows:EQUATION∇ θ L gem (ϕ, γ, θ)EQUATION∇ ϕ L gem (ϕ, γ, θ)EQUATION∇ γ L gem (ϕ, γ, θ)EQUATIONwhere the ϕ, θ and γ are the learnable parameters of E sem , D rec and DisGraph G, respectively. Let's say D(x, y) = log p θ (x,z) βq ϕ (x,z) , and the the gradients with respect to x and z can be obtained during backpropagation by the cross-entropy:EQUATIONwhere N bc and N m represent the number of samples and the posterior samples in a batch, respectively. Obviously, this loss resembles the adversarial loss utilized in Generative Adversarial Networks (GAN) [68] . Therefore, we employ the adversarial training strategy to optimize D(x, y). Combined with the disentanglement term from the original β-VAE indicated as L dis , the total loss for the attribute determining branch can be expressed as:L total = λ adv L adv + λ dis L dis + λ gem L gem (8)where the λ adv , λ dis and λ gem serve as hyperparameters to balance the disentanglement capability and reconstruction quality, with default values set to 0.8, 0.6 and 0.6, respectively. The detailed derivation process of the adversarial training strategy is provided in the supplementary material.Given a pre-processed image I 0 with n targeted attributes A = {1, 2, 3, ..., n} initialized by the β-VAE branch, our objective is to discover and rank the mutual relations for each pair within A. As represented by the brown blocks in Figure 2 , we employ MLLMs as a relation predictor P rel to discover and rank interrelations. Initially, the MLLM is required to score from 0 to 5 for each attribute, where 0 indicates the attribute's absence, and 5 denotes its highest expression. As shown in Figure 3 , the queries can be formulated as a question in natural language with the input image I 0 . Based on the attribute scores, we subsequently employ Somers' D algorithm [69] to rank the bidirectional impact scores of interrelations. For the attribute pair (A i ,A j ), we determine the number of concordant pairs N C and discordant pairs N D , as delineated by Kendall's Tau [70] algorithm. Subsequently, the impact score S ij within S = {1, 2, 3, ..., k} can be denoted as:EQUATIONFor the reversed relation of (A i , A j ), the impact score can be denoted as S ji or S ′ ij :EQUATION)where T i and T j is the number of ties only for the independent variable A i and A j , respectively. The calculated S and S ′ are used for initialization and refinement of DisGraph (see Section 3.3). As illustrated in Figure 4 , it is important to clarify that the primary goal of the MLLM branch in GEM is to discover interrelations, where the statistical relativity between two attributes is of primary concern, rather than the absolute scores for the individual attribute. For example, given a collection of facial images, it is acceptable if the scores of "age" and "bald" According to the definitions above, the model firstly constructs a sketched adjacency matrix A 0 ∈ R n×n upon the factors and relations initialized by the β-VAE branch and MLLM branch. Specifically, we treat the averaged impact scores of the first 1,500 images processed by MLLMs, as initial weights of relations. We further employ an unsupervised graph learner E gnn to dynamically refine the parameters of DisGraph by the structure bootstrapping mechanism [71] and multi-view graph contrastive learning [72] . The optimization function of E gnn can be formulated as:EQUATIONIt converts the sketched adjacency matrix A 0 into node embedding T via the GNN-based multilayer network, where h(l)w (•) is the embedding function with learnable parameters w of the l-th layer and T (l) is the output matrix. The augmented adjacency matrix A = A + I incorporates self-loops based on the initial matrix A 0 , and D is the degree matrix of A. Further, w (l) = Ω (l) ∈ R n×n denotes the parameter matrix of the l-th layer, with σ(•) as a non-linear function that enhances training stability. Figure 5 illustrates the comprehensive training algorithm of our model. The encoder processes input images and outputs the disentangled latent variable z, which subsequently initializes the embeddings of nodes in Dis-Graph. The adjacency matrix of DisGraph is calculated using Somer's D algorithm, which processes the attribute scores outputted by the MLLM. Following this initialization, a Graph Neural Network (GNN) refines the structure of DisGraph. The average of the feature matrix within DisGraph is then forwarded to the decoder to reconstruct images. Concurrently, the discriminator is trained to approximate the gradient of the loss function. Assuming that the model's performance is upper bounded by the norm of its gradient, which satisfies the Polyak-Lojasiewicz (PL) condition, this configuration ensures the suboptimality of the model.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 
experiments: 
 Datasets. We evaluate the GEM on two datasets: 1) CelebA [73] contains over 200,000 high-quality facial images. Each image is annotated with 40 binary attribute labels, making it a widely used benchmark for supervised DRL methodologies. Operating in an unsupervised manner, we do not utilize ground-truth labels from this dataset, yet we still conduct comparisons against the supervised approaches; 2) LSUN [74] consists of about one million images across various object categories such as cars, buildings, animals, etc. We select a typical subset from both scene categories and object categories, as bedroom and horse, respectively. We believe these two datasets are diverse enough to assess our method covering complex data of different object types.Implementation details. We implement GEM with PyTorch [75] . The landmark pre-processing settings follow the instructions of [66] and [67] . In addition, we employ the latest GPT-4o [60] as the interrelation predictor. For every experiment, the latent dimension size is set to 6. Concentrating on the disentanglement capacity of the framework, all experimental images are resized to a resolution of 64×64 to minimize computational resources. For high-definition outcomes at 256×256, refer to Appendix A.7. All the experiments are processed using the Adam optimizer with a learning rate of 1e-4, and conducted on the Nvidia Tesla A100 GPUs, with a batch size of 32.Baselines for Comparison. We evaluate the GEM with state-of-the-art DRL methods on the disentanglement capacity, reconstruction quality, and computational efficiency. The comparison encompasses supervised and unsupervised models, including standard VAE [12] , β-VAE [13] , β-TCVAE [14] , FactorVAE [41] and DEAR [27] . All baselines are trained using the complete CelebA dataset under the configurations previously specified.To evaluate the GEM's effectiveness of relation-aware and fine-grained disentanglement, we perform qualitative analyses with FactorVAE [41] and DEAR [27] . The experiments are conducted on CelebA, a standard benchmark that has been previously validated as compatible with these methods. We select the six fine-grained facial attributes from the database including Bangs, Bald, Gender, Beard, Blond, and Makeup. The disentanglement results are represented by traversals across various latent dimensions, where each dimension corresponds to distinct attributes. As illustrated in Figure 6 , GEM effectively achieves fine-grained and relation-aware disentanglement, via the integration of DRL and MLLMs. The interrelations determined by MLLMs are depicted as a heatmap in the bottom-right of Figure 6 , where deeper colors reflect stronger relations. Since DisGraph is bidirectional, the impact scores for bidirectional relations between a pair of attributes may vary, resulting in an asymmetric matrix. Specifically, in the first row of GEM's result, a person with heavier Bangs is less likely to be Bald, and the hair tends to be Blond, which is considered logical by MLLMs. Furthermore, as shown in the second and third rows, males (Gender) are more likely to be Bald and less likely to have Blond hair. The attribute Makeup is considered as relatively independent, with lower impacts scores among other attributes.In comparison, DEAR demonstrates limitations in learning specific attributes such as Bald (second row) and Gender (third row), while the relations between attributes appear to be tenuous. To our knowledge, this underperformance may stem from the stringent nature of causal relations, which are single-directional and heavily rely on the quality of prior. For FactorVAE, since it is a flat DRL framework, we employ the same causal relations used in DEAR to make it relation-aware. As shown in Figure 4 , GEM surpasses FactorVAE in both attribute disentanglement and relation discovery, which indicates the importance of specially-designed modules within our framework.Table 1 reports the results of Frechet Inception Distance (FID) [68] and Kernal Inception Distance (KID) [68] scores to verify the quality of reconstructed images. To ensure statistical significance, each comparison model undergoes three rounds of evaluations in the same configuration. The results indicate that GEM outperforms both typical unsupervised (VAE, β-VAE, β-TCVAE, FactorVAE) and supervised approaches (DEAR) in terms of reconstruction quality. To our understanding, this superior performance is attributed to the specialized training strategy implemented in the framework. Table 1 : Quantitative comparison results with typical DRL approaches in FID and KID. As shown in Table 1 , GEM surpasses baseline models in reconstruction quality on the datasets of CelebA, LSUN-horse, and LSUN-bedroom. However, the use of the disentanglement coefficient in the β-VAE branch leads to an inevitable trade-off in reconstruction quality, making the model less comparable to the models focused on generation quality (e.g., GAN and Diffusion [76] ). Therefore, the integration with leading generative models can be a direction for our future work. For additional comparison results, please refer to Appendix A.1. Furthermore, we evaluate four relation-aware models: FactorVAE, DEAR, GEM (Single), and GEM (Full), on quantitative comparisons of computational resources. Notably, GEM (Single) is the variant of GEM that incorporates single attribute determination branch (we only provide the initial relations to make it relation-aware). Table 2 shows that GEM outperforms DEAR and is comparable to FactorVAE on computational efficiency. This is mainly attributed to FactorVAE's utilization of a simple convolutional encoder, whereas GEM employs a β-VAE based encoder to strengthen disentanglement. In addition, the efficiency of full GEM is slightly inferior to GEM (Single), due to the extra modules for relation discovery and refinement.Method CelebA LSUN-horse LSUN-bedroom FID ↓ KID ×10 3 ↓ FID ↓ KID ×10 3 ↓ FID ↓ KID ×10 3 ↓ VAE [As a by-product, GEM inherits the interpretability and generalizability of MLLMs. Theoretically, owing to the commonsense reasoning faculties of MLLMs, our model can be generalized to discover any attributes and interrelations across various real-world objects and scenes. To demonstrate the robustness and generalizability of GEM, we perform extra experiments on more complex scenes in LSUN, specifically targeting the typical object subset LSUN-horse and the scene subset LSUN-bedroom. Furthermore, we test the attributes beyond the 40 specified in CelebA, collectively showcasing the model's superiority. To highlight the characteristics of bidirectional weighted DisGragh, we intentionally select paired fine-grained attributes exhibiting inconsistent bidirectional relations. As depicted in Figure 7 , GEM successfully achieve fine-grained disentanglement on complex scenes, while identifying bidirectional and weighted relations among attributes. Furthermore, the artifacts observed in the results of LSUN datasets are mainly due to the datasets' clutter (evidenced by the increase of FID and KID scores in Table 1 ). Nonetheless, despite the ambiguous and challenging nature of the data, GEM still obtain commendable disentanglement outcomes, affirming its robustness.Our model leverages the commonsense knowledge embedded in MLLMs to predict interrelations. This is predicated on the assumption that MLLMs, including their future iterations, are powerful and reliable enough to comprehend the physical rules of the real world (e.g., aging brings wrinkles, sunrise brings light, etc.). Therefore, before utilizing the interrelation discovery branch, it is imperative to evaluate the reliability of MLLMs. This evaluation guarantees that the identified interrelations and their associated impact scores are dependable and can be effectively applied to downstream modules.To this purpose, we evaluate three latest MLLMs including GPT-4o, GPT-4v and GLM-4-against the ground truth attributes of the CelebA dataset. The horizontal axis presents the targeted attributes selected from the CelebA, where the vertical axis presents the percentage of scoring accuracy. As reported in Figure 8 , GPT-4o outperforms other models on individual attribute scoring, achieving accuracy exceeding 90% for the majority of attributes. Specifically, it exhibits superior performance on attributes like Beard, Young, and Eyebrows, where other models yield significantly lower scores.In addition, GPT-4o achieves the highest average accuracy of 88.4% and the lowest zero-scoring rate at 0.25%, indicating a minimal rate of the meaningless predictions where all attributes are scored as zero. We conducted further evaluations on individual attributes, where GPT-4o also demonstrated superior performance (see Appendix A.7). Based on the evaluations, we employ GPT-4o as the interrelation predictor in the model.To As depicted in Figure 9 , replacing β-VAE encoder results in a declined disentanglement capability, albeit with an improvement in reconstruction quality. In addition, the removal of GNN-based graph learner prevents the parameter updating of DisGraph, leading to the inaccurate determination of relations (e.g., the relation between Bald and Gender weakens). It is worth noting that the removal of both graph learner and initialization process within the framework precludes the learning of interrelations. Furthermore, eliminating the adversarial training strategy in GEM and relying solely on the standard VAE loss function results in a significant decline in reconstruction quality. The aforementioned results highlight the effectiveness of each part of our framework. 
conclusion: 
 In this paper, we aim to explore the logical interrelations between semantic attributes within complex data, which is a critical challenge that existing DRL have yet to properly address. To this end, we introduce GEM, a β-VAE and MLLMs-based framework, designed to achieve fine-grained and relation-aware disentanglement. In this framework, DRL and MLLMs are integrated via a bidirectional and self-driven graph. Both qualitative and quantitative experiments demonstrate GEM's superior disentanglement and reconstruction capacities over typical DRL models. In addition, the model shows its enhanced interpretability and generalizability inherited from MLLMs.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The codes will be released in the camera-ready version, with detailed instructions for user to reproduce.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The experimental settings for our model, baseline, datasets and MLLMs can be found in Section 4 and Appendix A.2, A.4.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: The results of quantitative comparisons presented in this paper are accompanied by errors bar to achieve statistical significance (see Section 4.2).Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The computational efficiency of our framework is reported and compared with typical DRL models (see Section 4.2).Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The potential positive societal impacts are discussed in the paper and Appendix A.1, which will be included in the camera-ready version. To our best knowledge, this work does not cause any negative societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: Our framework rely on the VAE model, MLLMs and GNN model, so our safeguards are the same as theirs. To the best of our knowledge, these models poses no risks on misuse or dual-use.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All the public baselines and datasets used in this paper are properly credited (see Section 4).Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The assets of this project, primarily the codes, are well documented and will be released in the camera ready version.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The research presented in the paper exclusively utilizes public datasets and does not involve any crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Discussion Disentanglement Ability. As shown in Table 3 , we evaluate the disentanglement ability of our model (DRL branch only) against typical DRL models. Our model outperforms them across numerous metrics. However, it is really meaningful and interesting to discuss "what is a better disentanglement". If it means the better performance on independently decomposing factors, then the inclusion of interrelations might not seem beneficial; however, if it refers to a better performance/practicality for real and complex scenarios, our disentanglement paradigm excels by statistically capturing the logical rules of real world. Specifically, the inclusion of interrelations can be beneficial in model generalizability, counterfactual reasoning and practical usages.Trade-off between Quality and Interpretability. Even though our model achieved superior performance among DRL approaches, an inevitable trade-off between reconstruction and disentanglement remains, resulting in decreased reconstruction quality compared to the models focused on generation quality such as GANs and Diffusions (see Table 4 ). Since our model is oriented towards interpretability, we consider this trade-off acceptable. However, it is insightful to leverage the advantages of both DRL and non-DRL models within a mutually beneficial closed-loop architecture, and we will make efforts to improve our work in this direction.Current Limitations. Compared to existing DRL approaches, GEM emphasizes discovering underlying interrelations between attributes. This logical and effective framework can benefit a wide range of downstream tasks and practical applications such as controllable generation, medical image analysis, and automatic driving. However, there are still some limitations to this method. Firstly, the trade-off between reconstruction quality and disentanglement capacity, as a common challenge in the domain, is still not properly addressed in this work. To tackle it, we are currently investigating the integration of powerful generative models, e.g., diffusion models [76] and visual auto-regressive models [77] , into our framework. Secondly, the current implementation of GEM is not designed to work with 3D data, where 3D representations are much more complex. To understand our real world, it would be necessary to enhance the model with specific improvements to handle 3D scenes.CelebA. The Celebrity Faces Attributes (CelebA) dataset [73] is a widely-used large-scale face attributes dataset that contains more than 200,000 celebrity images, each annotated with 40 attributes.These annotations cover a wide range of facial attributes such as 'smiling', 'wearing Hat', 'young', 'wavy Hair', 'male', and 'mustache'. These attributes are labeled as present or absent in each image. This dataset is designed for various DRL and computer vision tasks, such as face recognition, face attribute disentanglement, and face editing. Commonly, we use employ the entirety of the CelebA dataset, which includes 162,770 images for training, 19,867 for validation, and 19,962 for testing.LSUN. The Large-scale Scene Understanding (LSUN) dataset [74] is a comprehensive collection for deep learning and computer vision, widely used in the domain of perceptual analysis and attribute disentanglement. This dataset consists of around one million labeled images for each of 10 scene categories and 20 object categories. The scene categories include diverse environments such as bedrooms, conference rooms, dining rooms, kitchens, living rooms, and etc. The object categories in LSUN include horse, car, church, etc. We select a typical subset, LSUN-horse, to demonstrate the model's generalizability. We believe these two datasets are diverse enough to verify our GEM.The β-VAE [13] is an extension of the standard VAE [12] , introducing an adjustable hyperparameter β prior to the KL term in vanilla VAE:EQUATION) where the penalty coefficient β balances the disentanglement capacity and reconstruction quality. When β > 1, this penalty increases the emphasis on learning disentangled representations in the latent space. However, increasing β can cause a trade-off on the reconstruction quality.The Total Correlation beta-VAE (β-TCVAE) [14] builds upon the β-VAE to further enhance the disentanglement of latent representations. It achieves this by the reduction of the total correlation (TC) term, extracted from the KL divergence term:EQUATION)where j represents the dimension of latent code z. The penalty coefficient β is selectively applied to the second term, i.e. TC term, on the right side of the loss function. It aims to make latent variables statistically independent of each other, therefore enhancing disentanglement. We include β-TCVAE in the quantitative comparisons, following the official implementation.The FactorVAE [41] is a typical variant of VAE, which employs a discriminator network in an adversarial manner to accurately estimate and minimize the total correlation term in the loss function. This adversarial strategy further enforces the factorization of the latent space, leading to improved disentanglement. We include β-VAE, β-TCVAE, FactorVAE in the quantitative comparisons following their official implementation. We evaluate the reconstruction quality by employing the Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) metrics for comparative assessment.DEAR [27] is the weakly supervised structural disengagement framework. It facilitates causal representation learning by adopting a structural causal model (SCM) as the prior distribution. This SCM prior is supervised by the information on the ground-truth factors and their underlying causal structure from the database. We integrate DEAR into the quantitative and qualitative comparisons, training it with the annotations provided by the CelebA. All in all, we evaluate our unsupervised framework against both unsupervised (β-VAE, β-TCVAE, FactorVAE) and supervised (DEAR) DRL methods, thereby rigorously evaluating the capabilities in reconstruction and disentanglement.Landmark detection are algorithm and technique utilized to detect and track specific key points in the image or video, encompassing a wide range of applications across various fields such as computer vision, robotics, and geospatial analysis. In this work, we employ landmark detection as a pre-processing method to extract the key points of the main object, thus removing the redundant parts in the image through cropping and resizing.Figure 10 : 68-points landmark pre-processing for data from the CelebA.We introduce the pre-processing phase for CelebA, where 68 landmark points are identified and extracted commonly. As shown in Figure 10 , points 1 to 17 present the jawline, points 18 to 27 for the eyebrows, points 28 to 36 for the nose, points 37-48 for the eyes, and points 49-68 identify the lips. Figure 11 illustrates some results of the pre-processing block. The normalized images benefit subsequent factor disentanglement and interrelation discovery.As described in Section 3.1, the prior assumption that p θ (z) ∼ N (0, 1) no longer holds, we reformulate the β-VAE loss as:EQUATIONSubsequently, the gradient of L with respect to θ can be derived as:EQUATIONwhere the first term on the right side needs to be approximated by a neural network due to the altered assumption. Therefore, we define:EQUATIONthenEQUATIONGiven the controllable proportion of the input images, denoted by α ∈ [0, 1], we define:EQUATIONthenEQUATIONwhere β ∈ (1, +∞) is proposed to enhance the disentanglement ability of β-VAE. We subsequently define the variable k:EQUATIONGiven that the solution for k is determined to be 1, under the condition that α = β = 1 (otherwise ∀(x, z) D(x, z) = 0 which is obviously impossible), it is logically imperative to require distribution p(x, z|(x, z) ∈ E ϕ ) and p(x, y|(x, z) ∈ D θ to closely approximate p(x, z). On this basis, we found:EQUATIONwhere if α = 1, we attain the optimization objective of p((x, y) ∈ D θ ) = p((x, y) ∈ E ϕ ) = 1 Nm . To this end, we can finally use the cross-entropy algorithm to optimize D:EQUATIONwhere sof tplus(x) = ln (1 + e x ) is a smooth activation function. N bc and N m represent the number of samples and posterior samples in a batch. The additional adversarial loss ensures the maintenance of reconstruction quality, tending to diminish as the capacity for disentanglement increases.In the interrelation discovery branch, we propose the Somers' Delta (Somers' D) [69] algorithm to determine and rank the bidirectional relations among the attributes extracted by the β-VAE branch. Somers' D is a statistical measure used to assess the strength and direction of the relation between variables. It is a nonparametric measure that can be considered a measure of rank correlation, similar to Kendall's tau [70] , but with a focus on asymmetric relations.Specifically, the calculation of impact scores based on Somers' D is upon the number of concordant pairs (C) and discordant pairs (D). The formula of Somers' D on variable Y and X can be given as:EQUATIONwhere T x is the number of ties only for the independent variable X, and T y is the number of ties only for the independent variable Y . The obtained attribute scores and interrelations from MLLMs are evaluated. For example, suppose we have the sample dataset S = (1,2), (3,1), (2,3):Variable Pairs Value N c (1,2) vs (2,3) 1 N d(1,2) vs (3,1) and (2,3) vs (3,1) 2 N y None 0Then the Somers' D indicator can be calculated as follows:EQUATIONThis obtains a value of approximately -0.33, signifying a negative correlation between variables X and Y . This calculation demonstrates that the Somers' D metric is straightforward to calculate and Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: To our best knowledge, this study does not involve any ethical issues.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: Reviewers noted the novel approach combining β-VAE with MLLM and a bidirectional weighted graph for disentangled representation learning, as well as enhanced interpretability, as the paper's principal strengths. Initially, the reviewers raised a list of concerns, notably that the paper occasionally lacks adequate definitions or explanations for technical terms and concepts, lacks a deeper theoretical analysis of why and how the inclusion of the interrelations leads to better disentanglement, lacks detailed explanations,  clear definitions for constraints and objectives,  lacks metrics to quantify the disentanglement capability of the DRL algorithm,  an explanation for the designed loss function, contains unclearness in the graph learner training, and an unclear description of how the node embeddings are associated with the losses. Additionally, there is a lack of comparison with more advanced generative models, such as diffusion models.

The authors' rebuttal and the discussion that followed have elucidated many of these concerns, and the authors have promised to include the proposed improvements in the final version of the paper. Based on this, acceptance is recommended. The authors are asked to thoroughly follow up with the requested adjustments, with special attention to the following points: 1) explain why such generalized disentangled representations, despite not fulfilling the principal standard requirement of having independent factors of variations, may nevertheless be useful in practical situations; 2) add more evaluations with disentanglement metrics such as DCI, FactorVAE, and SAP;  3) it would be beneficial to provide more experimental evidence of the advantages of the paper's approach on another challenging dataset.

Reviews: review: paper_summary: paper_summary: Objective
The paper presents a novel framework for disentangled representation learning, which aims to identify and separate the underlying factors of variation in complex data. The primary goal is to enhance the interpretability and robustness of data perception and generation models.

Methodology
The proposed method integrates a β-VAE (Variational Autoencoder) with a bidirectional weighted graph framework and utilizes Multimodal Large Language Models (MLLM) to discover and rank potential relationships between factors. This integrated approach ensures fine-grained, practical unsupervised disentanglement.

Contributions
1. Innovative Framework: Combining β-VAE with MLLM and a bidirectional weighted graph for enhanced disentangled representation learning.
2. Fine-Grained Disentanglement: Achieves more detailed and practical separation of data factors.
3. Improved Interpretability: The model inherits the interpretability and generalization capabilities of MLLMs.
4. Robust Evaluation: Shows superior performance in terms of disentanglement and reconstruction quality on benchmark datasets.
strength_weakness: strength_weakness: 1. Dependence on MLLM: The framework heavily relies on Multimodal Large Language Models (GPT-4) to discover and rank relationships between factors. This dependence can be problematic if the MLLM is not sufficiently trained on relevant data or if it introduces biases present in its training corpus. Additionally, relying on a single score from MLLM may not be sufficiently convincing, making this step overly dependent on the accuracy and reliability of MLLM. Despite the evaluations in section 4.4, which assess the accuracy of the scores, having specific guidelines or principles for MLLM scoring would be more convincing than relying solely on a single score.

2. Lack of Detailed Attribution: While the paper introduces a novel framework, it lacks detailed explanations on how each latent representation specifically maps to distinct attributes. This can make it challenging to understand and interpret the exact role of each variable in practical applications.
questions: questions: 1. I did not fully understand how the paper finds the correspondence between the latent representations output by the Encoder and the semantic attributes. For example, in Figure 1, how do the latent representations obtained through the Encoder correspond to attributes such as hat, eyes, etc.? Could you provide a detailed explanation of this process?

2. In the step where MLLMs are used to evaluate attributes (Figure 3), is there a specific principle guiding the MLLM to score the attributes? For instance, what is the difference when the MLLM scores the same attribute as 2, 3, or 4? Additionally, what would the results be if other MLLMs mentioned in your related work section were used for scoring instead of relying solely on the GPT-4 series?
limitations: limitations: Yes
paper_summary: paper_summary: The paper presents a novel framework that integrates β-VAE with multimodal large language models within a graph structure, DisGraph, to enhance disentangled representation learning. This approach allows for effective handling of complex and interdependent data attributes in an unsupervised manner. The model dynamically updates relationships between attributes, improving disentanglement and interpretability compared to traditional methods. Extensive experiments demonstrate its superior performance in both disentanglement and reconstruction.
strength_weakness: strength_weakness: 1. The paper would benefit significantly from clearer writing and better organization. The paper occasionally uses technical terms and concepts without adequate definitions or explanations.
2. While the paper addresses the unrealistic assumption of statistical independence in many DRL methods, the solution proposed still relies heavily on the disentanglement abilities of β-VAE, which itself often presupposes some level of independence or weak dependence among latent variables.
3. Also, the reliance on pre-trained multimodal large language models might introduce biases from these models into the disentangled representations.
4. A deeper theoretical analysis of why and how the inclusion of interrelations leads to better disentanglement would be valuable.
questions: questions: 1. How are the generated node embeddings T generated by DisGraph associated with the losses introduced in Section 3.1?
limitations: limitations: Yes
paper_summary: paper_summary: To achieve fine-grained, interpretable and unsupervised disentangled representation learning (DRL), this paper proposes a new framework by integrating $\beta$-variational autoencoder ( $\beta$-VAE), multimodal large language model (MLLM) and graph learning into a single pipeline. Experimental results show that the proposed framework can achieve a better trade-off in the capability of disentanglement and quality of reconstruction than the evaluated baselines under different datasets.
strength_weakness: strength_weakness: The weaknesses of the paper are listed as follows.    
1. It would be better if the authors could first formulate the problem as an optimization problem mathematically before introducing the details of the method to solve the problem. In the problem formulation, the input, output, constraints and objectives should be clearly defined.      
2. In the proposed framework, the input to the decoder $D_{rec}$ is not from the normal distribution but from the variable extracted from DisGraph. It is a critical point that needs to be highlighted in Figure 2. However, it is missing in Figure 2.   
3. Some technical details are not clear as listed below.   
A. The input to the decoder $D_{rec}$ is not from the normal distribution but from the variable extracted from DisGraph. But how to make this process differentiable for end-to-end training is unclear.      
B. It is not clear how to train the graph learner in an unsupervised way. What is the loss function? What is the dimension of the node feature? Moreover, how to update the adjacency matrix of the DisGraph given the updated weights of the GNN?    
C. In Figure 2, it is not clear about the usage of a set of extra images input to the landmark function.   
4. It would be better if the authors could use some metric to quantify the disentanglement capability of the DRL algorithms and show the quantitative results to evaluate the proposed framework and the baselines.   
5. It would be better if the authors could provide a more intuitive explanation of the loss function they designed. What is the usage of each term within the loss function? Why can they address the limitations of the existing challenges of DRL?
questions: questions: The questions of the paper are listed as follows.   
1. Is Equation (2) instead of Equation (1) the fundamental objective of vanilla VAE?   
2. What are the differences among $p_{\theta}(z|x)$, $p_{\theta}(x|z)$ and $p_{\theta}(z)$? Why can they share the same parameters $\theta$?   
3. Why can the MLLM have the capability to provide accurate attribute scores given the multimodal input provided by the framework?    
4. If it has been proved that unsupervised DRL is impossible without extra prior, why does the proposed unsupervised framework work? What is the extra prior or inductive bias here?  
5. Could you please give more explanation about why is the proposed algorithm to obtain the impact scores between the pair of attributes reasonable?   
6. What are the potential use cases of the disentanglement capability from DRL?
limitations: limitations: The limitations are shown in the section of Weaknesses and Questions.
paper_summary: paper_summary: Researchers introduced a bidirectional weighted graph-based framework to explore factorized attributes and their interrelations within complex data. They proposed a -VAE module for extracting initial factors and utilized a multimodal large language model (MLLM) to uncover latent correlations and update weighted edges. Integrating these modules enabled their model to achieve superior unsupervised disentanglement and reconstruction performance, inheriting interpretability and generalizability from MLLMs.
strength_weakness: strength_weakness: 1. The paper lacks significant innovation at the neural network and algorithmic levels. While the introduction of DisGraph and its optimization methods is proposed to be effective, insufficient explanation is provided regarding why they work. Strengthening this aspect of the description would enhance the paper's persuasiveness.
2. The paper could provide a brief explanation of some methods used, such as the Somers' D algorithm, even if included in an appendix. Currently, this aspect appears somewhat incomplete.
3. An explanation of the update mechanism for the entire model at the end of Section 3 would be beneficial.
questions: questions: 1. How is the effectiveness of DisGraph ensured? What problems could arise if errors are introduced into DisGraph?
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces a bidirectional weighted graph-based framework, to learn factorized attributes and their interrelations within complex data. The authors proposed a β-VAE based module to extract factors as the initial nodes of the graph, and leverage the multimodal large language model (MLLM) to discover and rank latent correlations, thereby updating the weighted edges. Experiments demonstrate evidence of effective performance in disentanglement and reconstruction.
strength_weakness: strength_weakness: * As a generative model, β-VAE is extensively discussed with many useful variants. While this work heavily borrows the existing advances of  β-VAE, I would respect ground-breaking findings or improvements on this model (e.g., closed-form derivation of the feature distribution z after employing a graphical model). Otherwise speaking about absolute performance, I expect a standard diffusion model can easily outperform β-VAE and most of its variants (And authors should include comparison and discussion with diffusion models, even it is less likely that the proposed method can outperform, but insights on what β-VAE is lacking would be insightful, although it’s less meaningful in the era of diffusion model). 
* Related to above, the technical novelty and motivation of this work is limited since the current model can be easily understood as the combination of β-VAE and Large Multimodal models. There is less strong justification while this combination is optimal rather than other obvious choices (e.g., integrating a vision-language model, or diffusion+MLLM (since diffusion models are also trained with variational inference)). Author should either provide strong justification (better if theoretical) or more empirical validation showing the proposed framework is optimal than other obvious variants to enhance their motivation and novelty. 
* Line 148, it is obvious that z dos not follow multivariate normal, and author proposed Eq. 4 as the solution. Surprisingly, the author only showed the derivation of this equation (which is trivial in my view) in the appendix, with no justification why this is optimal. Why α=1? Or how strong is this assumption? Is the resulting objective still convex? What is the distribution of z under the new objective? Overall the authors imposed made many conditions in the ease of derivation with many justifications missing.
* The authors only used GCN to learn the graph representation, which ruled out many other prominent choices (E.G., GAT, gatV2, GIN, GTN, GraphSAGE). authors should conduct thorough ablation study to select the optimal GNN architecture.
* As an applied paper, it is less acceptable that codes are not released in the current phase. As many papers release their codes in anonymous GitHub repos and recently there are higher requirements of reproducibility and codeavilability in top venues. We also need to check the overall soundness of the empirical implementation.
questions: questions: * For the bidirectional weighted graph, it is unclear why this is a good choice. From my knowledge, a causal DAG would be a better choice to represent the interrelation of the entities, and there are many works (e.g., [1]) available deriving the closed-form distributions of the features under this assumption.

[1] BayesDAG: Gradient-Based Posterior Inference for Causal Discovery. NIPS 2023.
limitations: limitations: The authors addressed some limitations of their works.

Comments: title: Official Comment by Reviewer qyA4
Hope for your further comments
comment: Thanks for the detailed rebuttal from the authors. I think it solved most of my concerns. Given the feedback, I think a thorough revision should be made for the final publication. So please remember to make the revisions you promised for the final version of the paper. I have raised my score for this paper.
I thank the authors for their responses. I will raise my score.
In general, I am satistifed with the author feedback. As reflected in my score, the paper has its merits, and is above the acceptance bar in my opinion.
Thank you for your continued efforts. We have provided comprehensive rebuttals and tried to address the concerns raised in your reviews. Please take the time to review if possible, if you have any further questions or require additional clarification, please let us know and we welcome discussions in any format . Thanks again.
I appreciate the authors' quick and convincing response. The empirical evaluation looks much more comprehensive now, which demonstrates the optimality of the proposed combinations and the sacrifice (i.e., compared to diffusion models) we need to make for the interpretability. My concerns are well-addressed and I will increase my scores.
Thanks for your quick and patient feedback. We apologize for our misunderstandings regarding your comments and hope to address your concerns effectively in this renewed discussion.

**1**: We would like to address your concerns of the optimal combination from both experimental and theoretical perspectives: 1) **Experimentally**, we have conducted extra experiments focusing on **relation-aware disentanglement** among typical DRL+MLLM/VLM combinations. The results are illustrated in the attached PDF accordingly:

**Table 1**

| Combinations| Results| Brief Comments| 
| --- | --- | --- |
| $\beta$-VAE + GLM-4v | **Figure A3** (in attachment) | Producing counterfactual and anomalous results.|
| FactorVAE + GPT-4o| **Figure A5** (in attachment)  | Suffering from inferior reconstruction quality.|
| $\beta$-TC-VAE+ GPT-4o| **Figure A5** (in attachment)  | Failing to learn attributes effectively.|
| $\beta$-VAE+ GPT-4o| **Figure A5** (in attachment)  | Current combination with most stable and effective performance|
|The optimal selection of MLLM| **Figure A1** (in attachment) |GPT-4o outperforms others on attribute identification (also see Figure 6 in paper)|

The visualization results demonstrate that the current combination yields the most stable and effective outcomes. We have also performed comparative experiments on **generation quality metrics** to evaluate the model performance (please refer to Table 2). 2) **Theoretically**, to our knowledge, these results can be attributed to the straightforward and effective design of the $\beta$-VAE. Since our model just leverages the DRL model to extract and initialize semantic factors, extensional designs can lead to instability (e.g., extra MLP classifier and discriminator in FactorVAE; the inflexible penalty in TC-$\beta$-VAE; the embedding codebook in VQ-VAE, etc.). These components, while potentially beneficial in certain contexts, may introduce unnecessary complexity and reduce the stability in our setting for relation-aware disentanglement.

*The exclusion of non-DRL combinations (Diffusion, GAN, etc. + MLLMs) from **disentanglement capability comparisons**, is due to **their inability to generate orthogonal and disentangled latent spaces,** as detailed in **R1**. However, the comparisons of generation quality are conducted involving DRL and non-DRL models, for evaluating the model trade-off between generation and interpretability (see next response).*

**2**. According to your response, we have included Diffusion and GAN models as the baselines in the generation quality experiments as shown in Table 2.

**Table 2**

| Model| CelebA (64x64)|  | CelebA (256x256) | | 
| --- | --- | --- | --- | --- | 
| | FID $\downarrow$| KID x $10^3$ $\downarrow$ | FID $\downarrow$| KID x $10^3$ $\downarrow$ |
| **FactorVAE + GPT-4o**| 112.08 | 101.54 | 126.58 | 130.12 |  
| **$\beta$-TC-VAE + GPT-4o** | 68.17 | 62.90 | 91.45 | 87.22 |  
| **GEM (Ours)** | 46.05 | 48.32 | 50.93 | 51.01 |
| **Vanilla VAE** | 53.39 | 51.48 |56.82 | 61.26 |
| **StyleGAN2** *(40k steps)*| 12.94| 9.20| 18.02 | 19.55  | 
| **DDPM** *(Diffusion, $T$ = 1k)*| 8.56| **6.56** | 15.93 | 10.01 | 
|**DDIM** *(Implicit Diffusion, $T$ = 1k)*| 10.04 | 8.15 | 16.24 |  13.62 | 
| **Stable Diffusion** *(fine-tuning)*| **7.72** | 7.22 | **10.63** | **9.17** |

*Due to time constraints, we present the results on the CelebA dataset, and we promise to provide comprehensive evaluations in the manuscript.*

Even though our model achieved superior performance among DRL approaches, an inevitable trade-off between reconstruction and disentanglement remains, resulting in decreased reconstruction quality compared to image generation models (GAN, Diffusion, etc.). Since our model is oriented towards interpretability, we consider this trade-off acceptable (see Lines 253-257 in the paper). However, it is insightful to leverage the advantages of both DRL and non-DRL models within a mutually beneficial closed-loop architecture (as detailed in **R1**), and we will make efforts to improve our work in this direction.

We are open to further discussions if you have any unresolved concerns. And we will conduct a comprehensive evaluation and analysis according to your concerns and discussion.
Thank you to the author for providing a thorough response to my question. I believe it has addressed my concerns regarding DisGraph. I will raise the score I have given and consider the paper to be an excellent one.
I thank the authors for their response. However, I think my messages were poorly delivered and many of my intended points were lost when the authors summarize my comments. Sorry to say but the authors are just reiterating their main contributions and strengths of their methods instead of utilizing this chance to address the key concerns. Some cases to illustrate:

1. For W2 I was asking why this combination is optimal to **all** non-trivial combinations. The proper way should be listing some potential non-trivial combinations and show (better with experiments) why these combinations would underperform. This would greatly enhance the motivations of this work. Sadly, the authors just restate their contributions and novelty and choose only one example (diffusion+MLLM) to illustrate, which is less convincing.

2. For W1, why not replace the $\beta$-VAE with advanced generative models is just one of my questions, while my major point is why diffusion-based models are included as a baseline. The authors have 7 days and I think it shouldn't be difficult to perform an image-generation experiment with a standard diffusion model. The author should also address this question on absolute generation performance, and how much the proposed method is worse than (if any) the standard diffusion model to sacrifice for the interpretability. These empirical evaluations should be performed to more holistically evaluate the proposed method.

I am open to challenges if I am wrong. But I cannot give a higher score based on the current version of the manuscript and authors' rebuttal. I think the authors should be candid when rephrasing the reviewers' comments, and make sure that every point raised is addressed properly.