Symmetry-induced Disentanglement on Graphs
Authors: Giangiacomo Mercatali
Andre Freitas
Vikas Garg
Venue: NeurIPS 2022 Accept (2022)
Keywords: Generative models
disentanglement
latent variable models
graph neural networks

Abstract: Learning disentangled representations is important for unraveling the underlying complex interactions between latent generative factors. Disentanglement has been formalized using a symmetry-centric notion for unstructured spaces, however, graphs have eluded a similarly rigorous treatment. We fill this gap with a new notion of conditional symmetry for disentanglement, and leverage tools from Lie algebras to encode graph properties into subgroups using suitable adaptations of generative models such as Variational Autoencoders. Unlike existing works on disentanglement, the proposed models segregate the latent space into uncoupled and entangled parts. Experiments on synthetic and real datasets suggest that these models can learn effective disengaged representations, and improve performance on downstream tasks such as few-shot classification and molecular generation.

Symmetry-induced Disentanglement on Graphs
Giangiacomo Mercatali" University of Manchester
André Freitas Idiap Research Institute University of Manchester
Vikas Garg Yai Ltd and Aalto University vgargacsail .mi..edt
Abstract
Learning disentangled representations important for unraveling the underlying complex interactions between latent generative factors. Disentanglement has been formalized using symmetry-centria notion for unstructured spaces, however, graphs have eluded similarly rigorous treatment. We fill this gap with new notion of conditional symmetry for disentanglement, and leverage tools from Lie algebras to encode graph properties into subgroups using suitable adaptations of generative models such as Variational Autoencoders. Unlike existing works on disentanglement, the proposed models segregate the latent space into uncoupled and entangled parts. Experiments on synthetic and real datasets suggest that these models can learn effective disengaged representations, and improve performance on downstream tasks such as few-shot classification and molecular generation.
Introduction
Disentanglement represents fundamental desideratum in learning with limited supervision because captures information about the salient (or explanatory) factors of variation in the data, and isolates information about each specific factor in only few dimensions, thus unraveling the interactions underlying complex data [2] [59]. Disentangled representations have often been deemed responsible for providing neural models with the ability improve performance on real-world downstream tasks 2 20 36 58 [5]. Empirically, they have been shown to enable, or facilitate, critical properties
such sample efficiency 21 and generalization 32
69
An interesting perspective on disentanglement has been recently proposed by Higgins et al. [20]. Specifically, they advocate viewing disentanglement decomposition of the latent space of em- beddings into subspaces, each of which can be transformed independently under the action of a single subgroup specific the subspace, without affecting others. These subgroups arise, in turn, from the decomposition of a symmetry group. This formalism bestows several benefits, e.g., (a) aligns with the idea that auxiliary information can be exploited for imposing structure the latent space (25] 40 and (b) leads to rigorous theoretical framework for analyzing disentangled
models 64
70 e.g., those based on Variational Autoencoders (VAEs) 27 50
While symmetry groups provide a suitable tool both for formalizing 20 and Iearning 57 64 70
disentangled representations in unstructured domains, these techniques have not been investigated in the context more general and complex data such as graphs. Furthermore, most works on
disentanglement for graphs [ 16 31
35 38, 65) not designed for generative settings, and the
existing deep generative models (DGMs) for graph disentanglement such as [10] [7] 56 do not consider group symmetries. Note that differently from other definitions of disentanglement 2 []]. leveraging group symmetries for disentanglement may unravel fundamental connections across several approaches based graph neural networks (GNNs), i.e., the state-of-the-art models for embedding graphs
63
first-name last-name @ postgrad. cta.eraa.mmaacceeera.....t
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

We, therefore, pursue two main goals in this work: (a) providing rigorous symmetry mameetr--based formalism for disentanglement on graphs, and 2) designing novel neural urchitectures that are able leverage symmetry groups to learn disengaged representations for graphs. However, accomplishing these objectives requires overcoming some challenges. In particular, graphs often abstract complex interactions, so the underlying latent space may not factorize completely into only disentangled subspaces. Thus, the symmetry-based formalism introduced 20] does not suffice for graphs. Therefore, we introduce new, more flexible notion of conditional disentanglementt advocating segregating the latent space into uncoupled and entangled parts. Translating this idea into an efficient algorithm requires further work, and we appeal to Lie algebra based parameterization to encode the graph properties into subgroups.
Contributions. Our contributions can be summarized as follows:
a novel notion of symmetry-induced conditional disentanglement (Section 3.1) that general- izes the previous definition from 20 (which we call unconditional disentanglement);
a parameterization centered on Lie algebra based on the intuition that each separable generative factor can correspond to Lie algebra coordinate element, and control a single graph property (Section 3.2
two algorithms for -ynduetr--Inuuedd Disentanglement under unconditional (SIDu) and conditional (SIDc) settings, both using two-level variational auto-encoding mechanism, namely, one level for the data and other for symmetry group representation (Section 4 and
The results demonstrate that the proposed models can outperform contemporary GNN baselines [38 65]; successfully learn to decouple the entangled part of latent space from the disentangled parts (Section 5. : and improve performance on downstream tasks such as few-shot classification, compression (Section 5.2) and molecular graph generation (Section 5.3)
systematic evaluation several disentanglement metrics [ 19 26 44
We begin with review of the relevant literature, and then proceed to the proposed framework.
2 Related work
Symmetry-based disentanglement (SBD). Initial works on SBD, such as Caselles- Caselles-Dupré et al.
(6). Quessard al. [48], propose reinforcement learning achieve irreducible representations of group elements through observation of action transitions. Painter et al. (44) introduce a VAE method that does not require labelled action-transition pairs, and demonstrate its validity within the environment from Caselles-Dupré et al. [6].
More recently, some algebraic approaches have been introduced. Zhu et al. [70] propose to decompose a Lie group into multiple subgroups, and enforce commutativity and independence between subgroups. Yang al. (64) impose cyclic group structure, encouraging the group be isomorphic to the ground truth. Bouchacourt et al. [4] invoke group representation theory for defining distributed latent operators that able learn disentangled representations. Tonnaer et al. [57] provide new metric for evaluation, and a VAE-base model for learning symmetry-baie based disentangled representations.
Some works leverage data manifolds for SBD. For example, Fumero et al. 12 view disentanglement as product low-dimensional sub-manifolds underlying the data space, such that each sub-man mannmanifold encodes an explainable data factor. Pfau et al. [46) propose a non-parametric algorithm that aims to discover a decomposition of the data manifold by investigating its holonomy group.
Other works explore SBD as well. For instance, Higgins et al. (2 demonstrate the importance of symmetry transformations and disentangled representation learning the context of neuroscience. Wang et al. [61] design a contrastive self-supervised learning method that iteratively updates partition, modeled as a symmetry group. until the considered factor is invariant.
To our knowledge, Ma et al. approach, which involves routing mechanism aimed at separating the underlying factors. More recently, Yang et al. 655 exploit an attention mechanism for GNNs, and provide two evaluation metrics for disentanglement graphs. Liu et al. [35] achieve disentanglement by encouraging
Disentangled GNNs.
38 provide the first graph disentanglement

---

independence in the latent space, while Bae and Jeon [ propose a method for disentangling multi- relational graphs motivated by an application on pedestrian trajectory prediction. Li al. [3] provides a contrastive framework for disentangling the latent factors in GNNs.
Differently from these methods, we provide formal definition of disentanglement for graphs, and leverage symmetry groups in a generative model setting.
Graph DGMs. Wu et
62 Yu al. 68] propose VAE-based Information Bottleneck methods for improving the performance of GNNs on classification tasks, by learning minimal sufficient representations. The work by Simonovsky and Komodakis (55] aims to generate molecular graphs using graph VAEs. Liu et al.
3 adapt VAEs for conditioning molecule generation on specific
properties. You et al. (67 propose a deep autoregressive model that learns to generate graphs by training representative set of graphs and decomposes the graph generation process into sequence of node and edge formations. Du et al. [T0] propose DGM for spatio-temporal graphs, and Guo et al. [18] investigate DGMs with spatial networks. Finally, Guo et al. []7 and Stoehr et al. [56] investigate disentangled representations with graph VAEs.
Proposed framework
In Section 3.1 we review the unconditional symmetry-induced disentanglement, and propose new definition of conditional disentanglement. In Section 3.2 we extend these notions to graphs. Finally,
in Section
3.3
we describe additional loss terms to encourage disentanglement.
3.1 Symmetry-based disentanglement
Definition (Unconditional symmetry-based disentanglement (20).. Consider generative process W - O mapping the world states W into observations (i.e. data) O, and an inference process h = 0 + Z mapping the data into latent vector space. By composing b and h, we obtain f= h b such that f W + Z. Given group actions on W, i.e., w G W + W and on Z. i..., 'z G Z Z, the function f is said to be equivariant between the actions on W and if the actions commute with f. We can express this equivariance mathematically g z fw) f(g "W w), Vg G, Yw E W. Assuming that the symmetry group G can be decomposed into subgroups G1 x... x Gn, a vector representation Z is disentangled with respect to this decomposition if all of the following conditions hold:
there is an action defined on the representation set G x Z + Z,
the map f W + Z is equivariant between the actions on W and Z, and
there is a decomposition Z = Zi x Zz... x Zn such that each Z; is affected only by G; and is invariant to (i.e., does not change due to) Gj for all j + i.
Definition I does not allow any part of the latent vector space to be entangled, which is likely to be restrictive for complex applications. Therefore, we propose conditional disentanglement.
Definition (Conditional symmetry- based equivariance). Consider a generative process b b= W E + O mapping the disentangle-abe world states W and non moo-diseetangled states E into observations (i.e. data) O, and an inference process = O - Z mapping the data into a latent vector space. By composing b and h. we obtain f bo h, which maps W x E to Z. Given group actions on W. namely, 'w : G x W - W and on Z, namely, .z G x -> Z, the function f is said to be conditionally equivariant between the actions on W and Z given E if the actions commute with f. That is, 9 z fw, e) f(g 'w w, e), Vi G. Vu W, Ve E E.
This definition of conditional equivariance naturally leads ment using a characterization similar to the three conditions for the unconditional case. Specifically, need to impose the condition that is conditionally equivariant between the actions of W and Z given E (instead of simply being equivariant between the actions of W and Z).
notion of conditional disentangle-
Note that while this definition results in a generalized framework for disentanglement, it does not directly translate into a learning algorithm. Toward that goal, we invoke tools from Lie algebra as described in the next section.
3

---

3.2 Disentanglement on graphs
Here, we aim to instantiate Definition D and 2 in the context of models that encode graph-structured data. Drawing inspiration from the previous work on Lic groups and algebras for symmetry-induce disentangled VAEs by Zhu et al. (70), we develop formalism for disentanglement on graphs induced by Lie groups
Lie groups and Lie algebras. A Lie group G is group of continuous symmetries 122 and associated with Lie algebra g. which the tangent space the identity element of G. We can thus parameterize a Lie algebra with a basis {A,) where every element in can be written as A = Ai4 ... Asty using coordinates t,. Elements of the Lie algebra can be mapped back into the Lie group with a matrix exponential map exp g + G.
We provide some intuition into how definition of disentanglement in a graph network, by associating each Lie algebra coordinate t; with a generative factor. A latent representation 2, obtained, e.g., from graph encoder, is disentangled with respect Lie group G, if change in the coordinate is associated with change in only the jth component of 2, i.e., only 2, In other words, disentanglement entails that the semantics (41, ...., (x) are equivariant with respect to the properties (ž1, .....,, , &x) that are being predicted.
connect the notion of Lie group G with the
Next, provide the formal definitions for unconditional and conditional graph disentanglement. The former expresses 2 as function of T without accounting for Z, whereas the latter expresses ă based on T after fixing an encoding Z.
Definition 3 (Lie-algebra based unconditional graph disentanglementt. The graph embedding Ž obtained by f(((T) is unconditionally disentangled with respect to the Lie group coordinates T = {41) if the following hold: (a) there is a group action 2 G x - 2 on 2, (b) the map f = (X)(A(T)) T - equivariant between actions on and 2, and (c) there is decomposition Ż = 2 x 8.... x 2g. where each coordinate t; affects only the corresponding component é,
Definition 4 (Lie-algebra based conditional graph disentanglement). The graph embedding 2 obtained by f(2(T, Z) is conditionally disentangled with respect to the Lie group coordinates T (41111 given Z if (a) there is group action :2 G x 2 Z on Z, (b) the map exp exp x + 2 is equivariant between actions on T and for any fixed Z, and (c) exp A(T) Z factorizes into the product IT :x,( x Z such that each component Ax)( is affected only by the corresponding coordinate & for i E
k} for any fixed Z.
3.3 Disentanglement constraints
It known that a symmetry group parameterization is usually not sufficient for group be decomposed into subgroups that are parameterized independently by single coordinate. Therefore, following Zhu et al. (70, we add two additional constraints for enforcing disentanglement. These constraints can be relaxed, and included in the loss function as regularization terms.
Commutative penalty. We would like to enforce coordinate t; to be identified by single group (t,Att and thus represent single property variation in the data &, Mathematically, one can state this desideratum as equivalence between the exponential map of the sum, and the product over exponential maps. [70] proved this to be achieved under commutativity over the Lie algebra basis.
Namely, if A,A, A,A, then exp
Eh 4AI)
I exp(t;A.).
Hessian penalty Furthermore, disentanglement can be encouraged using a Hessian penalty 45 based the fact the Hessian matrix with respect to disentangled representation is always (owing to the independence between different dimensions). Zhu et al. (70] adapt this penalty in Lie
(T)(T) where (T) = exp E t.A and show that algebra parameterization setup, as Hi 6t, A,A 0 Vi.j then Hyp 0.

---

Symmetry-Induced Disentanglement
Group
Group
A
GCN
Ax
Ecroph
Eccomp
Daruph
Ax
A
I
2
z
A
Z
t
GCN,
L
Conditionall disentanglement
Left (architecture): SID is composed of 4 blocks, i.e., 2 layers each for the auto-encoding modules for graph data and group structure. Middle (Egnou): the Lie algebra coordinates t; are obtained by applying ls graph convolutional networks (GCNs) to the graph embedding Z. Right (Dgrup): the coordinates l are combined with basis elements A to reconstruct the graph latent code. In the conditional case (red nodes) further condition on Z, by taking the mean (μ) of Z and Z, and passing μ through an activation function (ReLu for our experiments).
Figure I:
4 Method
We propose Symmetry-Indeee Induced Disentanglement (SID), a VAE framework for graphs 28 55 which encourages disentangled representations using Lie group parameterization and two regularization losses (commutative and hessian penalties). The architecture depicted in Figure]
4.1 Probabilistic formulation
We use the following notation for model variables: latent graph embeddings Z (and 2). graph features X, symmetry Lie group structure T,, and the graph adjacency matrix AA. We next derive lower bounds on log-likelihood for both the unconditional and conditional settings (see Appendix for details).
Proposition (Unconditional lower bound). Given two latent variables Z and T, we instantiate the unconditional disentanglement from Definition with probabilistic model that maximizes the log-likelihood of the graph data G = (X, by optimizing the following lower bound:
Lu E (Z))))((TZZA))) 2.4) log (TT
(L
Eg(z Φ) log q (Z19).
(1)
Proposition (Conditional lower bound). We extend Proposition to account for conditional disentanglement defined in Definition 4 We obtain the following lower bound
L. = (2 ()) E,(2)))
E log (Z(g).
(2)
Architecture. Propositions and ( are derived respectively in Appendix. The respective losses (2) are implemented with four neural network modules as follows. Two stochastic encoders (Egraph and Egrou) embed respectively the graph data into a latent variable Z via q( ()) and Z into the Lie group coordinates T via (Z) Two deterministic decoders (D (D groupp and D Dgraph) reconstruct, respectively, the graph embedding Ž from T via p p(2\TT for the unconditional case, and the graph from 2 via p(g (2). The decoder network Dyroup also plays a role implementing the conditional lower bound, via p( Z,T),, which computes 2 conditioned on Z and T.
4.2 Encoders
Graph encoder. We first encode the graph features X and the adjacency matrix in the latent space Z, using an inference network Egrap q (Z A). Similarly to Kipf and Welling (2 we use a single 2-layer network, where the first layer leverages graph convolutional network (GCN) to reduce the dimension of the graph features to the symmetry group size: X GCN(X, A). The second layer computes the mean and variance vectors (μ GCN (X) log GCN, (X, A)), which are
5

---

then used to sample the latent variable Z from a Gaussian distribution with the reparameteritzttion trick: qZ) A)
diag((())))
Group encoder. We employ Egrrup to map the latent vector Z into a vector T of l Lie algebra coordinates, where & is the subspace size (e.g., the number of disentangled elements). In our settings, each lie algebra coordinate {t,};- is the output of GCN with two layers. The first one computes graph embedding Z = GCN(Z, AA,, and the second one produces the vectors μ (Z, log a GCN, Z, A), which in turn yield produce {t, V with the reparameterization trick. Finally, {ti h are combined into single tensor T using a multilayer perceptron (MLP):
q (T) (LP(((4) - with q(4122, (4,lu, diag(a?)) for i 1....k (3)
4.3 Decoders
The module Dgroup reconstructs the latent variable using a Lie group G and Lie algebra Specifically, we first learn Lie algebra basis element {A.)== E for each coordinate t. We then aggregate the coordinates and basis elements A, with exponential map, to obtain group representation as
Group decoder.
p(2)1) g(T) = exp(A(T)) where A(T) Σ En= 4A. for g E G, Aeg.
(4)
The conditional version of D Deroup is obtained with a slight modification of Eq. 4 2 is computed by feeding the mean between Z (e.g. the graph embeddings obtained from Egraph), and exp(A(T)) into a mon-lineat activation function such as ReLU:
2( 7) a (mwaameeee(((AmT) Z), where = (=
Graph decoder. The network D Deraph reconstructs the graph data 9 from the latents 2. This is achieved by first passing the latent feature to GCN layer that maps Z into vector of the same dimension as the original feature vectors, and subsequently into a sigmoid activation function:
=(A X ((C((((())
(6)
4.4 Training
The proposed unconditional and conditional models are trained by optimizing the loss given by BL + Xh -C, where C is the lower bound from Eq. [ (unconditional) or Eq. (conditional), and h and are respectively the hessian and commutative penalties, obtained as regularization terms adapting the implementation from Zhu al. (70). We provide insights into the effect of parameters on the models and their performance with an ablation study in Section 5.1 For neural architecture, we set each encoder and decoder to consist of layers, where each layer takes as input the graph features from the previous layer.
Experiments
We conducted extensive experiments that decribe now. Section 5.1 evaluates the disentanglement capabilities of the models, Section (.cction5.22ppr provides a compression and few-shots classification experiment, and Section 5.3 assesses the generation capabilities molecular datasets.
5.1 Disentanglement evaluation
We evaluate disentanglement with a) an ablation study for assessing our models components, b) quali- tative investigations, and c) quantitative metrics. For metrics, we follow the evaluation protocols from Locatello al. (36), which involve being able to randomly combine a number of generative factors to sample new data. In our case, we provide the necessary generative factors to generate two types of synthetic random graphs (e.g. Erdos-Renyi and Watts-Strogatz) More details about our dataset construction procedure are provided in Appendix A. We compute 5 metrics, including 3-VAE [19 (Beta), FactorVAE (FVM) (26), Mutual Information Gap (MIG) [7], DCI Disentanglement
and
Factor Leakage (FL) 44

---

Disentanglemenn score
Metrics
60-
FVM
MIG
20
DCI
FL
36
10 15
30
20 Commutative penalty
10
25
8
70
Group size
Hessian penalty
Subspace size
Figure 2: Impact single model components on disentanglement metrics.
Ablation study. We start by evaluating the impact of our model components on the disentanglement metrics, by maintaining all parameters fixed, except from one that varies. We probe the effect from different group sizes, hessian penalties, commutative penalties, and subspace sizes. Figure we report the mean over 10 runs for our unconditional model on the synthetic Watts-Strogatz dataset. We observe that when the group size is increased, there is an improvement disentanglement across metrics, and this can be explained by the idea that with larger group size, the subgroups, represented by the Lie algebra coordinate t, can control different data variations. The hessian penalty results to be effectively enhancing disentangled representations, while the commutative penalty is less effective, and we can explain this result because the former requires the Lie algebra basis elements to have mutual products of zeros while subgroup decomposition only requires their commutators be zeros, which also confirms the results obtained for image datasets
we report the correlation matrix of the hidden graph features, the goal of which is to show whether the features capture mutually exclusive information, which is achieved when the there block-wise correlation pattern. In our models we group-size of 32 (to match with hidden features size of the other methods), a hessian penalty of 40 and commutative penalty 5. Figure we note that both our SID; and SIDc models achieve block-wise correlation pattern, however the conditional version has also some other highlighted regions, because this model involves fixing a latent vector to condition the reconstruction. As a result, the conditioning may be responsible for the visualized correlation on the hidden features.
Correlation analysis. Following previous work 65
38
SIDc
SIDu
FactorGCN
DisenGCN
GAT
GCN
Figure Correlation analysis
Learned graph variations. In this experiment, we train the SIDC model on Watts-Strogatz (WS) random graphs, which have 3 generative factors, n for number nodes, for connectivity (the probability of rewiring each edge), and & for neighborhood (k nearest neighbors in ring topology). adapt our model to the visualization implementations provided by Stoehr al. [56), which report adjacency matrices (black) and graph (white) information, where variations in node attribute values are indicated blue. In Figure 4 we report the results for the SIDe model by projecting pairs of latent variable traversals to see how the factors vary, and we observe the following. 1) The variable 20 controls the node attribute value. For example, in adje1 we see the color changing from blue to white along the horizontal axis. 2) The variable 21 controls connectivity, for example in grapho1, in the #1 axis (top to bottom) we see that the number of edges decrease, while in graph12. along the 21 axis (left to right), the number of edges increase. 3) The variable 22 controls the number of nodes, and see that in graph12 and graphe2 the number of nodes decreases along the 22 axis (bottom to top).
Disentanglement metrics. perform random seed runs, by training for 200 epochs on the proposed synthetic datasets and measure the quantitative metrics on 1000 data points. In terms of
7

---

adjon
grapho1
ad/12
graph12
adjo2
graphoz
Za
Figure 4: Graph traversals (attributes and node values) for the combinations of 3 latent variables.
Table I: Disentanglement results..
Erdos-Renyi
Model
FVM
Beta
MIG
DCI
FL
FVM
Beta
MIG
DCI
FL
GCN
69.620.8
66.5
23.2+2.5
43.8
84.1
57.0
61.021.4
22.5
54.1
71.7-2.6
GAT
72.6+1.3 66.2+3.4
17.4+1.0
53.2+2.7
79.0=1.1
56.0#32
66.3
31.441.9
54.0#2.9
70.2-2.1
FAC
74.2+1.a 65.7
26.4+2.0
53.4+4.0
80.3-u.9
67.2.1.7
66.3
35.1
53.8
78.1=a.9
DIS
73.0+3.5
69.342.1
27.1
52.2+3.9
85.2=2.b
68.1
64.1
41.3+1.4
53.5
72.4-2.
SIDu
80.5.15 71.644.5
28.5
55.412.0
91.0=2.3
83.0=2.9
71.843.2
53.2
58.621.
87.1
SIDe
81.911.2
14.5
49.1
59.7
=2.3
83.3.0.7
74.1
52.413.1
60.5
89.8
baselines, we consider graph VAEs with GAT [60 and GCN [29] encoders, as well as disentangled
encoders such as FAC (65) and DIS (38. We set our models have group size of 81, a hessian penalty of 40, and commutative penalty of 5.
In Table we observe that our models are able to improve on previous baselines on all metrics on the evaluated datasets, and the conditional version achieves the best performance in most cases. We notice that the FL metric shows significant performance increase, justified by its ability to capture the disentanglement on each generative factor, which, in our models is controlled by the group size, while in previous approaches is not considered. In terms of the other previously proposed disentangled approaches, DIS and FAC demonstrate higher disentanglement in most cases, when compared to GCN and GAT, however they are outperformed by our methods.
5.2 Compression and few-shots classification
Following the setup from Ge et al. [14, we consider the task of learning a compressed model via multiple pooling operations, and test on few-shots classification capabilities on datasets with multiple classes, including FRANKENSTEIN 43 classes), COLORS-3 30 (11 classes), Mu-
tagenicity [51] (2 classes), NCII 533 (2 c). The baselines include MIA [1), which propose an
iiteetiion-basee pooling for compression, as well as GCN (29) and GAT 600 based autoencoders, equipped with the same pooling layer. All models are set to have a pooling rate of 0.8, and depth of 3 layers. Our models are set with group size of 81, hessian penalty of 20 and commutative penalty of 5. The task consists in training first without labels to learn the representation, and then using the compressed model as input for training a GCN classifier a labelled subset of the datasct, which involves 100 samples for training and 100 for testing.
In Table we report the mean and variance across 10 runs for MSE, the classification accuracy, and the size of the trained model. The results show that SIDe and SIDu outperform the considered baselines in terms of few-shots classification accuracy on all datasets, while achieving a smaller compressed size. We hypothesize that the superior accuracy performance of our models is due to more efficient learning provided by disentangled representations, which have previously demonstrated the ability to enhance the accuracy of predictions in the context of generalization tasks 41 69 32 In terms of MSE, our models show an improvement compared to GCN- and GaT-based autoencoders, but they outperformed by MIA in most cases.
8

---

Table 2: Compression and few shot classification results.
TRANKENSTEIN
COLORS-3
Mutagenicity Size
NCII
Model
MSE
Size
Acc.
MSE
Size
Acc.
MSE
Acc.
MSE
Size
Acc.
MIA
1.6:0.5
4.9M
65
18
729K
33s2
4.1
753K
7812
3.2
777K
6111
GCN
12.6.2.5
4.2M
3913
11.2
657K
33.12
16.512.1
673K
72
7.421.2
697K
53
GAT
9.0
4.2M
2612
10.1
665K
30.1
11.342.8
681K
7914
8.1
705K
5012
SID
1.3
4.1M
68
5.1
625K
29
4.2
633K
8143
3.5
657K
63+2
SIDe
2.8.0.5
4.1M
69+2
1.9
625K
38
3.2
633K
84
3.9
657K
6642
Table 3: Random graph generation.
ZINC
QM9
MOSES
Model
Val
WR
Uni
Nov
Val
WR
Uni
Nov
Val
WR
Uni
Nov
JT-VAE
100
n/a
100
100
n/a
n/a
n/a
n/a
100
n/a
99.96
91.43
GCPN
100
20
99.97
100
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
GraphAF
100
99.1
100
100
94.15
88.83
100
99.99
100
GraphDF
100
89.03
99.16
100
100
82.67
97.62
98.1
100
87.58
99.55
100
SIDu
100
78.21
99.12
100
100
75.43
95.22
90.33
100
81.62
99.62
100
SIDe
100
86.02
99.14
100
100
82.92
96.43
92.28
100
84.29
99.79
100
5.3 Molecular graph generation
Molecular generation involves three tasks, random generation, property optimization and constrained optimization, and a suitable setup can be obtained following the approaches implemented via Torch- Drug and DIG 333 The generation process involves 1) learning the distribution of the molecular data and 2) fine-tuning the pre-trained model on one of the tasks. We employ our models during the training phase, using a group size of 81, a hessian penalty of 40 and commutative penalty of and then we follow the reinforcement learning fine-tuning method from GCPN [66]. Similar setups are used from previous molecule generation methods, which we include baselines: JT-VAE 24
GraphAF [54. GraphDF
37. GCPN [66]
Random generation. This task evaluates the quality of randomly generated samples on 4 standard metrics in percentage, including: valid molecules with resampling (Val), valid molecules without resampling (WR), unique molecules (Uni), novel molecules that not appear in the training data (Nov). We train our models for 10 epochs, with a batch size of 32 and the learning rate of 0.001, and compute the metrics over 10K generated molecules. In Table 3 we report results on ZINC 23 QM9 49 and MOSES 47). We observe that both SID and SIDe are able to improve the performance GraphAF and GCPN on most metrics, while achieving comparable performance with GraphDF, furthermore SIDe model achieves the highest score of validity without resampling on the QM9 dataset. Since our model is not optimized for molecule generation, the positive results of our models indicate that disentangled representations may represent a flexible tool for generating more realistic graphs, and that encouraging the separation of semantic factors into different latent codes, may be beneficial for improving the quality of the samples.
Property optimization The goal of the task generate novel molecules with high property scores for the penalized logP property, and the quantitative estimation of drug-likenes property (QED). The setup involves pretraining our models for 300 epochs on ZINC random generation, applying the RL fine-tuning procedure from GCPN, and measuring the scores from the top 3 generated molecules. In Table we observe that our models are able to outperform GCPN and GraphAF for both penalized logP and QED, and achieve comparable performance to the more advanced GraphDF model. In Figure 5 we show the molecules generated using SIDe. While our model does not outperform GraphDF, results are relevant because they demonstrate that the disentangled representations learned from our models can be leveraged to improve the performance of the downstream task of property optimization.
hhttss//oorchrrug.a/ (Apache license)

---

Table 4: Property optimization performance.
22.0
Penalized
log? 2nd
QED
1st
3rd
1st
2nd
3rd
ZINC
4.52
4.3
4.23
0.948
0.948
0.948
JT-VAE
5.3
4.93
4.49
0.925
0.911
0.910
SEDEM0
GCPN
7.98
7.85
7.80
0.948
0.947
0.946
GraphAF
12.23
11.29
11.05
0.948
0.948
0,947
GraphDF
13.7
13.18
13.17
0.948
0.948
0.948
SIDu SIDe
12.56
12.46
12.03
0.947
0.948
0.947
12.89
12.82
12.27
0.948
0.948
0.948
Figure 5: Molecules from SIDe
Table 5: Constrained optimization performance on 800 molecules used in GraphAR.
8
GraphAF Sim
GraphDF
SIDe
Imp
Suc
Imp
Sim
Suc
Imp
Sim
Suc
0.0
13, 66.69
2.29
100
14.15
0.29
100
13.26
0.28
100
0.2
11.90
0.
100
12.77 26.50
0.32
100
12.35
0.32
100
0.4
6..1
0.49
99.8
9.19
0.48 20.08
99.6
8.25
0.50
98.4
0.6
6.98
0.66
96.8
#5.80
0.65
92.1
4.67 20.39
0.66
93.2
Constrained Optimization. This task aims to modify the input molecular graph for improving its penalized logP score while keeping the similarity between the input and modified molecules higher than the threshold 6. We follow the setup of GraphAF and GraphDF, which select 800 molecules from ZINC with low penalized logP scores as the input molecules be optimized. We pretrain our model on the random generation task for 300 epochs, and then tune with the RL procedure from GraphAF. We report the mean and standard deviation of the largest property improvement (Imp), and similarities (Sim) between them and their corresponding input molecules, as well as the success rate (Suc). In 3 we compare the results for our SIDe model with GraphAF and GraphDF. We observe that SIDe is able to outperform GraphAF 3 out of 4 times for improvement values, and achieves comparable results with GraphDF, which we motivate by the fact that GraphDF provides further fine-tuning in order to improve the baseline. In terms of similarity and success rate, model has comparable results with the baselines for all å setups.
Discussion
Limitations. For quantitative evaluation, rely synthetic data, since current metrics based on the ability to sample the factors and combine them into observations Kim and Mnih [26). Future work should investigate how to provide metrics that evaluate disentanglement
real-world datasets. A recent line of work from Khemakhem et al. [25] shows that disentanglement is closely related to model identifiability, i.e., representation is disentangled when a of learned parameters uniquely identified with the parameters in the true distribution. Our approach has not considered or provided proofs for, identifiability. and we leave this as future work.
Conclusion. We formalize the notion of conditional disentanglement on graphs and propose novel framework for graph disentanglement by leveraging tools from Lie algebras. Based on the new definition of disentanglement, we design a graph VAE based on a Lie group parameterization, and provide a novel ELBO criteria for optimizing conditional disentanglement. Our method achieves superior performance on quantitative disentanglement benchmarks when compared to contemporary disentangled GNNs and other convolutional layers. Finally, we demonstrate strong capabilities on few-shots classification and molecular generation experiments.
7 Acknowledgements
We thank the anonymous reviewers for their helpful comments. GM's research was partially funded by EPSRC and the BBC under iCASE.
10

---

References
[1] Inhwan Bae and Hae-Gon Jeon. Disentangled multi-relational graph convolutional network for pedestrian trajectory prediction. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 911-919, 2021.
[2] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):
1798 1828, 2013.
[3] Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, Gopinath Balamurugan, Michael M. Bronstein, and Haggai Maron. Equivariant subgraph aggregation networks. International Conference on Learning Representations, 2022.
[4] Diane Bouchacourt, Mark Ibrahim, and Stéphane Deny. Addressing the topological defects of disentanglement via distributed operators. arXiv preprint arXiv:2102.05023, 2021.
[5] Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34 (4):18-42, 2017.
[6] Hugo Caselles- Michael Gareia Ortiz, and David Filliat. Symmetry-based disentangled representation learning requires interaction with environments. Advances in Neural Information Processing Systems, 32, 2019.
[7] Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentanglement in vaes. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pages 2615-2625, 2018.
[8] Zhengdao Chen, Soledad Villar, Lei Chen, and Joan Bruna. On the equivalence between graph isomorphism testing and function approximation with gnns. In NeurlPS, 2019.
[9] Kien Do and Truyen Tran. Theory and evaluation metries for learning disentangled representa- tions. In International Conference on Learning Representations, 2019.
[10] Yuanqi Du, Xiaojie Guo, Hengning Cao, Yanfang Ye, and Liang Zhao. Disentangled spatiotem- poral graph generative models. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[11] Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2.18.
[12] Marco Fumero, Luca Cosmo, Simone Melzi, and Emanuele Rodolà. Learning disentangled representations via product manifold projection. In International Conference on Machine Learning, pages 3530-3540. PMLR, 2021.
[13] V. Garg, S. Jegelka, and T. Jaakkola. Generalization and representational limits of graph neural networks. In International Conference on Machine Learning (ICML), 2020.
[14] Yunhao Ge, Yunkui Pang, Linwei Li, and Laurent Itti. Graph autoencoder for graph compression and representation learning. In Neural Compression: From Information Theory to Applications- Workshop @ ICLR 2021, 2021.
[15] M. Gori, G. Monfardini, and F. Scarselli. A new model for learning in graph domains. In IEEE International Joint Conference Neural Networks (IJCNN), 2005.
[16] Jingwei Guo, Kaizhu Huang, Xinping Yi, and Rui Zhang. Lgd-gen: Local and global disentan- gled graph convolutional networks. arXiv preprint arXiv:2104.11893 2021.
[17] Xiaojie Guo, Liang Zhao, Zhao Qin, Lingfei Wu, Amarda Shehu, and Yanfang Ye. Interpretable deep graph generation with node-edge co-disentanglement. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1697-1707, 2020.
11

---

[18] Xiaojie Guo, Yuanqi Du, and Liang Zhao. Deep generative models for spatial networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. pages 505-515, 2021.
[19] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
[20] Irina Higgins, David Amos, David Pfau, Sehastien Racaniere, Loie Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230 2018.
[21] Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P Burgess, Matko Bošnjak, Murray Shanahan, Matthew Botvinick, Demis Hassabis, and Alexander Lerehner. Scan:
Learning hierarchical compositional visual concepts. In International Conference Learning Representations, 2018
[22] Irina Higgins, Sébastien Racanière, and Danilo Rezende. Symmetry-based representations for artificial and biological general intelligence. Frontiers in Compulational Neuroscience, page 28, 2022.
[23] John Irwin, Teague Sterling, Michael M Mysinger, Erin S Bolstad, and Ryan G Coleman. Zinc: a free tool to discover chemistry for biology. Journal of chemical information and modeling, 52 (7):1757-1768, 2012.
[24] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning, pages 2323-2332. PMLR, 2018
[25] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational au- toencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217. PMLR, 2020.
[26] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference Machine Learning, pages 2649-2658, 2018.
[27] Diederik Kingma and Max Welling, Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[28] Thomas N Kipf and Max Welling. arXiv: (0:::611..00 1.07308 2016.
Variational graph auto-encoders. arXiv preprint
[29] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference Learning Representations (ICLR), 2017.
[30] Boris Knyazev, Graham W Taylor, and Mohamed Amer. Understanding attention and gener- alization in graph neural networks. Advances in neural information processing systems, 32, 2019.
[31] Haoyang Li, Xin Wang, Ziwei Zhang, Zehuan Yuan, Hang Li, and Wenwu Zhu. Disentangled contrastive learning on graphs. Advances in Neural Information Processing Systems, 34, 2021.
[32] Xiangyu Li, Zhe Xu, Kun Wei, and Cheng Deng. Generalized zero-shot learning via disentangled representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1966-1974, 2021.
[33] Meng Liu, Youzhi Luo, Limei Wang. Yaochen Xie, Hao Yuan, Shurui Gui, Haiyang Yu, Zhao Xu, Jingtun Zhang, Yi Liu, et al. Dig: a turnkey library for diving into graph deep learning research. Journal of Machine Learning Research, 22(240):1 9, 2021.
[34] Qi Liu, Miltiadis Allamanis, Mare Brockschmidt, and Alexander Gaunt. Constrained graph variational autoencoders for molecule design. Advances neural information processing systems, 31, 2018
12

---

[35] Yanbei Liu, Xiao Wang, Shu Wu, and Zhitao Xiao. Independence promoted graph disentangled networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4916-4923, 2020.
[36] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Ractsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. international conference on machine learning, pages 41 14 4124. PMLR, 2019.
[37] Youzhi Luo, Keqiang Yan, and Shuiwang Graphdf: A discrete flow model for molecular graph generation. In International Conference on Machine Learning, pages 7192-7203. PMLR, 2021.
[38] Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. Disentangled graph convolu- tional networks. In International conference on machine learning, pages 4212-4221. PMLR, 2019.
[39] Haggai Maron, Heli Ben- Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. Advances in Neural Information Processing Systems, 32:2156-2167 2019.
[40] Graziano Mita, Maurizio Filippone, and Pietro Michiardi. An identifiable double vae for disentangled representations. In International Conference on Machine Learning, pages 7769- 7779. PMLR, 2021.
[41] Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey Bowers. The role of disentanglement in generalisation. International Conference on Learning Representations, 2020.
[42] Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey Bowers. The role of disentanglement in generalisation. In International Conference on Learning Representations, 2021. URL
[43] Francesco Orsini, Paolo Frasconi, and Luc De Raedt. Graph invariant kernels. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
[44] Matthew Painter, Adam Prugel-Bennetee and Jonathon Hare. Linear disentangled representations and unsupervised action estimation. Advances Neural Information Processing Systems, 33: 13297-13307, 2020.
[45] William Peebles, John Peebles, Jun- Yan Zhu, Alexei Efros, and Antonio Torralba. The hessian penalty: A weak prior for unsupervised disentanglement. In European Conference Computer Vision, pages 581 381-597. Spriner, 2020.
[46] David Pfau, Irina Higgins, Alex Botev, and Sébastien Racanière. Disentangling by subspace diffusion. Advances in Neural Information Processing Systems, 33:17403-17415, 2020.
[47] Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy. Mark Veselov, et al. Molecular sets (moses): benchmarking platform for molecular generation models. Frontiers in pharmacology, 11:1931, 2020.
[48] Robin Quessard, Thomas Barrett, and William Clements. and group structure of dynamical environments. Advances in Neural Information Processing 3ystems,33: 13:19777-197777 2020.
Learning disentangled representations
[49] Raghunathan Ramakrishnan, Pavlo Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014.
[50] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. In International conference on machine learning, pages 1278-1286, PMLR, 2014.
13

---

[51] Kaspar Riesen and Horst Bunke. lam graph database repository for graph based pattern recognition and machine learning. In Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR), pages 287-297. Springer, 2008
[52] Wulf Rossmann. Lie groups: an introduction through linear groups, volume 5. Oxford University Press on Demand, 2006.
[53] Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-ehmann graph kernels. Journal of Machine Learning Research, 12(9), 2011.
[54] Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: a flow-based autoregressive model for molecular graph generation. In International Conference on Learning Representations, 2020.
[55] Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. In International conference on artificial neural networks, pages 412-422. Springer, 2018.
[56] Niklas Stoehr, Emine Yilmaz, Mare Brockschmidt, and Jan Stuchmer. Disentangling interpretable generative parameters of random and real-world graphs. arXiv preprint arXiv:/910.05639, 2019.
[57] Loek Tonnaer, Luis Armando Perez Rey, Vlado Menkovski, Mike Holenderski, and Jim Porte- gies. Quantifying and learning linear symmetry-based disentanglement, In P Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 21584-21608, 17-23 Jul 2022.
[58] Michael Tschannen, Olivier Bachem, and Mario Lucie. Recent advances in autoencoder-basee representation learning. arXiv preprint 1812.0081 2018
[59] Sjoerd Van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem, Are dis- entangled representations helpful for abstract visual reasoning? Advances in Neural Information Processing Systems, 32, 2019.
[60] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
[61] Tan Wang. Zhongqi Yue, Jianqiang Huang, Qianru Sun, and Hanwang Zhang. Self-supervised learning disentangled group representation feature. Advances Neural Information Process- ing Systems, 34, 2021.
[62] Tailin Wu, Hongyu Ren, Pan L.,, and Jure Leskovec. Graph information bottleneck. Advances in Neural Information Processing Systems, 33:20437-20448, 2020.
[63] K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? In International Conference Learning Representations (ICLR), 2019.
[64] Tao Yang, Xuanchi Ren, Yuwang Wang, Wenjun Zeng, and Nanning Zheng. Towards build- ing a group-based unsupervised representation disentanglement framework. In International Conference on Learning Representations, 2022.
[65] Yiding Yang. Zunlei Feng. Mingli Song, and Xinchao Wang. Factorizable graph convolutional networks. Advances in Neural Information Processing Systems, 33:20286-20296, 2020.
[66] Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for dirl-dir molecular graph generation. Advances in neural information processing systems, 31, 2018.
[67] Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. Graphrnn: Generat- ing realistic graphs with deep auto-regressive models. In International conference on machine learning, pages 5708-5717. PMLR, 2018.
14

---

[68] Junchi Yu, Tingyang Xu, Yu Rong, Yatao Bian, Junzhou Huang, and Ran He. Graph information bottleneck for subgraph recognition. In International Conference on Learning Representations. 2021. URL httpttts///peereeveeee... wtt.://ete/ooooummmdd//aaagg//
[69] Hanlin Zhang, Yi-Fan Zhang, Weiyang Liu, Adrian Weller, Bernhard Schölkopf, and Eric P Xing. Towards principled disentanglement for domain generalization. arXiv preprint arXiv:2111.13839, 2021.
[70] Xinqi Zhu, Chang Xu, and Dacheng Tao. Commutative lie group for disentanglement learning. In International Conference on Machine Learning, pages 12924-12934. PMLR, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(C) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Provided in Appendix
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as URL)? [Yes] Provided at https://merrctaall/ gitlab. 10/ sid/
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] We specify the experimental setups in Section
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See beginning of Section S
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)? [Yes] See beginning of Section 5
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section 5.3
(b) Did you mention the license of the assets? [Yes] See Section 5.3
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We include assets at https:: httpes//emeeraaallgi.......
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "ufSj8LuhjaV", "reviewer": "Reviewer_DqMr", "summary": "The paper proposes two (conditional and unconditional) symmetry-based and Lie-algebras-based disentanglement on Graph Neural Networks within the framework of Variational Autoencoder. As is defined in symmetry-based disentanglement, a symmetry group, which can be decomposed into subgroups, can help disentangle a vector representation affected by it if specific demands are satisfied. Therefore, the authors introduce a Lie group as the symmetry group to graph disentanglement. Besides, they add commutative penalty and hessian penalty to enforce further disentanglement. In the experiment part, they evaluate systematic disentanglement on multiple metrics based on synthetic datasets and model performance based on real datasets.", "questions": "1.\tThere are two minor errors in Conclusion. In line 319, “and” should be “an”. In line 320, “real-word” should be “real-world”.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThe disentanglement on graph is an important topic and the paper provides a new insight from symmetry-based one and introduces the Lie group to it.\n2.\tThe consideration of commutative penalty, hessian penalty is comprehensive.\n\nWeaknesses:\n1.\tAs is written in Conclusion, this paper doesn’t present the disentangled factors on real-world datasets. But both [1] and [2], which are mentioned in Related Work, have disentanglement evaluations based on feature correlation analysis. Therefore, it is possible to evaluate disentanglement on real-world datasets, although it is still open which one is the most convincing.\n[1] Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. Disentangled graph convolutional networks. In International conference on machine learning, pages 4212–4221. PMLR, 2019.\n[2] Yiding Yang, Zunlei Feng, Mingli Song, and Xinchao Wang. Factorizable graph convolutional networks. Advances in Neural Information Processing Systems, 33:20286–20296, 2020.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TBFCXHXxqFK", "reviewer": "Reviewer_3Dsd", "summary": "This paper emphasizes learning disentangled representations from graphs. In contrast to prior works, the authors formulate new definitions of unconditional/conditional symmetry-based graph disentanglements and developed a Lie group-based framework for learning with Graph VAE. The authors evaluate their methods on disentanglement metrics constructed on synthetic datasets and downstream tasks, including few-shot classiﬁcation, compression and molecular graph generation.", "questions": "**[Experimental setup]**\n\n- The derivation in section 3.2 use a single 2-layer GCN, does this architecture is used across all the experiments in section 5.3? I didn't find a clear description in the section. 5 nor the appendix. \n- The setting value of the communitative penalty is not mentioned.\n- What are the comparisons on the similarity and success rates (in line 307)? Are all the methods follow the same setting of distance threshold  $\\delta$ ? How to control that threshold in JT-VAE?\n\n**[Interpretations MIA on Table.2]** Its turns out MIA is a strong baseline that achieves the best MSE performance across all the datasets. Any interpretations on this?\n\n**[Scaling Up]** How would the proposed method perform on large graph task (macromolecules, community graphs)?", "limitations": "The authors adequately addressed the limitations (disentanglements can be measured only on synthetic datasets) of their work in section.6", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n\n- ***Originality***: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited. \n\n  Learning disentangled representaitons on graphs is a \n\n- ***Quality***: The work is complete, my concerns on the theoretical analysis and experimental results are put at the *clarity* and **Questions**.\n\n- ***Clarity***: The submission is clearly written and well organized. The experimental descriptions are not clear to me. My suggestions are as follows:\n\n  **[Adjacency in GNN architechture]** In lines 88-99, the notation of adjacency should be introduced at the beginning, otherwise how to conduct the graph convolution proceducre? The current formuation sees to be have space for improving readability.\n\n  **[Spelling]** (2 c) -> (2 classes) in line 252; in line 319 \"Since this is and open problem in representation learning\" is not a sentence, need to be rephrased.\n\n- ***Significance***: With the code provided in the supplementary, others researchers or practitioners can use the ideas or build on them. The constructed dataset should also be released (lines 217-219). Besides demonstrating the effectiveness of the proposed method, I have a question about the weakness of comparisons in the molecular generation and property optimization tasks. It seems that all the ways perform equally well/bad in these tasks, and many scores are precisely the same. Given the usages of different models and methods, it is a bit unclear to validate the improvements induced by disentangling the learned representations on these tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LduvfSHguEe", "reviewer": "Reviewer_Em3T", "summary": "The authors first propose a formalization of graph disentanglement in terms of graph embeddings building on existing definitions. They then propose a graph+group VAE architecture incorporating the graph disentanglement definition. This autoencoder first uses a graph encoder to map the original graph features to k latent features, which are then independently passed to k GCNs and aggregated by the group encoder to obtain the group coordinates. They are then decoded by the group decoder, combining the basis elements and passing them to the graph decoder. The authors then demonstrate significant improvements in several metrics, particularly disentanglement metrics. ", "questions": "How universal is the proposed definition - are there classes of graphs for which it may not be ideal?\n\nWhat are some practical implications of this kind of disentanglement on graphs? In what contexts might this be useful?", "limitations": "Aside from what is discussed above, I think the authors reasonably discussed their limitations.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper in general is clear, well-written, well-motivated, and sufficiently rigorous. Although I am not especially familiar with previous work on graph disentanglement in particular, having reviewed the related works, the proposed method seems novel in the context of graphs. The translation of their proposed definition to their GVAE seems particularly clever and does not seem to obviously follow from the definition alone: it captures the key features of their definition while still being highly general and customizable. The experiments are fairly convincing as well, with some impressive disentanglement performance on the datasets analyzed (though more disentanglement datasets would have been helpful). \n\nMy primary (albeit limited) concern with this paper’s definition is that it appears to more-or-less directly translate the concept of disentanglement in terms of underlying generative latent variables to graphs, embeddings and all, while seemingly largely disregarding graph-specific elements. However, graphs are potentially meaningfully different from the unstructured spaces for which disentanglement has previously been formalized. For example, graphs have often been used to express latent variable models, of which VAEs are a specific kind - though latent variable graphs are only a specific kind, it is not obvious that this framework would be the best way to characterize disentanglement in contexts where the graphs themselves represent generative processes. This is not to say that the proposed definition is not useful in this context: generative processes can themselves be the product of an underlying generative process. However, it is likely worth exploring and discussing the kinds of graphs where this definition may not cleanly apply. \n\nIn addition, it would also be helpful to provide some additional justification for the metrics used. While they are standard disentanglement metrics, it does not appear that many prior works specifically on graphs aside from a 2019 workshop paper use them (including the ones cited). In particular, a workshop paper that may be worth mentioning is “Disentangling Interpretable Generative Parameters of Random and Real-World Graphs” (Stoehr et al 2019) which much less formally proposes evaluating graph disentanglement in terms of equivariance between the underlying generative and corresponding encoding maps. Finally, some visualizations of the kinds of variations learned by disentangled models would have been helpful. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IrieMdO7tXA", "reviewer": "Reviewer_FmQN", "summary": "This paper brings group-based disentanglement learning to graph data domain. Two disentanglement cases, unconditional and conditional, are defined under this framework in this paper. The proposed architecture encodes group data into group latent representations with Lie algebra and exponential map, and leverages techniques like Hessian penalty to enforce disentanglement. Experiments on various data have shown the model’s effectiveness in disentanglement on graph data.", "questions": "Please see weaknesses.", "limitations": "Yes", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The idea of group-based disentanglement is relatively new, and it is interesting to see it being brought to the graph data domain.\n2. The proposed architecture is quite simple and the shown experimental results validate its effectiveness.\n\nCons:\n1. In Fig. 1 right, why is the addition from Z is used in the conditional case? Does it mean Z will contain extra information that is not encoded in the latent T?\n2. Is it possible to qualitatively show what information has been disentangled in the graph data?\n3. The section 2.2 GNN architecture part seems a little redundant to me as Sec. 3.2 and 3.3 will introduce the architecture in detail.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept