Permutation-Invariant Variational Autoencoder for Graph-Level Representation Learning

Abstract: Robin Winter, Frank Noe, Djork-Arné Clevert

Reviews and author responses:

 We thank the reviewer for the clarification of the mentioned references. We will add those with a discussion to the manuscript. Thanks for the suggestion.


 The first reference I referred to is “Provably Powerful Graph Networks“, NeurIPS 2019. The model there essentially is kind of the transformer on edges because 2-FWL naturally requires one to handle edge-edge interactions. A following-up work of this one "Global Attention Improves Graph Networks Generalization" that adopts the same trick explicitly claims using attention/transformer things. 

I think the difference between 2WL and 2FWL is that in 2WL, one computes the attention between a centric node-pair with each of its neighboring node-pairs and pool them together to update the centric node-pair representation, while in 2FWL, one computes the attention between every pair of the neighboring node-pairs of acentric node-pair and pool to update the centric node-pair representation. I think the authors' better clarify which model this paper indeed uses in the final version, and also add references. 
 

 We thank the reviewer very much for considering our responses again and raising their rating.

 Thanks for the detailed response again. Now I am fully aware that the proposed model is indeed invariant to the input node ordering, essentially solving the marginalization (over all node permutations) problem raised in [35].

This paper proposes a graph generative model based on VAE. The model differs from previous methods in that it has a permutation invariant encoder that learns the same representation for isomorphic input graphs. Then, a question is how to measure the reconstruction error. In principle, all n! possible generations (only differ in node permutations) should have equal reconstruction error to the input graph. Previous works usually use a graph matching process which is computationally inefficient. Some other works make the decoder generate a graph with exactly the same node order as the input graph, which makes the encoder permutation variant. This paper cleverly solves this problem by decoupling the encoder into two parts: a permutation invariant encoder that learns order-invariant graph representations, and a permutation sensitive "permuter" encoder which learns a permutation matrix to align the decoded graph's node order to the input graph. In this way, it can both learn permutation invariant graph representations as well as avoiding the expensive graph matching process.  Originality: The proposed permuter is novel and reasonable to me. The task of training a permutation-invariant graph encoder through a VAE framework is also desired.

Quality: The proposed model is technically sound. The experiments cover a range of unsupervised graph representation learning tasks.

Clarity: The paper is well written in general. However, some parts can be written better. For example, the discussion on solving the graph isomorphism problem can be made more clear. Why "if the decoder can perfectly reconstruct both graphs, two non-isomorphic graphs must have a different representation z"? Also, Section 2.3 puts the overall architecture to a later part, while the previous architecture designs are hard to understand without the overall architecture. I suggest re-organizing this section by introducing the overall architecture first.

Significance: I have some doubt on the significance of the results. If we only want a permutation invariant encoder, we have many ways to achieve this as stated in the related work (e.g., not relying on reconstructing the input graph). If we want a generative model, we also do not need to care about the node ordering as long as the the generative model learns the training graph distribution correctly (e.g., always generates graphs with canonical node ordering). In this sense, this work seems to simply combine a permutation invariant encoder (for graph representation learning) and a permutation dependent "permuter"+decoder (for generating the graph with the input node order). It does not really invent a permutation invariant decoder/generator---the generated graph's ordering still has to be dependent on the input graph.

---------After rebuttal-----------
I thank the authors for the nice discussion. Now I am aware that the training of the proposed model is also invariant to the input node ordering, essentially solving the marginalization (over all node permutations) problem raised in [35]. Therefore, the significance is high. I have raised my score by 2, and I encourage the authors to include our discussed points into the revised paper. The authors state in the related works that GAE-like approaches construct graph representations by concatenating node embeddings, thus is not permutation invariant. But why can't we sum/average the node embeddings like in graph pooling layers?

 We thank the reviewer again for considering our response and argument why our proposed method enables decoding a graph independent of the input graph’s node order. However, we apparently failed to clarify the permuter’s permutation-equivariant design properly and how it leads to an input order independent learning process. Please see our response to your comments bellow. 

### Comment 1:

„However, I do not agree with the authors that "the learning process (i.e., the learned weights) of the permuter is also invariant to the ordering of nodes in the input graphs." Suppose you have one training graph with 10 nodes. In the first case, the node of interest has index 1. In the second case, it has index 10. In both cases the message $m_{ii}$ of this node is the same. But in the first case the permuter needs to learn a scoring function $f_s$ that maps $m_{ii}$  to a small $s$, while in the second case it needs to learn a scoring function $f_s$ that maps $m_{ii}$  to a large $s$. Then, the learning process of the permuter will depend on the input node ordering.
That said, the proposed model does not fully solve the problem of [35] (i.e., the learning process is fitting a particular node ordering instead of marginalizing over all π) in my opinion. Can the authors comment on that?“

#### Response:

We would like to emphasize again that the permuter is by design permutation equivariant, i.e., the assigned scores for each node permute equivariantly with the input’s node order. This is a crucial property of our proposed model, enabling the learning process of the whole model to be independent of the input node order (i.e., the learned weights are invariant to the order of nodes in the input graphs of the training data set). Note, that for a equivariant function follows: $f(g\cdot x) = g\cdot f(x)$. Hence for the permutation equivariant permuter model $f_{\text{perm}}$ follows: 

$ f_{\text{perm}}(G_\pi) = f_{\text{perm}}(P_{\pi’\rightarrow \pi}G_{\pi’}) = P_{\pi’ \rightarrow \pi} f_{\text{perm}}(G_{\pi’}) $ .

The permuter is defined by a scoring function $f_s$ acting on the diagonal entries of the encoded graph

$f_{\text{perm}}(G_\pi) = f_s(\text{diag}(f_\text{enc}(G_\pi)))$, where $ f_\text{enc}: G_\pi \rightarrow m_{ij}$ and $f_s: m_{ii} \rightarrow s_i $. 

Both encoder $f_\text{enc}$ (self-attention without position embeddings) and $f_s$ (MLP individually applied (weight sharing) on each input $m_{ii}$) are permutation equivariant by design. Hence, the composition of both (i.e., $f_{\text{perm}}$) is permutation equivariant as well. 

Like the reviewer correctly points out, independent of the node order of the input graph, the encoded message $m_{ii}$ of a certain node will be the same. For the reviewers example, node $v_{1}^{(\pi)} = v_{10}^{(\pi’)}$ where $\pi$ and $\pi’$ correspond to case (i.e., node order) one and two respectively, follows $m_{1,1}^{(\pi)} = m_{10, 10}^{(\pi’)}$. However, other than the reviewer noted, the scoring function $f_s$ will map the node/message in both cases to the same score $s_1^{(\pi)} = f_s(m_{1,1}^{(\pi)}) = f_s(m_{10,10}^{(\pi’)})=s_{10}^{(\pi')} $. The only difference between both cases is the order of the scores in the output of the permuter. I.e., if we define a operator $P_{\pi' \rightarrow \pi}$ that exchanges/permutes node 1 and node 10 in a graph, follows for the permuter's output $\mathbf{s}=[s_1,...,s_{10}] $ (i.e., the sequence/vector of scores for the nodes in the input graph): 

$f_{\text{perm}}(G_\pi)=\mathbf{s}^{(\pi)}=[s_1, s_2, ..., s_9, s_{10}]^{(\pi)}=[s_{10}, s_2, ..., s_9, s_1]^{(\pi’)}=P_{\pi' \rightarrow \pi}[s_1, s_2, ..., s_9, s_{10}]^{(\pi’)}$

$=P_{\pi \rightarrow \pi’}\mathbf{s}^{(\pi’)}= P_{\pi' \rightarrow \pi}f_{\text{perm}}(G_{\pi’})= {f_\text{perm}}(P_{\pi' \rightarrow \pi}G_{\pi’})$ .


Based on this change in the order of scores, the permutation matrix which is defined by sorting the scores $s_i$ (compare Equation (6)), changes equivariantly.

From the reviewer’s comment it seems that they maybe misunderstood how the scores are used to construct the permutation matrix. Please note, that the permuter assigns a score based on the predicted node’s position in the decoded node order, not based on the input node order. The learned canonical output order is independent of the input node order. Hence, a node’s score should be invariant to its position in the input order. However, if a scores position will change in the permuter’s output (because the input node order changes), the argsort output will change equivariantly, which will lead to an equivariant change in the permutation matrix.

In the reviewer’s example, the node’s score ($s_1$ and $s_{10}$ for case 1 and 2 respectively) will be the same in both cases, e.g., w.l.o.g. the largest score (i.e., matches the first decoded node), then, the argsort operator will either match index 1 in the input with index 1 in the output (case 1) or index 10 in the input with index 1 in the output (case 2).
Please note how the same score at different positions in the permuter’s output will lead to the respective change in the permutation matrix (hence, $f_s$ does not need to learn to output different individual node scores for different input orders).


To summarize, by utilizing permutation equivariant methods in the permuter model, invariance to the input node order in the training dataset is given by design and is not part of the learning process (e.g., by marginalize over all possible permutations $\pi$ of a graph, which is computational intractable for larger graphs as discussed e.g. in [35]).
We hope that from this more detailed explanation it is now clearer how we utilize the permutation invariant and equivariant properties of the different parts of our proposed method to make the whole learning process independent of the input node order and actually fully addresse “the problem of [35]” (our learning process is not fitting a particular node ordering and marginalizing over all $\pi$ is not necessary as our proposed model’s learning process is invariant to the input node order).
We will rework the Key Architectural Properties section of our manuscript to make this point more clear. We thank the reviewer for making us aware of this. 

### Comment 2:

“Also, I acknowledge that [35] is not natively a VAE model. But it is easily generalizable to VAE by equipping it with a GNN encoder. And such models with permutation-invariant graph encoders and sequential decoders are explored in [1], [2] and many other papers. I expect the authors to discuss more on the differences between your model and the listed ones.“

#### Response:

We thank the reviewer for pointing this out. We acknowledge, that in its current form, the manuscript is missing a more detailed discussion of the related work on autoregressive decoder models. As discussed in our previous response, the main difference of autoregressive models to our proposed model is that those approaches train the decoder in a permutation variant way by sequentially decoding nodes/edges, maximizing the log likelihood of a graph in a certain node order. We will include a more detailed discussion of those works and their differences to our work to the related work discussion. We thank the reviewer for this suggestion. 


### Comment 3:

„Some minor points after my second read: The paper has many grammar errors or typos. For example, "58, represented -> represent"; "61, $ q_\phi(G|z)$" you mean "$q_\phi(z|G)$"?“

#### Response:

Yes indeed, we mean $q_\phi (z|G)$. Thanks for pointing this out! We will double check grammar and spelling.


 I thank the authors again for clarifying that the decoder learns its own canonical ordering, and the permuter only serves as reordering the decoded graph to align with the input graph for computing the reconstruction loss. This makes sense to me. In this sense, the decoupled design indeed enables decoding a graph independent of its input node order, in contrast to [35].

However, I do not agree with the authors that "the learning process (i.e., the learned weights) of the permuter is also invariant to the ordering of nodes in the input graphs." Suppose you have one training graph with 10 nodes. In the first case, the node of interest has index 1. In the second case, it has index 10. In both cases the message $m_{ii}$ of this node is the same. But in the first case the permuter needs to learn a scoring function $f_s$ that maps $m_{ii}$ to a small $s$, while in the second case it needs to learn a scoring function $f_s$ that maps $m_{ii}$ to a large $s$. Then, the learning process of the permuter will depend on the input node ordering.

That said, the proposed model does not fully solve the problem of [35] (i.e., the learning process is fitting a particular node ordering instead of marginalizing over all $\pi$) in my opinion. Can the authors comment on that?

Also, I acknowledge that [35] is not natively a VAE model. But it is easily generalizable to VAE by equipping it with a GNN encoder. And such models with permutation-invariant graph encoders and sequential decoders are explored in [1], [2] and many other papers. I expect the authors to discuss more on the differences between your model and the listed ones.

Some minor points after my second read:
The paper has many grammar errors or typos. For example, "58, represented -> represent"; "61, $q_\phi(G|z)$" you mean "$q_\phi(z|G)$"?

[1] Zhang, Muhan, et al. "D-vae: A variational autoencoder for directed acyclic graphs." arXiv preprint arXiv:1904.11088 (2019).
[2] Liu, Qi, et al. "Constrained graph variational autoencoders for molecule design." arXiv preprint arXiv:1805.09076 (2018).

 We thank the reviewer for their response. We are not sure which paper the reviewer is referring to with "Provably more powerful neural networks", Neurips 2019. The closest hits we found are “Distance Encoding: Design Provably More Powerful Neural Networks for Graph Representation Learning”, NeurIPS 2020 and “Provably Powerful Graph Networks“, NeurIPS 2019. In both we could not find a discussion of Transformers on edges and that they “can achieve 3-WL (2-forklore WL) power”. From our understanding, our proposed transformer working on edges can only be as powerful as the 2-WL test, as we work on edges (pairs of nodes, i.e., k=2) and not node triplets, while the aggregation of edge features of the self-attention mechanism is analogous to the WL color aggregation. As the reviewer correctly points out, 2-WL is as powerful as 1-WL. As 1-WL cannot distinguish d-regular graphs, 2-WL cannot as well. Hence, it is to be expected that our method fails on such regular graphs if no additional node or edge labels are added (such as the topological distance used in our proposed method).

We thank the reviewer very much for suggesting citing Li et al, "Distance Encoding: Design Provably More Powerful Neural Networks for Graph Representation Learning,". This work gives a thorough justification for adding the topological distance and explains why it enables us to handle regular graphs, while it seamlessly integrates in our message representation. 


 We thank the reviewer for considering our response. However, we still think that the raised concern is based on misunderstanding of our proposed model and hope that this time we can fully address the reviewer’s concern with our response (please also recheck our previous response to this concern, as we try not repeat our line of argument here again). 

### Comment 1:

“I thank the authors for the detailed response. My point "It does not really invent a permutation invariant decoder/generator---the generated graph's ordering still has to be dependent on the input graph" of course means that the decoder+permutor has to be dependent on the input order, and I understand that the pure generator part is not permutation variant because it does not depend on the input graph.
My concern is that, the decoding process still depends on the input graph order. In this sense, this paper is similar to [35]. [35] also learns a permutation invariant encoder (a GNN) as well as a permutation variant decoder (a sequential model generating nodes/edges following the input graph node order).“

#### Response:

We think the reviewer’s main misconception about our proposed method is that the learned decoding process cannot be permutation invariant (i.e. depends on the input node ordering) as we combine the permuter and decoder model to reconstruct the graph in a certain input node order. Hence, the reviewer argues, the overall “decoding process still depends on the input graph order“. However, please note that the permuter is NOT part of the decoding process. We only use the permuter’s output (i.e. a permutation matrix) to permute the decoded graph’s node ordering to match the input graph’s node ordering to calculate the reconstruction loss. The decoded graph order, before applying the permutation matrix predicted by the permuter model, is not dependent on the input graph order. For all n! permutations of the input graph the decoder will – by design - generate the same output graph in the same (learned canonical) order. Only the permuter’s output will change with the input graph order. It is correct to say that the combination of both decoder + permuter is dependent on input order, but this is by design to enable the evaluation of the permutation variant reconstruction loss. However, as we “move” the permutation dependent part (permuter) out of the actual decoding process (decoder), our proposed method still generates graphs independent of the input graph order. Hence, the decoding process does not depend on the input graph order. 

It might also worth mentioning that the permuter’s output - by design - will change equivariantly with the input graph node order. Exchanging two nodes in the input graph will result in the same permutation matrix only with the two corresponding rows exchanged. Hence, the learning process (i.e., the learned weights) of the permuter is also invariant to the ordering of nodes in the input graphs.

Overall, this is in stark contrast to the model proposed in [35]. There, the decoder is an autoregressive model that successively attaches nodes/edges to the decoded graph, maximizing the loglikelihood of nodes and edges of graphs in the same order as present in the training dataset. Hence, the sequential generation process is dependent on the order of nodes in the training data set. To maximize the loglikelihood objective, their model must learn the distribution $p_{data}(G, \pi)$ of graphs in a certain order $\pi$ present in the training dataset (compare equation (13)), such as RDKit’s canonical SMILES order for molecular graphs. Hence, the learned weights of the decoder model will depend on the node order of graphs in the training dataset. As discussed above, this is not the case for our proposed model (neither for the decoder nor for the permuter model), which is the crucial difference.
Interestingly, they showed that for smaller molecular graphs, where the otherwise intractable marginal likelihood $p(G)=\sum_\pi p(G, \pi)$ can be brute forced,  the canonical SMILES order was not always the best and suggest that there is a “big potential for actually learning an ordering” (compare section 5.2). We exactly address this with our proposed model, since in our framework, the decoder learns its own canonical order, independent of the input order, as the permuter decouples the decoding process from the input node order.

Also please note that [35] does not propose an autoencoder architecture. Their decoder is not initialized from an encoded graph representation (although this would be possible, e.g., as a seq2seq model). The graph embeddings extracted by a GNN with pooling as defined in their equations (3) and (4) are extracted for the current decoded graph at every iteration of the graph decoding procedure to inform the next decoding step about the “current state” of the decoded graph (compare section 4.1). Hence, in contrast to our work, [35] is not concerned with the extraction of meaningful graph representations and does not propose a permutation invariant graph generator.

We hope that our response makes our point clear that our proposed model actually learns a permutation invariant graph decoder/generator and is crucially different from decoders like defined in [35].

### Comment 2:

"That said, I now do not think the claim by authors "To the best of our knowledge, our work proposes the first approach for non-contrastive permutation invariant graph-level representation learning, that can also be used for graph generative modeling" is correct. I expect the authors to explain this.“

#### Response:

As mentioned above, [35] does not propose a method for graph-level representation learning. However, it could be adapted in that way, by initializing the decoding process with embeddings extracted by a GNN with node-wise pooling. Still, the issue of the permutation-variant (i.e., input order dependent) decoder would persist (even if the embedding is permutation-invariant), as the learning objective is defined as maximization of the loglikelihood of a nodes and edges in a certain order.

Nevertheless, as correctly pointed out by reviewer bsmb, the work "Conditional Structure Generation through Graph Variational Generative Adversarial Nets," Yang et al. NeurIPS 2019, proposes in theory a model to extract graph-level representations that are used to decode graphs in a permutation invariant way. As mentioned in our response to reviewer bsmb’s comment (please check for more details), they utilize adversarial training to learn a permutation invariant reconstruction loss. In contrast to our work, however, they never investigated the representative capabilities of their extracted embeddings or showed that they extract meaningful representations for downstream tasks. We argue, that unless perfectly trained, the adversarial trained reconstruction loss is inferior to the (true) MSE loss utilized in our work and will lead to worse reconstructions and thus worse graph representations. Still, we acknowledge that our claim about being the “first” is not correct and would like to remove this statement in favor of a discussion of this related work. 


 I thank the authors for the detailed response. My point "It does not really invent a permutation invariant decoder/generator---the generated graph's ordering still has to be dependent on the input graph" of course means that the decoder+permutor has to be dependent on the input order, and I understand that the pure generator part is not permutation variant because it does not depend on the input graph. 

My concern is that, the decoding process still depends on the input graph order. In this sense, this paper is similar to [35]. [35] also learns a permutation invariant encoder (a GNN) as well as a permutation variant decoder (a sequential model generating nodes/edges following the input graph node order). 

That said, I now do not think the claim by authors "To the best of our knowledge, our work proposes the first approach for non-contrastive permutation invariant graph-level representation learning, that can also be used for graph generative modeling" is correct. I expect the authors to explain this.

[35] Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning deep generative models of graphs

 I saw the point. This solves my doubts while introducing some further doubts.  

Transformers work on edges and can achieve 3-WL (2-forklore WL) power ("Provably more powerful neural networks", Neurips 2019). Indeed, they can handle most regular graph cases. Then, why do they still fail as the authors mentioned? Note that 2-WL (not 2-forklore WL) is as powerful as 1-WL.

For how topological distance improving the representation power of graph representation learning, I suggest the authors refer and cite
Li et al, "Distance Encoding: Design Provably More Powerful Neural Networks for Graph Representation Learning," which actually proves that standard GNNs (1-WL) paired with topological distance can handle almost all regular graphs. 

 We thank the reviewer again for reevaluating our work considering our clarifications and additional experiments and increasing their rating. Below you will find our responses to your comment.

### Comment:

„For the response to comment 5, Yes, I missed checking the regular graphs experiments in Figure 2. Sorry! This is also what makes me lean to accept this work. However, I still disagree with some of the authors' augments, which again makes me give a borderline score. From the perspective of representation power, I do not see the significant difference between transformers (dense graphs) and GNNs. They will be both limited by the 1-WL test. So it is not that clear to me how the model indeed helps to associate the nodes of a regular graph with different embeddings. If the node embeddings are the same, how can the model learn an permutation? My current understanding is that the randomness injected into the generative process kind of helps with this part. Things just get entangled here. I expect that the authors may clarify this by showing more details of the experiments or examples in the final version.“

#### Response:

We would like to clarify that the transformer in our proposed method does not simply act on a dense graph (i.e., each node attends to each other node) but rather acts on the edges between nodes (i.e., each edge attends on each other (incoming) edge). Hence, this approach will actually be limited by the 2-WL test as it acts on pairs of nodes and is strictly more powerful compared to regular GNNs that are limited by the 1-WL test (compare Morris et al. 2018 “Higher-order Graph Neural Networks”). But there is another crucial ingredient that makes our model work on regular graph that is unfortunately only mentioned in the supplementary: We include the topological distance between nodes as additional edge feature when initializing the message matrix in the encoder. In an additional experiment we now found that removing this information will make the model fail to learn the reconstruction task for regular graphs. As mentioned by the reviewer, this is since all embeddings will be the same in this case. However, by including the topological distance (i.e., length of shortest path in the original graph) between the nodes connected by the edge, this symmetry breaks and difference between regular graphs can be encoded, and reconstruction is possible. We thank the reviewer for raising this concern and hope this explanation helps understanding the properties of our proposed method. We would like to follow the reviewer’s suggestion and discuss this insight in the manuscript. 



 Okay. It is good to know the key differences. For the issues in their codes, I did not check the codes but if they are the case as the authors mentioned, the authors may want to include that discussion and comparison in the final version, which further demonstrates the contribution of the work under review. Even, the authors may drop a message to Yang et al., the authors of the referred work, to let them know the concerns. 

 We thank the reviewer again for reevaluating our work considering our clarifications and additional experiments and increasing their rating. Below you will find our responses to your comments.

### Comment:

“One paper just came to my mind during the discussion stage when the authors claimed that they are the first one to use graph-level representations to generate graphs:
"Conditional Structure Generation through Graph Variational Generative Adversarial Nets," Yang et al. NeurIPS 2019
It seem that the referred work also considers using a single vector to generate graphs. The decoder is built upon the graph embedding vectors, which may be a more efficient way than learning a permutation but lose some accuracy, compared to the work under review. Hope the authors can clarify the difference and the contributions.“

#### Response:

Indeed, the work referenced by the reviewer does condition a generative model on single-vector (permutation invariant) graph-level representations. The proposed method utilizes a GVAE architecture but averages the latent node features to extract a graph-level representation. They solve the ordering problem in the reconstruction from a graph-level representation by training alongside the GVAE a Generative Adversarial Neural Network (GAN), which’s discriminator network is used to embed input and output graph into a latent space in which a reconstruction loss (i.e. a L2 distance in this space) can be calculated. By parameterizing the discriminator as a permutation invariant function (i.e., a GNN with global pooling at the end), the reconstruction loss is permutation invariant as well. Although this formulation is an interesting approach to tackle the permutation invariance of the problem, it involves adversarial training of the reconstruction objective, which is known to be hard to train (e.g. might suffer from mode collapse). If no desirable discriminator is learned, the reconstruction loss used to train the GVAE will also not be desirable and the GVAE’s encoder and decoder might not be desirable. Only if trained perfectly, the reconstruction loss will be equivalent to the MSE loss used in our work. Hence, as the reconstruction objective has to be first learned in a possible unstable adversarial learning protocol, we would argue that the referenced approach is not necessarily more efficient than our proposed method. From the results in the referenced paper and its supplementary, input and output graph usually seem relatively unsimilar compared to results shown in this work, which could be attributed to the approximated/learned reconstruction loss. Hence, differences between similar graphs might not be represented well in the extracted representations. We think the referenced work’s main motivation was to generate graphs conditioned on a certain graph (i.e. similar to that graph) and neither to perfectly reconstruct the graph nor learn expressive graph-level representations (no analysis of extracted embeddings was done, input and output graphs were only compared based on observables like LCC (size of largest connected component), TC (triangle count), CPL (characteristic path length), MD (maximum node degree)). On the contrary, our main motivation is to learn expressive graph-level representations and we analyze exact reconstruction performance (which can be seen as a proxy how well even similar graphs can be distinguished in the embedding). Hence, we argue that our work is substantial different with respect to its scope and contributions.
Moreover, from carefully inspecting the published code of the referenced work it seems that, other than claimed, the node embeddings are not averaged as defined in Eq. (1) but individually used to sample initial node embeddings used by the decoder (compare Encoder class in https://github.com/KelestZ/CondGen/blob/master/models.py#L49 or code in supplementary material). This would make the approach again permutation variant and easier to train. We would be very interested to know what the reviewer thinks about this. 
Still, we acknowledge that our statement about our work being the first to address graph generation from a graph-level representation is not correct and would like to remove this statement from the related work section and add a discussion of the referenced work instead. We thank the reviewer for making us aware of this work.


 Thank authors for the clarification. It solves some questions in my mind though not all of them. I would increase my score to 6 to support the acceptance because of the further clarification. But I would like to clarify a few disagreements regarding authors' response. 

For the response to comment 1, I totally understand what the paper told and what the authors' response said. I think we are on the same page.

For the response to comment 2, yes, this is a point that makes me change my mind because I did not notice the node masking part. 

For the response to comment 3, I still do not agree with the authors' statement. It seems to me as a common practice to choose a more efficient/approximate way to compute Hungarian matching instead of the exact Hungarian matching, such as the sinkhorn algorithm and etc.. This is standard and not new. This is one reason why I rank this work in the borderline and cannot get a higher rate. 

For the response to comment 4, we are on the same page.  Different people evaluate the novelty of a work in different ways. 

For the response to comment 5, Yes, I missed checking the regular graphs experiments in Figure 2. Sorry! This is also what makes me lean to accept this work. However, I still disagree with some of the authors' augments, which again makes me give a borderline score. From the perspective of representation power, I do not see the significant difference between transformers (dense graphs) and GNNs. They will be both limited by the 1-WL test. So it is not that clear to me how the model indeed helps to associate the nodes of a regular graph with different embeddings. If the node embeddings are the same, how can the model learn an permutation? My current understanding is that the randomness injected into the generative process kind of helps with this part. Things just get entangled here. I expect that the authors may clarify this by showing more details of the experiments or examples in the final version.      
 

 Thanks for the detailed explanation. Reviewer oyaX also reminds me of the significance of interpolation, which I agree. I am also happy with the experiment results.

One paper just came to my mind during the discussion stage when the authors claimed that they are the first one to use graph-level representations to generate graphs:

"Conditional Structure Generation through Graph Variational Generative Adversarial Nets," Yang et al. NeurIPS 2019

It seem that the referred work also considers using a single vector to generate graphs. The decoder is built upon the graph embedding vectors, which may be a more efficient way than learning a permutation but lose some accuracy, compared to the work under review. Hope the authors can clarify the difference and the contributions.

This work proposes a new graph autoencoder approach, which is claimed to have invariant graph embedding. The work claims to solve the permutation matching challenge during the decoding stage.  Originality:
The idea of graph auto-encoder is not new, but I agree that decoding graph structures from an invariant graph representations is challenging. However, the proposed method does not actually solve the problem. Though the work is claimed to learn invariant embedding, the decoding procedure is actually generated from some non-invariant embedding (m_{ij}, i,j>0), which traditional GAE also does. 

Quality:
The claim seems to mismatch the method proposed. Specifically, the decoding procedure leverages much information that are not in the invariant embedding z (say using m_{ij}, i,j>0), which essentially implies that the so-called embedding z is not the graph embedding, while the entire matrix M (which is not invariant) is the graph embedding. Note that the proposed method also does not match the diagram in figure 1 because in Eq.13, the edges are generated from m_{ij}, i,j>0, i \noteq j, which is not reflected by figure 1. Therefore, the proposed method does not match the claim and I lean to reject this work. 

Because of this issue, I do not carefully check the experiments.

Clarity:
It is not hard to follow though the mismatching claims as explained above are kind of confusing. 

Significance:
Because of the technical issues of this work, the significance of this work cannot be fairly evaluated. It seems that the authors try to do extensive experiments to show the advantages of their model, which should be appreciated. But the technical problem severely hurts the significance of this work. 

 The work has adequately addressed the limitations and potential negative impact. 

 As some reviewers had concerns about the significance of unsupervised-learned graph-level representation over pooled unsupervised-learned node-level representation, we ran an additional experiment to demonstrate its superiority. In Section 4.1 we compare different graph-level representations for a graph classification problem. We repeated the experiment for representations learned from a classical GAE trained on the same dataset as our model (and the state-of-the-art graph constative learning method InfoGraph). Like the other methods we used the extracted representations of the trained GAE model (averaged node representations) to train a Support Vector Machine on predicting the different graph classes, following the same hyperparameter optimization scheme. This model achieved an average accuracy of 0.65 +/- 0.01, significant worse compared to our proposed model (0.83 ± 0.01). We think this experiment is an import addition to our manuscript, supporting our claims that pooled local node-level representations are inferior to a global graph-level representation for such tasks as discussed above. We would like to include these results and an additional discussion (of pooled local node-level vs global graph-level representations) to the manuscript and thank the reviewers for making us aware of this missing justification/discussion. 
We hope that the additional clarification and experiments address the reviewers concerns sufficiently and kindly ask the reviewer to consider increasing their evaluation score. 

 As some reviewers had concerns about the significance of unsupervised-learned graph-level representation over pooled unsupervised-learned node-level representation, we ran an additional experiment to demonstrate its superiority. In Section 4.1 we compare different graph-level representations for a graph classification problem. We repeated the experiment for representations learned from a classical GAE trained on the same dataset as our model (and the state-of-the-art graph constative learning method InfoGraph). Like the other methods we used the extracted representations of the trained GAE model (averaged node representations) to train a Support Vector Machine on predicting the different graph classes, following the same hyperparameter optimization scheme. This model achieved an average accuracy of 0.65 +/- 0.01, significant worse compared to our proposed model (0.83 ± 0.01). We think this experiment is an import addition to our manuscript, supporting our claims that pooled local node-level representations are inferior to a global graph-level representation for such tasks as discussed above. We would like to include these results and an additional discussion (of pooled local node-level vs global graph-level representations) to the manuscript and thank the reviewers for making us aware of this missing justification/discussion. 
We hope that the additional clarification and experiments address the reviewers concerns sufficiently and kindly ask the reviewer to consider increasing their evaluation score. 


 ### Comment 6:

„Compared with (V)GAE: The key technical difference between this work and (V)GAE is that (V)GAE keeps all the node correspondence between the input graph and output graph. This work instead pools node representations together and then uses [14][15] to learn a set matching to decode the pooled results. My feeling for this is that this work makes a harder problem and proposes a solution to this harder problem by combining previous set-mathcing techniques. However, it is unclear why the harder problem is indeed crucial and has practical usage. For this point, I kind of agree with reviewer 4.“

#### Response:

Indeed, our proposed model is harder to learn than a classical GAE. But the crucial difference between such a classical GAE and our proposed model is that we learn a single-vector permutation invariant graph-level representation, which is not the case in classical GAEs. 
We strongly believe that such a single-vector graph-level representation which can also be used for reconstruction/generation is a desirable property and has practical usage. We understand the reviewer’s concern that every node-level representation learning model can be made graph-level by simply pooling the node embeddings. However, pooling is a rather crude procedure that inherently assumes that a graph-level representation (global structure) can be expressed by a simple aggregation (average, sum, max) schematic over individual (local) node representations. This, however, is in general not the case and might lead to less expressive representations (e.g., discussed in the mentioned Set Transformer paper or Vinyals et al., https://arxiv.org/abs/1511.06391). Such pooling methods mainly have proven useful if used in a supervised setting, where node representations can be learned that can be pooled in a meaningful way. In an unsupervised setting this cannot be done easily. Recently, this has led to more sophisticated unsupervised graph-level representation learning methods, namely contrastive learning methods as discussed in the related work section. 
Contrastive learning objectives can be used, like supervised learning, to learn more meaningful aggregations of node features into a graph-level representation. However, those models cannot be used for graph generation. We think that for graph reconstruction/generation a single-vector (thus permutation invariant) graph-level representation is desirable as well. In the present manuscript we do not explore much the generative aspect of our proposed model, but one interesting property that can be exploited with our proposed model is that we are able to interpolate between graphs. Since we have one permutation invariant vector representing graphs that can be encoded from and decoded to graphs, independently of their node ordering, we can efficiently interpolate between graphs in the embedding space without considering their node order. In an autoencoder framework without permutation invariant embedding, there is no clear way how to interpolate, as graphs would need to be aligned first.
Since we are not able to upload figures to open review, we uploaded a depiction of an example interpolation between two graphs with a description of the experiment on figshare (private anonymous link: https://figshare.com/s/ee86702825d67aa959c1). To the best of our knowledge, such a graph interpolation in the latent space has not been report in other work yet and might show similar impact in the graph generation community as interpolation of latent spaces did in the NLP or image filed. 
Moreover, a single-vector graph representation naturally handles different sized graphs compared to node-level graph embedding that increases in size with the number of nodes.

As some reviewers had concerns about the significance of unsupervised-learned graph-level representation over pooled unsupervised-learned node-level representation, we run an additional experiment to demonstrate its superiority. In Section 4.1 we compare different graph-level representations for a graph classification problem. We repeated the experiment for representations learned from a classical GAE trained on the same dataset as our model (and the state-of-the-art graph constative learning method InfoGraph). Like the other methods we used the extracted representations of the trained GAE model (averaged node representations) to train a Support Vector Machine on predicting the different graph classes, following the same hyperparameter optimization scheme. This model achieved an average accuracy of 0.65 +/- 0.01, significant worse compared to our proposed model (0.83 ± 0.01). We think this experiment is an import addition to our manuscript, supporting our claims that pooled local node-level representations are inferior to a global graph-level representation for such tasks as discussed above. We would like to include these results and an additional discussion (of pooled local node-level vs global graph-level representations) to the manuscript and thank the reviewers for making us aware of this missing justification/discussion.


 Thank you very much for reevaluating our work considering our clarification. However, we still think that some essential parts of our proposed model are misunderstood by the reviewer.  We hope that after clarifying these raised concerns as well the reviewer will consider a higher evaluation score. Below you will find our responses to your comments.


### Comment1: 
“First, there is a permuter that learns a canonical order of the nodes in the input graph. Second, the decoder will leverage positional encodings to generate different representations of nodes, where positional encoding essentially imposes a canonical order. The learning procedure expects to match these two canonical orders.“

#### Response:

We would like to emphasize that the permuter’s task is to learn the decoders canonical order (output graph order not input graph order as stated by the reviewer) and match the input graph nodes to this canonical order. The input has no canonical order and the whole proposed model is invariant to any ordering bias in the input graphs. It is also not correct to say that the “positional encoding essentially imposes a canonical order“. The decoder learns a canonical order that is not imposed but aligned to the sequence of position embeddings used to initialize the decoders message matrix. The decoder is equivariant to permutations of its inputs, thus permuting the position embeddings will not change the learning behavior of the decoder and its learned canonical order. We only permute the position embeddings to align the decoded graph to the input graph order. We emphasize, that the permuter model (i.e. the permutation of the position embeddings) is not necessary for the decoding process in itself.

### Comment 2:

“The method requires that the output graph may have the exact number of nodes as the input graph, where essentially the distributions of node numbers and any scale-free properties of graphs will definitely be missed.“

#### Response:

We think this comment can also be explained by a misunderstanding of our proposed model architecture. The proposed method can model different sized graphs by utilizing masking of not present nodes. In general, a batch of graphs is represented by the message tensor with dimensions [batch_size, n, n, d], where n is the number of nodes of the largest graph in the batch and d the number of message features. The smaller graphs are padded to this size by “adding empty nodes” that are masked out in every operation. Since the proposed architecture is an autoencoder and thus trained on a node/edge-wise reconstruction loss, we have to somehow compare the same number of nodes in the input and output. The reviewer might be concerned that, during training, the decoder knows how many nodes to decode since this information is indirectly passed by defining the mask used by the decoder based on the input graph. This could lead to a model that does not learn the distribution of nodes in the graph dataset. However, to address this, we train alongside the autoencoder an additional MLP that predicts the number of nodes from the embedding (see line 187-190). That way we learn the distribution of nodes in the graphs and force the encoder to reflect this in the embedding. During inference/generation, the MLP’s prediction (e.g. for a sampled embedding) can then be used to define the mask used by the decoder and generation of different sized graphs can be done. Hence we would like to stress that the reviewer’s statement “the distributions of node numbers and any scale-free properties of graphs will definitely be missed.“ is incorrect. 


### Comment 3:

Given the same number of nodes of the input and output graphs, a permuter is not needed to learn anymore. Why don't we directly compute the matching between the two node representation sets, i.e., the node representations of the input graph and the node representations of the output graph. One may use Hungarian matching. 

#### Response:

Indeed, one could also use an optimal assigning/matching algorithm to find the permutation matrix $P_{\pi \rightarrow \pi’}$  that best matches input and output order to calculate the reconstruction loss. As discussed in the problem definition section (line 76-80), this was proposed by Simonovsky et al. 2018. However, such a matching, which involves up to O(n^4) re-ordering operations in the worst case (see Cho et al. 2014) (or O(n^3) for Hungarian algorithm), must be performed at EVERY backpropagation step for each individual graph and thus, is not computationally feasible, especially for larger graphs (as discussed in Simonovsky et al. 2018 or Cho et al. 2014). The motivation behind our proposed model is to solve this limitation, by learning a relaxed, continuous matching hat is differentiable and thus, can be trained alongside the graph autoencoder. In contrast to using an optimal assigning/matching algorithm which scales with O(n^4) or O(n^3) in the case of the Hungarian algorithm, our proposed (learned) matching algorithm only scales with O(n^2).

### Comment 4:

„My understanding of the goal of the permuter is to learn such a set matching. The techniques to build the permuter proposed in [14][15] are indeed proposed to learn set matching. They need to learn that because in their setting, the two sets are not accessible. They access to only the input set but not the output one, which I think is different from the setting considered in this work.“

#### Response:

It is correct, that the ultimate goal of the permuter is to learn a set matching between the set of input nodes and the set of output nodes. Please note, however, that the permuter model only has access to the input nodes and not to the output nodes. Hence, it would be more correct to say the permuter learns to sort the input nodes.
The works [14][15] are (also) not proposed to learn set matchings but to learn to sort sets (although one could argue that those two tasks are very similar, as both sorting and matching operations can be defined as permutations). For example, in one experiment they train a model to sort a set of MNIST images, predicting the permutation matrix that would sort these images. There is no set they are matching to, although one could argue that they are learning to match the set of MNIST images to the set of positions in the sorted sequence. This is very similar to our use case, as our proposed permuter model learns to match the set of input nodes to the set of position embeddings (which is a sorted sequence) that the decoder aligns its learned canonical node ordering to.

### Comment 5:

„Suppose we may use the above set matching. There are several issues when applied to graphs, as the set matching is in P while graph matching is not. For example, a direct result is as follows: if we want to learn the distribution of random regular graphs, due to the limited representation power of GNNs and the way proposed in this work to learn the permutation, there will be no meaningful canonical orderings. This will yield a failure to learn the distribution. I guess for random regular graphs, no results like Figure 2 can be generated.“

#### Response:

We want to clarify two wrong assessments in this comment:

First, we do not use GNNs in this work, but Transformers acting on the message matrix extracted from a graph. In contrast to GNNs our proposed model acts also on connections between nodes that are not connected in the graph. Moreover, the final embedding is extracted considering all these connections, which addresses some of the limitations of GNNs (e.g. as pointed out by Xu et al. 2019, “HOW POWERFUL ARE GRAPH NEURAL NETWORKS?”) such as representing symmetric nodes in graphs, which cannot be represented e.g. by GNNs that utilize max-pooling.

Second, the comment “I guess for random regular graphs, no results like Figure 2 can be generated.“ is incorrect, as we apply our model on random regular graphs and show results in mentioned Figure 2.

We are not sure what the reviewer means with “There are several issues when applied to graphs, as the set matching is in P while graph matching is not“. If we have a permutation matrix P that can match input and output nodes we can also match input and output graph by permuting node features and adjacency matrix as defined in Equation (1). 
We understand that for graphs with symmetric nodes/branches (e.g., in regular graphs) no single canonical order exist as we can permute symmetric nodes without changing the internal representation of that graph in the decoder. Or in other words, the decoder has no way of telling which of the symmetric nodes to decode first. However, this poses no problem if the node ordering with respect to such nodes stays consistent (which can be learned). For example, a 4-node k-regular graph with k=2 (i.e., a square) with nodes indexed from 1 to 4 clockwise is one possible canonical ordering that can be decoded. Even if the symmetric nodes 2 and 4 are exchange, the adjacency matrix stays the same and the reconstruction loss can be calculated. 




 We thank the reviewer for their review and the encouraging assessment our work. Below you will find our responses to your comments.

### Comment 1:
“Although experimentation is done on variety of tasks, in my opinion experimentation on graph property prediction (graph classification) tasks is lacking. It would really help if you could present the results on several open graph benchmark datasets.“

#### Response:

In the manuscript we focused on evaluating our proposed model on a variety of different tasks to highlight its many possible applications – not limited to graph property prediction (the reviewer might also be interested in the new experiment we conducted on graph interpolation with a figure of an example interpolation accessible on figshare (private anonymous link: https://figshare.com/s/ee86702825d67aa959c1)). Still, we acknowledge this reviewer’s comment and agree that a more thorough evaluation of the model on a standard graph property prediction benchmark would strengthen the manuscript. We ran the ogb-molhiv task utilizing the pretrained model used in section 4.2. for molecular property prediction. We used this pretrained model to extract graph representations for the molecules and trained a Support Vector Machines on this classification task. Following the OGB protocol we achieved a test accuracy of 0.793 corresponding to a Top-10 position on the leaderboard, outperforming regular GNN networks. Please note that we did not perform any fine-tuning on the training dataset nor extensive hyperparamter tuning of the SVM. We noticed, that the ogb-molhiv dataset also includes atom types like Cu that were not part of our training and had to be masked out and graphs of size up to over 200 atoms which also lies out of the training data domain (up to 32 atoms) of the utilized model. Hence, we argue that finetuning on this data would help increasing the performance significantly. However, we are unfortunately not sure if finetuning or pretraining on a more appropriate dataset can be finished within the rebuttal period. Similarly, the ogb-pcba task involves multi-task training, that is not easily done with SVMs but could be done with finetuning on these tasks. We plan to include these experiments once they are finished to the manuscript and thank the reviewer for this valuable suggestion.

### Comment 2:

„The diagonal elements of input message matrix to the decoder are initiated with the summation of positional embeddings and latent representation z. How are the off diagonal messages initialized?„

#### Response:

We think this part was not clear enough in the manuscript. The whole massage matrix M used by the decoder is initialized the following:
$ m_{i,j} = z + [PE(i)||PE(j)].$
Hence, the off-diagonal elements are initialized by the embedding z and the concatenation of two different position embeddings, inducing the information which two nodes are connected, as off-diagonal entries represent edges (connected and not connected) of the graph. We will include this equation to section 2.3 to make this part clearer. We thank the reviewer for pointing this out. 


 Thanks for the response.
I agree that the single-vector permutation-invariant representation can help to interpolate between graphs and to my knowledge it has not been reported in the prior works. 
I will consider it during the discussion period to make my final decision.

 We thank the reviewer for their comments. Below you will find our responses to your comments and proposed changes to our manuscript according to your valuable suggestions.

### Comment1: 
“The paper is well written in general. However, some parts can be written better. For example, the discussion on solving the graph isomorphism problem can be made more clear. Why "if the decoder can perfectly reconstruct both graphs, two non-isomorphic graphs must have a different representation z"?“

#### Response:
Thanks for pointing this out. We apologies if the line of argument in this section is hard to follow. The whole idea of this argument is that two isomorphic graphs will always have the same embedding since the encoder is permutation invariant by design. However, we must consider the possibility, that two non-isomorphic graphs could wrongly be encoded to the same embedding. Since, the decoder is a deterministic function, the same embedding will produce the same decoded graph up to a permutation as defined by the predicted permutation matrix. Thus, if both graphs have the same embedding z and can be perfectly reconstructed by the decoder, they must be isomorphic, as they can only differ in their node ordering. Consequently, if they can be perfectly reconstructed by the decoder but have a different embedding, they must be non-isomorphic. 

We hope this answers the reviewer’s question. We will add this additional explanation in the manuscript.

### Comment2: 
“Also, Section 2.3 puts the overall architecture to a later part, while the previous architecture designs are hard to understand without the overall architecture. I suggest re-organizing this section by introducing the overall architecture first.“

#### Response: 
This is a good point. We will reorganize the sections as suggested by the reviewer. Thank you for this comment, improving our manuscript!

### Comment3: 
“I have some doubt on the significance of the results. If we only want a permutation invariant encoder, we have many ways to achieve this as stated in the related work (e.g., not relying on reconstructing the input graph). If we want a generative model, we also do not need to care about the node ordering as long as the the generative model learns the training graph distribution correctly (e.g., always generates graphs with canonical node ordering). “

#### Response: 
We strongly believe that an unsupervised learned graph-level representation which can be also used for graph generation is desirable and significant to the graph representation learning and graph generation community.


One the one hand, the whole field of unsupervised representation learning has emerged from the believe and evidence that unsupervised learning can be used to extract meaningful representations for downstream tasks. Especially for tasks where labeled graphs are scarce but unlabeled graphs abundant (e.g. molecules), pretraining a model in an unsupervised way can be useful to learn powerful feature extractors that can then directly be used or finetuned to solve a task.

We acknowledge that every unsupervised-learned node-level representation learning model can be made graph-level by simply pooling the node embeddings. However, pooling is a rather crude procedure that inherently assumes that a graph-level representation (global structure) can be expressed by a simple aggregation (average, sum, max) schematic over individual (local) node representations. This, however, is in general not the case and might lead to less expressive representations (e.g., discussed in Lee et al. arXiv:1810.00825 and Vinyals et al. https://arxiv.org/abs/1511.06391).

In related works, pooling for graph-level representations is usually explored in a supervised context, where the model can learn to embed nodes in such a way to make a certain aggregation scheme work. In this work, however, we are concerned with unsupervised learned graph-representations. Hence, learning graph-level representations by pooling node-level representations would either require supervision, which is not the scope of this paper and might fail for data scarce problems or a simple aggregation of node-level representations (e.g. classical GAE-like approaches) that could miss important global structure.Thus, we argue that an unsupervised-learned representation of graphs that takes global structure into account as inductive bias as explored in this paper is desirable and significant. 


There exist other graph-level unsupervised representation learning methods that were recently proposed and show strong performance in representing graphs, namely contrastive learning methods as discussed in the related work section. However, those models cannot be used for graph generation. We argue that for graph reconstruction/generation a single-vector (thus permutation invariant) graph-level representation is desirable as well. In the present manuscript we do not explore much the generative aspect of our proposed model, but one interesting property that can be exploited with our proposed model is that we are able to interpolate between graphs. Since we have one permutation invariant vector representing graphs that can be encoded from and decoded to graphs, independently of their node ordering, we can efficiently interpolate between graphs in the embedding space without considering their node order. In an autoencoder framework without permutation invariant embedding, there is no clear way how to interpolate, as graphs would need to be aligned first.
Since we are not able to upload figures to open review, we uploaded a depiction of an example interpolation between two graphs with a description of the experiment on figshare (private anonymous link: https://figshare.com/s/ee86702825d67aa959c1). To the best of our knowledge, such a graph interpolation in the latent space has not been report in other work yet and might show similar impact in the graph generation community as interpolation of latent spaces did in the NLP or image filed. 
Moreover, a single-vector graph representation naturally handles different sized graphs compared to node-level graph embedding that increases in size with the number of nodes.

### Comment 4: 
“In this sense, this work seems to simply combine a permutation invariant encoder (for graph representation learning) and a permutation dependent "permuter"+decoder (for generating the graph with the input node order). It does not really invent a permutation invariant decoder/generator---the generated graph's ordering still has to be dependent on the input graph.“

#### Response: 
I’m not sure if I understand this comment correctly, but we would like to clarify and emphasize that the decoder does not depend on the input graph ordering. If we do not take the permuter into account, the generated/reconstructed graphs are independent of the input order. The decoder learns its own canonical ordering (that does not depend on any node order bias in the input graph distribution), as the embedding used to initialize the decoders message matrix is permutation-invariant. This canonical order is aligned to the position embeddings added to the initialized message matrix. We emphasize, that the permuter model (i.e. the permutation of the position embeddings) is not necessary for the decoding process in itself. We only need it to align input and output graphs to calculate the loss. During inference, i.e., using the trained decoder as generative model, we no longer need/use the permuter model (We do not permute the position embeddings). Thus, the trained decoder is permutation invariant, in that it does not depend at all on the distribution of node orderings in the input graphs. 
Hence, we would like to stress that this reviewer’s statement:

“It does not really invent a permutation invariant decoder/generator---the generated graph's ordering still has to be dependent on the input graph“

is incorrect. We think that the part explaining this (section 2.3 – Key Architectural Properties) was not clear enough and we would like to rework the manuscript accordingly. We thank the reviewer for making us aware of that. 


### Comment5: 
“The authors state in the related works that GAE-like approaches construct graph representations by concatenating node embeddings, thus is not permutation invariant. But why can't we sum/average the node embeddings like in graph pooling layers?“

#### Response: 
A classical GAE-like approach is trained on reconstructing the graph from node embeddings. E.g., in the original model proposed by Kipf and Welling, the adjacency matrix is reconstructed based on the dot product between the node embeddings. Thus, the model learns a local representation for each node to decide if two nodes are connected. Once the model is trained, these node-level representations could be summed/averaged to get a graph-level representation. However, as discussed in the response to Comment 3, this pooling of local representations can lead to a less expressive global representation. During training, one cannot average/sum node representations to get a graph-level representation, as the permutation-invariant graph cannot be reconstructed from such a pooled representation due to the permutation variance of the loss, as discussed in our manuscript. We address exactly this issue with our proposed permutation invariant GAE model. 



  #### Comment 5: 
“Sometimes the formulation is not neat. For example, P is the permutation matrix. But, it has been also used as a transformation on G in Eq (4) and Eq(5), which is not neat.“

#### Respond: 
We thank the reviewer for pointing this out, helping us to increase the quality of our manuscript. We will rework the notations to make them more concise (see also Respond to Reviewer bsmb). Regarding permutation matrix P, we will change the notation to $\hat{P}$ if we mean the transformation operator.

### Comment 6: 
„I am not sure Eq (5) is correct. Since G does not have any ordering (π), does it makes sense to apply P? Also, I am not sure using $g_\psi(G^{(i)})$ for taking the expectation is correct. P might be a deterministic function of G.“

#### Response: 
Thank you for this comment. I think the reviewers concern can be explained with the inconcise notation in Equation (5).  In Equation (5) we omit the subscripts defining the node order $\pi$ for G and the subscript $\pi’ \rightarrow \pi$ for P to make the equation more readable. Since, this understandably led to some confusion as it is less concise, we will reformulate the equation, including the subscripts. Moreover, as the reviewer noted, using $g_\psi(\mathbf{P}|G^{(i)})$ in the subscript of the expectation might be confusing in this context. We reformulate Equation (5) defining $\mathbf{P}$ separately as output from the permuter model $g_\psi(G^{(i)})$ and move this part out of the expectation. We hope that the reviewer finds this new formulation more concise and thank again for these remarks.

### Minor Comment 1:
“I did not understand how an additional node and edge "type" is added in the following line:

o	Line 152: "To distinguish the embedding node from other nodes, we add an additional node and edge type to represent this node and edges to and from this node."

•	Line 152, it is written "To distinguish the embedding node from other nodes, we add an additional node and edge type to represent this node and edges to and from this node."

o	I did not understand how an additional node and edge "type" is added“

#### Response: 
The additional node and edge type is added by including an additional dimension in the one-hot encoding of node and edge types. E.g., for graphs with three edge types, we describe each edge by a four dimensional one-hot encoded vector where the 4th element is only equal to one (and the rest zero) if the edge is connected to the embedding node $v_0$.

### Minor Comment 2:
“In line 168, it is written "We add to each entry in the message matrix mi,j the concatenated position embeddings [PE(i)||PE(j)]."

o	Is z also used in mi,j?"

#### Response: 
Yes, $m_{i,j}$ is the sum of z and [PE(i)||PE(j)]. In the present manuscript we write in line 165: „We initialize the message matrix for the decoder models input with the latent representation z at each entry”.
To make this point more clear, we will add another equation defining the initial messages $m_{i,j}$:

$ m_{i,j} = z + [PE(i)||PE(j)].$



 Thanks for your thoughtful review. Below you will find our responses to your comments and proposed changes to our manuscript according to your valuable suggestions.


### Comment 1:
“I think the main novelty is in the imputer model and using position embedding to reconstruct the graph nodes from the single representation vector. Using the self-attention on directed messages for encoding/decoding is incrementally novel. 
I think using the imputer model to match node orderings of the input and output graph, and using the position embedding to inject information about the position of the nodes for decoding can give a general framework for having permutation-invariant graph representation learning models.

•	Use any of the existing permutation-invariant graph models for graph encoding (If the model is for node-level encoding, the node representations can be aggregated using simple pooling or more complicated permutation-invariant pooling such as Pooling by Multi-head Attention used in Set Transformer paper).

•	Use the permuter to find the permutation matrix.

•	Add position encoding (according to the permutation matrix) to the graph representation to find initial node embeddings for decoder.

•	Use any existing equivariant decoding model for graph reconstruction.
“

#### Response: 
This is correct, we also think our main contribution is the proposed framework combining an encoder (with a graph-level permutation invariant embedding), permuter and decoder model as described in the manuscript. We chose transformers for the encoder and decoders, but there are other options to parameterize the different networks.

### Comment2: 
“In the abstract and introduction, it is written that graph-level unsupervised learning has not received much attention. I think this is not a correct statement to make. I think as the authors explained, any graph generation model or permutation-invariant node-representation learning model (with pooling) can be categorized under graph-level unsupervised learning. I think the right statement is that using unsupervised learning for single-vector graph-level representation which can be also used for reconstruction/generation has not been explored much.“

#### Response: 
We understand the reviewers concern with this statement, as every node-level representation learning model can be made graph-level by simply pooling the node embeddings. However, pooling is a rather crude procedure that inherently assumes that a graph-level representation (global structure) can be expressed by a simple aggregation (average, sum, max) schematic over individual (local) node representations. This, however, is in general not the case and might lead to less expressive representations (e.g., discussed in the mentioned Set Transformer paper or Vinyals et al., https://arxiv.org/abs/1511.06391).
In supervised learning (e.g. graph-level classification), the model can learn to assign the best node representations to make a certain aggregation work or even learn a aggregation from scratch (like the mentioned Multi-head Attention Pooling used in the Set Transformer paper), but in unsupervised learning more effort must be spend, such as contrastive learning approaches or the method proposed in this work. Thus, we would still argue that most attention in graph-level representation learning has been received in supervised learning. However, the reviewer correctly points out that generative modeling falls in the field of unsupervised learning. Hence, we would like to change the statement to “graph-level unsupervised representation learning”.

### Comment 3:
“I think the right statement is that using unsupervised learning for single-vector graph-level representation which can be also used for reconstruction/generation has not been explored much. But, I think one of the main reasons why this has not been investigated much is that it is not necessarily required to have a permutation-invariant single-vector representation to solve the downstream tasks. For example, any of the models which gives equivariant node-level representations may be used to solve the problems explored in this paper. For the problems such as graph classification/regression/similarity which needs single-vector graph representation, the node-level representations can be aggregated using simple pooling or more complicated permutation-invariant pooling such as Pooling by Multi-head Attention used in Set Transformer paper. On the other hand, for graph generation/reconstruction tasks, they can use their node-level encodings.“

#### Response: 
We strongly believe that “using unsupervised learning for single-vector graph-level representation which can be also used for reconstruction/generation“ will lead to a desirable entity to extract from graphs.

One the one hand, the whole field of unsupervised representation learning has emerged from the believe and evidence that unsupervised learning can be used to extract meaningful representations for downstream tasks. Especially for tasks where labeled graphs are scarce but unlabeled graphs abundant (e.g. molecules), pretraining a model in an unsupervised way can be useful to learn powerful feature extractors that can then directly be used or finetuned to solve a task. Following our line of arguments in our previous respond (Comment 2), apart from contrastive learning approaches, meaningful node embedding aggregation schemes for graph-level representations in current state-of-the-art method are all based on supervised learning. Hence, solving the tasks explored in our paper with “simple pooling or more complicated permutation-invariant pooling such as Pooling by Multi-head Attention used in Set Transformer paper“, would either require supervision, which is not the scope of this paper and might fail for data scarce problems or a simple aggregation of unsupervised learned node-level representations (e.g. classical GAE-like approaches) that could miss important global structure.

On the other hand, we think that for graph reconstruction/generation a single-vector (thus permutation invariant) graph-level representation is desirable as well. In the present manuscript we do not explore much the generative aspect of our proposed model, but one interesting property that can be exploited with our proposed model is that we are able to interpolate between graphs. Since we have one permutation invariant vector representing graphs that can be encoded from and decoded to graphs, independently of their node ordering, we can efficiently interpolate between graphs in the embedding space without considering their node order. In an autoencoder framework without permutation invariant embedding, there is no clear way how to interpolate, as graphs would need to be aligned first. We would like to add this experiment to the manuscript or supplementary to demonstrate this feature.
Since we are not able to upload figures to open review, we uploaded a depiction of an example interpolation between two graphs with a description of the experiment on figshare (private anonymous link: https://figshare.com/s/ee86702825d67aa959c1). To the best of our knowledge, such a graph interpolation in the latent space has not been report in other work yet and might show similar impact in the graph generation community as interpolation of latent spaces did in the NLP or image filed. 
Moreover, a single-vector graph representation naturally handles different sized graphs compared to node-level graph embedding that increases in size with the number of nodes.

 ### Comment 4: 
“In recent years, there have been various permutation-invarinat models which tries to give equivariant node representations. In addition to the ones cited in the paper (e.g. [21]), I recommend to discuss the following papers in Section 3 (Related works):

•	Tony Duan and Juho Lee, Graph Embedding VAE: A Permutation Invariant Model of Graph Structure.

•	Jenny Liu, et al. Towards Permutation-Invariant Graph Generation

•	Chenhao Niu, et al. Permutation Invariant Graph Generation via Score-Based Generative Modeling

It may be also good to cite these papers

•	Juho Lee, et al. "Set transformer: A framework for attention-based permutation-invariant neural networks." International Conference on 
Machine Learning. PMLR, 2019.

•	Rex Ying, et al. "Hierarchical graph representation learning with differentiable pooling." arXiv preprint arXiv:1806.08804 (2018).“


#### Response: 
We thank the reviewer for these suggestions. We will add them into the related work discussion.


 Based on the clarification of the notations, I understand the proposed method better. Basically, there are two important techniques that make the proposed autoencoder able to decode a graph from a single vector. First, there is a permuter that learns a canonical order of the nodes in the input graph. Second, the decoder will leverage positional encodings to generate different representations of nodes, where positional encoding essentially imposes a canonical order. The learning procedure expects to match these two canonical orders. 

Unfortunately, even after I understand this technique, I still cannot increase my evaluation score. The reasons are as follows:
1. The method requires that the output graph may have the exact number of nodes as the input graph, where essentially the distributions of node numbers and any scale-free properties of graphs will definitely be missed.

2. Given the same number of nodes of the input and output graphs, a permuter is not needed to learn anymore. Why don't we directly compute the matching between the two node representation sets, i.e., the node representations of the input graph and the node representations of the output graph. One may use Hungarian matching. My understanding of the goal of the permuter is to learn such a set matching. The techniques to build the permuter proposed in [14][15] are indeed proposed to learn set matching. They need to learn that because in their setting, the two sets are not accessible. They access to only the input set but not the output one, which I think is different from the setting considered in this work. 

3. Suppose we may use the above set matching. There are several issues when applied to graphs, as the set matching is in P while graph matching is not. For example, a direct result is as follows: if we want to learn the distribution of random regular graphs, due to the limited representation power of GNNs and the way proposed in this work to learn the permutation, there will be no meaningful canonical orderings. This will yield a failure to learn the distribution. I guess for random regular graphs, no results like Figure 2 can be generated. 

4. Compared with (V)GAE: The key technical difference between this work and (V)GAE is that (V)GAE keeps all the node correspondence between the input graph and output graph. This work instead pools node representations together and then uses [14][15] to learn a set matching to decode the pooled results. My feeling for this is that this work makes a harder problem and proposes a solution to this harder problem by combining previous set-mathcing techniques. However, it is unclear why the harder problem is indeed crucial and has practical usage. For this point, I kind of agree with reviewer 4. 

 




 We thank the reviewer for their review. However, it appears that, unfortunately, the reviewer misunderstood a crucial part of our proposed model architecture. We hope we can clarify this misunderstanding with our respond.

The reviewer writes:

“Though the work is claimed to learn invariant embedding, the decoding procedure is actually generated from some non-invariant embedding (m_{ij}, i,j>0), which traditional GAE also does.“ 

and 

“Specifically, the decoding procedure leverages much information that are not in the invariant embedding z (say using m_{ij}, i,j>0), which essentially implies that the so-called embedding z is not the graph embedding, while the entire matrix M (which is not invariant) is the graph embedding. Note that the proposed method also does not match the diagram in figure 1 because in Eq.13, the edges are generated from m_{ij}, i,j>0, i \noteq j, which is not reflected by figure 1.“


From these comments it seems that the reviewer thinks that the decoder has access to the whole message matrix $\mathbf{M}$ (not only $\mathbf{z}$) as encoded by the Encoder and thus access to the permutation variant information, similar to classical GAEs. This, however, is not the case. This misunderstanding is probably a consequence of the (unfortunately) ambiguous notation $\mathbf{M}$ and $m_{ij}$ used to describe both the message matrix acted on by the encoder as well as the message matrix acted on by the decoder (e.g. in Figure 1 or Equation 11 for the Encoder and Equation 13 for the decoder).

Please note that the encoder and decoder act on different message matrices $\mathbf{M}$.

To clarify this misunderstanding, we change in the following the notation for the message matrix in layer $l$ in the Encoder to $\mathbf{M}^{\text{enc}, l}$ and in the Decoder to $\mathbf{M}^{\text{dec}, l}$ and updated the manuscript accordingly.
Hence, for the definition of the invariant latent space in Equation 11 follows

$\mathbf{z}\sim\mathbb{N}(f_\mu(m_{0,0}^{\text{enc}, L}), f_\sigma(m_{0,0}^{\text{enc}, L}) \mathbf{I})$,

where L is the number of layers in the Encoder. We emphasize that the message matrix $\mathbf{M}^{\text{dec}, 0}$ acted on by the decoder is initialized solely from this latent space z:

$m^{\text{dec}, 0}_{ij} = z + [PE(i)||PE(j)]$

and is not identical to the encoded message matrix $\mathbf{M}^{\text{enc}, L}$.

Thus, edges decoded as defined in Equation 13 are based on decoded messages $m_{ij}^{\text{dec}, L}$ and do not depend on encoded messages $m_{ij}^{\text{enc}, L}$ as stated by the reviewer. The only information passed from $\mathbf{M}^{\text{enc},L}$ to $\mathbf{M}^{\text{dec}, 0}$ is the permutation invariant vector $m_{0,0}^{\text{enc}, L}$.

We apologies for this notational inconsistency and hope the reviewer acknowledges his misunderstanding and reevaluates our manuscript, taking into account that our proposed method matches our claim.


In this work authors propose a new Graph Autoencoder architecture which is a method graph-level unsupervised learning. Authors aim to learn effective permutation invarient graph representation but also learn to reconstruct a graph from a latent representation. The high representation complexity of graphs due to permutable Adjacency matrices poses a problem for graph level unsupervised learning. Most of the existing works by design are not invarient of input graph node order. This poses a problem in training an effective autoencoder because of the misalignment issues caused while calculating reconstruction loss (due to permutations).

The proposed method solves this problem by designing a permutation invarient autoencoder architecture which with the help of a *permuter* model learns to map input graph node ordering with the node ordering of the reconstructed graph. This permuter model is trained along with the autoencoder. Authors show the effectiveness of the model for various tasks including graph reconstruction and generation and also evaluate the model's expressive power by using extracted representations for downstream graph-level classification and regression tasks (molecular property prediction, graph classification).   - To the best of my knowledge, authors propose a new variational autoencoder architecture to solve graph-level representation learning problem. Proposed method predicts the relation between input and output graph ordering in order to make the resulting model permutation invarient. 
- The problem tackled in the paper is important to the graph representation learning community and this work can inspire various future works in this domain. 
- Authors show the effectiveness of the method for several tasks including molecular graph reconstruction and generation, molecular property prediction. 
- The paper is written clearly and is easy to follow.  Questions and Comments:
- Although experimentation is done on variety of tasks, in my opinion experimentation on graph property prediction (graph classification) tasks is lacking. It would really help if you could present the results on several open graph benchmark datasets. 
- The diagonal elements of input message matrix to the decoder are initiated with the summation of positional embeddings and latent representation *z*. How are the off diagonal messages initialized? 

This paper proposes a variational auto-encoder (VAE) to learn an imputation-invariant representation of a graph, which can be also used to reconstruct graph. In other words, a VAE model is proposed for unsupervised graph-level representation learning and reconstruction. In fact, graph-level representation learning using VAEs is challenging because of the non-unique representations of the graphs (n! possible node orderings) and complicated node matching in the reconstruction loss.  This is addressed in this paper mainly by introducing a permuter model, which learns how to permute the canonical ordering of the decoder according to the given input ordering.

The main contributions are as follows.
* Permutation-Invariant encoding of the graph, which can be also used to reconstruct the graph. The main novelty here is the permuter model to match node ordering of input and output graph.
* The model can also solve the graph isomorphism problem at least for the graphs it can reconstruct. If two graphs are isomorphic, the representation will be identical. But, the representation may be also identical for two non-isomorphic graph (if the decoder cannot perfectly reconstruct both graphs).
* Various experimental studies for graph reconstruction, graph generation, and evaluation of the learned representations using qualitative analysis, distance calculation, isomorphism test, and some supervised downstream tasks.  ### Pros
* I appreciate the author’s great effort to include diverse experimental studies.
* As the authors claim (and to my knowledge), this is the first approach for non-contrastive graph-level representation learning, where the resulting representation vector can be also used for graph reconstruction/generation. But, I will expand on this below.
* I think the main novelty is in the imputer model and using position embedding to reconstruct the graph nodes from the single representation vector. Using the self-attention on directed messages for encoding/decoding is incrementally novel.
    * I think using the imputer model to match node orderings of the input and output graph, and using the position embedding to inject information about the position of the nodes for decoding can give a general framework for having permutation-invariant graph representation learning models. This general framework can be like this: 
       * Use any of the existing permutation-invariant graph models for graph encoding (If the model is for node-level encoding, the node representations can be aggregated using simple pooling or more complicated permutation-invariant pooling such as Pooling by Multi-head Attention used in Set Transformer paper).
       * Use the permuter to find the permutation matrix.
       * Add position encoding (according to the permutation matrix) to the graph representation to find initial node embeddings for decoder.
       * Use any existing equivariant decoding model for graph reconstruction.

### Cons
* In the abstract and introduction, it is written that graph-level unsupervised learning has not received much attention. I think this is not a correct statement to make. I think as the authors explained, any graph generation model or permutation-invariant node-representation learning model (with pooling) can be categorized under graph-level unsupervised learning.  I think the right statement is that using unsupervised learning for single-vector graph-level representation which can be also used for reconstruction/generation has not been explored much. 
    * But, I think one of the main reasons why this has not been investigated much is that it is not necessarily required to have a permutation-invariant single-vector representation to solve the downstream tasks. For example, any of the models which gives equivariant node-level representations may be used to solve the problems explored in this paper. For the problems such as graph classification/regression/similarity which needs single-vector graph representation, the node-level representations can be aggregated using simple pooling or more complicated permutation-invariant pooling such as Pooling by Multi-head Attention used in Set Transformer paper. On the other hand, for graph generation/reconstruction tasks, they can use their node-level encodings.
    * In recent years, there have been various permutation-invarinat models which tries to give equivariant node representations. In addition to the ones cited in the paper (e.g. [21]), I recommend to discuss the following papers in Section 3 (Related works):
        * Tony Duan and Juho Lee, Graph Embedding VAE: A Permutation Invariant Model of Graph Structure.
        * Jenny Liu, et al. Towards Permutation-Invariant Graph Generation
        * Chenhao Niu, et al. Permutation Invariant Graph Generation via Score-Based Generative Modeling

    * It may be also good to cite these papers
        * Juho Lee, et al. "Set transformer: A framework for attention-based permutation-invariant neural networks." International Conference on Machine Learning. PMLR, 2019.
        * Rex Ying, et al. "Hierarchical graph representation learning with differentiable pooling." arXiv preprint arXiv:1806.08804 (2018).

* Sometimes the formulation is not neat.
    * For example, P is the permutation matrix. But, it has been also used as a transformation on G in Eq (4) and Eq(5), which is not neat.
    * I am not sure Eq (5) is correct. Since G does not have any ordering ($\pi$), does it makes sense to apply P? Also, I am not sure using $g_{\psi}(P|G)$ for taking the expectation is correct. P might be a deterministic function of G.

### Minor Comments
* I did not understand how an additional node and edge "type" is added in the following line:
    * Line 152:  "To distinguish the embedding node from other nodes, we add an additional node and edge type to represent this node and edges to and from this node."

* Line 152, it is written "To distinguish the embedding node from other nodes, we add an additional node and edge type to represent this node and edges to and from this node."
    * I did not understand how an additional node and edge "type" is added

* In line 168, it is written "We add to each entry in the message matrix $m_{i, j}$ the concatenated position embeddings [PE(i)||PE(j)]."
    * Is $\mathbf{z}$ also used in $m_{i, j}$?

### Final Thought
In overall, although this paper has some limitations (computational complexity and the points explained above), but I think it is an interesting paper for the community which can shed light on future works. In fact, because of the factorial growth of node orderings, it is intractable to learn the complete distribution of the graphs without having significant reduction in the permutations during representation learning.
 I do not think that this work has any potential negative societal impact.

The authors explained the computational complexity of $O(n^3)$ as one of the limitations.

Meta-review: This paper proposed a new variational encoder for unsupervised graph representation learning. The main contribution is the permutation invariance, which is missing in most existing works on graph generative modeling. After the rebuttal, the reviewers found that most of the concerns have been properly addressed, and during the committee discussion we all agree that the paper can be accepted. Despite that the are still challenges in this direction, this work has made a nontrivial contribution. The authors are encouraged to refine the paper according to the reviewers' comments in the final version.

Decision: accept