Title: The quest for the GRAph Level autoEncoder (GRALE)
Venue: NeurIPS 2025
Accepted: yes
Keywords: Graph

Graphs

AutoEncoder

Encoder-Decoder

Deep Learning

Optimal Transport

Attention

Representation Learning

Graph Matching

Combinatorial
Abstract: Although graph-based learning has attracted a lot of attention, graph representation learning is still a challenging task whose resolution may impact key application fields such as chemistry or biology. To this end, we introduce GRALE, a novel graph autoencoder that encodes and decodes graphs of varying sizes into a shared embedding space. GRALE is trained using an Optimal Transport-inspired loss that compares the source and reconstructed graphs and leverages a differentiable matching module, which is trained jointly with the encoder and decoder. The proposed attention-based architecture relies on Evoformer, the core component of AlphaFold, which we extend to support both graph encoding and decoding. We show, in numerical experiments on simulated and molecular data, that GRALE enables a highly general form of pre-training, applicable to a wide range of downstream tasks, from classification and regression to more complex tasks such as graph interpolation, editing, matching, and prediction.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes GRALE, a graph auto-encoder (GAE) to generate representations for graph structure and node features. Graphs with varying sizes can be encoded/decoded by padding. The graph representation can be well conducted on various downstream tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. Completeness: The work clearly defines the problem and outlines the modules in GRALE.\n\n2. Technical depth: The work provides a detailed design for the entire framework.\n\n**Weaknesses**\n\n1. Writing/Clarity: The paper does not adequately justify or clarify some content, making it difficult to follow.\n\n2. Novelty: While the framework design is comprehensive, its novelty should be further highlighted.\n\nQUESTIONS:\nThis work proposes a graph autoencoder to generate both graph and node level embeddings for graphs with different sizes; using padding to align with the max size of  graphs. This work provides a detailed block design for the entire encoder/decoder framework and clear flow of evaluation. Generally I appreciate the technical contribution and completeness of the GRALE in this work. To save authors’ time, I may directly express my confusions and questions on GRALE:\n\n1. The introduction of GRALE is not easy to follow for readers not focusing on the GAE area. Many concepts are not well explained and it takes me some time to understand them by reviewing the appendix and papers referred.\n\n1.1 In the Introduction, authors claimed GAE like Graph U-Net take advantage of the ground-truth information, including the graph size, in the decoding phase. Could you give more explanation on it? Why is this not a good choice in practice? For example in Graph U-Net, the graph size/structure is utilized the information to conduct the pooling/global-pooling function, and finally we get the graph embedding for downstream tasks. It seems to me a similar approach as GRALE where “pooling” is utilized in GRALE’s decoder. Please further justify that Graph U-Net-like structures are not an appropriate one in your scenario.\n\n1.2 Regarding the padding, “N is a maximum graph size; all graphs are padded to this size.” May I ask if the padding strategy (for example where the zeros should be padded, evenly on  two sides of the graph, $[…]+n+[…]\\to N$, or simply just after the graph $n+[…]\\to N$.) Will the padding detail influence the model performance?\n\n1.3 In Sec. 3.4, the Matcher $m$ is introduced to calculate and projected into a differentiable manner for the GRALE loss later. I understand the intension of this module, but have no idea of the mechanism it uses. Might be clearer to explicitly show the expression (or quick illustration) for readers like me without knowledge of it. Also, In Sec. 4.3, authors mention that most GAE works treat matching as a regression problem, but in GRALE, the matcher is trained without groundtruth but minimizing the cost. To my understanding, is this also a regression problem? I mean the $\\hat{T}=SINKHORN(K)$ seems still a regression because $K$ comes from the affinity between $X$ and $\\hat{X}$, where $X$ is the processed “ground-truth”.\n\n2. The evaluation is comprehensively conducted, but some concerns from my current understanding.\n\n2.1 In Table 1, the GRALE outperforms other methods a lot, which is quite impressive. Besides the results, can you give some interpretation on why GRALE can have this good performance? \n\n2.2 Quickly following, also in the main body of this manuscript, it’s much better to include more analysis and justification on the design of modules. For example in each module introduction, could you talk about why you need these operations? Why your choice is or should be the optimal one? Simply stating these might not be convincing enough.\n\n2.3 In Sec.5.2, great qualitative evlauations! But could you provide more analysis and explanation on the results; those would be more instructive to followed researchers.\n\n3. The limitation of the GRALE is on the max graph size. I think this stems from the large model size, as well the training effort, of the proposed GRALE? Is there some intuitive or thinking-loudly strategy could mitigate this issue? \n\n**Minor comments:**\n\ni. “This module first extract from data” -> “extracts” \n\nii. “Remplaced” in Table 2 -> “Replaced”\n\niii. In Figure 11, the reconstruction failure on symmetric structure seems similar to the issue mentioned in [1]. It is interesting. If possible, could you share more opinion on this? I found your manuscript (using random noise) and this paper (using two branches to generate two different embedding) seem address the same issue in two ways. \n\niv. The proposed new decoder in GRALE seems a good contribution. Authors might consider moving it to the main body and provide more interpretation on it.\n\n[1] Duan, Shijin, et al. \"GraphCroc: Cross-Correlation Autoencoder for Graph Structural Reconstruction.\" Advances in Neural Information Processing Systems 37 (2024): 45409-45437.\n\nLIMITATIONS:\nPlease see the questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA fully differentiable graph autoencoder is introduced, including a suggestion of how to make node matching differentiable.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is very clearly and well written.\nIt presents one of the first solutions to a long-standing problem.\nThe results are very good and convincing.\nEven though there is related work (PIGVAE), the authors clearly discuss differences and show significant empirical advantages.\nThe approach is original and combines multiple known as well as new aspects in an interesting way.\nThe approach presented establishes a significant step forward in self-supervised graph representation learning.\nThe computational cost associated to the cubic scaling might become a bottleneck and should be discussed more openly in the paper.\n\nQUESTIONS:\nFigure 4 seems cherry picked to me. All three properties scale with the graph size, as molecular weight and solubility are correlated, and also u0 as an extensive quantity is highly dependent on the graph size. It is not surprising that the embedding also scales with the graph size, as the decoder hast to reconstruct the mask, i.e. number of nodes from the latent embedding. How do these plots look like if you visualize intensive quantities, that are independent of the graph size (e.g. HOMO/LUMO energies or other intensive quantities in QM9). Showing plots of all QM9 properties in the appendix would be benefitial to support the claims in Section 5.2 or also to limit the applicability of those claims.\n\nThe same is true for Table 3 and Table 4(QM9): Which task was performed (there are many labels in QM9) for property prediction and graph prediction?\n\nWhat is the computational cost of training the model compared to previous models used in the benchmark? Please add that information to the paper. Please also discuss if the training scales cubic or even more than cubic with the typical graph size in the datasets (single forward/backward pass scales probably cubic, but if more data and/or more epochs are needed for training on larger graphs, one might expect above-cubic scaling).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a graph autoencoder that works at the graph level (ideally without directly using node embeddings, although in practice this is not entirely true). The main idea is to avoid the expensive matching step between the input graph (of the encoder) and the output graph (from the decoder) by using a loss inspired by Optimal Transport. Instead of computing a permutation matrix, the method computes a transport matrix. Since this operation can also be complex, the model includes a module that learns to predict the matrix. The whole system is trained end-to-end.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n- The paper is well structured and easy to follow; the transition between (almost all) sections is smooth and helps the reader follow the flow of ideas.\n- The choice of using different schemes to build the input and target graphs is interesting, as well as highlighting that breaking the symmetry between input and output can improve learning.\n- The entire system, including the prediction of the graph matching, is trained end-to-end, which makes the approach elegant and potentially efficient.\n\n*Weaknesses*\n\n- The paper claims to produce a graph-level embedding, but in reality, it also produces node embeddings.\n- The paper states that it produces a graph-level embedding Z, but in practice, it is a representation composed of K tokens. This is an important point, but it is only briefly discussed. What does it mean that these tokens represent abstract \"concepts\"?\n- Figure 2 could be very helpful, but in its current form, it is hard to read. In particular, it is unclear where X comes from, what the \"linear\" block actually does, and what the K \\times D input to the transformer decoder represents (or maybe I missed some parts). The best solution would be to expand the figure caption to clearly describe the architecture, including a proper explanation of notations such as n_F, n_C, etc.\n- The readability of Figure 3 could be improved, for instance, by adding the labels K and D.\n- Figure 4 does not provide much insight. It would be more informative to include a comparison with existing methods (e.g., PIGVAE) to better assess the qualitative performance of GRALE.\n- Unless I missed it, there seems to be no table reporting training time comparisons with other methods. \n- While the paper emphasizes the importance of the proposed matching-free loss, the ablation results in Table 2 suggest that its contribution to reconstruction quality is relatively limited compared to other architectural components such as the encoder, matcher, or featurizer. This raises a question: could this be considered a discrepancy? In other words, are the empirical differences between the proposed loss and previous alternatives negligible in practice? Apart from the observation that GRALE shows reduced standard deviation (suggesting more stable training), the actual impact of the loss function seems limited, and it is not entirely clear how the authors interpret this result.\n- Complex graph operations in the latent space are presented as an ambitious aspect of the paper, but they are only briefly introduced. Graph interpolation in latent space is not a novel idea; actually, it is well established in the graph generative modeling community. Its effectiveness depends on how well-structured the latent space is (ie embeddings must reflect meaningful semantics, where similar graphs correspond to nearby vectors), otherwise, interpolations may lead to invalid or meaningless graphs. In this work, the discussion remains theoretical, making this part of the contribution speculative.\n\nQUESTIONS:\n- How much of the performance depends on the preprocessing stage? This aspect is not clearly discussed and might deserve more attention\n- It’s not clear to me how the maximum number of nodes is fixed.\n\nFor other questions, please see above.\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes GRALE, a graph-level autoencoder that learns to encode and decode entire graphs into a shared embedding space using an optimal transport inspired loss and a differentiable matching module. It leverages Evoformer-based architectures to support flexible graph reconstruction and shows strong results on reconstruction, classification, and prediction tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. GRALE embeds graphs into a tokenized vector space that supports a wide range of downstream tasks. The design supports vector arithmetic over graph embeddings, enabling novel use cases such as latent space editing and interpolation.\n\n2 . The method achieves superior performance over GraphVAE and PIGVAE in reconstruction accuracy and downstream tasks. Ablation studies are well-conducted and demonstrate the contribution of each module.\n\n 3.The incorporation of the Evoformer module from AlphaFold for both graph encoding and decoding is a noteworthy attempt, which might help effective modeling of interactions between node and edge features.\n\n\nWeaknesses:\n1. The paper omits some relevant graph-level models that could serve as strong baselines or comparisons. For instance, recent masked graph autoencoders (e.g., GraphMAE) and Transformer-based architectures for graphs (e.g., Graphormer, TokenGT) are not discussed or evaluated.\n\n2 . The proposed “Evoformer decoder” part lacks comparative analysis with standard Transformer decoders or other graph generation decoders.\n\n3. Most experiments are performed on molecules or COLORING datasets. It remains unclear whether the method generalizes well to non-chemical graphs\n\nQUESTIONS:\n1. The paper claims that the graph size is predicted via the padding mask vector, yet the decoding process is not clearly explained. How does the model ensure the correct number of nodes is generated without explicit supervision on graph size?\n\n2. Lacks ablation against standard Transformer decoders. How critical is the Evoformer Decoder to performance, and what unique capabilities does it provide?\n\nLIMITATIONS:\n1.The method incurs relatively high computational overhead, primarily due to the use of Evoformer modules and Sinkhorn-based matching, which limits scalability to larger graphs.\n\n2. Although the OT-inspired loss is conceptually sound, the paper lacks theoretical analysis to guarantee that the learned embeddings faithfully preserve semantic or structural similarities between graphs.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces GRALE, a novel graph-level autoencoder that encodes and decodes entire graphs—of variable sizes—into a shared Euclidean latent space. GRALE employs a loss inspired by Optimal Transport and features a differentiable node-matching module, trained end-to-end with the encoder and decoder. It builds upon the Evoformer architecture from AlphaFold, adapting it for both encoding and decoding graphs. Extensive experiments show that GRALE achieves state-of-the-art performance in tasks including graph reconstruction, classification, regression, interpolation, and editing, especially in molecular and synthetic datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n- Novel Architecture: Utilizes and adapts Evoformer for graph-level encoding and introduces a novel Evoformer Decoder.\n\n- End-to-End Differentiable Matching: Avoids NP-hard graph matching by integrating a learnable Sinkhorn-based matcher.\n\n- Versatile Applications: Enables not only basic tasks (e.g., classification, regression) but also complex operations like graph interpolation and editing in latent space.\n\n- SOTA Performance: Demonstrates strong results across multiple benchmarks (e.g., PUBCHEM, QM9) and outperforms baselines like PIGVAE and GraphVAE.\n\n- Thorough Evaluation: Includes detailed ablations and empirical analysis of architecture choices, matching methods, and representation space.\n\n- Scalability Evidence: Trained on massive molecular datasets (up to 84M samples), showing GRALE’s robustness and generalization.\n\nWeaknesses：\n- High Computational Complexity: The use of edge-level attention and Sinkhorn projections makes training and inference computationally intensive (O(N³)), limiting applicability to graphs with ≤32 nodes.\n\n- Limited Theoretical Justification: While empirical results are compelling, theoretical guarantees (e.g., generalization bounds, robustness under graph perturbation) are not deeply explored.\n\n- Heavy Reliance on Pretraining: The effectiveness of GRALE in downstream tasks depends strongly on massive pretraining on large datasets.\n\nQUESTIONS:\n- Can GRALE scale to significantly larger graphs (e.g., protein networks or social graphs with hundreds of nodes), and what would be required to do so?\n\n- Would incorporating graph-specific domain priors (e.g., valency in molecular graphs) improve GRALE’s reconstructions?\n\n- Can the Evoformer Decoder be applied independently for graph generation tasks?\n\n- Could fine-tuning the matcher module separately improve matching accuracy for unseen graph pairs?\n\n- Is there a way to adapt GRALE to support dynamic or evolving graphs (e.g., temporal sequences)?\n\n- How about comparing GRALE with GraphsGPT[1]? \n\n[1] A Graph is Worth K Words: Euclideanizing Graph using Pure Transformer\n\nLIMITATIONS:\n- Scalability: The O(N³) cost for attention and matching restricts practical use to small graphs (≤32 nodes).\n\n- Data-Hungry Training: Requires very large training sets to reach high accuracy—less suitable for low-resource domains.\n\n- No Ground Truth Matching Supervision: Matching is learned implicitly, which may limit interpretability and performance on exact alignment tasks.\n\n- Fixed Graph Size Ceiling: Padding strategy introduces a hard constraint on maximum graph size.\n\n- Generalization to Unstructured Inputs: Limited discussion or experiments on how well GRALE performs with real-world, noisy graph inputs outside curated datasets.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}