Self-Supervised Learning on Molecular Graphs: A Systematic Investigation of Masking Design
Authors: Jiannan Yang
Veronika Thost
Tengfei Ma
Venue: Accepted by TMLR ()

Abstract: Self-supervised learning (SSL) plays a central role in molecular representation learning. Yet, many recent innovations in masking-based pretraining are introduced as heuristics and lack principled evaluation, obscuring which design choices are genuinely effective. This work cast the entire pretrain–finetune workflow into a unified probabilistic framework, enabling a transparent comparison and deeper understanding of masking strategies. Building on this formalism, we conduct a controlled study of three core design dimensions: masking distribution, prediction target, and encoder architecture, under rigorously controlled settings. We further employ information-theoretic measures to assess the informativeness of pretraining signals and connect them to empirically benchmarked downstream performance. Our findings reveal a surprising insight: sophisticated masking distributions offer no consistent benefit over uniform sampling for common node-level prediction tasks. Instead, the choice of prediction target and its synergy with the encoder architecture are far more critical. Specifically, shifting to semantically richer targets yields substantial downstream improvements, particularly when paired with expressive Graph Transformer encoders. These insights offer practical guidance for developing more effective SSL methods for molecular graphs.

Published in Transactions on Machine Learning Research (11/2025)
Self-Supervised Learning on Molecular Graphs: A Systematic Investigation of Masking Design
Jiannan Yang Stony Brook University
noamon..ongg@stonbbooo.com
Veronika Thost
veronika.thost@@kbmmcomm
MIT-IBM Watson AT Lab
Tengfei Ma
lengfel.ma@sstongrroo...h
Stony Brook University
Reviewed on OpenReview: https: // optpreveeeer.eeww..mmmm net/ forum? id= TE4uCYWRcc
Abstract
Self-supervised learning (SSL) plays central role in molecular representation learning. Yet, many recent innovations in masking-base based pretraining are introduced as hcuristics and lack principled evaluation, obscuring which design choices are genuinely effective. This work cast the entire pretrain-finttne workflow into a unified probabilistic framework, enabling a transparent comparison and deeper understanding of masking strategies. Building on this formalism, we conduct a controlled study of three core design dimensions: masking dis- tribution, prediction target, and encoder architecture, under rigorously controlled settings. measures to assess the informativeness of pretrain-
We further employ information-theorrt ing signals and connect them to empirically benchmarked downstream performance. Our findings reveal a surprising insight: sophisticated masking distributions offer no consistent benefit over uniform sampling for common node-level prediction tasks. Instead, the choice of prediction target and its synergy with the encoder architecture are far more critical. Specif- ically, shifting to semantically richer targets yields substantial downstream improvements, particularly when paired with expressive Graph Transformer encoders. These insights offer practical guidance for developing more effective SSL methods for molccular graphs.
1 Introduction
Graph neural networks (GNNs) have gained significant traction in chemistry due to their intrinsic compati- bility with molecular graph struct (Duvenaud et al. 2015 Gilmer et al. 2017). A key challenge in this domain is that obtaining molecular property labels often requires specialized and costly experimental proce- dures [Ramakrishnan et al. 2014 Wu et al. 2018), which inherently limits the scale of empirically labeled datasets and hinders the rapid exploration of the vast chemical space. Powerful computational models are es- sential for exploring vast chemical spaces and accurately predicting molecular properties at scale. To reduce the need for extensive experimental labeling, researchers have increasingly adopted self-supervised learning (SSL) DDrra et al. 2022). SSL leverages supervisory signals from abundant unlabeled molecular data to pre-train models that learn generalizable representations. These SSL methods for molecular GNNs are broadly categorized into two paradigms: masking- based pretraining (uu et al.) 2019 Hou et al. 2022 and contrastive learning YYou et al. 2020 Liu et. al.) 2021a). The former involves masking altributes of sampled nodes or edges within a molecular graph and training the model to recover this hidden information, often using the original atom or bond propertics as supervisory signals. The latter employs graph augmentations to generate positive and negative molecular pairs for contrastive learning. Both approaches aim maximize the extraction of chemically relevant information from molecular structures, thereby improving the inductive

---

Published in Transactions on Machine Learning Research (11/2025)
bias of GNNs for downstream tasks such molecular screening and drug discovery, where labeled data is scarce.
This work focuses on the masking-based pretraining paradigm. A seminaa work in this direction is Hu jet al. (2019), which pioneered the use of graph neural networks (GNNs) with a masked prediction objective for molecular representation learning, demonstrating the effectiveness of reconstructing masked node or edge features. This work laid the foundation for subsequent studies. Over the years, various modifications have been introduced. These innovations can be broadly categorized along three main axes: (1) model architectures, such as adopting alternative GNN encoders or reconfiguring the overall learning framework [Rong et al. 2020: Hou et al., 2022: Liu et al. 2023); (2) masking distributions, involving novel strategies for selecting which parts of the graph to mask @Liu et al. 2024 [nae et al. 2024); and (3) prediction targets, which alter the nature of the information the model aims to reconstruct during pretraining (Xia et al. 2023
Yang et al. 2024).
While new studies oflen claim to surpass prior methods on benchmark datasets, our comprehensive eval- uations demonstrated that many modifications to masking strategies do not yield significant performance gains when evaluated under more rigorously controlled settings. For instance, we find that replacing simple uniform sampling with more sophisticated distributions offers no consistent advantage. Furthermore, as noted by Koo & Kwon (2025), methodical comparisons that isolate the efficacy of specific masking strategies from other confounding factors remain limited. This makes challenging to ascertain which design choices are genuinely effective. To address these ambiguities and provide a clearer understanding of masking- based SSL in molecular graphs, our contributions are as follows:
1. We formalize the masking- pretraining pipeline for molecular graphs, factoring it into key design dimensions: masking distribution, prodiction target, and encoder architecture. This enables a structured calegorization and comparison of existing and novel approaches.
We conduct a rigorous comparative study by meticulously controlling experimental variables across these dimensions and hyperparameters, thereby isolating the true impact of different masking strate- gies and architectural choices downstream task performance.
3. We introduce a model-agnostic information-theoreii analysis, using mutual information and Jensen- Shannon Divergence, to quantify the alignment between pretraining proxy tasks and downstream molecular property prediction. This analysis provides deeper insights into the underlying mecha- nisms driving observed performance differences.
2 Related Works
Self-supervised learning (SSL) has become a pivotal paradigm for learning general-purpose representations from large-scale unlabeled data (Jing& & Tian 2020 Wu et al. 2020 Liu et al. 2021b). In the graph domain, SSL methods are broadly categorized into two main paradigms. Contrastive learning (CL) learns discrim- inative representations by maximizing the agreement between different augmented views of a graph (You) et al. 2020 Liu et al. (2021a] Wang et al. 2022). In parallel, a generative approach, often termed Masked Graph Modeling (MGM), learns by corrupting parts of the input graph and training a model to reconstruct the original information.
The underlying principle of MGM, learning representations by reconstructing masked portions of the input, first popularized in natural language processing by models like BERT (Devlin et al. 2019). This powerful self-supervised paradigm computer vision with Maskod Image Modeling (He et al.) 2022) and, central to this paper, the molccular domain. Within molecular learning, the masking principle has been applied across diverse data modalities of molecules. For instance, SMILES- BERT (Wang et al. 2019 treats molecules as 1D SMILES sequences and apply BERT-style token masking, directly leveraging advancements from NLP. Recent works such as MOAT (Long et al., 2024) and SMI-Editor (Zheng et al. (025) have further broadened the design space by transforming molecules into multi-gramularity prompts and integrating large language models.
subsequently and concurently adapted to other domains, including
2

---

Published in Transactions on Machine Learning Research (11/2025)
In the 2D visual domain, MaskMol CCheng et al. 2024) explores 2D molecular images, performing knowledge- guided pixel masking on atoms or functional groups to address specific challenges like activity cliffs. Further- more, EMPP {An et al.) 2025), physics-informee direction operates on 3D geometric structures, proposing to mask atomic positions and train equivariant GNNs to predict them, thereby learning about intramolccular lorces. While each modality offers unique research directions, our work focuses on a principled analysis of masking design choices specifically within the prevalent 2D molecular graph paradigm.
2.1 Evolving Designs in Masked Modeling for 2D Molecular Graphs
The central architectural component for processing 2D molecular graphs is the Graph Neural Network (GNN), which serves as powerful encoder that learns representations by operating directly on the graph topology and features. Applying masked modeling to these GNN-based systems began with the foundational framework of AttrMask (Hu et al., 2019), which masks atom bond attributes and uses simple MLP for reconstruction. Subsequent research has evolved this paradigm in multiple directions. Archilect GraphMAE (Hou et al.) 2022), uses more expressive decoder and introducing mechanisms like re-masking in the latent space. Concurrently, more powerful encoder backbones like Graph Transformers were also leveraged to better model long-range dependencies (Rong et al. 2020: Liu et al. 2023 Yang et al., 2024). Beyond standard node-centric homogeneous graphs, some works have even explored fundamentally different input representations, such as the heterogeneous atom-bond graphs in MGMAE (Feng et al. 2022). as well as edge-centric architectures such as ESA (Buterez al. 2025), which treat edges as primitives and interleave masked and vanilla sell-attention based on edge adjaceney, providing an alternative lo node-focused masking paradigms.
Innovation has also occurred in the masking strategy itself. Research has moved from simple uniform random masking to adaptive distributions based on graph heuristics or learnable: scorers to identify structurally important nodes (Liu et al. 2024). The masking granularity has also been focus, with a clear trend towards higher-level semantic units, such as masking entire chemically meaningful motifs (subgraphs) instead
of individual nodes (Zhang et al. 2021 Wu et al. 2023 Inae et al. 2024).
Another active rescarch direction involves designing more semantically rich prediction targets. These in- clude predicting a discrete index from a vocabulary representing structural subgraphs (Ma et al. 2024
or learned codebooks (Xia et al. 2023), as well as predicting pre-defined motif labels (Yang et al. 2024
Other approaches have also explored predicting high-dimensional continuous vector representations of local neighborhoods (Liu et al. 2023).
2.2 Current Challenges and The Need for Systematic Investigation
Despite the rapid proliferation of MGM methods on molecules, most. studies locus on proposing a novel model and demonstrating its superiority on specific benchmarks, leading to several challenges: a lack of systematic analysis of the interplay between different design choices, scarcity of controlled comparisons, and limited understanding of the mechanisms behind observed performance differences. This is exacerbated by the trend of creating complex, hybrid frameworks that combine different. SST paradigms. For instance, works like GCMAE (Wang et al.) 22024) and UGMAE (Tian et al. 2024) explicitly combine multiple SSL paradigıns and introduce numerous components and loss terms. While powerful, the complexity of such models makes it increasingly difficult to attribute performance gains to specific design choices.
This highlights the urgent need for systematic investigation. Some prior work has started this process. The study by Koo & Kwon (2025) provided comprehensive analysis of several lower-level masking design aspects, such as the masking phase (pretraining vs. fine-tuning), gramularity (e.g., node vs. subgraph), location (feature vs. embedding), and key hyperparamcters like masking ratio. While valuable, their analysis was conducted within a single architectural framework and did not cover higher-level design choices. Other works, like that of Wang et al. (2023): : Cintas et al. (2023), have proposed new evaluation methodologies to characterize pro-trained representations beyond simple downstream task performance. These efforts revcal a core challenge: a more fundamental understanding of the causal links between pretraining design choices and the properties of the learned representations is required. Our work aims to address this gap by proposing
3

---

Published in Transactions on Machine Learning Research (11/2025)
formal probabilistic framework, conducting rigorously controlled experiments, and employing information- theoretic measures to provide deeper, more principled insights and practical guidance for the field.
3
Methodology
This section details the methodology for our systematic investigation. We begin by casting the pretrain- finetune pipeline into a unifying probabilistic framework (Sec. 3.1 3.2 allowing us to deconstruct and systematically compare masking strategies (Soc. 3.3 All molccular graphs are treated as undirected, non- singleton graphs.
3.1 Analysis Dimensions
While many existing works highlight the goal of pretraining capturing the intrinsic chemical information embedded in molecular structures, they often lack formal account of how the pretraining task relates to downstream property prediction. To provide clearer perspective, we adopt probabilistic model to describe the two-step pipeline commonly used in evaluating mask prediction approaches.
We consider the pretrain-fineet finetune setting as applied to classification tasks. Formally, we define X, a random variable representing a label from a label space L that is assigned to a specific structural unit sampled from space possible units S(G) within a given graph G E);
X ::S(G)
(1)
This general formulation is flexible and powerful. For instance, in simple node-level tasks, the space of structural units S is simply the set of nodes V. For higher-lovel tasks, S can be family of subgraphs (c.g.. chemical motifs).
Meanwhile, the graph-level (global) property used for downstream prediction is denoted as
: + (0,1)
(2)
where G is the space of all molecular graphs in given downstream dataset.
Let M denote a specific masking strategy. This strategy defines how to sample masking indices M C (0, 1,....|||| -1 for a graph G. The application of mask M G from the dataset Gasta yields the prediction target, the vector of true labels X (SSM), and the model's input, the masked graph GMr. Conventionally, Gm is obtained by replacing the fundamental elements of G, such as node or edge features, to fixed, non-existent vector m of the same dimension. Using this notation, we can frame the entire pipeline in the following way.
1. In the pretraining stage, the objective is to learn an optimal encoder decoder pair (Je, Within the scope of our study, fe a GNN, and the decoder Jo typically an MLP or another GNN. The objective is to minimize the expected loss for masked label prediction
min GG 0.8
[L (go()o(GM)) X(SM)))
(3)
We define the overall loss L for masked set VM the mcan of a loss function E (c.g., cross-cntropy) over all nodes in that set:
L [(o.(f((G)).
Σ
SSM
WESH
2. At test time, the pretrained encoder fg is then used to initialize the backbone of the downstream classification model, which is further fine-tuned by minimizing:
min B(G.Y) /e (Ja())))) 8.0
(4)
to improve prediction performance on unseen molecular graphs. Here P(.,)) represents the joint distribution of graph G and graph-level label Y in the downstream dataset, and hy denotes the MLP classifier of graph-level properties.
4

---

Published in Transactions on Machine Learning Research (11/2025)
C.C....O.M
Unabaded Mcneraee Goes
Mosking M
Decoder
Preassiing
Updato Weights
Weigts
Updese Unaggtte
Enooder /5
Property Predkder by
Finetuiig Losa
Lareing
Probon Shartle
Figure 1: An Example of the Pretrain-Finetune Pipeline: Node Attribute Recovery
Based this formulation, it becomes clear that design choices made during the mask prediction pretraining stage (Eq. 3 can be broadly factored into three main categories, as described below.
3.1.1 Masking Distribution
This first design dimension concerns the choice of the masking strategy, M. Different strategics define different distribution to sample the masked nodes, which is why we refer this dimension as the masking distribution. In principle, the choice of M directly determines the marginal distribution, PM of the predicted labels X.
Uniform masking: The sampling distribution is uniform over the set of nodes V, where each node selected for masking with an equal and independent probability.
Heuristic masking: The sampling distribution is determined by pre-defined heuristics based on graph structure.
Learnable masking: The sampling distribution is dynamically learned during pretraining. This typically involves dedicated neural modules, that predict node importance scores, which are then used to parameterize the sampling probabilities.
3.1.2 Prediction Target
Another key design axis is to redefine the prediction larget X itself. In Figure 1 adopted the formulation representing random variable mapping from nodes to discrete label values. This formulation can be altered in several ways, for instance, changing the sample space V, modifying the label
X V + (1, nn.,
range, or redefining the correspondence between the two.
One important example of such an alteration to define X mapping from subsets of nodes,
x + (1, 11.,....,m.. m)
where F(V) denotes a lamily node subsets. This allows us to incorporate higher-level semantic labels, such as subgraph-level supervision.
In our experiments, we compare several classes of prediction targets. The specific methods corresponding these classes will be detailed in Section 3.
5

---

Published in Transactions on Machine Learning Research (11/2025)
Atomic Attribute Prediction: The target is to reconstruct the original, low-level attributes of individual masked atoms, such as atom type or formal charge. This represents the most direct form of feature recovery.
Learned Node-level Token Prediction: The target is a discrete token representing a learned, abstract representation of an atom. These tokens are typically derived from a separate, pretrained model like a vector-quantized encoder.
Structural Motif Prediction: The target is label corresponding to higher-level, chemically meaningful substructure (i.e., motif/subgraph) to which the masked atoms belong. This shifts the prediction from local atomic properties to broader structural semantics.
3.1.3 Encoder Architecture
The final design dimension we investigate the choice the encoder architecture, fo. The encoder's capacity to model different types of structural dependencies is crucial, as the component responsible for generating transferable representations. Our study focuses on comparing two dominant paradigms for graph-based molccular encoding:
Message Passing Neural Networks (MPNNs): This class of models iteratively updates node representations by aggregating information from their local neighborhoods. Due to their strong inductive bias for graph-structured data and computational efficiency, MPNNs have become the standard backbone for a wide range of molecular properly prediction lasks.
These architectures enhance message passing networks by incorporating global attention mechanisms, allowing every node to attend to every other node in the graph. This enables the direct modeling of long-range dependencies, which challenging for standard MPNNs. While more expressive in principle, whether this increased capacity translates to better performance in masking-based pretraining remains an open question evaluated in our study.
Graph Transformers:
Other Components Beyond the three core design dimensions, a complete pretraining pipeline involves choices about several auxiliary components. These often include the specific architecture of the decoder (c.g., simple MLP versus GNN-based decoder), the formulation of the loss function (e.g., standard enoss-entropy versus a scaled cosine error), and other techniques such applying re-masking step to the latent representations before decoding (Hou et al.) 2022). When comparing the main design dimensions in our study, we adopt a consistent configuration for these auxiliary components to ensure a fair comparison. Results from additional ablations on these components are provided in Appendix A.1 for completeness.
3.2 Principled Criteria for Signal Informativeness
Two hypotheses naturally arise in this framework: an effective pretraining signal can be engineered by either optimizing the masking distribution PM or enriching the prediction target X. These hypotheses rest on a common intuition: under controlled settings, if pretraining on the chosen definition of X is to provide more useful information for predicting Y, then X should exhibit stronger statistical dependence with Y. To formalize this intuition, employ mutual information between X and Y as a principled, model-agnostic measurement to compare different design choices.
3.2.1 Mutual Information
To this end, we propose using the mutual information (MI) between X and Y as information- th measure of alignment. The mutual information YY) quantifies how much information about the global label Y we can obtain by knowing the local label X, and is defined a
P(x, v)
I(X;Y)= = Σ P(a, y) log P(x)
(5)
2.9
Formalll, define the sample space of (X, Y) as Ω Liceo S(C) = Jocel(u, G) S(C)) that each structural unit is paired with its parent graph G, ensuring the joint, distribution welledefined.

---

Published in Transactions on Machine Learning Research (11/2025)
where 2 ranges over node or motif labels, and y is the graph-level label.
A higher value of I ()) indicates stronger statistical dependence between the sampled local label variable X and the downstream property Y, suggesting that the pretraining signal may provide richer information about the target task. This formulation allows us to compare the potential informativeness of different prediction targets and masking distributions a model-agnostic manner, independent of encoder architecture.
3.2.2 Analysis of Conditional Distributions for Low- Now-Frequency Labels
While MI provides a holistic measure of dependence, its averaging nature can obscure important details. Specifically, the influence of highly discriminative but infrequent local labels might be diluted by more common ones. This is a particularly relevant concern when comparing prediction targets of different semantic levels (c.g., common atoms vs. rare functional groups).
To probe the discriminative power of local labels beyond this average effect, our analysis leverages the Jensen-Shannon Divergence (JSD). The JSD allows for direct comparison of the conditional label distributions, P(X = 1) and P(X = 0). Based on the hypothesis that impaciful local labels are often infrequent, we focus our JSD analysis on a subset of low- wwefrequency labels, Sr, defined as:
S, = (x E P(x) <T)
where X' is the set of all unique local labels, P(x) is the empirical probability of label and T E (0, 1] is probability threshold. We then estimate the conditional probability distributions restricted to this subset, P(X = S,), as follows:
N(X = x, =x,Y= E'ES N(X = ,T,Y=
P(X = x = =, =
(7)
where N(X x, Y =) is the count of occurrences of label in graphs of class y. The JSD is then computed between P(X = S,) and P(X|Y- = O,S,) to evaluate how the distinguishability of rare labels varies across different prediction target types.
3.3 Instantiating the Design Dimensions
To ground our theoretical framework in practice, this section maps set of reproduced pretraining methods to the design dimensions outlined in Section 3 Instead of a simple chronological review, we organize this section to mirror our framework's structure. We first introduce AttrMask as the foundational bascline in detail. Subsequent subsections then explore how various methods have innovated upon this baseline along each of the three primary axes: masking distribution, prediction target, and architectural components. This approach allows for a clear, dimension-wise comparison while presenting each mothod in a logical, dependency-awail order. A complete overview of all method configurations provided in Appendix A.1 see Table g
The pioneering work of Hu et al. (2019) introduced Attribute Masking (AttrMask), which serves as the foundational baseline in our study. Within our probabilistic framework, AttrMask can be precisely defined by instantiating each core component. The prediction target
AttrMask.
Xtype V + 10, 118]
is the original atomic altribute (e.g,, atom type where the space of structural units S(G) for each graph G is the set of nodes V. The masking distribution PM is uniform distribution over these nodes, from which a subset Vm C V is sampled. To create the corrupted graph GM, the feature vectors Va of these selected nodes are replaced with a special mask token m of the same dimension. Finally, an encoder fe. typically an MPNN, processes Gm to produce node embeddings, and simple MLP acts as the decoder go to predict the original attributes X (v) for Vm by minimizing cross-entropy loss. See Figure !
2In the range Xatom, stands for the element class of unknown atoms.

---

Published in Transactions on Machine Learning Research (11/2025)
Building the AttrMask framework, GraphMAE (Hou et al. 2022 introduces key archi- tectural innovations to reformulate the task as a more complete masked graph aulo-encoder. It retains the same uniform masking distribution and atomic attribute prediction target as AttrMask, but enhances the architecture, primarily in the decoding process. Its key contributions arc:
GraphMAE.
(1) A Re-masking step, where the embeddings of masked nodes are again replaced by a special token before being passed to the decoder.
(2) A GNN-based decoder, where another GNN layer is used as part of the decoder Jo to further process latent codes before a final MLP predicts the node attributes.
Additionally, GraphMAE proposes using a scaled-cosine error (SCE) loss, shown in Equation instead of cross-entropy lo down-weight easy-to-predict examples.
1 V
LSCE
[
3.3.1 Innovations in Masking Distribution
The methods discussed far, AttrMask and GraphMAE, both rely simple uniform distribution for selecting nodes to mask. However, the hypothesis that non-uniform, structuro-aware masking distributions (PM) could provide a more effective pretraining signal has also been explored, with StructMAE (Liu el al., 2024) being a representative example. This approach builds upon the GraphMAE framework to introduce heuristic and learnable masking strategies.
2024) extends GraphMAE by replacing uniform node sampling with structure-aware masking distributions. Two variants are proposed:
StructMAE. StructMAE (Liu
StructMAE-D Nodes are ranked by PageRank scores, computed as (+++1) -n(()A(()) + a)p until convergence. To avoid always masking the same high-ranked nodes, perturbed top-k selection is applied, where random noise is added to candidate scores and the effective mask rate is gradually annealed during training.
MAE-L Instead of heuristics, node importance scores are learned jointly with the encoder via shallow GNN and MLP scorer. The same perturbed top-k mechanism is then applied to these learned scores, cnabling a flexible, data-driven masking distribution.
Both variants share the GraphMAE anto-encoding framework; only the masking distribution differs. The full algorithmic details are provided in Appendix A.12
MoAMa. Motif-aware Attribute Masking strategy (MoAMa) [Inae et al. 2024) partitions a molecular graph into multiple connected subgraphs, motifs, using the BRICS decomposition (Dgen al., 2008). Rather than uniformly sampling individual nodes, MoAMa samples at the motif level. It employs non- adjacent motif selection policy. Specifically, this algorithm iteratively samples motif from the pool of available candidates and then removes its direct neighbors from the pool for subsequent selections within the same graph. This principle is likely intended to prevent the creation of large, contiguous masked regions, which could sever information pathways for local message-passing encoders.
It is worth noting that the original MoAMa framework additionally incorporates a molecular fingerprint- based contrastive loss. However, since our study focuses solely the masking strategy, for fair comparison, exclude the auxiliary loss from our implementation.
3.3.2 Innovations in Prediction Target
We now turn to the second major axis of design: the prediction target itself. The following methods move beyond reconstructing simple atomic attributes, as done in the previously discussed methods, by proposing
8

---

Published in Transactions on Machine Learning Research (11/2025)
GNN Tokenizer
nng min Hll m
Gradient
Aood+Bond Recon, Loss
GNN Decoder
Update
Figure 2: Group VQ-VQE Pretraining for Tokenizer and VQ Codebook in MAM
semantically richer targets. These include approaches that predict learned, abstract node representations or explicit, chemically-mcaningtul substructures.
MAM. Inspired by VQ-VAE Van Den Oord et al. 2017 Xia et al. 2023 introduced Masked Atom Modeling (MAM) as an alternative to AttrMask, aiming to expand the prediction space beyond atomic types as part of their Mole-BERT framework. Specifically, it relies on GNN tokenizer, denoted as T'p and a learned VQ codebook, Qo, whose paramcters are fixed after a separate pretraining phase (sce Figure 2
In our study, we consider two instantiations of MAM based on how node labels are assigned. These variants differ in whether the pretrained VQ codebook is explicitly used during masking prediction.
(1) MAM-A (Argmax labeling): A discrete pseudo-label is generated by applying an arg max operation to the put of the GNN tokenizer T:p:
XA XA:V V + {0, 1, .,....,511)
+ argmax (T (b);
This variant does not rely the VQ codebook Qo.
3 Node labels are assigned by finding the index of the nearest entry in the VQ codebook Qe to the output of the tokenizer Tęp:
(2) MAM-VQ (Codebook labeling]"
XvQ {0,1, ..... 511} v + argmin (I (T)) - Qo :1
The pretraining and subsequent use of the VQ codebook in MAM-VQ involves nuanced two-stage process. For a detailed explanation, please refer to Appendix A.11
MotifPred. Based on the idea of motif-level supervision in CeaCTMask Yang et al. 2024), propose simplified motif prediction task, denoted as MotifPred. Specifically, we train the model to predict a unique pre-assigned label for each motif:
Xmottt F(G) - Lmotif
We implemented MAM-VQ based original paper's description, as only the MAM-A variant available the official codehase.

---

Published in Transactions on Machine Learning Research (11/2025)
where F(G) is the set of all motifs in G. To generate prediction for given motif, the final node repre- sentations of all atoms within that motif are first aggregated (e.g., via sum pooling) into single motif-level representation. This aggregated vector is then used by the decoder to predict the corresponding motif label.
To create a manageable set of labels L, we follow Yang et al. 2024 and pre-compute the motif decomposition using the refined BRICS algorithm (Zhang et al. (021). This pre-computation not only helps decrease the vocabulary size [C], but also enables more efficient sampling mechanism compared MoAMa's on-the-fly approach. In contrast MoAMa, MotifPred also masks only subset of atoms within selected motif rather than the entire substructure. This setting simplifies the original ReaCTMask, which was performed within disjoint union of molecular graphs in a chemical equation. To ensure a controlled comparison across different design dimensions, we implement MotifPred using both GraphGPS (Rampášek et al. 2022) and message passing networks.
3.3.3 Encoder Architecture
The final design dimension in our framework is the choice of the encoder fo. which dictates how structural information is processed and aggregated within the graph. Our study compares two GNN backbones used in the previous works, representing local and global information flow, respectively.
Message Passing Neural Networks (MPNNs) The architectural backbone employed by most dis- cussed methods in this study, is the Graph Isomorphism Network with Edge features (GINE) (Xu et all 2019 Hu et al.] 2019), Message Passing Neural Network. The GINE layer updates node's representation he by aggregating features from its neighborhood (()) according to the following rule for layer k:
(a+c (1+
ReL.U (k-k))) + tc..)) e
1 = MLPP(A)
1/ 1k-1) +
Σ (EN(()
(9)
where (((=-1) the representation of node from the previous layer, Ea, is the feature of the edge connecting nodes and v. This iterative, local aggregation provides strong inductive bias for graph topology but inherently limits the model's receptive field.
Graph Transformers To capture dependencies beyond local neighborhoods, we also employ more expressive Graph Transformer, specifically GraphGPS (Rampášek et al. 2022). These architectures augment the message- messsag-passing framework with a global attention mechanism, enabling any node in the graph to directly attend any other node. A conceptual representation of GraphGPS layer's update for node e
h( (k-1)) (FN(() (LocalMp(H) (k)
(h (B (k=)). + GlobalAttention (k) (1) (1)))
(10)
This capacity for modeling long-range dependencics makes them, in theory, better suited for pretraining tasks that require a global understanding of the graph. Indeed, this principle demonstrated by ReaCT- Mask (Yang al., 2024), which employs this mansformer-basee GNN encoder to enable information flow between the disconnected components (i.c., reactants and products) of reaction graph. The empirical comparison of these two encoder types in Section 5 is therefore key component of our investigation.
4 Experimental Protocol
We adopt standardized two-stage protocol: (1) self-supervised pretraining on 2M molecules sampled from ZINC15 (Sterling & Trwin 2015 Hu et al. 2019), and (2) fine-tuning and evaluation on 11 MoleculeNet benchmarks (Wu et al. 2018), with supplementary validation on curated datasets from Polaris (Wognum
ct al. 2024 Seee] A.5)
We primarily compare two encoder backbones: GIN and GraphGPS, both implemented with odge-aware GINE layers (Hu et al. 2019). The hidden dimension fixed to 300, trained for 100 epochs with Adam optimizer. To ensure fairness, mask ratios follow prior work but are aligned where needed: 0.15 for
Pretraining.
10

---

Published in Transactions on Machine Learning Research (11/2025)
AttrMask-family baselines, 0.25 for GraphMAE/StrnttMAA,, and 0.30 (with 50% intra-motif atom masking) for MotifPred. All other hyperperameters (batch size, dropout, learning rate) are summarized in Table The decoder is a single-layer MLP by default, except GrapIMAE/SttruttAAB which adopt GNN-based decoder (PRcLU + linear + GIN layer), see A.8 for more details. Method-specific modules such as vector quantizers and masking scorers are documented Table 2
Table 1: Pretraining configuration of two backbone models.
Component
GIN
GraphGPS
Encoder layers
5 GIN layers
5 GPS blocks
Hidden dineension
300
Dropout
0.0
0.0
Attention heads
Optimizer
Adam
Learning
1 10-3
Batch size
256
256
Dropout rate
0.0
0.0 (GIN) 0.5 (Artn)
Epochs
100
100
Table 2: Additional components used by specific pretraining methods.
Method
Component
Key Configuration
MAM
Vector Quantizer Qo
Codebook size 512, token dim 300, commitment cost 0.25.
Tokenizer To
5-layer GIN with hidden dim 300; trained jointly with VQ codebook for 60 epochs.
StructMAE-L
Node Importance Scorer
2-layer MLP and 1-layer GIN (each with in- put/output dim 300); their outputs are aggre- gated and pooled to produce scalar node scores.
MotifPred
Motif-Atom Map
Pre-computed mapping between motifs and con- stituent atoms.
Fine-tuning.
For downstream lasks, we attach a linear prediction head and fine-tune the encoder using scaffold-based 8:1:1 splits. Classification and regression tasks follow the settings in Table 3 To reduce variance, all experiments are repeated with random seeds.
Table 3: Fine-tuning configuration across task types.
Parameter
Classification
Regression
Prediction head
Linear layer (input dim 300)
Optimizer Learning rate
Adam
1 x 10-3
Epochs
100
100
Dropout rate
0.5
0.2
Batch size
32
256
Method Mapping. Finally, Table categorizes the key baselines along the three design dimensions (mask- ing distribution, prediction target, encoder). This provides a compact reference for how each method is positioned within our framework. Full implementation details are deferred to Table g
SupLearn denotes the randomly initialized model train on downstream labels from scratch. Methods with the suffix (T) indicate the encoder is GraphGPS; otherwise, GIN. Suffix -P indicates PageRank based masking, and -I. indicates learnable masking.
Remark:
11

---

Published in Transactions on Machine Learning Research (11/2025)
Table Categorization of methods
Method
Masking Dist.
Prediction Target
Encoder
Mask Ratio
SupLearn
GIN, GPS
AttrMask
Uniform
Atom Type
GIN, GPS
0.15
MAM-A
Uniform
Argmax Label
GIN, GPS
0.15
MAM-VQ
Uniform
VQ Code
GIN, GPS
0.15
MotifPred
Uniform
Motif Label
GIN, GPS
0.15
MoAMa
Intra-motif
Atom Type
GIN, GPS
0.15
GraphMAE
Uniform
Atom Type
GIN
0.25
StructMAE-I
PageRank
Atom Type
GIN
0.25
StruetMAB-L
Learnable
Atom Type
GIN
0.25
5 Experimental Results
5.1 Masking Distribution
We compare uniform, heuristic (PageRank), and learnable masking distributions across both classifica- tion and regression tasks. We additionally implemented heuristic and learnable variants (Aurraask-D /T. MotifPred-PP/LI implementation details are in A.10
80.0
Uniform
PageRank Learned
77.5
75.0
72.8
Average ROC ROC-AUT AUC
72.5
71.0711
71.3
71.6
71.6
71.4
70.6
70.0
65.0
62.5
GraphMAE
MotifPred
AttrMask (T)
MotifPred (T)
Figure 3: Effect of masking distributions on MoleculeNet classification
5.1.1 Discrete Molecular Properties
Figure reports the average ROC-AUC (%), with error bars denote the standard deviations over 5 random seeds. Each group of bars corresponds specific pretraining method (GraphMAE, MotifPred, Attr (T), and MotifPred (T)), where the prediction target and model architecture are fixed, and only the masking distribution is varied. It. shows that across all baselines, neither PageRank- nor learnable masking consistently outperforms uniform sampling. This is not an artifact of fine-tuning, we also conducted linear probing experiments, where the pretrained encoder is frozen and only a lincar head trained (see Table 12 in A.1.2). The results confirm the same trend: non-uniform masking does not provide improvements over uniform masking.
12

---

Published in Transactions on Machine Learning Research (11/2025)
We next examine the mutual information between the downstream label Y and the masked label X under each sampling strategy to further interpret these results.
0.4003
0.0007
0.001
0.0007
0.008
0.0010
0.0003
0.0008
0.4003
0.004
0.422
0.0002
0.0000
Unifom
Pegetank Sider
Leumzblee
Unlom
Pagefark Tox23
Leamublle
Uniferm
Pegefank ToxCast
Leumable
Usilam
Pagulark
Lammbbli
0.0823
0.0023
0.0015
6,00125
0.0015
Lucel Tape
5,00120
Uniform
0.0000
Pageflank
0.0013
Learnable
0.0055
0.0005
9,80925
0.0000
9,60000
0.0003
Unifmm
PageRank
Leamstlle
UnHom
PageRank
Leamable
Uniom
Pagelant
Learnable
Figure 4: MI Between Sampled Atom Labels and Property Labels Across Different Masking Strategies
Mutual Information Analysis We compute the mutual information between X and Y under each sampling strategy. Here, X defined as the sampled atom type label, while the sampling distribution varies over the node V. For each distribution, we sample [V] nodes from every graph G, pair each sampled node label with the corresponding graph-level property label y, and estimate the joint distribution of (X, (X, The mutual information is then computed separately for each classification datase
Each bar in Figure 4 shows the average MI over five random seeds, with error bars denoting standard deviation. Across datasets, MI scores under different masking strategies are very similar, with no consistent gain from PageRank-based or learnable masking. This indicates that structure-guided masking has little impact on the dependence between sampled atom labels X and downstream labels Y. The analysis is model-agnostic and complements Figure where prediction target and encoder are fixed. The consistently small MI variation across strategies explains why downstream performance remains stable despite changes in the sampling distribution.
5.1.2 Continuous Molecular Properties
Table reports the RMSE on four datasets with regression tasks.
*For multi-task datasets with multiple graph-level labels (e.g., Sider, ToxCast), compute MI single tractability.
13

---

Published in Transactions on Machine Learning Research (11/2025)
ESOI
Lipophilicity 4,200
Malaria
CEP
Average RMSE
of data
1,128
9,999
29,978
SupLearn
1.387 (0.087)
0.796 (0.019)
1.105 (0.011)
1,341 (0.010)
1.157
GraphMAE
1.195 (0.024)
0.781 (0.011)
1.116 (0.002)
1.384 (0.016)
1.119
StructMAE-P
1.195 (0.021)
0.762 (0.011)
1.119 (0.009)
1.385 (0.016)
1.115
StructMAB-L.
1.310 (0.029)
0.756 (0.015)
1.111 (0.015)
1.357 (0.007)
1.134
SupLearn (T)
1.036 (0.084)
0.744 (0.039)
1.130 (0.007)
1.689 (0.064)
1.150
AttrMask (T)
1.194 (0.073)
0.747 (0.015)
1.105 (0.013)
1.260 (0.027)
1.077
AttrMask-P (T)
1.010 (0.043)
0.693 (0.023)
1.127 (0.016)
1.482 (0.047)
1.078
AttrMask-L (T)
1.166 (0.031)
0.770 (0.018)
1.127 (0.014)
1.263 (0.043)
1.082
Table 5: MoleculeNet: Regression Tasks (RMSE) over Different Masking Distribution
Across all evaluated models, including GraphMAE variants, and AttrMask (T) variants, observe con- sistent benefit from using structure-aware masking strategies. In all cases, performance remains comparable across sampling methods, with variations falling within the expected range of random training noisc
The results on continuous molecular property prediction reinforce our conclusion that modilying the masking distribution yields limited benefits. In particular, on Malaria, none of the pretrained variants outperform the supervised bascline trained from scratch. On CEP, while the Transformer-bet based methods show bet- ter performance than the supervised baselines, the benefit appears orthogonal the choice of masking distribution.
Finally, beyond downstream performance and information- alignment, we also note the practical implications of computational cost. As quantified in our pre-training time comparison (see A.3 for details), sophisticated heuristic and learnable masking strategics introduce significant computational overhead, with some methods being 2-4x slower than the simple uniform baseline. This cost, combined with their lack of performance benefits, reinforces the practicality of uniform sampling.
5.2 Prediction Target
We next analyze how different prodiction targets affect pretraining effectiveness, by examining their informativeness as supervision signals.
Table 6: Formal definitions of prediction targets used in pretraining.
Target Type
Definition
Method Name
Atom Type
Xiyne : V + (0. 1...., 118) (Element class)
AttrMask
Argmax Label
X V + (0, (1...,,51]... arg max,/Tool (v)()) 1811
MAM-A MAM-VQ MotifPred
VQ Code
Xyo V + {0, 1,..,,11),, x -+ arg arg min - 2o 2.14.110
Motif Label
Xmott F(V) (l....m) (Motif class)
5.2.1 Impact Downstream Performance
The downstream performance of different pretraining targets reported in Table Full results with standard deviations provided in Appendix A.1
14

---

Published in Transactions on Machine Learning Research (11/2025)
Table 7: Classification: ROC-AUC (t); Regression: RMSE (L). Best results per row are in bold.
(a) GIN Encoder
(b) GraphGPS Encoder (T)
Dataset
SupLearn
AttzMask
MAM-A
MAM-VQ
MotifPred
Dalaset
SupLearn
AtarMask
MAM-A
MAM-VQ
MotifPred
Tox21
73.9
15.8
T1.9
75.7
76.6
Tox21
69.6
71.7
75.0
73.8
76.5
ToxCast
63.6
64.3
61.7
63.3
04.5
ToxCast
59.1
64.7
64.5
63.9
67.1
Sider
57.7
60.2
58.2
59.4
60.5
Sider
57.9
39.1
60.5
00.0
37.7
72.3
MUV
75.5
75.4
HIV
74.3
76.5
76.8
76.9
76.8
HTV
68.8
76.9
76.0
75.9
78.9
BEBP
67.7
63.4
65.4
64.6
64.7
BBEP
59.8
68.1
67.6
65.8
68.2
Bace
68.8
78.0
80.9
78.1
79.3
Race
70.3
81.2
79.9
80.5
84.3
Average
70.1
70.8
70.6
Average
64.9
71.3
70.8
ESOL
1.155
ESOL
1,036
1.194
1.356
Lipo
0,796
0.781
0.768
0.759
0.726
Lipo
0.744
0.747
0.862
0.793
0.688
Malaria
1.105
1.116
1.133
1.145
1.110
Malaria
1.130
1.105
1.123
1.115
1.084
CEP
1.311
1.381
1.367
1.334
1,338
CEP
1.689
1.260
1,108
1,337
1.139
Average
1.157
1.119
1.166
1.106
Average
1.150
1.077
1.187
1.136
0.974
While node-level targets yields similar results, motif prediction consistently perform best, especially with GraphGPS. To explain this gap; we again utilize MI to study the statistical dependence between local and graph-level labels.
Mutual Information Analysis Unlike Section 5.1 here MI is computed exactly by enumerating (X, over La S(G), which equivalent to taking expectation under the uniform distribution of X.
For all three node-level labels, it is naturally guaranteed that the atoms in the downstream datasets come from the same element set as those in the pretraining data. However, for motif-level labels, such consistency is not inherently ensured, since downstream molecules may contain motifs that were not observed during pretraining. To ensure the validity of this analysis also applied for motif labels, we evaluated the coverage of downstream motifs within the pretraining motif vocabulary. On average, 67.3% of motif classes in each downstream dataset are found in the pretraining vocabulary. More importantly, across all datasets, over 92.9 99%5%%%% of molecules contain least 80% pretraining-sedt motifs, demonstrating substantial overlap
5.30
0.036
0.00025
0.018
0.03
5.56
0.030
0.02
0.000
0.21
5.02
0.002
Atom
Argmax
Medl
Atom
Argina
V3
Moof
Abmm
Argiax
VQ
Mvol
Atom
Argiax
MVL
0.038
0.012
0.010
Type
0.008
Argmax Label
Vi Code
Arom
Argmax
vg
Hetr
Atom
Argias
vg
Meaf
Atmm
Argmaa
VQ
Figure 5: Ml between local labels and graph label (per dataset)
Sper-dataset statistics provided Appendix
15

---

Published in Transactions on Machine Learning Research (11/2025)
Figure 5 shows that motif labels yield consistently higher MI with graph-level labels than all node-level alternatives. Although absolute MT values are small due to the upper bound H(Y), motif-level labels consistently explain 4 11% of the label entropy (see A.7).
Analysis of Conditional Label Distributions Although VQ and Argmax achieve higher average MI than atom types, this does not translate into proportional downstream gains (Table 7 To probe this discrepancy, we analyze conditional label distributions using Jensen Shannon Divergence (Eq. to test our hypothesis about the importance of low- cowfrreuency labels.
Bace Dataset: Binary Graph Labels for &- S-Secretase 1 infibitor
HIV Dataset: Binary Graph Labels for ArdH-HV Activily Screering Resuls
Aom tyze Aegmas
Alom Type
Angrax Labe
0.20
Diveggess
6.15
0.15
0.05
0.03
5.00
6.4 Irqueeeyyyyooo......mm.nn
Figure 6: -hanenn-Shanon Divergence (JSD) between conditional local label distributions P(X = 1, and P(X = -,, for two representative datasets: Bace and HIV.
As shown in Figure we observe a consistent trend across both datasets: motif labels yield substantially higher JSD values than node-level labels when the threshold t decreases, i.e. when we focus on increasingly rare local labels. Additional results for other labels from these datasets are provided in Appendix A.66.
This trend becomes especially pronounced at lower frequency thresholds (e.g., S 0.1) where the divergence between P(X S,) and P(X 0, S,) for motif labels sharply increases, whereas the JSD values for node-level labels remain relatively flat. We further include shuffle-cnttrol experiment in Appendix A.6.2 to confirm that the observed gain is not attributable to vocabulary size. These results empirically support our hypothesis that low-frequency motifs carry more discriminative information with respect to molccular properties, likely because such motifs correspond to functional groups or structural patterns with specific bioactivity or chemical relevance.
In summary, motif labels provide more informative supervision than node-level alternatives. MI confirms stronger dependence on graph-level properties, and JSD highlights the discriminative power of rare mo- tifs. These results support the intuition that functional substructures encode richer, task-relevant chemical semantics than individual atoms.
5.3 Encoder Architecture
This section investigates how the choice of encoder architecture influences the effectiveness of different pretraining largets. In addition to previously discussed objectives, we include MoAMa, which employs motif-level masking while predicting node-level targets. This mismatch in semantic granularity offers a valuable case study for understanding encoder-target alignment.
5.3.1 Performance Comparison Across Encoders
Figure 7 summarizes the downstream performance of key pretraining strategies when implemented with GIN versus GraphGPS encoders. A few observations emerge.
Firstly, both AttrMask and MotifPred show consistent gains when moving from GIN to GraphGPS. However, their implications differ. For AttrMask, GraphGPS reaches about 71.4% ROC-AUC, level that already
16

---

Published in Transactions on Machine Learning Research (11/2025)
1.6
12.
71.4
1.4
HOCC-MAC (ho
64-4
1.2
1:577
3,750
1.119
62-
0.5
Alla Mask
McAMa
MotdPredd
Suplear
MoAMa
MotiPeed
(a) Average ROC-AUC (t) of Classification Tasks
(b) Average RMSE (L) of Regression Tasks
Figure 7: Comparison of downstream performance across encoder architectures (GIN vs. GraphGPS) for different masking designs.
be matched by GIN under alternative mask rates (see A.2 Thus, the observed improvement does not reflect a qualitatively new performance regime. In contrast, MotifPred with GraphGPS reliably achieves ~72.9% a higher regime that all examined methods cannot consistently reach with GIN.
Additionally, mothods that use motif-level masking but retain atom-level prediction targets (i.c., MoAMa and Moti(Pred-A) show only marginal improvements with GraphGPS.
5.3.2 Encoder- Target Compatibility
These results suggest that structural complexity in masking is insufficient by itself; what matters more is the semantic richness of the prediction target. Even with motif-aware input perturbations, if the supervision remains atom-level, the task is fundamentally local and does not compel GraphGPS to exploit its global receptive field.
This comparison also echoes our earlier analysis of masking distributions. MoAMa, as described Sec- tion 3.3 employs a 'non-adjacent" motif selection hcuristic that increases pretraining time by about 220%. Despite this computational overhead, it provides no performance advantage over simple uniform sampling and can even perform slightly worse GIN. The case of MoAMa reinforces central conclusion: merely engineering more complex masking distribution PM, without elevating the semantic richness of the pre- diction target X, is unlikely to be promising direction for improving pre-trained graph models.
6 Discussion
6.1 A Formal Framework for Principled Comparison
A primary contribution of this work is the introduction of a formal probabilistic framework to deconstruct and analyze the design space of basking-based SSL on molecular graphs. By modeling the pretraining task process of fitting random variable X S(G) - & from structural units to label space, we beyond ad-hoc comparisons and systematically investigate the distinct roles of its components: the masking distribution (PM), the prediction target (X), and the encoder architecture (fv). This principled formulation guides our entire investigation and enables us to isolate the impact of each design choice.
6.2 The Prediction Target Outweighs the Masking Distribution
Our systematic investigation reveals a clear hicrarchy of importance among the design dimensions. The cen- tral finding of this work is that the choice of what to predict (the prediction target)) is substantially
17

---

Published in Transactions on Machine Learning Research (11/2025)
more pivotal than the choice of where to mask (the masking distribution). Our probabilis- tic framework allows us to make the underlying hypothesis for sophisticated masking strategies explicit: that an optimal, non-uniform distribution PM should make the pre-training signal X more informative for the downstream task Y, which would manifest as a higher mutual information, I(X;Y). However, our information- analysis of several represenlative heuristic and learnable strategies (Sec. 5.1.1 finds no evidence to support this hypothesis. This lack of more informative signal, combined with their significant computational overhead (Appendix A.3) explains their failure to outperform simple uniform sampling in our experiments.
While this does not entirely preclude the existence of a more effective distribution, our work proposes more resource-effcient methodology for future explorations. Rather than relying solely on expensive downstream evaluations, researchers can first leverage our framework as a low-cost litmus test: if a novel distribution demonstrably increases Y( it warrants further investigation. Otherwise, our findings suggest that efforts are more fruitfully directed towards designing richer prodiction targets.
6.3 The Critical Synergy Between Encoder and Target
A second key insight the critical role of synergy between the encoder architecture and the pre- diction target. Our results consistently show that while standard MPNNs like GIN are well-suited for local, atom-level reconstruction tasks, their strong local inductive bias limits their ability to fully capitalize semantically richer, non-local targets. In contrast, expressive Graph Transformer architectures, with their global attention mechanism, unlock significant performance gains when paired with motif-level predic- tion. This highlights that the benefits of a more powerful encoder are not universal but are contingent on being paired with a pretraining task that requires its advanced capabilities, such as modeling long-range dependencies to understand the semantics a larger substructure.
6.4 Implications for Future Research: The Quest for Semantically Rich Targets
Our findings strongly advocate for shifting focus towards semantically richer prediction targets. This natu- rally raises the question: what constitutes semantic richness in the context of molccular SSL? Our work pro- vides a comparative answer. While learned discrete tokens from methods like MAM represent a data-driven form of semantics, they appear less effective than human-curated chemical concepts like BRICS-defined motifs. Our informationn-teorrticc analysis corroborates this, showing that motif labels have stronger sta- tistical dependence whose label space L is defined by explicit, chemically-awan potent supervisory signal than purely abstract, learned representations. An exciting avenue for future work could be the development of hybrid targets that combine the best of both worlds-I to discover novel, meaningful substructures that go beyond traditional, humnan-defined motifs.
downstream properties. This suggests that, at least for now, pretraining targets X structural knowledge provide a more
7 Conclusion
By formalizing the molecular graph masking pipeline within a probabilistic framework and leveraging inforeation-theorrticc measures to assess task alignment, we conducted a systematic investigation into the core design dimensions of self-supervisod lcarning. Our investigation concludes that the various design di- mensions do not. hold equal weight: the choice of a semantically rich prediction target is the most critical driver of performance, whose full potential is only realized through strong synergy with an expressive encoder architecture. In contrast, sophisticated masking distributions offer limited performance gains at a higher computational cost. These insights, derived from principled and reprodueible methodology, pro- vide a clearer and more resource-efficient roadmap for developing the next generation of SSL methods for molecular property prediction.
18

---

Published in Transactions on Machine Learning Research (11/2025)
References
Junyi An, Chao Qu, Yun-Fei Shi, XinHao Liu, Qianwei Tang, Fenglei Cao, and Yuan Qi. Equivariant masked position prediction for efficient molecular representation. In The Thirteenth International Conference on Learning Represeniations, 2025. URL https://openreeiex http://wweeeeo.rommm.ccceeeaaaaa
David Buterez, Jon Paul Janet, Dino Oglic, and Pictro Liò. An end-to-end attention-based approach for learning on graphs. Nature Communications, 16(1):5244, 2025.
Zhixiang Cheng, Hongxin Xiang, Pengsen Ma, Li Zeng, Xin Jin, Xixi Yang. Jianxin Lin, Yang Deng, Bosheng Song, Xinxin Feng, Changhui Deng, and Xiangxiang Zeng. Maskmol: Knowledge-guided molecular image pro-training framework for activity cliffs, 2024. URL https::/arxxv.org// strggabs/24005 12926
Celia Cintas, Payel Das, Jarret Ross, Brian Belgodere, Girmaw Abebe Tadesse, Vijil Chenthamarakshan, Jannis Born, and Skyler Speakman. Characterizing pre-trained and task-adapted molccular representa- tions. In UniReps: the First. Workshop on Unifying Representations in Neural Models, 2023.
Suresh Dara, Swetha Dhamercherla, Surender Singh Jadav, CH Madhu Babu, and Mohamed Jawed Ahsan.
Machine learning drug discovery: review. Artificial intelligence review, 55(3):1947-199) 1999, 2022.
Jorg Degen, Christof Wegscheid-Gerie using' drug-like'chemne chemical fragment spaces. ChemMedChemm, 3(10):1503, 2008.
Andrea Zaliani, and Matthias Rarey. On the art of compiling and
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short popers), pp. 4171 4416, 2019.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru- Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. Advances in neural information processing systems, 28, 2015.
Jonathan M Elkins, Vita Fedele, Marta Szklarz, Kamal R Abdul Azeez, Fidarus Salah, Jowita Mikolajezyk, Sergei Romanov, Nikolai Sepetov, Xi-Ping Huang, Bryan L Roth, et al. Comprehensive characterization of the published kinase inhibitor set. Nalure biolechmology, 34(1):95 103, 2016.
Jinjia Feng, Zhen Wang, Yaliang Li, Bolin Ding, Zhewei Wei, and Hongteng Xu. Mgmae: Molecular repre- sentation learning by reconstructing heterogeneous graphs with high mask ratio. In Proceedings of the 31st ACM International Conference Information & Knowledge Management, pp. 509 009,519, 2022.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263 1272. PMLR, 2017.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and paltern recognition, pp. 16000-16009, 2022.
Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pp. 594-604, 2022.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprint arXiv: 12265, 2019.
Eric Inae, Gang Liu, and Meng Jiang. Motif-aware attribute masking for molecular graph pre-training, In The Third Learning on Graphs Conference, 2024. URL ftttps://operreviee http:///ww....eeee/mom////////oa
Longlong Jing and Yingli Tian. supervip visual feature learning with deep neural networks: A survey. IEFE transactions on paltern analysis and machine intelligence, 43(11): (3(11):4037-400) 2020.
19

---

Published in Transactions on Machine Learning Research (11/2025)
Bonyou Koo and Sunyoung Kwon. Comprehensive analysis of masking techniques in molecular graph repre- sentation learning. IEEE Access, 1-4200--4303 2025. doi: 10.1109/ ACCCSSS.2225.3555
Chuang Liu, Yuyao Wang, Yibing Zhan, Xueqi Ma, Dapeng Tao, Jia Wu, and Wenbin Hu. Where mask: structure-guided masking for graph masked autoencoders. In Proceedings of the Thirly-Third International Joint Conference on Artificial Intelligence, IJCAI '24, 2024. ISBN 978-1-956792.0441 doi: URL
1h.29633/jcaii.0022
24963/ijcaii (02/////2aa.222/4111
Shengehao LIu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pro-training molecular graph representation with 3d geometry. arXiv preprinl arXiv:2100.0771 2021a.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self-supervised learning: Generative or contrastive. IEEE transactions on knowledge and data engineering, 35(1):857-877 876, 2021b.
Zhiyuan Liu, Yaorui Shi, An Zhang, Enzhi Zhang, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. Rethinking tokenizer and decoder in masked graph modeling for molecules. Advances in Neural Information Processing Systems, 36:25854 24225554-25875, 2023.
Qingqing Long, Yuchen Yan, Wentao Cui, Wei Ju, Zhihong Zhu, Yuanchun Zhou, Xuezhi Wang, and Meng Xiao. Moat: Graph prompting for 3d molecular graphs. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM '24, pp. 1586 1596, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704469. doi: https://doi. org/ 10. 1111/1127663...00 36796688
URL
Yuting Ma, Shuo Yu, and Yanning Shen. Pretraining molecules with explicit substructure information. In Proceedings of the 2024 SIAM International Conference on Data Mining (SDM), pp. 517-525 SIAM, 2024.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific dala, 1(1):1 7, 2014.
Ladislav Rampášck, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems, 35:14501-14515 2022.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self- supervised graph transformer on large-scale molecular data. Advances neural information processing systems, 33:12559-12571, 2020.
Teague Sterling and John J. Irwin. Zinc 15 ligand discovery for everyone. Journal of Chemical In Information and Modeling, ((11))2224-2331 2013. doi: 10.1021 /a/cc...jjje jct..cc..mmmm00555.. URL httphtttt///oooooooo 1021/
acs. hec.jccbm0ms........ PMTD: 26479676.
Yijun Tian, Chuxu Zhang, Ziyi Kou, Zheyuan Liu, Xiangliang Zhang, and Nitesh V Chawla. Ugmae: A unified framework for graph masked autoencoders. arXiv preprint arXiv:2402.0800 2024.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advonces in neural information processing systems, 30, 2017.
Hanchen Wang, Jean Kaddour, Shengchao Liu, Jian Tang, Joan Lasenby, and Qi Liu. Evaluating self- supervised learning for molccular graph embeddings. Advances in Neural Information Processing Systems, (1160088-60064 2023.
Sheng Wang, Yuzhi Guo, Yuhong Wang, Hongmao Sun, and Junzhou Huang. Smiles-bert: Large scale unsupervised pre-training for molecular property prediction. In Proceedings of the 10th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB '19, pp. 429 429-436, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450366663, doi: 10. 10.1145/ 1111/3307339. 3342186. URL htttp::///aott wrw.loo.r.o.1... w0w..0111//3/11......0 3342186
20

---

Published in Transactions on Machine Learning Research (11/2025)
Yuxiang Wang, Xiao Yan, Chuang Hu, Quanqing Xu, Chuanhui Yang, Fangcheng Fu, Wentao Zhang, Hao Wang, Bo Du, and Jiawei Jiang. Generative and contrastive paradigms are complementary for graph self-supervised learning. In 2024 IEEE 40th International Conference on Data Engineering (ICDE), pp.
3364-3378. IEEE, 2024.
Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molccular contrastive learning of representations ions via graph neural nełworks. Nature Machine Intelligence, 4(3):279 287, 2022.
Cas Wognum, Jeremy R Ash, Matteo Aldeghi, Raquel Rodriguez-Pérez, Cheng Fang, Alan C Cheng, Daniel J Price, Djork-Arné Clevert, Ola Engkvist, and W Patrick Walters. A call for industry-led initiative to critically assess machine learning for real-world drug discovery. Nature Machine Intelligence, 6(10):1120- 1121, 2024.
Yasu Wu, Changlong Fu, Manwen Yang, Haoran Duan, and Chong Xic. Motif masking-basee self-supervised learning for molecule graph representation learning. In 2023 IEEE International Conference on Bussinsss Engineering (ICEBE), pp.
IEEE, 2023.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenct: a benchmark for molecular machine learning. Chemical science, 9(2):513 530, 2018.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. A comprehensive survey on graph neural networks. IEEE transactions on newral networks and learning systems, 32(1):4-24 24, 2020.
Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z. Li. Mole-BERT: Rethinking pre-training graph neural networks for molecules. In The Eleventh International Conference on Learning Representations, 2023. URL https://opparreveeu wet.://orrur/ceeeeyyy-ttt..
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019. URL httpps:/ ottp://ppmmrriioooooooooorutoo
ryGs61A5Km
Jiannan Yang, Veronika Thost, and Tengfei Ma. Masking in molecular graphs leveraging reaction context. In ICML 2024 Al for Science Workshop, 2024. URL https::/ http::/pppaaeeeeeeeeeeee/ecoaaaa
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph castivvvive learning with augmentations. Advances in neural information processing systems, 33:5812-5823, 2020.
Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph self-supervised lcarning for molecular property prediction. Advances in Neural Information Processing Systems, 34:15870- 15882, 2021.
Kangjie Zheng, Siyue Liang, Junwei Yang, Bin Feng, Zequn Liu, Wei Ju, Zhiping Xiao, and Ming Zhang, SMI-editor: Edit-based SMILES language model with fragment-level supervision. In The Thirteenth International Conference on Learning Representations, 2025. URL httpp//rrrrree...tff.ffuumrdtdtt M29nUGozPal
21

---

Published in Transactions on Machine Learning Research (11/2025)
A Appendix
A.1 Complete Results
This appendix contains the comprehensive results of our downstream evaluation. For clarity and easy reference, we first present Table 8 which summarizes the design choices for every method implemented in this study. The subsequent tables then provide the full, unabridged performance metries (ROC-AUC for classification and RMSE for regression) for all variants of AttrMask across all MoleculeNe benchmark tasks discussed in the main paper. Results for the variants of GraphMAE are presented in latter sections.
Table 8: Configurations for all implemented pre-training methods and their variants, categorized by their primary design choices. Checkmarks (V) indicate the utilized components for each method.
Method
Distribution
Prediction Target
Loss GNN Decoder
Re-mask
Lewrn
Atom Type
Type
Token
Motif Label
Buselines lased on AttrMask
AttrMaakk
CF
AttzNask-B
Foriands with Levrned/ /Structored Torgets
KAM-A
CE
HAN-VQ
Kaz--A-B
CE
CE
MotifPred
CE
MotifPrea-P
CE
MotifPred-1
Mot.ifPred-A
CR
Auto Encoding Variants (GraphMAE' 8 StructMAE)
CraphMAR
SCE
GraphMAE-2
SCE
GraphMAS-CE
CE
StructMAK-P
SCE
StrmctMAR-P-R
SCB
StrucsMAB-P-QE
StructMAR-I.
SCB
StructMAB-L-R
SCE
StrucsMAB-I--E
CE
A.1.1
MoleculeNet: Full Fine-tuning
Tox21
ToxCast 8577
Sider 1427
MUV
HIV
BBBP
Bace
Average
# of data
93087
41127
2039
1513
SupLearn
(0.7)
63.6 (0.6)
57.7 (1.4)
(1.7)
74.3 (1.4)
67.7 (2.5)
68.8 (3.4)
SupLearn (T)
69.6 (0.6)
59.1 (1.3)
57.9 (1.7)
69.1 (0.9)
68.8 (3.7)
59.8 (3.4)
70.3 (5.8)
64.9
AttzMask
75.8 (0.5)
61.3
60.2 (1.1)
72.3 (2.0)
76.5 (1.6)
63.4 (2.3)
78.0 (1.0)
A
76.1 (0.7)
63.9 (0.5)
59.3 (0.6)
72.7 (1.5)
77.6 (0.3)
65.6 (1.8)
77.1 (1.2)
70.3
MAM-A
71.9 (0.8)
61.7 (0.5)
58.2 (0.3)
77.8 (2.3)
76.8 (1.8)
65.4 (1.4)
80.9 (1.1)
70.8
MAM-A-B
76.0 (0.3)
63.8 (0.3)
59.3 (0.8)
74.3 (2.0)
76.5 (1.0)
61.2 (2.7)
77.8 (1.2)
70.3
MAM-VQ
75.7 (0.4)
63.3 (0.3)
59.4 (0.7)
76.0 (1.3)
76.9 (1.1)
64.6 (1.8)
78.1 (1.2)
70.6
MotifPred-A
76.3 (0.3)
64.2 (0.9)
57.6 (0.7)
75.1 (1.3)
(1.5)
67.0 (0.9)
80.1 (0.6)
Moti(Pred
76.6 (0.6)
G1.5
60.5 (0.7)
76.8 (1.7)
76.8 (0.1)
61.7 (2.0)
79.3 (5.0)
MotifPred-P
75.6 (0.4)
63.4
58.2 (0.9)
75.2 (3.1)
77.3 (0.7)
64.4 (2.1)
76.1 (4.3)
70.2
MotifPred-L
77.0 (0.3)
64.4 (0.2)
60.3 (0.7)
77.6 (1.4)
77.1 (1.5)
64.1 (1.2)
80.7 (1.6)
71.6
AttrMask (T)
74.7 (0.4)
64.7 (0.8)
59.1 (0.9)
75.4 (1.9)
76.9 (1.3)
68.1 (0.8)
81.2 (2.5)
71.4
AttrMask.P (T)
75.4 (1.0)
65.6 (1.2)
53.5 (1:3)
78.8 (2.5)
76.7 (2.1)
70.4 (2.0)
82.5 (1.3)
71.6
AtteMakk-L (T)
75.3 (1.1)
64.9 (0.7)
56.1 (0.9)
77.3 (1.6)
75.4 (1.7)
67.6 (1.3)
83.2 (0.5)
71.4
MAM-A
(1.1)
(0.8)
60.5 (1.2)
75.5 (1.6)
(1.3)
67.6 (0.2)
79.9 (2.4)
MAM-VQ (T)
73.8 (1.0)
63.9 (0.4)
60.0 (1.6)
75.4 (0.9)
75.9 (1.8)
65.8 (3.8)
80.5 (2.0)
70.8
MotiPPeed-A (T)
75.3 (1.1)
61.9 (0.6)
56.8 (2.0)
74.6 (2.6)
74.2 (1.2)
69.7 (0.9)
83.0 (1.3)
71.2
MotifPred (T)
76.5 (0.4)
67.1 (0.9)
57.7 (1.6)
77.3 (2.0)
78.9 (1.1)
68.2 (1.5)
84.3 (2.2)
72.9
MotifPred-P (T)
76.4 (0.6)
66.1 (0.6)
58. 1 (1.6)
76. (2.2)
75.9 (0.1)
61.7 (1.6)
83.8 (2.0)
71.7
MottiPred-L (T)
76.6 (0.8)
65.4 (0.6)
57.7 (0.6)
78.1 (1.2)
76.8 (0.7)
69.8 (0.7)
85.2 (1.1)
72.8
Table 9:
Comparison among AttrMask-based approaches with modified objectives (with standard deviation)
We exclude the Clintox dataset. from our evaluation due its severe class imbalance and known data quality issues, which can lead to misleading performance metries.
22

---

Published in Transactions on Machine Learning Research (11/2025)
ESOL 1,128
Lipophilicity 4,200
Malaria
CEP
Average RMSE
# of data
9,999
29,978
SupLearn
1.387 (0.087)
0.796 (0.019)
1.105 (0.011)
1.341 (0.010)
1.157
SupLearn (T)
1.036 (0.084)
0.744 (0.039)
1.130 (0.007)
1.689 (0.0641)
1.150
AttrMask
1.195 (0.024)
0.781 (0.011)
1.116 (0.002)
1.384 (0.016)
1.119
AutrMask-B
1.191 (0.028)
0.759 (0.010)
1.124 (0.020)
1.343 (0.025)
1.104
MoAMa
1.212 (0.022)
0.773 (0.006)
1.125 (0.009)
1.314 (0.014)
1.114
MAM-A
1,386 (0.020)
0.768 (0.014)
1.143 (0.023)
1.367 (0.013)
1.166
MAM-A-B
1.191 (0.038)
0.759 (0.017)
1.121 (0.012)
1.340 (0.005)
1.103
MA
1,187 (0.025)
0.759 (0.009)
1.145 (0.008)
1.334 (0.015)
1.106
MotifPred
1.151 (0.027)
0.726 (0.008)
1.110 (0.009)
1.338 (0.035)
1.081
ArtrMask (T)
1.191 (0.073)
0.747 (0.015)
1.105 (0.013)
1.260 (0.027)
1.077
MoAMa (T)
1.041 (0.051)
0.777 (0.018)
1.100 (0.013)
1.279 (0.048)
1.049
MAM-A (T)
1.356 (0.077)
0.862 (0.034)
1.123 (0.006)
1,408 (0.028)
1.187
MA (T)
1.297 (0.029)
0.793 (0.020)
1.115 (0.010)
1.337 (0.032)
1.136
MotifPred (T)
0.984 (0.025)
0.688 (0.007)
1.084 (0.010) 1.139 (0.010)
0.974
Table 10: NoleculeNet: Regression Tasks (RMSE)
A.1.2 Addtional Comparisons:
Linear Probing
In the following tables, freeze the pretrained parameters of GNN backbones, only the MLP classifier is trained during fine-tuning. The results are collected with 5 random seeds.
Tox21 7831
ToxClast 8577
Sider 1427
MUV
Average
# of data
93087
41127
20339
1513
AttrMask
70.5 (0.1)
59.6 (0.1)
52.6 (0.3)
69.0 (0.7)
66.6 (1.3)
58.3 (0.1)
61.3(3.4)
62.6 (0.6)
MotifPred
69.1 (0.1)
61.5 (0.2)
50.5 (1.9)
72.1 (0.9)
64.7 (0.8)
60.3 (0.3)
65.5 (5.4)
63.4 (0.7)
AttzMask (T)
71.9 (0.1)
60.3 (0.1)
51.8 (0.0)
61.2 (0.3)
63.4 (0.8)
53.8 (0.2)
71.5 (0.4)
63.3 (0.1)
MotifPred (T)
(0.1)
(0.1)
59.0 (0.4)
70.6 (0.2)
74.4 (0.2)
66.8 (U.1)
77.4 (4.2)
(0.6)
Table 11: Linear Probing: among AttrMask- approaches
Tox21
ToxCast 8677
Sider 1427
MUV
HIV
BBBP 2039
Bace
Average
# of data
7831
93087
11127
1513
GraphMME
68.5 (0.2)
62.1 (0.3)
58.5 (1.3)
72.9 (1.9)
66.3 (0.6)
60.3 (2.9)
73.0 (2.2)
65.9 (0.7)
StructMAE.P
67.9 (0.3)
60.2 (0.3)
56.0 (0.5)
72.3 (1.6)
64.6 (0.6)
56.5 (0.4)
76.3 (0.9)
64.8 (0.2)
StrictMAFLL
70.6 (0.4)
59.3 (0.4)
56.4 (1.0)
74.9 (2.0)
61.9 (2.6)
59.1 (0.4)
64.2 (0.9)
63.8 (0.4)
Table 12: Linear Probing: among GraphMAE-based approaches
23

---

Published in Transactions on Machine Learning Research (11/2025)
A.2 Mask Ratio
Modi
Vlace ko00
(a) Masking Distribution
(b) Atom Type vs. Motif Label
Figure 8: Mask Ratio Sensitivity Analysis
The mask-ratio sensitivity analysis (Fig. 8) offers two key observations. First, although both atom-type prediction and motif prediction ultimately converge to comparable performance, their preferred mask ratios differ: MotifPred (T) achieves its peak at relatively low ratio (0.15), while higher ratios (e.g., 0.40) lead to a noticeable drop. For MotifPred, a 0.40 ratio corresponds to masking 40% of atoms while recovering 80% of motifs. This more aggressive corruption likely disrupts the structural context needed for motif-level labels, while atom-level targets remain less sensitive.
Second, for GraphMAE and its variants, performance remains closely aligned across different masking dis- tributions under all mask ratios. This consistency reinforces our earlier conclusion: modifying the sampling distribution provides little advantage over uniform masking, regardless of the corruption level.
A.3 Pretraining Time Comparison
All models were pre-trained for 100 epochs on 2 million molccules from ZINC15 using single NVIDIA A6000 GPU. For data loading, we utilized 8 parallel workers. The relative slowdown is calculated against the AttrMask (GIN) baseline.
Note that the training for GraphMAE is more efficient than AttrMask due to its pre-computation of one-hot vectors of atom attributes for training loss computation.
Table 13: Comparison of pretraining time for key model configurations.
Method
Key Design Choice
Pretraining Time (Hours)
Relative Slowdown
AttrMask (GIN)
Uniform Masking, Arom Target (Baseline)
11.7
L.xx
AttrMask (T)
Uniform Masking, Transformer Encoder
MoAMa (GIN)
Motif Masking (On-the-fly)
37.2
3.2x
MoAMa (T)
Motif Masking, Transformer Eneader
49.2
4.2x
GraphMAE (GIN)
Uniform Masking
7.8
0.7x
StricsMAEPP (GIN)
PageRank Masking (PageRankk)
24.4
StruciMAE-L (GIN)
Learnable Masking
27.5
2.4x
MonifProd (GIN)
Motif Target (Pre-rompated)
18.6
1.6x
MotifPred (I)
Motif Target, Transfrmer Encoder
23.5
2.0g
A.3.1 Asymptotic Cost of Masking Strategies
Let an input graph be G = E). The overall complexity of our pretraining pipeline is dominated by the encoder backbone, both GINE and GraphGPS scaling lincarly with the graph size, i.c., O(V]+ |E]) (Xu) et al. 2019 Rampášck al. 2022). Below we analyze the additional asymptotic cost introduced by different masking distributions.
24

---

Published in Transactions on Machine Learning Research (11/2025)
GraphMAE (Uniform). Random node indices are drawn uniformly and masked through vectorized assigmment adding at most O(VV) overhead. The total complexity remains O([V) + [F])
StructMAE-P (PageRank-based), Node importance scores are computed via power iteration for PageRank, ([) with T iterations to convergence. Two per-graph top-k selections for masking incur O(IV) log VI) each. The per-batch cost is therefore O + log
StructMAF-L (Tearnable). Replaces PageRank with an MLP + GNN scorer, O(IVI + [E],, followed by the same two top-k operations, giving O(IV) [V] + [E]) overall.
All variants share the same memory footprint as the backbone, O(VV) + [E]), since masking is performed online without pre-computation. In summary, non-uniform masking incurs additional (([V log (Vl) cost for node scoring and sorting, explaining the observed computational overhead. See AA.1 for details.
A.4 Ablation Studies on Auxiliary Components
In this section, we conduct ablation studies on several auxiliary components proposed in the reproduced works to assess their impact on downstream performance.
Tox21
ToxCast 8577
Sider 1427
MUV 93087
HIV 41127
BBBP 2039
Bace
Average
# of data
7831
1513
GraphMAE
75.3 (0.6)
64.1 (0.5)
58.4 (0.5)
74.9 (1.5)
76.5 (1.6)
67.2 (2.7)
80.7 (2.5)
71.0
GraphMAE-R
75.2 (0.4)
63.9 (0.5)
59.0 (0.9)
74.2 (2.2)
77.1 (1.3)
64.1 (1.3)
80.8 (1.4)
70.6
GraphMAF-CR
76.3 (0.4)
64.0 (0.4)
58.2 (0.7)
74.9 (3.4)
76.0 (1.2)
64.0 (1.9)
82.1 (1.0)
70.8
StructMAE-P
75.5 (0.6)
63.6 (0.3)
58.6 (U.8)
73.7 (2.7)
76.9 (0.9)
67.6 (3.5)
82.0 (1.1)
71.1
StructMAE-P-E
75.6 (0.3)
64.0 (0.2)
59.2 (0.8)
75.4 (1.0)
76.5 (1.8)
64.3 (1.9)
81.7 (1.2)
71.0
StrucIMAE-P-CE
75.1 (0.4)
64.2 (0.4)
59.9 (0.9)
74.6 (2.0)
76.3 (1.2)
68.4 (2.0)
83.2 (1.4)
71.7
StructMAAEL
75.4 (0.6)
63.9 (0.4)
59.6 (0.9)
73.6 (1.1)
76.7 (1.5)
65.2 (2.7)
79.9 (0.7)
70.6
StrucLMAE-L-R
75.2 (0.2)
63.3 (0.5)
59.6 (0.8)
75.8 (1.1)
76.0 (1.3)
61.3 (2.0)
78.1 (4.7)
69.9
StructMAE-L-CE
76.3 (0.4)
64.0 (0.4)
58.2 (0.7)
74.9 (3.4)
76.0 (1.2)
64.0 (1.9)
82.1 (1.0)
70.8
Table 14: Comparison among GraphMAF-based approaches
A.4.1 Edge Masking
Our investigation also included edge attribute masking, where the model is pre-trained to predict the type of masked bonds. Similar to findings on masking distributions, this strategy did not yield significant performance advantages (sce Table g and IO]. We attribute this to two primary factors. First, predicting a bond's type is a fundamentally local task. The graph topology and surrounding atoms often provide sufficient context for reconstruction. Second, the semantic information encoded in standard bond types (c.g.. single, double) inherently limited. Consequently, the supervisory signal generated from this task appears insufficient drive the learning of powerful representations needed for complex graph-level properties.
A.4.2 Decoder Architecture
The results show that replacing the MLP decoder with a GNN-basod one brings no consistent difference under our current settings. This is likely because the pretraining task relatively chaple-predicting discrete atom- type labels rather than high-dimensional targets. More expressive decoders may become beneficial more complex reconstruction settings, as suggested by SimSGT (Liu et al. 2023), though we leave such verification to future work.
A.4.3 Loss Function
We compare the standard Cross-Entropy (CE) loss with the Scaled Cosine Error (SCE) loss, which was proposed by GraphMAE to down-weight easy examples. The comparison is made across three different masking distributions. As shown in Table 14 we consistently observe that models trained with the CE loss
25

---

Published in Transactions on Machine Learning Research (11/2025)
(e.g., StructMAE-P-CE at 71.7%) outperform their counterparts trained with SCE (e.g., StructMAE-PPR at 71.0%). This suggests that for these atomic attribute reconstruction tasks, the standard CE loss remains more effective and robust choice.
A.4.4 Re-masking
The re-masking technique, introduced by GraphMAE, involves masking the latent representations of already- masked nodes before feeding them to the decoder. We evaluated this trick across the GraphMAE, StructMAE-P, and StructMAE-L frameworks. The results in Table 14 show no clear benefit from this technique. In three pairs, the model with re-masking (denoted by the (-R) sullix) performs either com- parably to or slightly worse than the model without it (e.g., GraphMAE: 71.0% vs. GraphMAE-R: 70.6%), We, therefore, conclude that the ro-masking step, least within experimental setup, does not provide consistent advantage and adds unnecessary complexity to the pretraining pipeline.
A.5 Polaris Benchmarks
A.5.1 A Case Study on Low-Data Regime: The Polaris PKIS Benchmark
Our experiments on the Polaris PKIS (Elkins 2016)
benchmark largely reinforced the main conclusions from broader study. As shown in Figure again found that sophisticated masking distributions (o.g.: PageRank-based vs. Uniform) and variations among different node-level prediction targets (e.g., Atom Type vs. Argmax) offered no significant advantage over their simplest counterparts.
GraphGGt
922
216
20.5
Average RDC.-at
Blaee lime
Masking Distribution
Prediction Target
Encoder Architecture
Figure 9: Average ROC-AUC (%)
- sid on PKIS Classification over 5 Runs
However, this benchmark revealed one notable exception. In direct reversal of the trend observed larger datascts, the simpler AttrMask(T) model empirically outperformed the more powerful MotifPred(T)) This phenomenon does not contradict our core findings. Instead, we altribute this performance inversion primarily to overfitting. Indeed, the PKIS dataset contains only 640 molecule. As illustrated in Figure [0] this reflocted the training curves: MotifPred(T) converges significantly faster and a near-perfect ROC-AUC on the training set, which indicates that its pre-trained features are indeed more powerfully aligned with the task.
26

---

Published in Transactions on Machine Learning Research (11/2025)
1.00
0.95
0.90
Train ROC-AU C-AUC (%)
0.85
0.80
0.75
0.70
0.65
MotifPred (T)
0.60
AttrMask (T)
0
20
40
60
80
100
Epoch
Figure 10: On PKIS, MotifPred(T) converges faster than AttrMask(1)
This result serves crucial case study highlighting that in data-scarce downstream applications, model's robustness to overfitting can be more decisive factor than the theoretical richness of its pretraining signal. The simpler AttrMask task may inadvertently act as regularizer, leading to less powerful but ultimately more generalizable model for this specific application.
A.5.2 Additional Regression Results: The Polaris ADME Benchmark
adme-microsomal
adme-sol 2,173
adme-ppb 115
adme-perm 2,642
Average
# of data
3,049
SupLearn
0.561 (0.018)
0.58 (0.010)
0.969
(0.174)
0.668 (0.044)
0.695
SupLearn (T)
0.534 (0.015)
0.556 (0.017)
0.771 (0.03)
0.558 (0.021)
0.605
AttrMask
0.536 (0.004)
0.645 (0.021)
0.706 (0.018)
0.665 (0.016)
0.638
AutrMask-B
0.534 (0.005)
0.595 (0.010)
0.593 (0.082)
0.672 (0.014)
0.399
MAM-A
0.567 (0.008)
0.594 (0.010)
0.622 (0.057)
0.656 (0.021)
0.610
MAM-A-B
0.545 (0.003)
0.621 (0.012)
0.521 (0.055)
0.657 (0.026)
0.586
MoAMA
0.579 (0.004)
0.598 (0.007)
0.570 (0.024)
0.642 (0.009)
0.597
MotifPred
0.582 (0.006)
0.599 (0.008)
0.621 (0.045)
0.665 (0.006)
0.617
MotifPred-A
0.558 (0.002)
0.589 (0.010)
0.672 (0.013)
0.630 (0.007)
0.612
GraphMAE
0.559 (0.014)
0.608 (0.021)
0.573 (0.078)
0.631 (0.011)
0.593
GraphMAE-R
0.557 (0.005)
0.601 (0.012)
0.567 (0.036)
0.637 (0.006)
0.591
GraphMAE-CK
0.562 (0.005)
0.609 (0.011)
0.626 (0.033)
0.661 (0.009)
0.615
StructMAE-P
0.559 (0.006)
0.596 (0.004)
0.546 (0.037)
0.635 (0.010)
0.584
StructMAE-P-R
0.587 (0.005)
0.605 (0.008)
0.599 (0.072)
0.655 (0.010)
0.611
StructMAE-P-CE
0.563 (0.011)
0.599 (0.006)
0.485 (0.027)
0.651 (0.013)
0.575
StruttMAE-L
0.540 (0.009)
0.598 (0.016)
0.480 (0.033)
0.638 (0.005)
0.561
StructMAE-L-R
0.514 (0.004)
0.610 (0.010)
0.531 (0.093)
0.631 (0.015)
0.580
StructMAE-L-CE
0.541 (0.003)
0.592 (0.011)
0.610 (0.075)
0.645 (0.005)
0.597
AutrMask (T)
0.545 (0.015)
0.588 (0.014)
0.601 (0.044)
0.596 (0.006)
0.583
MotifPred (T)
0.506 (0.009)
0.592 (0.017)
0.615 (0.017)
0.542 (0.012)
0.564
Table 15: Results ADME (RMSE)
27

---

Published in Transactions on Machine Learning Research (11/2025)
A.6 Additional Analysis of Conditional Label Distributions
A.6.1 JSD Results across All Classification Datasets
Figure 11: Jensen-Shamnon Divergence (JSD) between conditional local label distributions P(X = 1,S,) and P(X = S+) for all evaluated classification datasets. The x-axis represents the maximum frequency threshold (r) for including labels the analysis, and the y-axis represents the JSD value.
pesitive negative label
0.5
6.35
Atam
Argrisx Janel
Axyrex Labe
6.30
Nonit (inel
0.4
Veer Labe
Ovegeres
Jrre Shamen Deegerce
0.25
0.3
0.20
0.35
0.30
100000000000
Max Requency Co
Max Frequency Tiresheis CO
(a) BBBP Dataset
(b) MUV Dataset
J50 between positive and negative samples across label types for SIDER dataset
5D between positive and negative samples across label types for TOXCAST dataset
1Q Code
Iac VD Code
2oom lyse
Atom Type
Argman
Arg Labe
Shateen Ciaggnnn
0.25
0.30
0.08
0.05
0.04
5.00
3.2
02
0.4
2.8
08
100
0.0
0.2
0.4
0.0
0.5
1.0
Hrquenc
Presueicy Direshood
(c) Sider Dataset
(d) ToxCast Dataset
positive regative samples across types
neg atvaetiv samp types
Labe
HusT Laiel
Yetr Label
Joner Sterren
0.25
0.10
0.05
0.5
0.00
Max Roquany Tecodli
Prosency Ihcaloudel
(e) Tox21 Dataset
(I) PKIS Dataset
28

---

Published in Transactions on Machine Learning Research (11/2025)
A.6.2 Sanity Check with Shuffled Labels
To test whether the higher MI/JSD for motifs could be explained by the number of classes rather than seman- tic informativeness, we performed a samity check by randomly permuting the motif labels across molecules, while keeping the graph labels Y (positive vs, negative) fixed.
As shown Table 16 and Figure 12 motif MI/JSD collapses to the atom-level baseline. This confirms that the observed effect reflects semantic informativeness rather than mere class-cardinality.
Table 16: MT across prediction targets with shuffled motif MT (5 random seeds)
Dataset
Motif (orig.)
Motif (shuf.)
Atom type
Argmax
VQ
Bace
0.0433
0.0157=0.0004
0.0022
0.0064
0.0130
BBBP
0.0982
0.0250=000009
0.0127
0.0437
0.0575
HIV
0.0162
0.0057+000001
0.0009
0.0040
0.0053
MUV
0.0003
0,0002:1000000
0.0000
0.0001
0.0001
PKIS
0.0583
0.0287=0.0022
0.0005
0.0086
0.0141
Tox21
0.0134
0.00591100005
0.0006
0.0027
0.0063
ToxCast
0.0137
0.0067==000044
0.0017
0.0039
0.0057
Bace Datare:: Binary Graph Labels for Sccretase 1 inhibitor
HIV Datacet: Binary Labols for Andi HIV Activity Screcring Results
Cade (Tus)
(o. vc Gode Orue)
Acom Type (Tua)
Atom Top (Tua)
0.20
Argmax aoel [Fue]
Anprax (hhe)
Laze (hua)
0.22
Mcli (Tro)
Jerann D Diggerss
Lund brullhed
0.15
0.10
0.00
3.5
02
6.4
5.6
98
0.0
0.4
0.5
0.5
Vax Iequuecyy Threchals TrquurccTTTeeehatc
Prstuetry Thindoidi)
Figure 12: JSD curves with shuffled motif labels
A.7 Mutual Information Statistics
By definition, I(X;Y) S (YY). Here we summarize the HYY in the downstream datasets. This indicates the relative informativeness of motif labels is robust despite small absolute MI values.
Table 17: Upper bounds and relative gain of (Xmmotit, Y)
Dataset H(Y)
I(Xmotif; Y)
Relative Gain
Bace
0.999
0.043
4.3%
BBBP
0.918
0.098
10.7%
HIV
0.279
0.016
5.7%
MUV
0.0033
0.0003
9.1%
Tox21
0.210
0.013
5.7%
Toxcast
0.316
0.014
4.3%
Sider
0.999
0.054
5.4%
29

---

Published in Transactions on Machine Learning Research (11/2025)
A.8 More Configuration
Decoder Output Dimensions In our implementation, decoder modules are configured to predict masked node, edge motif labels. For each prediction task, the decoder's output dimension is set to match the number of possible categorical labels in the datasct. The following table summarizes the label dimensions for node-, edge and motif-level reconstruction:
Table 18: Decoder output dimensions used for different level of prediction tasks.
Prediction Target Label Type
Output Dimension
Node altribute
Atom type
119
Edge attribute
Bond type
Motif attribute
Motif label
35,082
A.9 Motif Statistics
Here we list the percentage of overlapping motif in the vocabularies of the pretraining and the down- stream sets used for analysis in Section 5.2
Table 19: Statistics of Motif Vocabularies
Tox21
ToxCast
Sider
MUV
HIV
BBBP
Bace
PKIS
ZINC
Vocab size
1,219
1,284
737
4,721
6,794
738
389
266
35,082
Intersection size
943
964
465
3,914
2,994
515
220
185
Overlap ratio (%)
77.4
75.1
63.1
82.9
44.1
69.8
56.6
69.6
The overlap ratio is computed as the percentage of motifs in each downstream dataset that also appear in This reflects how well the pretrained motif space covers [Vownl
the pretraining vocabulary (i.c., (Vaowa (erettral)) the downstream distributions.
To further understand how many motifs in the downstream datascts have been seen in the pretraining, compute the coverage ratio
r(G) #motifs in G seen in pretraining)
# (motiss in G}
Across all downstream datasets, over 92.9 99.5% molecules satisfy r(G) 2 0.8 (i.e., at least 80% of their motifs were seen during pretraining). Conversely, true cold-start cases are extremely rare: only 0 0.77% of molecules in most datasets (and most 2.5% in one dataset) have r(G) < 0.2).
Table 20: Per-molecule coverage statistics of r(G) across downstream datasets.
Dataset
Median
IQR
Mean
Std r
% r 2 0.8
% S 0.2
sider
1.0000
0.0000
0.9604
0.1648
95.30
2.52
toxcast
1.0000
0.0000
0.9764
0.1056
96.57
0.69
tox21
1.0000
0.0000
0.9791
0.1015
97.02
0.68
hiv
1.0000
0.0000
0.9648
0.1033
92.89
0.20
bbbp
1.0000
0.0000
0.9868
0.0559
97.06
0.00
bace
1.0000
0.1250
0.9542
0.0748
98.74
0.00
muv
1.0000
0.0000
0.9962
0.0284
99.45
0.00
pkis
1.0000
0.0000
0.9562
0.0793
97.66
0.00
30

---

Published in Transactions on Machine Learning Research (11/2025)
A.10 Additional Implementation Details for Masking Distributions
We provide additional details on how heuristic (PageRank-basee)) and learnable masking distributtons were adapted different pretraining settings.
For the hcuristic setting, starting from the original PageRank-based masking in StructMAE-P (sce Al- gorithm 1 implemented analogous variants for AttrMask and MotilPred. In the case of MotifPred, PageRank scores were computed on coarsened version of the molecular graph, where each node represents a motif.
For the learnable setting, recall that StructMAE-L trains its masking scorer by propagating gradients through the encoder's unmasked node embeddings. This strategy, however, is not directly applicable architectures with simple linear decoder that only consumes masked node embeddings. We therefore adapted the learning mechanism in two cases:
AttrMask-L (T): With linear decoder, gradient flow is enabled by attaching the learned scores directly to the masked node embeddings from the encoder.
MotifPred-L (T): A motif-level mask scorer is trained via minimax-style objective: it maximizes the prediction loss while keeping the encoder frozen. To ensure differentiability, Gumbel-softmax- based scores are atlached to the masked motif embeddings before loss computation.
These variants ensure that gradients can be propagated properly under different architectures. They are provided solely for completeness and reproducibility, and are not intended as novel contributions of this work.
A.11 Implementation Details of MAM-VQ
As mentioned in Section 3.3.2 the label generation for MAM-VQ involves nuanced two-stage process centered on its vector quantization (VQ) codebook.
Stage 1: Tokenizer and Codebook Pretraining. The VQ codebook and its corresponding GNN tokenizer are first jointly pretrained using a group VQ strategy. this stage, the codebook is partitioned into four sub-codebooks based on atom type: one each for Carbon, Nitrogen, and Oxygen, and fourth for all other elements. The scarch for the nearest codebook vector is constrained to the relevant partition (as illustrated in Figure 2
Stage 2: Main Encoder Pretraining. However, for the main pretraining of the GNN encoder the stage evaluated in our study a different approach is taken. The pretrained VQ codebook is frozen, and the group constraint is removed. The prediction target for a masked atom is determined by a global search for the nearest vector across the entire codebook. While the original paper does not elaborate on this design choice, it is presumably intended to create more challenging and effective reconstruction task for the main encoder. Our evaluation faithfully implements this two-stage procedure lo ensure fair comparison.
A.12 Detailed Procedures for StructMAE Variants
For completeness, we include the full pseudocode for the two StructMAE variants described in Section 3.3 Both variants rely on a perturbed top-k selection scheme: after computing importance scores for nodes, the top-k candidates are perturbed with random noise, and the final masked is chosen according the adjusted scores. The perturbation strength 3 and annealed mask rate % follow the original settings.
31

---

Published in Transactions on Machine Learning Research (11/2025)
Algorithm 1: Perturbed PageRank-Basce Masking (StructMAE-P)
Input: Graph G = (V, E); mask rate 7; epoch i (max E); perturbation , Output: Mask nodes VMr C
Compute PageRank scores p(e] for all V VEW V;
Anncal effective mask rate: Y/s Y. Vi/E:
Let C; 4 ve (p(u), m))
Sample random noise s(u] ~ 2( 1) for all os Add perturbation s(u] + s(u] + 3 for u E C;;
Select final nodes VM f Tv
return VM
Algorithm 2: Perturbed Learnable Masking (StrucLMAE-L)
Input: Graph G = (V, E); GNN/MLP scorers gnn_ gnn_scr, mlp_ scr; mask rate epoch Output: Mask nodes Vι C V, scores E RI
Compute learned scores [[u] = gnn_ _un_ssr(v) + À inlp_scr(r):
Anneal effective mask rate: 76 < 7 VitB;
Let C, + (
Sample random noise s(v] ~ 24(0, 1) for all i;
Add perturbation s[u] + s[u] + for v E C,;
Select final nodes VM Te (s)
return VM, s
In our implementation, 3 = 0.25 for StruetMAE-P and $ = 0.5, X = 1 for StructMAE-L. The differentiabillty of StructMAE-L relies on passing gradients through the unmasked node embeddings, which are multiplied by the learned scores before being decoded by the GNN.
A.13 Implementation Adaptations for Baselines
Table 21: Adaptations applied to baseline methods under our unified framework.
Method
Adaptations
MAM-A/VE /vQ (MoleBERT)
Remove TMCL loss; created only one masked view per batch
StructMAE-P/L.
Changed default mask rate from 0.5 to 0.25 (aligned with GraphMAB)
MoAMs
Omitted audiliary loss based on Tanimoto similarity
MotifPred (ResCTMask)
Pretrained only on single molecules; adapted to GIN encoder
32


Reviews:
[{"review_id": "3DoCZVNYb9", "reviewer": "Reviewer_gnVz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a unified framework for masked pre-training of self-supervised molecular embedding models in the context of molecular property prediction. The frameworks allows flexibility on a few key dimensions relevant to masked pre-training, and the authors analyse the impact of various instantiations of their framework on a number of downstream molecular property prediction tasks by fine-tuning the pre-trained models. The authors also introduce provide some mathematical analysis on the importance of some of these design decisions, such as methods for selecting how the masking is performed and for selecting what to use as a pre-training prediction target.", "claims_and_evidence_yn": "No", "claims_explanation": "- The setup of the pre-training framework seems mostly sound. They are a number of caveats, however. Firstly, the masking rates of various models used is not the same, and no attempt is made to investigate the importance of the amount of masking on the downstream performance. Without making an attempt to deconvolve the impact of this hyperparameter on the others, it's difficult to draw strong conclusions about the importance of the hyperparameters which are under investigation. Secondly, some modifications are made to the original models to make them fit into the framework, but it's difficult to see how much of an impact these changes make to the performance. For example, for the MotifPred-L (T) model, this involves using Gumbel-softmax to ensure differentiability, but this is known to have the potential of causing stability issues during training and may be detrimental to performance. It's therefore difficult to see whether the conclusions in the paper would also hold for the original models.\n\n- The pre-training dataset used is much smaller than many datasets used previously for pre-training molecular models. This doesn't mean the paper is wrong necessarily, but makes it difficult to see how the claims of the paper would transfer to more commonly used settings. The small model sizes used in this study are a related concern.\n\n- One of the core contributions of the paper is a mathematical analysis of the mutual information between the pre-training target, X, and the fine-tuning targets, Y (molecular properties). The basis of their analysis seems puzzling to me, however. They claim  (second last paragraph of page 5) \"This hypothesis implies that: if a non uniform strategy $\\mathcal{M}$ is indeed more effective, the information provided to $Y$ given $X \\sim P_{\\mathcal{M}} (. | G)$ should be higher than that given $X \\sim P_{\\text{uniform}} (. | G)$\". I don't think this holds however (and since it forms the core of your argument for conducting this analysis it would be very helpful to write this more formally). One can maximise the information in X by masking everything but this is not necessarily a useful pre-training target. Since pre-training and fine-tuning are done in separate phases a better question to ask is how useful are the weights of the model for fine-tuning immediately after pre-training. At the moment I am not convinced the mutual information results presented here are particularly meaningful.", "audience_yn": "Yes", "audience_explanation": "Pre-trained molecular embedding models are a very active area of research and would be of significant interest for various applications beyond molecular property prediction. This paper attempts to create a unified framework for studying how various design decisions within the pre-training setup can influence the downstream performance of pre-trained models for molecules.", "requested_changes": "- As mentioned above, a more thorough and formal explanation of why you have chosen to analyse the mutual information between X and Y.\n\n- The paper also currently does not include any discussion or analysis of models which encode edge information and apply masking to edges (such as [1] and others), which are state of the art for many of these tasks. It's not necessary to include these models within the framework but the paper should at a minimum make clear that techniques such as edge attention or masking is an important part of the discussion that is missing from the current framework.\n\n- Figure 3 should probably include error bars\n\n- In the section comparing GIN with GraphGPS, it would be useful to discuss whether there is a difference in sampling time or compute resource between these architectures. Could the benefit of GraphGPS simply be due to more compute?\n\n- Overall, I also think the introduction, background work, and framework sections could be significantly reduced in size. The first approx. 11 pages of the paper are devoted to outlining the details of the framework that is then used to test hyper parameters of existing model setups. A lot of this is very verbose and doesn't add much to the paper.  I think a lot of this could be condensed and more in depth details moved to the appendix. But I ultimately leave it up to the AE to decide whether this should be an extended length paper or not.\n\n[1] \"An end-to-end attention-based approach for learning on graphs\" https://arxiv.org/abs/2402.10793", "broader_impact_concerns": "NA", "additional_comments": "## Questions\n\n- In figure 5 the mutual information for the Motif task is significantly larger than other tasks. Couldn't this just be due to the larger size of these fragments? I assume the MI could also be influenced by the masking rate which, as mentioned above, is not controlled for in different setups.", "extra_scores": {}, "extra_text": {}}, {"review_id": "mcej990jhp", "reviewer": "Reviewer_rV3K", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides a comprehensive analysis of various masking strategies in self-supervised learning for molecular representation learning. It formalizes the pretraining process in a probabilistic framework and evaluates the impact of different masking distributions, prediction targets, and encoder architectures on downstream molecular property prediction tasks.\n\nPros:\n1. The paper introduces a unified probabilistic framework to systematically analyze and compare different masking strategies. 2. It conducts a thorough evaluation across multiple dimensions (masking distribution, prediction target, encoder architecture) under controlled settings. 3. The findings offer practical guidance for developing more effective SSL methods, particularly highlighting the importance of prediction targets and encoder architecture.\n\nCons:\n1. The study shows that complex masking strategies significantly increase computational costs. Analysis of time and space complexity is necessary to better understand the trade-offs. 2. The hypothesis that non-uniform masking distributions would outperform uniform sampling wasn't supported by the results. A deeper analysis of why these strategies fail to deliver benefits could be insightful. 3. Edge attribute masking did not provide significant performance advantages. Under what conditions or types of tasks might edge masking be more beneficial? 4. The article mainly discusses masking strategy. It is recommended to introduce related methods: MOAT: Graph Prompting for 3D Molecular Graphs; SMI-Editor: Edit-based SMILES Language Model with Fragment-level Supervision. 5. The impact of using a more expressive GNN-based decoder versus a simple MLP decoder was secondary to hyperparameter tuning. Is a more detailed comparison of different decoder architectures needed to clarify their respective roles?", "claims_and_evidence_yn": "Yes", "claims_explanation": "See above.", "audience_yn": "Yes", "audience_explanation": "The paper provides a comprehensive analysis of various masking strategies in self-supervised learning for molecular representation learning. It formalizes the pretraining process in a probabilistic framework and evaluates the impact of different masking distributions, prediction targets, and encoder architectures on downstream molecular property prediction tasks.", "requested_changes": "1. The study shows that complex masking strategies significantly increase computational costs. Analysis of time and space complexity is necessary to better understand the trade-offs. 2. The hypothesis that non-uniform masking distributions would outperform uniform sampling wasn't supported by the results. A deeper analysis of why these strategies fail to deliver benefits could be insightful. 3. Edge attribute masking did not provide significant performance advantages. Under what conditions or types of tasks might edge masking be more beneficial? 4. The article mainly discusses masking strategy. It is recommended to introduce related methods: MOAT: Graph Prompting for 3D Molecular Graphs; SMI-Editor: Edit-based SMILES Language Model with Fragment-level Supervision. 5. The impact of using a more expressive GNN-based decoder versus a simple MLP decoder was secondary to hyperparameter tuning. Is a more detailed comparison of different decoder architectures needed to clarify their respective roles?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "j73Jm169wn", "reviewer": "Reviewer_s6hh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper systematically investigates masking-based self-supervised learning for molecular graphs. It shows that masking distributions bring little benefit over uniform masking, while prediction targets—especially motif-level labels—are far more influential, especially combined with Graph Transformer encoders. Beyond performance, the use of an information-theoretic analysis to interpret core design is particularly impressive.\n\n**Weaknesses**\n\nThe interpretations of the results tend to be somewhat overstated and at times rely on unsupported inferences, which weakens the strength of the conclusions.\n\n- The relative MI comparison suggests motifs are more informative (Table 5), but the absolute MI values are quite small. It is questionable whether such differences are practically meaningful without stronger statistical/other evidence.\n- With about 40% of motifs unseen (64.3% overlap), this is not a negligible gap. In molecular tasks, such cold-start cases are critical and could significantly affect performance.\n- Figure 6 shows higher JSD for motifs, but this may partly result from their larger structural size. The authors should clarify whether the effect reflects true semantic informativeness or simply size.\n- In Figure 7, the performance gains from GIN to GraphGPS are quite modest—1.3 for AttrMask, 1.5 for MoAMa, and 1.6 for MotifPred. These differences are small, yet the paper presents them as “notable improvements,” which seems overstated.\n- While the emphasis is on human-curated motifs, the comparison was at the atom level. Further analysis is needed to determine whether this is due to size or whether it is clearly based on external knowledge.\n\nVarious sections are insufficiently elaborated, with important details either missing or not clearly presented.\n\n- Overall, the categorization/mapping between the core design and various models should be clear, but they are listed in a disorganized manner. A summary or categorization table organizing the models along the key dimensions—masking distribution (uniform, heuristic, learnable), prediction target (node or motif), and encoder architecture (GIN, Transformer)—would make the experimental design much clearer. (GraphMAE, StructMAE-P, StructMAE-L, MotifPred, MotifPred (T), AttrMask (T), AttrMask-L (T), AttrMask-P (T), MotifPred-L (T), SupLearn(?), …)\n- The abbreviations used in the manuscript (e.g., Table 6 Sup, AM, MA, VQ, MP, and (T)) are not clearly explained in the text.\n- Overall, the terminology in the paper is not well unified and feels inconsistent, which makes the presentation confusing. (Table 6: AM, MA, VQ, MP, in Figure 5: Atom, Argmax, VQ, Motif), and the order of factors is also disorganized (e.g., legend order in Figure 5, Figure 6)\n\n\n**Strengths**\n\nThe attempt of an information-theoretic analysis (MI and JSD) to quantify the alignment between pretraining and downstream finetuning is impressive.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This is a well-designed and carefully implemented study that provides valuable insights into the relative importance of masking design choices. However, the paper sometimes overstates its conclusions, and important details are either missing or insufficiently explained. Addressing these issues would improve both clarity and credibility.", "audience_yn": "Yes", "audience_explanation": "The work systematically investigates core design choices in masking-based self-supervised learning for molecular graphs. Some individuals in TMLR’s audience would be interested in this paper", "requested_changes": "The list of proposed adjustments to the submission is critical to enhance the quality of the manuscript and to recommend for acceptance.\nAuthors need to address the weaknesses and ambiguities mentioned above.", "broader_impact_concerns": "This work does not raise direct ethical concerns, as it primarily presents methodological advances in self-supervised learning for molecular graphs.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)