Synergistic Benefits of Joint Molecule Generation and Property Prediction
Authors: Adam Izdebski
Jan Olszewski
Pankhil Gawade
Krzysztof Koras
Serra Korkmaz
Valentin Rauscher
Jakub M. Tomczak
Ewa Szczurek
Venue: Accepted by TMLR ()

Abstract: Modeling the joint distribution of data samples and their properties allows to construct a single model for both data generation and property prediction, with synergistic benefits reaching beyond purely generative or predictive models. However, training joint models presents daunting architectural and optimization challenges. Here, we propose Hyformer, a transformer-based joint model that successfully blends the generative and predictive functionalities, using an alternating attention mechanism and a joint pre-training scheme. We show that Hyformer is simultaneously optimized for molecule generation and property prediction, while exhibiting synergistic benefits in conditional sampling, out-of-distribution property prediction and representation learning. Finally, we demonstrate the benefits of joint learning in a drug design use case of discovering novel antimicrobial peptides.

Published in Transactions on Machine Learning Research (12/2025)
Synergistic Benefits of Joint Molecule Generation and Property Prediction
Adam Izdebski
adam.izdebski Mll.m.tsdebbi@@@hhmhlltteeee
Institute of AT for Health, Helmholtz Zentrum Munchen Technical University of Munich, TUM School of Computation, Information and Technology Facully of Mathematics, Informatics and Mechonics, University of Warsaw
Jan Olszewski
Faculty of Mathematics, Informatics and Mechonics, University of Warsow
Pankhil Gawade
Institute of Al for Health, Helmholtz Zentrum Munchen
Krzysztof Koras
Ardigen SA
Serra Korkmaz
Institute of Al for Health, Helmholtz Zentrum Munchen
Valentin Rauscher
Technicol University of Munich
Jakub M. Tomezak
Lindhoven University of Technology
Ewa Szczurck
Institute of Al for Health, Helmholtz Zentrum Munchen
Faculty of Mathematics, Informatics and Mechonics, University of Warsow
Reviewed on OpenReview: https: // openreview. met/ forum? id= jnaCOLyGOA
Abstract
Modeling the joint distribution of data samples and their properties allows to construct single model for both data generation and property prediction, with synergistic benefits reaching beyond purely generative or predictive models. However, training joint models presents daunting architectural and optimization challenges. Here, we propose HYFORMER, transformer- based joint model that successfully blends the generative and predictive func- tionalities, using an alternating attention mechanism and a joint pre-training scheme. We show that HYFORMER simultancously optimized for molecule generation and property prediction, while exhibiting synergistic benefits in conditional sampling, out-of-distribution property prediction and representation learning. Finally, demonstrate the benefits of joint learning in drug design use case of discovering novel antimicrobial peptides.
Coda available http::///thtttcccoo///c/aaaaaa//

---

Published in Transactions on Machine Learning Research (12/2025)
1 Introduction
Developing models that simultaneously excel in both generative and predictive tasks is long-standing challenge in machine learning (Bishop, 1994; Jaakkola & Haussler, 1998; Lasserre et al., 2006). Joint models, which unify these tasks, offer synergistic benefits, including improved control over the generative process of the model, improved predictive robustness towards unseen, c.g,, newly generated or out-o--dist (OOD) data, and learning representations predictive of high-level molecular features (Nalisnick et al., 2019; Grathwohl et al., 2020; Cao & Zhang, 2022: Tomezak, 2022). These benefits are crucial for applications such as drug design, where success depends on balancing the generation of novel molccules from unexplored regions of the chemical space coupled with robust property prediction extrapolating towards the newly generated molecules (Grisoni, 2023; Steshin, 2023; van Tilborg et al., 2025).
However, molecule generation and property prediction are predominantly approached in separation. This division persists even though transformer-basse models are state-of-the-an across both tasks (Bagal et al., 2022; Gao et al., 2024b; Trwin et al., 2022; Xia et al., 2023; Zhou et al., 2023). A likely reason is that joint training poses daunting challenges, as combining a generative and a predictive part into a single model may over-renularize both parts (Lasserre et al., 2006) or cause gradient interference between the generative and predictive objectives (Nalisnick al., 2019). As a result, molecular models continue to forgo the potential benefits of joint learning. This raises natural question, whether one can develop transformer- -baseed joint model optimized for both generalive and prediclive performance, al the same lime offering the syneryislic benefits of joint learning?
To address this challenge, we introduce HYFORMER, a joint model that combines an autoregrossive transformer decoder with a bidirectional transformer encoder in single model with shared parameters. Upon training, we alternate between using the model as a decoder and as an encoder, with either a causal or bidirectional self-attention mechanism, alleviating problems typical for joint models. We evaluate the generative and predictive performance, well as synergistic benefits of joint learning using HYFORMER across a variety of molecular tasks (Wu et al., 2018; Brown et al., 2019; Steshin, 2023; Chen et al., 2023). Our contributions are:
1. We propose a novel joint model, HYFORMER, that unifies the generative and the predictive task in a single set of parameters.
2. We demonstrate the synergistic benefits of joint modeling, where HYFORMER outperforms basclines on (i) conditional molecule generation, (ii) out-of-distribution property prediction and (iii) molecular representation learning via probing.
3. We show that HYFORMER rivals the generative and predictive performance of state-of-the-art purely generative and predictive models.
4. We showcase the applicability of joint modeling in a real-world drug design use case of discovering novel antimicrobial peptides.
2 Related Work
Existing generative approaches can be categorized into sequence- and graph-based models. Sequence-based methods represent molecules as SMILES (Weininger, 1988) or SELFIES (Kremn et al., 2020) and process tokenized strings using recurrent transformer-based language models (Segler et al., 2018; Flam-Shepherd et al., 2022; Bagal et al., 2022). In contrast, mraph-based, models treat molecules as graphs and have been implemented using variational autocncoders (Liu et al., 2018; Jin et al., 2019; Maziarz et al., 2022; Hetzel et al., 2023), normalizing flows (Luo et al., 2021), energy-based models (Liu et al., 2021a), and graph transformers (Gao et al., 2024b). More recently, 3D-based generative models have been proposed to capture the spatial geometry of molocules (Hoogeboom et al., 2022; Guan et al., 2023; Gao et al., 2024a), however real world drug discovery pipelines continue to rely predominantly on 2D 2D-molecular representations (Xiang et al., 2024).
Molecule Generation
2

---

Published in Transactions on Machine Learning Research (12/2025)
Molecular Property Prediction Analogously, prediction models leverage distinct molecular representa- tions. Methods based pre-trained language models predominantly work with SMILES (Wang et al., 2019; Fabian et al., 2020; Irwin et al., 2022; Sultan et al., 2024), while other approaches represent molecules as graphs (Li et al., 2021; Wang et al., 2022). Recent methods leverage the three-dimensional spatial structure of a molecule, either using graph neural networks (Fang et al., 2022) or transformers (Zhou el. al., 2023). Finally, Yang et al. (2019); Fabian et al. (2020); Stokes al. (2020) incorporate pre-computed physicochemical descriptors of molecules into training.
Joint Models for Molecules
Early joint models combine variational autoencoders with latent-space predictors (Gómez-Bombarelli et al., 2018; Maziarz et al., 2022). Regression Transformer (Born & Manica, 2023) frames property prediction as conditional sequence generation, but. lacks unconditional generative capability. Graph2Seq (Gao et al., 2024b) is a graph-based encoder-decoder transformer, trained separately as generative or as a prodictive model, but evaluated on both molccule gencration and property prodiction. UniGEM (Feng et. al., 2024) is a diflusion-based model for unified generation and prediction, however specializing in 3D molecular modeling and not directly applicable to standard SMILES-based benchmarks.
Therefore, the question of whether the transformer architecture can be used to implement joint model for both SMILES-based generation and prediction, while enjoying synergistic benefits, remains open.
3 Background
Problem Formulation The aim of joint modeling is to learn the joint distribution of the data and its properties p(x, v), i.e., to identify model that at the same time generates new data and predicts its properties. We assume access to a labeled dalasel D [(xm: Dn ) n=1. sampled from the joint data distribution p(x, y), often accompanied with an unlabeled dataset Du = {x, J= sampled from p(x). Here, examples x can be thought of as molecules and labels y as molecular propertics.
In the general formulation Lasserre et al. (2006), joint modeling aims to learn the joint distribution p(x, v) by defining a joint model p y) that factorizes into a generative model pa(x) and a predictive model (y x) such that
Pa.s (x. 3) Pa(y x))o(xx),
(1)
where 0 denotes the parameters of the generative model, and o the parameters of the predictive model. Training of the joint model is equivalent to minimizing the negative log- lig-likelhhood, i.c., the joint loss
(0) (Ln Aln x)],
(2)
where x E IK weights the predictive and the generative parts.
Choosing the extent lo which parameters 0 and d are shared and the way the joint loss is optimized, is crucial for obtaining model with both a high generative and predictive performance, at the same time maintaining the synorgistic benefits of joint learning (Lassorre et al., 2006).
3.1 Transformer-based Models
Transformers (Vaswani et al., 2017) achieve state-of-the-art performance in both molecule generation (Bagal et al., 2022) and property prediction (Zhou et al., 2023) tasks.
Transformer Encoders and Decoders Transformers used for generation and for property prediction differ in the use of the self-allention mechanism. Transformer decoders, used for generative tasks, employ a causal aelf-attention
QK"" Vd
Att_, QQ V) softmax
M Vv,
3
where K,K,
K. V E Rxd are query, key and value matrices, respectively, M., E TxT is a cousal mask i..., a matrix such that (M,)ig 0, if i > j, and (M,)ú -00, otherwise, T' is the sequence length and is the
3

---

Published in Transactions on Machine Learning Research (12/2025)
head dimension." On the other hand, transformer encoders, used for predictive tasks, employ bidirectional self-allention
QQK"" + M V
AlL Q2 V) = softmax
(4)
vd
where M., E RTXT is bidirectional mask, L.., (M.))i = 0 for all i,j E [T].
Alternating attention The definition of the transformer decoder and encoder can be generalized by using an alternating attention scheme (Dong et al., 2019):
QK vd
ATT Type Q, V) = softmax
+ MATI_Type V
55)
where NTT_Type {->, ++) and Marr. Type M., is causal mask upon using the model as transformer decoder and MATI_Type- Me,, otherwise.
Training transformers Training transformers proceeds in two-step manner, by first pre- pre-training the model on an unlabeled dataset and then fine-tuning the pre-trained model on a downstream task. Transformer decoders and encoders are pre-trained using different losses.
Pre-training Transformer decoders, optimized for generative performance, are predominantly pre-trained using the negalive log-likelihode loss -Ex~p(x)] [In pa (x)]- As the causal mask induces a factorization of the transformer docoder into an autoregressive model po(x) I po( xxt), where x ;......;r).... the generalive loss reduces to the language modeling (LM) loss
Σ po / xx2))
(0)) = -Ex-p(x)
(6)
On the other hand, transformer encoders are usually pre- trainede using masked language modeling (MLM) loss xx)]
(M)M(()) EEx In po
where (I1, (31,-.-,æT)) TT), M is a set of indices drawn uniformly at random from the set of token indices [T. and the set of all tokens whose indices belongs to M are masked tokens XM. The rest of the tokens XR defined such that x XM U XR.
Fine-tuning Next, the pretrained model is fine-tuned by defining predictive head on top of the pretrained model and training it as a predictor on labeled dataset using the prediction loss
EYRED (Ø) -E
= -Exxx)) [In x)).
(8)
4 Hyformer
We propose HYFORMER, a joint transformer-based model that unifies generative decoder with predictive encoder in a single set of shared parameters, using an alternating training scheme.
4.1 Model Formulation
HYFORMER unifies decoder with encoder using transformer backbone fo(x; [TASK]) conditioned task token [TASK] E [LM)] (PRED), [MLM] The task token facilitates switching between respective losses during
We that the dimensions the query, key, and value matrices are equal.
4

---

Published in Transactions on Machine Learning Research (12/2025)
transformer block
x
Prediction head
Alternating Attention
Imput Embedding
Proparty
Layer Norm
Lay Norm
Mi P
Linear
Softmax
probabiittes
(1N6K)
A schematic representation of HYFORMER. Depending on the task token [TASK], HYFORMER uses either a causal or a bidirectional mask, outputting token probabilities or predicted property values.
Figure 1:
training (see Section 4.2) and determines whether the backbone Se processes input in an autoregressive manner using causal, or a bidirectional mask
if [TASK] = [LM],
ATT _yp_
if [TASK] E [[PRED], [MLM]
Finally, the generative pe (x) and predictive py x) parts of the joint model, factorized
pe (x. y) (o( x),
(9)
are implemented by adding a generative and a predictive head on the top of the shared backbone fo-
Algorithm Training of HYFORMER
Input: Dataset D (labeled or unlabeled); model parameters 0; task probabilities Р For pre-training: [TASK] E { {[LM) [PRED], [MLM], for fine-tuning: [TASK] [[LM) [PRED] }.
1: while stopping criterion not met do
2: Sample task [TASK] ~ CAT(P(TA Select loss [TASK] and the corresponding attention mask Update model parameters 0 using the gradient of [(TASK) end while
4.2 Hyformer Training
As with standard transformer- bascod models, the training of HYFORMER is divided into a pro-training and a fine-tuning stage.
Joint Pre-training To unify the generative and the predictive functionalities in a single model, we pre-train HYFORMER using a variant the joint loss (Eq. 2). For the generative part, we use the language modeling loss (LM, while for the predictive part, we use the masked language modeling loss EMLM and the predictive loss EyERD, with the combined loss being defined as:
HYFORRMER = LLM + WLMLM + N/PRED-
(10)
As pre-training lobels, we use values analytically computable from the input sequences, c.g., molecular descriptors, such as molecular weight for small molecules, or hydrophobicity for peptides. When the pre- training labels are not available, HYFORMER pre-trained without the predictive loss ÉRRED Analogously to multitask learning (Raffel ct al., 2023), the weighted loss HHvоoRuu (Eq. 10) effectively implemented using vector of task probabilities [[TASK] (P -PHLL) P(Pne))) which defines how the generative and predictive capabilities of the joint model are balanced.
5

---

Published in Transactions on Machine Learning Research (12/2025)
During training, the shared parameters 0 updated differently depending on the task token. If [TASK] E [[RRDD], [NLM] ), a bidirectional attention mask M., is applied and all attention module weights are updated, since the bidirectional mask does not restrict information flow. Conversely, if [TASK] = [LM), a causal mask M., is applied, restricting each token to attend only to its left context, altering the gradients of the attention module, due to the functional form of the Jacobian of the softmax function, alleviating gradient interference typical for joint modeling (Appendix E.1).
Fine-tuning
We fine-tune HYFORMER using the joint loss (Eq. 2), defined as
HYVORRRR LLM + AepuE
(11)
Analogously to pre-training, HYFORMER alternates between the generative and predictive task, to balance their objectives, based on pre-defined vector of task probabilities P = (Puw) (PouO,P((Raa)) P[PRED)) We assume that fine-tuning labels used in loss Eyar are different than in the pre-training phase and are defined by the downstream prediction task. Specifically, we omit the masked language modeling loss, to focus on the downstream task while retaining the generative capabilities of the model.
4.3 Sampling
Sampling from HYFORMER exploits the generative po(x) and predictive part po(y x) depending the sampling mode: unconditional or conditional.
Unconditional Generation In unconditional generation, sample x p=(x) using the autoregressive part of the model. This addresses limitation of conditionally trained generative models (Bagal et al., 2022) and joint models trained without a pure unsupervised objective (Born & Manica, 2023), where generating a single example requires conditioning on a fixed property value inferred from a dataset.
Conditional Generation To generate (x,y) ~ po(x, y) that satisfies a condition Y S y, HYFORMER samples K-many examples ; .K ~ (x() and, for every = l,..., K, accepts sample Xk, if the predictor poly x) classifies Xę as having property Y. As simple consequence of the Bayes rule, the above procedure yields correct conditional sampling procedure,
p(x y E (Y p x)p(x),
(12)
for E Y S Y such that p(y E Y) > O. Note that the conditional sampling procedure of HYFORMER is a variant of best-of-K sampling, a provably near-optimal solution to the KT-regularized RL problem (Yang et al., 2019) that has been shown to outperform other conditional sampling methods for LLMs, including state-of-the-an reinforcement learning methods like PPO and DPO (Touvron al., 2023; Mudgal al., 2023; Gao et al., 2023; Rafailov et al., 2023). Crucially, HYFORMER leverages a jointly trained predictor poly I) over unified representation space, resulting in tighter alignment between generation and control. This coherence is particularly valuable in drug discovery, where the primary objective is not throughput, but precision and sample efficiency, that is, generating a small mumber of high-quality candidates with minimal false positives.
5 Experiments
We evaluate HYFORMER across broad range of molecular modeling tasks. First, we demonstrate the synergistic benefits of joint modeling in three settings: (i) conditional generation GuacaMol dataset (Brown et al., 2019), (ii) out-of-distribution (OOD) property prediction on Hit Identification task from the Lo-Hi benchmark (Steshin, 2023) and (iii) representation learning via probing on MoleculeNet benchmark (Wu et al., 2018). Subsequently, we show that HYFORMER rivals state-of in both unconditional generation on GuacaMol and property prediction on MolcculeNet. Finally, we apply HYFORMER antimicrobial peptide (AMP) design, showcasing the benefits of our joint modeling approach. Experimental details and additional results are provided in Appendix H, I and J.
generative and predictive models

---

Published in Transactions on Machine Learning Research (12/2025)
Table I: Conditional generative performance on GuacaMol dataset. Best model is marked bold.
MODEL
JoiN
METRIC
QED
SA
LOGP
AvG.
MAD +
0.087
0.019
0.276
0.127
MoLGPT
SD 4
0.074
0.017
0.262
0.118
VALIDITY
0.985
0.986
0.982
0.981
MAD SD 4
0.039
0.011
0.158
0.069
GRAPHGPT
X
0.082
0.047
0.653
0.261
VALDITY f
0.998
0.997
0.992
0.995
MAD +
0.031 (0.003)
0.015 (0.001)
0.131 (0.010)
0.059 (0.004)
SD
0.045 (0. 004
0.020 (0.001)
0.170 (0:014)
0.078 (0.006)
VALIDITY t
0.993 (0. (0.003)
0.990 (0.001)
0.985 (0.014)
0.989 (0.007)
HYFORMER
MAD 4
0.008 (0.001)
0.005 (0.000)
0.041 (0.002)
0.018 (0.001)
SD
0.015 (0.002)
0.009 (0.002)
0.051 (0.004)
0.025 (0.003)
VALIDTY
0.990 (0.007)
0.985 (0.008)
0.987 (0.006)
0.987 (0.005)
5.1 Synergistic Benefits of Hyformer
5.1.1 Conditional Molecule Generation
To demonstrate the synergistic benefits of HYFORMER in generating molecules with specific molecular properties, we follow the setup of Bagal et al. (2022) and jointly pre-train HYFORMER scaled to 8,5M parameters on GuacaMol dataset with 1.3M molccules, using pro-computed molccular descriptors (Yang et al., 2019). We subsequently jointly fine-tune HYFORMER on GuacaMol dataset with QED, SA, and LogP molecular properties, as fine-tuning labels, and generate molecules with specifie properties using HYFORMER'S conditional sampling procedure. Pre-training and experimental details alongside results for all property settings can be found in Appendix H and I.1.
Following (Gao et al., 2024b), we compare HYFORMER to MolGPT (Bagal et al., 2022) and GraphGPT (Gao et al., 2024b) using: mean absolute deviation (MAD) from the target property value, standard deviation (SD) of the generated property values and validity of the generated molecules. Evaluation is averaged across three target values per each property: QFD::005, 0.7, 0.9), SA:(0.7, 0.8, 0.9), and logP:(0.0, 2.0, 4.0). Additionally, compare to a non-joint variant of HYFORMER, in which the predictive head is finc-tuned with prediction loss, on top of frozen, pre-trained generative part, i.e., without joint fine-tuning.
The jointly fine-tuned HYFORMER achieves the lowest MAD and SD across all properties, while maintaining high validity, outperforming all basclines. Notably, HYFORMER improves controllability over it's non- joint counterpart, confirming that joint fine-tuning enhances conditional generation. Although GraphGPT attains slightly higher validity, it does so at the cost of reduced controllability. These results demonstrate that joint modeling enables robust property-com conditoc-o molecular generation across range of chemically relevant targets.
5.1.2 Out-of-Disribution Molecular Property Prediction
To evaluate the ability of HYFORMER to predict molecular properties in an out-of-distribution (OOD) setting, we jointly pre-train HYFORMER scaled to 50M parameters on 19M molecules from (Zhou et al., 2023), together with pre-computed molecular descriptors (Yang et al., 2019), and benchmark on the Hit Identification (Hi) task from the Lo-Hi benchmark (Steshin, 2023). The Hi task requires generalization to molecular scaffolds not seen during training, with the test set constructed such that no molecule has a Tanimoto similarity greater than 0.4 (based on ECFP4 fingerprints) to any molecule in the training set. This setup mimics realistic drug discovery scenarios, where generalization beyond known chemical space is essential. For experimental details, see Appendix H and 1.2.
We follow the sctup of (Steshin, 2023) and compare jointly fine-tuned HYFORMER to all models reported in (Steshin, 2023); machine learning models: k-NN, gradient boosting (GB), SVM and MLP, trained on molecular fingerprints (ECFP4, MACCS) and deep learning models: Chemformer (Yang et al., 2019) and Graphformer (Ying et al., 2021; Shi al., 2022). Morcover, we compare to HYFORMER (no-joint), which is a version of our model pre-trained using MLM loss, hence without alternating attention, and fino-tuned using the prediction loss only.
7

---

Published in Transactions on Machine Learning Research (12/2025)
Table Predictive performance (AUPRC) on Hit Identification (Hi) task from Lo-Hi benchmark. Mean and standard deviation across random seeds.
DATASET, AUPRC
DRD2-HI
HIV-Hi
KDI-HH
Sot-HI
DUSMY BASELINE
0.67T (0.061)
0.040 (0.014)
0.609 (0.081)
0.215 (0.008)
KNN (ECFP4)
0.706 (0.047)
0.067 (0.029)
0.646 (0.048)
0.426 (0.022)
KNN (MACCS)
0.702 (0.042)
0.072 (0.036)
0.610 (0.072)
0.422 (0.009)
GB (ECFP4)
(0.050)
(0.038)
(0.067)
(0.006)
GB (MACCS)
0.751 (0.063)
0.058 (0.030)
0.603 (0.074)
0.502 (0.045)
SVM (ECFP4)
0.677 (0.061)
0.040 (0.014)
0.611 (0.081)
0.298 (0.047)
SVM (MACCS)
0.713 (0.050)
0.012 (0.015)
0.603 (0.082)
0.308 (0.021)
MLP (ECFP4)
0.717 (0.063)
0.049 (0.019)
0.626 (0.047)
0.403 (0.017)
MLP (MACCS)
0.896 (0.048)
0.052 (0.018)
0.613 (0:077)
0.462 (0.048)
CHEMPROP
(0.062)
(0.114)
(0.026)
(0.030)
GRAPHORMER
0.729 (0.039)
0.096 (0.070)
HYFORMER (NO-JOINT)
0.778 (0.070)
0.154 (0.108)
0.673 (0.046)
0.601 (0.040)
HYFORMER
0.784 (0.052)
0.158 (0.128)
0.701 (0.022)
0.640 (0.036)
HYFORMER achieves the highest mcan AUPRC across all datasots (Table 2), outperforming fingerprint-baood baselines and demonstrating the potential of deep learning methods real-world drug discovery. The consistent ranking in favor of HYFORMER shows the benefits of joint modeling in out-of-distribution molecular property prediction, although the differences are not statistically significant at the 95% confidence level.
5.1.3 Molecular Representation Learning
Table Molccular representation learning performance of predictive, generative and joint models on MoleculeNet benchmark, evaluated using linear and KNN probing. Best model within each probing method
is marked bold.
DATASET,
RMSE
DATASRT,
AUCROC
TYPE MODEL
Esot.
FREISOrV
LIPO
BBBP
BACE
CLINTOX
Tox21
ToxCAST
SIDER
ITV
P.
UNI-MOL
1.350
2.503
65.5
66.3
74.3
58.1
73.6
P.
HYFDRMER (N0-JOIVT)
1.256
2.840
0.894
68.4
73.6
98.8
73.4
61.2
58.8
75.9
G.
MOLGITT
1.299
4.110
1.033
66.8
79.1
97.8
71.9
60.5
59.2
77.5
J.
MOLER
1.223
4.935
0.938
67.8
79.5
84.6
71.1
59.3
58.3
74.6
J.
2.510
4.515
1.158
54.7
63.1
57.3
50.5
52.8
54.5
65.6
J.
CRAPH2SPQ
1.498
3.486
0.800
66.0
76.7
72.0
71.2
60.4
50.5
57.1
J.
HLFORMER
1.527
4.294
0.887
68.5
77.2
99.5
72.4
60.7
60.8
74.7
P.
UNI-MOL
1.579
3.403
1.025
60.0
75.9
78.0
64.7
37.5
61.0
61.3
(NO-JOrVT)
1.360
0.978
67.8
Y5.4
89.0
71.4
KNN
G.
MOLGPT
1.232
3.075
0.987
68.4
71.9
94.2
66.0
56.9
61.0
70.5
MoLeR
1.802
1.096
59.4
72.0
71.2
57.3
67.3
J.
RT
2.411
4.734
1.242
59.3
56.1
59.4
50.8
52.2
51.2
54.1
GRAPU2SEQ
49.9
52.4
J. HYFORMER
1.260
3.999
0.902
69.5
78.4
93.8
71.2
59.3
64.1
71.8
To assess the quality of molecular representations learned by HYFORMER, introduce novel probing protocol that emulates a typical drug discovery setting, where fixed molecular embeddings are used inputs to downstream predictive models. In this setup, we train simple linear models with L2 regularization, and k-nearest neighbor (KNN) predictors on the top of frozen embeddings extracted from the respective pre-trained models. To ensure comparability with MoleculeNet benchmark (Section 5.2.2), we reuse the same datasets, data splits, and model checkpoints. Implementation details are provided Appendix T.3.
We compare representations extracted from jointly pre-trained HYFORMER to those extracted from range of baselines, including saate-of-the-ane generative (MolGPT (Bagal et al,, 2022)), predictive (Uni-Mol (Zhou et al., 2023)), and joint models: MoLeR (Maziarz et al., 2022), Regression Transformer (RT) (Born & Manica, 2023) and Graph2Seq (Gao et al., 2024b). Moreover, to quantify the effect of alternating attention and joint pre-training, we compare to HyFORMER (no-joint),, the version of our model trained solely with MLM loss.
The jointly pre-trained representations from HYFORMER are the most predictive across both KNN and lincar probings, achieving the best performance on 4 out of 10 datasets for linear, and 5 out of 10 datasets for KNN, putfrrrrming all other baselines (Table 3). An additional analysis of linear probing Clin Tox shows
8

---

Published in Transactions on Machine Learning Research (12/2025)
Table 4: Unconditional generative performance on GuacaMol distribution learning benchmarks. The best. model in each category marked bold.
MODEL
FCD ScoRE
KL DIV. SCORE 1
VAL.
UNIQ.
Nov.
GRAPH-BASED
JT-VAE
0.750
0,940
1.000
MoLER
0.625
0.964
1.000
1.000
0.991
MAGNET
0.780
1.000
MICAM
0.731
0.989
1.000
0.994
0.986
SMILES-BASED
VAE
0.863
0.982
0.870
0.999
0.974
LSTM
0.959
MOLGPT
0,907
0,992
0,981
0.998
1.000
IYFORMER,=O.
0.897 (0.002)
0.995 (0.000)
0.986
(0.001)
0.999
(0.000)
0.879 (0.006)
HYFORMER,
0.918 (0.002)
0.959 (0.001)
0.978 (0.000)
0.999 (0.000)
0.908 (0.002)
HYFORMER
0.894 (0.002)
0.977 (0.001)
0.965 (0.001)
1.000 (0.000)
0.931 (0.001)
high per-tanget F1 scores of 0.98 and 0.90, indicating robust performance of HYFORMER across targets. The next best models, Hyformer (no-joint) and MoLeR for linear and MolGPT for KNN probing, rank first 2 and 3 out of 10 datasets, respectively. Notably, joint models outperform UniMol, the state-of-tho-an property predictor, on all datasets, except for Freesolv with linear probing, highlighting the effectiveness of joint modeling for transferable molecular representation learning.
5.2 Generative and predictive performance of Hyformer
We next confirm that HYFORMER effectively addresses the challenges of joint training, while it enjoys the synergistic benefits described above, it does not sacrifice generative prodictive performance compared to state-of-the-art models trained separately for these tasks.
5.2.1 Unconditional Molecule Generation
To evaluate the unconditional generative performance HYFORMER, we perform an evaluation on the Guacamol distribution learning benchmark (Brown al., 2019). We use HYFORMER scaled to 8.5M parameters and trained on GuacaMol dataset with 1.3M molecules, together with pre-computed molecular descriptors (Yang al., 2019), and investigate the impact of sampling temperature For experimental details, see Appendix 1.4.
We compare to state-of-the-art unconditional generative models; SMILES-based: VAE (Kingma & Welling, 2013), LSTM (Gers & Schmidhuber, 2001), MolGPT (Bagal et al., 2022) and graph-based: JT-VAE (Jin et al., 2019), MoLeR (Maziarz et al., 2022), MAGNet (Hetzel et al., 2023), MiCaM (Geng et al., 2023). We omit RT (Born & Manica, 2023) and GraphGPT (Gao al., 2024b) as they do not generate molecules unconditionally or provide results on the GuacaMol benchmark.
HYFORMER, with top FCD and KL div. score values, outperforms graph-based models, while achieving the highest validity among SMILES-basee models. Across varions sampling temperatures т, HYFORMER consistently lies on the Pareto front, balancing distributional fidelity (FCD Score, KL div. Score), validity and uniquencss. Overall, SMILES- models outperform those based on thcoretically more informative graph atties terms of FCD Score, at the expense of not al always sampling valid molecules.
5.2.2 Molecular Property Prediction
To evaluate the predictive performance of HYFORMER, molecules from (Zhou et al., 2023), together with pre-computed molecular descriptors (Yang et al., 2019), and fine-tune end-to-end on MoleculeNet benchmark (Wu et al., 2018). For experimental details, see Appendix L5..
HYFORMER scaled to 50M parameters 19M
We follow the experimental protocol (Zhou et al., 2023), use scaffold splitting and compare to predictive modcls: D-MPNN (Yang al., 2019), AttentiveFP (Xiong et al., 2019), N-gram (Liu et al., 2019) with Random Forest and XGBoost (Chen Guestrin, 2016), PretrainGNN (Hu et al., 2019), GROVER (Rong et al., 2020), MolCLR (Wang et al., 2022), Mole-BERT (Xia et al., 2023), GraphMVP (Liu et al., 2021b),
9

---

Published in Transactions on Machine Learning Research (12/2025)
Table 5: Predictive performance of predictive and joint models on the MoleculeNet benchmark. Mean and standard deviation across 3 random sceds. The best model in cach category, statistically significant at the 95% confidence level, is marked bold.
Dansser, RMSE 1
DanASRr; ACCROC 1
MODEL
Foor
FREssIOo
LIpO
BBRP
BACE
CLNNTON
Tox21
TOxCAST
SIDER
ITV
D-MPNN
1.050(0.008)
2,082/0.00 082)
0.883(0-010)
71. 11.000.33
80.0/0.6)
00.6;0.0)
75.8(0.7)
65. (5.5(0.3)
57.0:0.73
77. (00.5
ALTENNIIVE FP
0.877(00129)
2.073/0.183)
0.721(0.001)
64.3(1.8)
78.4(0.02)
84.7/0.3)
76.1(0.5)
63.7(0.2)
00.6/3.2)
78.4(1.4)
N GGSAaaRF
1.074/0.107
2.688/0.085)
0.812(0.008)
69.7(0.6)
77.9 1.5)
71.3/4.0)
14.300.)
06.8/0.77 7)
77. 27.1
N-GRAMIXGH
1.085(0.82))
(.66100.744
2.072(0.088)
69.1 69.1(0.8)
70.1(1.8)
87.5/2.71
T6.8/0.9)
65.5/0.7) 7)
7..4
Pamartte
PHATRAINGNY
1.100/0.000
2.764/0.000 002)
0.739/0-003)
68.7(1
$1.5/0 7)
72.6:155
78.1(0.6)
65-7(0.6)
62.7(0.8) 5)
79 79.9(0.7]
GROVERSASE
0.952(0.050)
2.176(0.0552
0.817(0.008)
70.000.1)
82.6(0.7)
81.2:3.0)
T4.3(0.1)
65.4(114)
64.8(0.6)
02 02.5(0.9)
GROVERLANE
(.017)
0.011)
0.823:0.0100 ..00
69.5(0.1
81.0(1.4)
76.2:5.7)
73.5(0.1)
65.3(0.5)
05.4(0.1)
68. (1.1
GRAPMMVP
1022(00053)
0.681(0-010)
72.4(1.6)
81.2(0.9)
10.1:2.8)
75.0(0.5)
63.1(0.4)
63.0.1.25 ()
77.011.21
MoRCLH
1.271(0.048)
2.694(0.249)
(..01)
72.2(2.3)
82.4(0.9)
01.2
75.0/0.2)
58.9(1.4)
78.1
MOu BORTT
(0.030)
(1.6)
(3.0)
(0.5)
(0.2)
GEM
0.796(0.029)
1.87/00009 0944)
0.060/0
72.400.4)
15.0(1.1)
90. 10.1:1.00
78.1(0.1)
09.2(0.4)
67.200.43
50 (0.6(0.9
UNI-Mot
0.785(0.129)
1.480(0.048)
00003().010)
72.9(0.5)
$5.7(0.2)
91.9(3.8)
70.6(0.6)
60.6(0.1)
65.0(1.3)
s0.8(0.8)
Jonr
GHAPH2Seq
0.860(0.024)
1,797/0 231)
0.716(0.010)
72.8(1.5)
$3.8(1.0)
76.0(0.3)
65.4(0.5)
68 20.000.00 9)
79 (0.9(3
HYFORMER
0.774(0.026)
2.047(0.076)
0.643(0.002)
75.900.9)
$3.8(1.33
99.2(0.5)
79.2(0.1)
63.0(1.6)
03.7(1.6)
80.0(1.0)
GEM (Fang et al., 2022), UniMol (Zhou et al., 2023) and a joint model: Graph2Seq (Gao et al., 2024b). We omit RT (Born & Manica, 2023) and other models that use random splitting.
HYFORMER obtains the lowest RMSE on Esol, and highest AUROC on BBBP and Tox, outperforming all models on 3 out of 10 datasets (Table 5). Moreover, HYFORMER performs better than Graph2Seq, the only other joint model capable of simultaneous molecule generation and property prediction, on 8 out of 10 datasets, Altogether, HYFORMER outperforms the other joint learning model, Graph2Seq, and successfully rivals the performance of purcly predictive models, demonstrating the efficiency of our joint lcarning strategy.
Table 6: Conditional generative performance on antimicrobial peptide design. Mean and standard deviation computed over 100 bootstrap iterations. The best model is marked bold.
MODEL
PERPLEXITY
DIVERSITY
FriNesS t
HyuRAMPxae
AMPLIEY f
AMPEPPy
PERCVAE
20.11 (0.14)
0.87 (0. (0.0003)
0.07 (0.0004)
0.20 (0.0016)
0.49 (0.0016)
0.52 (0.0007)
AMPGAN
18.58 (0.10)
0.81 (0.0005)
0.12 (005)
0.32 (0.0019)
0.64 (0.0018)
0.54 (0.0008)
HYDRAMP
20.14 (0.12)
0.86 (0. (0.0004)
0.09 (0.0003)
0.49 (0.0021)
0.59 (0.0016)
0.52 (0.0006)
AMP-DiePUSION
16.93 (0.18)
0.82 (0. (0.0004)
0.13 (0. (0.0006)
0.26 (0.0018)
0.20 (0.0014)
0.38 (0.0006)
HYFORMER
17.98 (0.06)
0.80 (0.0005)
0.19 (0.0006)
0.80 (0.0019)
0.94 (0:0027)
0.72 (0.0018)
5.3 Antimicrobial Peptide Design
To show the benefits of joint learning in a real-world use case related to drug discovery, we apply HYFORMER to the task of antimicrobial peptide (AMP) design (Chen et al., 2023), i.c., generating AMPs with low minimal inhibitory concentration values (MTC) against F. coli bacteria. We jointly pre-train HYFORMER on 3.5M general-purpont peptide sequences, and subsequently on 1M AMP sequences, together with 39 physicochemical descriptors from peptidy package (Özçelik et al., 2025). Next, we we jointly fino-tune HYFORMER 4,547 peptides with their MIC values (Szymezak et al., 2023) and conditionally sample 50K peptides with MIC regressor threshold set to < 100.3 ~ 2 p... For experimental details, see Appendix T.6.
We compare HYFORMER AMP generation baselines: PepCVAE (Das et al., 2018), AMPGAN (Van Oort et al., 2021), HydrAMP (Szymezak et al., 2023), and AMP Diffusifusin (Chen et al., 2023). Evaluation is based on four criteria: Perplexity (Torres et al., 2025), Diversity and Fitncss (Li et al., 2024), and success rates in generating AMPs and low-MIC candidates. For the latter, we use HydrAMP Amplify (Li et al., 2022), and amPEPpy (Lawrence et al., 2020) classifiers as of in-silico oracles.
HYFORMER outperforms all baseline models by large margin in terms of generating peptides with a high fitness and AMP probability, as evaluated by all oracle classifiers (Table 6). Despite the stringent conditioning MIC threshold of 2 pM, HYFORMER maintains competitive perplexity and high diversity. These results suggest that even when constrained to explore less charted regions of sequence space, HYFORMER is able to generate biologically plausible and novel peptide candidates.
We report perplexity, but do not seek to minimize it, as inherently balances plausibility and novelty.
10

---

Published in Transactions on Machine Learning Research (12/2025)
1:20
03
20-
0.6
When Alantan
0.2
Nylmm
MenuNO
APP
Figure 2: (a) Amino-acid distributions between the pre-training data and unconditionally generated sequences. (b) Distributions of charge, aromaticity, and isoclectric point (pI) for: non-AMP, AMP and conditionally generaled sequences. (c) Frequency of crossing an atlention threshold (x-axis) vs. mean atlention weight. (y-axis) for distinct amino-acids, colored by charge and sized by hydrophobicity.
To further validate the biological relevance of the generated peptides, show that both unconditional sampling from jointly pre-trained HYFORMER, and conditional sampling from the fine-tuned model produces amino-acid distributions in close agreement. with the training data (Figure 2a). Despite this very close agreement, the conditionally sampled peptides obtain a significant improvement of charge, aromaticity, and isoelectric point over the known non-AMPss as compared to known AMPs (Fig. 2b). Finally, to gain insight into which amino acids contribute most to antimicrobial activity, we analyze the attention weights of HYFORMER (Fig. 2c). The attention mechanism frequently prioritizes highly charged Arginine (R) and Lysine (K), which is expected as high AMP activity is associated with increased charge. The high attention frequency on Tryptophan (W) agrees with previous reports about this amino-acid's ' unique ability to interact with the interface of the bacterial membrane (Bi et al., 2014). Finally, the high attention that HYFORMER puts on Proline (P) agrecs with the known high potency of Proline-rich AMPs, which kill bacteria via a specific, non-lytic mechanism (Lai et al., 2019).
6 Discussion
In this paper, we introduced HYFORMER, a transformer- based joint model that combines an autoregressive decoder and a bidirectional encoder within a single set of shared parameters, using an alternating attention mechanism and joint pre-training. We showed that HYFORMER provides synergistic benefits in conditional sampling, representation learning and out-of-distribution property prediction, with ablations highlighting the specific contributions of alternating attention and joint training. Furthermore, we validated the utility of joint modeling in a real-world antimicrobial peptide design task. Our results indicate that HYFORMER successfully unifies molecular generation ion and property prediction for SMILES-based molecular representations, opening the avenue for the integration into real-world drug discovery pipelines, where informative molecular representations, robustness to OOD examples and robust conditional sampling are crucial.
Limitations & Future Work However, joint modeling introduces an inherent trade-off. While shared parameters promote synergistic benefits and learning unified representations, they may limit task-specific specialization. Therefore, promising direction for future work is designing dynamic modular attention architectures that allocate capacity across tasks more flexibly, while preserving synergistic benefits. Moreover, to ensure fair comparison with prior work and isolate the effect of joint learning, we deliberately restricted model scale and relied on a fixed set of analytically computed descriptors. The exlent to which the observed synergistic benefits carry over to other modalities, such as 3D structures, morphology or transcriptomics, remains an open question.
11

---

Published in Transactions on Machine Learning Research (12/2025)
References
Josep Arús-Pous, Simon Viet Johansson, Oleksii Prykhodko, Esben Jannik Bjerrum, Christian Tyrchan, Jean-Louis Reymond, Hongming Chen, and Ola Engkvist. Randomized smiles strings improve the quality of molecular generative models. Journal of cheminformatics, 11:1-13, 2019.
Viraj Bagal, Rishal Aggarwal, P. K. Vinod, and U. Deva Priyakumar. MolGPT: Molecular Generation Using Transformer- Deedddr Model. Journal of Chemical Information and Modeling, 62(9):2064 2076, May 2022. ISSN 1549-9596. doi: /.002//acs.jiiii wtm.je.m....000......
Xiaonan Bi, Che Wang, Weibing Dong, Wei Zhu, and Dejing Shang, Antimicrobial properties and interaction of two trp-substituted cationic antimicrobial peptides with a lipid bilayer. The Journal of Anlibiolics, 67 (5):361 368, 2014.
G Richard Bickerton, Gaia V Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L. Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90 98, 2012.
Christopher M Bishop. Novelty detection and neural network validation. IEE Proceedings- Vision Image and Signal processing, 141(4):217 222, 1994.
Esben Jannik Bjerrum. Smiles enumeration data augmentation for neural network modeling of molecules, 2017.
Jannis Born and Matteo Manica. Regression transformer enables concurrent sequence regression and generation for molecular language modelling. Nature Machine Intelligence, 5(4):432 444, 2023.
N. Brown, M. Fiscato, M. H. S. Segler, and A. C. Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of Chemical Information and Modeling, 59(3):1096-1108, 2019.
Gabriel Moras-Mora, Anamaría Daza, Nicole Soto-Gareía, Valentina Garrido, Diego Alvarez, Marcelo Navarrete, Lindybeth Sarmiento- Varón, Julieta H. Sepúlveda Yañez, Mehdi D. Davari, Frederic Cadet, Álvaro Olivera-Nappa, Roberto Uribe-Paredes and David Medina-Ortiz. Peptipedia v2.0: A peptide sequence database and user-friendly web platform. major update. bioRair, 2024. doi: 10.1101/2024.07.11 603053. URL https:: /www. biorxiv. org/ /ottte///eerll/2224/////////0000 07. 11. 603053.
Senqi Cao and Zhongfei Zhang. Deep hybrid models for out-of-distribution detection. In Proceedings of the IREE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1733-4743, 2022.
Tianlai Chen, Pranay Vure, Rishab Pulugurta, and Pranam Chatterjee. -ifffusfusoo:: Integrating latent diffusion with protein language models for antimicrobial peptide generation. In NewrIPS 2023 Generative Al and Biology (GenBio) Workshop, 2023.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
The UniProt. Consortium. Uniprot: the universal protein knowledgebase in 2025. Nucleic Acids Research, 53(DL))D609-DD)) D617, 11 2024. ISSN 1362-4962. doi: 10.1093/natt http///a//ggal00000 URL https://doo.oorg/10. 1093/nar/gkae1010.
Payel Das, Kahini Wadhawan, Oscar Chang, Tom Sercu, Cicero Dos Santos, Matthow Riemer, Vijil Chen- thamarakshan, Inkit Padhi, and Aleksandra Mojsilovic. Pepovae: Semi-supervised largeted design of antimicrobial peptide sequences, 2018
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-training for natural language understanding and generation. Advances in neural information processing systems, 32, 2019.
P. Ertl and A. Schuffenhaner. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. Journal of Cheminformatiis, 1(1):8, 2009.
12

---

Published in Transactions on Machine Learning Research (12/2025)
Benedek Fabian, Thomas Edlich, Héléna Gaspar, Marwin Segler, Joshua Meyers, Marco Fiscato, and Mohamed Ahmed. Molecular representation learning with language models and domain-relevant auxiliary tasks. arXiv preprint 0rXic:2011.132300 2020.
C. Fang et al. Prospective validation of machine learning algorithms for absorption, distribution, metabolism, and excretion prediction: An industrial perspective. Journal of Chemical Information and Modeling, 2023.
Xiaomin Fang, Lihang Liu, Jieqiong Lei, Donglong He, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. Geomelry-enhanceed molecular represenlation learning for properly prediction. Nature Machine Intelligence, 4(2)) 134, 2022.
Shikun Feng, Yuyan Ni, Yan Lu, Zhi-Ming Ma, Wei-Ying Ma, and Yanyan Lan. Unigem: A unified approach to generation and property prediction for molecules. arXiv preprint arXic:240. 10516, 2024.
Daniel Flam-Shepherd, Kevin Zhu, and Alán Aspur--Guzik. Language models can learn complex molecular distributions. Nature Communications, 13(1):3293, 2022.
Bowen Gao, Minsi Ren, Yuyan Ni, Yanwen Huang, Bo Qiang, Zhi-Ming Ma, Wei- Ying Ma, and Yanyan Lan. Rethinking specificity in sbdd: Leveraging delta score and energy-guided diffusion. or orXiv preprinl 0rXiv:2403.19989 2024a.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835 10866. PMLR, 2023.
Zhangyang Gao, Daize Dong, Cheng Tan, Jun Xia, Bozhen Hu, and Stan Z. Li. A graph is worth k words: Enclideanizing graph using pure transformer. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 14681-14701. PMLR, 21-27 Jul 2024b.
Zijie Geng, Shufang Xie, Yingoe Xia, Lijun Wu, Tao Qin, Jie Wang, Yongdong Zhang, Feng Wu, and Tie Yan Liu. De novo molecular generation via comnection-aware motif mining, 2023.
F.A. Gers and E. Schmidhuber. Lstm recurrent networks learn simple context-free and context-sensstive languages. IEEE Transactions on Neural Networks, 12(6))1333 1340, 2001. doi: 10.1109/72.963769.
Rafael GómezzBombarell,, Jennifer N. Wei, David Duvenaud, José Miguel Hernández-LLoato, Benjamín Sänchez-Lengelling Dennis Sheberla, Jorge Aguilera-Iparraguirre Timothy D. Hirzel, Ryan P. Adams, and Alán Aspuru-Guzik. Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules. ACS Central Science, 4(2):268 276, February 2018. ISSN 2374-7943. doi: ww.1021/acccenttee 7b00572.
Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky, Your classifier is secretly an energy based model and you should treat it like one, 2020.
Francesca Grisoni. Chemical language models for de novo drug design: Challenges and opportunities. Curvent Opinion in Structural Biology, 79:102527, 2023.
Jiaqi Guan, Wesley Wei Qian, Xingang Peng, Yuleng Su, Jian Peng, and Jianzhu Ma. 3d equivariant diff'usion for target-aware molecule generation and affinity prediction. arXiv preprint orXiv:2303.095// 2023.
Leon Hetzel, Johanna Sommer, Bastian Rieck, Fabian Theis, and Stephan Günnemann. Magnet: Motif-
agnostic generation of molecules from shapes, 2023.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling, Equivariant diffusion for molecule generation in 3d. In International conference on machine learning, pp. 8867 8887. PMLR, 2022.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. Xiu preprint ar arXio:1905. 12265, 2019.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Ross Irwin, Spyridon Dimitriadis, Jiazhen He, and Esben Jannik Bjerrum. Chemformer: a pre-trained transformer for computational chemistry. Machine Learning: Science and Technology, 3(1)011022, 2022.
Tommi Jaakkola and David Haussler. Exploiting generative models in discriminative classifiers. Advances in neural information processing systems, 11, 1998.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation, 2019.
J. Kim et al. Comprehensive survey of recent drug discovery using deep learning. International Journal of Molecular Sciences, 22(18):9983, 2021.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
Mario Krenn, Florian Häse, AkshaiKunis Nigam, Pascal Friederich, and Alan Aspuru-Gusik. Self-referencing embedded strings (selfies): A 100 Machine Learning: Science and Technology, 144)044502 October 2020. ISSN 2632-2153. doi: 10.1088/ 00.1088/2632- 20122113//// aba947.
Pin-Kuang Lai, Daniel T Tresnak, and Benjamin J Hackel. Identification and elucidation of proline-rich antimicrobial peptides with enhanced potency and delivery. Biotechnology and bioengineering, 116(10): 2439-2450, 2019.
Julia Lasserre, Christopher M Bishop, and Thomas P Minka. Principled hybrids of generative and discriminative models. In 2006 TEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'0G) 006), volume pp. 87 94. IEEE, 2006.
Travis J Lawrence, Dana L Carper, Margaret K Spangler, Alyssa A Carrell, Tomás A Rush, Stephen J Minter, David J Weston, and Jessy Labbé. ampeppy 1.0: a portable and accurate antimicrobial peptide prediction tool. Bioinformatics, 37(14):2058 2060, 11 2020. ISSN 1367-4803. doi: 10.1093 bioinformatics btaa917. URL https://dol..rg/ http://....0.aaaaaoorraaaaaaaaaa
Chenkai Li, Darey Sutherland, S Austin Hammond, Chen Yang, Figali Taho, Lauren Bergman, Simon Houston, René L Warren, Titus Wong, Linda M N Hoang, Caroline E Cameron, Caren C Helbing, and Inanc Birol. AMPlify: attentive deep learning model for discovery of novel antimicrobial peptides effective against WHO priority pathogens. BMC Genomics, 23(1):77, Jamuary 2022.
Pengyong Li, Jun Wang, Yixuan Qiao, Hao Chen, Yihuan Yu,, Xiaojun Yao, Peng Gao, Guotong Xie, and Sen Song. An effective self-supervised framework for learning expressive molecular global representations to drug discovery. Briesings in Bioinformaticcs, bbbbbll9,,,,9, 2021.
Tingting Li, Xuanbai Ren, Xiaoli Luo, Zhuole Wang, Zhenlu Li, Xiaoyan Luo, Jun Shen, Yun Li, Dan Yuan, Ruth Nussinov, Xiangxiang Zeng, Junfeng Shi, and Feixiong Cheng. A foundation model identifies broad-spectrum antimicrobial peptides against ttrug-resistan bacterial infection. Net. Commun., 15(1): 7538, August 2024.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hic, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shimeei, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Fvolutionary-scale prediction of atomic-level protein structure with language model. Science, (796))3)111 1130, 2023. doi: 10.1126 (0.1126/science.ade225 ade244...e.........
Meng Liu, Keqiang Yan, Bora Oztekin, and Shuiwang Ji. Graphebm: Molecular graph generation with cnsrgy-based models. arXie preprint orXiv:2102, 6, 2021a.
Qi Liu, Miltiadis Allamanis, Mare Brockschmidt, and Alexander Gaunt. Constrained graph variational autoencoders for molecule design. Advances in neural information processing systems, 31, 2018.
Shengehao Liu, Mchmet F Demirel, and Yingyu Liang. N-gram graph: Simple unsupervised representation for graphs, with applications to molecules. Advances in neural information processing systems, 32, 2019.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre-training molecular graph representation with 3d geometry. arXiv preprint arXim:210000722,, 2021b.
Youzhi Luo, Keqiang Yan, and Shuiwang Ji. Graphdf: A discrete flow model for molecular graph generation. In International conference on machine learning, pp. 7192-7203. PMLR, 2021.
Krzysztof Maziarz, Henry Jackson- Pashmina Cameron, Finton Sirockin, Nadine Schneider, Nikolaus Stiefl, Marwin Segler, and Mare Brockschmidt. Learning to extend molecular scaffolds with structural motifs, 2022.
David Mendez, Anna Gaulton, Patrícia Bento, Jon Chambers, Marlcen De Voij, Eloy Félix, María Paula Magariños, Juan F Mosquera, Prudence Mulowo, Michal Nowolka, et al. Chembl: towards direct deposition of bioassay data. Nucleic acids research, (7(DD):D930-DD) -940, 2019.
Sidharth Mudgal, Jong Lec, Harish Ganapathy, YaGuang Li, Tao Wang. Yanping Huang, Zhifong Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, al. Controlled decoding from language models. arXiv preprint Xiv:22000. 17022, 2023.
Eric Nalismick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Takshminarayanan. Hybrid models with deep and invertible features. In International Conference on Machine Learning, pp. 4723 4732. PMLR, 2019.
Riza Özgelik, Laura van Weesep, Sarah de Ruiter, and Francesca Grisoni. peptidy: A light-weight python library for peptide representation in machine learning, 2025.
Kaare Brandt Petersen, Michael Syskind Pedersen, et al. The matrix cookbook. Technical University of Denmark, 7(15):510, 2008.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is sccretly a reward model. Advances in Newral Information Processing Systems, 36:53728 53741, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer, 2023.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self- supervised graph transformer on large-scale molecular data. Advonces in neural information processing systemss 33:12559 12571, 2020.
Célio Dias Santos-Jinior, Yiqian Duan, Hui Chong, Thomas S.B. Schmidt, Anthony Fullam, Peer Bork, Xing-Ming Zhao, and Luis Pedro Coelho. Ampsphere the worldwide survey of prokaryotic antimicrobial peptides, May 2022. URL https://doi /ttp:o//doo.orr/1000// 10. 5281/zenodo 6511404.
Philippe Schwaller, Daniel Probst, Alain C. Vaucher, Vishnu H Nair, David Kroutter, Teodoro Laino, and Jean-Louis Reymond. Mapping the space of chemical reactions using allention-based neural networks. ChemRzie, 2020. doi: 10.26434/ 00.26434/chemrxxv httpc/hhwemw.v./eeeeeme 99w.9969755....
Marwin HS Segler, Thierry Kogej, Christian Tyrchan, and Mark Waller. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS central science, 4(1):120 131, 2018.
Yu Shi, Shuxin Zheng, Guolin Ke, Yifei Shen, Jiacheng You, Jiyan He, Shengjie Luo, Chang Liu, Di He, and Tic-Yan Liu. Benchmarking graphormer on large-scale molecular modeling datasets. Xix preprint Xrxiv:2003.0 200.2200304400, 2022.
Simon Steshin. Lo-hi: Practical ml drug discovery benchmark. In Advances in Neural Information Processing Systems, 2023.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Jonathan M Stokes, Kevin Yang, Kyle Swanson, Wengong Jin, Andres Rubillos-Ruiz, Nina M Donghia, Craig R MacNair, Shawn French, Lindsey A Carfrae, Zohar Bloom-Ackermann, et al. A deep learning approach to antibiotic discovery. Cell, 180(4):688 702, 2020.
A Sultan, J Sieg, M Mathea, and A Volkamer. Transformers for molecular property prediction: Lessons learned from the past five years. arxiv preprint arxiv: 240403969. 2024.
Paulina Saymezak, Marcin Možejko, Tomasz Grzegorzek, Radostaw Jurczak, Marta Bauer, Damian Neubauer, Karol Sikora, Michal Michalski, Jacek Sroka, Piotr Setny, Wojciech Kamysz, and Ewa Szczurek. Discovering highly potent antimicrobial peptides with deep generative model hydramp. bioRrio, 2023. doi: 10.1101/
2022.01.77.478052
Jakub M Tomezak. Deep generative modeling for neural compression. In Deep Generative Modeling. Springer, 2022.
Marcelo D. T. Torres, Tianlai Chen, Fangping Wan, Pranam Chatterjee, and Cesar de la Fuente-Nunez, Generative latent diffusion language modcling yiclds anti-infective synthotic peptidos Riorrie, 2025. doi: 10.1101/202.001 : URL https: http::::///wwwbwooiiioo// biorxiv. http:///tteeeeeea/////////////// 636003.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Sounya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. XiXie preprint 2xXw:2337.00228. 2023.
Colin M. Van Oort, Jonathon B. Ferrell, Jacob M. Remington, Safwan Wshah, and Jianing Li. Ampgan v2: Machine seermiing-giieed design of antimicrobial peptides. Journal of Chemical Information and Modeling, 61 (1(5)22198-2207, 2021. doi: 10.1021 /ttp//cc..ciiii icc.j.im.00004411.
PMID: 33787250.
Derck van Tilborg, Luke Rossen, and Francosca Grisoni. Molecular deep learning at the edge of chemical space. 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Mllia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Sheng Wang, Yuzhi Guo, Yuhong Wang, Hongmao Sun, and Junzhou Huang. Smiles-bert. large scale unsupervised pre-training for molecular property prediction. In Proceedings of the 10th ACM international conference on bioinformatics, computational biology and health informatics, pp. 429 436, 2019.
Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3):279 287, 2022.
David Weininger. SMILES, chemical language and information system. 1. Introduction to methodology and encoding rules. Journal of Chemicol Information and Computer Sciences, 28(1):31-36, February 1988. ISSN 0095-2338. doi: 10.1021 1000///000577000
S. A. Wildman and G. M. Crippen. Prediction of physicochemical parameters by atomic contributions. Journal of Chemical Information and Computer Sciences, (55588888-873, 1999.
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Ancesh S. Pappu, Karl Leswing, and Vijay Pande. Moleculenet: A benchmark for molecular machine learning, 2018.
Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z. Li. Mole-BERT: Rethinking pre-training graph neural networks for molecules. In The Eleventh International Conference on Learning Representations, 2023.
Yu- Ti-Ting Xiang, Guang-Y- Yi Huang, Xing-Xing Shi, Ge-Fei Hao, and Guang-Fu Fu Yang. 3d molecular generation models expand chemical space exploration in drug design. Drug Discovery Today, pp. 104282, 2024.
16

---

Published in Transactions on Machine Learning Research (12/2025)
Zhaoping Xiong, Dingyan Wang, Xiaohong Liu, Feisheng Zhong, Xiaozhe Wan, Xutong Li, Zhaojun Li, Xiaomin Luo, Kaixian Chen, Hualiang Jiang, et al. Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. Journal of medicinal chemistry, 63(16))8749 8760, 2019.
Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Cuzman-Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, el al. Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling, 59(8):3370-3388 2019.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Yan Liu. Do transformers really perform badly for graph representation? Advances in neural information processing systems, 34:28877 28888, 2021.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. In The Eleventh International Conference on Learning Representations, 2023.
17

---

Published in Transactions on Machine Learning Research (12/2025)
A Impact Statement
The goal of this this work is to improve the field of deep generative modeling and, potentially, drug design. An example of potential malicious use of our approach would be training a deep generative model for generating new toxic molecules. However, the intention of this paper is to provide tools that will facilitate designing new potential medications.
Acknowledgments
We thank Hassan Akell for insightful discussions and careful review the theoretical part of this paper. Their leedback substantially improved the clarity, rigor, and presentation the theoretical analysis.
C Extended Discussion
Extended Novelty Statement The alternating self-attention scheme closely related to prior multitask transformer work (c.g., Dong et al. (2019)). However, to the best of our knowledge, HYFORMER is the first model to employ an alternating attention scheme during both pre-training and fine-tuning, resulting in a joint model that unifies molecular generation and property prediction. Tn contrast, Dong al. (2019) apply alternating attention only during pre-training. Moreover, HYFORMER explicitly combines reconstruction- based losses: LM and MLM, with prediction loss (Eq. 10), while Dong et al. (2019) rely exclusively on reconstruction-based losses, without incorporating any supervision based on labeled data. Together, these architectural and objective-level differences enable joint generative and predictive modcling, and distinguish HYFORMER from prior alleenattig-attention--be altention-basee models.
Extended Future Work interesting direction for future work to cast molccular property prodic- tion as purely generative task. While such a formulation could further unify generation and prediction within single modeling paradigm, it introduces nontrivial challenges, most notably the principled tokeniza- tion and representation of continuous molecular properties. Moreover, the proposed framework naturally extends to modeling molecular interactions. Since HYFORMER natively supports multimodal inputs, small- polecule-protein
molecule protein interactions can be incorporated by conditioning the decoder on protein embeddings and augmenting training with additional pretraining objectives. Exploring such extensions to jointly model multiple molccular modalitics represents promising avenue for future rescarch.
D Notation
Symbol
Meaning
[N]
Set of integers 1,...,N N
A
Malrix
AT
Transposed matrix A Matrix indexed for some purpose
A; Agj, A
(A), A A
The i-th row of matrix A The i-th, j-th entry of matrix A Vector (column-vector)
(A)i [.. An
at aij, a
Vector indexed for some purpose
(a),, ali),
The i-th entry of vector Scalar
X
input space, iee. the space of all possible inputs, data examples target space iee the space of all possible property values joint data distribution
y
p(x, y)
Po (x. y)
joint model paramotrized by parameters f
po(y x)
predictive model parametrized by parameters 0 E θ generative model parametrized by parameters 0 θ
po (x)
18

---

Published in Transactions on Machine Learning Research (12/2025)
E Proofs
E.1 Gradient Interference
Lemma E.1. Let x E R' and define
0; softmax(x);
expxy
for i 1,..., I.
Σ - exp
The Jacobian of the softmar is given by
Da,
a: dig - ag),
i,j I....,I
0x;
where dig is the Kronecker delta, i.e., dig = if j j and 0 otherwise.
Proof. Differentiate the quotient a; exp2: 2: Ex exp z4 using the product and chain rules (Petersen al., 2008).
Corollary E.2. Let Q,K Txd and the attention score matriz S., with causal mask M., be defined
QK S. Vd + M., where (M,)i²
ffizs
00 i.
For a fized row inder t E [T), define the attention score row-vector s, (S), E R and the corresponding row-wise softmax oulgrut as a, = softmax(s:)) E R²² The Jucobian of the soltmax outpul as with respect to masked attention score St is given by
0(ar): (aa)) (at)g). 0(st);
Hence, i < or j < t, while i + j, then a(8.),
8(ac).
O.
Proof. Lemma E.1 gives the derivative of the softmax. As the causal mask sets (st) = -o for every j < t, the corresponding probabilities satisfy (at); = 0.
F Benchmark Task Definitions
F.1 Conditional Molecule Generation
Quantitative Estimate of Drug-likeness (QED). A continuous metric of the drug-likeness of a molecule basod on physicochemical propertics such as molocular weight and hydrophobicity, with values ranging from to 1. (Bickerton al., 2012)
Synthetic Accessibility (SA). A continuous metric quantifying how difficult molecule is to synthesize, derived from structural complexity, where lower values indicate casier synthesis. (Ertl & Schuffenhauer, 2009)
Partition Coefficient (logP). A contimmous metric of molecular hydrophobicity, defined as the logarithm of the partition coefficient between octanol and water, where higher values denote greater affinity for lipophilic environments. (Wildman & Crippen, 1999)
Metric values calculated using rdkit 2023.09.2.
19

---

Published in Transactions on Machine Learning Research (12/2025)
F.2 Out-of- of-Distribution Molecular Property Prediction
DRD2-Hi. Binary classification dataset of 8482 compounds with labels indicating dopamine receptor inhibition, with therapeutic relevance in schizophrenia and Parkinson's 's disease; dataset obtained from ChEMBL30. (Mendez et al., 2019)
HIV-Hi. Binary classification dataset of 41127 compounds from the Drug Therapeutics Program AIDS Antiviral Screen, with labels indicating the inhibition of HIV replication; dataset obtained from MoleculeNet. (Wu et al., 2018)
KDR-Hi. Binary classification dataset with labels indicating VEGFR2 (vascular endothelial growth factor receptor 2) inhibition, kinase target in cancer therapy, with training restricted to 500 compounds to simulate low-data regimes; dataset obtained from Chembl30. (Mendez et al., 2019)
Sol-Hi. Binary classification dataset 2173 compounds with labels indicating solubility; dataset obtained at Biogen. (Fang et al., 2023)
For further dataset and train/test splitting details, see (Steshin, 2023). Data accessed from https://github. StheshhnSS//ohh_rruurpp003333331 neurips2023/tree http://reemmmaaaaaaaaaaaaaaaaaaa [accessed 20.03.2023)
Sem//ttehhnnS//oh...
F.3 Molecular Representation Learning and Property Prediction
ESOL, Regression datasct containing water solubility measurements for 1128 compounds.
FreeSolv. Regression dataset containing experimentally measured hydration free energy values in water for 642 compounds.
Lipophilicity. Regression dataset containing experimentally measured octanol/water distribution coefficients (logD at pH 7.4), key indicator of membrane permeability and solubility, for 4,200 compounds.
BACE. Binary classification dataset of 1513 compounds with experimentally determined qualitative binding results for a set of inhibitors of human 8-secretase (BACE-1).
BBBP. Binary classification datasot of 2039 compounds with binary labels indicating blood-brain barrier
permcability.
Tox. Multitask classification dataset of 1478 compounds with labels indicating whether compound is (i) FDA-approved and/or (ii) failed clinical trials due to toxicity reasons.
HIV. Binary classification dataset of 41127 compounds from the Drug Therapeutics Program AIDS Antiviral Screen, measuring inhibition of HIV replication.
Tox21. Multitask classification dataset of 7831 compounds with qualitative toxicity measurements across 12 biological targets, including nuclear receptors and stress response pathways.
ToxCast. Multitask classification dataset of 8575 compounds with qualitative toxicity results across over 600 in vitro assays, derived from high-throughhut screening.
SIDER. Multitask classification dataset of 1427 approved drugs, with side effects grouped into 27 system organ classes according to ModDRA classifications, capturing adverse drug reactions across organ systems.
For further details, see Table 1 in Wu al. (2018). To ensure comparability with Uni-Mol (Zhou al., 2023), we accessed data from data/finetune/noleclll orr.leeullapppppptyyrrroocoooooo tar gz [accessed 20.03.2023).
tos- cn---n--iie beijjing.nn...e...e....e..e.ii.t volces. com/unimol_ beijinn..vocce.
G Benchmark Metric Definitions
MAD. Mean Absolute Deviation between predicted and target property values; lower better.
SD. Standard Deviation of generated property values from the target; lower better.
Validity. Fraction of syntactically valid molecules generated by the model; higher is better.
20

---

Published in Transactions on Machine Learning Research (12/2025)
Uniqueness. Fraction of unique molecules among generated samples; higher is better.
Novelty. Fraction of generated molecules not present in the training set; higher is better.
KL Div. Score. Score based on the Kullback-Leibler Divergence between various descriptor distributions of generated and training molecules; values normalized the range [0, 1]; higher values indicate a closer match between descriptor distributions between generated and training molecules. (Brown et al., 2019)
FCD Score. Score based on the Fréchet ChemNet Distance between the generated and reference (training) molecule embedding distributions, calculated in ChemNet feature space; values normalized in the range [0, 4]; higher values indicate closer resemblance of the generated to reference molccules. (Brown et al., 2019)
Perplexity. Exponentiated negative log-likelihodd of a sequence, with the log-likelihodd being calculated per token, using BoGGen2-meding (Torres et al., 2025); lower values indicate greater model-based plausibility of the generated peptides.
Diversity. Average pairwise Levenshtein distance between the generated sequences; higher values indicate greater diversity of the generated samples. For details, see Eq. 6 in Kim et al. (2021), where Hyformer replaces Socrgel with Levenshtein distance.
Fitness. A measure quantifying to what extent a peptide forms a stable, amphipathic a-helix, computed according. (Li et al., 2024)
HydrAMP MIC. The probability of a peptide being active against E.Col, bacteria strain predicted with HydrAMP. (Szymezak et al., 2023)
AMPlify. The probability of peptide being antimicrobial predicted with AMPlify. (Li et al., 2022)
amPEPy. The probability of a peptide being antimicrobial predicted with amPEPy. (Lawrence et al., 2020)
Pre-training Details
We implement HYFORMER using a LLAMA backbone (Touvron et al., 2023). Depending on the size of the pretraining dataset, we scale HYFORMER to 8. 7.7M paramelers for GuacaMol and 50M parameters for the UniMol and peptide datasets 5. These configurations align model capacity with dataset size and ensure a fair comparison with prior work: the 8.7M model is comparable to MolGPT (Bagal et al., 2022), while the 50M variant matches the scale of Uni-Mol (Zhou et al., 2023) and Graph2Seq (Gao al., 2024b). For GuacaMol, we apply 2x data augmentation using non-canonical SMILES enumeration (Bjerrum, 2017: Arús- Pous et al., 2019) to increase molecular diversity. All models pretrained using pre-computed molecular descriptors (Yang et al., 2019). The balancing of the lasks (prwn P P[Paen] is set to (0.90, 0.05, 0.05) and (0.80, 0.10, 0.10), respectively.
We use SMILES (Weininger, 1988) or amino acid sequences as molccular representations across all experiments. For tokenization, we adopt an exlended character-level tokenizer for SMILES, based on Schwaller el al. (2020), and use the ESM-2 tokenizer (Lin et al., 2023) for peptides.
We pro-trrin HYFORMER using batch size of 1024 for up to 50K or 250K iterations, depending model size. Training is performed with the AdamW optimizer (3) = 0.9, ,3 = 0.95, = x 10-5, weight decay = 1 x 10- using a peak learning rate of 6 x 10-4 with cosine decay and 5000 warm-up steps. We use gradient clipping with a maximum norm of 1.0. All input sequences are padded to a fixed length of 128 tokens. Training is conducted using bftoat16 precision on a single NVIDIA H100 S0GB HBM3 GPU.
DDta accessed from https: cota/pprooectt///accaaa////////e
20.03.2025.
4Data accessed from http:////boooobemmee-eeeeeeeeeee
tos coa-cn-toe -n- -nw-ecc--eeeeeeeeeee beijing. volces. com/uninol haw//atooo..tt///ttttuuu/llllllu
property.prediction. tar gz 20.03.2023
DData accesse from attps://app
http:////apppppppeia..poo peptipedia.cll
https: //wwv www.mmtttttgouiiiipaa... hutpppoo.oo.uuupppottttrr////s/u
big-data-bbologg .rgg ddommladdd
and
https://drive. gggglee.cmm/rrive//foleeee
17.04.2025 with train/tes constructed using standard scikit's train/test.
splitting and random seed 44.
21

---

Published in Transactions on Machine Learning Research (12/2025)
Table 7: Architectural details of HYFORMER.
NUM. PARAM.
EMBED. DIM:
HIDDEN
# LAVERS
ArT. HEADS
8.7M
256 512
1024
50M
2018
12
Experimental Details
All fine-tuning and inference is conducted using float32 precision single NVIDIA V100 32GB GPU.
1.1 Conditional Molecule Generation
We jointly fine-tune HYFORMER, pretrained on GuacaMol dataset, for 10 epochs with a batch size of 256. The peak learning rate selected from the set {1e-4, 2e- 3e-4, 4c-4, 5c-4, 5e-4, 6e-4), based on root mean squared error (RMSE) with respect to the target property. During fine-tuning, we set the task probability vector to (Piz0oPPPPED)) P(PReD) = (0.5, 0.5) and do not perform hyperparameter search over this setting, as it yields satisfactory performance by default. For the non-joint variant of HYFORMER, freeze the pretrained model and fine-tune only the prediction head. This avoids catastrophic forgetting of the generative capability when removing the generative loss during training. For each target property value, we sample 100K unique molecules, with a wall-clock time of 78 + 1 seconds, and retain those passing manually defined threshold, using multinomial top-k sampling with 7 = 0.9 and 10. Note that reported SA scores are normalized, following (Gao et al., 2024b).
To further characterize the selectivity of conditional sampling and the calibration of the predictive heads, we report acceptance rates in the conditional molecule generation experiment in Table 9. The results confirm that conditional sampling with HYFORMER is highly selective across all target values.
Table 8: Conditional generative performance on GuacaMol dataset across all targets. Best model marked bold.
ProemAx Jonn
Menoe
QKD.07
Qun...
SA..02
S4..0.5
5A 9
soctt...00
Avt:
Gorsartrtto Saware
0.365
Vauanery
0.983
0.081
0.975
0.987
0.883
0.954
MAU 4
0.341
6400
2307
0.02
00225
0.001
0.303
0.189
0.201
X
SD 1
0.079
6.677
1.221
0.055
0.02
0.070
0.400
0.636
0.855
0.230
VALIorYY
0.383
0,080
0.992
MAD
0.082
0.051
0.000
0.017
0,190
0.050
SD
0.200
0,073
5100
01042
110022
0.463
0.703
0.796
0.251
Valanrry
0.906
0.998
0.999
0.955
01.000
0.084
0,900
0,952
0.995
MAD 1
C.05 (0.000)
8.82 (0.001)
0.022 (0.007)
0,020
(0.001)
0.015 (0.000
0.009 (0 0010
0.131 (0.012)
0.156 (0.007)
0.127
(0.011)
0.009
(00004
cutt (1.00)
8016 (0/000))
0:00 (0.013)
01,022 (01812)
(0000)
0:012
(0:000
1.102 (0.035)
(0.000)
0.375
0.072 (14006)
(0.000)
(0.000)
00.00)
0.586 (0.000)
(0.001)
(0.009)
0.078 (0.081)
(0.004)
(0,007)
0,509 (0.007)
MAD
6.010 (0.001)
aunno (0.000)
0,006 (0.001)
01:00 (0.091)
(1085 (0.0000
0.001 (0.000)
0/033 (0.005)
Buost (0.001)
0.046 (0.00))
0J018 (0.001)
SD
0.018 (0.002)
0.015 (0.002)
0.0L0 (0.003)
0.015 (0.008)
0.000 (0.002)
0.004 (0.000)
0.057 (0.007)
0.067 (0.008)
0,060 (0.001)
0,025 (0.005)
0.855 (0.010)
1:800 (0.240)
0.906 (0.1015)
0.975 (0.1455
0.986: (11003)
0.500 (0.0022)
1.000 (o.000)
0.901 (0.0017)
0.971 (0.013)
0.987 (0.1105)
Algorithm Conditional sampling with HYFORMER
Input: Number of examples to sample K, batch size B, condition Y, model parameters 0.
1: D Dsampled = 0
2: while (Daamptedl= < K do
3: Sample B many examples (x, y) ~ -e(xx,)) 8)
Accept examples Dbatch = {(x, v) y Y
Append dataset D Psampled D Psamlled U D batch end while
22

---

Published in Transactions on Machine Learning Research (12/2025)
Table 9: Number of accepted samples per 100,000 generated molecules in the conditional generation experiment.
Mean and standard deviation across three random sceds. The average (Avg.) is computed over all target values.
Model
QED=0.5
QBD=0.7
QED=b.9
8A=0.7
8A=0.8
SA=0,9
log|==0.0
log!=2.0
log|==4.0
Ava.
HYFORMER (no-joint)
315 (7)
367 (11)
162 (11)
144 (6)
448 (15)
229 (23)
14 (3)
76 (7)
70 (2)
203 (81)
Bvronaet (joint)
(19)
330 (4)
(17)
(10)
426 (13)
(7)
11
70 (7)
(4)
(71)
1.2 Out- -iitr----Ditiiuuiion Molecular Property Prediction Task
We use HYFORMER pre-trained on UniMol dataset and perform a grid search over hyperparameters, as detailed Table 10, with end-to-end joint fine-tuning, with early stopping triggered if the validation loss
does not improve for 5 consecutive epochs. Results in Table 2 are reported from (Steshin, 2023).
Table 10: Hyperparameter ranges for the grid search hyperparameter optimization on out-of-distribution molecular property prediction task.
HYPERPARAMETER
SEARCH HLNNE
MAX Erocus
(20. 50, 100)
BATCH SIZE
(64, 128, 256]
LRARNING RATE
1E-5. 68.4
WEIGHT DECAY
[x-2, Is-1]
[0.0, 0.2]
GRARNING RATE DECAY
(TRUR, FAUSK)
(Prz)) (trua)
(00.0, 1.0), (0.1, 0.9)
1.3 Molecular Representation Learning Task
For KNN probe, we use the euclidean norm to pick K most similar molecules. For each dataset, we search the parameter K the set (1,3, 5, 100,300 500, 1000, 3000, 5000) and pick K with the best performance on the validation split. For linear probe, we report the results of linear probe with L2 regularization added. If the validation loss between the epochs does not decrease by more than 0.0001 for 10 consecutive epochs, we terminate the training process early. All results in Table 3 are ours.
1.4 Molecule Generation Task
For generation, we use HYFORMER pre-trained on GuacaMol and sample using multinomial top-k sampling, with k = 10 and varying temperature f0.9, 1.0, 1.1].
In Table 4, baseline results for JTVAF and MAGNeT are reported from (Hetzel et al., 2023), for MoLeR and MiCaM from (Geng et al., 2023), for VAE, LSTM from (Brown et al., 2019), for MolGPT from (Bagal et al., 2022).
1.5 Molecular Property Prediction Task
We use HYFORMER pre-trained on UniMol dataset and perform a grid search over hyperparameters, as detailed in Table 11, with end-to-end predictive fine-tuning run for a maximum of 20 epochs, with early stopping triggered if the validation loss does not improve for consccutive epochs. Results in Table 5 are reported from (Zhou et al., 2023; Gao al., 2024b).
1.6 Antimicrobial Peptide Design
We construct a general-purpose peptide dataset and an AMP-specific dataset. For the general purpose dataset, we collect 3459247 peptide sequences with length 8-50 from the combined Peptipedia (Cabas-Mora et al., 2024) and UniProt (Consortium, 2024) datascts and apply CDHIT filtering with a similarity threshold of 90%. For the ApP-specific dataset, we collect 1056321 sequences from combining the Peptipedia (Cabas-Mora et al., 2024), filtered with Antigram (-), Antigram (+), Antibacterial and
Dataset
23

---

Published in Transactions on Machine Learning Research (12/2025)
Table 11: Hyperparameter ranges for the grid search hyperparameter optimization on molecular property prediction task.
HYPERPARAMETIR
SEARCH RANGE
BATCH SIxE
[16. 64, 128, 256)
LEARNING RATE
1x-5, 1E-3]
WEIGHT DECAY
1E-2. 36.
POOLER DROPOUT
[0.0, 0.2]
LEARNING RATE DECAY
(TRUE, FALSE]
Table 12: Unconditional generative performance on MOSES benchmark. The best model in each category is marked bold.
MODRL
VALIITY
UNIQUR
NOVEETY
INTDIVI
INtDiv2 1
UNCONDITIONAL
HMM
0.076
0.567
0.999
0.847
0.810
NGRAM
0,238
0,922
0.909
0.874
0.864
COMBINATORIAL
1.000
0.991
0.958
0.573
0.867
CEARRNN
0.975
0.999
0.842
0.856
0.850
VAE
0.977
0,998
0,856
0,850
AEE
0.937
0.997
0.793
0.856
0l850
LATRNTGAN
0.850
JT-VAE
1.000
0.999
0.914
0.855
0.849
MoLGPT
0.99-
1.000
0.797
0,857
0.851
HYFORMER,
0.996
1.000
0.701
0.851
0.845
HYFORMER, -10
0.991
1.000
0.749
0.856
0.850
HYFORMER,
0.986
1.000
0.791
0.861
0.855
FEW-SHOT
PRAPUGPT-IW. T-1..
0.995
0.995
0.255
0.854
0.850
GRAPHGPT-II 50.5
0.993
0.996
0.334
0.856
0.848
GRAPHGPT-IW.
0.978
0.997
0.871
0.860
0.8157
GRAPUGPT-IW:
0.972
1.000
1.000
0,850
0.847
Antimicrobial keywords, Uniprot with the keywords antimicrobial and AMPSphere (Santos-Júnior et al., 2022), and applying CDHIT filtering with similarity threshold of 90%.
Pre-trainig We pre-train HYFORMER in a two-stage manner, by first training on the general-purpose, followed by training on the AMP specific dataset with peak learning rate equal to 4c-4. All additional details follow Appendix
Fine-tuning We fine-tune HYFORMER for maximum of 10 epochs, with batch size 64, peak learning rate 5e-5 and early stopping, with task probabilities (M(n)) P(Prm) equal to (0.6, 0.4). Additionally, we freeze the first four layers of the model.
Conditional Sampling For the antimicrobial peptide design experiment, we unconditionally sample around 6.7M sequences and accept 50K of them, which results an acceptance rate of around 0.7%. Note that our sampling procedure intentionally prioritizes high selectivity over throughput. All peptides have a maximum length of 50 AAs.
J Additional Experiments
J.1 Unconditional Molecule Generation MOSES benchmark
To additionally evaluate the unconditional generative performance of HYFORMER, perform an evaluation the MOSES benchmark. Analogously to unconditional molecule generation in Section 5.2.1, we scale HYFORMER to 8.5M parameters and follow all the training details in Appendix H for GuacaMol dataset. We compare HYFORMER, across various sampling temperatures т, to baseline unconditional and few-shot. generative models, as reported in (Gao et al., 2024b).
HYFORMER successfully generates valid, unique, novel and diverse molecules, rivaling other unconditional and few-shot generative models.
24

---

Published in Transactions on Machine Learning Research (12/2025)
J.2 Qualitative Evaluation of Generated Molecules
To igges the effect of sampling lemperature the structural diversity and chemical quality of generated molecules, we show molecules sampled in the unconditional generation task (Section 5.2.1), at temperatures 7= 0.9, 1.0, and 1.1. For each sampled molecule, we additionally report four chemical properties: molecular partition coefficient (LogP), topological polar surface area (TPSA), quantitative estimate of diru-likeness (QED) and molecular weight (MW). At. r 0.9, the model generates drug-like molecules, with the majority exhibiting QED 2 0.7 and MW 500 g/ g/mol (Fig. 3). At 7 1.0, the sampling process yields molecules with greater structural diversity (Fig. 4). Despite the increased exploration of chemical space, some molecules exhibit lower QED values. At T = 1.1, the model produces molecules with less common substituent patterns. Some of these structures exceed traditional drug-likenoss thresholds, such as MW > 500 g/ g/mol or LogP > 5, according to Lipinski's Rule of Five (Fig. 5). Additionally, we investigate molecules generated in the conditional generation task in Section 5.1.1 (Figure 6, and 8).
J.3 Qualitative Evaluation of Learned Representations
We next examine the Hyformer embeddings in the context of the chemical properties of the molecules (Fig. 9). To this end, we randomly sample 20,000 molecules and pass them through HyFORMER's encoder, pre-trained for molecular property prediction Section 5.2.2, to obtain molecule embeddings. We visualize the embeddings in two dimensions through principal components analysis (PCA) and color them according to their four chosen chemical properties (LogP, TPSA, QES, MW).
Qualitatively, the spatial arrangement of molecules is clearly connected to their chemical properties. Further- more, embeddings exhibit a smooth profile of change w.r.t. each property. These observations indicate that HYFORMER learns well-behaved, information-rioe molecular representations.
25

---

Published in Transactions on Machine Learning Research (12/2025)
MW: 448.55 gimol
MW: 218.30 gimal
MW: 254.29 g/mol
QED: 0.69 TPSA: 107 10
LogP: 1.00
OED: 0.76 TPSA: 40.46
Logg: 2.91
QED: 0.70
TPSA: 52:31 | Logp: 2.42
MW. 298.29 29 g/mol
MW: 392.50 gimal
MW: 497.62 697-62 g/mol
QED: 0.81 TPSA: 46.15
Loge: 3.03
QED: 0.79
TPSA: 49.65 | LogP: 3.81
QED: 0.54
TPSA: 97.71
LogP: 3.52
MW: 351.44 41 g/mol QED: 0.71 | TPSA: 62. 62.97
MW: MW: 338.32 gimal
MW: MWV 376.42 g/mol
LogP: 1.52
QED: 0.71
TPSA: 96.97 | Logg: 3.20
QED: 0.73
TPSA: 95.94 | LogP: 1.39
MW: 331 37 g/mol QED: 0.73 TPSA: 66.40 LogP: 4.61
MW: 363.46 g/mal
MW: 311.77 gimal g/mal
QED: 0.88 TPSA: 55.40 LogP: 3.84
QED: 0.76
TPSA: 38.33 LogP: 4.75
Figure 3: Structures of the twelve generated molecules with Hyformer when the sampling temperature is 0.9, visualized using RDKit, together with their properties.
26

---

Published in Transactions on Machine Learning Research (12/2025)
MW: 625.29 g/mol
MW: 480.56 g/rmal
MW:: 430.51 g/mol
QED: 0.32
TPSA: 83.83
LogP: 9.81
QED: 0.15 TPSA: 110.79 | LogP: 5.05
QED: 0.56
TPSA: 92:35 | Logp: 3.03
MW: 349.84 g/mol
MW: 392 46 g/mol
MW: 421.95 g/mol g/mo!
QED: 0.81 TPSA: 46.61
LogP: 4.42
QED: 0.64 | TPSA: 69.48
LogP: 3.39
QED: 0.32
TPSA: 31.35 | LogP: 6.93
MW: 408.47 g/mol
MW: 443.46 g/mol
MW: 285. 34 g/mol
QED: 0.48
TPSA: 75.42
LogP: 4.39
QED: 0.29
TPSA: 112.40 | LogP: 2.52
QED: 0.82
TPSA: 72.55 | Logg: 2.78
MW: 364.52 g/mal QED: 0.84 TPSA: 64.68 LogP: 3.49
MW: 258.31 g/mal
MW:: 342. 79 g/mol
QED: 0.74 TPSA: 70.67 LogP: 2.58
QED: 0.56
TPSA: 81.92 LogP: 2.99
Figure 4: Structures of the twelve generated molecules with Hyformer when the sampling temperature is 1.0, visualized using RDKit, together with their properties.
27

---

Published in Transactions on Machine Learning Research (12/2025)
QED:
LogP:
QED:
LogP: 3.02
TPSA: 115.43
LogP:
MW: 380.47 gimol QED: 0.64
MW: 583.683666 64 g/mol
MW: 391. 39 g/mol
TPSA: 68.30
4.62
QED: TPSA: 105.92
Logg: 7.14
QED: 0.59
TPSA: 48.65
LogP: 4.99
MW: 390.26 26 gimol
MW: 606.8 61 g/mol
MW 221. 34 g/mol
QED: 0.59 | TPSA: 50.22|
LogP: 3.98
QED: 0.29 | TPSA: 86.95
LogP: 4.84
QED: 0.71 TPSA: 32.26
LogP: 1.95
MW: 340.34 gimol
MW: 512.616666 a/mol
MW: 252.34 g/mel
QED: 0.49 TPSA: 109.54 54 LogP: 2.57
QED: 0.34 TPSA: 115.39 LogP: 4.23
QED: 0.74 TPSA: 49.33 0o77 0.77
Figure 5: Structures of the twelve generated molecules with Hyformer when the sampling temperature is 1.1, visualized using RDKit, together with their propertics.
28

---

Published in Transactions on Machine Learning Research (12/2025)
HyN
Stooit
QED=0.5
MW: 384.89 g/mol TPSA: 54.38
MW: 481.57 g/mol
MW: 403.48 g/mol
QED: 0.51
LogP: 3.16
QED: 0.5
TPSA: 96.02
LogP: 3.62
QED: 0.5
TPSA: 79.49) | LogP: 4.12
QED=0.7
MW: 396.47 g/mol TPSA: 102.01
MW: 220. g/mol
MW: 371.44 g/mol
QED: 0.7
LogP: 1.31
QED: 0.7
TPSA: 44.89
LogP: 2.11
QED: 0.69
TPSA: 64.11 | LogP: 3.74
QED=0.9
MW: 260.34 g/mol QED: 0.9 | TPSA: 49.41 Logg: 1.82
255.27 g/mol TPSA: 47.56
MW: 370.42 g/mol
QED: 0.9
LogP: 2.98
QED: 0.89 | TPSA: 50.8 | LogP: 3.76
Figure 6: Structures of molecules generated by Hyformer conditioned QED values, visualized using RDKit, along with their chemical properties.
29

---

Published in Transactions on Machine Learning Research (12/2025)
SA=0.7
MW: 380.49 g/mol
589.69 g/mol
583.13 g/mol
QED: 0.71 TPSA: 36.36 LogP: 4.85
QED: 0.46 | TPSA: 133.32
LogP: 4.34
QED: 0.09
TPSA: 166.27
LogP: 3.09
SA=0.8
MW: 440.95 g/mol
MW: 385.5 g/mol
MW: 288. 74 g/mol
LogP:
QED: 0.64 70.0 LogP: 4.03
LogP:
SA=0.9
MW: g/mol
g/mol
351.43 g/mol QED: 0.87 |
QED: 0.54 TPSA: 51.22 LogP: 5.22
QED: 0.85 | TPSA: 34.14 | LogP: 3.48
TPSA: 86.79 LogP: 0.89
Figure 7: Structures of molecules generated by Hyformer conditioned on SA score, visualized using RDKit, along with their chemical properties.
30

---

Published in Transactions on Machine Learning Research (12/2025)
LogP=0.0
306.32 g/mol
275.33 g/mol TPSA: 90.29
MW: g/mol
QED: 0.63
TPSA: 104.58
LogP: 0.58
QED: 0.78
LogP: 0.29
QED: 0.71
TPSA: 94.7
LogP: 0.48
LogP=2.0
MW: 281.36 g/mol
MW: 488.57 g/mol
MW: 504.65 g/mol
QED: 0.81
TPSA: 64.1
LogP: 2.33
QED: 0.49
TPSA: 122.09
LogP: 3.25
QED: 0.4
TPSA: 133.47
LogP: 2.17
LogP=4.0
MW: 392.48 g/mol QED: 0.65 TPSA: 68.29 LogP: 4.51
MW: 276.42 g/mol
MW: 358.51 51.51 g/mol
QED: 0.79 TPSA: 37.3 LogP: 4.02
0.48 TPSA: 53.38 LogP: 4.16
Figure 8: Structures of molecules generated by Hyformer conditioned on LogP values, visualized using RDKit, along with their chemical properties.
31

---

Published in Transactions on Machine Learning Research (12/2025)
LogP
TPSA
10
10
0.8
PCA
0
0.6
Loge
PCA 2
0.6
TPSA
-5
-5
0.2
0.2
-10
-15
-5
PCA
QED
MW
10
0.8
0.8
0.6
PCA
0
PCA
0.4
0.4
0.2
0.2
-10
-10
-15
-10
-5
10
-15
-10
-5
0
5
10
PCA 1
PCA
Figure 9: Hyformer's molecular embeddings. The considered chemical properties normalized to lie in the [0, 1] interval.
32


Reviews:
[{"review_id": "lUvhPuVjpi", "reviewer": "Reviewer_ookq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "A great contribution unifying generative and predictive models for molecules.\n\nThe authors provide a great solution for SMILES-based generation, which does not yet exist in the literature.\n\nThey propose and alternating mask for the same transformer model, which is a clever and simple approach to achieve a seamless joint training. The authors give mathematical explanations of why there can be no conflicts, typical for joint learning.\n\nThe model excels or is on par to representation learning tasks, conditional and unconditional generation, as well as property predictions.\n\nWell done!", "claims_and_evidence_yn": "Yes", "claims_explanation": "Very clear presentation overall.", "audience_yn": "Yes", "audience_explanation": "Very appropriate contribution on generative modeling and property prediction of molecules. I think the most interesting aspect is that for realistic scenarios (e.g. OOD testing and/ or AMP design), it works well.", "requested_changes": "What does validity measure in Table 1? Could you clarify in the text?\n\nWhat are the confidence intervals in Table 1?\n\nClean up the \"the the\" in the text.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "W3LZYBVSgA", "reviewer": "Reviewer_fGqQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose Hyformer, a transformer-based joint model for molecular generation and property prediction. Unlike prior approaches that train separate models, Hyformer shares a single backbone across both tasks and switches between autoregressive (causal) and bidirectional attention modes via task tokens. Two specialized heads handle generative and predictive outputs. The model is evaluated on standard molecular ML benchmarks (e.g., GuacaMol, MoleculeNet, Lo-Hi) and demonstrates strong performance in unconditional/conditional generation, out-of-distribution property prediction, and molecular representation learning. The paper also includes a case study on antimicrobial peptide (AMP) design.\n\n# Key strengths include:\n- A simple but effective architectural unification of predictive and generative tasks.\n- Broad evaluation across widely used benchmarks.\n- Strong empirical performance, including improved OOD prediction.\n\n# Key weaknesses include:\n- Alternating attention is closely related to prior multitask transformer work (e.g., UniLM, Dong et al. 2019).\n- Some dataset details are insufficiently documented or missing.\n- Certain benchmark metrics (FCD, KL div., etc.) are used without definition, potentially confusing to readers outside the molecular ML community.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The evidence is generally convincing, with experiments, baselines, and supplementary details. However, there are a few outstanding issues requiring clarification:\n\n- Exact version or data of acquisition of the used resources for the GuacaMol and AMP datasets.\n- Lacking information of the generation process of the AMPs ( at least number and size of generated sequence).\n- Missing definition for the used metrics \"Diversity\" even in the referenced Li et al., 2024 paper.\n- Explanation for the higher than expected ClinTox scores (99.5) in Table 3.", "audience_yn": "Yes", "audience_explanation": "The work addresses a long-standing challenge in molecular ML which is jointly unifying generative and predictive modeling. Both the methodological contribution and the practical results are relevant for TMLR’s readership in machine learning and computational biology.", "requested_changes": "# Critical fixes:\n- Dataset version or acquisition date (also state which AMP training examples are predicted only)\n- Resolve issue with \"Diversity\" in section 5.3\n- Provide justification for the higher than expected ClinTox scores (99.5) in Table 3.\n- KL div. in an ML paper should be only used for Kullback–Leibler divergence (not a derived benchmark score)\n\n# Benchmark metric definitions:\n- Explicitly define or reference all benchmark metrics used (at least in the supplement). For example:\n- FCD, KL div., Val., Uniq., Nov. in Table 4 (GuacaMol).\n- Esol, Freesolv, Lipo, BBBP, BACE, ClinTox, Tox21, ToxCast, SIDER, HIV datasets in Tables 2, 3, 5.\n- Even if these are standard in chemistry/ML, they should be clarified for a computer science audience.\n\n# Novelty statement:\nClearly distinguish Hyformer’s alternating-attention mechanism from prior work such as Dong et al. 2019. A transparent discussion of similarities/differences will strengthen positioning.\n\n# Minor changes:\n- Fix the typo in Sec. 5.3: “Arginine (R) and Arginine (K)”  - (“Arginine (R) and Lysine (K)” ??)\n- Ensure all acronyms are defined at first mention (e.g., FCD, SD, QED).\n- Add explicit reporting of acceptance rates and predictor calibration for conditional sampling.", "broader_impact_concerns": "I do not see significant broader impact concerns beyond what the authors have already noted. The paper briefly acknowledges potential dual-use (e.g., generation of toxic molecules), which seems sufficient. The primary contributions are methodological and benchmark-driven. I do not think an expanded Broader Impact Statement is strictly necessary.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "igo5VowxIf", "reviewer": "Reviewer_xgqr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper tackles joint generative and predictive modelling for small molecular structures. The proposed Hyformer is based on Transformer encoder for prediction with molecule string inputs, and jointly with Transformer decoder for generative modelling of molecular strings. The joint approach yields strong results on both generative and predictive molecular benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, I believe that all the claims are accurate and backed up by experimental results. The experimental setup is well described in detail in the appendix, and appears to largely follow previous published papers in terms of how the benchmarks are set up, etc. At this point, molecular benchmarks are pretty standardized, so I am fairly confident that the authors implemented them correctly.", "audience_yn": "Yes", "audience_explanation": "Yes, I believe there is now a significant community of machine learners interested in molecular modelling applications, who will find this work exciting and forward looking for its unified perspective on prediction + generation.", "requested_changes": "I realise that some past work may have ignored this, but I would suggest adding standard deviations to tables which currently miss them. As the paper claims to be the top performing model on several benchmarks, it is expected that future work will be asked to compare to Hyformer. It would be good to know how much variation there is in the generative model's performance when varying random seed, for instance, in table 1 on conditional generation.", "broader_impact_concerns": "No concerns.", "additional_comments": "I would be interested in scaling studies where Hyformer is trained on very large scale molecular datasets annotated with properties. Can the model zero-shot generalize to all other molecular benchmarks in that case? If yes, that would be extremely exciting.\n\nAdditionally, currently Hyformer needs Transformer encoder layers. What if these are removed, and property prediction is also framed as a generative task? If that works, it would also be extremely exciting.\n\nLastly, how would Hyformer generalize to considering molecular interactions? E.g. small molecules + protein partners.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)