Repurformer: Transformers for Repurposing-Aware Molecule Generation (2024)

Abstract: Generating as diverse molecules as possible with desired properties is crucial for drug discovery research, which invokes many approaches based on deep generative models today. Despite recent advancements in these models, particularly in variational autoencoders (VAEs), generative adversarial networks (GANs), Transformers, and diffusion models, a significant challenge known as the sample bias problem remains. This problem occurs when generated molecules targeting the same protein tend to be structurally similar, reducing the diversity of generation. To address this, we propose leveraging multi-hop relationships among proteins and compounds. Our model, Repurformer, integrates bi-directional pretraining with Fast Fourier Transform (FFT) and lowpass filtering (LPF) to capture complex interactions and generate diverse molecules. A series of experiments on BindingDB dataset confirm that Repurformer successfully creates substitutes for anchor compounds that resemble positive compounds, increasing diversity between the anchor and generated compounds.

heading: 1 Introduction
text: The design of valid and novel molecules with desired biological properties, known as de novo molecule generation, is vital to modern drug discovery. Recent advancements in deep generative models, particularly variational autoencoders (VAEs) (Kingma and Welling, 2022), generative adversarial networks (GANs) (Goodfellow et al., 2014), Transformers (Vaswani et al., 2017), and diffusion models (Ho et al., 2020), have significantly enhanced our ability to generate chemically valid and novel molecules. However, these models need to be further refined to generate molecules that interact with specific target proteins.
Target-specific molecule generation addresses this challenge by producing drug-like molecules that are more likely to bind with specific target
proteins (Grechishnikova, 2021; Qian et al., 2022; Tan et al., 2022). Nonetheless, there remains a significant issue known as the sample bias problem, where reliance on existing protein-compound pairs results in the generation of structurally similar molecules. This phenomenon limits the diversity of generated molecules and hinders the discovery of novel compounds.
To address this, we propose leveraging multi-hop relationships among proteins and compounds to expand the generative space and increase the diversity of the generated molecules. Our method introduces the concept of repurposing-aware molecule generation, designed to identify and utilize latent multi-hop relations within the protein-compound interaction network.
In this paper, we present Repurformer, a novel model that integrates bi-directional pretraining and advanced signal processing techniques to overcome the limitations of existing models. Repurformer captures complex relationships between proteins and compounds by pretraining encoders in both protein-to-compound and compound-to-protein directions and applying Fast Fourier Transform (FFT) with low-pass filtering (LPF) to the latent space. This approach allows the model to distinguish the different scales of interactions. By focusing on lowfrequency components, which correspond to the longer propagation through the multi-hop proteincompound interaction network, Repurformer generates as diverse compounds as possible with desired properties. In summary, the contributions of our work are threefold:
• We introduce a framework for repurposingaware molecule generation to address the sample bias problem by leveraging multi-hop relations between proteins and compounds.
• We develop Repurformer, a model that integrates a bi-directional pretraining and an FFTbased approach to capture and utilize latent
117
multi-hop relations in an end-to-end manner.
• We demonstrate that Repurformer successfully generates valid and diverse molecules, creating substitutes for anchor compounds that resemble positive compounds.

heading: 2 Preliminaries
text: 2.1 De novo Molecule Generation
De novo molecule generation is the process of exploring vast chemical space and producing novel molecules with desired biological properties. With the rapid advancement of artificial intelligence, recent deep generative models have been widely used in molecule generation tasks.
For example, several VAE variants have been introduced thanks to its manipulable latent space, such as charVAE (Gómez-Bombarelli et al., 2018), SD-VAE (Dai et al., 2018), and JT-VAE (Jin et al., 2018). GAN has been adopted due to their capability to generate new molecules that are highly similar in structure to existing ones, including ORGAN (Guimaraes et al., 2018), ORGANIC (Sanchez-Lengeling et al., 2017), and MolCycleGAN (Maziarka et al., 2020). More recently, Transformers and diffusion models have been utilized, based on their success in language modeling and image generation, respectively, such as MolGPT (Bagal et al., 2022), MDM (Huang et al., 2023), and GeoLDM (Xu et al., 2023).

heading: 2.2 Target-specific Molecule Generation
text: In drug discovery, identifying drug-target interactions (DTI) is crucial for understanding the bioactivity and therapeutic effects of drugs for specific diseases. Although the deep generative models have proved useful in generating novel and chemically valid molecules, further screening is necessary to evaluate their potential to bind with specific protein targets. Building on this notion, several researchers have developed target-specific molecule generation models to produce novel, drug-like molecules that are highly likely to interact with specific target proteins, including Transformer-based generation (Grechishnikova, 2021), AlphaDrug (Qian et al., 2022), SiamFlow (Tan et al., 2022) and POLYGON (Munson et al., 2024).

heading: 2.3 Repurposing-Aware Molecule Generation
text: Drug repurposing is a strategy that identifies new therapeutic uses for approved drugs beyond their
original indications. This approach offers significant advantages over developing entirely new drug, such as lower failure risk and development costs. The concept of drug repurposing can be defined as multi-hop relationships in the protein-compound interaction network, which is not directly connected but can be accessed through intermediaries. In chemical spaces, proteins and compounds have many-to-many relationships based on their structural coordination. This leads to the assumption that if a compound can reach a specific protein through another compound that shares a common protein (i.e., in the multi-hop relationship), there is potential for repurposing the focal compound.
The repurposability in chemical spaces can introduce a new paradigm for molecule generation, by serving as a key to expanding the generative space and increasing molecular diversity. Previous approaches for target-specific molecule generation tend to generate structurally similar molecules for a specific protein target due to their dependence on known protein-compound interactions. While incorporating randomness in the generation process can contribute to molecular diversity, it may neglect structural coordination with targets, possibly resulting in a trade-off between diversity and binding affinity. In this context, leveraging latent repurposability within the multi-hop relationships among proteins and compounds can provide a reasonable boundary for molecule generation, broadening the generative space and enhancing molecular diversity without sacrificing their drug potency.

heading: 3 Problem Statement
text: The discovery of new compounds often relies on existing protein-compound pairs. This results in that the compounds targeting the same protein exhibit similar structures. In other words, the generative space of models tends to be bounded in limited regions, reducing the diversity of the generation. We refer to this as a sample bias problem.
To address this problem, we leverage multi-hop relations among proteins and compounds. Specifically, given a pair of protein p and compound c that are known to interact, we assume that the compound relates to p within a 3-hop relation, i.e., a positive compound c+, has a potential interaction with p. Definitions from 3.1 to 3.3 describe the key concepts of our approach, and Figure 1 visually represents the rationale. Note that both protein and compound are represented by
amino-acid and SMILES sequences, respectively: p = [p1 , · · · , pTp ] and c = [c1 , · · · , cTc ] with Tp and Tc being the fixed length of each sequence.
Definition 3.1 (Protein-Compound Graph). The relations between proteins and compounds can be represented as a bipartite graph G(P ∪ C, E), where P and C denote the sets of protein and compound nodes, respectively. Specifically, p(i) ∈ P represents the i-th protein and c(j) ∈ C represents the j-th compound, for i = 1, · · · ,M and j = 1, · · · , N . Definition 3.2 (Protein-Compound Pair). A pair of nodes in G is represented by an edge eij = {(p(i), c(j))|p(i) ∈ P, c(j) ∈ C} ∈ E . The presence of an edge eij indicates a link between the i-th protein and the j-th compound, such that eij = 1 if they are linked and eij = 0 otherwise.
Definition 3.3 (Anchor/Positive Compounds). Given a target protein p(i), a compound c(j) is defined as an anchor compound ć if eij = 1. For another protein p(k) where ekj = 1, any compound c(l) (l ̸= j) that satisfies ekl = 1 is regarded as a positive compound c+ for the target protein p(i).

heading: 4 Repurformer
text: In this section, we propose Repurformer, a novel method designed to address the sample bias problem by leveraging multi-hop relations among proteins and compounds. Figure 2 illustrates how Repurformer seamlessly integrates the concepts of drug discovery and repurposing.
Bi-directional Pretraining To capture the manyto-many relationships between proteins and compounds, we employed bi-directional pretraining for
the protein and compound encoders. Specifically, we built two Transformers with identical encoderdecoder structures but opposite training directions: one was trained in the protein-to-compound direction, and the other in the compound-to-protein direction (see Figure 2a). By doing so, we expect the protein encoder fp(c|p) and the compound encoder fc(p|c) to extract latent relations, zp and zc, that encompass both cases where proteins and compounds are the head and tail of an edge, and vice versa, i.e., fp : c|p → zp and fc : p|c → zc. For example, given a pair of p(2) and c(1) as shown in Figure 2b, zp and zc will represent the edges from p(2) to c(1) (i.e., p(2) → c(1)) and from c(1) to p(1) (i.e., c(1) → p(1)), respectively.
Transformer with Bi-directional Encoders The pretrained bi-encoders are then used as feature extractors; they are frozen and followed by a new compound decoder. The compound decoder π(·), parameterized by θ, receives a sum of the encoding vectors h = zp + zc and a positive compound c+ as inputs:
ĉ+t+1 = πθ(·|c+1:t, ht) where ht = zpt + zct .
Here, ht ∈ R|d| represents a |d|-dimensional latent vector of 2-hop relation, e.g., p(2)
1-hop→ c(1) (= ć)
2-hop→ p(1) (see Figure 2b), from a t-th token perspective. Accordingly, feeding the compound decoder with a positive compound as a label enables it to learn potential repurposing relationships that emerge from an additional third-hop edge, e.g., · · · 2-hop→ p(1) 3-hop→ c(2) (= c+). Putting it all to-
gether, the loss function is defined as follows:
lnπθ(c +|p, c) = ln
Tc∏
t=1
πθ(c + t+1|c+1:t, p, ć)
=
Tc∑
t=1
lnπθ(c + t+1|c+1:t, p, ć)
Fast Fourier Transform (FFT) The Fourier transform decomposes a function into its constituent frequencies using complex exponentials (sinusoids) as basis functions (Heckbert, 1995; Lee-Thorp et al., 2021). Given a sequence {x1, · · · , xT }, the discrete Fourier Transform (DFT) is defined by the formula:
Xk = T−1∑
t=0
xte − 2πi T tk , 0 ≤ k ≤ T − 1
where Xk is the k-th frequency component, xt is the t-th time-domain signal, and i is the imaginary unit. Calculating the DFT directly has a complexity of O(T 2), which can be inefficient for large datasets. To address this, the Fast Fourier Transform (FFT) algorithm was proposed, reducing the complexity to O(T log T ) (Cooley and Tukey, 1965; Brigham, 1988). In this study, we apply the FFT to h ∈ RT×|d| to construct eigenvectors along which the 2-hop propagation occurs. To be specific, the 2D DFT is utilized: one 1D DFT along the sequence dimension,Fseq, and another 1D DFT along
the feature dimension, Fdim, keeping real-valued parts only as in Lee-Thorp et al. (2021):
H = R(Fseq(Fdim(h))) ∈ RT×|d| .
Note that T is set to the length of a longer sequence; if Tp > Tc, then T is set as Tp and vice versa.
Low-Pass Filter (LPF) The Fourier-transformed features H comprise low frequencies that represent a globally smoothed signal and high frequencies that indicate a locally normalized signal. This separation of frequency components allows for distinct interpretations at different scales. For example, Tamkin et al. (2020) applied the discrete cosine transform (DCT) (Rao and Yip, 2014), which is closely related to the DFT, to separate latent information at different scales. They found that low frequencies capture topic-scale context while high frequencies capture word-scale context.
In our setting, a scale can be understood as the number of hops. Specifically, the lower frequency implies a longer propagation through multi-hop relations while the higher one implies a shorter propagation within a single-hop relation. From the repurposing perspective, we need to leverage the longer propagation so that only the multi-hop relations are considered. To achieve this, we can apply the low-pass filtering (Pollack, 1948; Costen et al., 1996), which removes the frequency components above a certain cutoff parameter α by setting
Hk,d ← 0 for all k, d > α. This filtering can be easily implemented using a binary mask:
HLPF = H ⊙M where M = {mt,d|mt,d ∈ {0, 1}, 1 ≤ t ≤ T, 1 ≤ d ≤ |d|} is an one-hot matrix, with mt,d = 1 for low-frequency components and mt,d = 0 otherwise. Lastly, we transformed HLPF back to the features of an original domain using the inverse FFT (IFFT), before passing it to the compound decoder:
h̃ = F−1dim(F−1seq (HLPF)) ∈ RT×|d| . Implementation Details The structure of the Repurformer is essentially identical to that of pretrained transformers. It consists of encoder and decoder networks, each linearly stacked with 4 layers of 256 dimensions, with each layer divided into 4 heads of 64 dimensions. To tokenize the protein and compound sequences, we utilized existing vocabularies from previous works—the protein vocabulary from Rao et al. (2019) and the compound vocabulary from Honda et al. (2019). For training, we set the number of epochs, batch size, and learning rate to 20, 64, and 5e-05, respectively.

heading: 5 Experiments
text: Experiment Setup We collected data from BindingDB (Gilson et al., 2016) which contains over 2.8 million measured binding affinities of interactions between proteins and drug-like molecules. The collected dataset was then preprocessed to filter out missing values, duplicates, and proteins and compounds with excessively long or short sequences. In particular, given the many-to-many nature of protein-compound relationships, we selected compounds that interact with a reasonable number of individual proteins between 10 and 100, to enable our model to learn various compound structures reacting with different proteins. The resulting dataset comprised 60,719 protein-compound pairs derived from 3,006 proteins and 7,803 compounds. We split this dataset into train and test datasets with 8:2 ratio, ensuring that the proteins interacting with each compound did not overlap between the two sets. Our model was then trained on proteincompound pairs from the train set, representing proteins with amino acid sequences and compounds with canonicalized SMILES strings. We tokenized individual characters from amino acid sequences and SMILES strings, resulting in vocabularies containing 30 characters for proteins and 46 characters for compounds.
Evaluation Metrics To thoroughly assess the effectiveness and reliability of Repurformer, we employed several evaluation metrics, focusing on the generative performance of the model and physicochemical properties and drug-likeness of the molecules it generated. In terms of generative performance, we applied widely accepted metrics for sequence generation tasks: BLEU (Papineni et al., 2002), GLEU (Wu et al., 2016), and F1 score of ROUGE (Lin, 2004). In particular, we used 1- and 2-gram units as the evaluation basis for these generative metrics. We utilized physicochemical properties, specifically molecular weights and log of octanol-water partition coefficients (LogP) (Wildman and Crippen, 1999), to assess the feasibility of molecular structures as drugs. Furthermore, we used other widely used drug-likeness metrics, such as QED (Bickerton et al., 2012), SA (Ertl and Schuffenhauer, 2009), and NP (Ertl et al., 2008), to evaluate the potential effectiveness of the generated molecules as drug-like compounds.
Configurations This study aims to analyze whether the configuration of Repurformer is effective. Given that the distinguishing configuration of Repurformer is the application of FFT with LPF in the embedding space, we conducted comparative experiments with different configuration options:
• SUM Only: This is the baseline configuration. It directly passes h to the compound decoder.
• +FFT: This configuration transforms h to H but does not revert it to h̃.
• +MLP: This configuration adds a single fullyconnected layer that mixes the values of h feature-wise.
• +FFT+MLP: This configuration mixes the frequencies of H .
• +FFT+MLP+IFFT w/ auxiliary losses: This configuration mixes the frequencies of H and reverts the mixed H to h̃. Note that L1, L2, and Frobenius norm are added as auxiliary losses to minimize the distance between the MLP output and h̃.

heading: 6 Results
text: Main Results To evaluate Repurformer, we conducted a comparative analysis of 11 configurations, focusing on generative performance, physicochemical properties, and drug-likeness.
Table 1 shows the similarity of the generated compounds ĉ+ to both the anchor ć and positive c+ compounds, calculated using BLEU, GLEU, and ROUGE scores. The results indicate that the “+MLP” and “Repurformer with α = 4” exhibit remarkable performance compared to other configurations. Notably, the Repurformer (α = 4) generated compounds with higher structural similarity to the anchor compounds than those generated by the +MLP configuration. This suggests that Repurformer successfully generates compounds that are potentially repurposable to the target proteins.
In Tables 2 and 3, we can compare the molecular properties of the generated compounds from different configurations. Table 2 shows that Repurformer with α = 4 generates compounds that are the most physicochemically desirable. On the other hand, Table 3 shows that the Repurformer with α = 4, with α = 2, and “+FFT+MLP” configu-
rations had comparative advantages in QED, SA, and NP, respectively. Given that QED is generally considered the most important metric for measuring drug similarity and efficacy, we can emphasize that Repurformer (α = 4) excels in generating compounds with the highest potential for effective drug discovery. Figure 3 compares the generation results of the ‘+MLP’ and ‘Repurformer (α = 4)’ configurations.
Performance Comparison To assess the effectiveness of Repurformer as a target-specific molecule generation model, we compared its performance with the existing protein-specific generative approaches as external baseline models, including Transformer-based model (Grechishnikova, 2021) and AlphaDrug (Qian et al., 2022). Transformer-based model utilized the vanilla Transformer architecture (Vaswani et al., 2017) to generate compounds based on target proteins. This model viewed the target-specific molecule genera-
tion as a translational task, converting amino acid sequence into SMILES strings. AlphaDrug modified the vanilla Transformer by introducing skipconnections between its encoders and decoders, facilitating the joint embedding of target proteins and molecules. In addition, it employed a Monte Carlo tree search algorithm for the conditioned generation of novel molecules based on specific target proteins.
To ensure a fair comparison, we trained the external baseline models using our dataset using the same experiment setup and evaluation metrics as for Repurformer. Table 4 presents the performance comparison between our best configuration (Repurformer with α = 4) and the external baseline models. The comparison results demonstrate that the Repurformer (α = 4) outperformed the existing approaches on most evaluation metrics. In particular, our model generated compounds with high structural similarity to both the anchor and positive compounds than those generated by the external baseline models. This suggests that Repurformer can generate not only realistic but diverse compounds with methodological considerations for
drug repurposability. Regarding drug-likeness, our model achieved the highest performance only on QED. Although the Transformer-based model excelled in SA and NP, the feasibility of its generated compounds is questionable due to its exceptionally high scores in physicochemical properties, which indicate the compounds might not be suitable as medicines. This is further validated by the evaluation of compound validity, as illustrated in Figure 8 in the Appendix. The compounds generated by the Transformer-based model were significantly less valid compared to those generated by Repurformer.
Mitigation of Sample Bias Figure 4a shows that the distance distribution of the generated compounds to the anchor compounds is similar to that of the positive compounds. We calculated the distance over the fingerprint domain to consider the patterns of molecular substructure. The result implies that the generated and positive compounds have different substructures from the anchor compounds to the same extent. However, Figure 4a compares “the relative distances” of the generated and positive compounds to the anchor compounds
“at the substructure level,” making it difficult to directly compare the absolute distances to each other at the holistic level.
Figure 4b visualizes the overlapping representations among the anchor, positive, and generated compounds, “directly comparing their absolute distances at the holistic level.” To do this, we extracted SMILES word embeddings (e.g., C, N, F, =, +, [, ], etc.) using Word2Vec (Mikolov et al., 2013) and defined the holistic representation of each molecule as the summation of these word embeddings. We then projected the holistic representation of each molecule into a 2-dimensional space by t-SNE (Van der Maaten and Hinton, 2008). Since t-SNE embeddings preserve pairwise similarities of high-dimensional data as neighboring points in
a low-dimension, it allows for direct comparison of absolute distances between samples. Finally, we applied Gaussian kernel density estimation (KDE) to visualize the distribution of t-SNE embeddings.
The results from Figures 4a and 4b indicate that the generated compounds are more similar to the positive compounds than to the anchor compounds, both relatively and absolutely, and at substructural and holistic levels. This suggests that Repurformer successfully addressed the sample bias problem, creating substitutes for anchor compounds that resemble positive compounds.
Existence of Mode Collapse Mode collapse refers to a phenomenon where the generative model creates high-quality samples at the expense of indistribution diversity (Adiga et al., 2018). In this
section, we demonstrate that Repurformer suffers from mode collapse and thus the “internal” diversity of generated compounds is relatively lower than anchor and positive compounds.
Figure 5 illustrates the negative relationship between the validity and uniqueness of the generated compounds by different values of α. Validity represents the ratio of samples that can be depicted as 2D molecular drawings by RDKit (i.e., the quality of generation), while uniqueness represents the ratio of non-duplicated samples (i.e., the diversity of generation). As α increases, we observe that uniqueness increases but validity decreases. This is an expected outcome given that the low-frequency signals represent global structure whereas the highfrequency signals represent local structure. For example, low-frequency signals (i.e., lower α) focus on the most fundamental structures, increasing the validity of generated compounds but reducing their uniqueness. Conversely, high-frequency signals (i.e., higher α) focus on local details, increasing the uniqueness of generated compounds but reducing their structural validity. In short, Figure 5 demonstrates that Repurformer may be susceptible to the validity-uniqueness trade-off, i.e., mode collapse, and thus α must be carefully selected.
Figure 6 describes the “internal” diversity of valid compounds generated by Repurformer with α = 4, along with anchor and positive compounds that share the same target proteins with the generated ones. Following Pereira et al. (2021), we evaluated the internal diversity within each compound group using two metrics: Levenshtein distance (Ld) and maximum common substructure (MCS). The Levenshtein distance (Levenshtein et al., 1966), also known as edit distance, measures the difference between two SMILES strings by calculating the minimum number of insertions, deletions, and replacements needed to make the strings identical. On the other hand, the MCS (Cao et al., 2008) measures the ratio of the number of atoms in the maximum common substructure of two compounds to their total number of atoms. Since the MCS represents a similarity score normalized between 0 and 1, the MCS distance can be obtained by 1- MCS, which captures the atom-level dissimilarities of compounds. The diversity of each compound group was computed by averaging the pairwise distances of all compounds. The result indicates that the internal diversity of the generated compounds is lower than that of the anchor and positive compounds, suggesting mode collapse in Repurformer.
Note that the existence of mode collapse does not contradict the mitigation of sample bias. Mode collapse refers to less internal diversity among generated compounds, while mitigating sample bias involves creating substitutes for anchor compounds that resemble positive ones, thus increasing diversity between the anchor and generated compounds.

heading: 7 Limitations
text: This study has some limitations. First, due to inconsistencies between the tokens in our dataset and those we borrowed from previous research, some generated outputs contained <UNK> tokens, which had to be excluded. Second, the study lacks experiments on binding affinity, which are necessary to evaluate how strongly the generated compounds bind to proteins. These limitations must be addressed in future research.

heading: 8 Concluding Remarks
text: In this study, we introduced Repurformer, a novel model designed to address the sample bias problem in de novo molecule generation by leveraging multi-hop relationships. Repurformer integrates bidirectional pretraining with Fast Fourier Transform and low-pass filtering, to capture complex interactions between proteins and compounds. This approach focuses on low-frequency components, corresponding to longer propagation through multihop protein-compound interactions. The results show that Repurformer successfully generates valid and diverse molecules.
Building on these positive results, there are several promising directions for future improvement. Enhancing the backbone architecture by incorporating advanced models like diffusion or graph neural networks and techniques such as contrastive learning could further improve Repurformer’s ability to capture multi-hop protein-compound interactions. The results from Figures 5 and 6 also suggest promising directions to improve Repurformer such as leveraging reinforcement learning to maximize diversity rewards or introducing Wasserstein loss to address mode collapse. Additionally, while our current experiments have shown the potential of Repurformer, it is critical to validate its applicability in real-world scenarios. Therefore, we need to verify the performance of Repurformer on existing drug repurposing cases. Considering these aspects will strengthen the practical implications and utilities of Repurformer.

heading: Acknowledgments
text: This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (RS-2023-00275796).

heading: A Supplementary Materials

heading: Future Work
text: Building on these positive results, there are several promising directions for future improvement. Enhancing the backbone architecture by incorporating advanced models like diffusion or graph neural networks and techniques such as contrastive learning could further improve Repurformer’s ability to capture multi-hop protein-compound interactions. The results from Figures 5 and 6 also suggest promising directions to improve Repurformer such as leveraging reinforcement learning to maximize diversity rewards or introducing Wasserstein loss to address mode collapse. Additionally, while our current experiments have shown the potential of Repurformer, it is critical to validate its applicability in real-world scenarios. Therefore, we need to verify the performance of Repurformer on existing drug repurposing cases. Considering these aspects will strengthen the practical implications and utilities of Repurformer