# Dual Embedding: A Fine-Tuned Language Model Approach for Accurate Polymer Glass Transition Temperature Prediction

Aymar Tchagoue, Véronique Eglin, Jean-Marc Petit, Sébastien Pruvost, Jannick Duchet-Rumeau, Jean-François Gérard

<figure><img src="image_1.png" /></figure>

## To cite this version:

Aymar Tchagoue, Véronique Eglin, Jean-Marc Petit, Sébastien Pruvost, Jannick Duchet-Rumeau, et al. Dual Embedding: A Fine-Tuned Language Model Approach for Accurate Polymer Glass Transition Temperature Prediction. Journal of Chemical Information and Modeling, 2025, 10.1021/acs.jcim.5c02469. [hal-05355618](https://hal.science/hal-05355618v1)

HAL Id: hal-05355618

https://hal.science/hal-05355618v1

Submitted on 8 Nov 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Dual Embedding: A Fine-Tuned Language Model Approach for Accurate Polymer Glass Transition Temperature Prediction

Aymar Tchagoue¹,², Véronique Eglin¹, Jean-Marc Petit¹, Sébastien Pruvost², Jannick Duchet-Rumeau², Jean-François Gérard²

¹INSA Lyon, UCBL, CNRS, LIRIS UMR 5205, Villeurbanne, France

²INSA Lyon, UCBL, Université Jean Monnet, CNRS, IMP UMR 5223, Villeurbanne, France

## Abstract

Recent years have witnessed major advances in polymer informatics, yet accurately predicting polymer properties, such as the glass transition temperature ($T_g$), remains a challenge. Language models like BERT have been leveraged to derive embeddings from polymer representations (e.g., SMILES). However, similarity between embedding vectors in these latent spaces primarily reflects chemical structural similarity, with limited alignment to physicochemical properties.

Here, we introduce a dual-embedding framework that enhances $T_g$ prediction by combining a conventional BERT-based embedding with a fine-tuned counterpart explicitly trained so that vector similarity reflects proximity in $T_g$ values.

We evaluate our approach across four benchmarks: a heterogeneous dataset compared against 25 machine-learning baselines, along with three additional datasets focused on homopolymers and polyimides. The dual embedding outperforms standard BERT-based embeddings, achieving up to a 20% reduction in RMSE and surpassing alternative models such as graph-based and descriptor-based approaches. These results demonstrate that embedding molecular properties directly into representations can advance polymer informatics beyond structure-centric paradigms.

## 1 Introduction

Recent advances at the intersection of artificial intelligence and materials science have enabled the modeling of complex structure–property relationships from textual representations of molecules, such as SMILES, Simplified Molecular Input Line Entry System introduced in 1988 in [25]. This compact chemical language has opened new avenues for representing molecular structures computationally and predicting their associated physicochemical properties. In polymer informatics, this synergy is particularly promising: data-driven models offer the potential to predict key physicochemical properties without costly and time-consuming experiments.

This growing opportunity has already led to significant advances in the prediction of polymer properties [2, 11, 14, 17, 22]. Among these properties, one stands out for its particular interest: the glass transition temperature ($T_g$). This temperature represents the point at which a polymer transitions from a glassy state to a rubbery state. It is a crucial property for selecting materials for practical applications, yet it remains extremely difficult to predict. Many studies have aimed to predict it based on chemical structure [3, 6, 11, 21, 23]. Among these approaches, the use of language models

for polymer encoding is becoming increasingly predominant [4, 11, 17, 24, 27].

However, existing language-model embeddings group polymers in the latent space mainly according to structural similarity. While this organization can enable predictions, it does not fully exploit the potential of BERT-based embeddings for property prediction, particularly for $T_g$.

As illustrated in the graphical abstract Fig. 1, we propose a dual-embedding approach that augments standard polymer embeddings with a fine-tuned version, in which vector proximity reflects similarity in $T_g$ values rather than solely structural similarity derived from SMILES. By aligning the geometry of the latent space with property-based similarity, this method establishes a property-aware embedding strategy that more effectively captures relevant functional relationships.

We validate this approach on several datasets, including heterogeneous polymers, polyimides, and homopolymers, comparing our results with existing state-of-the-art models. Our findings demonstrate consistent improvements in prediction accuracy, with reductions in RMSE of up to 20% over standard embeddings. This shows the promise of fine-tuned language models not only for structure-based but also for property-driven representation learning in polymer design.

To demonstrate the effectiveness of our approach, this paper is organized as follows: Section 2 reviews the related work that is used as baselines; Section 3 presents the proposed dual embedding methodology with a proof-of-concept example; Section 4 evaluates its performance across multiple datasets and compares it to existing methods; Section 5 provides additional analyses to further explore its potential; Section 6 offers a broader discussion of the results, including their limitations; and Section 7 concludes the study. The data and software used in this work are available in Section 8.

## 2 Related Work

### 2.1 Standard polymer embeddings

RDKit [12] is a widely used open-source cheminformatics toolkit that offers a broad range of tools for molecular modeling, data analysis, and machine learning in chemistry. One of its main features is the calculation of molecular descriptors, which quantitatively represent different aspects of molecular structure.

These RDKit descriptors are commonly employed in Quantitative Structure–Property Relationship (QSPR) models. They span a broad range of types, including topological, geometric, electrostatic, and solubility-related descriptors. While powerful, their effective use often requires careful feature engineering tailored to the specific property of interest. In the context of polymer informatics, such

Manuscript accepted in the *Machine Learning and Deep Learning* section of the *Journal of Chemical Information and Modeling*, Published November 3, 2025, DOI: 10.1021/acs.jcim.5c02469.
<figure><img src="image_3.png" /><figcaption>Figure 1: Graphical Abstract of the dual embedding method</figcaption></figure>

descriptors have been successfully used to predict properties like the glass transition temperature ($T_g$) [3, 23]. However, a known limitation is that QSPR descriptors may overlook subtle structural motifs or patterns not captured by predefined numerical features.

To address this, alternative representations such as Extended-Connectivity Fingerprints (ECFP) [19] have become increasingly popular. Implemented in RDKit as Morgan fingerprints, these representations encode molecular substructures using a circular hashing algorithm. Each substructure, defined by a central atom and its surrounding atoms within a chosen radius, is mapped to a unique identifier. For each molecule, the result is a vector representation in which each component corresponds to the frequency of a distinct substructure. Morgan fingerprints are generated directly from SMILES strings and have proven effective in property prediction tasks, particularly when used with deep learning models such as graph neural networks (GNNs) and transformers [21]. Nevertheless, they primarily capture local structural patterns and may struggle to encode more global molecular features. Additionally, their dimensionality increases with the diversity of patterns in the dataset, raising the risk of the curse of dimensionality.

Beyond handcrafted or pattern-based descriptors, unsupervised representation learning has emerged as a powerful alternative. Techniques like Word2Vec [15], originally developed for natural language processing, have been adapted for chemistry. A notable example is Mol2Vec [10], which learns vector representations of molecular substructures in an unsupervised manner. Building further on this idea, deep sequence-based models such as Smiles2Vec [5] use Recurrent Neural Networks (RNNs) to directly process SMILES strings and learn features optimized for property prediction.

More recently, BERT-based models adapted to molecular data [4, 11, 17, 24, 27, 30] have achieved state-of-the-art results in property prediction, encoding SMILES into fixed-size latent spaces where

cosine similarity reflects sequence similarity. However, these embeddings remain globally unsupervised and do not guarantee alignment with property-based relationships, such as $T_g$ proximity, a limitation we directly address in this work.

## 2.2 PolyBERT

While general-purpose molecular language models like SMILES-BERT [24] and ChemBERTa [4] have demonstrated promising results, most remain trained on generic molecular datasets and do not account for the structural and functional specificity of polymers. This gap has recently been addressed by PolyBERT, a dedicated model tailored to polymeric structures. PolyBERT [11] is a polymer chemical embedding model based on DeBERTa [8] that treats the chemical structure of polymers as a chemical language. The proposed approach surpasses the best available methods for polymer property prediction based on handcrafted fingerprint schemes. It is trained on 100 million hypothetical PSMILES (Polymers SMILES). To the best of our knowledge, PolyBERT is the most recent open-source advancement in polymer language models. In the following, we analyze our novel approach applied to this model.

## 2.3 Sentence transformers

To guide the embedding space toward capturing property-driven similarities, we draw inspiration from methods developed in natural language processing, specifically, Sentence-BERT (SBERT) [18], which aligns embeddings based on semantic closeness rather than surface form. Sentence-BERT is a modification of the pretrained BERT model that employs siamese and triplet network architectures to generate semantically meaningful sentence embeddings, which can be compared using cosine similarity. This approach significantly reduced the time to compute semantic similarity from 65 hours with BERT to just 5 second, while maintaining similar accuracy. SBERT outperformed prior sentence embedding methods on semantic similarity and transfer learning benchmarks upon its release in 2019, [18]. We draw on the core principles of this approach
to design embeddings where vector similarity reflects property relevance rather than structural similarity between SMILES.

## 3 Dual Embedding Method

<figure><img src="image_4.png" /><figcaption>Figure 2: Cosine similarity (in red) of polyBERT embeddings for two polymer groups: (a) polyimides, shown by their repeating units with I denoting the junction, and (b) hypothetical polymers.</figcaption></figure>

From a chemical standpoint, it may seem counterintuitive to claim that structural similarity does not always correspond to similarity in properties, since molecular structure determines material properties. For instance, the glass transition temperature ($T_g$) mainly depends on two physical factors: chain stiffness and the strength of cohesive interactions [28].

However, it is crucial to clarify what is meant by structural similarity, within the PolyBERT latent space, structural similarity is quantified using the cosine similarity between polymer embedding vectors. This metric essentially measures the resemblance between two character strings: the more similar the strings, the closer their cosine similarity is to 1. Yet, this measure carries an inherent bias with respect to property similarity. Small structural modifications—such as replacing a functional group, introducing a double bond instead of a single bond, or shifting a substituent from the meta to the para position—can have minor effects on string similarity, but substantial effects on a property. For example, Fig. 2a illustrates the inequality between structural modifications, cosine similarity, and $T_g$ variations in polyimides. Similarly, Fig. 2b highlights the disproportion between PolyBERT latent space geometry and $T_g$ differences: two polymer pairs may exhibit nearly identical cosine similarity values (0.59 vs. 0.63), while their $T_g$ differences are drastically distinct (263°C vs. 1°C).

Therefore, predicting properties such as $T_g$ directly from structural embeddings is theoretically consistent, but these embeddings do not explicitly encode the structural modifications that drive property changes. Thus, it is essential to exploit both raw and fine-tuned embeddings within predictive models to preserve complementary structural information that may otherwise be lost during fine-tuning. Similar strategies have been successfully implemented in recent studies. For instance, MolPROP [20], which integrates

SMILES syntax with graph-based structural features, demonstrated that simple fine-tuning of a general embedding is often insufficient. Likewise, dual-channel models for drug property prediction [26] effectively learn molecular representations that better capture property variations and fragment semantics. Finally, multi-task generative modeling [13] has shown that training on multiple complementary objectives produces synergistic representations, enhancing the capture of complex structure–property relationships.

<figure><img src="image_5.png" /><figcaption>Figure 3: Dual embedding method for $T_g$ prediction</figcaption></figure>

The proposed Dual Embedding Method, illustrated in Fig. 3, consists of training a glass transition temperature ($T_g$) prediction model using both a standard language model embedding (E) and a fine-tuned counterpart (E') that has been optimized to reflect $T_g$ similarity, thereby capturing the sub-structural patterns that influence variations in $T_g$.

Let A and B, be the embedding vectors of polymers $P_1$ and $P_2$ in E and A' and B' their corresponding embeddings in E'. We define the cosine similarity between two vectors V and W in an embedding space of dimension n as (Eq.1):

$$ \cos(V, W) = \frac{V \cdot W}{\sqrt{\sum_{i=1}^{n} V_i^2} \cdot \sqrt{\sum_{i=1}^{n} W_i^2}} \quad (1) $$

Our objective is to create the embedding E' such that $\cos(A', B')$ is correlated with the difference in $T_g$ values between $P_1$ and $P_2$, while the original $\cos(A, B)$ primarily reflects the syntactic similarity of the polymers' SMILES representations.

### 3.1 Hypothetical polymers dataset

To perform our proof of concept, we used a subset of the open-source PolyBERT dataset [11], which comprises 100 million hypothetical polymers derived from 13,000 experimental polymers data (not publicly available).

This subset contains data on more than 4,000 polymers, including their PSMILES and nine properties, such as the $T_g$, the degradation temperature ($T_d$), the heat capacity ($C_p$), and their electron affinity (Eea). The histogram of the $T_g$ values is provided below in Fig. 4.

### 3.2 Building a dataset for embedding fine-tuning

#### 3.2.1 Embedding fine-tuning dataset constitution.
To avoid bias and ensure fair evaluation, the embedding fine-tuning dataset is either distinct from the test set or drawn as a subset of the training
<figure><img src="image_6.png" /><figcaption>Figure 4: Combined Tg Histograms with Overlap</figcaption></figure>

set. Nandan Thakur and Gurevych [16] demonstrated that with more than 3,000 data pairs, it is possible to reach the performance plateau when fine-tuning a well-structured encoder model. Our experiments confirmed this observation for the PolyBERT model. By selecting 200 polymers whose $T_g$ distribution spans the entire range of the training set, we construct, through pairwise combination, a fine-tuning dataset containing up to $\binom{200}{2} = 19,900$ unique pairs, which is sufficient to enable effective fine-tuning. For each pair, the difference between their $T_g$ values is computed using various metrics. Table 1 illustrates the structure of the fine-tuning dataset, where $(x)$ denotes the difference in $T_g$, and $(\text{Score})$ represents the target similarity between the polymer vectors in the latent space of embedding $E'$.

<table><caption>Table 1: Example of fine-tuning dataset</caption><thead><tr><th>Polymer 1</th><th>Polymer 2</th><th>x</th><th>Score</th></tr></thead><tbody><tr><td>*OCcccc1...*</td><td>*OCCOC...*</td><td>15</td><td>0.8879</td></tr><tr><td>..................</td><td>..................</td><td>....</td><td>....</td></tr><tr><td>*OCOccc1...*</td><td>*OCOcc1...*</td><td>181</td><td>0.1321</td></tr></tbody></table>

### 3.2.2 Definition of difference metrics on polymer $T_g$ pairs.
The most natural way to compute the difference between two elements $a > 0$ and $b > 0$ is to use the absolute Euclidean (Eu) difference. However, other difference metrics can also be considered, such as the logarithmic (ln) difference, which emphasizes proportional rather than absolute differences and helps to relativize variations in $T_g$ values. Similarly, the quadratic (Q) difference provides additional insight by penalizing large discrepancies more heavily. Below are the corresponding metrics (Eq. 2) for (Eu), (ln), and (Q):

$$|a - b|, \quad |\ln(a/b)|, \quad \sqrt{|a^2 - b^2|} \qquad (2)$$

### 3.2.3 Normalizing the difference between $T_g$ values.
The $T_g$ differences vary widely in scale depending on the dataset and the measurement units. To ensure meaningful comparison with cosine similarity, it is necessary to rescale these differences to the interval

$]0, 1[$, such that smaller differences correspond to values closer to 1 (higher similarity), while larger differences approach 0 (lower similarity). Let $x$ denote the difference between two $T_g$ values. To prevent degradation during training, the data are analyzed using quantiles to remove outlier pairs with extreme values of $x$. Once this cleaning step is completed, let $x_m$ denote the maximum $T_g$ difference. We then define two normalization approaches, using a factor $\lambda$ to control the sensitivity of the transformation.

We define the parametric scaling methods **Min-max** and **Exponential** as follows:

$$MM(x) = 1 - (1 - \lambda) \frac{x}{x_m}, \quad Exp(x) = e^{\frac{x}{x_m}} \cdot \ln(\lambda)$$

where $Exp(x_m) = MM(x_m) = \lambda$, with $\lambda < 1$. While **MM** provides a linear mapping, **Exp** penalizes larger $x$ more heavily than smaller ones. Depending on the dataset structure, one approach may be more advantageous than the other.

## 3.3 Embedding finetuning : models and results

<figure><img src="image_7.png" /><figcaption>Figure 5: Fine-tuning Embedding (E) with Cosine Similarity Loss [18]</figcaption></figure>

After constructing the fine-tuning dataset with multiple scores, all using a fixed $\lambda = 0.90$, the BERT-based model is fine-tuned as illustrated in Fig. 5, within a Siamese network. The loss function is the mean squared error between the cosine similarity of the embedding vectors $V$ and $W$ and the target score, which corresponds to the normalized difference of the polymers' $T_g$ values. The fine-tuning is conducted using the pretrained checkpoint kuelumbus/polyBERT over 5 epochs with 10 warm-up steps and a fixed random seed of 42 to ensure reproducibility. We denote this by:

* **PB**: Original PolyBERT model.

* **PB_Eu_MM** and **PB_Eu_Exp**: Fine-tuned PolyBERT using Euclidean difference with Min-Max normalization and Exponential normalization, respectively.

* **PB_1n_MM** and **PB_1n_Exp**: Fine-tuned PolyBERT using logarithmic difference with Min-Max normalization and Exponential normalization, respectively.
* **PB_Q_MM** and **PB_Q_Exp**: Fine-tuned PolyBERT using quadratic difference with Min-Max normalization and Exponential normalization, respectively.

To visualize the effectiveness of our fine-tuning, we formed two groups of polymer pairs ($P_1, P_2$) through filtering: one group contains polymers with similar glass transition temperatures ($T_g$), where $|T_{g1} - T_{g2}| < 1°C$, and the other group contains polymers with significantly different $T_g$ values, where $|T_{g1} - T_{g2}| > 250°C$.

Figure 6 shows the cosine similarity of polymer pairs from both groups in the original PB embedding space and in the fine-tuned embedding spaces PB_Eu_MM and PB_Eu_Exp. In the fine-tuned space, polymer pairs with similar $T_g$ values exhibit cosine similarities close to 1, unlike in the original embedding. Conversely, pairs with different $T_g$ values show similarities approaching 0 in the fine-tuned space, further confirming the effectiveness of the fine-tuning process.

Figure 7 shows the t-SNE projection of polymer vectors in both the PB and PB_Eu_MM embeddings. Two groups of polymer pairs are highlighted: those with similar $T_g$ values, which become closer in the fine-tuned embedding, and those with very different $T_g$ values, which are now placed in diametrically opposite regions. The t-SNE visualization reveals a clear rearrangement of the latent space, indicating that the fine-tuned embeddings capture a different geometry of polymer relationships compared to the original PolyBERT representations.

<figure><img src="image_8.png" /><figcaption>Figure 6: Cosine similarity of polymer pair embeddings from PB and fine-tuned PB (PB_Eu_MM, and PB_Eu_exp), for two groups: $\Delta T_g < 1°C$ and $\Delta T_g > 250°C$.</figcaption></figure>

## 3.4 $T_g$ prediction models

Given $n$ samples with true values $y_i$ and predicted values $\hat{y}_i$, we define:

The Mean Squared Error (MSE): $MSE = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$

The Root Mean Squared Error (RMSE): $RMSE = \sqrt{MSE}$

The Mean Absolute Error (MAE): $MAE = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|$

<figure><img src="image_9.png" /><figcaption>Figure 7: t-SNE of polymer embeddings from PB and the fine-tuned model (PB_Eu_MM). Two groups of polymer pairs are highlighted: $\Delta T_g < 1°C$ and $\Delta T_g > 250°C$.</figcaption></figure>

The Coefficient of Determination ($R^2$): $R^2 = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2}$ where $\bar{y}$ is the mean of the true values.

After obtaining the fine-tuned models ($E'$) from the initial embeddings ($E$), each SMILES string is converted into two 600-dimensional vectors, each corresponding to the structural and $\Delta T_g$-based embeddings. We then design a model that takes these vectors as input and predicts the $T_g$ as output. To integrate these complementary representations, we designed a dual-branch feedforward neural network (FFNN) in which each embedding is processed separately through two dense layers before being merged into a joint latent layer for $T_g$ prediction. This design allows the model to capture both the full chemical structure (from $E$) and the property-based structure (from $E'$), while also enabling an explicit assessment of their individual contributions.

The choice of the number of neurons per layer reflects the complexity of the information to be processed. Since the first layer corresponds to the initial PB structural embedding, it is assigned more neurons per layer in its branch compared to the fine-tuned embedding branch. This modeling strategy, illustrated in Fig. 8, was compared with alternative architectures, and the results indicate that the dual-branch design provides optimal performance across all evaluations.

For the hypothetical polymer dataset, the model was trained with dense architecture of sizes $a = 32$, $b = 8$, and $c = 2$, using a ReLU activation function. Training was performed with a learning rate of 0.0001, a batch size of 16, and optimized using the MAE loss. The dataset was split into 80% training and 20% testing, with a validation split of 20% applied to the training set. The number of epochs is not specified because to avoid overfitting we use Keras early stopping with a patience of 5 to restore the best weights, and a maximum epochs of 1,000. The use of early stopping ensures that hyperparameters such as batch size or learning rate do not exert a
<figure><img src="image_10.png" /><figcaption>Figure 8: $T_g$ prediction model</figcaption></figure>

significant influence on the final performance, as training is halted once convergence is achieved regardless of these variations.

The following table presents the results of $T_g$ prediction, trained on 2,566 samples, validated on 641 samples, and tested on 803 samples. The reported RMSE and $R^2$ values correspond to the test set. In Tab.2, we denote ($E|E'$) as the model built using both embeddings (one per branch) and ($E|E'$) or ($|E'|$) as the model built using only one of the two branches.

<table><caption>Table 2: Ablation study on $T_g$ prediction using different embedding configurations on the hypothetical polymer dataset</caption><thead><tr><th>Models/Embeddings</th><th>R<sup>2</sup></th><th>RMSE</th></tr></thead><tbody><tr><td><b>PB</b></td><td><b>0.893</b></td><td><b>19.31</b></td></tr><tr><td>|PB_Eu_MM</td><td>0.837</td><td>23.91</td></tr><tr><td>|PB_Eu_Exp</td><td>0.853</td><td>22.72</td></tr><tr><td>|PB_Q_MM</td><td>0.870</td><td>21.33</td></tr><tr><td>|PB_Q_Exp</td><td>0.865</td><td>21.73</td></tr><tr><td>|PB_In_MM</td><td>0.846</td><td>23.25</td></tr><tr><td>|PB_In_Exp</td><td>0.873</td><td>21.07</td></tr><tr><td>PB|PB_Eu_MM</td><td>0.910</td><td>17.74</td></tr><tr><td><b>PB|PB_Eu_Exp</b></td><td><b>0.918</b></td><td><b>16.93</b></td></tr><tr><td>PB|PB_Q_MM</td><td>0.917</td><td>17.08</td></tr><tr><td>PB|PB_Q_Exp</td><td>0.917</td><td>17.07</td></tr><tr><td>PB|PB_In_MM</td><td>0.912</td><td>17.53</td></tr><tr><td>PB|PB_In_Exp</td><td>0.910</td><td>17.70</td></tr></tbody></table>

Mean and Std of the test set: 135 and 59 °C

We can observe in Tab.2 that when considering only the branch consisting of a fine-tuned model ($|E'|$), the RMSE values are all worse compared to the original model ($E|PB$). However, the combination of fine-tuned embeddings with the original one results

in a direct improvement in the performance of all models containing a Dual Embedding ($E|E'$). Moreover, all these models exhibit an RMSE performance superior to that of the original model.

From this proof of concept, we conclude that the fine-tuned embedding based on $T_g$ provide complementary information in the Dual Embedding model.

We also conclude that the difference metric does not necessarily have a significant influence. Therefore, moving forward, we consider only the Euclidean difference to quantify differences in $T_g$ values.

# 4 Dual Embedding Applications

## 4.1 Benchmark models

<figure><img src="image_11.png" /><figcaption>Figure 9: Benchmark $T_g$ prediction model</figcaption></figure>

In order to compare our work with existing studies on $T_g$ (°C) prediction, which cover various models and embeddings, we referred to an established benchmark on heterogeneous polymer $T_g$ prediction [21].

This benchmark stands out because it presents 25 different methods for predicting $T_g$ using diverse embeddings, such as the Morgan fingerprint [9], a molecular embedding (ME) based on Word2Vec, molecular graph embeddings, the conversion of polymers into 2D images, and the use of RDKit descriptors. Furthermore, this study combines these embeddings with various models, including RNN, Support Vector Machines (SVM), Feed-Forward Neural Networks (FFNN), 1D and 2D Convolutional Neural Networks (CNN), Graph Convolutional Neural Networks (GCNN), and many others.

Only the benchmark test set, their labels, and the predictions are provided as open source. To enable comparison with this benchmark without access to their training set, we propose as presented in Fig. 9 to use a pretrained model based on the architecture and data described in Section 3.4, which we fine-tune on a portion of the benchmark test set.

To ensure a fair evaluation, we performed a random split of the benchmark test set into fine-tuning (30% (170)) and test subsets (70% (396)), without stratification based on polymer family, chemical structure, or property distribution. Consequently, the fine-tuning and test sets are fully disjoint at the sample level, avoiding any direct data leakage. The Kolmogorov-Smirnov test revealed no significant difference between the two distributions ($D = 0.067$, $p = 0.63$), indicating that neither set was favored.
### 4.1.1 Embedding fine-tuning.
To fine-tune our embedding while ensuring fair evaluation, we select 100 samples each from the hypothetical polymer training set and the benchmark fine-tuning set, in such a way that their $T_g$ distribution spans the entire range of the dataset, resulting in 200 representative data. This is essential because we need to adjust our embedding model to the two types of data, which are considerably different. Once this is done, we apply, as in our proof of concept, a pairwise combination of the polymers to create the fine-tuning dataset for the embedding.

We note the following: PB_Merge_MM and PB_Merge_Exp refer to PolyBERT fine-tuned on 200 merged data, using euclidean difference with Min-Max and Exponential normalization, respectively.

### 4.1.2 $T_g$ model prediction fine-tuning.
The overall $T_g$ prediction model is illustrated in Fig. 9, where a pretrained model is constructed following the approach presented in Section 3.4, except that it utilizes the fine-tuned embeddings PB_Merge_MM or PB_Merge_Exp. The pretrained models are then fine-tuned on the benchmark fine-tuning subset (30%) and evaluated on the remaining (70%). To optimize prediction performance, the only modifications applied to all models were the use of mean squared error as the loss function (loss = MSE) and a reduced learning rate ($lr = 0.001$).

Table 3 summarizes the results, where BM refers to the best model among the 25 proposed in [21], and BM_70 denotes the performance of BM on the selected 70% test subset.

<table><caption>Table 3: Ablation study on $T_g$ prediction using different embedding strategies on the benchmark polymer dataset</caption><thead><tr><th>Models/Embeddings</th><th>R<sup>2</sup></th><th>RMSE</th></tr></thead><tbody><tr><td>BM [21]</td><td>0.64</td><td>66.58</td></tr><tr><td>BM_70</td><td>0.63</td><td>64.46</td></tr><tr><td>PB</td><td>0.64</td><td>64.20</td></tr><tr><td>|PB_Merge_Exp</td><td>0.62</td><td>66.21</td></tr><tr><td>|PB_Merge_MM</td><td>0.64</td><td>63.76</td></tr><tr><td>PB|PB_Merge_Exp</td><td>0.65</td><td>63.66</td></tr><tr><td><b>PB|PB_Merge_MM</b></td><td><b>0.68</b></td><td><b>60.76</b></td></tr></tbody></table>

Mean and Std of the test set: 98 and 108 °C

We observe that PolyBERT alone already performs competitively, slightly outperforming the best model (BM) from the benchmark, which uses a Word2Vec-based molecular embedding combined with a Deep Neural Network (DNN). While the training datasets are not identical, the comparison remains informative, as both approaches are evaluated on the same test set (BM_70). Nevertheless, the core argument remains valid: the dual embedding approach outperforms the single-embedding model, with the best results obtained when using Min-Max normalization.

## 4.2 Polyimides models

Polyimides are a class of polymers characterized by the presence of at least one *imide linkage*, which can appear either as part of an open-chain structure or as a heterocyclic unit integrated into the polymer backbone. An imide linkage is typically represented as: 'R-CO-NH-CO-R', where two carbonyl groups (CO) are bonded to the same nitrogen atom (NH).

Volgin et al. [23] develop the prediction of polyimide glass transition temperature $T_g$ (°K), focusing on GCNN. They also provide, as open-source data, two datasets: a synthetic dataset containing more than 6 million data and an experimental dataset with 214 data.

They developed multiple approaches, in this study we consider three: a first predictive model built on 1,000 synthetic data (GCNN/Syn), a second one based on experimental data (GCNN/Exp), and a third one consisting of the fine-tuning of (GCNN/Syn) on the experimental data, referred to as (GCNN/Poly). All these models are built using the 10-fold cross-validation technique.

In order to perform our dual-embedding approach, we selected 200 unique data whose $T_g$ distribution spans the entire range of the synthetic dataset for embedding fine-tuning. We denote by PB_PI_MM and PB_PI_Exp the PolyBERT model fine-tuned on polyimide synthetic data using euclidean difference with Min-Max and Exponential normalization, respectively.

We constructed the corresponding dual embedding $T_g$ prediction model as described in Section 3.4, using the following modified parameters for optimization: We use $lr = 0.001$ for models trained on synthetic data (syn) or pretrained on synthetic data and then trained on experimental data (syn+exp), and $lr = 0.0001$ for models trained only on experimental data (exp). The results provided in Tab. 4 represent the mean ± standard deviation (Std) from the 10-fold cross-validation.

As presented in Tab. 4, in terms of MAE, the model built by considering only the PolyBERT branch already surpasses the GCNN approach developed in [23]. Furthermore, all the dual embedding approaches perform better on all three datasets, with a significant improvement observed in the $R^2$ and RMSE values on both the synthetic and experimental datasets, with up to 20% reduction in RMSE (from 23.65 to 18.54).

## 4.3 Homopolymers models

Homopolymers are polymers whose structure is made up of repeating units of a single kind of monomer. Casanola-Martin et al. [3] address the prediction of homopolymers $Log(T_g)$ using quantitative structure-property relationship (QSPR) descriptors. They utilized various machine learning models, including multiple linear regression (MLR), and support vector machine (SVM). The open-source dataset contains 683 samples for training and 219 for testing.

We extracted 200 data from the training set to perform PolyBERT fine-tuning. We denote by PB_Hm_MM and PB_Hm_Exp the PolyBERT model fine-tuned on homopolymer data using euclidean difference with Min-Max and Exponential normalization, respectively.

We constructed the corresponding dual embedding $Log(T_g)$ prediction model as indicated in Section 3.4 with the following modified parameters for optimization: $lr = 0.02$, and $batch=13$.

Table 5 presents the results, which we explain as follows. One of the particularities of this study is that the prediction is performed on $Log(T_g)$ rather than $T_g$ directly, this is done to ensure comparability, as the publication with which we compare developed their model using $log(T_g)$ to optimize prediction performance.

The PolyBERT model alone achieves the lowest performance. However, when incorporated into the dual embedding process, its performance improves significantly, surpassing QSPR-MLR and approaching that of QSPR-SVM in terms of $R^2$ on this dataset. These
<table><caption>Table 4: Ablation study on $T_g$ prediction using different embedding configurations on polyimides dataset</caption><thead><tr><th>Models</th><th>Data</th><th>R<sup>2</sup></th><th>RMSE</th><th>MAE</th></tr></thead><tbody><tr><td>GCNN/Syn [23]</td><td>syn</td><td>/</td><td>/</td><td>14.9</td></tr><tr><td>PB</td><td>syn</td><td>0.67 ± 0.1</td><td>23.65 ± 3.2</td><td>4.15 ± 0.25</td></tr><tr><td>|PB_PI_MM</td><td>syn</td><td>0.77 ± 0.06</td><td>20.19 ± 3.3</td><td>3.77 ± 0.25</td></tr><tr><td>|PB_PI_Exp</td><td>syn</td><td>0.71 ± 0.07</td><td>22.42 ± 2.85</td><td>4.07 ± 0.19</td></tr><tr><td>PB|PB_PI_MM</td><td>syn</td><td>0.76 ± 0.1</td><td>19.69 ± 4.0</td><td>3.69 ± 0.38</td></tr><tr><td><b>PB|PB_PI_Exp</b></td><td><b>syn</b></td><td><b>0.80 ± 0.05</b></td><td><b>18.54 ± 2.4</b></td><td><b>3.65 ± 0.19</b></td></tr><tr><td>GCNN/exp [23]</td><td>exp</td><td>/</td><td>/</td><td>28.1 ± 5.7</td></tr><tr><td>PB</td><td>exp</td><td>0.56 ± 0.18</td><td>33.13 ± 8.8</td><td>4.85 ± 0.64</td></tr><tr><td>|PB_PI_MM</td><td>exp</td><td>0.31 ± 0.31</td><td>41.74 ± 11.87</td><td>5.30 ± 0.75</td></tr><tr><td>|PB_PI_Exp</td><td>exp</td><td>0.50 ± 0.25</td><td>35.59 ± 11.33</td><td>4.93 ± 0.67</td></tr><tr><td><b>PB|PB_PI_MM</b></td><td><b>exp</b></td><td><b>0.74 ± 0.09</b></td><td><b>26.04 ± 8.0</b></td><td><b>4.13 ± 0.51</b></td></tr><tr><td>PB|PB_PI_Exp</td><td>exp</td><td>0.70 ± 0.09</td><td>27.78 ± 7.9</td><td>4.27 ± 0.49</td></tr><tr><td>GCNN/Poly [23]</td><td>syn+exp</td><td>/</td><td>/</td><td>22.5 ± 5.1</td></tr><tr><td>PB</td><td>syn+exp</td><td>0.71 ± 0.10</td><td>27.36 ± 9.0</td><td>4.39 ± 0.64</td></tr><tr><td>|PB_PI_MM</td><td>syn+exp</td><td>0.58 ± 0.21</td><td>31.91 ± 7.71</td><td>4.64 ± 0.62</td></tr><tr><td>|PB_PI_Exp</td><td>syn+exp</td><td>0.65 ± 0.13</td><td>29.31 ± 6.68</td><td>4.44 ± 0.57</td></tr><tr><td>PB|PB_PI_MM</td><td>syn+exp</td><td>0.74 ± 0.10</td><td>25.44 ± 7.6</td><td>4.16 ± 0.54</td></tr><tr><td><b>PB|PB_PI_Exp</b></td><td><b>syn+exp</b></td><td><b>0.74 ± 0.13</b></td><td><b>24.82 ± 7.0</b></td><td><b>4.13 ± 0.54</b></td></tr></tbody></table>

Mean and Std of the synthetic data: 500 and 42 °K

Mean and Std of the experimental data: 546 and 53 °K

<table><caption>Table 5: Ablation study on Log($T_g$) prediction using different embedding configurations on homopolymers dataset</caption><thead><tr><th>Models</th><th>R<sup>2</sup></th><th>RMSE</th></tr></thead><tbody><tr><td>QSPR-SVM [3]</td><td>0.77</td><td>/</td></tr><tr><td>QSPR-MLR [3]</td><td>0.617</td><td>0.086</td></tr><tr><td>PB</td><td>0.547</td><td>0.093</td></tr><tr><td>|PB_Hm_Exp</td><td>0.570</td><td>0.091</td></tr><tr><td>|PB_Hm_MM</td><td>0.563</td><td>0.091</td></tr><tr><td>PB|PB_Hm_Exp</td><td>0.661</td><td>0.080</td></tr><tr><td><b>PB|PB_Hm_MM</b></td><td><b>0.667</b></td><td><b>0.080</b></td></tr></tbody></table>

Mean and Std of the test set: 2.552 and 0.139

results demonstrate that the dual embedding approach consistently outperforms the original embedding, highlighting the effectiveness of property-aware fine-tuning.

# 5 Additional Analyses

## 5.1 Other properties predictions with the Dual Embedding

While our approach has shown promising results in predicting $T_g$ in previous models, it is crucial to assess whether the same methodology can generalize to other polymer properties. Given that the $T_g$ prediction relies heavily on molecular structure and interactions, it stands to reason that other properties may also benefit from the dual embedding approach. To investigate this, we use the hypothetical polymers subset (4,000) with three additional properties introduced in Sec. 3.1, namely: degradation temperature

$T_d$ (°K), heat capacity $C_p$ (J g⁻¹ K⁻¹), and electron affinity $E_{ea}$ (eV). We denote by:

* **PB_Cp_MM** and **PB_Cp_Exp** the PolyBERT model fine-tuned on 200 hypothetical polymers data using euclidean difference on $C_p$ with Min-Max and Exponential normalization, respectively.

* **PB_Eea_MM** and **PB_Eea_Exp** the PolyBERT model fine-tuned on 200 hypothetical polymers data using euclidean difference on $E_{ea}$ with Min-Max and Exponential normalization, respectively.

* **PB_Td_MM** and **PB_Td_Exp** the PolyBERT model fine-tuned on 200 hypothetical polymers data using euclidean difference on $T_d$ with Min-Max and Exponential normalization, respectively.

For these new properties, we applied the same dual embedding methodology as described in Section 3.4, adjusting the hyperparameters for the $C_p$ and $E_{ea}$ prediction models. More specifically, we set the learning rate to $lr = 0.01$ for the $C_p$ models and adjusted the batch size for the $E_{ea}$ models to batch = 3 to optimize performance. The corresponding results, along with the mean and standard deviation (Std) of the test set, are presented in Tab. 6.

Tab. 6 shows that the dual-embedding framework performs well for $T_d$, provides a minor improvement for $C_p$, and remains constant for $E_{ea}$. To understand the limited performance for $C_p$ and $E_{ea}$, further analysis and additional experiments are necessary to verify the generality of this observation, as was done for $T_g$. We therefore consider this a direction for future studies. However, for this specific dataset configuration, the prediction using a single embedding already appears to have reached a performance plateau for $E_{ea}$,
<table><caption>**Table 6:** Ablation study on Td (°K), Cp (J/gK), and Eea (eV) prediction using different embedding configurations on hypothetical polymers dataset</caption><thead><tr><th>Models</th><th>R<sup>2</sup></th><th>RMSE</th></tr></thead><tbody><tr><td colspan="3"><em>Td (°K), Mean = 652, Std = 50</em></td></tr><tr><td>PB</td><td>0.64</td><td>30.46</td></tr><tr><td>|PB_Td_MM</td><td>0.67</td><td>29.34</td></tr><tr><td>|PB_Td_Exp</td><td>0.67</td><td>29.06</td></tr><tr><td><strong>PB|PB_Td_MM</strong></td><td><strong>0.71</strong></td><td><strong>26.97</strong></td></tr><tr><td>PB|PB_Td_Exp</td><td>0.71</td><td>27.05</td></tr><tr><td colspan="3"><em>Cp (J/gK), Mean = 1.31, Std = 0.128</em></td></tr><tr><td>PB</td><td>0.85</td><td>0.05</td></tr><tr><td>|PB_Cp_MM</td><td>0.76</td><td>0.06</td></tr><tr><td>|PB_Cp_Exp</td><td>0.68</td><td>0.07</td></tr><tr><td><strong>PB|PB_Cp_MM</strong></td><td><strong>0.87</strong></td><td><strong>0.05</strong></td></tr><tr><td>PB|PB_Cp_Exp</td><td>0.85</td><td>0.05</td></tr><tr><td colspan="3"><em>Eea (eV), Mean = 1.90, Std = 0.46</em></td></tr><tr><td>PB</td><td><strong>0.92</strong></td><td><strong>0.13</strong></td></tr><tr><td>|PB_Eea_MM</td><td>0.87</td><td>0.16</td></tr><tr><td>|PB_Eea_Exp</td><td>0.84</td><td>0.18</td></tr><tr><td><strong>PB|PB_Eea_MM</strong></td><td><strong>0.92</strong></td><td><strong>0.13</strong></td></tr><tr><td>PB|PB_Eea_Exp</td><td>0.91</td><td>0.14</td></tr></tbody></table>

with R² = 0.92. Nevertheless, implementing the dual-embedding approach remains worthwhile.

## 5.2 T<sub>g</sub> prediction by finetuning embeddings with properties different from T<sub>g</sub>

Another interesting idea we wanted to verify is whether fine-tuning the embedding on a property different from T<sub>g</sub> captures relevant complementary information for the dual embedding approach on T<sub>g</sub> prediction. This is most likely the case for properties that are not correlated with T<sub>g</sub> across a wide range of polymers, including T<sub>d</sub> (thermal degradation temperature), C<sub>p</sub> (heat capacity at constant pressure), and E<sub>ea</sub> (electron affinity), which reflect distinct physicochemical dimensions.

To verify our hypothesis, we applied the same methodology as in the proof of concept (Sec. 3), using different fine-tuned embeddings centered around the properties T<sub>d</sub>, C<sub>p</sub>, and E<sub>ea</sub>. The results of this analysis presented in Tab. 7 show that all dual embeddings based on **properties different from T<sub>g</sub>** outperform the native embedding in predicting T<sub>g</sub>, although they do not perform as well as the embedding fine-tuned specifically on T<sub>g</sub> values. This demonstrates that fine-tuning language model embeddings on a specific property helps capture relevant patterns that enhance the prediction of T<sub>g</sub>.

We also tested models incorporating more than two embeddings, combining PolyBERT with all fine-tuned embeddings on properties other than T<sub>g</sub>. This expanded the architecture from two to four branches, improving performance. However, it still did not outperform the model using the T<sub>g</sub>-fine-tuned embedding. Given the additional computational cost, this approach may not be the most efficient for T<sub>g</sub> prediction.

<table><caption>**Table 7:** T<sub>g</sub> prediction with fine-tuned embedding on T<sub>d</sub>, C<sub>p</sub> and E<sub>ea</sub></caption><thead><tr><th>Models</th><th>R<sup>2</sup></th><th>RMSE</th></tr></thead><tbody><tr><td>PB</td><td>0.893</td><td>19.31</td></tr><tr><td>PB|PB_Eu_MM</td><td>0.910</td><td>17.74</td></tr><tr><td><strong>PB|PB_Eu_Exp</strong></td><td><strong>0.918</strong></td><td><strong>16.93</strong></td></tr><tr><td>PB|PB_Td_MM</td><td>0.900</td><td>18.71</td></tr><tr><td>PB|PB_Td_Exp</td><td>0.902</td><td>18.51</td></tr><tr><td>PB|PB_Cp_MM</td><td>0.905</td><td>18.26</td></tr><tr><td>PB|PB_Cp_Exp</td><td>0.906</td><td>18.13</td></tr><tr><td>PB|PB_Eea_MM</td><td>0.912</td><td>17.49</td></tr><tr><td>PB|PB_Eea_Exp</td><td>0.905</td><td>18.24</td></tr><tr><td>PB|PB_Td|PB_Cp|PB_Eea (MM)</td><td>0.913</td><td>17.43</td></tr><tr><td>PB|PB_Td|PB_Cp|PB_Eea (Exp)</td><td>0.913</td><td>17.41</td></tr></tbody></table>

Mean and Std of the test set: 135 and 59 °C

# 6 Discussion

We have presented a dual embedding method which leverages two embeddings: a standard language model embedding trained on polymer SMILES, and a fine-tuned version where vectors are organized according to the proximity of T<sub>g</sub> values. Our results demonstrate consistent improvements across heterogeneous polymer datasets, as well as for specialized families such as polyimides and homopolymers, with reductions in RMSE of up to 20% in the best case. This highlights the potential of the proposed property-aware fine-tuning.

The effectiveness of the dual embedding strategy lies in the complementarity of the information captured by each branch. While the structural embedding preserves global molecular information, the fine-tuned embedding emphasizes property-relevant dimensions. Their combination yields more accurate predictions than either embedding alone.

Furthermore, extending this framework to other physicochemical properties shows that the model has the potential to improve prediction, as observed for the degradation temperature (T<sub>d</sub>), although performance stagnated for the heat capacity (C<sub>p</sub>) and electron affinity (E<sub>ea</sub>). A complete study similar to the one carried out for T<sub>g</sub>, along with experiments on multiple datasets, would provide a more global understanding and conclusion.

**Limitations.** While the performance gain may appear modest in some cases, it is significant in the context of polymer property prediction, where even small improvements are valuable. Such accuracy gains are particularly important for generative polymer design with reinforcement learning, where the calibration of reward functions depends critically on precise property prediction [29].

The dual embedding framework increases computational cost compared to single-embedding models, as fine-tuning must be performed separately for each property. This additional complexity remains manageable, but it highlights a trade-off between performance and efficiency. Moreover, generalization to very small datasets remains challenging, since a portion of the training data must be used for fine-tuning. Indeed, we cannot guarantee the effectiveness of the fine-tuning procedure with fewer than 200 polymer samples; therefore, a specific analysis would be required.
*Future directions.* Future work could focus on integrating explicit physico-chemical descriptors—such as chain rigidity, cohesive energy density, or bonding energy—into the learning process to better capture the molecular mechanisms governing $T_g$. Incorporating uncertainty quantification would also allow the model to account for experimental variability in polymer property measurements. Moreover, extending the approach to multi-task or multi-property training (e.g., $T_g$, $T_d$, $C_p$) could yield a unified latent representation linking structure and thermodynamic behavior. Finally, improving model interpretability by associating latent dimensions with structural motifs and validating predicted trends through new experiments represents a promising step forward.

# 7 Conclusion

In this work, we introduced a dual embedding framework that combines a standard polymer embedding with a fine-tuned counterpart aligned with $T_g$ similarity. This property-aware representation consistently outperforms conventional embeddings across heterogeneous polymer datasets as well as for specialized families such as polyimides and homopolymers, achieving substantial reductions in prediction error and demonstrating the benefits of embedding physicochemical information directly into the latent space.

Beyond improving property prediction, our approach opens promising avenues for materials discovery. Embedding spaces fine-tuned on specific properties could provide a foundation for inverse design, enabling the generation of novel polymer structures tailored to target properties. Multi-property fine-tuning strategies may further enhance flexibility, while coupling dual embeddings with generative models, such as transformer-based SMILES generators [7], could pave the way toward fully automated polymer design systems guided by property specifications.

# 8 Declarations

## 8.1 Data and Software Availability

The code and data used in this article are available on GitHub.

## 8.2 Acknowledgments

This work benefited from the state aid managed by the National Research Agency under the France 2030 program, bearing the reference ANR-22-PEXD-0004 [1].

## 8.3 Author Contributions

This work was made possible through a close collaboration between the LIRIS and IMP research teams. The LIRIS team (A.T., V.E., and J.-M.P.) contributed to the design of the machine learning methodologies and the analysis of model behavior. The IMP team (A.T., S.P., J.D.-R., and J.-F.G.) provided expertise in polymer science and contributed to the interpretation of the predicted results. All authors participated in the manuscript preparation and provided valuable feedback throughout the study.

## 8.4 Conflict of Interest Statement

The authors declare no competing financial interests.

# References

[1] [n. d.]. *Projet AMETHYST, France 2030 PEPR DIADEM*. https://www.pepr-diadem.fr/projet/amethyst/

[2] Nagababu Andraju, Greg W. Curtzwiler, Yun Ji, Evguenii Kozliak, and Prakash Ranganathan. 2020. Machine-Learning-Based Predictions of Polymer and Post-consumer Recycled Polymer Properties: A Comprehensive Review. *ACS Applied Materials and Interfaces* (September 2020). doi:10.1021/acsami.2c08301

[3] Gerardo M. Casanola-Martin, Anas Karuth, Hai Pham-The, Humbert González-Diaz, Dean C. Webster, and Bakhtiyor Rasulev. 2024. Machine learning analysis of a large set of homopolymers to predict glass transition temperatures. (2024). doi:10.1038/s42004-024-01305-0

[4] S. Chithrananda, G. Grand, and B. Ramsundar. 2020. Chemberta: large-scale self-supervised pretraining for molecular property prediction. *arXiv preprint arXiv:2010.09885* (2020).

[5] Garrett B. Goh, Nathan O. Hodas, Charles Siegel, and Abhinav Vishnu. 2017. SMILES2Vec: An Interpretable General-Purpose Deep Neural Network for Predicting Chemical Properties. *arXiv abs/1712.02034* (2017).

[6] Subhasish Goswami, Rajdeep Ghosh, Arohan Neog, and Bitopan Das. 2021. Deep learning based approach for prediction of glass transition temperature in polymers. *Materials Today: Proceedings* (2021). doi:10.1016/j.matpr.2021.02.730

[7] Z.-Y. Sun H. Qiu. 2024. On-demand reverse design of polymers with PolyTAO. (2024). doi:10.1038/s41524-024-01466-5

[8] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. DeBERTa: Decoding-enhanced BERT with Disentangled Attention. *arXiv preprint arXiv:2006.03654* (2020). doi:10.48550/arXiv.2006.03654

[9] Morgan HL. 1965. The generation of a unique machine description for chemical structures—a technique developed at chemical abstracts service. (1965). doi:10.1021/c160017a018

[10] Sabrina Jaeger, Simone Fulle, and Samo Turk. 2018. Mol2vec: Unsupervised Machine Learning Approach with Chemical Intuition. *Journal of Chemical Information and Modeling* 58, 1 (2018), 27–35. doi:10.1021/acs.jcim.7b00616

[11] Christopher Kuen Kenneth and Rampi Ramprasad. 2023. polyBERT: a chemical language model to enable fully machine-driven ultrafast polymer informatics. *Nature* (2023). doi:10.1038/s41467-023-39868-6

[12] Greg Landrum. 2013. RDKit: Open-source cheminformatics. (2013). https://www.rdkit.org/.

[13] Rachel K. Luu, Marcin Wysokowski, and Markus J. Buehler. 2023. Generative Discovery of Novel Chemical Designs using Diffusion Modeling and Transformer Deep Neural Networks with Application to Deep Eutectic Solvents. *Applied Physics Letters* 122, 23 (2023), 234103. doi:10.1063/5.0155890

[14] Luis A. Miccio and Gustavo A. Schwartz. 2020. From chemical structure to quantitative polymer properties prediction through convolutional neural networks. *Polymer* 202 (2020), 122341. doi:10.1016/j.polymer.2020.122341

[15] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. *Advances in Neural Information Processing Systems (NeurIPS)* 26 (2013). https://arxiv.org/abs/1310.4546

[16] Johannes Daxenberger Nandan Thakur, Nils Reimers and Iryna Gurevych. 2021. Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks. (2021). doi:abs/2010.08240

[17] Haoke Qiu, Lunyang Liu, Xuepeng Qiu, Xuemin Dai, Xiangling Ji, and Zhao-Yan Sun. 2024. PolyNC: a natural and chemical language model for the prediction of unified polymer properties. *Chemical Science* 15 (2024), 534–544. doi:10.1039/D3SC05079C

[18] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. (2019). doi:10.48550/arXiv.1908.10084

[19] David Rogers and Mathew Hahn. 2010. Extended-Connectivity Fingerprints. *Journal of Chemical Information and Modeling* 50, 5 (2010), 742–754. doi:10.1021/ci100050t

[20] Zachary A. Rollins, Alan C. Cheng, and Essam Metwally. 2024. MolPROP: Molecular Property Prediction with Multimodal Language and Graph Fusion. *Journal of Cheminformatics* 16 (2024), 56. doi:10.1186/s13321-024-00846-9

[21] Lei Tao, Vikas Varshney, and Ying Li. 2021. Benchmarking Machine Learning Models for Polymer Informatics: An Example of Glass Transition Temperature. *J Chem Inf Model.* (2021). doi:10.1021/acs.jcim.1c01031

[22] Huan Doan Tran, Chiho Kim, Lihua Chen, Anand Chandrasekaran, Rohit Batra, Shruti Venkatram, Deepak Kamal, Jordan P. Lightstone, Rishi Gurnani, Pranav Shetty, Manav Ramprasad, Julia Laws, Madeline Shelton, and Rampi Ramprasad. 2020. Machine-learning predictions of polymer properties with Polymer Genome featured. *Journal of Applied Physics* 128, 17 (2020), 171104. doi:10.1063/5.0023759

[23] Igor V. Volgin, Pavel A. Batyr, Andrey V. Matseevich, Alexey Yu. Dobrovskiy, Maria V. Andreeva, Victor M. Nazarychev, Sergey V. Larin, Mikhail Ya. Goikhman, Yury V. Vizilter, Andrey A. Askadskii, and Sergey V. Lyulin. 2022. Machine Learning with Enormous "Synthetic" Data Sets: Predicting Glass Transition Temperature of Polyimides Using Graph Convolutional Neural Networks. (2022). doi:10.1021/acsomega.2c04649
[24] Sheng Wang, Yuzhi Guo, Yuhong Wang, Hongmao Sun, and Junzhou Huang. 2019. SMILES-BERT: Large Scale Unsupervised Pre-Training for Molecular Property Prediction. (2019). doi:10.1145/3307339.3342186

[25] David Weininger. 1988. SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules. *Journal of Chemical Information and Modeling* 28, 1 (1988), 31–36. doi:10.1021/ci00057a005

[26] Yue Wu, Xinran Ni, Zhihao Wang, and Weike Feng. 2023. Enhancing Drug Property Prediction with Dual-Channel Transfer Learning Based on Molecular Fragments. *BMC Bioinformatics* 24 (2023), 293. doi:10.1186/s12859-023-05413-x

[27] Changwen Xu, Yuyang Wang, and Amir Barati Farimani. 2023. TransPolymer: a Transformer-based language model for polymer property predictions. npj

*Computational Materials* 9, 1 (2023), 105. doi:10.1038/s41524-023-01016-5

[28] Wen-Sheng Xu, Jack F. Douglas, and Zhao-Yan Sun. 2021. Polymer Glass Formation: Role of Activation Free Energy, Configurational Entropy, and Collective Motion. *Macromolecules* 54, 5 (2021), 3001–3016. doi:10.1021/acs.macromol.0c02740

[29] Tianle Yue, Lei Tao, Vikas Varshney, and Ying Li. 2025. Benchmarking study of deep generative models for inverse polymer design. *Digital Discovery* 4, 4 (2025), 910–926. doi:10.1039/D4DD00395K

[30] X. Zheng and Y. Tomiura. 2024. A BERT-based pretraining model for extracting molecular structural information from a SMILES sequence. *Journal of Cheminformatics* 16, 1 (2024), 71. doi:10.1186/s13321-024-00848-7