# A Deep Learning-Based Approach for Predicting Michaelis Constants from Enzymatic Reactions
## 3 Materials and Methods
### Enzyme Sequence and Enzyme Reaction Feature Extraction Modules  
In biological reactions, enzyme sequences and enzymatic reactions contain complex, long-distance dependencies and contextual patterns between amino acids and atoms. This complexity makes it difficult for traditional methods to capture all the relevant information. Pre-trained large language models were trained on large-scale biological sequences through self-supervised learning, which possess strong feature representation abilities. These models can efficiently identify key information and improve the accuracy of feature expression to provide biologically meaningful representations for downstream prediction tasks. Among these, the pre-trained model BERT uses the Transformer encoder architecture and is pre-trained on a vast amount of unsupervised prior data [40]. This enhances its ability to represent enzyme sequences and enzymatic reactions, which provides strong \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Organisms** & \begin{tabular}{c} **Downloaded** \\ **Samples** \\ \end{tabular} & \begin{tabular}{c} **Processed** \\ **Samples** \\ \end{tabular} & \
\begin{tabular}{c} **Training** \\ **Samples** \\ \end{tabular} &
\begin{tabular}{c} **Test Samples** \\ \end{tabular} \\ \hline _H. sapiens_ & 10,749 & 7160 & 5728 & 1432 \\ _E. coli_ & 5573 & 2962 & 2369 & 593 \\ Total & 16,322 & 10,122 & 8097 & 2025 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Number of data samples. Figure 2: Data processing steps for _E. coli_ and _H. sapiens_. support for the prediction of enzyme kinetic parameters. As shown in Figure 3A, the main structure of BERT consists of N encoder layers, each of which is composed of a multi-head attention mechanism, a normalization layer, and a feed-forward layer. For any input matrix \(\mathbf{X}\in\mathbb{R}^{L\times F}\), as shown in Figure 3B, the multi-head attention mechanism in the \(m\)-th encoder captures the contextual relationships of the sequence based on \(\mathbf{Q}\in\mathbb{R}^{L\times H}\), \(\mathbf{K}\in\mathbb{R}^{L\times H}\), \(\mathbf{V}\in\mathbb{R}^{L\times H}\) of each input token. The feature matrices obtained from multiple heads are concatenated and processed through residual connections to obtain \(\mathbf{X}_{m}\). Furthermore, the normalized feature matrix is fed into the feed forward neural (FFL) network. After residual connection and normalization, the output matrix \(\mathbf{X}_{e}\) is obtained. As shown in Figure 3C, FFL represents the feed-forward neural network. This operation consists of Linear, ReLU, and Linear layers. The details of the encoder layer are provided in Supplementary Note S1. The FFL operator is defined as follows:  
\[\text{FFL}=\text{Linear}(\text{ReLU}(\text{Linear}(\cdot))). \tag{1}\]  
To build the ESM-2 model, the Meta-AI research team [29] used a 33-layer transformer encoder structure and treated the output matrix of the final layer as the representation of the protein. The model consists of approximately 650 million parameters and is trained in an unsupervised manner using around 50 million protein amino acid sequences from the UR50D database. This allows the ESM-2 model to effectively learn the internal relationships of amino acid sequences. The training method involves randomly masking a portion of the amino acid sites in the input protein sequence and using the built ESM-2 model to predict the unknown amino acids. In this paper, as shown in Figure 1B, we use the model (ESM2_t33_650M_UR50D) as the enzyme sequence feature extractor for enzyme-catalyzed reactions to obtain high-quality feature representations \(\mathbf{X}_{sy}\in\mathbb{R}^{L_{s}\times D}\) of the first 400 amino acids in the enzyme sequence. Similarly, the RXNFP [28] pre-trained model consists of multiple transformer encoders and uses the vector of the first token from the model output as the representation of the enzyme-catalyzed reaction SMILES string.