# A Formula-Structure-Embedding-Based GNN for Formula Equivalence Determination
## 4 Experiments and Analysis
### Experimental Results  
To demonstrate the training process and convergence behavior of the proposed model, the training loss curve in Figure 10 is analyzed: the curve exhibits a consistent downward trend, indicating that the model is effectively minimizing the objective function; during the initial stages of training, the loss decreases rapidly, suggesting a quick learning from the training data by the model, while as the number of epochs increases, the rate of decrease in the loss slows down and gradually stabilizes, reflecting that the model parameters are approaching an optimal solution. This steady decline and plateau in the loss curve provide empirical evidence that the training process is both stable and effective, ensuring the reliability of the learned representations. To demonstrate the superiority of our proposed model, comparative experiments among the pre-trained Bert model, the baseline model SimGNN, the GNN model NAGSL, and our proposed model FSCMGNN are conducted. As shown in Figure 11. The accuracy of the Bert model is 5.29%, 7.04%, and 13.31% lower than that of NAGSL, SimGNN, and FSCMGNN, respectively. This performance gap can be attributed to the nature of the models and the data: Bert, being optimized for natural language processing, excels at capturing semantic relationships in linearly structured text. However, when dealing with formulas represented as ASTs, its capabilities are limited. In contrast, GNNs are inherently better suited for such structure representations, as they operate directly on graph structures and can effectively aggregate both local and global structural information through message-passing mechanisms. These findings validate the suitability and effectiveness of GNN-based models for formula equivalence determination tasks. To verify the effectiveness of each structural pre-processing module, we introduced Formula Structure Embedding (FSE) and Standardization (SD) operations in the Bert and SimGNN models, respectively, and conducted comparative experiments. The experimental results show that with only the introduction of formula structure embedding module, all indicators of SimGNN model have been improved, among which the F1 value has increased by 2.62%. Furthermore, after adding standardized operations on this basis, the F1 value increased by 2.1% again (see in Table 3). Similarly, the Bert model also shows some improvement after incorporating the same module, but overall performance still lags significantly behind the SimGNN model that introduced the same module. These  
Figure 10: Training loss curve of FSCMGNN. results indicate that pre-processing structure information has a positive effect on improving model performance, especially in the graph neural network architecture. FSE and SD can more fully explore the structural and semantic features of formulas, further confirming the significant advantages of graph neural network-based models in mathematical formula equivalence determination. To verify the effectiveness of our optimization modules, we conducted an ablation experiment based on the baseline model SimGNN, with results shown in Figure 12. Specifically, FSCMGNN integrates multi-level node feature embedding and several optimisation modules. To examine the impact of each component, we designed different variants: one without formula structure embedding (FSCMGNN-ff), one without the node-level similarity module (FSCMGNN-node), one without the similarity score calculation module (FSCMGNN-score), and one without standardisation processing (FSCMGNN-sd). As the results indicate, the formula structure embedding module contributes the most significantly to the overall performance, demonstrating that effectively incorporating structural information of formulas can greatly enhance similarity measurement. Standardization also plays a notable role by reducing redundant information and improving the consistency of feature representations, thereby facilitating better learning.