# A Knowledge and Semantic Fusion Method for Automatic Geometry Problem Understanding
## 6 Experiment and Application
### Experiment and Analysis
#### 6.1.2 Application Verification Test and Error Analysis
Evaluation: Performance was assessed through manual statistics and by calculating precision, recall, and F1 scores. Suppose there are p geometry problems in the dataset with a total of m relationships mentioned; the problem understanding method extracts n relationships, among which k are correct. The metrics are defined as follows: (1) Precision (P) = k/m, (2) Recall (R) = k/n, (3) F1 = 2k/(m + n). Relationship extraction performance: In this dataset, there are a total of 1462 clauses and 1823 relationships, of which 1744 relationships were correctly extracted, resulting in a relationship extraction accuracy of 95.23%. All entities and relationships for the processed entity and relationship types were correctly extracted, successfully completing the text-level ERE task. \begin{table}
\begin{tabular}{l c c c} \hline \hline
**RMT** & **Total** & **Average** & **Proportion** \\ \hline SC-B & 458 & 1.99 & 0.25 \\ SC-T & 367 & 1.60 & 0.20 \\ SI-B & 183 & 0.80 & 0.10 \\ SI-T & 277 & 1.20 & 0.15 \\ MC-S & 329 & 1.43 & 0.18 \\ MC-D & 146 & 0.63 & 0.08 \\ MT & 72 & 0.31 & 0.04 \\ Total Relationships & 1832 & 7.97 & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 11: Statistics on the Relationship Mention Type distribution of Dataset B. For text-oriented relationship extraction, the proposed method was compared with the baseline methods. The evaluation results of the four baseline methods alongside the proposed method are presented in Table 12.  
Compared with the baseline methods, the proposed method achieved the highest precision (0.996) and F1 score (0.974) in relationship extraction, demonstrating excellent performance in accurately identifying relationships while balancing precision and recall. Notably, it showed advantages in handling problems with complex multi-relationship expressions, where baseline methods either encountered difficulties or required cumbersome procedures. In the baseline methods, STM relies on predefined sentence-level templates, which, while precise for simple relationships, suffer from limited coverage of complex relationship combinations and unseen types. Moreover, redundant template designs (including unnecessary matching information) further reduce fault tolerance and generalization ability, thereby restricting recall. In multi-relationship scenarios, templates of varying complexity may produce redundant results, necessitating additional deduplication processes. Inadequate post-processing can severely impact accuracy, resulting in overall lower performance. In the baseline methods, TBC mitigates template redundancy to some extent through expression normalization and the Bi-LSTM-CRF model, partially improving generalization. However, since its relationship extraction is still fundamentally based on sentence templates, it struggles with complex RMTs (especially MC-D and MT types). Additionally, the normalization process for sentences containing new types is prone to information loss, and the imbalance in training data makes the model sensitive to low-frequency relationships. Although TBC performs better than STM in terms of precision and recall, it still does not overcome the inherent limitations of template-based methods. In the baseline methods, BBC's end-to-end extraction avoids the limitations of static template design but is highly sensitive to data distribution. The model produces false negatives due to distribution shift (e.g., failing to detect perpendicular relationships), which severely affects recall. Furthermore, its ability to handle complex multi-step reasoning is limited, leading to possible errors in splitting relationship chains (such as relationships among expressions), and unhandled or insufficiently trained types result in recognition failures. Its performance is constrained by data quality and the coverage of annotations.