# Intelligent Evaluation of Chinese Hard-Pen Calligraphy Using a Siamese Transformer Network
## 4 Results and Discussion
### Analysis of Calligraphy Evaluation Experiments
Notably, the ChangeFormer performs better when the epoch is less than 10. Observing Figure 15c and its consistency with Figure 14 in terms of loss trends, the network's  
Figure 14: Calligraphy evaluation model training process loss changes. minimum loss reaches 0.319. Around the 80 th epoch, a relatively slight variation in loss is observed, indicating a proximity to optimal convergence. This network exhibits relatively minor differences in performance compared to the network selections of this study and demonstrates an improved effectiveness compared to the model using ResNet50 as the backbone network. Finally, when comparing Figure 15d with Figure 14, it shows the loss trend for the Siamese network based on ResNet50. However, the minimum loss for this network only reaches 0.331. By the 80 th epoch, the loss variation stabilizes, approaching optimal convergence. The comparative experimental results are presented in Table 2. Without the inclusion of an enhanced feature extraction network, the Siamese network with ResNet50 as the backbone performs better. However, upon adding the transformer-enhanced feature extraction network, the network structure and parameters are finetuned. In this case, the Siamese network with VGG16 as the backbone outperforms the one with ResNet50 as the backbone. The results of the comparative experimental testing are illustrated in Figure 16, showing the effects of varying model structures on handwriting evaluations. The left side shows the handwriting sample under evaluation, while the right side shows the standard sample. Although there are slight differences in stroke forms and brushwork between the left handwriting sample and the standard one, the overall structural framework of the characters is quite similar to the standard sample. Given that hard-pen writing is relatively constrained in stroke forms and brushwork compared to brush writing, the structural framework is more critical in evaluating hard-pen calligraphy. From the results of the comparative experimental testing, it can be observed that the lowest score is 0.354 for ResNet50 + transformer, the highest score is 1 for VGG16, and the score for VGG16 + transformer is 0.714, which aligns better with the intuitive perception of the handwriting sample. Figure 15: Calligraphy evaluation models training process loss variation. In the figure, these models are arranged in the following order: (**a**) ResNet50 + Transformer, (**b**) ChangeFormer, (**c**) VGG16, and (**d**) ResNet50. To further evaluate the model performance, we randomly selected 20 handwritten samples for each of the six Chinese characters from the dataset for manual assessment. A total of 10 evaluators participated, assigning scores based on the similarity between each handwritten sample and the standard sample of each Chinese character. For comparative analysis with evaluation models, scores were standardized from 0 to 1, rounded to two decimal places, resulting in 1200 manual scores. Handwritten samples were input into the evaluation models to obtain similarity scores. We employed Mean Absolute Error (MAE) and Pearson Correlation Coefficient (PCC) as evaluation metrics to compare the performance of the different structural models, as detailed in Table 3. Experimental results indicate that models incorporating a Transformer (VGG16 + Transformer and ResNet50 + Transformer) outperform others in terms of MAE and PCC. Particularly, VGG16 + Transformer demonstrated the lowest MAE (0.076) and the highest PCC (0.681), highlighting the positive impact of the Transformer, especially on the VGG16 model. \begin{table}
\begin{tabular}{c c c c} \hline
**Model** & **Precision (\%)** & **Recall (\%)** & **mAP50 (\%)** \\ \hline VGG16 & 57.86 & 71.80 & 99.90 \\ ResNet50 & 60.20 & 71.80 & 99.60 \\ VGG16 + transformer & 75.00 & 83.33 & 99.90 \\ ResNet50 + transformer & 66.66 & 75.00 & 99.90 \\ transformer & 75.00 & 50.00 & 86.20 \\ \hline \end{tabular}
\end{table} 
Table 2: The results of comparative experiments. Figure 16: Comparison experiment test chart.