## 6 Discussion and Limitations  
Intrinsic evaluation for word embeddings. Our pre-trained SAE model is competent in word embeddings. We use the architecture based on Vision Transformer pre-trained on 3755 printed Chinese character sequences to output word embeddings. We notice that the output of the encoder is 25 vectors in 64 dimension, and each word vector corresponds to a spatial position of the image. A 1600 (\(25\times 64\)) vector is too large for semantic embedding. For Chinese characters that have the same radical part at different locations, the embeddings of words cannot clearly show their relationship. In addition, each output of the decoder corresponds to an image of Chinese character's stroke sequence, and therefore the middle layer of the decoder contains the stroke information of Chinese characters. For a Chinese character and its stroke number \(m\), the \(m\)-th output corresponds exactly to its original image. Based on the above analysis, we use the \(m\)-th output patch of the third Transformer block of the decoder as its embedding vector, since it can reflect not only the stroke variation but also the overall structure of Chinese characters. To validate the representation ability of word embeddings of Chinese character structure, we calculate the word similarity of Chinese characters and their radical parts as an intrinsic evaluation. As shown in Figure 12, the Chinese characters and their corresponding radical parts have a higher cosine similarity in that row and column. This illustrates that our encoder enhances the representation of Chinese characters with their abundant morphological information. Moreover, we notice that the bold numbers in the first column do not reflect the strong relevance of Chinese characters and their radical parts. This is because all Chinese characters in the examples are composed of the same radical component '\(\square\)', except '\(\square\)'. It can be seen that '\(\square\)' has the lowest cosine similarity in the first column, which is consistent with the above observation. We use the t-SNE visualization [49] for the 256-dim embeddings of some Chinese characters. The K-means clustering algorithm [50] with cosine distance is applied to divide these characters into three categories, and each category is marked with the same color. As shown in Figure 13, we can see that these Chinese characters in the same category really have some common radical and stroke parts. Thus, the clustering results show great potential for our word embedding ability in other Chinese NLP-related applications, such as Chinese Text Classification [30] and Chinese Named Entities [6]. Our discussion of word embeddings from SAE architecture only scratches the surface of the Chinese character structure in semantic enhancement. How to replace the image representation  
Figure 12: The similarity between the Chinese characters and their radical parts. Bold numbers indicate the cosine similarity between Chinese characters and their corresponding radical components, and those enclosed in red boxes represent some outliers. method in [30] or the Five-Stroke method in [6] with SAE is beyond the scope of this work, but worthy of further exploration. **Complexity analysis. ** We analysis the model size of two different models on training, pre-training and fine-tuning. For simplicity, we denote SAE based on Vision Transformer by SAE-ViT and SAE based on Resnet and Transformer by SAE-RT. The two training models are as large as stroke-based [1] and MAE [21] respectively. The details are shown in Table 3.  
**Limitations. ** Our model successfully captures the morphological and semantic information of Chinese characters with self-supervised learning. However, since our model implements the self-supervised method by reconstructing stroke image sequences, how to assemble a high-quality dataset still remains challenging when we migrate the model to ancient Chinese characters or other hieroglyphs. That is, we need the stroke image sequences of the corresponding characters to implement the model training.