## 4 Approach
### The Pre-Training Model  
Our stroke-based autoencoder (SAE) is a self-supervised model that predicts the original signal \(\hat{z}=(z_{1},\ldots,z_{n}),z\in\mathbb{R}^{n\times d_{3}}\) given its encoded patches, as shown in Figures 1 and 5. Similar to other autoencoders, our model has an encoder that maps an input sequence of symbol representations \(x=(x_{1},\ldots,x_{n}),x\in\mathbb{R}^{n\times d_{1}}\) to a sequence of continuous representations \(y=(y_{1},\ldots,y_{n})\in\mathbb{R}^{n\times d_{2}}\), and a decoder that generates an output sequence \(z=(z_{1},\ldots,z_{n})\in\mathbb{R}^{n\times d_{3}}\), where \(d_{1}\) is the number of pixels in the input patches, \(d_{3}\) is the dimension of the outputs and \(d_{2}\) is the dimension embedded of the encoded patches. Unlike the Masked Autoencoder [21], which uses the spatial segmentation of pictures, we divide Chinese pictures by time according to their writing orders, since every Chinese character has its unique writing order. In this way, we reconstruct the stroke sequence of character images from the encoded patches. **SAE based on Vision Transformer:** Our encoder based on Vision Transformer(ViT) [22] is applied to the image patches of a Chinese character picture, as shown in Figure 4. For a Chinese character, we set its stroke number to \(m\). In our Chinese character datasets, the maximum stroke number of characters is 24. Thus, we split each image into \(n=5\times 5\) patches \(x=(x_{1},\ldots,x_{n})\). For these characters whose stroke number is less than 25, we supplement their stroke sequence images with blank pictures. We apply a linear projection(Proj) with added positional embeddings(Pos) on the input patches; after that, a series of Transformer blocks are used to process the encoded patches. The encoder has depth 4 and width 256 of Transformer blocks. \[y= Transformer\_Blocks\_1(Proj(x)+Pos) \tag{1}\] \[y= Transformer\_Blocks\_i(y),i=2,3,4 \tag{2}\]  
Our decoder is similar to the decoder of Transformer [36] applied on encoded patches, but only consists of multihead self-attention. That is, we directly reconstruct the image sequence of a Chinese character with encoded patches. The decoder also has depth 4 and width 256 of the Transformer blocks. \[z= Transformer\_Blocks\_1(y+Pos) \tag{3}\] \[z= Transformer\_Blocks\_j(z)),j=2,3,4 \tag{4}\]  
**SAE based on Resnet and Transformer:** If we consider the stroke sequence of images as a sentence, each image can be viewed as a word. Thus, the reconstruction task can be modeled as a sequence-to-sequence (seq2 seq) task. In this regard, we exploit the Transformer [36] to decode the stroke sequence of images from the previous images and the original image of a Chinese character. Considering that Resnet [37] plays an important role in recognition tasks and performs better on smaller datasets than Vision Transformer, we propose an SAE architecture based on Resnet and Transformer, the overall framework is illustrated in Figure 5. For an input image \(x_{m}\in\mathbb{R}^{C\times H\times W}\), we employ Resnet as the encoder to output \(\hat{y}\in\mathbb{R}^{\frac{H}{2}\times\frac{W}{2}\times 256}\) encoded patches. For input image sequences \(x\), a linear projection(Proj) is added after Resnet to map \(n\times\frac{H}{2}\times\frac{W}{2}\times 256\) to output \(y\in\mathbb{R}^{n\times 256}\). \[\hat{y} =Encoder(x_{m}) \tag{5}\] \[y =Proj(Encoder(x)) \tag{6}\]  
The encoder here has the effect of generating word vectors from image sequences. The decoder has depth 1 and width 256 of Transformer blocks. The detailed architecture of our encoder and decoder is shown in Figure 6. We apply a masked multi-head attention (\(Masked\_MultiHead\)) with added positional embeddings(Pos) on input sequence \(y\). Then, a layer normalization [38] is employed after residual connection. \[y^{(1)} =y+Pos \tag{7}\] \[y= Transformer\_Blocks\_1(Proj(x)+Pos) \tag{1}\]0 (8) \[y= Transformer\_Blocks\_1(Proj(x)+Pos) \tag{1}\]1  
Next, we employ multi-head attention (MultiHead) on \(\hat{y}\) and \(y^{(3)}\).