# A Pipeline Approach to Context-Aware Handwritten Text Recognition
## 3 Proposed Method
### Implementation Details  
This section describes the training procedure of the proposed pipeline. Firstly, the YOLO-v5 model is used for multi-text type classification. In this study, the YOLO-v5x model was chosen as it has shown a very promising result from the established results. An annotation software called Labellmg [25] was used to annotate the collected dataset to train the YOLOv5 model. The image is segmented into three classes (printed, handwritten, and non-text), as shown in Figure 7. The non-text category contains unreadable texts such as logos, signatures, and others. Three hundred receipt images were used, with a train and validation split of 5:1. After that, the model was trained for 100 epochs, where the best model with the highest precision score was saved. The total number of characters included in the experiment was 96, with a maximum length of 64.  
After the handwritten segments were obtained from YOLOv5, we combined the ResNet-101 and Transformer models, named ResNet-101T, for HTR. ResNet-101 is used as the feature extractor, the backbone of the proposed architecture where the linear projection layer is excluded, and Transformer is used to analyze the extracted features, as inspired by [26]. In [26], CNN is used together with Transformer to achieve the object detection goal. In this paper, ResNet-101 is responsible for learning 2D representations that encompass shape/outline and positional information of the texts/words in the image. The last feature map by ResNet-101 serves as the input to Transformer, which is then flattened with 2D-positional encoding and passed to the encoder. The ResNet-101 feature map also serves as the input for the decoder, which is the target in this case. The output embeddings of the decoder are then passed to the final linear layer for predictions.  
Both the ResNet-101 and Transformer (ResNet-101T) are jointly trained. Two inputs are fed into the model: handwritten segments and the label. The image is fed into ResNet-101T with the shape 30 \(\times\) 375 \(\times\) 3. The label length is 64 characters. The input shape and the corresponding number of parameters for each layer of ResNet-101T are given in Table 2. The total number of text segments used was 15,297, and the train, test, validation split ratio was 8:1:1. The model was trained for 100 epochs, and the hyperparameters were tuned to find the optimal result in terms of character error rate (CER) and word error rate (WER). The hyperparameters that are tuned in the experiments include cell units of the ResNet-101 and the number of decoders, encoders, attention heads of the Transformer model.  
In training the NER model, a tag editor is used to annotate the text data, where the text data is obtained from the annotated text labels. Unlike the labeled data for the HTR  
\begin{table}
\begin{tabular}{l l l} \hline \hline
**Layer (Type)** & **Input Shape** & **Parameter** \\ \hline ResNet & [1; 3; 30; 375] & 9408 \\ Embedding layer & [1; 63] & 25,344 \\ Positional Encoding & [1; 63; 256] & 0 \\ Transformer Encoder & [12; 1, 256] & 5,260,800 \\ Transformer Decoder & [12; 1, 256], [63; 1, 256] & 6,315,520 \\ Linear & [1; 63; 256] & 25,443 \\ \hline Total parameter: 12,151,651 & \\ Trainable parameter: 12,151,651 & \\ Non-trainable parameters: 0 & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Output shape and number of parameters of each layer in the proposed ResNet-101T model.  
Figure 7: Sample segmented images: (a) printed (b) handwritten (c) non-text.  
model recognition, the text labels are annotated for both printed and handwritten texts. The model is trained for 100 epochs, and the best model with the highest accuracy is saved. The dataset contains a total of 54,522 tokens, with 954 sentences. Tokens refer to chunks within a sentence, or the string between spaces and punctuation symbols, while sentences represent the sequence of tokens. Eleven attributes, such as the clinic's name, address, and contact details, were identified in the study.