## 3 Key Character Detection
### CNN-Based Key Character Detection
#### 3.2.1 RetinaNet Architecture  
Figure 4 illustrates the block diagram of RetinaNet [27]. As shown in Figure 4, the RetinaNet architecture has some sub-parts: ResNet [8], Feature Pyramid Network (FPN) [29], classification subnet, and box regression subnet. In the first sub-part, ResNet, which is one of the famous CNNs, is exploited to extract multi-level feature maps from different convolutional layers for image classification and object detection. In the second subpart,  
Figure 3: Illustration of ROI within an input image.  
the concept of image pyramid used in SIFT, SURF, HOG, YOLOv3 [30], and YOLOv4 [31] is adopted here. To raise the performance of detecting objects with different sizes, FPN is adopted in RetinaNet. The main reason is that FPN is utilized to combine multi-scaled feature maps and then obtain rich feature maps. In addition to feature-point extraction and object detection, FPN is also used to achieve instance segmentation [32]. After FPN, the classification subnet based on the full convolution network (FCN) predicts the probability of a present object to obtain the object class information in the multi-scale feature maps. Similarly, the box regression subnet is used to estimate the coordinate information of objects with different sizes in the multi-scale feature maps. Then, it is expected that RetinaNet can effectively detect key characters with different sizes for document images.  
To correctly classify and localize the key characters, the total loss function of RetinaNet is the sum of the classification loss and the bounding box regression loss for network training. For object classification, a common loss function, cross-entropy loss, is described as follows:  
\[CE(p,y)=\begin{cases}-\log(p)&\text{if }y=1\\ -\log(1-p)&\text{if }y=0\end{cases} \tag{1}\]  
where \(y=\{0,1\}\) denotes whether the ground-truth class exists, \(p\in[0,1]\) represents the probability output of the proposed scheme with \(y=1\) (i.e., the ground-truth class exists), and \(\log(\cdot)\) denotes the logarithmic function. Furthermore, to avoid class imbalance and easy/hard example problems during the training, the loss weight for well-classified samples (i.e., easy samples) can be reduced, and the weight of hard samples can be increased. Then, the focal loss function in RetinaNet is designed as the classification loss in the following:  
\[FLoss(p,y)=\alpha\cdot(1-p_{t})^{\gamma}\cdot CE(p,y) \tag{2}\]  
where \(\alpha\) is a weighting factor, \(p_{t}=y\cdot p+(1-y)(1-p)\), and \(\gamma\) is a constant used in a modulating factor \((1-p_{t})^{\gamma}\). As we can see in Equation (2), the effect of the modulating factor is enlarged when \(\gamma\) is increased. This means that the focal loss contains more errors from the hard samples, and then RetinaNet can learn more experiences from the hard samples for raising the performance of object detection when \(\gamma\) is bigger. On the other hand, RetinaNet is also needed to evaluate whether an object is correctly located. To evaluate whether each true object is detected by a bounding box, the smooth \(L_{1}\) loss [26] applied as the loss function to the box regression subnet for object localization is described as follows:  
\[L_{1-smooth}(z)=\begin{cases}|z|&\text{if }|z|>\beta\\ \frac{1}{|\beta|}z^{2}&\text{if }|z|\leq\beta\end{cases} \tag{3}\]  
where \(z\) denotes the input of the smooth \(L_{1}\) loss and \(\beta\) is a hyper-parameter. It is expected that the detected bounding box is close to the true object when the smooth \(L_{1}\) loss is small. Therefore, the trained RetinaNet can be obtained by minimizing the total loss function.  
Figure 4: Illustration of key character detection based on RetinaNet.