Title: OBJECT DETECTION DEEP LEARNING NETWORKS FOR OPTICAL CHARACTER RECOGNITION

Abstract: In this article, we show how we applied a simple approach coming from deep learning networks for object detection to the task of optical character recognition in order to build image features taylored for documents. In contrast to scene text reading in natural images using networks pretrained on ImageNet, our document reading is performed with small networks inspired by MNIST digit recognition challenge, at a small computational budget and a small stride. The object detection modern frameworks allow a direct end-to-end training, with no other algorithm than the deep learning and the non-max-suppression algorithm to filter the duplicate predictions. The trained weights can be used for higher level models, such as, for example, document classification, or document segmentation.

1. INTRODUCTION:

Document images make the use of deep learning networks a complex task, since most deep learning network architectures have been designed and trained for natural images, making them useless for document images which are mainly white and black characters and figures. This is in particular the case for classification networks (VGG, ResNets, ...), object detection networks (Fast RCNN, Faster RCNN, Yolo, SSD, ...), segmentation networks (FCN, U-Net, SegNet, DeconvNet, Dilated-Net, ParseNet, DeepLab...) which cannot be applied directly, even with finetuning.

Two challenges arise with deep learning and document data. First, we need to train specific features for the type of data. Second, the available datasets can be of smaller size than classical datasets for computer vision (ImageNet, COCO, ...), in particular when it is required to annotate images for a specific purpose.

To reduce the amount of data to train high level descriptions of the document images, such as document zones, segments, lines, or elements, the idea is to train a smaller network on OCR data which exists at massive scale, and use the weights of this small network as pretrained early layers in bigger networks to perform high level tasks with less required data.

In our experiments, we show that best performing approaches currently available for object detection on natural images can be used with success at OCR tasks. Code will be released on Github, so that the open research community can bring the best model architectures in terms of accuracy and speed/size efficiency.

2. RELATED WORK:

In this section we quickly review the literature on OCR and object detection.

2.1. APPROACHES FOR OCR:

Most deep learning approaches using Object Detection methods for OCR are applied to the task of scene text recognition also called text spotting, which consists in recognizing image areas of text, such as a sign or a wall plaque. Once the text area is recognized, a reading method is applied inside the zone. Some approaches use weakly supervised training either using a CTC loss leaving the alignment between the character positions and the output result to a recurrent network such as bidirectionnal LSTM (He et al. (2015) , Jaderberg et al. (2014b) , Wang et al. (2018) , Goodfellow et al. (2013 Goodfellow et al. ( ), dro (2017)) , Liao et al. (2016) ) or using a fixed number of softmax classifiers (Jaderberg et al. (2015) , Bartz et al. (2017) ) ; some other approaches use guided learning (He et al., 2018) . These approaches are mainly driven by the Street View SVHN, Uber-Text (Zhang et al., 2017) , FSNS (Smith et al., 2017) , Coco-text (Veit et al., 2016 ), ICDAR 2003 (Lucas et al., 2003) and 2015 (Karatzas et al., 2015) , SVT and IIIT5K (Mishra et al., 2012) , Synth90k (Jaderberg et al., 2014a) datasets.

Rather than recognizing at word level or scene text level, few approaches concern direct detection of characters in natural images, using a localization network in ST-CNN (Jaderberg et al., 2015) , or modern object detection approach in yolo-digits (Redmon & Farhadi, 2018) to recognize digits in natural images.

This work is the first to apply modern object detection deep learning approaches to document data with small convolutional networks, without converting them to natural images as in (Gilani et al., 2017) . (Tensmeyer & Martinez, 2017) shows that document classification accuracy decreases with deeper networks.

2.2. APPROACHES FOR OBJECT DETECTION:

Modern object detections approaches are divided into two classes.

The first class yields to the highest accuracy object detectors, such as Fast-RCNN (Girshick, 2015) , Faster-RCNN (Ren et al., 2015) , Mask-RCNN (Detectron) (He et al., 2017) , and is based on the twostage approach of R-CNN (Girshick et al., 2013) . In the first stage, an algorithm, such as Selective Search, or a deep learning model, generates a set of candidate proposals for object regions. In the second stage, a deep learning network classifies the presence of the object (the objectness), its class, as well as estimates the precise object bounding box.

In the second class of object detectors, the objectness, the class as well as the bounding box regression, are directly predicted by a single dense deep learning network. These approaches include OverFeat (Rowley et al., 1995) , Yolo (Redmon et al. (2015) , Redmon & Farhadi (2016) , Redmon & Farhadi (2018) ) or SSD (Liu et al., 2015) .

3. OUR APPROACH TO OCR:

In our work, as a first attempt to use object detection networks to OCR, we design a single stage object detector, predicting the confidence of an object presence, the class, and the regression for the bounding box. In order to cope with multiple scales we use the feature pyramid approach of SSD (Liu et al., 2015) .

3.1. ARCHITECTURES:

Our 1-scale models are inspired by the LeCun model for digit classification except that the dense layer have been converted to convolutions (locally linear) in order to compute a prediction at multiple positions in the image on a grid defined by the stride of the whole network. These models are composed of 2 convolution layers of kernel 3 and 32 and 64 features respectively, followed by a max pooling of stride 2 and 1 convolution layers of kernel 12 and 128 features, so that the receptive field of the network is 28 pixel large and wide. Offset of the model with valid paddings will be 14. We consider the stride of the last convolution as a parameter, stride scale, to adjust the stride of the whole model which will be 2 × stride scale. On top of these features, 4 stacks of dense layers are used for objectness, classification, position and scale regressions. We named this 1-scale model CNN_C32_C64_M2_C128_D.

Our 2-scale models are composed of 2 convolution layers of kernel 3 and 32 and 64 features respectively, followed by a max pooling of stride 2 and 2 other convolution layers of kernel 3 and 64 features and another max pooling of stride 2. Each max pooling layer is followed by a convolution layer of kernel 11 and 12 respectively, so that the receptive field for each output is 28 and 56 pixel large and wide. Offset for each output is 14 and 28 respectively. We consider the stride of the output convolutions as a variable parameter, stride scale, to adjust the stride of the whole model which will be 2 × stride scale for the first output, and 4 × stride scale for the second output. On top of these 2 stages of features, 4 stacks of dense layers are used as well. We name this 2-scale model CNN_C32_C64_M2_C64_C64_M2_C128_D_2.

Each layer is followed by a ReLU activation, except for the outputs: objectness is computed with a sigmoid, classification with a softmax, position with hyperbolic tangent and scale with sigmoid.

1-scale model: 2-scale model:

3.2. LOSS:

For objectness, we need to consider the abundance of negative positions compared to positive positions. That is why we use the Tensorflow expression of weighted crossentropy designed to ensure stability and avoid overflow:

(1 -z) × x + l × log(1 + exp(-| x |) + max(-x, 0)

where l = (1 + (q -1) × z) and x = logits, z = targets, q = pos weight. We found that a positive weight of 1000 works well on our OCR dataset.

The loss for classification and regression are crossentropy loss and MSE loss. We found that adding a multiplier of 100 to the regression loss help converge faster.

3.3. COMPUTATION OF AVERAGE PRECISION:

It is common to use the mAP score as the final metric for object detection. In our case, we consider all classes as one class in order to use average precision as metric to measure the capacity of the models in terms of objectness and not classification. We use the name object mAP to distinguish it from classical mAP score.

The reason for this choice is that we focus on character detection in this work. For full character recognition, early results suggest that two-stage detectors might be of better fit than a one-stage detector, because in our 1-stage setting, classification accuracy drops when the classification network is not well positioned on the character (see Stride experiments on Mnist), and this argument could give an advantage to 2-stage detectors.

Later on, we might add a second stage on top of this network as in Mask RCNN or Faster RCNN and this network might become a region proposal network. We leave this as future work, which purpose will be to improve the classification accuracy.

4. DATASETS:



4.1. TOY DATASET:

We build a toy dataset in order to test our implementations on a simpler task and check that the implementation is correct. For that purpose, we use the MNIST handwritten digits dataset to create pages with handwritten digits, at fixed or variable scales, with or without noise. The number of object classes is 10, the digits ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"].

Our MNIST dataset is composed of 1600 images of size 728x448, consisting of 28x28 square digits randomly placed on a 2D grid of stride 28.

MNIST (default digit size 28):

MNIST with white prob .4 ( .9) MNIST with noise MNIST with digit size in 14-28 range random digits at scale 28 at different positions on a grid more digits per positions cluttered with noise added randomly random digit scale between 14 and 28. position is random

MNIST digit size 56:

MNIST with digit size in 28-56 range MNIST with 2 digit sizes 28,58 MNIST with 2 digit ranges 14-28,28-56 random digits at scale 56 at different positions on a grid random digits scale between 28 and 56. random position digits at scales 28 and 56. Positions on a grid random digit scales between 14 and 56. random positions.

Our MNIST dataset with noise adds random distortions to create a high level of noise on the images and test the robustness of the models.

To check the performance of the position prediction, we set a different network stride, for example 12 (setting stride scale to 6), so that the network grid of positions where the model is evaluated in the convolutions, do not fall exactly on the grid of characters. That way, some digits will appear cropped, up to 6 pixels off horizontally and vertically, in the viewpoint of the network, ie its 28x28 receptive field.

To check the performance of the scale prediction, we build a MNIST dataset with digits randomly resized in the [14-28] range.

Before adding a layer to our network architecture as in SSD (Liu et al., 2015) , we also check larger models at a bigger scale, with a MNIST dataset of 56 pixel wide digits.

Last, we build a two-scale dataset for two-layer predictions as in SSD (Liu et al., 2015) , with digits at size 28 and 56, and add a middle output to CNN_C32_C64_M2_C64_C64_M2_C128_D to build CNN_C32_C64_M2_C64_C64_M2_C128_D_2, a two-scale network.

For a continuous scale between 14 pixels and 56 pixels, we build another two-scale dataset with 2 digit size ranges, 14-28 and 28-56.

4.2. OCR DATA:

We build our own dataset of 8000 document pages, split into train (90%) and validation (10%) sets, for a fast check of our approach. Document PDF are converted to images with a resolution chosen automatically to have normal sized characters. To have fixed-sized image input for the network batched training, document images are then randomly cropped on a 728x448 area with characters, to have the same sized inputs as our mnist dataset.

We consider uppercase and lowercase letters, digits, the two parenthesis and the % sign. The number of classes is 65: ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", "A", "B", "C", "D", "E", "F", "G", "H", "I", "J", "K", "L", "M", "N", "O", "P", "Q", "R", "S", "T", "U", "V", "W", "X", "Y", "Z", "(", ")", "%"] Letters are filtered by their size to fall in the range of [14-56] pixels and we start with two-scale networks ([14-28] and ) tested on our MNIST dataset.

Character widths:

Character heights 5 EXPERIMENTS To test the network capability to predict position, we need to use a network stride different than the data grid stride. For example, with stride 12 instead of 28, most digits are not anymore in the center of the network reception field (except first row and column of our image).

Also, most digits will appear cropped in the field of view of the network and the IOU threshold defines how much crop ratio will be allowed to still consider the position on the grid as positive.

In the worst case, the network stride can be so large that some digits do not appear on the output grid, equivalent to an Intersection Over Union (IOU) of zero (intersection area is zero). Usually, the stride is not larger than the receptive field, and under this condition, the maximal intersection area between any digit and any network field is 50% times 50% = 0.25, while the union is 1.75, leading to a minimal IOU of 0.14. In the case of a smaller stride, for example 12 as below, the IOU threshold can be set higher without losing any digit for reconstruction: The large drop in classification accuracy for a small stride suggests that classification would benefit from better localized digits in the receptive field, which would encourage the use of 2-stage detectors.

IOU

To reduce the impact of the stride, we set a stride margin (see Experiment section on OCR data) on the digit max size to consider at a layer scale so that there is always one position on the network grid for which the character is fully seen by the network.

Reconstruction of ground truth from target data at 100% is only possible until an IOU threshold of 0.4, after which the network stride should be decreased. With a smaller stride of 4, reconstruction at 100% is possible at most IOU range: The images below show the target for an IOU of .2 for digits at scale between 7 and 14 pixels. The right image shows that with a large stride, small digits cut in the receptive field are dropped because of a too small IOU with the anchor, while with smaller stride, the IOU threshold does remove good candidates. A smaller stride enables to work with higher IOU and better mAP scores.

IOU

With stride 14, final object mAP is 72.31 With stride 4, final object mAP is 98.61

5.2.3. DIGIT SCALE PREDICTION:

The second reason (after the cropped digits) to use a smaller IOU threshold is to capture small digits.

For example, for digits two times smaller, the maximal intersection of a digit with a network receptive field is 0.5 times 0.5 times 0.25 (the maximal intersection area for full size digits of 20), hence 0.0625, while the union is 1 + 0.5 × 0.5 × (1 -0.25) = 1.1875 leading to a minimal IOU of 0.052. About 3 times smaller than for the full digit size.

With a range scale of 14-28 for the digit sizes, the target object mAP (obtained when we reconstruct the bounding boxes from the target) remains 100% at IOU 0.25 for a stride of 12 pixels. The predicted object mAP is 99.58. The classification accuracy drops down to 89.37%.

5.2.4. HIGHER CAPACITY NETWORKS:

Lets double the number of kernels to create CNN_C64_C128_M2_C256_D model. In order to capture digits in a bigger range than 28 pixels, we try networks with double reception field size, adding more layers (CNN_C32_C64_M2_C64_C64_M2_C128_D model), and possibly, multiple outputs at multiple layer stages (CNN_C32_C64_M2_C64_C64_M2_C128_D_2 model) as in SSD (Liu et al., 2015) .

First, we check our model with bigger field, CNN_C32_C64_M2_C64_C64_M2_C128_D model, on the MNIST dataset of 56 pixel wide digits. object mAP score is 1 while classification accuracy is 99.2% at 12 epochs, meaning this first architecture 56x56 receptive field deals well with digits twice big.

Then we add a second output to our network architecture as in SSD (Liu et al., 2015) to build CNN_C32_C64_M2_C64_C64_M2_C128_D_2 model, and on a 2-scale dataset with digits at size 28 and 56, object mAP scores remain stable at 99.44 and 99.64 for network strides 12 and 4 respectively.

On a 2-scale dataset with digits at size ranges 14-28 and 28-56, object mAP score with our CNN_C32_C64_M2_C64_C64_M2_C128_D_2 model is 98.82% and for the double size CNN_C64_C128_M2_C128_C128_M2_C256_D_2 is 99.11%. In order to train full document image rather than a 700 pixel high crop of the images, resolution has to be smaller to fit in the GPU. For that reason, we look at models to recognize digits at a maximum size of 14 pixels instead of 28. We build a model CNN_C32_C64_C128_D by removing the max pooling layer. The network input fields becomes 14 pixel wide, and the stride is divided by 2. Target average precision is better when the IOU is low, since the grid misses no ground truth boxes, nevertheless, the model possibly learns better at an higher IOU, which also leads to better classification results.

Model:

We also tried considering as positive any character that fall in the receptive field of the network. Target average precision is very close to 1 but final average precision remains below 0.48.

5.3.2. STRIDE MARGIN:

Since the network performs a strided analysis of the input image, we consider that characters should fall entirely into the receptive field of the network on one positive position. For that reason, we consider a stride margin, ie filter characters with a size lower than the receptive field dimension minus the stride. Deactivating this setting, some characters are not being seen completely by the network anymore and the prediction in position and scale should be harder to perform. object mAP score becomes 78.5%.

with stride margin (d0644c) without stride margin (6aa6c8)

5.3.3. POSITIVE WEIGHT AND LOSS WEIGHTS:

pos weight 1000 (4ca89a) pos weight 100 (b9bfbf) pos weight (df9c83)

Best results are obtained with pos weights=100.

5.3.4. KERNEL WIDTH:

To see the influence of the width of the layers, we try to double the number of filters for each layer, leading to the CNN_C64_C128_M2_C128_C128_M2_C256_D_2 model. A wider architecture does not seem to help much.

CNN_C32_C64_M2_C64_C64_M2_C128_D_2 Since document images are wide, we used a generator for the preprocessing, and do not have ground truth for mAP computation. Results are evaluated qualitatively. Best results with convolution of 28 pixel wide reception fields are obtained with images of max size 1000 pixels, since most characters fall at the right to be recognized.

At 1000 pixels:

While at size 1500 pixels, it misses the address :

More results at scale 1000: very small characters (below 7 pixels) as well as too big characters are missed, but main invoice information (amounts, headers, client,...) is recognized correctly.

6. CONCLUSION:

Object detection architectures sound promising for high quality character reading and the development of document features through end-2-end training. Classical rule-based algorithms can reuse these features to solve higher level tasks, such as line extraction.

This opens the way for best model architectures search by the community. Future work includes reusing improvements from object detection models, such as multiple anchors, two-stage detection, focal loss, optimization tuning for larger images, batches or other input resolution.

:



Figures and Tables:

At stride 12 and IOU .3, classification accuracy increases from 92.78 to 96.84, while objectness remains roughly perfect. At stride 6 and IOU .2, it increases from 78.53 to 95.78%.

Table content: <table><tr><td colspan="2">5.2.5 MULTI-STAGE NETWORKS</td><td/><td/><td/><td/></tr><tr><td>Parameters</td><td>Obj acc</td><td colspan="2">Class acc Reg acc</td><td>Obj</td><td>Target</td></tr><tr><td/><td/><td/><td/><td>mAP</td><td>mAP</td></tr><tr><td>Stride 12, IOU .5</td><td>99.59</td><td>98.02</td><td>0.00078</td><td>92.32</td><td>94.89</td></tr><tr><td>Stride 12, IOU .4</td><td>99.17</td><td>97.23</td><td>0.0047</td><td>99.79</td><td>100</td></tr><tr><td>Stride 12, IOU .3</td><td>99.74</td><td>96.84</td><td>0.00043</td><td>100</td><td>100</td></tr><tr><td>Stride 12, IOU .2</td><td>97.57</td><td>91.14</td><td>0.0016</td><td>99.98</td><td>100</td></tr><tr><td>Stride 12, IOU .15</td><td>98.02</td><td>83.85</td><td>0.0083</td><td>99.95</td><td>100</td></tr><tr><td>Stride 4, IOU .5</td><td>99.80</td><td>98.87</td><td>0.00053</td><td>100</td><td>100</td></tr><tr><td>Stride 4, IOU .25</td><td>99.48</td><td>95.78</td><td>0.00054</td><td>100</td><td>100</td></tr><tr><td>14-28 pixel wide,</td><td>96.58</td><td>91.42</td><td>0.0045</td><td>99.85</td><td>100</td></tr><tr><td>Stride 12, IOU .25</td><td/><td/><td/><td/><td/></tr></table>

f0f2f2) CNN_C64_C128_M2_C128_C128_M2_C256_D_2 (b0d7d3) Parameters Obj acc Class acc Reg acc Obj mAP Target mAP Stride 6+12, IOU 0.2 98.45/98.66 83.27/85.42 0.018/0.0097 70.11 78.15 5.3.5 FULL DOCUMENT, AT LOW RESOLUTION

Table content: <table/>

Peer review (iclr2019):
 This paper lacks any novelty/contribution as it just applies well-known and standard architectures for object detection (SSD) and image classification (LeNet) trained with standard algorithms and losses. Moreover, I fail to see what is the purpose of the proposed pipeline and it is not clear at all how it may help improving existing OCR engines in any particular scenario (handwriting recognition, printed text, historical documents, etc.). No demonstration or comparison with state of the art is provided. The authors claim “This work is the first to apply modern object detection deep learning approaches to document data” but there are previously published works. For example: Tuggener, Lukas, et al. "DeepScores--A Dataset for Segmentation, Detection and Classification of Tiny Objects." ICPR 2018. Pacha, Alexander, et al. "Handwritten music object detection: Open issues and baseline results." DAS 2018. Actually, in my opinion Music Object Detection in musical scores would be a much better test-bed/application for the proposed pipeline than any of the datasets used in this paper. The datasets used in the experimental section seem to be created ad-hoc for the proposed pipeline and do not come from any real world application. Finally, the presentation of the paper is marginal. Data plots have very bad resolution, there are no captions in any table or figure and they are not correctly referenced within the text. There seem to be also missing content in the last sections which makes them impossible to read/understand.