SUGISAKAMASANORI

Journal of Robotics, Networking and Artificial
Life Vol. 10(4); March (2024), pp. 362–367
ISSN (Online): 2352-6386; ISSN (Print): 2405-9021
https://alife-robotics.org/jrnal.html

Research Article

ASLA: Automatic Segmentation and Labeling by Deep Learning
for Document Pictures
Kanta Kakinoki1, Tetsuro Katayama1, Yoshihiro Kita2, Hisaaki Yamaba1, Kentaro Aburada1, Naonobu Okazaki1
Department of Computer Science and Systems Engineering, Faculty of Engineering, University of Miyazaki, 1-1 Gakuen-kibanadai nishi, Miyazaki, 889-2192,
Japan2
Department of Information Security, Faculty of Information Systems, Siebold Campus, University of Nagasaki, 1-1-1 Manabino, Nagayo-cho, Nishi-Sonogi-gun,Nagasaki, 851-2195, Japan
1ARTICLE INFO
Article History
Received 24 November 2023
Accepted 22 July 2024
Keywords
Document picture
Area segmentation
Label generation
Rule-based area correction

ABSTRACTIn this paper, we propose ASLA, a segmentation and label generation system for document
pictures. ASLA reduces the duration needed for separating document pictures into areas and label
generation. By using the application example, we have verified that ASLA operates properly. We
have evaluated the usefulness of ASLA in regard to time and accuracy. We have assessed theefficacy of the rule-based area correction method. As a result, we have verified that ASLA isuseful.

© 2022 The Author. Published by Sugisaka Masanori at ALife Robotics Corporation Ltd.
This is an open access article distributed under the CC BY-NC 4.0 license(http://creativecommons.org/licenses/by-nc/4.0/).

content of these areas. Furthermore, our approach
incorporates the tracking of the reader's gaze movements,
mapping it. The implementation of this approach canenable the following significant benefits.
• Enhancing sales strategies through a deeper
understanding of client interests and concerns• Enhancing compliance by ensuring accurate
communication of critical information to service
subscribers and product consumers
Documentation in Portable Document Format orpicture format is convenient for human readers, but it isdifficult to understand the layout of the document andextract information from these formats. Therefore, thefollowing two tasks are needed to solve the above.
• Area segmentation that separates a digital document
into areas based on their constituent elements.
• Label generation that generates descriptive keywords
and phrases from the content of areas.

1. Introduction
As the quantity of printed texts grows, their storage
becomes increasingly challenging. The maintenance andmanagement of print-based records require significant
expenses. Moreover, there's always the possibility for
documentation to be misplaced or unlawfully removed.
Digital documentation offers solutions to these issues.
Transitioning to digital formats can substantiallydecrease the costs associated with maintenance andmanagement of documentation. Digital files also have
many advantages, such as the easy sharing of data.
Additionally, implementing a digital document
management system can reduce the time spent searchingfor the needed documentation. As a result, the adoptionof digital documentation is gaining widespread
popularity across various sectors.
A new approach to utilizing digital documentation
should be proposed because they mainly serve as
substitutes for printed texts at present. This approach
involves separating digital documentation into areas
based on their constituent elements, and subsequently
generating descriptive keywords and phrases from theManual execution of these processes requires timeconsuming and labor-intensive. To deal with thisinefficiency, we propose a prototype system named

Corresponding author E-mail: kakinoki@earth.cs.miyazaki-u.ac.jp, kat@cs.miyazaki-u.ac.jp, kita@sun.ac.jp, yamaba@cs.miyazaki-u.ac.jp,
aburada@cs.miyazaki-u.ac.jp, oka@cs.miyazaki-u.ac.jp362

ASLA, a segmentation and label generation system bydeep learning for document pictures.

two primary outputs. These outputs are the separated
picture and analyzed file. The separated picture isrendered as a JPEG (Joint Photographic Experts Group)
picture, which is the original document overlaid withcolor-coded rectangles. The analyzed file is a structured
data file in CSV (Comma Separated Value) format. In
this paper, the area is the smallest rectangle surrounding
the components of a document.

2. ASLA
2.1. Functions
ASLA is designed to process Japanese document picturesin either single or dual column layouts. ASLA generates

Table 1 Classes and colors of rectangles.

ClassesCaption

Footnote

FormulaColorsvioletolive

cyan

Listitem
teal

Pagefooter
purplePageheader
orangePicture
blue

Sectionheader
magenta

Table

Text

Title

green

redbrown

Fig. 2 The flow of rule-based area correction.

Fig. 1 ASLA’s structure.
2.2. Structure•

Fig. 1 presents ASLA’s structure. ASLA is composed of
the three following processing sections.
• Area Segmentation Section:
This section adopts Cascade R-CNN [1] known as a
sophisticated object detection architecture forimages. To enhance its accuracy, the section
integrates LayoutLMv3 [2], a specialized pre-trainedmodel designed for document AI applications, as thebasis for its object detection algorithm. First, thissection generates a training model. The system
refines LayoutLMv3 model by using DocLayNet [3]that is a diverse dataset containing a wide range of
document formats and their corresponding layout
annotations. Next, Cascade R-CNN processes theinput document picture by utilizing the refinedmodel, delimiting areas within it. For each identifiedarea, this section determines and records thecoordinates (top-left and bottom-right) along withthe appropriate classification. Thereafter, the rulebased area correction is applied to adjust any
misalignments in the separated areas. This process isdiscussed in more detail in Section 2.3 of this paper.
Finally, it sends the final segmentation data to thelabel generation section.
• Label Generation Section:
This section processes the areas separated by the area
segmentation section to generate appropriate labels.

For visual elements such as picture and table, thissection extracts the associated captions and utilizes
these as labels. For textual areas, the section
generates all nouns included in the area as labels. For
these processes, the OCR tools Tesseract [4] and themorphological analysis tool Mecab are used.
Area Analysis Section:This section outputs the segparating picture and theanalyzed file. The separated picture is a picture
which is drawn rectangles surrounding an area incolors defined for each area on the input picture. The
analyzed file is the CSV format output of the analysis
results obtained from the area segmentation andlabeling sections. Table 1 presents the definitions ofclasses and colors of rectangles in ASLA.

2.3. Rule-based Area CorrectionSome recent researches have considered the applicationof object detection frameworks for document layout
analysis [2], [5]. These researches primarily focus on
broad separating document into areas. In such contexts,
misalignments between separated and actual areaboundaries are generally considered inconsequential.
However, approach of ASLA differs from these previous
research in a crucial aspect. ASLA requires precise
character extraction within each segmentation areas.
Consequently, misalignments between separated andactual area boundaries are unacceptable in ASLA.
Furthermore, existing object detection architectures often
struggle with the detection of smaller document elements
[5].

363

Fig. 3 The area reseparating process in rule-based
area correction.

Fig. 5 Separated picture

Fig. 4 Input document picture sample
To deal with these limitations, ASLA incorporates therule-based area correction. This process serves to bridge
the gap between initial object detection-based
segmentation and the need for highly accurate areasegmentation. Fig. 2 presents the workflow of thiscorrection process. This process adjusts the areas initiallyidentified by the object detection architecture based on
the connected components extracted from the document
areas for refinement. Fig. 3 presents a visualdemonstration of this refinement process. In Fig. 3, an
example of correction misalignments in text areas thatwere initially detected by Cascade R-CNN architecture.

Fig. 6 Analyzed fileand table. Fig. 5 presents the separated picture generated
by inputting the document picture of Fig. 4 into ASLA.
Fig. 6 presents the analysis results file. From Fig. 5, wehave verified that ASLA is able to separate the picture
into each area, as defined in Table 1. Also, we haveverified that the small areas are separated and that there
are no misalignments in the segmentation areas.
4. Evaluation
We evaluate the usefulness of prototype ASLA. It isevaluated in the execution time, accuracy of area
segmentation, and accuracy of text extraction. In thispaper, we evaluate the accuracy of area segmentation and

3. Application Example
Fig. 4 presents a document picture that serves as our test
case. Fig. 4 is a document picture containing thecomponents of title, caption, text, section-header, picture,364

text extraction by calculating the F score and Levenshtein
distance.

analysis field. Our evaluation derives IoU value for theareas and classes defined in Table 1 to calculate the Fscore. In addition, we calculate the F score with andwithout rule-based area correction to evaluate theusefulness of rule-based area correction. The equations
for the F score are shown below.

4.1. Evaluation of the Execution TimeTo evaluate the efficiency of ASLA, we conducted acomparative analysis of processing times between ourtool and hand-operated methods. This evaluation focusedon two key tasks: segmentation and label generation.
Our segmentation evaluation utilized two document
layouts that are a single-column format and a dualcolumn
arrangement.
Forthehand-operated
segmentation, we engaged five subjects to perform area
segmentation on these documents. Table 2 presents acomprehensive comparison of the time requirements forboth methods. From Table 2, We have verified areduction in processing time of approximately 3 minutes
Table 2 Evaluating the duration needed for
ASLA versus hand-operated segmentation.
Handoperated(Average)
ASLA

One column

Dual column

Average

2m6s

4m8s

3m7s

7s

9s

8s

𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃 =𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅 =𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 𝑤𝑤𝑤𝑤𝑤𝑤ℎ 𝐼𝐼𝐼𝐼𝐼𝐼 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 𝑇𝑇ℎ𝑟𝑟𝑟𝑟𝑟𝑟ℎ𝑜𝑜𝑜𝑜𝑜𝑜
𝐴𝐴𝐴𝐴𝐴𝐴 𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴

𝐹𝐹 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 =2 × 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃 × 𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅
𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃 + 𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅

(1)
(2)
(3)

Table 4 presents the computed F score. In this result,
we have verified that the integration of Cascade R-CNN
with area correction implemented in ASLA's area
segmentation section is the most favorable F score.
Moreover, when the IoU threshold is elevated to astringent 0.9, the rule-based area correction continues toexhibit superior performance, maintaining the highest Fscore among the compared methods. These observations
provide strong evidence for the effectiveness of the rulebased area correction approach.

Table 3 Evaluating the duration needed for
ASLA versus hand-operated labeling.
Hand-operated(Average)ASLA

𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 𝑤𝑤𝑤𝑤𝑤𝑤ℎ 𝐼𝐼𝐼𝐼𝐼𝐼 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 𝑇𝑇ℎ𝑟𝑟𝑟𝑟𝑟𝑟ℎ𝑜𝑜𝑜𝑜𝑜𝑜
𝐴𝐴𝐴𝐴𝐴𝐴 𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴

Time4.3. Evaluation of Text Extraction Accuracy

8m40s
5s

After area segmentation, ASLA uses OCR to extract thecharacters in the area and generates labels. Therefore, theaccuracy of character extraction has a significant impact
on the accuracy of label generation. Thus, to evaluate theaccuracy of character extraction, we calculate theLevenshtein distance. The Levenshtein distance is ametric for the degree of similarity between two strings.
The distance is the number of deletions, insertions, andreplacements needed to convert one string into another[7]. Fig. 4 presents the test data is the document picture.
We calculate the Levenshtein distance between thecharacter strings in this document picture and thecharacter strings generated by the Label Generation
Section of ASLA.
Table 5 shows the calculated Levenshtein distances.
From Table 5, we have verified that the combined use of
Cascade R-CNN and area correction in the area
segmentation section of ASLA gives the best accuracy incharacter extraction. In addition, we have verified thatarea correction improves the accuracy of character
extraction for both Mask R-CNN and Cascade R-CNN.

(95%) per document. In other words, we gain a striking
efficiency with ASLA in area segmentation.
In label generation evaluation, we measured the time
efficiency of label generation. The hand-operated
labeling involved five subjects who were tasked withgenerating labels to pre-separated document pictures.
Table 3 presents the comparative results of this label
generation task. From Table 3, ASLA achieved a time
reduction of approximately 8 minutes (99%) perdocument in the label generation. In other words, we gaina striking efficiency with ASLA in label generation.
4.2. Evaluation of Area Segmentation Accuracy
In the realm of document picture analysis, various
research has explored the object detection architectures
for area segmentation [5]. One of these architectures is
Mask R-CNN [6] that is a sophisticated object detection
architecture. Based on these previous research efforts,
this evaluation aims to provide an all-inclusivecomparative evaluation.
We evaluate segmentation accuracy by contrasting theperformance of two architectures. Two architectures are
Cascade R-CNN which forms the backbone of ASLA and
Mask R-CNN utilized in previous research. To ensureassessment, we adopt DocLayNet test dataset which is awell-recognized benchmark in the document layout

5. Conclusion
This paper has presented a prototype system named
ASLA, a segmentation and label generation system bydeep learning for document pictures, to reduces theduration needed for segmentation document pictures into
areas and label generation. ASLA operates by365

segmentation input document pictures into distinct areas,classifying these areas, generating appropriate labels, andsubsequently producing both a separated picture and ananalysis file.
Empirical testing with various document pictures has
validated ASLA's functional accuracy. Our evaluation ofASLA's efficiency reveals significant time savings. Area
segmentation time per document is reduced byapproximately 3 minutes (95%). In addition, label

generation time is reduced by approximately 8 minutes(99%). Further assessment of ASLA's segmentation
accuracy demonstrates its ability to maintain high Fscores, even under stringent IoU threshold conditions.
Moreover, an analysis of text extraction accuracy using
Levenshtein distance metrics indicates that ASLA
achieves superior performance in character extraction
tasks.

Table 4 Evaluating the accuracy of area segmentation by F score.
Methodclass

IoU
CaptionFootnote
FormulaList-item
Page-footer
Page-header
PictureSectionheader
Table
Text
Title
Average

MethodLevenshtein
distances

Only Mask
R-CNN

Mask R-CNN and area
correction

0.8
0.58
0.69
0.61
0.83
0.68
0.73
0.71

0.9
0.24
0.31
0.27
0.38
0.25
0.290.28

0.8
0.71
0.75
0.69
0.84
0.72
0.76
0.77

0.9
0.68
0.72
0.69
0.80
0.71
0.73
0.75

Only
Cascade
R-CNN
0.8
0.9
0.60
0.27
0.71
0.37
0.61
0.33
0.83
0.54
0.71
0.31
0.74
0.35
0.79
0.42

0.62

0.21

0.69

0.69

0.66

0.69
0.85
0.74
0.70

0.26
0.32
0.34
0.29

0.82
0.90
0.71
0.76

0.80
0.88
0.67
0.74

0.73
0.87
0.74
0.72

Only Mask RCNN
219

Cascade R-CNN and area correction(ASLA)
0.8
0.75
0.76
0.69
0.88
0.73
0.79
0.81

0.90.71
0.76
0.68
0.87
0.72
0.78
0.77

0.28

0.71

0.69

0.34
0.39
0.32
0.36

0.83
0.91
0.78
0.79

0.80
0.89
0.75
0.77

Table 5 Evaluating Levenshtein distances.
Mask R-CNN and area
correction

Only Cascade RCNN

Cascade R-CNN and area
correction (ASLA)

5183

3The future issues include the following.
• Support non-white backdrop document pictures.
ASLA treats the background of document pictureswith non-white backgrounds as an area. This
prevents appropriate area segmentation. Therefore, itis necessary to support document pictures with nonwhite backdrops.
• Automatically create nouns or phrases that describe
thedetailsofpictures
ortables.
ASLA generates the text in the caption below thepicture and above the table as the label. This meansthat the target of analysis includes the condition that
there is a caption below the picture and above thetable. In recent documentation, there are often nocaptions for pictures or tables. Therefore, the current
ASLA lacks versatility. Therefore, it is necessary toautomatically generate labels from pictures andtables.
• Consider processing of meaningless parts inJapanese.
ASLA cannot generate appropriate labels if the text
in the area does not make sense as Japanese. Thismeans that it cannot generate labels for areas such as
page numbers. Documentation often has these kindsof areas. Therefore, it is necessary to handle areas•that do not make sense as Japanese in an exceptional
way.
Improve Accuracy of Optical Character Recognition.
ASLA uses Tesseract in the label generation section.
However, the current Tesseract cannot extract
characters from pictures 100% accurately. This may
result in incorrect extraction of characters. Therefore,it is necessary to improve accuracy of OCR.

References
1.

2.

3.

366

Zhaowei Cai and Nuno Vasconcelos, “Cascade R-CNN:
Delving into high quality object detection,” Proc. of 2018
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), pp. 6154-6162, 2018.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, andFuru Wei, ”LayoutLMv3: Pre training for Document AI
with Unified Text and Picture Masking”, MM '22:
Proceedings of the 30th ACM International Conference onMultimedia, pp. 4083-4091, 2022.
Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed
S. Nassar, and Peter Staar, “DocLayNet: A Large HumanAnnotated Dataset for Document-Layout Segmentation”,
KDD '22: Proceedings of the 28th ACM SIGKDD
Conference on Knowledge Discovery and Data Mining, pp.
3743-3751, 2022.

4.
5.

6.
7.

”tesseract”, https://github.com/UB-Mannheim/tesseract(Accessed 2024-06-05)
Sanket Biswas, Pau Riba, Josep Llados, Umapada Pal,
“Beyond document object detection: instance-levelsegmentation of complex layouts”, IJDAR 24, pp.269-281,
2021.
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross B.
Girshick: "Mask R-CNN", International Conference on
Computer Vision(ICCV), pp.2961-2969, 2017.
Vladimir I. Levenshtein., “Binary codes capable ofcorrecting deletions, insertions and reversals”, Soviet
Physics, Doklady, Vol.10, pp.707-710, 1966.

Dr. Kentaro Aburada
He received the B.S., M.S, and Ph.D.
degrees in computer science and systemengineering from the University ofMiyazaki, Japan, in 2003, 2005, and2009, respectively. He is currently anAssociate Professor with the Faculty ofEngineering, University of Miyazaki,
Japan. His research interests include
computer networks and security. He is amember of IPSJ and IEICE.
Dr. Naonobu Okazaki
He received his B.S, M.S., and Ph.D.
degreesinelectricalandcommunication engineering fromTohoku University, Japan, in 1986,
1988 and 1992, respectively. He joined
the Information Technology Researchand Development Center, Mitsubishi
Electric Corporation in 1991. He iscurrently a Professor with the Faculty ofEngineering, University of Miyazaki
since 2002. His research interests include mobile networkand network security. He is a member of IPSJ, IEICE andIEEE.

Authors Introduction
Mr. Kanta Kakinoki
He received the Bachelor's degree inengineering (computer science andsystemsengineering)
fromtheUniversity of Miyazaki, Japan in 2023.
He is currently a Master's student inGraduate School of Engineering at theUniversity of Miyazaki, Japan. His
research interests include softwaredevelopment support method
andpicture processing.

Dr. Tetsuro Katayama

He received a Ph.D. degree inengineering from Kyushu University,Fukuoka, Japan, in 1996. From 1996 to2000, he has been a Research Associate
at the Graduate School of Information
Science, Nara Institute of Science andTechnology, Japan. Since 2000 he has
been an Associate Professor at theFaculty of Engineering, Miyazaki
University, Japan. He is currently a Professor with theFaculty of Engineering, University of Miyazaki, Japan. His
research interests include software testing and quality. He isa member of the IPSJ, IEICE, and JSSST.
Dr. Yoshihiro Kita

He received a Ph.D. degree in systems
engineering from the University ofMiyazaki, Japan, in 2011.
He iscurrently an Associate Professor with theFaculty of Information Systems,
University of Nagasaki, Japan. His
research interests include software
testing and biometrics authentication.

Dr. Hisaaki Yamaba

He received the B.S. and M.S. degrees
in chemical engineering from theTokyo Institute of Technology, Japan,in 1988 and 1990, respectively, and thePh D. degree in systems engineeringfrom the University of Miyazaki, Japanin 2011. He is currently an AssistantProfessor with the Faculty ofEngineering, University of Miyazaki,
Japan. His research interests include network security anduser authentication. He is a member of SICE and SCEJ.

367