<figure><img src="image_1.png" /></figure>

HAL
open science

# Encoder-Decoder Language Model for Khmer
## Handwritten Text Recognition in Historical Documents

Seanghort Born, Dona Valy, Phutphalla Kong

► To cite this version:

Seanghort Born, Dona Valy, Phutphalla Kong. Encoder-Decoder Language Model for Khmer Handwritten Text Recognition in Historical Documents. 2022 14th International Conference on Software, Knowledge, Information Management and Applications (SKIMA), Dec 2022, Phnom Penh, Cambodia. pp.234 - 238, 10.1109/skima57145.2022.10029532. hal-04789871

HAL Id: hal-04789871

https://hal.science/hal-04789871v1

Submitted on 27 May 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Encoder-Decoder Language Model for Khmer Handwritten Text Recognition in Historical Documents

Seanghort Born, Dona Valy, and Phutphalla Kong

Department of Information and Communication Engineering, Institute of Technology of Cambodia, Cambodia

bornseanghort5@gmail.com, dona@itc.edu.kh, kongphutphalla@gmail.com

**Abstract**—Correcting spelling errors in texts extracted from Khmer palm leaf manuscripts by handwritten text recognition (HTR) systems can be very challenging. A Khmer Language Model developed in this study aims to facilitate the task mentioned above. The proposed model utilizes long short-term memory (LSTM) modules applicable for improving the performance of text recognition which is to predict a sequence of characters as output. The architecture of the language model is based on an encoder-decoder mechanism which is composed of two parts: an encoder to capture the context of the input erroneous word and a decoder to decode and predict the correctly spelt output word. Experimental evaluations are conducted on a text corpus consisting of Khmer words extracted from Sleuk-Rith set.

**Keywords**—Khmer Language Model, Natural Language Processing, Recurrent Neural Network, Encoder-Decoder, Long Short-Term Memory (LSTM)

## I. INTRODUCTION

Khmer palm leaf Manuscript or Sleuk-Rith is one of Cambodia's significant heritages. The documents are facing destruction and are in need for preservation. For the past decades, there have been ongoing efforts to recover and preserve these treasured documents. The digitization of Khmer historical documents [1] is being conducted and a text recognition system is to be developed to extract texts from the digitized documents. A Khmer Language Model (KLM) is to be implemented to assist and improve the performance of the text recognition system.

To train a machine capable of understanding the contents of historical documents, including the contextual nuances of a language, a method that matches between that natural language and machine is needed. The KLM is a Natural Language Processing (NLP) model implemented to improve the misspelling correction system which is very beneficial for the preservation of Khmer Palm Leaf Manuscripts. NLP is a sub-field of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human languages. They are used in different applications including text recognition [2], speech recognition [3], machine translation [4], part-of-speech tagging [5],

parsing [6], Optical Character Recognition [7], and information interval [8] to name a few.

A deep learning approach with recurrent modules (LSTM) adopting encoder-decoder architecture is proposed because of its efficiency. The proposed approach is not only trained depending on individual characters in a word, but also on the context of where the characters are located which helps the model to produce a better predicting accuracy. The KLM is an important support to research on the digitization of Khmer Palm Leaf Manuscripts or Sleuk-Rith. The contributions of this study are as follows:

* Development of a Khmer language model using an encoder-decoder algorithm applicable for the Khmer misspelling word correction.

* The developed model is to be integrated in the text recognition system which will be used in the preservation project of Khmer Palm Leaf Manuscripts.

## II. KHMER PALM LEAF MANUSCRIPTS

The concept of writing has been revolutionized throughout many civilizations in order to communicate, keep the important information safe, or transfer knowledge to the next generations. Before papers were invented in the nineteenth century, natural materials were chosen as writing media based on the geographical conditions of each nation. Palm leaf manuscript is one of the major writing systems that were used in Southeast Asia including Nepal, Sri Lanka, Burma, Thailand, Indonesia, and Cambodia [9].

Palm leaf manuscripts, or shortly called Sleuk-Rith, were used during and after the Khmer Empire, from at least the twelfth century [10]. They are made from dried, matured palm leaves which are cut into a long rectangle shape approximately forty-eight-by-four-centimeter (48x4 cm) [11], are bound and tied together to create a binding like a book. On each page, letters are carved using a sharp edge stylus (Fig. 1).

Sleuk-Rith documents are used to preserve rich knowledge on a variety of topics and disciplines of study that

<figure><img src="image_3.png" /><figcaption>Fig. 1. Sample images of Khmer palm leaf manuscript pages</figcaption></figure>
<figure><img src="image_4.png" /><figcaption>Fig. 2. The different spellings between Khmer words on Sleuk-Rith and modern Khmer (a. original words on Sleuk-Rith, b. extracted text from Sleuk-Rith, and c. corresponding words in modern Khmer)</figcaption></figure>

have been passed down through generations [12]. The contents of these ancient documents are related to many fields of study such as history, religion, art, and even scientific discovery including mathematics, architecture, and medicine [13].

## A. Khmer Language on Sleuk-Rith

Khmer is the official language spoken by Cambodian people. The letters are written horizontally from left to right and then downwards. Khmer language was influenced significantly by languages such as Sanskrit and Pali amid Hindu and Buddhist influences throughout Cambodia's history. Khmer is one of the oldest written languages from the Mon-Khmer family, and was used during historical empires of Chenla, Angkor, and possibly even Funan a thousand years earlier [14]. There are four periods in the history of the Khmer language, the Old Khmer pre-Angkorian period (from 6th through early 9th centuries), the Old Khmer Angkorian period (from 9th to 13th centuries), the Middle Khmer period (from 14th to early 18th centuries), and the Modern Khmer period (from the middle 18th to present), according to [15]. The Khmer that was written on Sleuk-Rith is the Middle period language which is partially different from the current language, Modern Khmer. Fig. 2 illustrates an example of the different spelling words that were written on Sleuk-Rith (Fig. 2.b) from the current language (Fig. 2.c).

The content in Sleuk-Rith consists of a variety of Khmer texts. Even though Khmer alphabet is used, certain scripts are written in the Pali language (a middle Indo-Aryan liturgical language native to the Indian subcontinent [16]), Sanskrit language (an Indo-European language, a member of the Indo-Aryan brand of the Indo-Iranian subgroup of that family [17]), and some in Khmer. Pali is used when the texts are predominantly religious or are a collection of folk tales [11], and Khmer is used instead when the texts are generally impressive variety of didactic poems, short, aphoristic verse compositions. When a palm leaf document ages and decays, their contents are transferred to fresh new leaves [13]. They were traditionally studied, copied, and recited by children studying at Khmer Buddhist temples [18].

# III. METHOD

The proposed method is developed taking into consideration the properties and characteristics of Khmer writing specifically for historical palm leaf manuscripts.

## A. Long Short-Term Memory Encoder-Decoder

The proposed KLM is implemented with a deep learning approach using Long Short-Term Memory (LSTM) in an encoder-decoder architecture. RNN is the state-of-the-art algorithm for processing sequential data in which the information about the order is important. It can be explained by three different layers: an input layer, hidden layers, and an output layer (Fig. 3). The input layer $x_t$ refers to the initial state where data is input. The hidden layers $h_t$ are the

<figure><img src="image_5.png" /><figcaption>Fig. 3. Recurrent neural network architecture</figcaption></figure>

combination of the current input $x_t$ with the previous output $h_{t-1}$. The output layer $y_t$ is the final result of the model.

A simple RNN has a short-term memory, which is extremely forgetful and reduce its effectiveness in most tasks. Contrarily, Long Short-Term Memory (LSTM) can keep contexts longer and can recall prior information beneficial to the model in predicting the current output. LSTMs enable RNNs to retain inputs for an extended length of time which in turn is able to produce more accurate results.

However, to enable the model to predict words based on their context, an encoder-decoder algorithm is added to the model. This mechanism is used to solve sequence-to-sequence prediction problems by utilizing two RNNs. The first RNN, which is called encoder, is used to encode the input sequence $x_t$ to the intermediate context vector $X$. Then the second RNN, decoder, decodes the encoded sequence or intermediate vector $X$ to the target sequence $y_t$ (Fig. 6).

## B. Generation of Word Pairs

Four operations are used to generate word input-output pairs to train and to evaluate the KLM: insertion, deletion, substitution, and copy. The first three operation are primitive editing operations which are used to synthetically create words with spelling errors. The copy operation is used to enable the model to recognize a correct word with no spelling mistakes and to make no change to the input word. For example, Fig. 4 shows three different wrong words and a copied word which are generated after applying these four operations to a Khmer word “សូស្តី” (which means hello in English).

To generate the word pairs for training, one of these four operations is randomly selected to apply to an original word.

<figure><img src="image_6.png" /><figcaption>Fig. 4. Four operations used to generate input-out word pairs</figcaption></figure>
<figure><img src="image_7.png" /><figcaption>Fig. 5. Overview of the architecture of the proposed model</figcaption></figure>

We apply this process repeatedly to create a word pairs until all words in the dataset are processed.

### C. Proposed Model

The goal of this research is to develop a Khmer Language Model (KLM) that can correct the misspelling words, and the model is to be used to support the text recognition system in research of Khmer Palm Leaf Manuscripts digitization [1]. The proposed model is shown in a simple diagram below (Fig. 5) with an example of a Khmer word “ស្កាត្ក”. The KLM is constructed using RNNs and LSTMs to aid in its performance. The model can recall prior outputs by using LSTM, which enables more accurate results.

In this example, the KLM gets an input which is a synthetic wrong word “ស្កាត្ក” created from applying one of the editing operations to the ground truth word “ស្កាត្ក”. The model then encodes the input word and decodes it back to output the ground truth word.

## IV. EXPERIMENTS AND RESULTS

### A. Dataset

The dataset used in the experiments is composed of Khmer synthetic words extracted from 657 annotated pages of Sleuk-Rith Set [13]. The dataset is divided into one training set and two test sets (Test 1 and Test 2) with 557, 50, and 50 pages

respectively. Each synthetic word is constructed by merging consecutive sub-syllables to form a sub-syllable group where each group contains 5 sub-syllables. The construction of sub-syllable group is explained in detail in [1].

Table I shows the number of pages and words according to each sub set.

<table><caption>TABLE I. NUMBER OF PAGES AND WORDS IN TRAINING AND TEST SETS</caption><thead><tr><th></th><th>Train</th><th>Test 1</th><th>Test 2</th></tr></thead><tbody><tr><td>Pages</td><td>557</td><td>50</td><td>50</td></tr><tr><td>Words</td><td>133,589</td><td>12,552</td><td>11,912</td></tr></tbody></table>

### B. Implementation

The KLM is implemented with a deep learning approach by using LSTM modules adopting an encoder-decoder architecture. In addition, this model is trained and tested at character level where each input word is transformed into a sequence of characters. We use 91 distinct characters including 88 of the Khmer characters (codes from 1780 to 17D7) taken from the standard Khmer Unicode [19] and three additional special characters to represent START, END, and UNK (unknown or empty) tokens.

The implementation is composed of two parts:

* Encoder: a small batch of the generated word pairs from the dataset is selected. The wrong word is transformed into an input sequence of characters and then to corresponding sequence of one-hot encoding vectors. Padding is used so that the words in each batch are of the same length. The encoder of the KLM outputs an intermediate context vector X for each

<figure><img src="image_8.png" /><figcaption>Fig. 6. Complete pipeline of the encoder-decoder architecture (an example of an input word ស្កាត្ក encoded by the encoder to predict the correct word ស្កាត្ក decoded by the decoder)</figcaption></figure>
input word, and it will be used as an input to the
decoder in the next step.

* Decoder: in this step, the model takes the intermediate vector *X* generated from the encoder as the initial hidden vector concatenate with a special character START as the first input, then it produces an output *y*₁ and a hidden state *h*₁. After that, the model takes the hidden vector *h*₁ with the previous output *y*₁ as the next input then it produces a new output *y*₂ and another hidden state *h*₂. It repeats this process until it meets a special character END. Finally, KLM ends its job here with the output sequence representing a list of the character of the predicted correct word. The overall encoder-decoder architecture is shown in Fig. 6. During training, we adopt a teacher forcing technique where the sequence of characters from the ground truth is passed directly to the decoder to fasten its learning.

After empirical fine-tuning, the following configurations
are used in the experiments:

* batch size of 200 words

* training epochs of 100 epochs

* Adam optimizer [20] with default learning rate

The model is implemented using Pytorch, and is trained
using a machine equipped with Nvidia GeForce RTX 3080.

C. Evaluation Method

The model performance is evaluated by measuring the
error rate of its prediction. Two metrics are used: character
error rate (CER) and word error rate (WER). CER is computed
by summing the minimum edit distance (Levenshtein
distance) and divide it with the total number of characters of
all words in the test set. On the other hand, WER is computed
by comparing the number of incorrectly predicted words to the
number of all testing words.

D. Experimental Results

The evaluation results conducted on the two test sets are
illustrated in Table II. We experiment with 3 different
numbers of hidden units for the LSTM layers in the encoder
and the decoder of the proposed model.

<table><caption>TABLE II. EXPERIMENTAL RESULTS</caption>
  <thead>
    <tr>
      <th></th>
      <th>Number of Hidden Units (N)</th>
      <th>N = 64</th>
      <th>N = 128</th>
      <th>N = 256</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td rowspan="2">Test 1</td>
      <td>CER (%)</td>
      <td>28.34</td>
      <td>12.82</td>
      <td>32.91</td>
    </tr>
    <tr>
      <td>WER (%)</td>
      <td>75.42</td>
      <td>57.97</td>
      <td>80.08</td>
    </tr>
    <tr>
      <td rowspan="2">Test 2</td>
      <td>CER (%)</td>
      <td>28.06</td>
      <td>12.26</td>
      <td>32.15</td>
    </tr>
    <tr>
      <td>WER (%)</td>
      <td>75.14</td>
      <td>57.22</td>
      <td>79.62</td>
    </tr>
  </tbody>
</table>

Due to the random nature of the generation of the synthetic
wrong words, 10 different evaluation experiments are
conducted. From the 10 experiments, a mean CER and WER
are reported (Table II). By using the encoder-decoder
mechanism with LSTM, the KLM is able to capture the
context as well as the sequential information of the input
words, to detect the errors, and finally to output the correct
words sufficiently well. According to the results, selecting 128

as the number of hidden units of LSTM modules of both the
encoder and decoder provides the best performance.

V. CONCLUSION

Language Models (LM) are critical components of NLP
tasks. In this work, a KLM is developed to address misspelling
correction concerns by utilizing a deep learning approach with
LSTM consisting of two modules: encoder and decoder.
Trained with a dataset containing over 133K words extracted
from 557 pages of Sleuk-Rith pages, the model is able to
performs sufficiently well in detecting misspelling and
generating corresponding correct words. The model is
evaluated on two test sets containing over 10K words each,
and the lowest (CER, WER) pair (12.82%, 57.97%) and
(12.26%, 57.22%) respectively for the two test sets Test 1 and
Test 2 are obtained. The proposed KLM is to be incorporated
with a text recognition system so that its performance can be
improved. Also, alternative language model approaches such
as transformer are to be considered and will be tackled in our
future work.

ACKNOWLEDGMENT

The work was funded by Cambodia Higher Education
Improvement Project (Credit No. 6221-KH).

REFERENCES

[1] D. Valy, M. Verleysen and S. Chhun, "Data Augmentation and Text Recognition on Khmer Historical Manuscripts," in 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), 2020.

[2] J. Howard and S. Ruder, "Universal language model fine-tuning for text classification," *arXiv preprint arXiv:1801.06146*, 2018.

[3] S.-H. Chiu and B. Chen, "Innovative BERT-based reranking language models for speech recognition," in 2021 IEEE Spoken Language Technology Workshop (SLT), 2021.

[4] J. Andreas, A. Vlachos and S. Clark, "Semantic parsing as machine translation," in *Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, 2013.

[5] H. Schmid, "Part-of-speech tagging with neural networks," *arXiv preprint cmp-lg/9410018*, 1994.

[6] N. Aussel, Y. Petetin and S. Chabridon, "Improving performances of log mining for anomaly prediction through nlp-based log parsing," in 2018 IEEE 26th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), 2018.

[7] L. Lyu, M. Koutraki, M. Krickl and B. Fetahu, "Neural OCR post-hoc correction of historical corpora," *Transactions of the Association for Computational Linguistics*, vol. 9, p. 479–493, 2021.

[8] V. Pham, T. Bluche, C. Kermorvant and J. Louradour, "Dropout improves recurrent neural networks for handwriting recognition," in 2014 14th international conference on frontiers in handwriting recognition, 2014.

[9] D. U. Kumar, G. Sreekumar and U. Athvankar, "Traditional writing system in southern India—palm leaf manuscripts," *Design Thoughts*, vol. 7, p. 2–7, 2009.

[10] J. Igunma, "Aksoon Khoom: Khmer Heritage in Thai and Lao Manuscript Cultures," *Tai Culture*, vol. 13, p. 25–32, 2013.

[11] J. Dean, "The preservation of books and manuscripts in Cambodia," *The American Archivist*, vol. 53, p. 282–297, 1990.

[12] D. Valy, M. Verleysen, S. Chhun and J.-C. Burie, "Character and text recognition of khmer historical palm leaf manuscripts," in *2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR)*, 2018.

[13] D. Valy, M. Verleysen, S. Chhun and J.-C. Burie, "A new khmer palm leaf manuscript dataset for document analysis and recognition:
Sleukrith set," in *Proceedings of the 4th International Workshop on Historical Document Imaging and Processing*, 2017.

[14] J. J. Jacobs and D. Smyth, Cambodian Linguistics, Literature and History: Collected Articles, Routledge, 2013.

[15] C. Sak-Humphry, Syntax of Nouns and Noun Phrases in Dated Pre-Angkorian Inscriptions, University of Hawai'i at Manoa, 1992.

[16] "Pali," [Online]. Available: https://en.wikipedia.org/w/index.php?title=Palioldid=1031475499.

[17] "Sanskrit inscription of ancient cambodia," [Online]. Available: https://www.iskconcambodia.com/2013/01/18/sanskrit-inscription-of-ancient-cambodia.

[18] T. Walker, "Treasures from cambodia," [Online]. Available: https://www.bdrc.io/blog/2020/09/30/treasures-from-cambodia/?lang=bo.

[19] J. D. Allen, D. Anderson, J. Becker, R. Cook, M. Davis, P. Edberg, M. Everson, A. Freytag, L. Iancu and R. Ishida, "The unicode standard," *Mountain view, CA*, p. 660–664, 2012.

[20] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," *arXiv preprint arXiv:1412.6980*, 2014.