Khmer Word Segmentation Using
Conditional Random Fields
Vichet Chea*† , Ye Kyaw Thu† , Chenchen Ding† , Masao Utiyama† , Andrew Finch† , Eiichiro Sumita†*

Research and Development Center, NIPTICT, Phnom Penh, Cambodia
E-mail: vichet.chea@niptict.edu.kh†
Advanced Speech Translation Research and Development Promotion Center, NICT, Kyoto, Japan
E-mail: {yekyawthu, chenchen.ding, mutiyama, eiichiro.sumita}@nict.go.jp

Abstract—Word Segmentation is a critical task thatis the foundation of much natural language processing
research. This paper is a study of Khmer word segmentation using an approach based on conditional random
fields (CRFs). A large manually-segmented corpus wasdeveloped to train the segmenter, and we provide detailsof a set of word segmentation strategies that were usedby the human annotators during the manual annotation.
The trained CRF segmenter was compared empirically toa baseline approach based on maximum matching thatused a dictionary extracted from the manually segmented
corpus. The CRF segmenter outperformed the baseline interms of precision, recall and f-score by a wide margin.
The segmenter was also evaluated as a pre-processing stepin a statistical machine translation system. It gave rise to
substantial increases in BLEU score of up to 7.7 points,
relative to a maximum matching baseline.

I. I
In the writing system of the Khmer language, spaces
are not used to separate words, but spaces are used
occasionally for easier reading. There are no standard
rules for using spaces in the Khmer writing system.
These large contiguous blocks of unsegmented words
can cause major problems for natural language processing applications such as machine translation, speechsynthesis, information extraction, and therefore word
segmentation techniques need to be developed. Although
Khmer native speakers are easily able to determine
the the positions of word boundaries, developing an
automatic word segmentation is not a trivial task.
Word boundary ambiguities have two main causes.
The first one is concerned with the lexical semantics. A
single sentence can be segmented in several ways basedon its meaning in context. For example:- ខញុំ ចង់ឱយ 〈 អនក��ប់ 〉 យល់ ពី ប � េនះ- I want listener to understand this problem- ខញុំ ចង់ឱយ 〈 អនក 〉 〈 ��ប់ 〉 យល់ ពី ប � េនះ- I want you to listen in order to understand thisproblem
The second cause is unknown words. Unknown words
are words that are not found in dictionaries or training
data and are often named entities such as personal names
and locations.
An overview of the contents of the paper is a follows.
The CRF++ toolkit [1] was used to build a CRF [2]models to learn the word formation patterns of Khmer
words. Training data for the CRF and maximum matching baseline model was manually segmented based onfour sets of segmentation patterns based on word types: a
set of patterns for a single words (Section III-A); patterns
for compound words (Section III-B); patterns for prefixwords (Section III-C); and patterns for suffix words (Section III-D). Section IV describes the maximum matchingbaseline approach, and Sections V and VI present experiments on segmentation quality and statistical machinetranslation respectively. Finally, Section VII concludes.
II. R

W

The first published word segmentation approach to
Khmer word segmentation was presented in [3]. Theirmethod was developed for use with a speech recognition
system, and was based on the method of longest matching which is in essence a sub-optimal greedy versionof the maximum matching method used in this paper
as a baseline. [4] proposed the word segmentation of
Khmer written text based on a combination of dictionary
matching and a bi-gram model. First, the input sentence
was segmented into compound orthographic symbols
called Khmer Character Clusters (KCC) and converted
into Khmer Common Expressions (KCE) [5]. Then,dictionary look up is done using KCE to produce thepossible word segmentation hypotheses for the original

input text. Finally a bigram model is applied to resolve
segmentation ambiguities. Bigram models over KCC
units and words were studied. Word bigrams proved to be
the most effective, achieving 91.56% precision 92.14%
recall and 91.85% F-score.
In [6] a rule-based approach was obtained by statistical analysis to tackle the issue of out-of-vocabulary
words, and to detect compound words, proper names,
acronyms, derivatives words and new words for Khmer
word segmentation. Their experimental study showed
their proposed approach to be superior to a baseline
based on [4]. Specifically, the reported precision was
77.7% (5.9% higher), the recall was 75.5% (3.3% lower),and the f-score was 76.5% (1.4% higher).
[7] proposed a Khmer word segmentation methodthat used a Bi-Directional Maximal Matching (BiMM)
approach. The study also focused on how to implement
Khmer word segmentation on both Khmer plain text and
Khmer in Microsoft word documents. Their approachachieved a segmentation accuracy of 98.13%.
III. KW

S

As in most other languages, the Khmer vocabulary
consists of single words and compound words. A single
word is a word that is not composed from other words,
while a compound word is composed of two or more single words, prefixes or/and suffixes. Word concatenation
has often been used in language as a means of creating a
new word that draws on the semantics of its components.
Four classes of segment (word) types were observed
during the manual segmentation of the corpus of Khmer
text, each representing a different type of word, thesewere:• Word Type 1: Single Words• Word Type 2: Compound Words• Word Type 3: Compound Words with Prefix• Word Type 4: Compound Words with Suffix①

Samlor Bang Prohok②

Samlok Bang Prohor③
Samlorsamlok

Fig. 1. Formation of a word using a borivasab.

are rhythm words used to make the spoken language
comfortable for listener. Khmer people have a habitof speaking a phrase or word by interchanging syllables. An example of the formation of a word using a
borivasab is given in Figure 1: the word is សម�សម�ុក
pronounced as Samlorsamlok. The root of this word
comes from the phrase សម�បង់�បហុក (pronounced Samlor
Bang Prohok). It means that soup cooked with Prohok.
Prohok is the name of Khmer cheese. In this case,
Samlor Bang Prohok the final syllables of the first andlast words are interchanged ( 1 in Figure 1) Samlok
Bang Prohor; forming the borivasab Samlok in the first
word. Then, the pseudocompound word Samlorsamlok
is formed by combining the main word in the originalphrase Samlor ( 1 ) with the borivasab ( 3 ). This kind ofword formation process can be used freely at anytime inthe daily speaking and writing of Khmer, and eventuallythese words can enter common usage even they are notfound in the dictionary.
Another example of a borivasab is shown below:- កេមទចកមទី (debris) = kamtickamteeកេមទច ជ ធូលី (bits of dust) = kamtic chea tulee−→ កមទី ជ ធូលចិ = kamtee chea tulic

B. Word Type 2: Compound Words
The word segmentation was annotated using the tagset
used to train the CRF models shown in Table VI withthe exception of the ’0’ tag which remained implicit.
A. Word Type 1: Single Words
In Khmer language, a borivasab (បរ�ស័ិ ព)ទ [8] is subordinate word that is combined with a main word to forma single word. Words formed using borivasab are classed
as pseudocompound words [9] because the subordinate
word has no meaning in isolation.
[10] describes borivasab as entourage words that areused to facilitate verbal speaking and sometimes these

A compound word is a word that is composed of two
or more single words. Most compound words can beclassified into 7 patterns as in Table I. Here, N denotes
Noun, Adj denotes Adjective, and V denotes Verb.
For the remaining cases, that we call collocation
words, two or more words that are always written or used
next to each other, are also considered to be compoundwords, and some examples are given in Table II.
C. Word Type 3: Compound Word with Prefix
The most 12 frequent words in the corpus were used asprefixes in the formation of compound words. They canTABLE IW
PCPatterns
N_N
N_Adj
N_V
Adj_Adj
Adj_N
V_V
V_N

G

ExamplePrefix Group�ន_�កង (city_car→bus)

Group Oneកំហុស_ឆគង (mistake_improper→mistake)កប៉ល់_េ�ះ (ship_fly→airplane)
ខូច_ខត (broken_lose→damage)�ត_ី េកណ (three_angle→triangle)
ផ�ល_់ ឱយ (provide_give→provide)ពេិ �គះ_េយបល់ (consult_optinion→to consult)

E

TABLE IIC

Group Two
W

ExamplesExplanation and Definition

ក៏_ប៉ុែន�

or_but → but

ខញុំ_បទ
នង_ខញុំ

េ�យ�រ_ែតផង_ែដរ

ទទួល_ខុស_�តវ
ឥឡូវ_េនះ

CGroup Three

I_yes → I (for man)

TABLE III
W

P

Examples- ករ + អប់រ ំ (educate)→ ករ~អប់រ ំ(education)- េសចក�ី + សុខ (fine)→ េសចក�ី~សុខ (happiness)- ភព + ឯេក (lonely)→ ភព~ឯេក (loneliness)- អតថ (idea) + ន័យ (meaning)→ អតថ~ន័យ (definition)- �ពះ (god) + �ទតិ យ (sun)→ �ពះ~�ទិតយ (the sun)- អនក (person) + េបីកបរ (to drive)→ អនក~េបកី បរ (driver)- អ (not) + សកមម (active)→ អ~សកមម (inactive)

she_I → I (for woman)because_only → becauseC

also_also → as well

TABLE IVW

S

receive_wrong_right → be responsible for

SuffixExamplenow_this → nowកិចច- អភិបល (govern) + កិចច (task)→ អភបិ ល^កច
ិ ច (governance)

be partitioned into three groups. The first group contains
eight prefixes ករ, ក�,ី ភព, េសចក�,ី អត�, អតថ, អនុ, អំេពី formaking compound nouns. The second group contains
three prefixes �ពះ, េ�ក, អនក that are used for nouns
representing humans and deities. The third contains only
one prefix អ for negation. Table III shows examples ofcompound words constructed with prefixes from eachgroup.
D. Word Type 4: Compound Words with Suffix
Only 3 words were used as suffices of compoundwords. They are កិចច, កមម, ភព and examples of compoundwords formed with them are shown in Table IV.
Some rare compound words are formed by both prefix
and suffix. For example, អ~ទទួល_មរណ^ភព (death).
E. Annotation Tags
When words of these four types were identified in
the corpus by the annotators, they were considered as asingle token of segmented text and tagged according to
the following scheme:{} a pair of bracket characters were used for wordboundaries;_ the underline character was used to delimit thecomponent words within compound words;កមមភព- ទំេនីប (moden) + កមម (action)→ ទំេនីប^កមម (modernization)- ឯក�ជយ (independent) + ភព (state/condition)→ ឯក�ជយ^ភព (independence)~ the tilde symbol was used to indicate prefix words;^ the caret symbol was used to indicate suffix words.
The following is an example of some Khmer text withword boundary annotation:{សុខ}{និយយ}{ថ}{៖}{«}{ករ~សិក�}{នំ_មក}{នូវ}{ចំេណះ_ដឹង}{»}{េតី}{ឯក^ភព}{េទ}{?}
IV. M

M

(MM)

Maximum matching is one of the most popular structural segmentation algorithms and it is often used as abaseline method in word segmentation [11]. This method
segments using segments chosen from a dictionary. The
method strives to segment using the longest possible
segments. It is a greedy algorithm and is therefore suboptimal. The segmentation process may start from either
end of the sequences. We ran the maximum matchingexperiments with a dictionary that contained 27,070
unique words extracted from the manually annotatedcorpus used to train the CRF models. This was a setof every segment that occurred in the corpus.
V. S

E

A. Data Setup
1) Training Data: In order to train the CRF segmentation models, we needed to construct a manuallysegmented corpus in accordance with the guidelines set
out in Section III. This corpus was constructed overa six-month period using 4 human annotators in thefollowing manner. First, manual word segmentation was
done on 5,000 randomly selected sentences from the general Khmer web domain. For manual word segmentation,
segmented according to four types of Khmer words as
explained in Section III. We used the CRF++ toolkit [1]to build the CRF model with the 5,000 segmented Khmer
sentences. We then used this CRF model to annotate a
new set of unsegmented data. This automatic annotationof this data was hand-corrected and used to train a new
CRF model. In this manner the annotation for the full
97,340-sentence corpus was bootstrapped.
The training data set included 3,435 sentences from
Agriculture domain, 67,725 sentences from the Basic
Travel Expression (BTEC) corpus [12], 2,915 sentencesfrom the Buddhist religious domain, 2,052 sentencesfrom the economic domain, 126 sentences from the
medical domain, 98 sentences from the history domain,
665 sentences from law, 746 sentences from the management domain, 9,923 sentences from the news domain,
3,284 sentences from the scientific research domain,
6,009 sentences from stories, and 362 sentences frommiscellaneous other domains. All of the training data
was web data, except for the BTEC or Travel domaindata.
2) Test Data: The 12,468-sentence test set was randomly selected from the full corpus, and consistedof: 490 sentences (14,879 words) from the agriculture
domain, 9,989 sentences (75,902 words) from BTEC,
1,400 sentences (34,371 words) from the history domain,
393 sentences (12,488 words) from the news domain, 90
sentences (1,775 words) from stories, and 106 sentences(2,921 words) from miscellaneous other domains.
B. CRF Training
The feature set used in the models (character unigrams) was as follows (where t is the index of thecharacter being labeled):{wt−2 , wt−1 , wt , wt+1 , wt+2 }These n-grams were combined with label unigrams to
produce the feature set for the model.

T

TABLE V
T

R

No

Tags

Meaning

1

C

Consonant

2

V

Vowel

C

Charactersកខគឃងចឆជឈញដឋ
ឌឍណតថទធនបផពភ
មយរលវឝឞសហឡអ◌ា ◌ិ ◌ី ◌ឹ ◌ឺ ◌ុ ◌ូ ◌ួ េ◌ីេ◌ េ◌ េ◌ ែ◌ ៃ◌ េ◌ េ◌◌ំ ◌ះ ◌ៈ

3

IV

Independence
Vowel

4

US

Upper Sign

5

AN

Atak
Number(astrology)

6

LN

Lunar Number

SUB

Subscript
Sign

8

END

End
of
sentence
signs etc.

9NS

No Space

10

UNK

UnknownR◌៉ ◌៊ ◌់ ◌៌ ◌៍ ◌៎ ◌៏ ◌័ ◌៑៰» ៱៲៳៴៵៶៷៸៹

᧠᧡᧢᧣᧤᧥᧦᧧᧨᧩᧪᧫
᧬᧭᧮᧯᧰᧱᧲᧳᧴᧵᧶
᧷᧸᧹᧺᧻᧼᧽᧾᧿
◌្

7Tឤឥឦឧឩឳឰឬឫឭឮឯឱ
ឲឪ◌៓ ។ ៕ ៖ ៗ ៘ ៙ ៚ ៛ ៜ ៝០១២៣៤៥៦៧៨៩01
23456789
Characters outside the Khmer
unicode set

TABLE VI
S

No

Tag

Meaning after corresponding character10

Zero: no space2}{

Right and Left brace: Space

3_

Underscore sign: compound word4~

Tilde sign: prefix

5^

Caret sign: suffix

The CRF models were trained on character segmented
Khmer. We used 10 tags for type of Khmer charactersand they are C (Consonant), V (Vowel), IV (Independent
Vowel), US (Upper Sign), AN (Atak Number), SUB(Subscript Sign), END (End of Sentence), NS (NoSpace), UNK (Unknown) (Refer Table V). Two separate
models that used two different tag sets were trained.
These two tag sets were: {1,2} and {1,2,3,4,5} using
the tag number notation in Table VI.

TABLE VII
CRF

W
Maximum Matching
Precision Recall F-score

CRF with 2 tags
Precision Recall F-score

Type of data

1

Agriculture

0.921

0.934

0.927

0.993

0.981

0.987

0.989

0.992

0.991

2

BTEC

0.913

0.919

0.916

0.990

0.992

0.991

0.989

0.992

0.991

3

History

0.881

0.940

0.910

0.959

0.962

0.961

0.958

0.963

0.961

4

News

0.894

0.934

0.914

0.986

0.981

0.983

0.985

0.982

0.984

5

Story

0.928

0.933

0.930

0.996

0.986

0.991

0.996

0.994

0.995

6

Others

0.926

0.916

0.921

0.995

0.994

0.994

0.997

0.984

0.991

AVERAGE

0.910

0.929

0.920

0.986

0.983

0.985

0.986

0.983

0.985

The evaluation method used was the Edit Distanceof the Word Separator (EDWS) [13]. As an example,
given a non-segmented Khmer sentence (ខញុំេឈមះសុីថ)represented as a sequence of 15 characters:ខ ◌្ ញ ◌ុ ◌ំ ឈ ◌្ ម េ◌ ◌ះ ស ◌៊ ◌ីិ ថ ◌ាThe CRF segments this sentence into a sequenceof words, and we indicate the segment boundaries by
the space character ’ ’. For example, an output mightbe: ខញុំ េឈមះ សុី ថ. Let X = x1 x2 x3 ...xn represent an
unsegmented string of characters. A segmented string(with n = 15) may look like:x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 x14 x15
The EDWS measures how many edit operations (insertions, deletions and substitutions) are needed to jointly
segment a given segmentation and a reference segmentation. Here the edit operations are based only on thesegmentation token ’ ’: a substitution ⟨ , ⟩ being a correct segmentation point, insertions ⟨ , ∅⟩ corresponding
to segmentation points in the reference that are not in thegiven segmentation, and deletions ⟨∅, ⟩ corresponding
to segmentation points in the given segmentation that are
not in the reference. For example:

CRF:
Ref:
CRF:

CRF with 5 tags
Precision Recall F-score

No

C. Evaluation

Ref:

.

ខញុំ េឈមះ សុីថខញុំ េឈមះ សុី ថx1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 x14 x15x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 x14 x15

In this example there are two substitutions after x5
and x10 , and one deletion after x13 . The segmentation
precision, recall and harmonic mean F-score are definedas:Precision (P ) = N
H
Recall (R)
= N
S (
)
F-score= 2 × PP ×R+R
where:
N is the number of substitutions;
H is the number of separators in the hypothesis;
S is the number of separators in the reference.

D. Results and Discussion
Table VII presents the main results of our segmentation evaluation. It is clear from the results that there
were almost no difference in the performance of the CRF
systems built using a 2-tag set and those that used a 5-tag
set.
It can also be seen that the CRF models substantiallyoutperform the MM systems in terms of the overall
precision, recall and F-score. We believe that much ofthis difference is caused by out-of-vocabulary (OOV)
words such personal names, location names, foreign
words.
In order to study how the CRF models behave withvarying amounts of training data, we ran a sequence ofexperiments that trained two CRF models (2-tag and 5tags) on 10K, 20K, 30K, 40K, 50K, 60K, 70K, 80K and
90K sentences sampled randomly without replacementfrom the full manually segmented corpus. The results areshown in Figure 2. 10-fold jackknifing experiments were
performed for each data point on the graph. It is clear
from the figure that both of the CRF models perform
almost identically, with the 5-tags model is slightly
outperforming the 2-tags CRF model. We measured thelabeling quality of 3 tags (prefix, suffix, compoundword) for the 5-tag CRF model on all 6 domains.

1) OOV: The following is an example of an OOVsegmentation error:
Gloss
Ref
MM
CRF

Fig. 2. F-Scores from training with CRF models on varying data set
sizes.

E

P,S

PrefixSuffixCompound word

TABLE VIII,C

W

Precision

RecallF-score

0.911
0.975
0.923

0.958
0.814
0.896

0.934
0.887
0.909

The result in VIII shows f-scores of prefix, suffix, andcompound word labeling, 0.934, 0.887, and 0.909 respectively. Noticeably, we got lower recall than precision
on both suffix and compound word labels because only
three suffices were defined in our segmentation rule set
and there are a lot of ambiguities among compoundwords.
When we inspected the word segmentation errors of
the CRF model, we found two common errors: OOV errors and compound word errors. We analyzed the resultsin order to measure how well each of the approaches
dealt with these cases. In each case we measured theaccuracy of the segmentation for words of the respective
types. Explicitly, in the case of OOVs, for each OOVin the reference the accuracy is given by the ratio oftimes the word is correctly segmented to the numberof occurrences in the reference set. We found that the
maximum matching method was unable to segment anyof the OOVs correctly, this was because if a word is
not in the dictionary, the default character segmentation
was used. The CRF model had an OOV segmentation
accuracy of 0.44. The CRF also showed much better
performance on segmenting compound words, here theaccuracy was 0.88, compared to only 0.57 with the
maximum matching method.

story

AmericanAirlineេរ ង�េមរេខន
ិែអ៊រ�ញេរ ងេរ ង�ម េ◌ រ ◌ិ េខន�េមរេខន
ិែអ៊រ�ញ

ែអ៊រ�ញ

The word for ’American’ was not in the dictionary
used for maximum matching, and therefore a the word
has been segmented into characters, whereas the CRF
has been able to segment the OOV correctly.
2) Compound words: The following is an example ofan compound word segmentation error:
Gloss
Ref
MM
CRF

these

have

mistake

a lot

very

ទំងេនះ

មនភពខុសឆគង

�ស់ទំងេនះ

មន

េ�ចីន
េ�ចន
ី

ទំងេនះ

មនភពខុស ឆគងភពខុសឆគង

េ�ចីន

�ស់

�ស់

The compound meaning ’mistake’ was not in the
dictionary, however its component words were. The
method of maximum matching has therefore segmented
the word into its component words whereas the CRF hascorrectly segmented the word.
However, for some long compound words, the CRF
model made errors where the maximum matching
method did not. For example, the compound Khmer
word កថខណ� (meaning ’paragraph’ in English) is splitby the CRF into two words as កថ (’word’ in English)and ខណ� (’section’ in English). This error was caused
because the compound word only occurred once in the
training corpus, whereas the components occurred 11
and 207 times respectively. Therefore the CRF had apreference to segment (incorrectly) into the components.
The maximum matching technique on the other hand,segmented (correctly) into the longest possible word.
Numbers written with words also caused ambiguities.
Numbers from five to nine are expressed relative to five.
For example, six (�បំមួយ) is formed from five (�បំ) plus
one (មួយ). If numbers are written in words continuously
without using space (for example for a phone number)they can be segmented in several ways, therefore spaces
must be used to disambiguate. For example the Khmer�បំមួយ is literally ‘five one’, and could the number five(�បំ) followed by the number one (មួយ), or it could be
the number six. This problem can not be solved by themethods studied in this paper.
However, if the numbers contain units such as រយ(hundred) or ពន់ (thousand), then it is possible todisambiguate them, and in fact we observed that the CRF
segmenter was able to learn to do this. Of course themethod based on maximum matching was not able togeneralize to be able to handle this issue.
VI. M

T

E

A. Corpora
We used a selection of five language pairs from themultilingual Basic Travel Expressions Corpus (BTEC),
which is a collection of travel-related expressions [12].
The language pairs were selected to include languages
from a variety of language groups. The languages were
Chinese (zh), English (en), Japanese (ja), Khmer (km),Myanmar (my), and Vietnamese (vi). 155,121 sentenceswere used for training, 5,000 sentences for development
and 2,000 sentences for evaluation.
B. Methodology
We used the phrase based SMT system provided
by the Moses toolkit [14] for training the phrasebased machine statistical translation system. The Khmer
was aligned with the word segmented target languages(except for the Myanmar language that was syllable
segmented) using GIZA++ [15]. The alignment wassymmetrized by grow-diag-final-and heuristic [16]. The
lexicalized reordering model was trained with the msdbidirectional-fe option [17]. We use SRILM for training
the 5-gram language model with interpolated modified Kneser-Ney discounting [18]. Minimum error rate
training (MERT) [19] was used to tune the decoder
parameters and the decoding was done using the Moses
decoder (version 2.1) [14].
C. Segmentation Schemes
Four types of segmentation were used in these experiments. One baseline segmentation scheme was obtainedusing the syllable segmentation scheme described in [20]which is similar to the KCC scheme mentioned earlier.
Two baselines based on segmentation derived using
the method of maximum matching were also studied:the first used a publicly available dictionary [8] which
represents the current state-of-the-art in Khmer word
segmentation; the second was based on the dictionary
composed of the full set of types in the manuallysegmented corpus we created. The final segmentation
scheme was that arising from the proposed method; that
it was the segmentation from a CRF segmenter trainedon the manually segmented corpus using a 5-tag tag set.

T

K

TABLE IX
(BLEU).

SyllableMM(p-dict)MM(c-dict)

CRF(5 tags)

49.07
23.46
27.43
45.67
23.72

32.35
22.36
21.63
29.56
16.78

51.82
29.96
33.61
46.59
28.40

59.51
34.27
38.08
53.39
32.09

km-en
km-ja
km-my
km-vi
km-zh

T

K

TABLE X
(BLEU).

SyllableMM(p-dict)MM(c-dict)

CRF(5 tags)

49.86
32.57
33.82
47.93
32.21

48.08
33.55
31.49
44.14
32.25

56.38
37.44
36.89
53.13
37.61

58.85
38.49
38.25
54.26
39.20

en-km
ja-km
my-km
vi-km
zh-km

For the experiments in which translation was into
Khmer, the output was re-segmented into sequencesof syllables before evaluation in order to keep thesegmentation the same, and therefore the BLEU scorescomparable.
D. Results
The results of the machine translation experiment are
shown in Tables IX and X. In this table, ’MM(p-dict)’
denotes the maximum matching method based on the
publicly available Khmer dictionary [8]. ’MM(c-dict)’refers to the maximum matching method based on the
dictionary extracted from the human segmented corpus
used to train the CRF model. The highest performing
systems are indicated in bold font. The proposed CRF
word segmentation scheme gave rise to the highest
BLEU scores in all experiments, the improvements in
BLEU were substantial, ranging from 1.1 to 7.7 BLEU
points. The differences between the best baseline, MM(cdict), and the proposed CRF method were tested forsignificance using the paired bootstrap method [21]. All
differences were significant (p < 0.01).
VII. CIn this paper we proposed a segmentation method
for the Khmer language based on a supervised CRFbased segmentation method. In order to train the segmenter we constructed a large word-segmented corpus
of Khmer text. We evaluated the performance of oursegmenter in terms of both segmentation quality andalso in terms of its effect when applied to statisticalmachine translation. The experiments show it is possible
to segment very precisely using the word segmentation
scheme that we defined. The proposed method achieved
an average f-score of 0.99 on test data, exceeding theperformance of a maximum matching baseline which
achieved an average f-score of 0.92. Furthermore, ourexperiments on machine translation show that the highlevels of segmentation quality can be translated into tolarge improvements in end-to-end performance of realworld NLP application. In a set of statistical machinetranslation tasks, the segmenter was able to improvesystem performance over the best maximum matchingbaseline by a wide margin (from 1.1 to 7.7 BLEU
points).
In summary the primary conclusions that can be
drawn from the work in this paper are that accurate
word segmentation looks likely to be a necessary prerequisite for natural language processing in the Khmer
language. Simple segmentation strategies based on maximum matching do not appear to be sufficient. We havedemonstrated that it is possible to achieve a high level ofsegmentation performance on Khmer with a supervised
CRF model.
A
We would like to express our gratitude to our colleagues Mr. Mech Nan, Mr. Sorn Kea and Mr. Tep Raksa
from National Institute of Posts, Telecommunications
and Information Communication Technology, Cambodia
for their help in segmenting 97,340 sentences of BTEC
Khmer Corpus and General Domain Corpus manually.
R[1] Taku Kudo. Crf++ an open source toolkit for crf (2005),
https://taku910.github.io/crfpp/.
[2] John D. Lafferty, Andrew McCallum, and Fernando C. N.
Pereira. Conditional random fields: Probabilistic models forsegmenting and labeling sequence data. In Proceedings ofthe Eighteenth International Conference on Machine Learning,
ICML ’01, pages 282–289, San Francisco, CA, USA, 2001.
Morgan Kaufmann Publishers Inc.
[3] Sopheap Seng, Sethserey Sam, Laurent Besacier, Brigitte Bigi,
and Eric Castelli. First broadcast news transcription system for khmer language. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco, may 2008. European
Language Resources Association (ELRA). http://www.lrecconf.org/proceedings/lrec2008/.
[4] Chea Sok Huor, Top Rithy, Ros Pich Hemy, and Vann Navy.
Word bigram vs orthographic syllable bigram in khmer word
segmentation.

[5] Chea Sok Huor, Top Righty, Row Pich Hemy, and Vann Navy.
Detection and correction of homophonous error word for khmer
language. In PAN Localization Cambodia, 2006.
[6] Channa Van and Wataru Kameyama. Khmer word segmentation and out-of-vocabulary words detection using collocation
measurement of repeated characters subsequences. GITS/GITI
Research Bulletin, 2012-2013:21–31, 2013.
[7] Narin Bi and Nguonly Taing. Khmer word segmentation basedon bi-directional maximal matching for plaintext and microsoft
word document. In Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA
2014, Chiang Mai, Thailand, December 9-12, 2014, pages 1–9.
IEEE, 2014.
[8] Chuon Nath. Dictionnaire Cambodgien. Edition de L’institut
bouddhique, Phnom Penh, 1967.
[9] Madeline E. Ehrman. Contemporary Cambodian Grammatical
Sketch. US Goverment Printing Office, 1972.
[10] Sisovat Poraksy. The Compound words in Khmer language(បរ�រស័ពទ កនុង ភ� ែខមរ េ�យ សុីសុវតថិ ប៉ូរកខសុី)(in Khmer).
ិ
Phnom Penh Sisowath Poraksy, 1972.
[11] Qiang Tan Yuan Liu and Kun Xu Shen. The Word Segmentation Methods for Chinese Information Processing (inChinese). Quing Hua University Press and Guang Xi Scienceand Technology Press, 1994.
[12] G. Kikui, E. Sumita, T. Takezawa, and S. Yamamoto. Creating
corpora for speech-to-speech translation. In Proceedings of
EUROSPEECH-03, pages 381–384, 2003.
[13] Joy.
Chinese word segmentation evaluation toolkit,
http://projectile.sv.cmu.edu/, 2004.
[14] Philipp Koehn and Barry Haddow. Edinburgh’s Submission to
all Tracks of the WMT2009 Shared Task with Reordering and
Speed Improvements to Moses. In Proceedings of the FourthWorkshop on Statistical Machine Translation, pages 160–164,
2009.
[15] F. J. Och and H. Ney. Improved statistical alignment models.
In ACL00, pages 440–447, Hong Kong, China, 2000.
[16] Philipp Koehn, Franz Josef Och, , and Daniel Marcu. Statisticalphrase-based translation. In In Proceedings of the Human
Language Technology Conference, Edmonton, Canada, 2003.
[17] Christoph Tillmann. A unigram orientation model for statistical machine translation. In Proceedings of HLT-NAACL
2004: Short Papers, HLT-NAACL-Short ’04, pages 101–104,
Stroudsburg, PA, USA, 2004. Association for Computational
Linguistics.
[18] Andreas Stolcke. SRILM - An Extensible Language Modeling
Toolkit. In Proceedings of the International Conference on Spoken Language Processing, volume 2, pages 901–904, Denver,
2002.
[19] Franz J. Och. Minimum error rate training for statisticalmachine translation. In Proceedings of the 41st Meeting of
the Association for Computational Linguistics (ACL 2003),
Sapporo, Japan, 2003.
[20] Ye Kyaw Thu, Vichet Chea, Andrew Finch, Masao Utiyama,and Eiichiro Sumita. A large-scale study of statistical machinetranslation methods for khmer language. In Proceedings of the
29th Pacific Asia Conference on Language, Information andComputation (PACLIC 2015), Shanghai, China, October 2015.
[21] Philipp Koehn. Statistical significance tests for machine translation evaluation. In Dekang Lin and Dekai Wu, editors, Proceedings of EMNLP 2004, pages 388–395, Barcelona, Spain,
July 2004. Association for Computational Linguistics.