Language Resources and Evaluation manuscript No.
(will be inserted by the editor)

A Comparative Study of Dictionaries and Corpora as
Methods for Language Resource Addition
Shinsuke Mori · Graham Neubig

Received: date / Accepted: date

Abstract In this paper, we investigate the relative effect of two strategies forlanguage resource addition for Japanese morphological analysis, a joint task ofword segmentation and part-of-speech tagging. The first strategy is adding entries
to the dictionary and the second is adding annotated sentences to the training
corpus.
The experimental results showed that addition of annotated sentences to thetraining corpus is better than the addition of entries to the dictionary. In particular,
adding annotated sentences is especially efficient when we add new words with
contexts of several real occurrences as partially annotated sentences, i.e. sentencesin which only some words are annotated with word boundary information.
According to this knowledge, we performed real annotation experiments oninvention disclosure texts and observed word segmentation accuracy.
Finally we investigated various language resource addition cases and introduced the notion of non-maleficence, asymmetricity, and additivity of language
resources for a task. In the WS case, we found that language resource addition is
non-maleficent (adding new resources causes no harm in other domains) and sometimes additive (adding new resources helps other domains). We conclude that itis reasonable for us, NLP tool providers, to distribute only one general-domain
model trained from all the language resources we have.
Keywords Partial annotation · Domain adaptation · Dictionary · Word
segmentation · POS tagging · Non-maleficence of language resources
The current paper describes and extends the language resource creation activities, experimental
results, and findings that have previously appeared as an LREC paper (Mori and Neubig, 2014).
Shinsuke Mori
Academic Center for Computing and Media Studies, Kyoto University
Yoshidahonmachi, Sakyo-ku, Kyoto, Japan
Tel.: +81-75-753-7486 Fax: +81-75-753-7475 E-mail: forest@i.kyoto-u.ac.jp
Graham Neubig
Nara Institute of Science and Technology
8916-5 Takayamacho, Ikoma, Nara, Japan
E-mail: neubig@is.naist.jp

2

Shinsuke Mori, Graham Neubig

1 Introduction
The importance of language resources continues to increase in the era of natural
language processing (NLP) based on machine learning (ML) techniques. Well defined annotation standards and rich language resources have enabled us to build
very accurate ML-based analyzers in the general domain. Representative tasks include word segmentation for languages without word boundaries such as Japaneseand Chinese, part-of-speech (POS) tagging, and many others. These are the firststep in NLP for many languages and have a great impact on subsequent processes.
However, while the accuracies are more than 97% for texts from the same domain
as the training corpus, large drops in accuracy are seen when using these models in a different domain. To cope with the problem of adapting to new domains,there are many attempts at semi-supervised training and active learning (Tomanek
and Hahn, 2009; Settles et al, 2008; Sassano, 2002). However, the simple strategies of corpus annotation or dictionary expansion are still highly effective and notunbearably costly. In fact, according to authors’ experience with annotation forJapanese word segmentation, it only took 7 hours × 10 days to annotate 5,000
sentences (about 40 words per sentence) with word-boundary information including two check processes.1 As shown in the subsequent parts of this paper, 5,000
annotated sentences are often enough to achieve large gains in domain adaptation
for sequence labeling.
The 5,000 sentences mentioned above were so-called fully annotated sentences,where all the positions between two characters are annotated with word-boundary
information. Within the context of sequence labeling, however, a variety of resources can be used, including partially annotated sentences and dictionaries. In
contrast to fully annotated sentences, partially annotated sentences lack labels atsome points. These annotated sentences give us information about word use incontext, without requiring the annotator to annotate the full sentence. On theother hand, dictionaries lack context information but are often available at large
scale.
In this paper, we first investigate the relative effect of dictionary expansion
and annotated corpus addition (full annotation and partial annotation) for theJapanese morphological analysis (MA) problem (a joint task of word segmentation
and POS tagging) and the word segmentation problem. Then we present somereal adaptation cases, including invention disclosures and recipe texts. Finally we
introduce a notion of non-maleficence of language resources, which means that theaddition of language resources in a certain domain causes no harm to segmentation
accuracy for another domain.

2 Related Work
The task we have chosen is Japanese MA, a joint task of word segmentation
and POS tagging. Although in Japanese most of the ambiguity in MA lies in
1 In the first check process the annotator focused on words appearing only in the newly
annotated 5,000 sentences. In the second process we divide annotated sentences into several
parts and the annotator checked the differences between the manual annotations of each partand the machine decisions by a model trained on the corpus including the other parts, similarly
to cross validation.

Title Suppressed Due to Excessive Length3

word segmentation,2 in the first half of the experimental evaluation we address
the full MA task for comparison between two standard methods in the field: thejoint sequence-based method (Kudo et al, 2004) and the 2-step pointwise method(Neubig et al, 2011). In the second half, we focus on word segmentation.
After a long history of rule-based MA methods, the first statistical method
was proposed by Nagata (1994). It was based on a hidden Markov model whosestates correspond to POS tags. Then Mori and Kurata (2005) extended it bylexicalizing the states like many works in that era, grouping the word-POS pairs
into clusters inspired by the class-based n-gram model (Brown et al, 1992), andmaking the history length variable like it has been done for a POS tagger in English(Ron et al, 1996). In parallel, Kudo et al (2004) applied conditional random fields(CRFs) (Lafferty et al, 2001) to this task and showed that it achieved better
performance than a POS-based Markov model. This CRF-based method does nothave an unknown word model and large drops in accuracy are seen when it isapplied to a different domain from the training data. For unknown words Nakagawa(2004) and Kruengkrai et al (2009) proposed a word-based model (not CRF-based)
equipped with an unknown word model represented by position-of-character tags
and reported comparable accuracies to the CRF-based method.
Along with the model evolution, the NLP community has been becoming more
and more aware of the importance of language resources. In addition we haveobserved a drastic degradation in word segmentation accuracy in domain adaptation cases. Because most unknown words are nouns and the rest fall into othercontent word categories such as verbs, adjectives, it is less difficult to estimate
the POS given the correct segmentation than word segmentation of sentences including unknown words. Thus the Japanese MA research focus has been shifted
to word segmentation. Given this background Mori and Oda (2009) proposed amethod for referring to the words in a dictionary prepared for humans, not computers. The entries in these dictionaries tend to be compound words but not words
defined according to the annotation standard.3 Another important extension toenlarge language resource availability is CRFs trainable from partially annotatedsentences (Tsuboi et al, 2008). This is applied to Chinese word segmentation capitalizing on so-called natural annotations such as tags in hyper-texts (Yang andVozila, 2014). Because the training time of sequence-based methods tends to belong, a simple method based on pointwise classification has been shown to becomparable to sequence-based methods (Neubig et al, 2011). Since the pointwise
method decides whether there is a word boundary or not between two characterswithout referring to the decisions on the other points, we can train the model frompartially annotated sentences in a straightforward way.4
2 We had run some experiments. BCCWJ consists of six domains. We split each of theminto the test and train. Then we built a model from five training data and tested it on therest of the data in the other domain. When we use Yahoo! QA as test, WS and MA accuracies
are 98.64 and 97.78, respectively. The WS errors are 61.3% of those of MA. When the test isYahoo! blogs, the most difficult domain among six, the accuracies are 96.98 and 95.77, so the
WS errors are 71.4% of those of MA.
3 For example an entry “フラン ス語” (French language) is a combination of “フラン ス”(France) and “語” (language).
4 Note that it is also possible to learn sequence-based models from partial annotations(Tsuboi et al, 2008; Yang and Vozila, 2014), which may provide an increase of accuracy at thecost of an increase in training time (the total time for training CRFs on partially annotateddata scales in the number of words in sentences with at least one annotation, in contrast

4

Shinsuke Mori, Graham Neubig

Fig. 1 Joint MA (a) performs maximization over the entire sequence, while two-step MA (b)maximizes the 4 boundaries and 4 POS tags independently.
Type
1-gram2-gramFeature strings
tj , tj wj , c(wj ), tj c(wj )tj−1 tj , tj−1 tj wj−1 ,tj−1 tj wj , tj−1 tj wj−1 wj

Table 1 Features for the joint model using tags t and words w. c(·) is a mapping function
onto the sequence of one of the six character types mentioned in the body text.

3 Morphological Analysis
Japanese MA takes an unsegmented string of characters xI1 as input, segments it
into morphemes wJ1 , and annotates each morpheme with a part of speech tJ1 . Thiscan be formulated as a two-step process of first segmenting words, then estimating
POS (Ng and Low, 2004; Neubig et al, 2011), or as a single joint process of finding
a morpheme/POS string from unsegmented text (Nagata, 1994; Mori and Kurata,
2005; Kudo et al, 2004; Nakagawa, 2004; Kruengkrai et al, 2009). In this section
we explain these approaches briefly, and contrast their various characteristics.

3.1 Joint Sequence-Based MA
Japanese MA has traditionally used sequence-based models, finding the highestscoring POS sequence for entire sentences as in Fig. 1 (a). The CRF-based method
presented by Kudo et al (2004) is a widely used baseline in this paradigm. CRFs
are trained over segmentation lattices, which allows for the handling of variablelength sequences that occur due to multiple segmentations. The model is able totake into account arbitrary features, as well as the context between neighboringtags.
to the pointwise approach, which scales in the number of annotated words). A comparison
between these two methods is orthogonal to our present goal of comparing dictionary andcorpus addition, and thus we use pointwise predictors in our experiments.

Title Suppressed Due to Excessive Length
Type
Charactern-gramCharactertype
n-gram
WS only
POS only

5

Feature strings
xl , xr , xl−1 xl , xl xr ,xr xr+1 , xl−1 xl xr , xl xr xr+1c(xl ), c(xr )c(xl−1 )c(xl ), c(xl )c(xr ), c(xr )c(xr+1 )c(xl−2 )c(xl−1 )c(xl ), c(xl−1 )c(xl )c(xr )c(xl )c(xr )c(xr+1 ), c(xr )c(xr+1 )c(xr+2 )ls , rs , iswj , c(wj ), djk

Table 2 Features for the two-step model. xl and xr indicate the characters to the left and rightof the word boundary or word wj in question. For example, when estimating word boundarybi , l = i − 1 and r = i. ls , rs , and is represent the features expressing whether a word in the
dictionary exists to the left, right, or spanning the current boundary, while djk indicates thattag k exists in the dictionary for word j.

The main feature of this approach in the context of the current paper is thatit relies heavily on a complete and accurate dictionary. In general when building
the lattice of candidates from which to choose, it is common to consider only
candidates that are in a pre-defined dictionary, only adding character sequencesthat are not in the dictionary when there are no in-vocabulary candidates.5 Thus,
if the dictionary contains all of the words present in the sentences we want toanalyze, these methods will obtain relatively high accuracy, but any words notincluded in the dictionary will almost certainly be given a mistaken analysis.
We follow (Kudo et al, 2004) in defining our feature set, as summarized inTable 16 . Lexical features were trained for the top 5,000 most frequent words inthe corpus. It should be noted that these are word-based features, and information
about transitions between POS tags is included. When creating training data, theuse of word-based features indicates that word boundaries must be annotated,while the use of POS transition information further indicates that all of thesewords must be annotated with POS.

3.2 Two-step Pointwise MA
In the two-step approach (Neubig et al, 2011), on the other hand, we first segment
character sequence xI1 into the word sequence w J1 with the highest probability,
then tag each word with POS tJ1 . This approach is shown in Fig. 1 (b).
Word segmentation is formulated as a binary classification problem, estimatingboundary tags bI−1
. Tag bi = 1 indicates that a word boundary exists between
1characters xi and xi+1 , while bi = 0 indicates that a word boundary does not exist.
POS estimation can also be formulated as a multi-class classification problem,
where we choose one tag tj for each word wj . These two classification problems
can be solved by tools in the standard machine learning toolbox such as logistic
regression (LR), support vector machines (SVMs), or CRFs.
5 It should be noted that there has been a recently proposed method to loosen this restriction,although this adds some complexity to the decoding process and reduces speed somewhat (Kajiand Kitsuregawa, 2013).
6 More fine-grained POS tags have provided small boosts in accuracy in previous research(Kudo et al, 2004), but these increase the annotation burden, which is contrary to our goal.

6

Shinsuke Mori, Graham Neubig

As features for these classification problems, it is common to use information
about the surrounding characters (character and character-type n-grams), as wellas the presence or absence of words in the dictionary (Neubig et al, 2011). The
character n-gram features fire binary features for each character n-grams of length
1-3 in the neighborhood of the current segmentation boundary. The character typen-gram features similarly are binary features, but with each character generalized
from its surface form to one of six different types of characters widely used in theJapanese language: kanji, hiragana, katakana, arabic number, alphabet, and symbol.
kanji is ideograms, while hiragana and katakana are phonograms mainly used forfunction words and imported words, respectively. Dictionary features include ls
and rs which are active if a string of s characters included in the dictionary ispresent directly to the left or right of the present word boundary, and is which isactive if the present word boundary is included in a dictionary word of s characters.
Dictionary feature djk for POS estimation can indicate whether the current word
wj occurs as a dictionary entry with tag tk .
Compared to the joint sequence-based method described in Subsection 3.1(Kudo et al, 2004), the two-step approach is a dictionary-light method. In fact,given a corpus of segmented and POS-tagged sentences, it is possible to perform
analysis without the dictionary features, relying entirely on the information about
the surrounding n-grams learned from the corpus. However, as large-coverage dictionaries often exist in many domains for consumption by either computer orhuman, having the possibility to use these as additional features is expected togive a gain in accuracy, which we verify experimentally in the following section.
Previous work using this two-stage approach has used sequence-based prediction methods, such as maximum entropy Markov models (MEMMs) or CRFs (Ngand Low, 2004; Peng et al, 2004). However, as Liang et al (2008) note, sequencebased predictors are often not necessary when an appropriately rich feature set isused. One important difference between our formulation and that of Liang et al(2008) and all other previous methods is that we rely only on features that are
directly calculable from the surface string, without using estimated informationsuch as word boundaries or neighboring POS tags7 . This allows for training fromsentences that are partially annotated practically.

4 Experimental Evaluation
To observe the difference between the addition of annotated sentences to the training corpus, and addition of entries to the dictionary, we conducted the experiments
described below.

4.1 Experimental Setting
The task we use as our test bed is the domain adaptation of Japanese MA. We
use the Core part of the Balanced Corpus of Contemporary Written Japanese(BCCWJ) (Maekawa, 2008) as the data for our experiments. The BCCWJ Core
7 Dictionary features for word segmentation are active if the string exists in the original
unsegmented input, regardless of whether it is segmented as a single word in wJ
1 , and thus canbe calculated without the word segmentation result.

Title Suppressed Due to Excessive Length

DomainGeneralGeneral + Web
Web for test
DomainGeneralGeneral + Web

7

Corpus#words784,000
898,000
13,000Dictionary#words Coverage (word-POS pairs)
29,70096.3%
32,50097.9%

Table 3 Language Resource Specifications.

data is divided into six sections, each from a different source, so this is ideal fordomain adaptation experiments.
As our target domain, we use data from the Web (Yahoo! Chiebukuro in BCCWJ) and as the general domain we use a joint corpus built by putting together
the other five domains of BCCWJ Core data. Table 3 shows the specifications of
the corpus and dictionary.
As morphological analyzers, we use the following two publicly available tools.8
1. MeCab: CRF-based joint method (Kudo et al, 2004)
2. KyTea: two-step pointwise method (Neubig et al, 2011)
We compare the following adaptation strategies for the two morphological analyzers.
– No adaptation: Use the corpus and the dictionary in the general domain.
– Dictionary addition (no re-training): Add words appearing in the Web training
corpus to the dictionary. As the dictionary includes weights, we set the weightof all new words to the same value as infrequent words of the same POS tag,following the instructions on the MeCab Web page9 (MeCab only 10 ).
– Dictionary addition (re-training): Add words appearing in the Web corpus to
the dictionary and estimate the weights of the model on the general domain
training data again.
– Corpus addition: Combine the training corpora from the general and Web domains, and train the parameters on the result.
4.2 Evaluation Criterion
As an evaluation criterion we follow Nagata (1994) and use precision and recall
based on word-POS pairs. First the longest common subsequence (LCS) is foundbetween the correct answer and system output. Then let NREF be the number ofword-POS pairs in the correct sentence, NSYS be that in the output in a system,
and NLCS be that in the LCS of the correct sentence and the output of the system,so the recall R and precision P are defined as follows:
R=

NLCS
,
NREF

P =

NLCS
.
NSYS

8 We did not precisely tune the parameters, so there still may be room for further improvement.
9 http://mecab.sourceforge.net/dic.html
10 KyTea requires re-training.

8

Shinsuke Mori, Graham Neubig
Adaptation strategy
No adaptation
Dict. addition (no re-training)
Dict. addition (re-training)
Corpus addition

MeCab
95.20%
96.59%
96.55%
96.85%

KyTea
95.54%
96.75%
97.15%

Table 4 Word Segmentation Accuracy (F-measure).

Finally we calculate F-measure defined as the harmonic mean of the recall and theprecision:
ﬀ−1

2NLCS
1 −1
−1
(R + P )=.
F =2
NREF + NSYS

4.3 Results and Discussion
Table 4 shows the experimental result. From this table, we can see that justadding entries to the dictionary has a large positive effect on the accuracy. By
adding entries to the dictionary (no re-training in the MeCab case11 ) the accuraciesof MeCab and KyTea increase by 1.39% and 1.21% respectively. However, by actuallyadding annotated sentences to the training corpus we can further increase by 0.26%
and 0.40% respectively. That is to say, 75∼84% of the accuracy increase can beachieved through dictionary expansion and the remaining 16∼25% can be realized
only by adding the context information included in the corpus.
The following are the examples of increases realized only by the corpus addition
for MeCab.
na

n

kan

ji

– な / ん ⇒ なん (freq.=4)
In books and newspaper articles “なん ”(what) is written in the Chinese character “何” instead of the hiragana “なん .” Note that every Chinese character(ideogram) can be written by hiragana, (phonogram). Thus the morphologicalanalyzer divides the string into the auxiliary verb “な”(become, get) and its
inflectional ending “ん ” which appear many times in these domains.
– ＾ / ＾⇒ ＾＾ (freq.=3)
Smiley faces are rare in the general domain but often used in Web domain. And
characters including “＾” are a single word in many cases. Thus we need to add
a Web domain training corpus to estimate that the smiley face is sufficiently
common as a single word and should not be divided.
– 感 / じ ⇒ 感じ (freq.=2)“感じ ”(feeling) as a noun does not appear in the general domain corpus and issegmented into a verb “感” and inflectional endings “じ ”, but using this word
as a noun is common in the Web domain.
Another remark is that the accuracy gain is almost the same in CRF-based
joint method (MeCab) and two-step pointwise method (KyTea) contrary to our
expectation that MeCab depends more on the dictionary than KyTea. Thus bothmorphological analyzers are making good use of dictionary information, but also
can be improved with the context provided by the corpus.
11

As we can see in Table 4, renewing CRF parameters decreased the accuracy.

Title Suppressed Due to Excessive Length

Training
Test#sent.
1,760
724

9#r-NEs13,197–#words
33,08813,147#char.
50,002
19,975

Table 5 Specifications of the recipe corpus.

各(each)/ホット ド ッグ /F(hot dog)

に
(cmi)

を
(cmd)

/ふりかけ /Ac
(sprinkle)

る(infl.)/ホット ド ッグ /F(hot dog)

を
(cmd)

/チリ/F
(chili)/アルミ ホイル /F(aluminum foil)、 /チーズ /F,(cheese)

で
(cmc)

/覆/Ac
(cover)、,/オニオン /F
(onion)う(infl.)English is added for explanation only. cmc, cmd, and cmi stand for the case marker forcomplement, direct object, and indirect object, respectively. infl. stands for inflectional
ending. F and Ac are type tags and mean food and action by chef, respectively.
Fig. 2 Example sentences in the r-FG corpus.

5 Realistic Cases
The experimental results that we described in the previous section are somewhat
artificial or in-vitro. In the corpus addition case, it is assumed that the sentences
are entirely annotated with word-boundary information and all the words areannotated with their POS.
In this section, we report results under two other adaptation methods used inreal or in-vivo adaptation scenarios. In both cases, the language resources to be
added are partially annotated corpora (Neubig and Mori, 2010). Because MeCabis not capable of training a model from such corpora, we only report the result of
KyTea.
As the problem, we focus on word segmentation, because in Japanese mostambiguity in MA lies in word segmentation as we mentioned in Section 2, especiallyin the domain adaptation situation where most of unknown words are nouns andthe rest fall into other content word categories such as verbs, adjectives, etc.

5.1 Recipe Domain
The first case we examine is the adaptation to cooking recipe texts. A cooking
recipe text consists of sentences describing procedures. Because they do not containdifficult language phenomena for NLP such as tense, aspect, viewpoint, etc., they
can be thought to be relatively easy for computers to understand. Thus they aresuitable as a benchmark for the language understanding research. In addition,
there is a large demand for more accurate NLP in recipe domains for intelligent
recipe processing (Wang et al, 2008; Yamakata et al, 2013).
In the following experiment we used the recipe flow graph corpus (r-FG corpus)(Mori et al, 2014). Table 5 shows the specifications of the r-FG corpus relating
to the word segmentation experiment. In the corpus word sequences importantfor cooking are annotated with types (recipe named entities; r-NEs) and they arecorrectly segmented into words. Fig. 2 shows two example sentences.

10

Shinsuke Mori, Graham Neubig
Adaptation strategy
No adaptation
Dictionary

Partial annotation#occurrences
Maximum (n)Average
–
–
–
–
1
1.00
2
1.60
3
2.02
4
2.36
8
3.26
16
4.26
32
5.10
64
5.77
∞
6.60

#words
0
1,999
1,999
3,191
4,046
4,727
6,523
8,512
10,203
11,542
13,197

WS F-measure
BCCWJ
Recipe99.01%
95.56%
99.00%
95.78%
99.01%
95.80%
99.01%
96.06%
99.01%
96.15%
99.01%
96.27%
99.01%
96.28%
99.01%
96.33%
99.01%
96.36%
99.02%
96.38%
99.01%
96.43%

Table 6 Word segmentation accuracy in the partial annotation case.

5.1.1 Experimental Setting
As the adaptation strategies, we used the following two methods in addition to“No adaptation.” The examples are taken from Fig. 2. F and Ac are type tags andmean food and action by the chef, respectively.
Dictionary: Use the training data as a dictionary.
1. Extract r-NEs from the training data,ex.) /ホット ド ッグ /F, /チリ/F, /チーズ /F,/オニオン /F, /ふりかけ /Ac,/ホット ド ッグ /F, /アルミ ホイル /F, /覆/Ac
2. Make a dictionary containing the words in these r-NEs,ex.) ホット , ド ッグ , チリ, チーズ , オニオン ,ふりかけ , アルミ, ホイル , 覆
3. Use the dictionary as the additional language resource to train the model.
Partial annotation: Use the training data as partially annotated data.
1. Extract n occurrences at maximum of the r-NEs from the training data
(see Fig. 2, where the r-NE in focus is ホットド ッグ and n = 2),
2. Convert them into partially segmented sentences in which only both edgesof the r-NEs and the inside of the r-NEs are annotated with word-boundary
information. If the r-NE in focus is ホットド ッグ composed of two words,
then the partially segmented sentences areex.) 各|ホ-ッ-ト |ド -ッ-グ |に チ リ 、…,
ex.) |ホ-ッ-ト |ド -ッ-グ |を ア ル ミ…,
where the symbols “|,” “-,” and “ ” mean word boundary, no word boundary, and no information, respectively.
3. Use the partially annotated data as an additional language resource to train
the model.

5.1.2 Result and Discussion
Table 6 shows the word segmentation accuracies (WS F-measure) of No adaptationand the strategies that we explained above. The results of the partial annotation
strategy vary depending on the parameter n (the maximum number of occur-

Title Suppressed Due to Excessive Length11

rences). The table shows these results with the real average number of occurrencesin the partially segmented sentences.
From the result we can note several things. The BCCWJ results show that
adding language resources in the recipe domain has no effect on the general domain
accuracy. Regarding the results of the first recipe, the addition of new words as thedictionary to the training data improves the word segmenter. This is consistent
with the results shown in Table 4. Second, the partial annotation strategy with
one occurrence (n = 1) is as good as the dictionary addition strategy. And as
we increase the number of occurrences (n), the segmenter improves. The degreeof improvement, however, shrinks as n increases. In a real situation, we have toprepare such partially annotated data and the annotation cost is proportional to
the number of occurrences to be annotated. Therefore it is good to start annotating
new words in descending order of frequency, selecting a threshold based on thenumber of occurrences. We describe a concrete way of doing so for real unannotateddata in the following section.

5.2 Invention Disclosure Domain
Finally we report the result of a real adaptation experiment that we performed.
The target domain is invention disclosure texts from patents, which are an important domain for NLP, especially information extraction (Nanba et al, 2011, interalia) and machine translation (Goto et al, 2011, inter alia).
5.2.1 Setting
Based on the knowledge we described above, we adopted the partial annotation
strategy. Concretely, we performed the following procedure.
1. Extract unknown word candidates based on the distributional similarity froma large raw corpus in the target domain (Mori and Nagao, 1996),
2. Annotate three occurrences with word-boundary information to make partiallysegmented sentences for each unknown word candidate in the descending orderof the expected frequencies.12
For frequent word candidates, i.e. in the beginning of the annotation work, thethree-occurrence annotation corresponds to the case of those with the maximum
occurrence count of 4 (average: 2.36) and 8 (average: 3.26) in Table 6, because
the average number of the occurrences is expected to be three.
In practice, we first assign each word a default annotation, then ask an annotator to check unknown word candidates with three different contexts in the raw
corpus and correct the word boundary information if the default is incorrect. Fig.
3 shows two example word candidates with their three occurrences. The default
segmentation gyo
assumesthat the candidate word is really a word. In the first exbu
ample case,gyo“ 御 部”seigyo
is assumed to be a word, but
it is a concatenation of a wordbu
fragment “ 御 ” in “制御” (control) and a suffix “部” (part) at all three occurrences.
12 The expected frequency of a word candidate is the frequency as a string in the raw corpus
multiplied by the word likelihood estimated by the comparison between the distribution of theword candidate and that of the words. See (Mori and Nagao, 1996) for more detail.

12

Shinsuke Mori, Graham Neubig

Test#sent.
500#words
20,658#char.
32,139

Table 7 Specifications of the invention disclosure corpus.

gyobu

Freq. = 200, candidate = “ 御 部”
Default annotation by the system
こ の 場 合 、 第 ２ の 制|御-部|２ ２ は 初 期 動 作 ル ー チ ン の み を 行 う 。
ま た 、 第 １ の 制|御-部|２ １ は 、 Ｓ ８ ０ で 、 ホ ス ト か ら 記 録 再 生 …
プ リ ン タ ９ ５ Ｂ の プ リ ン タ 制|御-部|か ら 開 始 信 号 が 帰 っ て く る …
After annotation work
こ の 場 合 、 第 ２ の 制-御|部|２ ２ は 初 期 動 作 ル ー チ ン の み を 行 う 。
ま た 、 第 １ の 制-御|部|２ １ は 、 Ｓ ８ ０ で 、 ホ ス ト か ら 記 録 再 生 …
プ リ ン タ ９ ５ Ｂ の プ リ ン タ 制-御|部|か ら 開 始 信 号 が 帰 っ て く る …
seki soFreq. = 80, candidate = “ 積 層”
Default annotation by the system… ｐ 型 半 導 体 基 板 上 に 電 荷 蓄|積-層|（ 浮 遊 ゲ ー ト ） と 制 御 ゲ ー …
… バ イ モ ル フ 形 の も の の 他 、 |積-層|形 の も の を 採 用 し て も 良 い 。
… モ リ ブ デ ン 等 の 金 属 を １ ０ ０ ０ Å の 膜 厚 に |積-層|し て 、 金 属 …
After annotation work
… ｐ 型 半 導 体 基 板 上 に 電 荷 蓄-積|層|（ 浮 遊 ゲ ー ト ） と 制 御 ゲ ー …
… バ イ モ ル フ 形 の も の の 他 、 |積-層|形 の も の を 採 用 し て も 良 い 。
… モ リ ブ デ ン 等 の 金 属 を １ ０ ０ ０ Å の 膜 厚 に |積-層|し て 、 金 属 …
Fig. 3 KWIC (key word in context) of unknown word candidates.

F measure (word segmentation)

96.8
96.6

96.4

96.2

96.0
0

2

4
6
8
Work time [hour]

1012Fig. 4 Accuracy increase.

So the annotator changed the word-boundary information
as shown in “After ansekiso
notation work.” In the second example
case,
“
積層
”
in
the
first line (context) issekichikuseki
asou
concatenation of a word fragment “ 積 ” in “ 蓄 積 ” (accumulate) and a suffix“ 層 ” (layer). So the annotator changed the word-boundary information as shownsekisoin “After annotation work.” In the second and third line (context), however, “積層”(lamination) is a word and the annotator leaves it as the default without change.

Title Suppressed Due to Excessive Length

13

Table 8 WS F-measure of the adapted models
Test domain#sentences
Adaptation method

General
3,680–

No adapatation
Adaptation torecipepatenttwitterall

99.01

Recipe
724
r-NE
n=8
95.56

Patent500
KWIC
12 hours
96.15

Twitter542
KWIC
47 hours
85.90

99.01
99.02
99.01

96.28
95.54
95.73

96.56
96.63
96.22

85.99
85.94
87.63

99.01

96.35

97.05

88.35

5.2.2 Result and Discussion
The learning curve is shown in Fig. 4. The leftmost point corresponds to the“No adaptation” case. The accuracy in this case is high compared with the recipedomain (Table 6) because the invention disclosure domain is not as stylisticallydifferent from the general domain containing newspaper articles etc. The mostimportant thing to note is that the accuracy gets higher as we add more unknown
word candidates to the training data as partially annotated sentences. After 12hours of annotation work, we succeeded to eliminate 12% of the errors. The absolute F-measure is almost the same as that of the state-of-the-art word segmenteron the test set in the same domain as the training data (Neubig et al, 2011). This
improved word segmenter model is capable of contributing to various NLP applications in the invention disclosure domain in Japanese. In addition the accuracy
does not seem to be saturating, thus we can improve more by only more annotator
work based on the partial annotation strategy.

6 Non-maleficence of Language Resources
In this section we introduce a notion of non-maleficence13 of language resources.
Briefly this means that the effects of a language resource in a certain domain does
no harm in another domain.
Our experiments indicate that non-maleficence holds for Japanese word segmentation. In order to show this empirically, we provide results for an experimentin which we executed the adaptation of word segmenter KyTea to Twitter (microblog) for 47 hours in addition to two domains which we described in Section 5.
The adaptation method is exactly the same as the invention disclosure domain(patent).
In the experiment we made five models of KyTea in total. The first one isthe default model without adaptation. The next three are models adapted to
the recipe domain, invention disclosure (patent) domain, and Twitter. They aretrained from the language resources in each domain in addition to the default
language resources. In the recipe case we set n = 8 because it is realistic as
we discussed above. In the patent case we used the maximum size of partially
13 We borrow this terminology from medicine, where non-maleficence indicates the propertyof “doing no harm.”

14

Shinsuke Mori, Graham Neubig

annotated sentences, i.e the 12-hour work result corresponding to the rightmost
point in Fig. 4. The final KyTea model is trained from the language resources inall three adaptation domains in addition to the default language resources. Then
we measured the accuracy of each model on the test data in the general domain,recipe domain, patent domain, and Twitter.
Table 8 shows the results. From these results we first see that the corpus
addition to each domain improves the performance in that domain. In addition it
does not degrade the performance in other domains.14 We call this characteristicof the pair of a task and language resources non-maleficence. Its conditions are as
follows:– Language resource addition in a target domain is efficient for the target domain– Language resource addition in other domains is not harmful for the target
domainWe also see that in some cases language resource addition in other domains has apositive effect on the target domain. We term this additivity of language resources.
A clear example is the model trained from the recipe tested on the patents. The
reason may be that the recipe texts covers how-to expressions and the many frequent technical terms have already been covered by the general texts. On thecontrary, language resource addition in the patent domain does not improve theperformance on the recipe domain. Thus, we can say that the language resource
addition is not symmetric. The model adapted to all the three domains performs
almost always the best.15 In this case as well, language resource addition is nonmaleficent. The users of the NLP tool can always use the model trained from the
maximum language resources, but not the model trained only from the language
resource in the target domain.16

7 Conclusion
In this paper, we first reported to what extent two strategies of language resource
addition contributed to improvements in word segmentation and POS tagging accuracy in Japanese. The first strategy is adding entries to the dictionary and thesecond is adding annotated sentences to the training corpus. In the experimental
evaluations, we first showed that the corpus addition strategy allows for achievement of better accuracy than the dictionary addition strategy in the Japanesemorphological analysis task.
We then introduced the partial annotation strategy, in which only important
points are annotated with word-boundary information, and reported the real cases
focusing on word segmentation in Japanese. The experiment showed that adding
word candidates to the training data as partially annotated data with about threedifferent contexts is efficient to improve a word segmenter.
14 A very slight degradation is observed in case of recipe WS by the model trained frompatent texts (from 95.56% to 95.54%). This is not statistically significant.
15 The only exception is that the model adapted to the patent tested on the general domain
is better than the others (from 99.01% to 99.02%). The change is, however, not significant.
16 ML technologies have a possibility to adapt the model to an unexpected input automatically.

Title Suppressed Due to Excessive Length15

Finally we investigated various language resource addition cases and introduced the notion of non-maleficence, additivity, and asymmetricity of language
resources for a task. Briefly non-maleficence means that language resource addition in a certain domain does no harm in another domain, additivity means that
language resource addition in other domains has a positive effect on the target
domain, and asymmetricity means that the effect of language resource addition
is not symmetric. In the WS case, language addition is non-maleficent and sometimes additive. So it is enough for us, NLP tool providers, to distribute the only
one model trained from all the language resources we have.
Acknowledgements This work was supported by JSPS Grants-in-Aid for Scientific Research
Grant Numbers 26280084 and 26540190, and NTT agreement dated 05/23/2013.

References
Brown PF, Pietra VJD, deSouza PV, Lai JC, Mercer RL (1992) Class-based ngram models of natural language. Computational Linguistics 18(4):467–479
Goto I, Lu B, Chow KP, Sumita E, Tsou BK (2011) Overview of the patentmachine translation task at the NTCIR-9 workshop. In: Proceedings of NTCIR9 Workshop Meeting, pp 559–578
Kaji N, Kitsuregawa M (2013) Efficient word lattice generation for joint word
segmentation and POS tagging in Japanese. In: Proceedings of the Sixth International Joint Conference on Natural Language Processing, Nagoya, Japan, pp
153–161
Kruengkrai C, Uchimoto K, Kazama J, Wang Y, Torisawa K, Isahara H (2009) An
error-driven word-character hybrid model for joint Chinese word segmentation
and POS tagging. In: Proceedings of the 47th Annual Meeting of the Association
for Computational Linguistics, pp 513–521
Kudo T, Yamamoto K, Matsumoto Y (2004) Applying conditional random fields toJapanese morphological analysis. In: Proceedings of the Conference on Empirical
Methods in Natural Language Processing, pp 230–237
Lafferty J, McCallum A, Pereira F (2001) Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In: Proceedings of theEighteenth ICML, pp 282–289
Liang P, Daumé III H, Klein D (2008) Structure compilation: trading structure
for features. In: Proceedings of the 25th ICML, pp 592–599
Maekawa K (2008) Balanced corpus of contemporary written Japanese. In: Proceedings of the 6th Workshop on Asian Language Resources, pp 101–102
Mori S, Kurata G (2005) Class-based variable memory length Markov model. In:
Proceedings of the InterSpeech2005, pp 13–16
Mori S, Nagao M (1996) Word extraction from corpora and its part-of-speech estimation using distributional analysis. In: Proceedings of the 16th International
Conference on Computational Linguistics, pp 1119–1122
Mori S, Neubig G (2014) Language resource addition: Dictionary or corpus? In:
Proceedings of the Nineth International Conference on Language Resources and
Evaluation, pp 1631–1636

16

Shinsuke Mori, Graham NeubigMori S, Oda H (2009) Automatic word segmentation using three types of dictionaries. In: Proceedings of the Eighth International Conference Pacific Association
for Computational Linguistics
Mori S, Maeta H, Yamakata Y, Sasada T (2014) Flow graph corpus from recipe
texts. In: Proceedings of the Nineth International Conference on Language Resources and Evaluation, pp 2370–2377
Nagata M (1994) A stochastic Japanese morphological analyzer using a forwardDP backward-A∗ n-best search algorithm. In: Proceedings of the 15th International Conference on Computational Linguistics, pp 201–207
Nakagawa T (2004) Chinese and Japanese word segmentation using word-level andcharacter-level information. In: Proceedings of the 20th International Conference
on Computational Linguistics, pp 466–472
Nanba H, Fujii A, Iwayama M, Hashimoto T (2011) Overview of the patent mining
task at the ntcir-8 workshop. In: Proceedings of NTCIR-8 Workshop Meeting,pp 293–302
Neubig G, Mori S (2010) Word-based partial annotation for efficient corpus construction. In: Proceedings of the Seventh International Conference on Language
Resources and Evaluation, pp 2723–2727
Neubig G, Nakata Y, Mori S (2011) Pointwise prediction for robust, adaptable
Japanese morphological analysis. In: Proceedings of the 49th Annual Meetingof the Association for Computational Linguistics, pp 529–533
Ng HT, Low JK (2004) Chinese part-of-speech tagging: one-at-a-time or all-atonce? word-based or character-based. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp 277–284
Peng F, Feng F, McCallum A (2004) Chinese segmentation and new word detection using conditional random fields. In: Proceedings of the 20th International
Conference on Computational Linguistics, pp 562–568
Ron D, Singer Y, Tishby N (1996) The power of amnesia: Learning probabilistic
automata with variable memory length. Machine Learning 25:117–149
Sassano M (2002) An empirical study of active learning with support vector machines for Japanese word segmentation. In: Proceedings of the 40th Annual
Meeting of the Association for Computational Linguistics, pp 505–512
Settles B, Craven M, Friedland L (2008) Active learning with real annotation costs.
In: NIPS Workshop on Cost-Sensitive Learning
Tomanek K, Hahn U (2009) Semi-supervised active learning for sequence labeling.
In: Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics, pp 1039–1047
Tsuboi Y, Kashima H, Mori S, Oda H, Matsumoto Y (2008) Training conditionalrandom fields using incomplete annotations. In: Proceedings of the 22nd International Conference on Computational Linguistics, pp 897–904
Wang L, Li Q, Li N, Dong G, Yang Y (2008) Substructure similarity measurementin Chinese recipes. In: Proceedings of the 17th International Conference on
World Wide Web, pp 978–988
Yamakata Y, Imahori S, Sugiyama Y, Mori S, Tanaka K (2013) Feature extraction and summarization of recipes using flow graph. In: Proceedings of the 5thInternational Conference on Social Informatics, LNCS 8238, pp 241–254
Yang F, Vozila P (2014) Semi-supervised Chinese word segmentation using partiallabel learning with conditional random fields. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp 90–98