Pointwise Prediction and Sequence-based Reranking
for Adaptable Part-of-Speech Tagging
Shinsuke Mori
Academic Center for Computing and Media Studies,
Kyoto University
Yoshidahonmachi, Sakyo-ku, Kyoto, 606-8501, Japan
forest@i.kyoto-u.ac.jp
Graham Neubig
Nara Institute of Science and Technology,
8916-5 Takayamacho, Ikoma, Nara, 630-0192, Japanneubig@is.naist.jp

Abstract—This paper proposes an accurate method for partof-speech (POS) tagging that is highly domain-adaptable. The
method is based on an assumption that the POS transition tendencies do not depend on domains, and has the following threecharacteristics: 1) it is trainable from partially annotated data,
2) it uses efficiently trainable pointwise POS taggers to allow for
active learning, and 3) is more accurate than the pointwise or
sequence-based POS taggers. The proposed method estimates
POS tags by stacking pointwise and sequence-based predictors.
In the experiments we deal with the joint problem ofword segmentation and POS tagging in Japanese. We showthat our proposed stacking process improves over pointwiseand sequence-based methods (hidden Markov models andconditional random fields) both in the general domain andthe target domain. In addition we show the learning curve in adomain adaptation scenario. The result shows that our method
outperforms state-of-the-art methods in the same domain as thetraining data and is better than them in domain adaptation
situations as well.
Keywords-Active learning; Reranking; Word segmentation;
Part-of-speech tagging; Pointwise prediction;

I. I NTRODUCTION
Part-of-speech (POS) tagging [1], [2, and many others]
is a fundamental step of natural language processing (NLP)
in many languages, and many NLP applications use POS
tagging results. Thus POS tagging accuracy has a great
impact on these NLP applications. With large annotated corpora [3, inter alia] and methods based on machine learningtechniques, the NLP community achieved a high accuracy
around 97% or more in various languages. However, with thediversification of the domains to which NLP is applied, suchas medical texts or texts in user generated contents (blog,
twitter, etc.), we sometimes observe a severe degradation in
POS tagging accuracy in text domains different from that of
the training data
* This work was done when he was at Kyoto University.

Yosuke Nakata∗
NTT Communications
1-1-6 Uchisachicho, Chiyoda-ku, Tokyo, 100-8019, Japan
ruberukuraku@gmail.com

Tetsuro Sasada
Academic Center for Computing and Media Studies,
Kyoto University
Yoshidahonmachi, Sakyo-ku, Kyoto, 606-8501, Japansasada@ar.media.kyoto-u.ac.jp. Therefore we can say that there is still a large demandfor improving POS tagging accuracy, especially in domain
adaptation situations.
In addition to the machine learning techniques, the NLP
community is increasingly aware of importance of language
resources, as the easiest way to improve an NLP based on aparticular machine learning technique for a certain domain
is to just add annotated texts in that domain to the training
data. This strategy does require time and money, however, so
we are interested in reducing annotation work by allowing
annotators to focus on informative points that will provide
a good performance/cost trade-off. Thus, there is a large
amount of research on training NLP systems from partiallyannotated data or incomplete data, in which only some points
are annotated with labels [4]. In this setting, some points lackthe correct labels, and some may have multiple labels.
One of the major tasks to which these methods havebeen applied is word segmentation (WS) for languages
without obvious word boundaries. A method for training
conditional random fields (CRFs) from partially annotateddata has been proposed and tested in Japanese WS [4].
This CRF extension is used to improve Chinese WS byreferring to so-called natural annotations, such as partiallysegmented sentences converted from Wikipedia assumingthat HTML tags are word boundaries [6]. A word segmenter
based on a binary classifier [7] is another implementation
trainable from partially annotated data. In this method the
WS system decides whether there is a word boundary or not
at each point between two characters without referring to
the estimated labels surrounding the decision point. This iscalled a pointwise prediction method (or simply a pointwise
method). Compared with sequence prediction methods like
Markov models or CRFs, pointwise prediction requires
less time to estimate model parameters even from partiallyannotated data, and is thus suitable for active learning, whichis performed by alternating rounds of selecting uncertain
points for annotation, performing annotation, and retraining
the classifier [8].
Given this background, in this paper we propose a POS
tagger equipped with following three characteristics:
1) It is trainable from partially annotated data.
2) Training is as fast as pointwise POS taggers to allow
active learning.
3) It is more accurate than the pointwise and sequencebased POS taggers.
Our method performs POS estimation by stacking pointwiseand sequence-based predictors, using pointwise prediction
followed by reranking using sequence-based predictors [9].
The first module, pointwise POS estimation, is trainablefrom partially annotated sentences. The second module,
sequence-based POS reranking, is efficiently trainable only
from fully annotated sentences. Thus the training data of
the second module is a subset of the first module. However,sequence-based predictors can use POS sequence information, and thus there may be room for improvement byreferring to the combination of label candidates. We assume
that these POS transition tendencies do not depend on thedomain, and thus even if the sequence-based labeler istrained only on general domain data, it might be able tocontribute to improve POS tagging accuracy even on texts inthe adaptation target domain where only partially annotateddata is available.
In the experiment we deal with the joint problem of WS
and POS tagging in Japanese, traditionally called morphological analysis (MA).1 We show that our stacking process
improves over pointwise MA [8] and sequence-based MA[10] [11] [12] both in the general domain and in the target
domain. In addition we show the learning curve in a domain
adaptation scenario, which finds that the proposed method
is as domain adaptable as purely pointwise approaches.
II. J OINT P ROBLEM S OLUTION BASELINE
Our method, which we propose in this paper, uses MA bypointwise classifiers [8] as the first step. MA by pointwiseclassifiers (PW-MA) solves the problem step-by-step. First,
it segments an input sentence into a word sequence. Then, it
estimates the POS of each word like an English POS tagger.
At each step PW-MA refers only to the input but not to any
estimation results (or dynamic information) as features. In[8] linear support vector machines (SVMs) [13] are usedbecause of their classification accuracy and speed. In thissection we describe PW-MA in detail.
A. Word Segmentation by Pointwise Classification
The two-step approach [8] segments character sequencex = x1 x2 · · · xk into the word sequence w. Word segmentation is formulated as a series of binary classification
1 MA often also performs recovery of word base forms, but we do nothandle this element in the present work.

problems, estimating boundary tags b1 , b2 , . . . , bk−1 .
Tag bi = 1 indicates that a word boundary exists between
characters xi and xi+1 , while bi = 0 indicates that a wordboundary does not exist.
As features it uses information about the surrounding
characters (character and character-type n-grams), as wellas the presence or absence of words in the dictionary. The
details of the features are as follows:
1) Character n-grams: substrings surrounding the decision point i. There are two parameters: the window
width m and the length n. Features are all the substrings of the length up to n in the 2m long substring xi−m+1 , · · · , xi−1 , xi , xi+1 , · · · , xi+m . Figure 1
shows an example.
2) Character type n-grams: the same as the character ngrams but the characters in the substring are converted
into the character type. The character types are Chinese character (K), katakana (k), hiragana (H), Roman
alphabet (R), Arabic number (N), or other (O). Figure
1 shows an example.
3) Dictionary: three flags indicating that the word starting
at i, ending at i, or containing i are included in the
dictionary, and the length of that word.
As the above explanation indicates, PW-MA is trained fromonly the annotated points between two characters and it
does not require any modification to estimate its parametersfrom partially annotated data. Thus it is both simple and fastenough to make active learning realistic.
B. POS Tagging by Pointwise Classification
POS tagging by pointwise classification performs one of
the following four processes depending on the target word.
1) If the word appears as more than one POS in the
training corpus, estimate the POS by a classifier,2) If the word appears as only one POS in the training
corpus, return its POS,
3) If the word does not appear in the training corpus but
in the dictionary, return the POS of the first entry,
4) Otherwise, return noun.
In the first case, POS estimation is formulated as a multiclass classification problem, where we choose one tag tj
for each word wj . The input is a word sequence but theclassifier regards it as the target word and the character
sequences preceding it (x− ) and following it (x+ ). The
POS of wj is estimated from x− , wj , and x+ . When
the window width is m0 , then the information referred tois x−m0 · · · x−2 x−1 , wj , x1 x2 · · · xm0 . Putting it in anotherway, it only refers to the fact that there are word boundaries
on both sides of wj and that there is no word boundary
inside wj , and two character sequences x− and x+ .
The features for POS estimation are as follows (see Figure2):1) Word in focus,

Text:(vaccinate a healthy child with this medicine)xi−2 xi−1 xi xi+1 xi+2 xi+3
健 康 児
に 本 剤
を
接
種
↑
ti : Decision point

しCharacter (type) 1-gram: -3/児 (K), -2/に (H), -1/本 (K), 1/剤 (K), 2/を (K), 3/接 (K)Character (type) 2-gram: -3/児に (KH), -2/に本 (HK), -1/本剤 (KK), 1/剤を (KH), 2/を接 (HK)
Character (type) 3-gram: -3/児に本 (KHK), -2/に本剤 (HKK), -1/本剤を (KKH), 1/剤を接 (KHK)
Figure 1.

Features referred to in word segmentation (window width m = 3, n = 1, 2, 3).

Text:(vaccinate a healthy child with this medicine)x−3 xi−2 x−1 w xi+1 xi+2 xi+3
健 康児に 本剤 を
接種↑
Word in focus

しCharacter (type) 1-gram: -3/康 (K), -2/児 (K), -1/に (H), 1/を (H), 2/接 (K) 3/種 (K),Character (type) 2-gram: -3/康児 (KK), -2/児に (KH), -1/にを (HH), 1/を接 (HK), 2/接種 (KK)
Character (type) 3-gram: -3/康児に (KKH), -2/児にを (KHH), -1/にを接 (HHK), 1/を接種 (HKK)
Figure 2.

Features referred to in POS tagging (window width m0 = 3, n = 1, 2, 3).

2) Character n-grams included in x− x+ ,3) Character type n-grams included in x− x+ .
Similar to PW-MA, POS tagging based on pointwise prediction is trained from only the words annotated with their
POS and it does not require any modification to estimate its
parameters from partially annotated data and is enough fast
to make active learning realistic.
C. Flexible Language Resource Usage by Pointwise Prediction
WS or POS tagging based on the pointwise prediction
allows us to use the following new types of language
resources, making it possible to more efficiently adapt thetagger to new domains.
1) Partially annotated corpora: Only some points between two characters in a sentence are annotated withword boundary information or only some words areannotated with POSs. For MA a corpus annotatedonly with word boundaries is also a partial annotation
corpus. Partially segmented or partially POS-annotatedcorpora also fall in this category.
2) Word dictionary: A list of words without POSs. This
type of dictionary is often available in many domains.
We can use this for automatic WS.
Of course the pointwise prediction can use a fully annotatedcorpus in which all the sentences are completely segmented
into words and all the words are annotated with their POSs,
and a list of words with POSs. These fully annotated corporaand dictionaries are sometimes difficult to prepare in a target
domain, but partial annotations are relatively easy to prepare.
Thus, MA based on the pointwise prediction makes it easier

to adapt to new domains by making it possible to retrieve
information even from these various language resources.
III. 2-S TEP POS E STIMATIONThe PW-MA described in the previous section can not use
the POS sequence information in the training corpus. This
information may be, however, important for POS estimation.
In this paper we assume that the domain dependency of POS
transition tendencies is low and propose a new method for
POS estimation based on this assumption. In this method,
we use stacking to combine pointwise and sequence-based
predictors, with the domain-specific pointwise predictor
capturing domain knowledge, and the domain independent
sequence-based predictor reranking the POS estimation result of PW-MA.
A. Overview of the Proposed Method
The proposed method combines the following three processes in a cascade:1) word boundary estimation by a pointwise prediction,2) POS-confidence pair estimation by pointwise prediction, and
3) POS reranking by sequence prediction.
Given an input sentence, first we segment it into a word
sequence by word boundary estimation based on a pointwise
prediction. This process is completely the same as the one
described in Section II-A. Then we estimate a POS for eachword in the word sequence. This process is similar to theone described in Section II-B, but we enumerate all thepossible POSs with confidences. Finally we rerank the POS
sequences based on sequence-based prediction referring to
the confidences.

two types: context features and confidence features, which
we describe in detail in the subsequent section. In the prediction step the CRFs output the most likely POS sequence
taking the output of the pointwise prediction results withconfidence as the input. In the example shown in Figure
3, the CRFs output the POS sequence connected by thesolid line, where the POS of the word “児” (child) has beenchanged into prefix from noun.
D. Features

Figure 3.

POS reranking by sequence labeling.

B. POS Estimation with Confidence by the Pointwise Prediction
The pointwise POS estimation described in Subsection
II-B outputs only one POS for each word. In the proposed
method, however, we calculate the confidences for all thepossible POSs for a word and we use these confidences in
the reranking process (see Figure 3).
The confidence of a POS for a word is defined as follows.
First let dr be the distance (margin) from the separation
hyper-plane of the r-th (r ≥ 1) POS candidate. And
we define the confidence of the r-th POS candidate as
cr = dr −d2 . As a result, the confidence of the first candidate
is a positive value (in almost all cases cr  100 becauseof L2 regularization), that of the second candidate is 0,
and those of the other candidates are negative values. If
there is no POS candidate (the case 4 in Section II-B), thisprocess returns a noun with confidence 0. And if there is
only one POS candidate (the case 2 or 3 in Section II-B),
this process returns that POS with confidence 100 (a special
value). Figure 3 shows an example of POS candidates andtheir confidences.
C. POS Reranking by a Sequential Prediction
We have a word sequence and all the possible POSs withconfidences as the output of the process above. Then we
search for the best POS sequence among all the possible
POS sequences by referring to the POS-confidence pairestimation result and the POS sequence statistics taken in the
training corpus. Note that the word boundaries estimated by
the pointwise word segmentation are not changed, because
we do not rerank the word boundaries.
As a sequential prediction method we use CRFs [14], astandard method for sequence labeling problems because of
their flexible feature design and high classification accuracy.
The correct labels in the training data are the POS sequencesin the full annotation corpus. The features are divided into

As we mentioned, the CRFs for POS reranking refer tocontext features and confidence features. The confidence features are the followings calculated from the POS-confidence
pairs output by the pointwise prediction.
Rule 1:
If the word has multiple POS candidates, the t-thfeature (1 ≤ t ≤ T ) is the confidence of the t-th
POS.
Rule 2:
If the t-th POS is not a candidate, the (T + t)-thfeature (T + 1 ≤ T + t ≤ 2T ) is set be 1.
Rule 3:
If the t-th POS is the only candidate, the (2T + t)th feature (2T + 1 ≤ 2T + t ≤ 3T ) is set be 1.
When the condition of each rule is not satisfied, the feature
value is set to be “NULL” (i.e. many features are NULL).
The rationale of the rule 2 is to provide information about
the POSs not in the candidate list. That of the rule 3 isto indicate POSs with high confidence according to thepointwise prediction, that may not need to be changed.
The other feature set is the context. We list them asfollows:
1) word n-grams in the window width m00 including theword in focus at the center.
2) character type set n-grams of the words in the window
width m00 including the word in focus at the center.
The character type set is a set of character types includedin the spelling of a word. We set 6 character types, which
are the same as those used in the word boundary prediction(Subsection II-A). Thus the character type set has 26 − 1
combinations. The character type set n-grams are sequencesof the character types for a word sequence.
E. Training Data Creation
As the training data of the CRFs for POS reranking, weneed the correct POS tag sequence and those estimated by
the pointwise prediction for a word sequence for feature
creation. The estimated POS tag sequence should be similar
to that given at the runtime. Thus the confidence estimation
target has to be different from the training data of thepointwise prediction for the POS-confidence pair estimation.
So we propose the following procedure similar to deletedinterpolation [15].

Training corpus (word/POS full annotation)

Test

1st 1/3 sub-corpus

2nd 1/3 sub-corpus

3rd 1/3 sub-corpus

Tra

in

Word segmentation by
pointwise prediction

1st 1/3 sub-corpus
with confidence

Word segmentation by
pointwise prediction

2nd 1/3 sub-corpus
with confidence

Word segmentation by
pointwise prediction

3rd 1/3 sub-corpus
with confidence

Tra

in

POS reranking by
sequence prediction

Figure 4. Procedure for generating the training corpora for POS reranking
by sequence labeling (k = 3).

1) Divide the training corpus C into k subsets
C1 , C2 , . . . , Ck .
2) For each i ∈ {1, 2, ..., k}a) Train the i-th pointwise MA from k − 1 subsets
except for Cib) Estimate POS-confidence pairs on Ci by the i-th
pointwise MA with the model obtained by thestep a)Figure 4 illustrates the above procedure in the case of k =3. The above procedure produces the subsets annotated with
POS candidates and their confidences C10 , C20 , . . . , Ck0 . By
adding the correct POS tag sequence in C to them, we havethe training data of our CRFs for POS reranking.
F. Proposed Method and Language Resource
At the end of this section, we discuss the relationshipbetween the proposed method and the corpus types. In
the general domain, many fully annotated corpora (GTF in
Figure 5), in which the sentences are divided into words
completely and all the words are annotated with POSs, areavailable. Almost all annotated corpora produced throughWord boundary
only (W)

Examples of various types of corpora.

GWF
Full
Annotation

Word boundary
and POS (T)Figure 6.

GTF
Full
Annotation

Sentence

GWP
PartialAnnotation

Word boundary estimation
by pointwise prediction

GTP
Full
Annotation

POS estimation
by pointwise prediction
Word boundary
only (W)

Full (F):Partial (P):Full (F):Partial (P):

AWF
Full
Annotation

Word boundary
and POS (T)Word boundary(W)
Word boundary/POS (T)General domain (G)|文-化|交-流|使|事-業|を |文 化|交-流|使 事 業 を|文-化/Noun|交-流/Noun|使/PostF|事-業/Noun|を /PP|
文 化|交-流/Noun|使 事 業 をTarget domain (A)|血|小-板|の|減-少|が |血|小-板|の 減 少 が|血/PreF|小-板/Noun|の/PP|減-少/Noun|が /PP|
血|小-板/Noun|の 減 少 がGeneral domain corpus

Full (F):Partial (P):Full (F):Partial (P):

Adaptation domain corpus

Word boundary(W)
Word boundary/POS (T)ATF
FullAnnotation

AWP
PartialAnnotation

ATP
PartialAnnotation

POS reranking
by sequence prediction

Word/POS seq.

Figure 5. Relationship between the proposed method and various typesof corpora. Theoretically we can use language resources connected by bothdotted and solid lines for each process indicated by the ovals. Practically
we use language resources connected by solid lines.

corpus annotation research [16], [17] fall in this category.
The annotation work for the target (adaptation) domain
corpus requires, however, domain knowledge in addition to
the linguistic knowledge of the annotation standard. Thus
the full annotation corpus in the target domain is costly.
But a partial annotation corpus, in which some words are
identified and some are annotated with POSs, is relatively
easy to prepare. Figure 6 shows examples. In Figure 6, we
use a notation called the extended 3-valued notation, which
is our extension of the following 3-valued notation [18].
| : There is a word boundary.
- : There is no word boundary.
: There is no information.
As an extension we add “/” to denote the POS of the word
after it like |s-p-e-l-l-i-n-g/POS| if annotated. The

Table I
C ORPUS SPECIFICATION .
Name
BCCWJ

SourceWhite paper, Book, Newspaper(General domain as GTF)
Yahoo!QA(Target as ATF or ATP)following are the information that we can extract fromvarious types of corpora.
• GWF, AWF: Word sequence and surrounding charactersof word boundaries• GWP, AWP: Surrounding characters of word boundaries• GTF, ATF: Word sequence, surrounding characters ofword boundaries, POS sequence, word-POS pair sequence, and surrounding characters of word-POS pairs• GTP, ATP: Word sequence, surrounding characters ofword boundaries, and surrounding characters of wordPOS pairs
Theoretically speaking the WS based on a pointwise
prediction can use any type of corpora containing one ormore word boundaries with the characters surrounding them.
So the pointwise WS can be trained from all types ofcorpora. The POS tagging based on a pointwise prediction
can use any type of corpora containing one or more words
annotated with their POSs with the characters surrounding
them. In the above list, this applies to GWP, GWF, AWP, andAWF. POS tagging based on a sequence prediction can use
corpora in which sentences are divided into a word sequenceand the words are annotated with their POSs without any
missing elements. In the above list, this applies to GTF and
ATF.
In practical domain adaptation situations the available
training data are a large GTF and AWP or ATP which
are relatively easy to build. Full annotation corpora (AWF
and ATF) are costly because it requires both linguistic anddomain knowledge to build them. Figure 5 summarizes these
remarks. In this figure the corpora connected by solid lines
are usable by the processes listed on the right hand side: theWS or POS tagging based on pointwise prediction and the
POS reranking based on sequence prediction.
IV. E VALUATION
In order to test the effectiveness of the proposed method,
we conducted two experiments. One is a comparison on thegeneral domain among existing methods and the proposed
method. The other is a comparison among the major methods
in a domain adaptation situation. We set the parameters n
in n-gram to 2 and the window width m, m0 , and m00 to 5
in all cases based on the results of preliminary experiments.
We divided the training corpus into 9 parts in the training
data creation for the POS reranking (see Section III-E). For
the sequence labeling we used CRFsuite [19].

Usage
Training
Test
Training
Test#Sent.
27,338
3,038
5,800
645

#Words782,584
87,458114,265
13,018#Char.
1,131,317
126,154
158,000
17,980A. Corpus
The corpus we used is the core part of Balanced Corpus of Contemporary Written Japanese (BCCWJ) [17] The
sentences are divided into words and each word is annotated with a POS. We only used 21 coarse grained POS
tags. The sources are white papers, books, newspapers, andYahoo!QA. As [17] states, Yahoo!QA is different from theothers. Thus we regard Yahoo!QA as the target domain andthe others as the general domain. Table I shows the corpus
specifications.
B. Evaluation Criterion
As an evaluation criterion we follow [10] and use precision and recall based on word-POS pairs. First the longest
common subsequence (LCS) is found between the correct
answer and system output. Then let NREF be the numberof word-POS pairs in the correct sentence, NSY S be that in
the output in a system, and NLCS be that in the LCS of thecorrect sentence and the output of the system, so the recall
R and precision P are defined as follows:
NLCS
NLCS
, P =
.
NREF
NSY S
Finally we calculate F-measure defined as the harmonicmean of the recall and the precision:
}−1
{
2NLCS
1 −1
(R + P −1 )
=
.
F =
2
NREF + NSY S
R=

C. Evaluation 1: Comparison with Existing Methods
First we compared our method with popular existing
methods in the general domain. The methods are basedon POS 2-grams model2 [10], word-POS pair n-grams (n= 2,3) [11], CRFs (MeCab) [12], or pointwise prediction(KyTea) [8]. In this experiment, we assumed that only thefull annotation corpus (GTF in Figure 5 and 6) is available
to compare our method with existing ones trained from thesame language resources.
To train the CRFs for reranking in the proposed method,
we used the corpus generated from the general domain
corpus produced by the procedure described in Subsection
III-E. We tested the methods on the corpora in generaldomain and in the target domain. First we performed MA
using pointwise prediction and then reranked the resulted
POSs using sequence-based prediction.
2 [10] reports POS 3-gram model but POS 3-gram model is less accurate
than word-POS pair 3-gram model.

Table II
ACCURACIES OF WS AND THE JOINT PROBLEM ON THE GENERAL DOMAIN .

MethodPOS 2-gram model (HMM)
Pair 2-gram model
Pair 3-gram model
CRFs (MeCab)
Pointwise (KyTea)
Pointwise + Reranking

Word boundary estimation
Precision [%]Recall [%]F-measure
96.32
96.84
96.58
97.44
98.52
97.98
97.49
98.53
98.00
97.19
98.30
97.74
98.73
98.71
98.72
98.73
98.71
98.72

Precision [%]
93.77
96.58
96.70
96.72
98.07
98.38

Joint
Recall [%]
94.27
97.65
97.73
97.84
98.06
98.37

F-measure
94.02
97.11
97.21
97.28
98.06
98.38

Table III
ACCURACIES OF WS AND THE JOINT PROBLEM ON THE TARGET DOMAIN (YAHOO !QA).
Joint
Recall [%]
87.96
94.09
94.24
95.65
95.51
96.18

Training corpus in the general domain(full annotation)
Test corpus in thetarget domain

1. Train1.

Word segmenter and
POS tagger based on
pointwise prediction
Am
big
u

POS reranker basedon sequence prediction

F-measure
87.36
93.04
93.16
94.66
95.35
96.02

Training corpus in the target domain(partial annotation)

Test
Test

Table II and III show the accuracies in the generaldomain and the target domain respectively. In these tables,“pointwise” means the results of “pointwise prediction,”
the second oval from the top in Figure 5. “pointwise +reranking” means the results of the POS reranking by the
proposed method, that is the third oval in Figure 5. Since
we do not rerank the WS results, word boundary estimation
accuracies of these two methods are the same. From thetables we can say that the proposed method improves thejoint problem accuracy both in the general domain andthe target domain. The improvement is larger in the target
domain. From these results, our assumption that the POS
transition tendencies does not depend on the domain (seeSection III) is plausible and we can improve PW-MA basedon this assumption without losing the flexibility in choosing
language resources. From the above observations, we cansay that the proposed method is effective.

Precision [%]
86.78
92.01
92.10
93.69
95.19
95.86

ain
Tr

Training data addition

Word boundary estimation
Precision [%]Recall [%]F-measure
93.17
94.44
93.80
94.52
96.65
95.57
94.52
96.71
95.60
94.89
96.87
95.87
96.93
97.26
97.09
96.93
97.26
97.09

Test

MethodPOS 2-gram model (HMM)
Pair 2-gram model
Pair 3-gram model
CRFs (MeCab)
Pointwise (KyTea)
Pointwise + reranking

ou

sp

oin

ts
ele

cti

on
Manual annotationof 100 points

Result for
Evaluation

Figure 7. Domain adaptation scheme based on active learning using partial
annotation.

D. Evaluation 2: Adaptation Case
Second, we evaluate our method in a domain adaptation
scenario. The existing method that is the most flexible inthis scenario is pointwise MA, as it is trainable from partial
annotations. In the experiment, we emulated active learningby adding partially annotated sentences. Along with theproposed method we tested pointwise MA and sequencebased MA. We started with the training corpus in the generaldomain and added partially annotated sentences gradually.
The concrete procedure is as follows (see Figure 7).
1) Train the pointwise MA from the training corpus in
the general domain (GTF in Figure 5 and 6),2) Estimate confidences of the training corpus in thetarget domain by the above obtained model without
referring to the correct tags.

3) Annotate 100 points of low confidence in the corpus
in the target domain with word boundary or POS
producing a partial annotation corpus in the target
domain (ATP in Figure 5 and 6), and
4) Add the above partial annotation corpus to training
corpus and train the model again and go to 2).
We repeated this procedure for 200 iterations. Each time wemeasured the accuracies on the target domain. The baselines
are the pointwise MA (pointwise:part) trained from the same
corpus as the proposed method (pointwise+CRFsuite:part)and the CRFs with new words appearing in the partiallyannotation corpus added to the dictionary (CRF:part).
The result is shown in Figure 8. From this graph wesee that the proposed method outperforms the pointwise

Pointwise:part

96.60

F値
値
F-measure[3] M. P. Marcus and B. Santorini, “Building a large annotatedcorpus of English: The Penn treebank,” Computational Linguistics, vol. 19, no. 2, pp. 313–330, 1993.

Pointwise+CRFsuite:part97.10[4] Y. Tsuboi, H. Kashima, S. Mori, H. Oda, and Y. Matsumoto,“Training conditional random fields using incomplete annotations,” in Proceedings of the 22nd International Conference
on Computational Linguistics, 2008.

96.10

95.60

CRF:part

95.10

94.60
0

20

40

60

80

100

120

140

160

180

200

アノテーション形態素数（××100）） words)
アノテーション形態素数（
Numberof iterations (x100 annotatedFigure 8.

Learning curve in the case of domain adaptation.

MA consistently. In addition the proposed method improves
the accuracy in the domain adaptation case. Putting it inother words, the proposed method successfully increased theaccuracy without losing the domain adaptability of pointwise
MA. Therefore we can say that the proposed method issuperior to existing ones in this case as well.
V. C ONCLUSION
In this paper we have proposed a POS tagging method
allowing flexible usage of language resources. The method
is based on pointwise prediction and reranking by sequencebased prediction combined in the cascaded manner. The experimental results showed that the accuracy in the resourcerich domain is higher than existing methods. In a domain adaptation scenario where we add partially annotatedcorpora, the proposed method outperformed the existing
pointwise method constantly. These results showed thatthe proposed method is capable of providing high domain
adaptability while keeping high accuracy in the generaldomain.
Interesting research directions include testing POS tagging
in other languages and the application of our reranking
technique in various sequence labeling problems in NLP orother fields.
ACKNOWLEDGMENTS
This work was supported by JSPS Grants-in-Aid for
Scientific Research Grant Numbers 26280084, 26540190,
24240030, and 26280039.
R EFERENCES[1] S. J. DeRose, “Grammatical category disambiguation bystatistical optimization,” Computational Linguistics, vol. 14,
no. 1, pp. 31–39, 1988.
[2] B. Merialdo, “Tagging English text with a probabilistic
model,” Computational Linguistics, vol. 20, no. 2, pp. 155–
171, 1994.

[5] F. Pereira and Y. Schabes, “Inside-outside reestimation frompartially bracketed corpora,” in Proceedings of the 30th
Annual Meeting of the Association for Computational Linguistics, 1992, pp. 128–135.
[6] Y. Liu, Y. Zhang, W. Che, T. Liu, and F. Wu, “Domain
adaptation for CRF-based Chinese word segmentation using
free annotations,” in Proceedings of the 2014 Conference on
Empirical Methods in Natural Language Processing, 2014,
pp. 864–874.
[7] G. Neubig and S. Mori, “Word-based partial annotation for
efficient corpus construction,” in Proceedings of the SeventhInternational Conference on Language Resources and Evaluation, 2010.
[8] G. Neubig, Y. Nakata, and S. Mori, “Pointwise prediction
for robust, adaptable Japanese morphological analysis,” inProceedings of the 49th Annual Meeting of the Association
for Computational Linguistics, 2011, pp. 529–533.
[9] E. Charniak and M. Johnson, “Coarse-to-fine n-best parsing
and maxent discriminative reranking,” in Proceedings of the
43rd Annual Meeting of the Association for Computational
Linguistics, 2005, pp. 173–180.
[10] M. Nagata, “A stochastic Japanese morphological analyzer
using a forward-DP backward-A∗ n-best search algorithm,”in Proceedings of the 15th International Conference on Computational Linguistics, 1994, pp. 201–207.
[11] S. Mori and G. Kurata, “Class-based variable memory length
Markov model,” in Proceedings of the InterSpeech, 2005, pp.
13–16.
[12] T. Kudo, K. Yamamoto, and Y. Matsumoto, “Applying conditional random fields to Japanese morphological analysis,”in Proceedings of the Conference on Empirical Methods in
Natural Language Processing, 2004, pp. 230–237.
[13] R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J.
Lin, “LIBLINEAR: A library for large linear classification,”
Journal of Machine Learning Research, vol. 9, pp. 1871–
1874, 2008.
[14] J. Lafferty, A. McCallum, and F. Pereira, “Conditional random fields: Probabilistic models for segmenting and labelingsequence data,” in Proceedings of the Eighteenth ICML, 2001.
[15] F. Jelinek, “Self-organized language modeling for speech
recognition,” IBM T. J. Watson Research Center, Tech. Rep.,
1985.
[16] EDR Electronic Dictionary Technical Guide, Japan Electronic
Dictionary Research Institute, Ltd., 1993.

[17] K. Maekawa, M. Yamazaki, T. Maruyama, M. Yamaguchi,
H. Ogura, W. Kashino, T. Ogiso, H. Koiso, and Y. Den,
“Design, compilation, and preliminary analyses of balanced
corpus of contemporary written Japanese,” in Proceedings of
the Seventh International Conference on Language Resources
and Evaluation, 2010.
[18] S. Mori and H. Oda, “Automatic word segmentation using
three types of dictionaries,” in Proceedings of the Eighth International Conference Pacific Association for Computational
Linguistics, 2009.
[19] N. Okazaki, “Crfsuite: a fast implementation of conditionalrandom fields,” 2007. [Online]. Available: http://www.
chokkan.org/software/crfsuite/