2015 Pacific Association for Computational Linguistics

Named Entity Recognizer Trainable from Partially Annotated Data
Tetsuro Sasada∗ , Shinsuke Mori∗ , Tatsuya Kawahara∗ and Yoko Yamakata†∗ Academic Center for Computing and Media Studies, Kyoto University,
Yoshidahonmachi, Sakyo-ku, Kyoto, 606-8501, Japan
Email: {sasada@ar.media.kyoto-u.ac.jp, forest@i.kyoto-u.ac.jp, kawahara@i.kyoto-u.ac.jp}† Graduate School of Informatics, Kyoto University,
Yoshidahonmachi, Sakyo-ku, Kyoto, 606-8501, Japan
Email: yamakata@dl.kuis.kyoto-u.ac.jp

addition, NLP users started to notice that task dependent
deﬁnitions of NEs are useful for a special purpose insteadof the general deﬁnition. Famous one is medical NE [5].
Obviously body part names, disease names, and proteinnames are important for information retrieval or text mining
in medical texts. Nowadays there are many applications of
NLP. For the reputation analysis of a company it is important
to distinguish the product names of the company fromthose of its competitors. For recipe search it is importantto recognize food name correctly in a certain context. For
example, a recipe entitled “hamburger of steak house” is nota steak recipe. So we want to ﬁgure out that “steak” in thiscontext is not a food. For these special NE deﬁnitions in the
beginning only a small annotated corpus is available and a
rapid and low cost development of an NER is called for.
In this background we propose an NER which we cantrain from partially annotated data. In NER case partiallyannotated data is a set of sentences in which only somewords are annotated with NE tags and others are not. In
practical cases they may be new NEs not appearing in asmall fully annotated data. Our method is composed of two
modules: (1) Enumerate NE tag with conﬁdence for eachword independently from the tags for other words and (2)NE tag sequence search referring to the tag-conﬁdence pairs.
We conducted several experiments comparing state-of-theart NERs in various scenarios. The results showed that our
method is effective both in the normal case and in adaptation
cases.

Abstract—In this paper we propose a named entity recognizer (NER) which we can train from partially annotateddata. As the natural language processing is getting to be
applied to diverse texts, there arise high demands for the NER
for new named entity (NE) deﬁnition in different domains.
For these special NE deﬁnitions, only a small annotatedcorpus is available in the beginning, and a rapid and low-costdevelopment of an NER is needed in practice. To satisfy theneeds, we propose the use of partially annotated data, which
is a set of sentences in which only a limited number of words
are annotated with NE tags. Our NER method uses two-passsearch for sequential labeling of NE tags: (1) enumerate NE
tags with conﬁdences for each word independently from thetags for other words and (2) the best NE tag sequence searchreferring to the tag-conﬁdence pairs by CRFs. For the ﬁrst-passmodule, our method uses partially annotated data to improve
the accuracy in the target domain. By this two-pass searchframework, our method is expected to incorporate tag sequence
statistics and to outperform state-of-the-art NERs based on asequence labeling while keeping the high domain adaptability.
We conducted several experiments comparing state-of-the-art
NERs in various scenarios. The results showed that our method
is effective both in the normal case and in adaptation cases.
Keywords-Partial annotation; Incomplete data; Named entity
recognition; Pointwise prediction; Sequence labeling; Recipe

I. I NTRODUCTION
One of the important natural language processing (NLP)
is to recognize spans of words in a text corresponding to
the real world and classify them into one of the classes
deﬁned in advance. In newspaper articles, which were themain target of NLP for long time, the classes are personnames, company names, amount of money, etc. as deﬁned
in [1]. In the researches they are called named entities (NEs)and the task of automatically recognizing them is called the
NE recognition (NER). NERs are useful for information
retrieval from newspaper articles, question and answering
about the world knowledge, and others. The NER task can
be considered as a sequence labeling and tried many methods
such as hidden markov model, conditional markov model,
support vector machine (SVM) with dynamic programming(DP), conditional random ﬁelds (CRFs), etc. [2] [3] [4].
As the NLP is getting to be used more and more widely,
the NER is applied to various texts in many languages. In

II. R ELATED WORK
The task we solve in this paper is NER. NER is a sequencelabeling problem and many solutions have been proposed[2], [6, inter alia]. To our best knowledge one of the stateof-the-art methods is based on CRFs [7]. In this method
ﬁrst they convert the training corpus annotated with NE tags
into so-called extended BIO system. B, I, and O stand forbegin, intermediate, and other, respectively. Let us assume
that there are NE types T1 , T2 , . . . , TJ , they annotate aword sequence w1 , w2 , . . . , wn of an NE of type Tj as
w1 /Tj -B, w2 /Tj -I, · · · , wn /Tj -I and a word not included in
any NE as w/O. In the BIO system, there are 2J +1 tags and10

2015 Pacific Association for Computational Linguistics

Table Ia word is annotated with one of them. The problem is similar
to POS tagging which assign a grammatical category tag toeach word, but in NER there are constraints on tag sequence.
For example, (wi /O, wi+1 /Tj -I), (wi /Tj -B, wi+1 /Tk -I) and(wi /Tj -I, wi+1 /Tk -I) (j = k), are illegal and we cannot
interpret them. The NER based on CRFs automatically
captures these constraints and outputs a legal sequence for
an input word sequence. Some researches use a pointwise(PW) classiﬁer such as a SVM or a logistic regression(LR) combined with a tag sequence search module basedon dynamic programing (DP) .
These NERs based on SVM+DP or LR+DP has anadvantage that they can use a partially annotate data fortraining, in which only some words are annotated with BIO
tags and many other words are not. This advantage is very
beneﬁcial especially in resource-poor situations such as NER
for a new NE deﬁnition or a new language. As it is wellknown, the coverage has a strong relationship with the NER
accuracy. And the trainability from partial annotations allows
annotators to focus on new NEs or an active learning [8][9] [10] [11] [12] to select annotation unit smaller thana sentence. These methods help us to build an NER for
shorter time and lower cost. This advantage may also allow
researchers to try to devise an NER method for using natural
annotations like HTML tags in Wikipedia, which is a hot
topic in the word segmentation research recently [13] [14].
Our method extends these pointwise NERs with a reranker
based on CRFs. Our BIO tag sequence search is moreaccurate than DP, so our method is expected to be better
than the pointwise NERs without losing their advantage,trainability from partially annotate data.
As the input of our NER we assume a word sequence
but not tagged with a part-of-speech (POS) tag. So when
we apply our method to languages without obvious wordboundary we need a word segmenter [15] [16]. The reason
why we do not assume POS tagger results is that someresearch has reported a severe degradation in the POS tagger
accuracy on texts in a new domain [17]. By assuming a word
sequence as the input, we can skip an adaptation of a POS
tagger to the target domain. As a result we can avoid thatan entire NLP system including our NER loses its domain
adaptability in real use.
The domain in which we test our NER in the experimentis cooking recipe. In the past the main target of NLP wasnewspaper articles but nowadays NLP is used in various
texts. For example, a special deﬁnition of NE for medical
texts has been deﬁned and medical NER had a great success[18] [5]. Our application, recipe texts, is one of the user
generated contents and have many potential applications
ranging from researches to real uses: recipe search [19],recipe summarization [20], cooking help system [21], procedural text understanding [22] [23], computer vision [24],[25], cooking robot [26], etc. The recipe NER has not been
as mature as the medical NER and only small training data

R -NE TAGS .

r-NE tag

F
T
D
Q
Ac
Af
Sf
St

Meaning
Food
Tool
Duration
Quantity
Action by the chef
Action by foods
State of foods
State of tools

is available. So it is a good test bet for an NER trainablefrom various types of training data, which is important inresource-poor domain and/or language. The NER method
which we propose in this paper is not limited to this domain
but is applicable to others.
III. R ECIPE NAMED E NTITY
The test data we use in the experiment is named entity
specially deﬁned for recipe texts (r-NE) [28]. Their structureis the same as the general NE [1]. An r-NE is a span of one
or more words without overlap. No NE boundary occurs in
the middle of a word. So a word in a sentence belongs to at
most one r-NE. An r-NE has one type label listed in Table I.
So we can say that only the type deﬁnition is different fromthe general NE. The types for the general NE are designed
to be useful for information retrieval from the newspaper
articles. Contrary r-NE types are useful to recognize actions,objects, and their status in the recipe texts. They are important for recipe text search [19], its understanding [26], andsymbol grounding for cooking videos [24], [25]. Similar to
NER for the general NE, NER for r-NE can be formalized
as a sequence labeling problem and many solutions for thegeneral NE [2], [6, inter alia] are applicable.
The reason why we use r-NE instead of the general NE
is that we want to solve a practical problem, in whichsome NLP application is under development and we want
to increase the NER accuracy in a resource-poor situation.
Our NER method is, however, applicable to the general NE
and other NE such as medical NE etc.
IV. 2- STEP NAMED E NTITY R ECOGNITION
The NER method which we propose in this paper iscomposed of two modules:
1) Enumerate BIO tag with conﬁdence for each word
independently from the tags for other words,
2) BIO tag sequence search referring to the tagconﬁdence pairs.
The procedure is similar to a POS tagger trainable frompartially annotated sentences [27]. In NER, however, thesecond process is necessary to output a consistent tag
seqquence. In this section we explain these one by one.

11

2015 Pacific Association for Computational Linguistics

Type
Charactern-gramCharactertype n-gram

Table II

Table IIIFEATURE SET OF THE LR.

FEATURE SET OF THE CRF S .

Feature templates
x−1 , x+1 ,x−2 x−1 , x−1 x+1 , x+1 x+2x−2 x−1 x+1 , x−1 x+1 x+2
c(x−1 ), c(x+1 ),c(x−2 )c(x−1 ), c(x−1 )c(x+1 ), c(x+1 )c(x+2 ),c(x−3 )c(x−2 )c(x−1 ), c(x−2 )c(x−1 )c(x+1 ),c(x−1 )c(x+1 )c(x+2 ), c(x+1 )c(x+2 )c(x+3 )Type
Word n-gram
Word type n-gramTag-conﬁdence pair
(LR+CRF only)

A. Tag-conﬁdence Pair Enumeration
Given an input word sequence, the ﬁrst module provides
pairs of a tag and its conﬁdence for each word to the second
module. In order to make this module trainable from partially annotated data, we propose to use a pointwise classiﬁer
which refers, as features, only to the information containedin the input word sequence but not to the estimation results(so-called dynamic features).
As it is clear from the above design, this module istrainable from partially annotated data, because we can justuse only the annotated words and its context as the training
data. The following example:ex.) Sprinkle black/F-B pepper/F-I and salt,
where only two words are annotated with BIO tags, isconverted into the training data as follows:
left context
BOS
BOS Sprinkle
Sprinkle black
black pepper
pepper andword
Sprinkle
black
pepperandsaltright context
black pepper
pepper andand salt
salt EOSEOS

PLR (t|w)
F-B
F-I
Ac-B
t Ac-I
T-B..
.
O
Figure 1.

Feature templates
w−1 , w+1 ,w−2 w−1 , w−1 w+1 , w+1 w+2c(w−1 ), c(w+1 ),c(w−2 )c(w−1 ), c(w−1 )c(w+1 ),c(w+1 )c(w+2 ),c(w−2 )c(w−1 )c(w+1 ),c(w−1 )c(w+1 )c(w+2 )t1 , si,1 , t2 , si,2 , · · · , t2J+1 , si,2J+1 

Sprinkle
0.00
0.00
0.99
0.01
0.00
..
.

black
0.40
0.10
0.00
0.00
0.50
..
.

w
pepper
0.37
0.63
0.00
0.00
0.00
..
.

and
0.00
0.00
0.00
0.00
0.00
..
.

salt
0.80
0.20
0.00
0.00
0.00
..
.

0.00

0.00

0.00

1.00

0.00

DP search for the most likely tag sequence.

1) DP Search: We can use a DP search to select the most
likely tag sequence, where the likelihood can be deﬁned the
product of the probabilities as follows:

tag
–
F-B
F-I
–
–

m

t̂1 = argmax

t1 , t2 , ..., tmmsi,j .

j=1

Figure 1 illustrates this DP search. The numbers in the bold
face indicate the selected node. In the search, we take theconstraints on the BIO tag sequence into consideration. For
example, as shown in Figure 1, “black/T-B pepper/F-I” isillegal. This is one of the baselines and we test this in theexperiments.
2) Sequence Labeling: Instead of the naive DP search,
we propose to use a sequence labeling to search for the best
sequence. By using a sequence labeling based on machinelearning techniques we can take more context information
into consideration such as tag sequence tendency for acertain word sequence, etc.
The input of this module at runtime is a word sequenceannotated with tag-conﬁdence pairs provided by the ﬁrst
module. Since at runtime the word sequence in focus is new
for the ﬁrst module, we have to emulate this situation at
the training time of the second module for the model tobe effective for new texts. Thus we execute the following
procedures:(i) Divide the training corpus into N parts of equal size,(ii) Build the i-th pointwise classiﬁer from N − 1 parts oftraining corpus excluding the i-th part, and(iii) Enumerate all the BIO tags with their conﬁdence foreach word in the i-th part by using the i-th pointwise

We train a pointwise classiﬁer such as SVM or LR [29],which estimate the tag for a word.
At runtime, different from the normal classiﬁcation task,
the classiﬁer enumerates all the possible tags and theirconﬁdence. As the conﬁdence we can use the margin fromthe separation hyper-plain in the SVM case or probability
in the LR case. In this paper we use an LR as the classiﬁer
and the conﬁdence si,j for each tag tj for a word wi in thecontext of x− , wi , x+ is calculated as follows:si,j = PLR (tj |x− , wi , x+ ).
The features are listed in Table II. c(·) is a function, whichmaps the character type of a word or a character (Chinese character, hiragana, Arabic number, etc.). So we have(t1 , si,1 , t2 , si,2 , · · · , t2J+1 , si,2J+1 ), where 2J + 1 isthe size of the BIO tag set and si,j is the conﬁdence of BIO
tag tj for word wi .
B. Search for the Best Sequence
The second module is to search the best tag sequence
given a word sequence annotated with tag-conﬁdence pairs
provided by the ﬁrst module.

122015 Pacific Association for Computational Linguistics

Training corpus (word/BIO full annotation and partial annotation)
2nd 1/3 sub-corpus

3rd 1/3 sub-corpus

Estimate tag-confidence pairsby pointwise classifier (2nd+3rd)

Estimate tag-confidence pairsby pointwise classifier (1st+3rd)

Estimate tag-confidence pairsby pointwise classifier (1st+2nd)

1st 1/3 sub-corpus
with tag-confidence pairs

2nd 1/3 sub-corpus
with tag-confidence pairs

3rd 1/3 sub-corpus
with tag-confidence pairs

Test

1st 1/3 sub-corpus

Train

Train
Test

BIO-tagging by
sequence labeling(1st + 2nd + 3rd)Figure 2.

Test corpus

Procedure for generating the training corpora for BIO tagging by the sequence labeling (N = 3).
Table IVC ORPUS SPECIFICATION .

classiﬁer.
Figure 2 shows these procedures. As a result we can annotate
the words in the training corpus with tag-conﬁdence pairs(t1 , si,1 , t2 , si,2 , · · · , t2J+1 , si,2J+1 ) estimated by apointwise classiﬁer built from training data not containing
the words in focus1 . That is to say, we can successfully
emulate the runtime situation.
Now we are ready to train the second module. This partis formulated as a sequence labeling. The training data is aset of word sequences annotated with tag-conﬁdence pairs
provided by the ﬁrst module. The training data is similar
to Figure 1 except that the correct tags for each word (label
sequence) are attached in addition. As the sequence labelingwe use CRFs [7], but we can use any other sequence labelingmethods such as structured SVM [31]. Table III lists thefeatures referred to by the second module. Note that in many
NER researches POSs of the word in focus or in the context
are also referred to as features, but we do not do it to keep
the domain adaptability of our method in the entire system as
we stated in Section II. If we assumed the POS as the input,
we would have to spend time and cost to adapt a POS tagger
to the target domain in order to build a practically valuable
NER system.

Usage
Train
Test
(Total)#recipes
386
50
436

#sentences
2,946
371
3,317

#r-NEs17,243
1,996
19,239#words54,4706,072
60,542#characters82,393
9,167
91,560

Table VE XPERIMENTAL SETTINGS ABOUT TRAINING CORPUS .
training corpus set
1/2 FULL
1/2 FULL + 1/2 PART
1/1 FULL#sentences
1,473
2,946
2,946

#r-NEs
8,543
10,81017,243#BIO tags
27,11931,77054,470V. E VALUATIONAs evaluations of our NER, we measured the accuraciesof our NER and other methods under various settings. In
this section we present the results and evaluate our NER.
A. Experimental Settings
The domain in the experiments below is cooking recipe.
The NE deﬁnition is described in Section III which isdifferent from the general one for newspaper articles [1].
So we test our method mainly in a relatively resourcepoor situation. The corpus we used is procedural text sentences fully annotated with r-NE [28]. Table IV shows the

1 We can also use so-called leaving-one-out technique [30], but it iscomputationally too costly because we have to build as many models as
the number of words in the training data.

13

2015 Pacific Association for Computational Linguistics

Table VIIR ESULT: 1/2 FULL AND 1/2 PARTIAL ANNOTAION CORPUS .

Table VIR ESULT:1/2 FULL ANNOTAION CORPUS .
methodCRF
LR
LR+DP
LR+CRF (proposed)

BIO Accu.
0.89490.89300.89510.8989Precision
0.84910.84410.83970.8591

Recall
0.83720.84070.84770.8402F-measure
0.84380.84240.84370.8495methodCRF
LR
LR+DP
LR+CRF (proposed)

BIO Accu.
0.89900.89950.90120.9112

Precision
0.86120.85590.85390.8773

Recall
0.84520.84520.85520.8632F-measure0.85310.85050.85460.8702

Table VIIIR ESULT: 1/1 FULL ANNOTAION CORPUS .

speciﬁcations of the corpus. We see in this paper that the
number of sentences, 2,946 + 371, is much smaller than the
corpus annotated with general domain NE (normally morethan 10,000 sentences). In addition the sentences tend to
be much shorter than newspaper articles, thus the numberof annotated NE instances is much smaller that the general
NER case. For a detailed description about the NE deﬁnition
and the corpus the readers may refer to [28].
In the experiments we divided the training data into
two parts in order to test our NER and others simulating
resource-poor situations, or the beginning of a project which
NLP is applied to. The concrete settings are as follows:

methodCRF
LR
LR+DP
LR+CRF (proposed)

BIO Accu.
0.90650.90560.90690.9157

Precision
0.87590.87130.86960.8853

Recall
0.86270.85820.86520.8742F-measure0.86930.86470.86740.8798

B. Evaluation Criterion
We adopt two criteria. The ﬁrst one is the tag accuracy,
the percentage of the BIO tags correctly estimated by the
NER system. The second is F-measure, which is the standard
criterion for the NER task. The F-measure is the harmonicmean of precision and recall. Let Nsys , Nref , and Nint be
the number of the estimated NEs, the gold standard NEs, andtheir intersection, respectively. Then precision = Nint /Nsys ,recall = Nint /Nref , and F-measure = 2Nint /(Nref +Nsys ),the harmonic mean of them.

1/2 FULL: The ﬁrst half of training data is available as
fully annotated corpus,• 1/2 FULL + 1/2 PART: In addition to the ﬁrst half, thesecond half is available as a partially annotated corpus,• 1/1 FULL: The entire training data is available as fully
annotated corpus, that is the training data size is twiceas large as the 1/2 FULL case.
•

C. Evaluation

Table V shows the numbers of r-NEs and those of BIO
tags in the above settings. In the partially annotated corpus,
1/2 PART, we emulated the situation where new r-NEs notcontained in 1/2 FULL are annotated three times (if thefrequency is less than 3, that number of times).
The methods we compared are as follows:

We compared our method LR + CRF with three methods:
CRF, LR, and LR + DP under three settings, 1/2 FULL,
1/2 FULL + 1/2 PART, and 1/1 FULL. Table VI, VII, and
VIII show the results. And Figure 3 shows the F-measuresof the same results in graph form. As we see in Figure 3, theproposed method, LR + CRF, outperforms the other threemethods, CRF, LR, and LR + DP) in all the cases. Below
we discuss the results in detail.
When a large full annotation corpus is available, that is thecase of Table VIII, CRF is better than LR, and LR + DP.
This is the reason why CRF is used as the state-of-the-art
method for NER task in recent researches [33]. However,in case where the size of the full annotation corpus issmall (Table VI) or a partially annotated corpus is available
additionally (Table VII), LR + DP is better than CRF. LR +DP is simple and not so bad because the machine learningpart is pointwise, not sequence labeling, thus its training
time is much shorter than CRF especially when a partiallyannotated corpus is available. As we see in the paper [32],training CRF from a partially annotated corpus requires anumber of iterations calculating the expected values of thepossible tags for each words without annotation and the timeneeded for training tends to be long. Contrary LR is basedon a pointwise classiﬁer and we can train it for short time
just by using the annotated words [34]. Thus we can saythat in the beginning of an NE tagging project with a new

CRF: Sequence labeling by conditional random ﬁelds
trainable from partially annotated data [32],• LR: Pointwise classiﬁcation by a logistic regression[29] without DP search,• LR + DP: LR with DP search,• LR + CRF: LR with the best tag sequence searchby conditional random ﬁelds trained from the fully
annotate data only (proposed method; see Section IV).
•In LR + CRF, we divided the fully annotate training into
3 to create the corpus containing sentences of words with
tag-conﬁdence pairs (see Section IV-B and Figure 2).
As the implementation of CRFs which we can train frompartially annotated data [32] we used partial-crfsuite toolkit 2[14]. As an LR classiﬁer we adopt KyTea toolkit3 [16]. Table
III and II show the feature sets of CRF and LR, respectively,
respectively. The 2nd module of LR + CRF uses the tagconﬁdence pairs as features in addition.
2 https://github.com/ExpResults/partial-crfsuite
3 http://www.phontron.com/kytea/

14

2015 Pacific Association for Computational Linguistics&ͲŵĞĂƐƵƌĞ
Ϭ͘ϴϵ
Ϭ͘ϴϴ
Ϭ͘ϴϳ
Ϭ͘ϴϲ
Ϭ͘ϴϱ
Ϭ͘ϴϰ
Ϭ͘ϴϯ
Ϭ͘ϴϮ

ϭͬϮ&h>>нϭͬϮWZd

ϭͬϮ&h>>Figure 3.

ϭͬϭ&h>>NER accuracies.

NE deﬁnition, LR + DP is suitable allowing frequent model
updates especially when use an active learning technique [8][16].
In real situations, we want to maximize the accuracy fora certain annotation cost. As we have pointed out, it is goodto concentrate annotation work on informative words. One
simple strategy is to annotate new r-NEs for a few times toincrease the coverage. 1/2 FULL + 1/2 PART represents thissituation, where new r-NEs are annotated at most for threetimes4 . As we have pointed out above, our method, LR +CRF, is the best in this case as well. The important point is,however, the differences in F-measure between LR + CRF
and the others are very large in this case (see Figure 3). This
result indicates that our method is effective for constructing
an NE recognizer in real situations. Surprisingly Figure 3
clearly shows that LR + CRF trained from 1/2 FULL + 1/2
PART is better than the others trained from 1/1 FULL. As
shown in Table V, the number of additional r-NE annotationsfor 1/2 FULL + 1/2 PART from 1/2 FULL (2,267 = 10,810- 8,543) is around a quarter of 1/1 FULL from 1/2 FULL
(8,700 = 17,243 - 8,543). So we can say that with LR + CRF
we need less annotation work to achieve a higher accuracy.
In addition the time needed for training LR + CRF froma partially annotated corpus is as short as LR and LR +DP, and much shorter than CRF, because we only need to
update the ﬁrst part, the pointwise classiﬁer which is thesame as those in LR and LR + DP. Therefore we can saythat after development of a small fully annotated corpus it
is a good strategy to annotate new NEs providing a partiallyannotated corpus and to use our method, LR + CRF, which

is trained from that partially annotated corpus.
VI. C ONCLUSION
In this paper we have proposed a method for recognizing named entities. Our method is trainable from partiallyannotated data and we have experimentally shown that our
method is better than extisting ones in both the situations
where only fully annotated data is available and wherepartially annotated data is additionally available. Thus our
method is useful not only for the normal setting but also forresource-poor domains and/or languages.
An interesting research direction is to try to improve NER
by using partially annotated texts converted from wikipediaor other hyper texts. Active learning is another good research
direction. Our method allows more ﬂexible units to beannotated selection to make active learning more effective.
ACKNOWLEDGEMENT
This work was supported by JSPS Grants-in-Aid for
Scientiﬁc Research Grant, and JSPS Grant-in-Aid for Young
Scientists Grant. We are grateful to the annotators fortheir contribution to the design of the guidelines and theannotation effort.
R EFERENCES[1] N. A. Chinchor, “Overview of muc-7/met-2,” in Proceedingsof the Seventh Message Understanding Conference, 1998.
[2] A. Borthwick, “A maximum entropy approach to named entity
recognition,” Ph.D. dissertation, New York University, 1999.
[3] J. R. Finkel, T. Grenager, and C. Manning, “Incorporating
non-local information into information extraction systems bygibbs sampling,” in Proceedings of the 43rd Annual Meetingof the Association for Computational Linguistics, 2005, pp.
363–370.

4 This is a simulation and does not include real annotation work. An
experiment with the real annotation time is a future work.

15

2015 Pacific Association for Computational Linguistics[4] A. McCallum and W. Li, “Early results for named entity
recognition with conditional random ﬁelds, feature induction
and web-enhanced lexicons,” in Proceedings of the Seventh
Conference on Computational Natural Language Learning,
2003.

[17] G. Kevin, S. Nathan, O. Brendan, D. Dipanjan, M. Daniel,
E. Jacob, H. Michael, Y. Dani, F. Jeffrey, and S. N. A,“Part-of-speech tagging for twitter: Annotation, features, andexperiments,” in Proceedings of the ARPA Workshop on
Human Language Technology, 2011, pp. 42–47.

[5] A. Ben Abacha and P. Zweigenbaum, “Medical entity
recognition: A comparaison of semantic and statisticalmethods,” in Proceedings of BioNLP 2011 Workshop.
Portland, Oregon, USA: Association for Computational
Linguistics, June 2011, pp. 56–64. [Online]. Available:
http://www.aclweb.org/anthology/W11-0207[18] L. Ratinov and D. Roth, “Design challenges and misconceptions in named entity recognition,” in Proceedings of theThirteenth Conference on Computational Natural Language
Learning (CoNLL-2009). Boulder, Colorado: Association for
Computational Linguistics, June 2009, pp. 147–155. [Online].
Available: http://www.aclweb.org/anthology/W09-1119[6] E. F. T. K. Sang and F. D. Meulder, “Introduction to theconll-2003 shared task: Language-independent named entity
recognition,” in Proceedings of the Seventh Conference on
Computational Natural Language Learning, 2003, pp. 142–
147.

[19] L. Wang, Q. Li, N. Li, G. Dong, and Y. Yang, “Substructure
similarity measurement in chinese recipes,” in Proceedings ofthe 17th International Conference on World Wide Web, 2008,
pp. 978–988.

[7] J. Lafferty, A. McCallum, and F. Pereira, “Conditional random ﬁelds: Probabilistic models for segmenting and labelingsequence data,” in Proceedings of the Eighteenth ICML, 2001.

[20] Y. Yamakata, S. Imahori, Y. Sugiyama, S. Mori, andK. Tanaka, “Feature extraction and summarization of recipes
using ﬂow graph,” in Proceedings of the 5th International
Conference on Social Informatics, ser. LNCS 8238, 2013, pp.
241–254.

[8] B. Settles, M. Craven, and L. Friedland, “Active learning withreal annotation costs,” in NIPS Workshop on Cost-Sensitive
Learning, 2008.

[21] A. Hashimoto, N. Mori, T. Funatomi, Y. Yamakata,K. Kakusho, and M. Minoh, “Smart kitchen: A user centric cooking support system,” in Proceedings of the 12th
Information Processing and Management of Uncertainty inKnowledge-Based Systems, 2008, pp. 848–854.

[9] M. Sassano, “An empirical study of active learning withsupport vector machines for japanese word segmentation,” inProceedings of the 40th Annual Meeting of the Association
for Computational Linguistics, 2002, pp. 505–512.

[22] Y. Momouchi, “Control structures for actions in proceduraltexts and pt-chart,” in Proceedings of the Eighth International
Conference on Computational Linguistics, 1980, pp. 108–114.

[10] K. Tomanek and U. Hahn, “Semi-supervised active learningfor sequence labeling,” in Proceedings of the 47th Annual
Meeting of the Association for Computational Linguistics,
2009, pp. 1039–1047.

[23] H. Maeta, T. Sasada, and S. Mori, “A framework for recipe
text interpretation,” in Proceedings of the SixthInternational
Workshop on Cooking and Eating Activities, 2014.

[11] M. Tang, X. Luo, and S. Roukos, “Active learning forstatistical natural language parsing,” in Proceedings of the40th Annual Meeting of the Association for Computational
Linguistics, 2002, pp. 120–127.

[24] I. Naim, Y. C. Song, Q. Liu, H. Kautz, J. Luo, and D. Gildea,“Unsupervised alignment of natural language instructions
with video segments,” in Proceedings of the 28th National
Conference on Artiﬁcial Intelligence, 2014.

[12] Y. S. Chan and H. T. Ng, “Domain adaptation with active
learning for word sense disambiguation,” in Proceedings ofthe 45th Annual Meeting of the Association for Computational
Linguistics, 2007, pp. 49–56.

[25] M. Rohrbach, W. Qiu, I. Titov, S. Thater, M. Pinkal, and
B. Schiele, “Translating video content to natural language
descriptions,” in Proceedings of the 14th International Conference on Computer Vision, 2013.

[13] F. Yang and P. Vozila, “Semi-supervised chinese word segmentation using partial-label learning with conditional random ﬁelds,” in Proceedings of the 2014 Conference on
Empirical Methods in Natural Language Processing, 2014,
pp. 90–98.

[26] M. Bollini, S. Tellex, T. Thompson, N. Roy, and D. Rus,
“Interpreting and executing recipes with a cooking robot,”in Proceedings of The 13th International Symposium on
Experimental Robotics, 2013, pp. 481–495.

[14] Y. Liu, Y. Zhang, W. Che, T. Liu, and F. Wu, “Domain
adaptation for crf-based chinese word segmentation using
free annotations,” in Proceedings of the 2014 Conference on
Empirical Methods in Natural Language Processing, 2014,
pp. 864–874.
[15] R. Sproat and C. S. W. G. N. Chang, “A stochastic ﬁnite-state
word-segmentation algorithm for chinese,” Computational
Linguistics, vol. 22, no. 3, pp. 377–404, 1996.

[27] S. Mori, Y. Nakata, G. Neubig, and T. Sasada, “Pointwise
prediction and sequence-based reranking for adaptable partof-speech tagging,” in Proceedings of the Eleventh International Conference Paciﬁc Association for Computational
Linguistics, 2015.

[16] G. Neubig, Y. Nakata, and S. Mori, “Pointwise prediction
for robust, adaptable japanese morphological analysis,” inProceedings of the 49th Annual Meeting of the Association
for Computational Linguistics, 2011, pp. 529–533.

[28] S. Mori, H. Maeta, Y. Yamakata, and T. Sasada, “Flow graphcorpus from recipe texts,” in Proceedings of the Nineth International Conference on Language Resources and Evaluation,
2014, pp. 2370–2377.

16

2015 Pacific Association for Computational Linguistics[29] R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J.
Lin, “LIBLINEAR: A library for large linear classiﬁcation,”
Journal of Machine Learning Research, vol. 9, pp. 1871–
1874, 2008.
[30] R. Kneser and H. Ney, “Improved clustering techniques forclass-based statistical language modelling,” in Proceedings of
the Third European Conference on Speech Communication
and Technology, 1993, pp. 973–976.
[31] I. Tsochantaridis, T. Joachims, T. Hofmann, and Y. Altun,“Large margin methods for structured and interdependent
output variables,” Machine Learning, vol. 6, pp. 1453–1484,
2005.
[32] Y. Tsuboi, H. Kashima, S. Mori, H. Oda, and Y. Matsumoto,“Training conditional random ﬁelds using incomplete annotations,” in Proceedings of the 22nd International Conference
on Computational Linguistics, 2008.
[33] A. Neelakantan and M. Collins, “Learning dictionaries for
named entity recognition using minimal supervision,” in Proceedings of the 14th European Chapter of the Association for
Computational Linguistics, 2014, pp. 452–461.
[34] G. Neubig and S. Mori, “Word-based partial annotation for
efﬁcient corpus construction,” in Proceedings of the SeventhInternational Conference on Language Resources and Evaluation, 2010.

17