gaof23/citation_sentiment_corpus huggingface dataset:
section 20 Majority voting (Mufioz et al. , 1999) (Tjong Kim Sang and Veenstra~ 1999) (Ramshaw and Marcus, 1995) (Argarnon et al. , 1998) accuracy precision O:98.10% C:98.29% 93.63% O:98.1% C:98.2% 93.1% 97.58% 92.50% 97.37% 91.80% 91.6% recall FZ=I 92.89% 93.26 92.4% 92.8 92.25% 92.37 92.27% 92.03 91.6% 91.6 section 00 accuracy precision Majority voting 0:98.59% C:98.65% 95.04% r (Tjong Kim Sang and Veenstra, 1999) 98.04% 93.71% (Ramshaw and Marcus, 1995) 97.8% 93.1% recall FB=I 94.75% 94.90 93.90% 93.81 93.5% 93.3 Table 3: The results of majority voting of different data representations applied to the two standard data sets put forward by (Ramshaw and Marcus, 1995) compared with earlier work.
And third, 1This (Ramshaw and Marcus, 1995) baseNP data set is available via ftp://ftp.cis.upenn.edu/pub/chunker/ 2Software for generating the data is available from http://lcg-www.uia.ac.be/conl199/npb/ 50 with the FZ=I rate which is equal to (2*precision*recall)/(precision+recall).
(1999) 91.6% 91.6% F/3=1 93.86 93.26 92.8 92.03 91.6 Table 3: The overall pertbrmance of the majority voting combination of our best five systems (selected on tinting data perfbrnmnce) applied to the standard data set pnt tbrward by Ramshaw and Marcus (1995) together with an overview of earlier work.
177 Proceedings of EACL '99 IOB1 IOB2 IOE1 IOE2 \[+\] \[+ IO IO +\] (Ramshaw and Marcus, 1995) (Veenstra, 1998) (Argamon et al. , 1998) (Cardie and Pierce, 1998) accuracy 97.58% 96.50% 97.58% 96.77% 97.37% 97.2% precision 92.50% 91.24% 92.41% 91.93% 93.66% 91.47% 91.25% 91.80% 89.0% 91.6 % 90.7% recall F~=I 92.25% 92.37 92.32% 91.78 92.04% 92.23 92.46% 92.20 90.81% 92.22 92.61% 92.04 92.54% 91.89 92.27% 92.03 94.3% 91.6 91.6% 91.6 91.1% 90.9 Table 6: The F~=I scores for the (Ramshaw and Marcus, 1995) test set after training with their training data set.
All formats 2The data described in (Ramshaw and Marcus, 1995) is available from ftp://ftp.cis.upenn.edu/pub/chunker/ 175 Proceedings of EACL '99 word/POS context chunk tag context IOB1 L=2/R=I IOB2 L--2/R=I IOE1 L=I/R=2 IOE2 L=I/R=2 \[ +\] L=2/R=I + L=0/R=2 \[ + IO L=2/R=0 + L=I/R=I IO +\] L=I/R=I+L=0/R=2 F~=I 1/2 90.12 1/0 89.30 1/2 89.55 0/1 89.73 0/0 + 0/0 89.32 0/0 + I/I 89.78 1/1 + 0/0 89.86 Table 3: Results second experiment series: the best F~=I scores for different left (L) and right (R) chunk tag context sizes for the seven representation formats using 5-fold cross-validation on section 15 of the WSJ corpus.
If one reduces the problem of entity mention detection to the detection of its head, the nature of the problem changes and the annotation of data becomes at; The [GPE Jordanian] [ORG military] [PER spokesman]  This allows us to consider the problem as a tagging/chunking problem and describe each word as beginning (B) an entity mention, inside (I) an entity mention or outside (O) an entity mention (Ramhsaw and Marcus, 1995; Sang and Veenstra, 1999).
(2002) 94.17 Li and Roth (2001) 93.02 94.64 Table 2: Baseline results on three shallow parsing tasks: the NP-Chunking task (Ramshaw and Marcus, 1995); the CoNLL-2000 Chunking task (Sang and Buchholz, 2000); and the Li & Roth task (Li and Roth, 2001), which is the same as CoNLL-2000 but with more training data and a different test section.
Toward a Task-based Gold Standard for Evaluation of NP Chunks and Technical Terms Nina Wacholder Rutgers University nina@scils.rutgers.edu Peng Song Rutgers University psong@paul.rutgers.edu Abstract We propose a gold standard for evaluating two types of information extraction output -noun phrase (NP) chunks (Abney 1991; Ramshaw and Marcus 1995) and technical terms (Justeson and Katz 1995; Daille 2000; Jacquemin 2002).
NP chunks (Abney 1991; Ramshaw and Marcus 1995; Evans and Zhai 1996; Frantzi and Ananiadou 1996) and technical terms (Dagan and Church 1994; Justeson and Katz 1995; Daille 1996; Jacquemin 2001; Bourigault et al. 2002) fall into this difficult-toassess category.
On the same dataset as that of (Chen et al. , 1999), our new supertagger achieves an accuracy of a2a4a3a6a5a8a7a10a9a12a11 . Compared with the supertaggers with the same decoding complexity (Chen, 2001), our algorithm achieves an error reduction of a22a23a5a26a9a12a11 . We repeat Ramshaw and Marcus Transformation Based NP chunking (Ramshaw and Marcus, 1995) test by substituting supertags for POS tags in the dataset.
Much previous work has been done on this problem and many different methods have been used: Church's PARTS (1988) program uses a Markov model; Bourigault (1992) uses heuristics along with a grammar; Voutilainen's NPTool (1993) uses a lexicon combined with a constraint grammar; Juteson and Katz (1995) use repeated phrases; Veenstra (1998), Argamon, Dagan & Krymolowski(1998) and Daelemaus, van den Bosch & Zavrel (1999) use memory-based systems; Ramshaw & Marcus (In Press) and Cardie & Pierce (1998) use rule-based systems.
Among the machine learning algorithms studied, rule based systems have proven effective on many natural language processing tasks, including part-of-speech tagging (Brill, 1995; Ramshaw and Marcus, 1994), spelling correction (Mangu and Brill, 1997), word-sense disambiguation (Gale et al. , 1992), message understanding (Day et al. , 1997), discourse tagging (Samuel et al. , 1998), accent restoration (Yarowsky, 1994), prepositional-phrase attachment (Brill and Resnik, 1994) and base noun phrase identification (Ramshaw and Marcus, In Press; Cardie and Pierce, 1998; Veenstra, 1998; Argamon et al. , 1998).
Adverbs/adverbial phrases becorae part of the VP chunk (as long as they are in front of the main verb): (VP could (ADVP very well) (VP show  )) "-+ \[ve could very well show \]  In contrast to Ramshaw and Marcus (1995), predicative adjectives of the verb are not part of the VP chunk, e.g. in "\[NP they \] \[vP are \] \[ADJP unhappy \]'.
00: the current input token and the previous one have the same parent  90: one ancestor of the current input token and the previous input token have the same parent  09: the current input token and one ancestor of the previous input token have the same parent  99 one ancestor of the current input token and one ancestor of the previous input token have the same parent Compared with the B-Chunk and I-Chunk used in Ramshaw and Marcus(1995)~, structural relations 99 and 90 correspond to B-Chunk which represents the first word of the chunk, and structural relations 00 and 09 correspond to I-Chunk which represents each other in the chunk while 90 also means the beginning of the sentence and 09 means the end of the sentence.
Type Precision Recall Fa=l Overall 96.40 96.47 96.44 NP 96.49 96.99 96.74 VP 97.13 97.36 97.25 ADJP 89.92 88.15 89.03 ADVP 91.52 87.57 89.50 97.13 97.36 PP 97.25 Table 16: Results of 25-fold cross-validation chunking experiments with the merged context-dependent lexicon Tables 14 and 16 shows that our new chunk tagger greatly outperforms other reported chunk taggers on the same training data and test data by 2%-3%.(Buchholz S. , Veenstra J. and Daelmans W.(1999), Ramshaw L.A. and Marcus M.P.(1995), Daelemans W. , Buchholz S. and Veenstra J.(1999), and Veenstra J.(1999)).
The concept of baseNP has undergone a number of revisions (Ramshaw and Marcus, 1995; Tjong Kim Sang and Buchholz, 2000) but has previously always been tied to extraction from a more completely annotated treebank, whose annotations are subject to other pressures than just initial material up to the head . To our knowledge, our gures for inter-annotator agreement on the baseNP task itself 169 (i.e. not derived from a larger annotation task) are the rst to be reported.
Introduction Since Eric Brill first introduced the method of Transformation-Based Learning (TBL) it has been used to learn rules for many natural language processing tasks, such as part-of-speech tagging \[Brill, 1995\], PPattachment disambiguation \[Brill and Resnik, 1994\], text chunking \[Ramshaw and Marcus, 1995\], spelling correction \[Mangu and Brill, 1997\], dialogue act tagging \[Samuel et al. , 1998\] and ellipsis resolution \[Hardt, 1998\].
1 Full Morphological Tagging English Part of Speech (POS) tagging has been widely described in the recent past, starting with the (Church, 1988) paper, followed by numerous others using various methods: neural networks (Julian Benello and Anderson, 1989), HMM tagging (Merialdo, 1992), decision trees (Schmid, 1994), transformation-based error-driven learning (Brill, 1995), and maximum entropy (Ratnaparkhi, 1996), to select just a few.
We can find some other machine-learning approaches that use more sophisticated LMs, such as Decision Trees (Mhrquez and Rodrfguez, 1998)(Magerman, 1996), memory-based approaclms to learn special decision trees (Daelemans et al. , 1996), maximmn entropy approaches that combine statistical information from different sources (Ratnaparkhi, 1996), finite state autonmt2 inferred using Grammatical Inference (Pla and Prieto, 1998), etc. The comparison among different al)t)roaches is dif ficult due to the nmltiple factors that can be eonsid614 ered: tile languagK, tile mmfl)er and tyt)e of the tags, the size of tilt vocabulary, thK ambiguity, the diiticulty of the test ski, Kte.
POS tagger: The maximum entropy POS tagger developed by Ratnaparkhi (Ratnaparkhi, 1996) and the rule-based POS tagger developed by Brill (Brill, 1994) are trained with 1200 abstracts extracted from the GENIA corpus, which achieve accuracies of 97.97% and 98.06% respectively, when testing on the rest 800 abstract of the GENIA corpus.
Second, several tagging experiments on newspaper language, whether statistical (Ratnaparkhi, 1996; Brants, 2000) or rule-based (Brill, 1995), report that the tagging accuracy for unknown words is much lower than the overall accuracy.2 Thus, the lower percentage of unknown words in medical texts seems to be a sublanguage feature beneficial to POS taggers, whereas the higher proportion of unknown words in newspaper language seems to be a prominent source of tagging errors.
The target set is built using the 88-89 Wall Street Journal Corpus (WSJ) tagged using the (Ratnaparkhi, 1996) tagger and the (Bangalore & Joshi, 1999) SuperTagger; the feedback sets are built using WSJ sentences con330 Algorithm 1 KE-train: (Karov & Edelman, 1998) algorithm adapted to literal/nonliteral classification Require: S: the set of sentences containing the target word Require: L: the set of literal seed sentences Require: N: the set of nonliteral seed sentences Require: W: the set of words/features, w  s means w is in sentence s, s owner w means s contains w Require: epsilon1: threshold that determines the stopping condition 1: w-sim0(wx,wy) := 1 if wx = wy,0 otherwise 2: s-simI0(sx,sy) := 1, for all sx,sy  S S where sx = sy, 0 otherwise 3: i := 0 4: while (true) do 5: s-simLi+1(sx,sy) := summationtextwxsx p(wx,sx)maxwysy w-simi(wx,wy), for all sx,sy  S L 6: s-simNi+1(sx,sy) := summationtextwxsx p(wx,sx)maxwysy w-simi(wx,wy), for all sx,sy  S N 7: for wx,wy  W W do 8: w-simi+1(wx,wy) := braceleftBigg i = 0 summationtextsxownerwx p(wx,sx)maxsyownerwy s-simIi(sx,sy) else summationtextsxownerwx p(wx,sx)maxsyownerwys-simLi (sx,sy),s-simNi (sx,sy)} 9: end for 10: if wx,maxwyw-simi+1(wx,wy)w-simi(wx,wy)}  epsilon1 then 11: break # algorithm converges in 1epsilon1 steps.
Models of that form include hidden Markov models (Rabiner, 1989; Bikel et al. , 1999) as well as discriminative tagging models based on maximum entropy classification (Ratnaparkhi, 1996; McCallum et al. , 2000), conditional random fields (Lafferty et al. , 2001; Sha and Pereira, 2003), and large-margin techniques (Kudo and Matsumoto, 2001; Taskar et al. , 2003).
The unknown word tokens are with respect to Training I. Data set Sect'ns Token Unknown Training I 26-270, 600-931 213986 Training II 600-931, 500-527, 1001-1039 204701 Training III 001-270, 301-527, 590-593, 600-1039, 1043-1151 485321 Devset 23839 2849 XH 001-025 7844 381 HKSAR 500-527 8202 1168 SM 590-593, 1001-1002 7793 1300 Test set 23522 2957 XH 271-300 8008 358 HKSAR 528-554 7153 1020 SM 594-596, 1040-1042 8361 1579 5.2 The model Our model builds on research into loglinear models by Ng and Low (2004), Toutanova et al. , (2003) and Ratnaparkhi (1996).
(Hakkani-Tur et al. , 2000)), and Basque (Ezeiza et al. , 1998), which pose quite different and in the end less severe problems, there have been attempts at solving this problem for some of the highly inflectional European languages, such as (Daelemans et al. , 1996), (Erjavec et al. , 1999) (Slovenian), (Hajic and Hladka, 1997), (Hajic and Hladka, 1998) (Czech) and (Hajic, 2000) (five Central and Eastern European languages), but so far no system has reached in the absolute terms a performance comparable to English tagging (such as (Ratnaparkhi, 1996)), which stands around or above 97%.
The standard split of the corpus into training (sections 222, 9,753 sentences), validation (section 24, 321 sentences), and testing (section 23, 603 sentences) was performed.2 As in (Henderson, 2003; Turian and Melamed, 2006) we used a publicly available tagger (Ratnaparkhi, 1996) to provide the part-of-speech tag for each word in the sentence.
The part of speech tags for the development and test data were automatically assigned by MXPOST (Ratnaparkhi, 1996), where the tagger was trained on the entire training corpus; to generate part of speech tags for the training data, we used 10-way jackknifing.8 English word clusters were derived from the BLLIP corpus (Charniak et al., 2000), which contains roughly 43 million words of Wall Street Journal text.9 The Czech experiments were performed on the Prague Dependency Treebank 1.0 (Hajic, 1998; Hajic et al., 2001), which is directly annotated with dependency structures.
8http://svmlight.joachims.org 9Our replication of the Wan et al. model is approximate, because we used different preprocessing tools: MXPOST for POS tagging (Ratnaparkhi, 1996), MSTParser for parsing (McDonald et al., 2005), and Dan Bikels interface (http://www.cis.upenn.edu/dbikel/ software.html#wn) to WordNet (Miller, 1995) for lemmatization information.
Since the advent of manually tagged corpora such as the Brown Corpus and the Penn Treebank (Francis(1982), Marcus(1993)), the efficacy of machine learning for training a tagger has been demonstrated using a wide array of techniques, including: Markov models, decision trees, connectionist machines, transformations, nearest-neighbor algorithms, and maximum entropy (Weischedel(1993), Black(1992), Schmid(1994), Brill(1995),Daelemans(1995),Ratnaparkhi(1996 )).
A possible solution to his problem might be the use of more general morphological rules like those used in part-of-speech tagging models (e.g. , 1 2 3 4 530 40 50 60 70 80 90 100 level error RAND BASE Boost_S NNtfidf NB Boost_M Figure 6: Comparison of all models for a129 a48a51a95a66a97a98a97a180a222 . Ratnaparkhi (1996)), where all suffixes up to a certain length are included.
3.1 Part-of-Speech (POS) of Neighboring Words We use 7 features to encode this knowledge source: a0a2a1a4a3a6a5a7a0a8a1a10a9a11a5a7a0a8a1a13a12a14a5a15a0a17a16a6a5a15a0a2a12a18a5a7a0a19a9a20a5a15a0a17a3, where a0a8a1 a21 (a0 a21 ) is the POS of thea6 th token to the left (right) ofa0, and a0a17a16 is the POS of a0 . A token can be a word or a punctuation symbol, and each of these neighboring tokens must be in the same sentence asa0 . We use a sentence segmentation program (Reynar and Ratnaparkhi, 1997) and a POS tagger (Ratnaparkhi, 1996) to segment the tokens surroundinga0 into sentences and assign POS tags to these tokens.
Output sequence optimization Rather than basing classifications only on model parameters estimated from co-occurrences between input and output symbols employed for maximizing the likelihood of point-wise single-label predictions at the output level, classifier output may be augmented by an optimization over the output sequence as a whole using optimization techniques such as beam searching in the space of a conditional markov models output (Ratnaparkhi, 1996) or hidden markov models (Skut and Brants, 1998).
)|(maxarg* STPT T = (1) Then we assume that the tagging of one character is independent of each other, and modify formula 1 as  == = = = n i ii tttT nn tttT ctP ccctttPT n n 1 2121 * )|(maxarg )|(maxarg 21 21 (2) Beam search (n=3) (Ratnaparkhi,1996) is applied for tag sequence searching, but we only search the valid sequences to ensure the validity of searching result.
Finally, we show in Section 7.3 that our SCL PoS 124 (a) 100 500 1k 5k 40k75 80 85 90 Results for 561 MEDLINE Test Sentences Number of WSJ Training Sentences Accuracy supervised semiASO SCL (b) Accuracy on 561-sentence test set Words Model All Unknown Ratnaparkhi (1996) 87.2 65.2 supervised 87.9 68.4 semi-ASO 88.4 70.9 SCL 88.9 72.0 (c) Statistical Significance (McNemars) for all words Null Hypothesis p-value semi-ASO vs. super 0.0015 SCL vs. super 2.1 1012 SCL vs. semi-ASO 0.0003 Figure 5: PoS tagging results with no target labeled training data (a) 50 100 200 500 86 88 90 92 94 96 Number of MEDLINE Training Sentences Accuracy Results for 561 MEDLINE Test Sentences 40kSCL 40ksuper 1kSCL 1ksuper nosource (b) 500 target domain training sentences Model Testing Accuracy nosource 94.5 1k-super 94.5 1k-SCL 95.0 40k-super 95.6 40k-SCL 96.1 (c) McNemars Test (500 training sentences) Null Hypothesis p-value 1k-super vs. nosource 0.732 1k-SCL vs. 1k-super 0.0003 40k-super vs. nosource 1.9 1012 40k-SCL vs. 40k-super 6.5 107 Figure 6: PoS tagging results with no target labeled training data tagger improves the performance of a dependency parser on the target domain.
(b) MEDLINE DT JJ VBN NNS IN DT NN NNS VBP The oncogenic mutated forms of the ras proteins are RB JJ CC VBP IN JJ NN NN . constitutively active and interfere with normal signal transduction . Figure 1: Part of speech-tagged sentences from both corpora we investigate its use in part of speech (PoS) tagging (Ratnaparkhi, 1996; Toutanova et al. , 2003).
Step Description mean stddev % 1.5 Sample 1.5s 0.07s 0.7% 1.6 Extraction 38.2s 0.13s 18.6% 1.7 Build tree 127.6s 27.60s 62.3% 1.8 Percolation 31.4s 4.91s 15.3% 1.911 Leaf updates 6.2s 1.75s 3.0% 1.511 Total 204.9s 32.6s 100.0% 2004),10 the only one that we were able to train and test under exactly the same experimental conditions (including the use of POS tags from Ratnaparkhi (1996)).
The features we used are as follows:  Direct and inverse IBM model;  3, 4-gram target language model;  3, 4, 5-gram POS language model (Ratnaparkhi, 1996; Schmid, 1994); 96  Sentence length posterior probability (Zens and Ney, 2006);  N-gram posterior probabilities within the NBest list (Zens and Ney, 2006);  Minimum Bayes Risk probability;  Length ratio between source and target sentence; The weights are optimized via MERT algorithm.
The training samples are respectively used to create the models PT^G, PCHUNK, PBUILD, and PCMECK, all of which have the form: k p(a, b) = II _ij(o,b ~j (1) j----1 where a is some action, b is some context, ~" is a nor4 Model Categories Description Templates Used TAG See (Ratnaparkhi, 1996) CHUNK chunkandpostag(n)* BUILD CHECK chunkandpostag(m, n)* cons(n) cons(re, n)* cons(m, n,p) T punctuation checkcons(n)* checkcons(m,n)* production surround(n)* The word, POS tag, and chunk tag of nth leaf.