gaof23/citation_sentiment_corpus huggingface dataset:
NER proves to be a knowledgeintensive task, and it was reassuring to observe that System Resources Used F1 + LBJ-NER Wikipedia, Nonlocal Features, Word-class Model 90.80 (Suzuki and Isozaki, 2008) Semi-supervised on 1Gword unlabeled data 89.92 (Ando and Zhang, 2005) Semi-supervised on 27Mword unlabeled data 89.31 (Kazama and Torisawa, 2007a) Wikipedia 88.02 (Krishnan and Manning, 2006) Non-local Features 87.24 (Kazama and Torisawa, 2007b) Non-local Features 87.17 + (Finkel et al., 2005) Non-local Features 86.86 Table 7: Results for CoNLL03 data reported in the literature.
CRF (baseline)] 97.18 97.21  Table 7: POS tagging results of the previous top systems for PTB III data evaluated by label accuracy system test additional resources JESS-CM (CRF/HMM) 95.15 1G-word unlabeled data 94.67 15M-word unlabeled data (Ando and Zhang, 2005) 94.39 15M-word unlabeled data (Suzuki et al., 2007) 94.36 17M-word unlabeled data (Zhang et al., 2002) 94.17 full parser output (Kudo and Matsumoto, 2001) 93.91  [supervised CRF (baseline)] 93.88  Table 8: Syntactic chunking results of the previous top systems for CoNLL00 shared task data (F=1 score) 30-31 Aug. 1996 and 6-7 Dec. 1996 Reuters news articles, respectively.
test additional resources JESS-CM (CRF/HMM) 94.48 89.92 1G-word unlabeled data 93.66 89.36 37M-word unlabeled data (Ando and Zhang, 2005) 93.15 89.31 27M-word unlabeled data (Florian et al., 2003) 93.87 88.76 own large gazetteers, 2M-word labeled data (Suzuki et al., 2007) N/A 88.41 27M-word unlabeled data [sup.
2.4 Comparison with Hybrid Model SSL based on a hybrid generative/discriminative approach proposed in (Suzuki et al., 2007) has been defined as a log-linear model that discriminatively combines several discriminative models, pDi , and generative models, pGj , such that: R(y|x;,,) = producttext i p Di (y|x;i)i producttext j p Gj (xj,y;j)j summationtext y producttext i p Di (y|x;i)i producttext j p Gj (xj,y;j)j , where ={i}Ii=1, and ={{i}Ii=1,{j}I+Jj=I+1}.
Networks (Toutanova et al., 2003) 97.24 SVM (Gimenez and M`arquez, 2003) 97.05 ME based a bidirectional inference (Tsuruoka and Tsujii, 2005) 97.15 Guided learning for bidirectional sequence classification (Shen et al., 2007) 97.33 AdaBoost.SDF with candidate features (=2,=1,=100, W-dist) 97.32 AdaBoost.SDF with candidate features (=2,=10,=10, F-dist) 97.32 SVM with candidate features (C=0.1, d=2) 97.32 Text Chunking F=1 Regularized Winnow + full parser output (Zhang et al., 2001) 94.17 SVM-voting (Kudo and Matsumoto, 2001) 93.91 ASO + unlabeled data (Ando and Zhang, 2005) 94.39 CRF+Reranking(Kudo et al., 2005) 94.12 ME based a bidirectional inference (Tsuruoka and Tsujii, 2005) 93.70 LaSo (Approximate Large Margin Update) (Daume III and Marcu, 2005) 94.4 HySOL (Suzuki et al., 2007) 94.36 AdaBoost.SDF with candidate featuers (=2,=1,=, W-dist) 94.32 AdaBoost.SDF with candidate featuers (=2,=10,=10,W-dist) 94.30 SVM with candidate features (C=1, d=2) 94.31 One of the reasons that boosting-based classifiers realize faster classification speed is sparseness of rules.
For instance, word alignment models are often trained using the GIZA++ toolkit (Och and Ney, 2003); error minimizing training criteria such as the Minimum Error Rate Training (Och, 2003) are employed in order to learn feature function weights for log-linear models; and translation candidates are produced using phrase-based decoders (Koehn et al. , 2003) in combination with n-gram language models (Brants et al. , 2007).
1 Introduction Very large corpora obtained from the Web have been successfully utilized for many natural languageprocessing(NLP)applications, suchasprepositional phrase (PP) attachment, other-anaphora resolution, spellingcorrection, confusablewordsetdisambiguation and machine translation (Volk, 2001; Modjeska et al., 2003; Lapata and Keller, 2005; Atterer and Schutze, 2006; Brants et al., 2007).
To scale LMs to larger corpora with higher-order dependencies, researchers Work completed while this author was at Google Inc. have considered alternative parameterizations such as class-based models (Brown et al., 1992), model reduction techniques such as entropy-based pruning (Stolcke, 1998), novel represention schemes such as suffix arrays (Emami et al., 2007), Golomb Coding (Church et al., 2007) and distributed language models that scale more readily (Brants et al., 2007).
Table 2 shows the total space and number of bytes required per n-gram to encode the model under different schemes: LDC gzipd is the size of the files as delivered by LDC; Trie uses a compact trie representation (e.g., (Clarkson et al., 1997; Church et al., 2007)) with 3 byte word ids, 1 byte values, and 3 byte indices; Block encoding is the encoding used in (Brants et al., 2007); and randomized uses our novel randomized scheme with 12 error bits.
We have also used TPTs to encode n-gram count databases such as the Google 1T web n-gram database (Brants and Franz, 2006), but are not able to provide detailed results within the space limitations of this paper.4 5.1 Perplexity computation with 5-gram language models We compared the performance of TPT-encoded language models against three other language model implementations: the SRI language modeling toolkit (Stolcke, 2002), IRSTLM (Federico and Cettolo, 2007), and the language model implementation currently used in the Portage SMT system (Badr et al., 2007), which uses a pointer-based implementation but is able to perform fast LM filtering at load time.
2 Related Work There have been various efforts to integrate linguistic knowledge into SMT systems, either from the target side (Marcu et al., 2006; Hassan et al., 2007; Zollmann and Venugopal, 2006), the source side (Quirk et al., 2005; Liu et al., 2006; Huang et al., 2006) or both sides (Eisner, 2003; Ding et al., 2005; Koehn and Hoang, 2007), just to name a few.
c2009 Association for Computational Linguistics Improving Mid-Range Reordering using Templates of Factors Hieu Hoang School of Informatics University of Edinburgh h.hoang@sms.ed.ac.uk Philipp Koehn School of Informatics University of Edinburgh pkoehn@inf.ed.ac.uk Abstract We extend the factored translation model (Koehn and Hoang, 2007) to allow translations of longer phrases composed of factors such as POS and morphological tags to act as templates for the selection and reordering of surface phrase translation.
Chiang (2005) distinguishes statistical MT approaches that are  syntactic in a formal sense, going beyond the  nite-state underpinnings of phrasebased models, from approaches that are syntactic in a linguistic sense, i.e. taking advantage of a priori language knowledge in the form of annotations derived from human linguistic analysis or treebanking.1 The two forms of syntactic modeling are doubly dissociable: current research frameworks include systems that are  nite state but informed by linguistic annotation prior to training (e.g., (Koehn and Hoang, 2007; Birch et al., 2007; Hassan et al., 2007)), and also include systems employing contextfree models trained on parallel text without bene t of any prior linguistic analysis (e.g.
Furthermore, the BLEU score performance suggests that our model is not very powerful, but some interesting hints can be found in Table 3 when we compare our method with a 5-gram language model to a state-of-the-art system Moses (Koehn and Hoang, 2007) based on various evaluation metrics, including BLEU score, NIST score (Doddington, 2002), METEOR (Banerjee and Lavie, 2005), TER (Snover et al., 2006), WER and PER.
Unlike with factored models (Koehn and Hoang, 2007) or additional translation lexicons (Schwenk et al., 2008), we do not generate the surface form back from the lemma translation, which means that tense, gender and number information are 151 news-dev2009a representation OOV % METEOR BLEU NIST baseline surface form only 2.24 49.05 20.45 6.135 decoding lemma backoff 2.13 49.12 20.44 6.143 word alignment lemma+POS for all 2.24 48.87 20.36 6.145 lemma+POS for adj 2.25 48.94 20.46 6.131 lemma+POS for verbs 2.21 49.05 20.47 6.137 decoding + alignment backoff + all 2.10 48.97 20.36 6.147 backoff + adj 2.12 49.05 20.48 6.140 backoff + verbs 2.08 49.15 20.50 6.148 news-dev2009b representation OOV % METEOR BLEU NIST baseline surface form only 2.52 49.60 21.10 6.211 decoding lemma backoff 2.43 49.66 21.02 6.210 word alignment lemma+POS for all 2.53 49.56 21.03 6.199 lemma+POS for adj 2.52 49.74 21.00 6.213 lemma+POS for verbs 2.47 49.73 21.10 6.217 decoding+alignment backoff + all 2.44 49.59 20.92 6.194 backoff + adj 2.43 49.80 21.03 6.217 backoff + verbs 2.39 49.80 21.03 6.217 Table 2: Evaluation of the decoding backoff strategy, the modified word alignment strategy and their combination Input Meme sil demissionnait, la situation ne changerait pas.
Many strategies have been proposed to integrate morphology information in SMT, including factored translation models (Koehn and Hoang, 2007), adding a translation dictionary containing inflected forms to the training data (Schwenk et al., 2008), entirely replacing surface forms by representations built on lemmas and POS tags (Popovic and Ney, 2004), morphemes learned in an unsupervised manner (Virpojia et al., 2007), and using Porter stems and even 4-letter prefixes for word alignment (Watanabe et al., 2006).
While this is certainly a daunting task, it is possible that for annotation studies that do not require expert annotators and extensive annotator training, the newly available access to a large pool of inexpensive annotators, such as the Amazon Mechanical Turk scheme (Snow et al., 2008),4 or embedding the task in an online game played by volunteers (Poesio et al., 2008; von Ahn, 2006) could provide some solutions.
While early machine learning approaches for the task relied on local, discriminative classifiers (Soon et al., 2001; Ng and Cardie, 2002b; Morton, 2000; Kehler et al., 2004), more recent approaches use joint and/or global models (McCallum and Wellner, 2004; Ng, 2004; Daume III and Marcu, 2005; Denis and Baldridge, 2007a).
In contrast, globally optimized clustering decisions were reported in (Luo et al. , 2004) and (DaumeIII and Marcu, 2005a), where all clustering possibilities are considered by searching on a Bell tree representation or by using the Learning as Search Optimization (LaSO) framework (DaumeIII and Marcu, 2005b) respectively, but the first search is partial and driven by heuristics and the second one only looks back in text.
We have already shown in Section 3 how to solve (a); here we avoid (b) by maximizing conditional likelihood, marginalizing out the hidden variable, denotedz: max vector summationdisplay x,y p(x,y)log summationdisplay z pvector(y,z | x) (17) This sort of conditional training with hidden variables was carried out by Koo and Collins (2005), for example, in reranking; it is related to the information bottleneck method (Tishby et al. , 1999) and contrastive estimation (Smith and Eisner, 2005).
Some researchers (Lappin and Leass 1994; Kennedy and Boguraev 1996) use manually designed rules to take into account the grammatical role of the antecedent candidates as well as the governing relations between the candidate and the pronoun, while others use features determined over the parse tree in a machine-learning approach (Aone and Bennett 1995; Yang et al. 2004; Luo and Zitouni 2005).
Thispaperfocusesontheframeworkintroduced in Figure 2 for two reasons: (a) cautious al50 gorithms were shown to perform best for several NLP problems (including acquisition of IE patterns), and (b) it has nice theoretical properties: Abney (2004) showed that, regardless of the selection procedure, sequential bootstrapping algorithms converge to a local minimum of K, where K is an upper bound of the negative log likelihood of the data.
Formally, by distributional similarity (or co-occurrence similarity) of two words w 1 and w 2 , we mean that they tend to occur in similar contexts, for some definition of context; or that the set of words that w 1 tends to co-occur with is similar to the set that w 2 tends to co-occur with; or that if w 1 is substituted for w 2 in a context, its plausibility (Weeds 2003; Weeds and Weir 2005) is unchanged.
2 Evaluating SR measures Various approaches for computing semantic relatedness of words or concepts have been proposed, e.g. dictionary-based (Lesk, 1986), ontology-based (Wu and Palmer, 1994; Leacock and Chodorow, 1998), information-based (Resnik, 1995; Jiang and Conrath, 1997) or distributional (Weeds and Weir, 2005).
in at with use teacher school 11894.47020.1 28.9 0.0 teacher handbook 2.5 0.0 3.2 10.1 soldier gun 2.8 10.3 105.9 41.0 Table 5: A fragment of the CCxL space We use this space to measure relational similarity (Turney, 2006) of concept pairs, e.g., finding that the relation between teachers and handbooks is more similar to the one between soldiers and guns, than to the one between teachers and schools.
Congress of the Italian Association for Artificial Intelligence, Palermo, 1991 B. Boguraev, Building a Lexicon: the Contribution of Computers, IBM Report, T.J. Watson Research Center, 1991 M. Brent, Automatic Aquisition of Subcategorization frames from Untagged Texts, in (ACL, 1991) N. Calzolari, R. Bindi, Acquisition of Lexical Information from Corpus, in (COLING 1990) K. W. Church, P. Hanks, Word Association Norms, Mutual Information, and Lexicography, Computational Linguistics, vol.
The results of these studies have important applications in lexicography, to detect lexicosyntactic regularities (Church and Hanks, 19901 (Calzolari and Bindi,1990), such as, for example~ support verbs (e.g. "make-decision") prepositional verbs (e.g. "rely-upon") idioms, semantic relations (e.g. "part_of") and fixed expressions (e.g. "kick the bucket").
While we have observed reasonable results with both G 2 and Fisher's exact test, we have not yet discussed how these results compare to the results that can be obtained with a technique commonly used in corpus linguistics based on the mutual information (MI) measure (Church and Hanks 1990): I(x,y) --log 2 P(x,y) (4) P(x)P(y) In (4), y is the seed term and x a potential target word.
Probably the most widely used feature weighting function is (point-wise) Mutual Information (MI) (Church and Patrick 1990; Hindle 1990; Luk 1995; Lin 1998; Gauch, Wang, and Rachakonda 1999; Dagan 2000; Baroni and Vegnaduzzo 2004; Chklovski and Pantel 2004; Pantel and Ravichandran 2004; Pantel, Ravichandran, and Hovy 2004; Weeds, Weir, and McCarthy 2004), dened by: weight MI (w,f)=log 2 P(w,f) P(w)P(f) (1) We calculate the MI weights by the following statistics in the space of co-occurrence instances S: weight MI (w,f)=log 2 count(w,f) nrels count(w) count(f) (2) where count(w,f) is the frequency of the co-occurrence pair w,f  in S, count(w)and count(f) are the independent frequencies of w and f in S,andnrels is the size of S.High MI weights are assumed to correspond to strong wordfeature associations.
METRIC FORMULA Frequency (Guiliano, 1964) x yf Pointwise Mutual Information [PMI] (Church & Hanks, 1990) ( )xy x y2log /P P P True Mutual Information [TMI] (Manning, 1999) ( )xy 2 xy x ylog /P P P P Chi-Squared ( 2 ) (Church and Gale, 1991) { }{ },, 2( ) i X X Y Y i j i j i j j f     T-Score (Church & Hanks, 1990) 1 2 2 2 1 2 1 2 x x s s n n  + C-Values4 (Frantzi, Anadiou & Mima 2000) 2 is not nested 2 log ( ) log ( ) 1 ( ) ( ) a a b T a f f f b P T         where is the candidate string f( ) is its frequency in the corpus T is the set of candidate terms that contain P(T ) is the number of these candidate terms 609 1,700 of the three-word phrases are attested in the Lexile corpus.
A variety of methods have been applied, ranging from simple frequency (Justeson & Katz 1995), modified frequency measures such as c-values (Frantzi, Anadiou & Mima 2000, Maynard & Anadiou 2000) and standard statistical significance tests such as the t-test, the chi-squared test, and loglikelihood (Church and Hanks 1990, Dunning 1993), and information-based methods, e.g. pointwise mutual information (Church & Hanks 1990).
To examine the effects of including some known AMs on the performance, the following AMs had a 50% chance of being included in the initial population: pointwise mutual information (Church and Hanks, 1990), the Dice coefficient, and the heuristic measure defined in (Petrovic et al., 2006): H(a,b,c) =    2log f(abc)f(a)f(c) if POS(b) = X, log f(abc)f(a)f(b)f(c) otherwise.
INTRODUCTION Word associations have been studied for some time in the fields of psycholinguistics (by testing human subjects on words), linguistics (where meaning is often based on how words co-occur with each other), and more recently, by researchers in natural language processing (Church and Hanks, 1990; Hindle and Rooth, 1990; Dagan, 1990; McDonald et al. , 1990; Wilks et al. , 1990) using statistical measures to identify sets of associated words for use in various natural language processing tasks.
Statistical data about these various cooccurrence relations is employed for a variety of applications, such as speech recognition (Jelinek, 1990), language generation (Smadja and McKeown, 1990), lexicography (Church and Hanks, 1990), machine translation (Brown et al. , ; Sadler, 1989), information retrieval (Maarek and Smadja, 1989) and various disambiguation tasks (Dagan et al. , 1991; Hindle and Rooth, 1991; Grishman et al. , 1986; Dagan and Itai, 1990).
The mutual information of a cooccurrence pair, which measures the degree of association between the two words (Church and Hanks, 1990), is defined as (Fano, 1961): P(xly) I(x,y) -log 2 P(x,y) _ log 2 (1) P(x)P(y) P(x) = log 2 P(y\[x) P(Y) where P(x) and P(y) are the probabilities of the events x and y (occurrences of words, in our case) and P(x, y) is the probability of the joint event (a cooccurrence pair).
Thus, given a hyponym definition (O) and a set of candidate hypernym definitions, this method selects the candidate hypernym definition (E) which returns the maximum score given by formula (1): SC(O, E) : E cw(wi, wj) (I) 'wIEOAwj6E The cooccurrence weight (cw) between two words can be given by Cooccurrence Frequency, Mutual Information (Church and Hanks, 1990) or Association Ratio (Resnik, 1992).
For instance, Church and Hanks (1990) calculated SA in terms of mutual information between two words wl and w2: N * f(wl,w2) I(wl, w2) = log2 (1) f(wl)f(w2) here N is the size of the corpus used in the estimation, f(Wl, w2) is the frequency of the cooccurrence, f(wl) and f(w2) that of each word.
We then rank-order the P X|Y MI XY M Z Pr Z|Y MI ZY G092log [P X P Y P X P Y ] f Y [P XY P XY ] f XY [P XY P XY ] f XY M iG13X,X} jG13Y,Y} (f ij G09 ij ) 2 ij f XY G09 XY XY (1G09( XY /N)) f XY G09 XY f XY (1G09(f XY /N)) Table 1: Probabilistic Approaches METHOD FORMULA Frequency (Guiliano, 1964) f XY Pointwise Mutual Information (MI) (Fano, 1961; Church and Hanks, 1990) log (P / PP) 2XY XY Selectional Association (Resnik, 1996) Symmetric Conditional Probability (Ferreira and Pereira, 1999) P / PP XY X Y 2 Dice Formula (Dice, 1945) 2 f / (f +f ) XY X Y Log-likelihood (Dunning, 1993; (Daille, 1996).
There are several distance measures suitable for this purpose, such as the mutual information(Church and Hanks, 1990), the dice coefficient(Manning and Schueutze 8.5, 1999), the phi coefficient(Manning and Schuetze 5.3.3, 1999), the cosine measure(Manning and Schueutze 8.5, 1999) and the confidence(Arrawal and Srikant, 1995).
a11a29a9 thea13 thea15 a1a4a3a6a5 a11a29a9 thea13 thea15 a11a29a9 thea15 a11a29a9 thea15a1a0 a2 since a11a2a9 thea13 thea15a4a3 a11a29a9 thea15 a11a29a9 thea15 . Also note that in the case of phraseness of a bigram, the equation looks similar to pointwise mutual information (Church and Hanks, 1990), but they are different.
Such studies follow the empiricist approach to word meaning summarized best in the famous dictum of the British 3 linguist J.R. Firth: You shall know a word by the company it keeps. (Firth, 1957, p. 11) Context similarity has been used as a means of extracting collocations from corpora, e.g. by Church & Hanks (1990) and by Dunning (1993), of identifying word senses, e.g. by Yarowski (1995) and by Schutze (1998), of clustering verb classes, e.g. by Schulte im Walde (2003), and of inducing selectional restrictions of verbs, e.g. by Resnik (1993), by Abe & Li (1996), by Rooth et al.
A broad view of the possible scope of lexical semantics would thus be one which tries to chart out the systematic, generalizable aspects of word meanings, and of the relations between words, drawing on readily accessible sources of lexical knowledge, such as machine readable dictionaries, encyclopedias, and representative corpora, coupled with the kind of analytic apparatus that is needed to fruitfully explore such sources, for instance custom-built parsers to cope with dictionary definitions (Vossen 1990), statistical programs to deal with the distributional properties of lexical items in large corpora (Church & Hanks 1990) etc. At the same time this kind of massive data-acquisition should be made sensitive to the borders between perceptual experience, lexical knowledge and expert knowledge.
s e, the window to consider when extracting words related to word w, should span from postttuon w-5 to w+5 Maarek also defines the resolwng power of a parr m a document d as P = ~'Pd log Pc where Pd is the observed probabshty of appearance of the pan" m document d, Pc the observed probabdny of the pmr recorpus, and -log Pc the quantity of mformauon assocmted to the pmr It Is easdy seen that p wall be h|gher, the higher the frequency of the pmr m the document and the lower sts frequency m the corpus, which agrees wlth the sdea presented at the begmnmg of this sectton Church and Hanks (1990) propose the apphcatlon of the concept of mutual mformatton e(x,y) ~,(x.y) = hog2 ecx)e(y) 51 to the retrieval, ro a corpus, of pairs of lextcally related words They alsoconslder a word span of :e5 words and observe that "roterestrog" pmr, s generally present a mutual mformatxon above 3 Salton and.Allan (1995) foc~as on paragraph level Each paragraph Is represented by a weighed vector, where each element is a term (typically.
of the works of (Kuplec, Pedersen, and Chen, 1995) and (Brandow, Mltze, .and Ran, 1995), and advances summarmatlon technology by applynag corpus-based statistical NLP teehmques, robust information extraction, and readily avaalable on-hne resources Our prehxmnary experiments with combining different summarization features have been reported, and our current effort to learn to combine these features to produce the best summaries has been described The features derived by these robust NLP techmques were also utihzed m presentmg multiple summary.vtews to the user m a novel way References Advanced Research Projects Agency 1995 Proceed:rigs of S:zth Message Understanding Conference (MUC-6) Morgan Kanfmann Pubhshers Brandow, Ron, Karl Mltze, and Lisa Ran 1995 Automatic condensation of electromc pubhcatlous by sentence selection Information Processing and  Management, 31, forthcoming .Bull, Eric 1993 A Comps-based Approach to Language Learning Ph D thesm, Umverslty of Pennsylvania Church, Kenneth and Patrick Hanks 1990 Word  Aesoclatlon Norrns, Mutual Information, and Lexicography Computational Lmgmstscs, 16(1) Church, Kenneth W 1995 One term or two 9 In Proceedings of the 17th Annual International SIGIR Conference on Research and Development In Informatzon Retrzeral, pages 310-318 Edmundson, H P 1969 New methods m automatic abstracting Journal of the ACM, 16(2) 264-228 Fum, Dando, Glovanm Gmda, and Carlo Tasso 1985 Evalutatmg Importance A step towards text surnmarlzatlon In I3CAI85, pages 840-844IJCAi, AAAI Hahn, Udo 1990 Topic parsing Accounting for text macro structures m full-text analysm In format:on Processing and Management, 26(1)135170 Harman, Donna 1991 How effective is suttixang ~ Journal of the Amerlcan Sot:cry for Informatwn Sc:ence, 42(1) 7-15 Harman, Donna 1996 Overview of the fifth text retrieval conference (tree-5) In TREC-5 Conference Proceedings Jmg, Y and B Croft 1994 An Assoc:atwn Thesaurns for Informatzon Retrseval Umass Techmcal Report 94-I7 Center for Intelligent Information Retrieval, University of Massachusetts Johnson, F C, C D Prate, W J Black, and A P Neal 1993.