gaof23/citation_sentiment_corpus huggingface dataset:
Previous studies have shed light on the predictability of the next unix command that a user will enter (Motoda and Yoshida, 1997; Davison and Hirsch, 1998), the next keystrokes on a small input device such as a PDA (Darragh and Witten, 1992), and of the translation that a human translator will choose for a given foreign sentence (Nepveu et al. , 2004).
As a result, the empirical approach has been adopted by almost all contemporary part-of-speech programs: Bahl and Mercer (1976), Leech, Garside, and Atwell (1983), Jelinek (1985), Deroualt and Merialdo (1986), Garside, Leech, and Sampson (1987), Church (1988), DeRose (1988), Hindle (1989), Kupiec (1989, 1992), Ayuso et al.
Model Bits / Character ASCII Huffman code each char Lempel-Ziv (Unix TM compress) Unigram (Huffman code each word) Trigram Human Performance 8 5 4.43 2.1 (Brown, personal communication) 1.76 (Brown et al. 1992) 1.25 (Shannon 1951) The cross entropy, H, of a code and a source is given by: H(source, code) = ~ ~ Pr(s, h I source) log 2 Pr(s I h, code) s h where Pr(s, h I source) is the joint probability of a symbol s following a history h given the source.
Successful approaches aimed at trying to overcome the sparse data limitation include backoff (Katz 1987), Turing-Good variants (Good 1953; Church and Gale 1991), interpolation (Jelinek 1985), deleted estimation (Jelinek 1985; Church and Gale 1991), similarity-based models (Dagan, Pereira, and Lee 1994; Essen and Steinbiss 1992), Pos-language models (Derouault and Merialdo 1986) and decision tree models (Bahl et al. 1989; Black, Garside, and Leech 1993; Magerman 1994).
Much research has been carried out recently in this area (Hughes and Atwell 1994; Finch and Chater 1994; Redington, Chater, and Finch 1993; Brill et al. 1990; Kiss 1973; Pereira and Tishby 1992; Resnik 1993; Ney, Essen, and Kneser 1994; Matsukawa 1993).
The smoothing methods proposed in the literature (overviews are provided by Dagan, Lee, and Pereira (1999) and Lee (1999)) can be generally divided into three types: discounting (Katz 1987), class-based smoothing (Resnik 1993; Brown et al. 1992; 364 Computational Linguistics Volume 28, Number 3 Pereira, Tishby, and Lee 1993), and distance-weighted averaging (Grishman and Sterling 1994; Dagan, Lee, and Pereira 1999).
5.2 Pseudo-Disambiguation Task Pseudo-disambiguation tasks have become a standard evaluation technique (Gale, Church, and Yarowsky 1992; Sch utze 1992; Pereira, Tishby, and Lee 1993; Sch utze 1998; Lee 1999; Dagan, Lee, and Pereira 1999; Golding and Roth 1999; Rooth et al. 1999; EvenZohar and Roth 2000; Lee 2001; Clark and Weir 2002) and, in the current setting, we may use a nouns neighbors to decide which of two co-occurrences is the most likely.
Several authors (for example, Krovetz and Croft \[1989\], Guthrie et al. \[1991\], Slator \[1992\], Cowie, Guthrie, and Guthrie \[1992\], Janssen \[1992\], Braden-Harder \[1993\], Liddy and Paik \[1993\]) have attempted to improve results by using supplementary fields of information in the electronic version of the Longman Dictionary of Contemporary English (LDOCE), in particular, the box codes and subject codes provided for each sense.
Regardless of whether it takes the form of dictionaries (Lesk 1986; Guthrie et al. 1991; Dagan, Itai, and Schwall 1991; Karov and Edelman 1996), thesauri (Yarowsky 1992; Walker and Amsler 1986), bilingual corpora (Brown et al. 1991; Church and Gale 1991), or hand-labeled training sets (Hearst 1991; Leacock, Towell, and Voorhees 1993; Niwa and Nitta 1994; Bruce and Wiebe 1994), providing information for sense definitions can be a considerable burden.
The second approach (Sekine et al. 1992; Chang, Luo, and Su 1992; Resnik 1993a; Grishman and Sterling 1994; Alshawi and Carter 1994) takes triples (verb, prep, noun2) and (nounl, prep, noun2), like those in Table 10, as training data for acquiring semantic knowledge and performs PP-attachment disambiguation on quadruples.
Table 2: Three types of class-based MSLMs on Switchboard-I (swbd) and ICSI Meeting (mr) corpora # of swbd mr classes BROWN MMI MCMI BROWN MMI MCMI 100 68.9 0.3 68.4 0.3 68.2 0.3 78.9 3.0 77.3 2.8 76.8 2.8 500 68.9 0.3 68.3 0.3 67.9 0.3 78.7 3.1 77.1 2.8 76.7 2.8 1000 68.9 0.3 68.2 0.3 67.9 0.3 79.0 3.1 77.2 2.7 76.9 2.8 1500 69.0 0.3 68.2 0.3 68.0 0.3 79.6 3.1 77.4 2.7 77.4 2.7 2000 69.0 0.3 68.3 0.3 68.0 0.3 80.1 3.1 77.6 2.7 77.9 2.7 jV j 68.5 0.3 78.3 2.7 Table 3: Class-based MSLM on Switchboard Eval-2003 size 100 500 1000 1500 2000 jV j 3-gram 4-gram ppl 65.8 65.5 65.6 65.7 66.1 67.9 72.1 76.3 % reduction 8.6 8.9 8.8 8.7 8.3 5.8 0 -5.8 Class-based language models (Brown et al. , 1992; Whittaker and Woodland, 2003) yield great bene ts when data sparseness abounds.
We consider three class models, models S, M, and L, defined as pS(cj|c1cj1,w1wj1)=png(cj|cj2cj1) pS(wj|c1cj,w1wj1)=png(wj|cj) pM(cj|c1cj1,w1wj1)=png(cj|cj2cj1,wj2wj1) pM(wj|c1cj,w1wj1)=png(wj|wj2wj1cj) pL(cj|c1cj1,w1wj1)=png(cj|wj2cj2wj1cj1) pL(wj|c1cj,w1wj1)=png(wj|wj2cj2wj1cj1cj) Model S is an exponential version of the class-based n-gram model from (Brown et al., 1992); model M is a novel model introduced in (Chen, 2009); and model L is an exponential version of the model indexpredict from (Goodman, 2001).
4.2 Models with Prior Distributions Minimum discrimination information models (Della Pietra et al., 1992) are exponential models with a prior distribution q(y|x): p(y|x) = q(y|x)exp( summationtextF i=1 ifi(x,y)) Z(x) (14) The central issue in performance prediction for MDI models is whether q(y|x) needs to be accounted for.
We compare the following model types: conventional (i.e., non-exponential) word n-gram models; conventional IBM class n-gram models interpolated with conventional word n-gram models (Brown et al., 1992); and model M. All conventional n-gram models are smoothed with modified Kneser-Ney smoothing (Chen and Goodman, 1998), except we also evaluate word n-gram models with Katz smoothing (Katz, 1987).
While we can only compare class models with word models on the largest training set, for this training set model M outperforms the baseline Katzsmoothed word trigram model by 1.9% absolute.6 4 Domain Adaptation In this section, we introduce another heuristic for improving exponential models and show how this heuristic can be used to motivate a regularized version of minimum discrimination information (MDI) models (Della Pietra et al., 1992).
2 Related Work A large amount of previous research on clustering has been focused on how to find the best clusters [Brown et al. , 1992; Kneser and Ney, 1993; Yamamoto and Sagisaka, 1999; Ueberla, 1996; Pereira et al. , 1993; Bellegarda et al. , 1996; Bai et al. , 1998].
Proceedings of the Conference on Empirical Methods in Natural 2 Automatic Thesaurus Extraction The development of large thesauri and semantic resources, such as WordNet (Fellbaum, 1998), has allowed lexical semantic information to be leveraged to solve NLP tasks, including collocation discovery (Pearce, 2001), model estimation (Brown et al. , 1992; Clark and Weir, 2001) and text classi cation (Baker and McCallum, 1998).
This approach to term clustering is closely related to others from the literature (Brown et al. , 1992; Clark, 2000).2 Recall that the mutual information between random variables a0 and a1 can be written: a2a4a3a6a5a8a7a10a9a11a13a12a15a14a17a16a19a18a21a20a23a22a25a24a27a26a29a28 a14a17a16a19a18a21a20a23a22a25a24 a14a17a16a19a18a30a24a31a14a17a16a19a22a32a24 (1) Here, a0 and a1 correspond to term and context clusters, respectively, each event a18 and a22 the observation of some term and contextual term in the corpus.
The technique is based on word class models, pioneered by (Brown et al., 1992), which hierarchically 151 CoNLL03 CoNLL03 MUC7 MUC7 Web Component Test data Dev data Dev Test pages 1) Baseline 83.65 89.25 74.72 71.28 71.41 2) (1) + Gazetteer Match 87.22 91.61 85.83 80.43 74.46 3) (1) + Word Class Model 86.82 90.85 80.25 79.88 72.26 4) All External Knowledge 88.55 92.49 84.50 83.23 74.44 Table 4: Utility of external knowledge.
61 Distributional cluster (Brown et al. , 1992): tie, jacket, suit Word 'tie' (7 alternatives) 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 draw, standoff, tie, stalemate affiliation, association, tie, tie-up: a social or business relationship tie, crosstie, sleeper: subconcept of brace, bracing necktie, tie link, linkup, tie, tie-in: something that serves to join or link drawstring, string, tie: cord used as a fastener tie, tie beam: used to prevent two rafters, e.g., from spreading apart Word 'jacket' (4 alternatives) 0.0000 book jacket, dust cover: subeoncept of promotional material 0.0000 jacket crown, jacket: artificial crown fitted over a broken or decayed tooth 0.0000 jacket: subconceptofwrapping, wrap, wrapper 1.0000 jacket: a short coat Word 'suit' (4 alternatives) 0.0000 suit, suing: subconcept of entreaty, prayer, appeal 1.0000 suit, suit of clothes: subconcept of garment 0.0000 suit: any of four sets of13" cards in a paek 0.0000 legal action, action, case, lawsuit, suit: a judicial proceeding This cluster was derived by Brown et al. using a modification of their algorithm, designed to uncover "semantically sticky" clusters.
Distributional cluster (Brown et al. , 1992): cost, expense, risk, profitability, deferral, earmarks, capstone, cardinality, mintage, reseller Word 'cost' (2 alternatives) 0.5426 cost, price, terms, damage: the amount of money paid for something 0.4574 monetary value, price, cost: the amount of money it would bring if sold Word 'expense' (2 alternatives) 1.0000 expense, expenditure, outlay, outgo, spending, disbursal, disbursement 0.0000 expense: a detriment or sacrifice; "at the expense of" Word 'risk' (2 alternatives) 0.6267 hazard, jeopardy, peril risk: subconeept of danger 0.3733 risk, peril danger: subeonceptofventure Word 'profitability' (1 alternatives) 1.0000 profitableness, profitability: subconcept of advantage, benefit, usefulness Word 'deferral' (3 alternatives) 0.6267 abeyance, deferral, recess: subconcept of inaction, inactivity, inactiveness 0.3733 postponement, deferment, deferral, moratorium: an agreed suspension of activity 0.3733 deferral: subconeeptofpause, wait Word 'earmarks' (2 alternatives) 0.2898 earmark: identification mark on the ear of a domestic animal 0.7102 hallma.k, trademark, earmark: a distinguishing characteristic or attribute Word 'capstone' (1 alternatives) 1.0000 capstone, coping stone, stretcher: used at top of wall Word 'eardinality' Not in WordNet Word 'mintage' (1 alternatives) 62 1.0000 coinage, mintage, specie, metal money: subconcept of cash Word 'reseller' Not in WordNet This cluster was one presented by Brown et al. as a randomly-selected class, rather than one hand-picked for its coherence.
In all other respects, our work departs from previous research on broad--coverage 16 I t I I I I I i ! I i I I I I I I I I I I I i I 1, I. I I I I i I 1 I I I I probabilistic parsing, which either attempts to learn to predict gr~rarn~tical structure of test data directly from a training treebank (Brill, 1993; Collins, 1996; Eisner, 1996; Jelinek et al. , 1994; Magerman, 1995; S~kine and Orishman, 1995; Sharman et al. , 1990), or employs a grammar and sometimes a dictionary to capture linguistic expertise directly (Black et al. , 1993a; GrinBerg et al. , 1995; Schabes; 1992), but arguably at a less detailed and informative level than in the research reported here.
In contrast, approaches to WSD attempt to take advantage of many different sources of information (e.g. see (McRoy, 1992; Ng and Lee, 1996; Bruce and Wiebe, 1994)); it seems possible to obtain benefit from sources ranging from local collocational clues (Yarowsky, 1993) to membership in semantically or topically related word classes (arowsky, 1992; Resnik, 1993) to consistency of word usages within a discourse (Gale et al. , 1992); and disambignation seems highly lexically sensitive, in effect requiring specialized disamhignators for each polysemous word.
Though several algorithms (Brown et al. , 1992; Pereira, Tishby, and Lee, 1993) have been proposed 100( 9o( 80( 4O( 20( 1000 goo 80~ 41111 2@ 5 10 15 20 25 30 5 10 15 20 25 30 iteration of EM iteration of EM (a) (b) Figure 1: Plots of (a) training and (b) test perplexity versus number of iterations of the EM algorithm, for the aggregate Markov model with C = 32 classes.
(~(e) = max ((fl ,f~) ~ e) (23) (11  Y~) Now, the problem of learning probabilistic subcategorization preference is stated as: for every verb-noun collocation e in C, estimating the probability distribution P((fl, 6Resnik (1993) applys the idea of the KL distance to measuring the association of a verb v and its object noun class c. Our definition of ekt corresponds to an extension of Resnik's association score, which considers dependencies of more than one case-markers in a subcategorization frame.
toilet/bathroom Since the word "facility" is the subject of "employ" and is modified by "new" in (3), we retrieve other words that appeared in the same contexts and obtain the following two groups of selectors (the log A column shows the likelihood ratios (Dunning, 1993) of these words in the local contexts):  Subjects of "employ" with top-20 highest likelihood ratios: word freq, Iog,k word freq ORG" 64 50.4 plant 14 31.0 company 27 28.6 operation 8 23.0 industry 9 14.6 firm 8 13.5 pirate 2 12.1 unit 9 9.32 shift 3 8.48 postal service 2 7.73 machine 3 6.56 corporation 3 6.47 manufacturer 3 6.21 insurance company 2 6.06 aerospace 2 5.81 memory device 1 5.79 department 3 5.55 foreign office 1 5.41 enterprise 2 5.39 pilot 2 537 *ORG includes all proper names recognized as organizations 18  Modifiees of "new" with top-20 highest likelihood ratios: word freq log,k post 432 952.9 issue 805 902.8 product 675 888.6 rule 459 875.8 law 356 541.5 technology 237 382.7 generation 150 323.2 model 207 319.3 job 260 269.2 system 318 251.8 word freq log )~ bonds 223 245.4 capital 178 241.8 order 228 236.5 version 158 223.7 position 236 207.3 high 152 201.2 contract 279 198.1 bill 208 194.9 venture 123 193.7 program 283 183.8 Since the similarity between Sense 1 of "facility" and the selectors is greater than that of other senses, the word "facility" in (3) is tagged "Sense The key innovation of our algorithm is that a polysemous word is disambiguated with past usages of other words.
Several techniques and results have been reported on learning subcategorization frames (SFs) from text corpora (Webster and Marcus, 1989; Brent, 1991; Brent, 1993; Brent, 1994; Ushioda et al. , 1993; Manning, 1993; Ersan and Charniak, 1996; Briscoe and Carroll, 1997; Carroll and Minnen, 1998; Carroll and Rooth, 1998).
Using the values computed above: Pl -7tl k2 P2 --= -77, 2 kl +k2 p -7z 1 .-\]'It 2 Taking these probabilities to be binomially distributed, the log likelihood statistic (Dunning, 1993) is given by: 2 log A = 2\[log L(pt, k:l, rtl) @ log L(p2, k2, rl,2) -log L(p, kl, n2) log L(p, k2, n2)\] where, log L(p, n, k) = k logp + (,z -k)log(1 p) According to this statistic, tile greater the value of -2 log A for a particular pair of observed frame and verb, the more likely that frame is to be valid SF of the verb.
For example, in John saw Mary yesterday at the station, only John and Mary are required arguments while the other constituents are optional (adjuncts).3 The problem of SF identification using statistical methods has had a rich discussion in the literature (Ushioda et al. , 1993; Manning, 1993; Briscoe and Carroll, 1997; Brent, 1994) (also see the refences cited in (Sarkar and Zeman, 2000)).
Using the values computed above: p1 = k1n 1 p2 = k2n 2 p = k1+k2n 1+n2 Taking these probabilities to be binomially distributed, the log likelihood statistic (Dunning, 1993) is given by: 2 log = 2[log L(p1;k1;n1)+log L(p2;k2;n2) log L(p;k1;n2) log L(p;k2;n2)] where, log L(p;n;k)=k log p+(n k) log(1 p) According to this statistic, the greater the value of 2 log for a particular pair of observed frame and verb, the more likely that frame is to be valid SF of the verb.
The value of Dist(D(T)) can be defined in various ways, and they found that using log-likelihood ratio (see Dunning 1993) worked best which is represented as follows: 0 # log )(# log D K k TD k k i M ii i i M ii i  == , where k i and K i are the frequency of a word w i in D(W) and D 0 respectively, and {w 1,,w M } is the set of all words in D 0 . As stated in introduction, Dist(D(T)) is normalized by the baseline function, which is referred as B Dist () here.
Each word i in the context vector of w is then weighted with a measure of its association with w. We chose the loglikelihood ratio test, (Dunning, 1993), to measure this association the context vectors of the target words are then translated with our general bilingual dictionary, leaving the weights unchanged (when several translations are proposed by the dictionary, we consider all of them with the same weight) the similarity of each source word s, for each target word t, is computed on the basis of the cosine measure the similarities are then normalized to yield a probabilistic translation lexicon, P(t|s).
The candidates were then ranked according to the scores assigned by four association measures: the log-likelihood ratio G2 (Dunning, 1993), Pearsons chi-squared statistic X2 (Manning and Schutze, 1999, 169172), the t-score statistic t (Church et al. , 1991), and mere cooccurrence frequency f.4 TPs were identified according to the definition of Krenn (2000).
Almost all of these measures can be grouped into one of the following three categories: a0 frequency-based measures (e.g. , based on absolute and relative co-occurrence frequencies) a0 information-theoretic measures (e.g. , mutual information, entropy) a0 statistical measures (e.g. , chi-square, t-test, log-likelihood, Dices coefficient) The corresponding metrics have been extensively discussed in the literature both in terms of their mathematical properties (Dunning, 1993; Manning and Schutze, 1999) and their suitability for the task of collocation extraction (see Evert and Krenn (2001) and Krenn and Evert (2001) for recent evaluations).
However, while similarity measures (such as WordNet distance or Lins similarity metric) only detect cases of semantic similarity, association measures (such as the ones used by Poesio et al. , or by Garera and Yarowsky) also find cases of associative bridg497 Lin98 RFF TheY TheY:G2 PL03 Land (country/state/land) Staat Staat Kemalismus Regierung Kontinent state state Kemalism government continent Stadt Stadt Bauernfamilie Prasident Region city city agricultural family president region Region Landesregierung Bankgesellschaft Dollar Stadt region country government banking corporation dollar city Bundesrepublik Bundesregierung Baht Albanien Staat federal republic federal government Baht Albania state Republik Gewerkschaft Gasag Hauptstadt Bundesland republic trade union (a gas company) capital state Medikament (medical drug) Arzneimittel Pille RU Patient Arzneimittel pharmaceutical pill (a drug) patient pharmaceutical Praparat Droge Abtreibungspille Arzt Lebensmittel preparation drug (non-medical) abortion pill doctor foodstuff Pille Praparat Viagra Pille Praparat pill preparation Viagra pill preparation Hormon Pestizid Pharmakonzern Behandlung Behandlung hormone pesticide pharmaceutical company treatment treatment Lebensmittel Lebensmittel Praparat Abtreibungspille Arznei foodstuff foodstuff preparation abortion pill drug highest ranked words, with very rare words removed : RU 486, an abortifacient drug Lin98: Lins distributional similarity measure (Lin, 1998) RFF: Geffet and Dagans Relative Feature Focus measure (Geffet and Dagan, 2004) TheY: association measure introduced by Garera and Yarowsky (2006) TheY:G2: similar method using a log-likelihood-based statistic (see Dunning 1993) this statistic has a preference for higher-frequency terms PL03: semantic space association measure proposed by Pado and Lapata (2003) Table 1: Similarity and association measures: most similar items ing like 1a,b; the result of this can be seen in table (2): while the similarity measures (Lin98, RFF) list substitutable terms (which behave like synonyms in many contexts), the association measures (Garera and Yarowskys TheY measure, Pado and Lapatas association measure) also find non-compatible associations such as countrycapital or drugtreatment, which is why they are commonly called relationfree.
Proceedings of EACL '99 Determinants of Adjective-Noun Plausibility Maria Lapata and Scott McDonald and Frank Keller School of Cognitive Science Division of Informatics, University of Edinburgh 2 Buccleuch Place, Edinburgh EH8 9LW, UK {mlap, scottm, keller} @cogsci.ed.ac.uk Abstract This paper explores the determinants of adjective-noun plausibility by using correlation analysis to compare judgements elicited from human subjects with five corpus-based variables: co-occurrence frequency of the adjective-noun pair, noun frequency, conditional probability of the noun given the adjective, the log-likelihood ratio, and Resnik's (1993) selectional association measure.
5http://cl.cs.okayama-u.ac.jp/rsc/ jacabit/ a4a6a5 which gathers the set of co-occurrence units a7 associated with the number of times that a7 and a2 occur together a8a6a9a10a9 a5 a11 . In order to identify speci c words in the lexical context and to reduce word-frequency effects, we normalize context vectors using an association score such as Mutual Information (Fano, 1961) or Log-likelihood (Dunning, 1993).
All the enumerated segment pairs are listed in the following table: Feature x,y Feature x,y AM1+1 c1, c0 AM2+1 c2c1, c0 AM1+2 c1, c0c1 AM2+2 c2c1, c0c1 AM1+3 c1, c0c1c2 AM3+1 c3c2c1, c0 We use Dunnings method (Dunning, 1993) because it does not depend on the assumption of normality and it allows comparisons to be made between the signiflcance of the occurrences of both rare and common phenomenon.
A boundary-based model of co-occurrence assumes that both halves of the bitext have been segmented into s segments, so that segment Ui in one half of the bitext and segment Vi in the other half are mutual translations, 1 < i < s. Under the boundary-based model of co-occurrence, there are several ways to compute co-occurrence counts cooc(u, v) between word types u and v. In the models of Brown, Della Pietra, Della Pietra, and Mercer (1993), reviewed in Section 4.3, s COOC(R, V) = ~ ei(u) .j~(V), (12) i=1 where ei and j5 are the unigram frequencies of u and v, respectively, in each aligned text segment i. For most translation models, this method produces suboptimal results, however, when ei(u) > 1 and )~(v) > 1.
Dunning (1993) argues for the use of G 2 rather than X 2, based on the claim that the sampling distribution of G 2 approaches the true chi-square distribution quicker than the sampling distribution of X 2 . However, Agresti (1996, page 34) makes the opposite claim: The sampling distributions of X 2 and G 2 get closer to chi-squared as the sample size n increasesThe convergence is quicker for X 2 than G 2 . In addition, Pedersen (2001) questions whether one statistic should be preferred over the other for the bigram acquisition task and cites Cressie and Read (1984), who argue that there are some cases where the Pearson statistic is more reliable than the log-likelihood statistic.
The measures2  Mutual Information (a0a2a1 ) (Church and Hanks, 1989), the log-likelihood ratio test (Dunning, 1993), two statistical tests: t-test and a3a5a4 -test, and co-occurrence frequency  are applied to two sets of data: adjective-noun (AdjN) pairs and preposition-noun-verb (PNV) triples, where the AMs are applied to (PN,V) pairs.
Substituting the probabilities in the PMI formula with the previously introduced Web statistics, we obtain: a15a17a16a25a18a26a11a22a21 Qspa49a6a50a22a51a6a52 Aspa24 a15a17a16a25a18a26a11a22a21 Qspa24a56a55a57a15a33a16a19a18a26a11a6a21 Aspa24 a55 a38 a1a6a39a17a34a40a1a8a41a45a43a46a11 Maximal Likelihood Ratio (MLHR) is also used for word co-occurrence mining (Dunning, 1993).