gaof23/citation_sentiment_corpus huggingface dataset:
This shows that hypothesis features are either not discriminative enough, or that the reranking model is too weak This performance gap can be mainly attributed to two problems: optimization error and modeling error (see Figure 1).1 Much work has focused on developing better algorithms to tackle the optimization problem (e.g. MERT (Och, 2003)), since MT evaluation metrics such as BLEU and PER are riddled with local minima and are difficult to differentiate with respect to re-ranker parameters.
A path in a translation hypergraph induces a translation hypothesis E along with its sequence of SCFG rules D = r1,r2,,rK which, if applied to the start symbol, derives E. The sequence of SCFG rules induced by a path is also called a derivation tree for E. 3 Minimum Error Rate Training Given a set of source sentences FS1 with corresponding reference translations RS1, the objective of MERT is to find a parameter set M1 which minimizes an automated evaluation criterion under a linear model: M1 = argmin M1  SX s=1 Err`Rs, E(Fs; M1 ) ff E(Fs; M1 ) = argmax E  SX s=1 mhm(E, Fs) ff . In the context of statistical machine translation, the optimization procedure was first described in Och (2003) for N-best lists and later extended to phrase-lattices in Macherey et al.
As discussed in (Och, 2003), the direct translation model represents the probability of target sentence English e = e1eI being the translation for a source sentence French f = f1 fJ through an exponential, or log-linear model p(e|f) = exp( summationtextm k=1 k  hk(e,f))summationtext eprimeE exp( summationtextm k=1 k  hk(eprime,f)) (1) where e is a single candidate translation for f from the set of all English translations E,  is the parameter vector for the model, and each hk is a feature function of e and f. In practice, we restrict E to the set Gen(f) which is a set of highly likely translations discovered by a decoder (Vogel et al. , 2003).
To model p(t,a|s), we use a standard loglinear approach: p(t,a|s)  exp bracketleftBiggsummationdisplay i ifi(s,t,a) bracketrightBigg where each fi(s,t,a) is a feature function, and weights i are set using Ochs algorithm (Och, 2003) to maximize the systems BLEU score (Papineni et al. , 2001) on a development corpus.
Given a source sentence f, the preferred translation output is determined by computing the lowest-cost derivation (combination of hierarchical and glue rules) yielding f as its source side, where the cost of a derivation R1 Rn with respective feature vectors v1,,vn  Rm is given by msummationdisplay i=1 i nsummationdisplay j=1 (vj)i. Here, 1,,m are the parameters of the loglinear model, which we optimize on a held-out portion of the training set (2005 development data) using minimum-error-rate training (Och, 2003).
The real-valued features include the following: a block translation score derived from phrase occurrence statistics a4a9a113a77a11, a trigram language model to predict target words a4a179a112a229 a78a204a11, a lexical weighting score for the block internal words a4a127a202a204a11, a distortion model a4a0a207a229 a218a147a11 as well as the negative target phrase length a4a60a36a87a11 . The transition cost is computed as a19 a4a20a6 a23 a6 a39 a11a224a15 a27 a28 a30a89a32 a4a7a6 a83 a6a20a39a34a11, where a27 a199a230a227 a228 is a weight vector that sums up to a113a89a35a116 : a228 a13a26a17 a10 a27 a13a217a15a231a113a25a35a116 . The weights are trained using a procedure similar to (Och, 2003) on held-out test data.
To model p(t,a|s), we use a standard loglinear approach: p(t,a|s) ??exp bracketleftBiggsummationdisplay i ifi(s,t,a) bracketrightBigg (1) where each fi(s,t,a) is a feature function, and weights i are set using Och?s algorithm (Och, 2003) to maximize the system?s BLEU score (Papineni et al. , 2001) on a development corpus.
Unfortunately, longer sentences (up to 100 tokens, rather than 40), longer phrases (up to 10 tokens, rather than 7), two LMs (rather than just one), higher-order LMs (order 7, rather than 3), multiple higher-order lexicalized re-ordering models (up to 3), etc. all contributed to increased system?s complexity, and, as a result, time limitations prevented us from performing minimum-error-rate training (MERT) (Och, 2003) for ucb3, ucb4 and ucb5.
This is seen in that each time we check for the nearest intersection to the current 1-best for some n-best list l, we Algorithm 1 Och (2003)s line search method to find the global minimum in the loss, lscript, when starting at the point w and searching along the direction d using the candidate translations given in the collection of n-best lists L. Input: L, w, d, lscript I {} for l L do for e  l do m{e} e.features d b{e} e.features w end for bestn argmaxel m{e}{b{e} breaks ties} loop bestn+1 = argminel max parenleftBig 0, b{bestn}b{e}m{e}m{bestn} parenrightBig intercept  max parenleftBig 0, b{bestn}b{bestn+1}m{bestn+1}m{bestn} parenrightBig if intercept > 0 then add(I, intercept) else break end if end loop end for add(I, max(I)+2epsilon1) ibest = argminiI evallscript(L,w+(iepsilon1)d) return w+(ibest epsilon1)d must calculate its intersection with all other candidate translations that have yet to be selected as the 1-best.
For example, in IBM Model 1 the lexicon probability of source word f given target word e is calculated as (Och and Ney, 2003): p(f|e) = summationtext k c(f|e;e k,fk) summationtext k,f c(f|e;e k,fk) (1) c(f|e;ek,fk) = summationdisplay ek,fk P(ek,fk)summationdisplay a P(a|ek,fk) (2) summationdisplay j (f,fkj )(e,ekaj) Therefore, the distribution of P(ek,fk) will affect the alignment results.
4.1 Baseline Our baseline system is a fairly typical phrasebased machine translation system (Finch and Sumita, 2008a) built within the framework of a feature-based exponential model containing the following features: Table 1: Language Resources Corpus Train Dev Eval NC Spanish sentences 74K 2,001 2,007 words 2,048K 49,116 56,081 vocab 61K 9,047 8,638 length 27.6 24.5 27.9 OOV (%)  5.2 / 2.9 1.4 / 0.9 English sentences 74K 2,001 2,007 words 1,795K 46,524 49,693 vocab 47K 8,110 7,541 length 24.2 23.2 24.8 OOV (%)  5.2 / 2.9 1.2 / 0.9 perplexity  349 / 381 348 / 458 EP Spanish sentences 1,404K 1,861 2,000 words 41,003K 50,216 61,293 vocab 170K 7,422 8,251 length 29.2 27.0 30.6 OOV (%)  2.4 / 0.1 2.4 / 0.2 English sentences 1,404K 1,861 2,000 words 39,354K 48,663 59,145 vocab 121K 5,869 6,428 length 28.0 26.1 29.6 OOV (%)  1.8 / 0.1 1.9 / 0.1 perplexity  210 / 72 305 / 125 Table 2: Testset 2009 Corpus Test NC Spanish sentences 3,027 words 80,591 vocab 12,616 length 26.6  Source-target phrase translation probability  Inverse phrase translation probability  Source-target lexical weighting probability  Inverse lexical weighting probability  Phrase penalty  Language model probability  Lexical reordering probability  Simple distance-based distortion model  Word penalty For the training of the statistical models, standard word alignment (GIZA++ (Och and Ney, 2003)) and language modeling (SRILM (Stolcke, 2002)) tools were used.
3.2 Translation Scores The translation scores for four different systems are reported in Table 1.5 Baseline: In this system, we use the GIZA++ toolkit (Och and Ney, 2003), a suffix-array architecture (Lopez, 2007), the SRILM toolkit (Stolcke, 2002), and minimum error rate training (Och, 2003) to obtain word-alignments, a translation model, language models, and the optimal weights for combining these models, respectively.
In fact, when the perceptron update rule of (Dekel et al. , 2004)  which modifies the weights of every divergent node along the predicted and true paths  is used in the ranking framework, it becomes virtually identical with the standard, flat, ranking perceptron of Collins (2002).5 In contrast, our approach shares the idea of (Cesa-Bianchi et al. , 2006a) that if a parent class has been predicted wrongly, then errors in the children should not be taken into account. We also view this as one of the key ideas of the incremental perceptron algorithm of (Collins and Roark, 2004), which searches through a complex decision space step-by-step and is immediately updated at the first wrong move.
For comparison to previous results, table 2 lists the results on the testing set for our best model (TOP-Efficient-Freq20) and several other statistical parsers (Collins, 1999; Collins and Duffy, 2002; Collins and Roark, 2004; Henderson, 2003; Charniak, 2000; Collins, 2000; Shen and Joshi, 2004; Shen et al. , 2003; Henderson, 2004; Bod, 2003).
This algorithm and its many variants are widely used in the computational linguistics community (Collins, 2002a; Collins and Duffy, 2002; Collins, 2002b; Collins and Roark, 2004; Henderson and Titov, 2005; Viola and Narasimhan, 2005; Cohen et al., 2004; Carreras et al., 2005; Shen and Joshi, 2005; Ciaramita and Johnson, 2003).
3 Incremental Parsing Method Based on Adjoining Operation In order to avoid the problem of infinite local ambiguity, the previous works have adopted the following approaches: (1) a beam search strategy (Collins and Roark, 2004; Roark, 2001; Roark, 2004), (2) limiting the allowable chains to those actually observed in the treebank (Collins and Roark, 2004), and (3) transforming the parse trees with a selective left-corner transformation (Johnson and Roark, 2000) before inducing the allowable chains and allowable triples (Collins and Roark, 2004).
Each queue Hi stores the only N-best 43 Table 1: Parsing results LR(%) LP(%) F(%) Roark (2004) 86.4 86.8 86.6 Collins and Roark (2004) 86.5 86.8 86.7 No adjoining 86.3 86.8 86.6 Non-monotonic adjoining 86.1 87.1 86.6 Monotonic adjoining 87.2 87.7 87.4 partial parse trees.
With this model, we can provide not only qualitative textual summarization such as good food and bad service, but also a numerical scoring of sentiment, i.e., how good the food is and how bad the service is. 2 Related Work There have been many studies on sentiment classification and opinion summarization (Pang and Lee, 2004, 2005; Gamon et al., 2005; Popescu and Etzioni, 2005; Liu et al., 2005; Zhuang et al., 2006; Kim and Hovy, 2006).
We chose a dataset that would be enjoyable to reannotate: the movie review dataset of (Pang et al. , 2002; Pang and Lee, 2004).3 The dataset consists of 1000 positive and 1000 negative movie reviews obtained from the Internet Movie Database (IMDb) review archive, all written before 2002 by a total of 312 authors, with a cap of 20 reviews per author per 2Taking Ccontrast to be constant means that all rationales are equally valuable.
3 Data Sets We used three opinion-related data sets for our analyses and experiments: the OP data set created by (Wiebe et al. , 2004), the Polarity data set5 created by (Pang and Lee, 2004), and the MPQA data set created by (Wiebe et al. , 2005).6 The OP and Polarity data sets involve document-level opinion classi cation, while the MPQA data set involves 5Version v2.0, which is available at: http://www.cs.cornell.edu/people/pabo/movie-review-data/ 6Available at http://www.cs.pitt.edu/mpqa/databaserelease/ sentence-level classi cation.
3 CLaC-NB System: Nave Bayes Supervised statistical methods have been very successful in sentiment tagging of texts and in subjectivity detection at sentence level: on movie review texts they reach an accuracy of 85-90% (Aue and Gamon, 2005; Pang and Lee, 2004) and up to 92% accuracy on classifying movie review snippets into subjective and objective using both Nave Bayes and SVM (Pang and Lee, 2004).
In addition to CFG-oriented approaches, a number of richer treebank-based grammar acquisition and parsing methods based on HPSG (Miyao et al. , 2003), CCG (Clark and Hockenmaier, 2002), LFG (Riezler et al. , 2002; Cahill et al. , 2004) and Dependency Grammar (Nivre and Nilsson, 2005) incorporate non-local dependencies into their deep syntactic or semantic representations.
F (Cahill et al. , 2004) overall 95.98 57.86 72.20 73.00 40.28 51.91 90.16 54.35 67.82 65.54 36.16 46.61 args only 98.64 42.03 58.94 82.69 30.54 44.60 86.36 36.80 51.61 66.08 24.40 35.64 Basic Model overall 92.44 91.28 91.85 63.87 62.15 63.00 63.12 62.33 62.72 42.69 41.54 42.10 args only 89.42 92.95 91.15 60.89 63.45 62.15 47.92 49.81 48.84 31.41 32.73 32.06 Basic Model with Subject Path Constraint overall 92.16 91.36 91.76 63.72 62.20 62.95 75.96 75.30 75.63 50.82 49.61 50.21 args only 89.04 93.08 91.02 60.69 63.52 62.07 66.15 69.15 67.62 42.77 44.76 44.76 Table 7: Evaluation of trace insertion and antecedent recovery for C04 algorithm, our basic algorithm and basic algorithm with the subject path constraint.
In order to resolve all Chinese NLDs represented in the CTB, we modify and substantially extend the (Cahill et al. , 2004) (henceforth C04 for short) algorithm as follows: Given the set of subcat frames s for the word w, and a set of paths p for the trace t, the algorithm traverses the f-structure f to: predict a dislocated argument t at a sub-fstructure h by comparing the local PRED:w to ws subcat frames s t can be inserted at h if h together with t is complete and coherent relative to subcat frame s traverse f starting from t along the path p link t to its antecedent a if ps ending GF a exists in a sub-f-structure within f; or leave t without an antecedent if an empty path for t exists In the modified algorithm, we condition the probability of NLD path p (including the empty path without an antecedent) on the GF associated of the trace t rather than the antecedent a as in C04.
3.2 The parsers The parsers that we chose to evaluate are the C&C CCG parser (Clark and Curran, 2007), the Enju HPSG parser (Miyao and Tsujii, 2005), the RASP parser (Briscoe et al., 2006), the Stanford parser (Klein and Manning, 2003), and the DCU postprocessor of PTB parsers (Cahill et al., 2004), based on LFG and applied to the output of the Charniak and Johnson reranking parser.
Most of this work has so far focused either on post-processing to recover non-local dependencies from context-free parse trees (Johnson, 2002; Jijkoun and De Rijke, 2004; Levy and Manning, 2004; Campbell, 2004), or on incorporating nonlocal dependency information in nonterminal categories in constituency representations (Dienes and Dubey, 2003; Hockenmaier, 2003; Cahill et al. , 2004) or in the categories used to label arcs in dependency representations (Nivre and Nilsson, 2005).
We have also applied our more general unification grammar acquisition methodology to the TIGER Treebank (Brants et al. 2002) and Penn Chinese Treebank (Xue, Chiou, and Palmer 2002), extracting wide-coverage, probabilistic LFG grammar 361 Computational Linguistics Volume 31, Number 3 approximations and lexical resources for German (Cahill et al. 2003) and Chinese (Burke, Lam, et al. 2004).
Because treebank annotation for individual formalisms is prohibitively expensive, there have been a number of efforts to extract TAGs, LFGs, and, more recently, HPSGs, from the Penn Treebank (Xia 1999; Chen and Vijay-Shanker 2000; Xia, Palmer, and Joshi 2000; Xia 2001; Cahill et al. 2002; Miyao, Ninomiya, and Tsujii 2004; ODonovan et al. 2005; Shen and Joshi 2005; Chen, Bangalore, and Vijay-Shanker 2006).
Statistical parsers have been developed for TAG (Chiang 2000; Sarkar and Joshi 2003), LFG (Riezler et al. 2002; Kaplan et al. 2004; Cahill et al. 2004), and HPSG (Toutanova et al. 2002; Toutanova, Markova, and Manning 2004; Miyao and Tsujii 2004; Malouf and van Noord 2004), among others.
Even robust parsers using linguistically sophisticated formalisms, such as TAG (Chiang, 2000), CCG (Clark and Curran, 2004b; Hockenmaier, 2003), HPSG (Miyao et al. , 2004) and LFG (Riezler et al. , 2002; Cahill et al. , 2004), often use training data derived from the Penn Treebank.
c2006 Association for Computational Linguistics Robust PCFG-Based Generation using Automatically Acquired LFG Approximations Aoife Cahill1 and Josef van Genabith1,2 1 National Centre for Language Technology (NCLT) School of Computing, Dublin City University, Dublin 9, Ireland 2 Center for Advanced Studies, IBM Dublin, Ireland {acahill,josef}@computing.dcu.ie Abstract We present a novel PCFG-based architecture for robust probabilistic generation based on wide-coverage LFG approximations (Cahill et al. , 2004) automatically extracted from treebanks, maximising the probability of a tree given an f-structure.
1 Introduction Parsers have been developed for a variety of grammar formalisms, for example HPSG (Toutanova et al. , 2002; Malouf and van Noord, 2004), LFG (Kaplan et al. , 2004; Cahill et al. , 2004), TAG (Sarkar and Joshi, 2003), CCG (Hockenmaier and Steedman, 2002; Clark and Curran, 2004b), and variants of phrase-structure grammar (Briscoe et al. , 2006), including the phrase-structure grammar implicit in the Penn Treebank (Collins, 2003; Charniak, 2000).
1 Introduction A recent theme in parsing research has been the application of statistical methods to linguistically motivated grammars, for example LFG (Kaplan et al. , 2004; Cahill et al. , 2004), HPSG (Toutanova et al. , 2002; Malouf and van Noord, 2004), TAG (Sarkar and Joshi, 2003) and CCG (Hockenmaier andSteedman,2002; ClarkandCurran,2004b).
6 Related Work The most relevant previous works include word sense translation and translation disambiguation (Li & Li 2003; Cao & Li 2002; Koehn and Knight 2000; Kikui 1999; Fung et al. , 1999), frame semantic induction (Green et al. , 2004; Fung & Chen 2004), and bilingual semantic mapping (Fung & Chen 2004; Huang et al. 2004; Ploux & Ji, 2003, Ngai et al. , 2002; Palmer & Wu 1995).
6 Discussion Noting that adding latent features to nonterminals in unlexicalized context-free parsing has been very successful (Chiang and Bikel, 2002; Matsuzaki et al. , 2005; Prescher, 2005; Dreyer and Eisner, 2006; Petrov et al. , 2006), we were surprised not to see a 3Czech experiments were not done, since the number of features (more than 14 million) was too high to multiply out by clusters.
Building upon the large body of research to improve tagging performance for various languages using various models (e.g., (Thede and Harper, 1999; Brants, 2000; Tseng et al., 2005b; Huang et al., 2007)) and the recent work on PCFG grammars with latent annotations (Matsuzaki et al., 2005; Petrov et al., 2006), we will investigate the use of fine-grained latent annotations for Chinese POS tagging.
(2005) 86.6 86.7 1.19 61.1 Collins (1999) 88.7 88.5 0.92 66.7 Charniak and Johnson (2005) 90.1 90.1 0.74 70.1 This Paper 90.3 90.0 0.78 68.5 all sentences LP LR CB 0CB Klein and Manning (2003) 86.3 85.1 1.31 57.2 Matsuzaki et al.
3.1 A Note on State-Splits Recent studies (Klein and Manning, 2003; Matsuzaki et al. , 2005; Prescher, 2005; Petrov et al. , 2006) suggest that category-splits help in enhancing the performance of treebank grammars, and a previous study on MH (Tsarfaty, 2006) outlines specific POS-tags splits that improve MH parsing accuracy.
Although several methods have already been proposed to incorporate non-local features (Sutton and McCallum, 2004; Bunescu and Mooney, 2004; Finkel et al. , 2005; Roth and Yih, 2005; Krishnan and Manning, 2006; Nakagawa and Matsumoto, 2006), these present a problem that the types of non-local features are somewhat constrained.
Method dev test Finkel et al. , 2005 (Finkel et al. , 2005) baseline CRF 85.51 + non-local features 86.86 Krishnan and Manning, 2006 (Krishnan and Manning, 2006) baseline CRF 85.29 + non-local features 87.24 Table 5: Summary of performance with POS/chunk tags by TagChunk.
F-me. 1 CBC-NER system M 71.67 23.47 35.36CBC-NER system A 70.66 32.86 44.86 2 XIP NER 77.77 56.55 65.48 XIP + CBC M 78.41 60.26 68.15 XIP + CBC A 76.31 60.48 67.48 3 Stanford NER 67.94 68.01 67.97 Stanford + CBC M 69.40 71.07 70.23 Stanford + CBC A 70.09 72.93 71.48 4 GATE NER 63.30 56.88 59.92 GATE + CBC M 66.43 61.79 64.03 GATE + CBC A 66.51 63.10 64.76 5 Stanford + XIP 72.85 75.87 74.33 Stanford + XIP + CBC M 72.94 77.70 75.24 Stanford + XIP + CBC A 73.55 78.93 76.15 6 GATE + XIP 69.38 66.04 67.67 GATE + XIP + CBC M 69.62 67.79 68.69 GATE + XIP + CBC A 69.87 69.10 69.48 7 GATE + Stanford 63.12 69.32 66.07 GATE + Stanford + CBC M 65.09 72.05 68.39 GATE + Stanford + CBC A 65.66 73.25 69.25 Table 1: Results given by different hybrid NER systems and coupled with the CBC-NER system corpora (CoNLL, MUC6, MUC7 and ACE): ner-eng-ie.crf-3-all2008-distsim.ser.gz (Finkel et al., 2005) (line 3 in Table 1),  GATE NER or in short GATE (Cunningham et al., 2002) (line 4 in Table 1),  and several hybrid systems which are given by the combination of pairs taken among the set of the three last-mentioned NER systems (lines 5 to 7 in Table 1).
Some stem from work on graphical models,includingloopybeliefpropagation(Suttonand McCallum, 2004; Smith and Eisner, 2008), Gibbs sampling (Finkel et al., 2005), sequential Monte Carlo methods such as particle filtering (Levy et al., 2008), and variational inference (Jordan et al., 1999; MacKay, 1997; Kurihara and Sato, 2006).
Corpus Time Period Size Articles Words New Indian Express (English) 2007.01.01 to 2007.08.31 2,359 347,050 Dinamani (Tamil) 2007.01.01 to 2007.08.31 2,359 256,456 Table 1: Statistics on Comparable Corpora  From the above corpora, we first extracted all the NEs from the English side, using the Stanford NER tool [Finkel et al, 2005].
Starting out with a chunking pipeline, which uses a classical combination of tagger and chunker, with the Stanford POS tagger (Toutanova et al., 2003), the YamCha chunker (Kudoh and Matsumoto, 2000) and the Stanford Named Entity Recognizer (Finkel et al., 2005), the desire to use richer syntactic representations led to the development of a parsing pipeline, which uses Charniak and Johnsons reranking parser (Charniak and Johnson, 2005) to assign POS tags and uses base NPs as chunk equivalents, while also providing syntactic trees that can be used by feature extractors.
We thus introduce a multiplier  to form the actual objective function that we minimize with respect to :4 summationdisplay iL logp,i(yi ) +  Nsummationdisplay inegationslashL H(p,i) (4) One may regard  as a Lagrange multiplier that is used to constrain the classifiers uncertainty H to be low, as presented in the work on entropy regularization (Brand, 1999; Grandvalet and Bengio, 2005; Jiao et al. , 2006).
For example, minimum entropy regularization (Grandvalet and Bengio, 2004; Jiao et al., 2006), aims to maximize the conditional likelihood of labeled data while minimizing the conditional entropy of unlabeled data: summationdisplay i logp(y(i)|x(i)) 122bardblbardbl2H(y|x) (3) This approach generally would result in sharper models which can be data-sensitive in practice.
We divided these case roles into four types by location in the article as in (Iida et al., 2006), i) the case role depends on the predicate or the predicate depends on the case role in the intra-sentence (dependency relations), ii) the case role does not depend on the predicate and the predicate does not depend on the case role in the intra-sentence (zeroanaphoric (intra-sentential)), iii) the case role is not in the sentence containing the predicate (zeroanaphoric (inter-sentential)), and iv) the case role and the predicate are in the same phrase (in same phrase).
Others have introduced alternative discriminative training methods (Tillmann and Zhang, 2006; Liang et al., 2006; Turian et al., 2007; Blunsom et al., 2008; Macherey et al., 2008), in which a recurring challenge is scalability: to train many features, we need many train218 ing examples, and to train discriminatively, we need to search through all possible translations of each training example.
item form: [i,j,ueve] goal: [I,j,ue] rules:     [i,j,ue] R(fifiprime/ejejprime) [iprime,j,ejejprime] [i,j,ueejve] [i,j + 1,ueejve] ej+1 = rj+1 (Logic MONOTONE-ALIGN) Under the boolean semiring, this (minimal) logic decides if a training example is reachable by the model, which is required by some discriminative training regimens (Liang et al., 2006; Blunsom et al., 2008).
For this reason, to our knowledge, all discriminative models proposed to date either side-step the problem by choosing simple model and feature structures, such that spurious ambiguity is lessened or removed entirely (Ittycheriah and Roukos, 2007; Watanabe et al., 2007), or else ignore the problem and treat derivations as translations (Liang et al., 2006; Tillmann and Zhang, 2007).