gaof23/citation_sentiment_corpus huggingface dataset:
We evaluate the string chosen by the log-linear model against the original treebank string in terms of exact match and BLEU score (Papineni et al., 821 Syntactic feature Type Definites Definite descriptions SIMPLE DEF simple definite descriptions POSS DEF simple definite descriptions with a possessive determiner (pronoun or possibly genitive name) DEF ATTR ADJ definite descriptions with adjectival modifier DEF GENARG definite descriptions with a genitive argument DEF PPADJ definite descriptions with a PP adjunct DEF RELARG definite descriptions including a relative clause DEF APP definite descriptions including a title or job description as well as a proper name (e.g. an apposition) Names PROPER combinations of position/title and proper name (without article) BARE PROPER bare proper names Demonstrative descriptions SIMPLE DEMON simple demonstrative descriptions MOD DEMON adjectivally modified demonstrative descriptions Pronouns PERS PRON personal pronouns EXPL PRON expletive pronoun REFL PRON reflexive pronoun DEMON PRON demonstrative pronouns (not: determiners) GENERIC PRON generic pronoun (man  one) DA PRON da-pronouns (darauf, daruber, dazu, ) LOC ADV location-referring pronouns TEMP ADV,YEAR Dates and times Indefinites SIMPLE INDEF simple indefinites NEG INDEF negative indefinites INDEF ATTR indefinites with adjectival modifiers INDEF CONTRAST indefinites with contrastive modifiers (einige  some, andere  other, weitere  further, ) INDEF PPADJ indefinites with PP adjuncts INDEF REL indefinites with relative clause adjunct INDEF GEN indefinites with genitive adjuncts INDEF NUM measure/number phrases INDEF QUANT quantified indefinites Table 5: An inventory of interesting syntactic characteristics in IS phrases Label 1 (+ features) Label 2 (+ features) B/A Total D-GIVEN-PRONOUN INDEF-REL 0 19 PERS PRON 39 INDEF ATTR 23 DA PRON 25 SIMPLE INDEF 17 DEMON PRON 19 GENERIC PRON 11 D-GIVEN-PRONOUN D-GIVEN-CATAPHOR 0.1 11 PERS PRON 39 SIMPLE DEF 13 DA PRON 25 DA PRON 10 DEMON PRON 19 GENERIC PRON 11 D-GIVEN-REFLEXIVE NEW 0.11 31 REFL PRON 54 SIMPLE INDEF 113 INDEF ATTR 53 INDEF NUM 32 INDEF PPADJ 26 INDEF GEN 25  Table 6: IS asymmetric pairs augmented with syntactic characteristics 822 2002).
In order to create the necessary SMT language and translation models, they used:  Giza++ (Och & Ney, 2003);2  the CMU-Cambridge statistical toolkit;3  the ISI ReWrite Decoder.4 Translation was performed from EnglishFrench and FrenchEnglish, and the resulting translations were evaluated using a range of automatic metrics: BLEU (Papineni et al. , 2002), Precision and Recall 2http://www.isi.edu/och/Giza++.html 3http://mi.eng.cam.ac.uk/prc14/toolkit.html 4http://www.isi.edu/licensed-sw/rewrite-decoder/ 185 (Turian et al. , 2003), and Wordand Sentence Error Rates.
Och showed thatsystemperformanceisbestwhenparametersare optimizedusingthesameobjectivefunctionthatwill be used for evaluation; BLEU (Papineni et al. , 2002) remains common for both purposes and is often retained for parameter optimization even when alternative evaluation measures are used, e.g., (Banerjee and Lavie, 2005; Snover et al. , 2006).
5http://opennlp.sourceforge.net/ We use the standard four-reference NIST MTEval data sets for the years 2003, 2004 and 2005 (henceforth MT03, MT04 and MT05, respectively) for testing and the 2002 data set for tuning.6 BLEU4 (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005) and multiple-reference Word Error Rate scores are reported.
De-En En-De Baseline 26.95 20.16 Factored baseline 27.43 20.27 Submitted system 27.63 20.46 Table 1: Bleu scores for Europarl (test2007) De-En En-De Baseline 19.54 14.31 Factored baseline 20.16 14.37 Submitted system 20.61 14.77 Table 2: Bleu scores for News Commentary (nc-test2007) 5 Results Case-sensitive Bleu scores4 (Papineni et al., 2002) for the Europarl devtest set (test2007) are shown in table 1.
2 Syntactic-oriented evaluation metrics We investigated the following metrics oriented on the syntactic structure of a translation output:  POSBLEU The standard BLEU score (Papineni et al., 2002) calculated on POS tags instead of words;  POSP POS n-gram precision: percentage of POS ngrams in the hypothesis which have a counterpart in the reference;  POSR Recall measure based on POS n-grams: percentage of POS n-grams in the reference which are also present in the hypothesis;  POSF POS n-gram based F-measure: takes into account all POS n-grams which have a counter29 part, both in the reference and in the hypothesis.
After a brief period following the introduction of generally accepted and widely used metrics, BLEU (Papineni et al., 2002) and NIST (Doddington, 2002), when it seemed that this persistent problem has finally been solved, the researchers active in the field of machine translation (MT) started to express their worries that although these metrics are simple, fast and able to provide consistent results for a particular system during its development, they are not sufficiently reliable for the comparison of different systems or different language pairs.
2.2 Automatic evaluation metric Since the official evaluation criterion for WMT09 is human sentence ranking, we chose to minimize a linear combination of two common evaluation metrics, BLEU and TER (Papineni et al., 2002; Snover et al., 2006), during system development and tuning: TERBLEU 2 Although we are not aware of any work demonstrating that this combination of metrics correlates better than either individually in sentence ranking, Yaser Al-Onaizan (personal communication) reports that it correlates well with the human evaluation metric HTER.
In showing how DLTAG and an interpretative process on its derivations operate, we must, of necessity, gloss over how inference triggered by adjacency or associated with a structural connective provides the intended relation between adjacent discourse 578 Computational Linguistics Volume 29, Number 4 units: It may be a matter simply of statistical inference, as in Marcu and Echihabi (2002), or of more complex inference, as in Hobbs et al.
7 Automated Sense Labelling of Discourse Connectives The focus here is on automated sense labelling of discourse connectives (Elwell and Baldridge, 2008; Marcu and Echihabi, 2002; Pitler et al., 2009; Wellner and Pustejovsky, 2007; Wellner, 679 Total Density of Intra-Sentential Intra-Sentential Total Intra-Sentential Intra-Sentential Subordinating Coordinating Discourse Genre Sentences Connectives Connectives/Sentence Conjunctions Conjunctions Adverbials ESSAYS 4774 1397 0.293 808 (57.8%) 438 (31.4%) 151 (10.8%) SUMMARIES 2118 275 0.130 166 (60.4%) 99 (36.0%) 10 (3.6%) LETTERS 739 200 0.271 126 (63.0%) 56 (28.0%) 18 (9.0%) NEWS 40095 9336 0.233 5514 (59.1%) 3015 (32.3%) 807 (8.6%) Figure 4: Distribution of Explicit Intra-Sentential Connectives.
There are many research directions, e.g., sentiment classification (classifying an opinion document as positive or negative) (e.g., Pang, Lee and Vaithyanathan, 2002; Turney, 2002), subjectivity classification (determining whether a sentence is subjective or objective, and its associated opinion) (Wiebe and Wilson, 2002; Yu and Hatzivassiloglou, 2003; Wilson et al, 2004; Kim and Hovy, 2004; Riloff and Wiebe, 2005), feature/topic-based sentiment analysis (assigning positive or negative sentiments to topics or product features) (Hu and Liu 2004; Popescu and Etzioni, 2005; Carenini et al., 2005; Ku et al., 2006; Kobayashi, Inui and Matsumoto, 2007; Titov and McDonald.
Measures of attributional similarity have been studied extensively, due to their applications in problems such as recognizing synonyms (Landauer and Dumais 1997), information retrieval (Deerwester et al. 1990), determining semantic orientation (Turney 2002), grading student essays (Rehder et al. 1998), measuring textual cohesion (Morris and Hirst 1991), and word sense disambiguation (Lesk 1986).
Such techniques are currently being applied in many areas, including language identification, authorship attribution (Stamatatos et al. , 2000), text genre classification (Kesseler et al. , 1997; Stamatatos et al. , 2000), topic identification (Dumais et al. , 1998; Lewis, 1992; McCallum, 1998; Yang, 1999), and subjective sentiment classification (Turney, 2002).
Sentiment analysis includes a variety of different problems, including: sentiment classification techniques to classify reviews as positive or negative, based on bag of words (Pang et al. , 2002) or positive and negative words (Turney, 2002; Mullen and Collier, 2004); classifying sentences in a document as either subjective or objective (Riloff and Wiebe, 2003; Pang and Lee, 2004); identifying or classifying appraisal targets (Nigam and Hurst, 2004); identifying the source of an opinion in a text (Choi et al. , 2005), whether the author is expressing the opinion, or whether he is attributing the opinion to someone else; and developing interactive and visual opinion mining methods (Gamon et al. , 2005; Popescu and Etzioni, 2005).
2 Related Work There has been a large and diverse body of research in opinion mining, with most research at the text (Pang et al., 2002; Pang and Lee, 2004; Popescu and Etzioni, 2005; Ounis et al., 2006), sentence (Kim and Hovy, 2005; Kudo and Matsumoto, 2004; Riloff et al., 2003; Yu and Hatzivassiloglou, 2003) or word (Hatzivassiloglou and McKeown, 1997; Turney and Littman, 2003; Kim and Hovy, 2004; Takamura et al., 2005; Andreevskaia and Bergler, 2006; Kaji and Kitsuregawa, 2007) level.
The research of opinion mining began in 1997, the early research results mainly focused on the polarity of opinion words (Hatzivassiloglou et al., 1997) and treated the text-level opinion mining as a classification of either positive or negative on the number of positive or negative opinion words in one text (Turney et al., 2003; Pang et al., 2002; Zagibalov et al., 2008;).
2 Motivation In the past, work has been done in the area of characterizing words and phrases according to their emotive tone (Turney and Littman, 2003; Turney, 2002; Kamps et al. , 2002; Hatzivassiloglou and Wiebe, 2000; Hatzivassiloglou and McKeown, 2002; Wiebe, 2000), but in many domains of text, the values of individual phrases may bear little relation to the overall sentiment expressed by the text.
The third exploits automatic subjectivity analysis in applications such as review classification (e.g. , (Turney, 2002; Pang and Lee, 2004)), mining texts for product reviews (e.g. , (Yi et al. , 2003; Hu and Liu, 2004; Popescu and Etzioni, 2005)), summarization (e.g. , (Kim and Hovy, 2004)), information extraction (e.g. , (Riloff et al. , 2005)), 1Note that sentiment, the focus of much recent work in the area, is a type of subjectivity, specifically involving positive or negative opinion, emotion, or evaluation.
To evaluate the polarity and strength of opinions, most of the existing approaches rely either on training from human-annotated data (Hatzivassiloglou and McKeown, 1997), or use linguistic resources (Hu and Liu, 2004; Kim and Hovy, 2004) like WordNet, or rely on co-occurrence statistics (Turney, 2002) between words that are unambiguously positive (e.g. , excellent) and unambiguously negative (e.g. , horrible).
2 Motivation Automatic subjectivity analysis methods have been used in a wide variety of text processing applications, such as tracking sentiment timelines in online forums and news (Lloyd et al. , 2005; Balog et al. , 2006), review classification (Turney, 2002; Pang et al. , 2002), mining opinions from product reviews (Hu and Liu, 2004), automatic expressive text-to-speech synthesis (Alm et al. , 2005), text semantic analysis (Wiebe and Mihalcea, 2006; Esuli and Sebastiani, 2006), and question answering (Yu and Hatzivassiloglou, 2003).
291 3.1 Level of Analysis Research on sentiment annotation is usually conducted at the text (Aue and Gamon, 2005; Pang et al., 2002; Pang and Lee, 2004; Riloff et al., 2006; Turney, 2002; Turney and Littman, 2003) or at the sentence levels (Gamon and Aue, 2005; Hu and Liu, 2004; Kim and Hovy, 2005; Riloff et al., 2006).
Some existing resources contain lists of subjective words (e.g. , Levins desire verbs (1993)), and some empirical methods in NLP have automatically identified adjectives, verbs, and N-grams that are statistically associated with subjective language (e.g. , (Turney, 2002; Hatzivassiloglou and McKeown, 1997; Wiebe, 2000; Wiebe et al. , 2001)).
accuracy Training data Turney (2002) 66% unsupervised Pang & Lee (2004) 87.15% supervised Aue & Gamon (2005) 91.4% supervised SO 73.95% unsupervised SM+SO to increase seed words, then SO 74.85% weakly supervised Table 7: Classification accuracy on the movie review domain Turney (2002) achieves 66% accuracy on the movie review domain using the PMI-IR algorithm to gather association scores from the web.
Identifying subjectivity helps separate opinions from fact, which may be useful in question answering, summarization, etc. Sentiment detection is the task of determining positive or negative sentiment of words (Hatzivassiloglou and McKeown, 1997; Turney, 2002; Esuli and Sebastiani, 2005), phrases and sentences (Kim and Hovy, 2004; Wilson et al. , 2005), or documents (Pang et al. , 2002; Turney, 2002).
Lexical cues of differing complexities have been used, including single words and Ngrams (e.g. , (Mullen and Collier, 2004; Pang et al. , 2002; Turney, 2002; Yu and Hatzivassiloglou, 2003; Wiebe et al. , 2004)), as well as phrases and lexico-syntactic patterns (e.g, (Kim and Hovy, 2004; Hu and Liu, 2004; Popescu and Etzioni, 2005; Riloff and Wiebe, 2003; Whitelaw et al. , 2005)).
Between these two extremes, there has been a relatively modest amount of work in sentence simplification (Chandrasekar, Doran, and Bangalore 1996; Mahesh 1997; Carroll et al. 1998; Grefenstette 1998; Jing 2000; Knight and Marcu 2002) and document compression (Daume III and Marcu 2002; Daume III and Marcu 2004; Zajic, Dorr, and Schwartz 2004) in which words, phrases, and sentences are selected in an extraction process.
1 Introduction Hyponymy relations can play a crucial role in various NLP systems, and there have been many attempts to develop automatic methods to acquire hyponymy relations from text corpora (Hearst, 1992; Caraballo, 1999; Imasumi, 2001; Fleischman et al. , 2003; Morin and Jacquemin, 2003; Ando et al. , 2003).
1 Motivation Question Answering has emerged as a key area in natural language processing (NLP) to apply question parsing, information extraction, summarization, and language generation techniques (Clark et al. , 2004; Fleischman et al. , 2003; Echihabi et al. , 2003; Yang et al. , 2003; Hermjakob et al. , 2002; Dumais et al. , 2002).
In contrast, the idea of bootstrapping for relation and information extraction was first proposed in (Riloff and Jones, 1999), and successfully applied to the construction of semantic lexicons (Thelen and Riloff, 2002), named entity recognition (Collins and Singer, 1999), extraction of binary relations (Agichtein and Gravano, 2000), and acquisition of structured data for tasks such as Question Answering (Lita and Carbonell, 2004; Fleischman et al. , 2003).
Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 585592 Manchester, August 2008 Random Restarts in Minimum Error Rate Training for Statistical Machine Translation Robert C. Moore and Chris Quirk Microsoft Research Redmond, WA 98052, USA bobmoore@microsoft.com, chrisq@microsoft.com Abstract Ochs (2003) minimum error rate training (MERT) procedure is the most commonly used method for training feature weights in statistical machine translation (SMT) models.
3 Baseline MT System The phrase-based SMT system used in our experiments is Moses, phrase translation pro ing probabilities, and languag ties are combined in the log-linear model to obtain the best translation best e  of the source sentence f :  =  = M p | )(maxarg fee ebest  (2) m mm h 1 ,(maxarg f)e e  The weights are set by a discriminative training method using a held-out data set as describ in (Och, 2003).
2 Summary of approaches Given a source language sentence f, statistical machine translation defines the translation task as selecting the most likely target translation e under a model P(e|f), i.e.: e(f) = argmax e P(e|f) = argmax e msummationdisplay i=1 hi(e,f)i where the argmax operation denotes a search through a structured space of translation ouputs in the target language, hi(e,f) are bilingual features of e and f and monolingual features of e, and weights i are trained discriminitively to maximize translation quality (based on automatic metrics) on held out data (Och, 2003).
To model p(t,a|s), we use a standard loglinear approach: p(t,a|s)  exp bracketleftBiggsummationdisplay i ifi(s,t,a) bracketrightBigg where each fi(s,t,a) is a feature function, and weights i are set using Ochs algorithm (Och, 2003) to maximize the systems BLEU score (Papineni et aal.
on test BLEU BP BLEU BP pair-CI 95% BLEU BP 3 01  03 32.98 0.92 33.03 0.93 [ -0.23, +0.34] 33.60 0.93 4 01  04 33.44 0.93 33.46 0.93 [ -0.26, +0.29] 34.97 0.94 5 01  05 33.07 0.92 33.14 0.93 [ -0.29, +0.43] 34.33 0.93 6 01  06 32.86 0.92 33.53 0.93 [+0.26, +1.08] 34.43 0.93 7 01  07 33.08 0.93 33.51 0.93 [+0.04, +0.82] 34.49 0.93 8 01  08 33.12 0.93 33.47 0.93 [ -0.06, +0.75] 34.50 0.94 9 01  09 33.15 0.93 33.22 0.93 [ -0.35, +0.51] 34.68 0.93 10 01  10 33.01 0.93 33.59 0.94 [+0.18, +0.96] 34.79 0.94 11 01  11 32.84 0.94 33.40 0.94 [+0.13, +0.98] 34.76 0.94 12 01  12 32.73 0.93 33.49 0.94 [+0.34, +1.18] 34.83 0.94 13 01  13 32.71 0.93 33.54 0.94 [+0.39, +1.26] 34.91 0.94 14 01  14 32.66 0.93 33.69 0.94 [+0.58, +1.47] 34.97 0.94 15 01  15 32.47 0.93 33.57 0.94 [+0.63, +1.57] 34.99 0.94 16 01  16 32.51 0.93 33.62 0.94 [+0.62, +1.59] 35.00 0.94 3.2 Non-Uniform System Prior Weights As pointed out in Section 2.1, a useful property of the MBR-like system selection method is that system prior weights can easily be trained using the Minimum Error Rate Training (Och, 2003).
5.1 Baseline System We trained Moses on all Spanish-English Europarl sentences up to length 20 (177k sentences) using GIZA++ Model 4 word alignments and the growdiag-final-and combination heuristic (Koehn et al., 2007; Och and Ney, 2003; Koehn, 2002), which performed better than any alternative combination heuristic.13 The baseline estimates (Heuristic) come fromextractingphrasesuptolength7fromtheword alignment.
Assuming that the corpusbased error count for some translations eS1 is additively decomposable into the error counts of the individual sentences, i.e., ED4rS1 ,eS1D5 AG EWSs AG1 ED4rs,esD5,the MERT criterion is given as: M1 AG argmin M1 AZ S F4 sAG1 EA0rs,eD4fs;M1 D5A8 B7 (3) AG argmin M1 AZ S F4 sAG1 K F4 kAG1 ED4rs,es,kD5A0eD4fs;M1 D5,es,kA8 B7 with e D4fs;M1 D5 AG argmaxe AZ M F4 mAG1 mhmD4e,fsD5 B7 (4) In (Och, 2003), it was shown that linear models can effectively be trained under the MERT criterion using a special line optimization algorithm.
Starting from an initial point M1 , computing the most probable sentence hypothesis out of a set of K candidate translations Cs AG D8e1,,eKD9 along the line M1 A0  A4 dM1 results in the following optimization problem (Och, 2003): e D4fs;D5 AG argmax eC8Cs AX D4 M 1 A0  A4 d M 1 D5 C2 A4 hM1 D4e,fsD5 B5 AG argmax eC8Cs AY F4 m mhmD4e,fsD5 D0D3D3D3D3D3D3D3D3D1D3D3D3D3D3D3D3D3D2 AGaD4e,fsD5 A0 A4 F4 m dmhmD4e,fsD5 D0D3D3D3D3D3D3D3D3D1D3D3D3D3D3D3D3D3D2 AGbD4e,fsD5 B6 AG argmax eC8Cs AWa D4e,fsD5 A0  A4 bD4e,fsD5 D0D3D3D3D3D3D3D3D3D3D3D3D1D3D3D3D3D3D3D3D3D3D3D3D2 D4A6D5 B4 (5) Hence, the total score D4A6D5 for any candidate translation corresponds to a line in the plane with  as the independent variable.
The upper envelope is a convex hull and can be inscribed with a convex polygon whose edges are the segments of a piecewise linear function in  (Papineni, 1999; Och, 2003): EnvD4fD5 AG max eC8C AWa D4e,fD5 A0  A4 bD4e,fD5 :  C8 RB4 (6) 726 Score  Error count  0 0 e1 e2 e5 e6 e8 e1e 2 e3 e4 e5e6e 7 e8 Figure 1: The upper envelope (bold, red curve) for a set of lines is the convex hull which consists of the topmost line segments.
Their transliteration probability is: P(t|s)  PE(s|t)max[PT(t),PL(t)] (1) Inspired by the linear models used in SMT (Och, 2003), we can discriminatively weight the components of this generative model, producing: wE logPE(s|t)+wT logPT(t)+wL logPL(t) with weights w learned by perceptron training.
The alignment a J 1 that has the highest probability (under a certain model) is also called the Viterbi alignment (of that model): a J 1 = argmax a J 1 p   (f J 1, a J 1 | e I 1 ) (8) A detailed comparison of the quality of these Viterbi alignments for various statistical alignment models compared to human-made word alignments can be found in Och and Ney (2003).
The rules extracted from the training bitext have the following features: a114 P( | )andP( | ), the latter of which is not found in the noisy-channel model, but has been previously found to be a helpful feature (Och and Ney 2002); 210 Chiang Hierarchical Phrase-Based Translation a114 the lexical weights P w ( | )andP w ( | ), which estimate how well the words in  translate the words in  (Koehn, Och, and Marcu 2003); 4 a114 a penalty exp(1) for extracted rules, analogous to Koehns phrase penalty (Koehn 2003), which allows the model to learn a preference for longer or shorter derivations.
294 Fraser and Marcu Measuring Word Alignment Quality for Statistical Machine Translation 2.2 Measuring Translation Performance Changes Caused By Alignment In phrased-based SMT (Koehn, Och, and Marcu 2003) the knowledge sources which vary with the word alignment are the phrase translation lexicon (which maps source phrases to target phrases using counts from the word alignment) and some of the word level translation parameters (sometimes called lexical smoothing).
This is the shared task baseline system for the 2006 NAACL/HLT workshop on statistical machine translation (Koehn and Monz, 2006) and consists of the Pharaoh decoder (Koehn, 2004), SRILM (Stolcke, 2002), GIZA++ (Och and Ney, 2003), mkcls (Och, 1999), Carmel,1 and a phrase model training code.
3 Experiments We built baseline systems using GIZA++ (Och and Ney, 2003), Moses phrase extraction with grow-diag-finalend heuristic (Koehn et al., 2007), a standard phrasebased decoder (Vogel, 2003), the SRI LM toolkit (Stolcke, 2002), the suffix-array language model (Zhang and Vogel, 2005), a distance-based word reordering model Algorithm 5 Rich Interruption Constraints (Coh5) Input: Source tree T, previous phrase fh, current phrase fh+1, coverage vector HC 1: Interruption  False 2: ICount,VerbCount,NounCount  0 3: F  the left and right-most tokens of fh 4: for each of f  F do 5: Climb the dependency tree from f until you reach the highest node n such that fh+1 / T(n).
In order to incorporate a new dependency which contains extra information other than the bilingual sentence pair, we modify Eq.2 by adding a new variable v: Pr(a|e,f,v) = exp[ summationtextM m=1 mhm(a,e,f,v)]summationtext aprime exp[ summationtextM m=1 mhm(aprime,e,f,v)](4) Accordingly, we get a new decision rule: a = argmax a braceleftbigg Msummationdisplay m=1 mhm(a,e,f,v) bracerightbigg (5) Note that our log-linear models are different from Model 6 proposed by Och and Ney (2003), which defines the alignment problem as finding the alignment a that maximizes Pr(f, a|e) given e. 3 Feature Functions In this paper, we use IBM translation Model 3 as the base feature of our log-linear models.
as follows: p(synI1|trgI1) = ( Iproductdisplay i=1 p(syni|trgi) (4)  pprime(trgi|syni)prime  pw(syni|trgi)w  pwprime(trgi|syni)wprime  pd(syni,trgi)d)  lw(synI1)l  c(synI1)c  pLM(synI1)LM For estimation of the feature weights vector defined in equation (4) we employed minimum error rate (MER) training under the BLEU measure (Och, 2003).
Metrics in the Rouge family allow for skip n-grams (Lin and Och, 2004a); Kauchak and Barzilay (2006) take paraphrasing into account; metrics such as METEOR (Banerjee and Lavie, 2005) and GTM (Melamed et al. , 2003) calculate both recall and precision; METEOR is also similar to SIA (Liu and Gildea, 2006) in that word class information is used.