gaof23/citation_sentiment_corpus huggingface dataset:
Our baseline model follows Chiangs hierarchical model (Chiang, 2007) in conjunction with additional features:  conditional probabilities in both directions: P(|) and P(|);  lexical weights (Koehn et al., 2003) in both directions: Pw(|) and Pw(|); 21  word counts |e|;  rule counts |D|;  target n-gram language model PLM(e);  glue rule penalty to learn preference of nonterminal rewriting over serial combination through Eq.
4 Experiments and evaluation We carried out an evaluation on the local rephrasing of French sentences, using English as the pivot language.2 We extracted phrase alignments of up to 7 word forms using the Giza++ alignment tool (Och and Ney, 2003) and the grow-diag-final-and heuristics described in (Koehn et al., 2003) on 948,507 sentences of the French-English part of the Europarl corpus (Koehn, 2005) and obtained some 42 million phrase pairs for which probabilities were estimated using maximum likelihood estimation.
4 Architecture of the SMT system The goal of statistical machine translation (SMT) is to produce a target sentence e from a source sentence f. It is today common practice to use phrases as translation units (Koehn et al., 2003; Och and Ney, 2003) and a log linear framework in order to introduce several models explaining the translation process: e = argmaxp(e|f) = argmaxe {exp(summationdisplay i ihi(e,f))} (1) The feature functions hi are the system models and the i weights are typically optimized to maximize a scoring function on a development set (Och and Ney, 2002).
As a result, they are being used in a variety of applications, such as question answering (Hermjakob, 2001), speech recognition (Chelba and Jelinek, 1998), language modeling (Roark, 2001), language generation (Soricut, 2006) and, most notably, machine translation (Charniak et al., 2003; Galley et al., 2004; Collins et al., 2005; Marcu et al., 2006; Huang et al., 2006; Avramidis and Koehn, 2008).
Since no such corpus exists, researchers have used coarser features learned from smaller sets through supervised learning (Soon et al. , 2001; Ng and Cardie, 2002), manually-de ned coreference patterns to mine speci c kinds of data (Bean and Riloff, 2004; Bergsma, 2005), or accepted the noise inherent in unsupervised schemes (Ge et al. , 1998; Cherry and Bergsma, 2005).
We consider the outputs of the top 3 allwords WSD systems that participated in Senseval-3: Gambl (Decadt et al. , 2004), SenseLearner (Mihalcea and Faruque, 2004), and KOC University (Yuret, Nouns Verbs Adjectives F-SCORE 0.4228 0.4319 0.4727 Feature F-Score Ablation Difference TOPSIG 0.0403   OED 0.0355 0.0126 -0.0124 DERIV 0.0351 0.0977 0.0352 RES 0.0287 0.0147  TWIN 0.0285 0.0109 -0.0130 MN 0.0188 0.0358  LESK 0.0183 0.0541 -0.0250 SENSENUM 0.0155 0.0146 -0.0147 SENSECNT 0.0121 0.0160 0.0168 DOMAIN 0.0119 0.0082 -0.0265 LCH 0.0099 0.0068  WUP 0.0036 0.0168  JCN 0.0025 0.0190  ANTONYM 0.0000 0.0295 0.0000 MAXMN -0.0013 0.0179  VEC -0.0024 0.0371 -0.0062 HSO -0.0073 0.0112 -0.0246 LIN -0.0086 0.0742  COUSIN -0.0094   VERBGRP  0.0327  VERBFRM  0.0102  PERTAINYM   -0.0029 Table 4: Feature ablation study; F-score difference obtained by removal of the single feature 2004).
A variety of synset similarity measures based on properties of WordNet itself have been proposed; nine such measures are discussed in (Pedersen et al. , 2004), including gloss-based heuristics (Lesk, 1986; Banerjee and Pedersen, 2003), information-content based measures (Resnik, 1995; Lin, 1998; Jiang and Conrath, 1997), and others.
We use eight similarity measures implemented within the WordNet::Similarity package5, described in (Pedersen et al. , 2004); these include three measures derived from the paths between the synsets in WordNet: HSO (Hirst and St-Onge, 1998), LCH (Leacock and Chodorow, 1998), and WUP (Wu and Palmer, 1994); three measures based on information content: RES (Resnik, 1995), LIN (Lin, 1998), and JCN (Jiang and Conrath, 1997); the gloss-based Extended Lesk Measure LESK, (Banerjee and Pedersen, 2003), and finally the gloss vector similarity measure VECTOR (Patwardan, 2003).
1 Introduction Estimating the degree of semantic relatedness between words in a text is deemed important in numerous applications: word-sense disambiguation (Banerjee and Pedersen, 2003), story segmentation (Stokes et al. , 2004), error correction (Hirst and Budanitsky, 2005), summarization (Barzilay and Elhadad, 1997; Gurevych and Strube, 2004).
Extensive research concerning the integration of semantic knowledge into NLP for the English language has been arguably fostered by the emergence of WordNet::Similarity package (Pedersen et al. , 2004).1 In its turn, the development of the WordNet based semantic similarity software has been facilitated by the availability of tools to easily retrieve 1http://www.d.umn.edu/a0 tpederse/similarity.html data from WordNet, e.g. WordNet::QueryData,2 jwnl.3 Research integrating semantic knowledge into NLP for languages other than English is scarce.
The implementation includes path-length (Rada et al. , 1989; Wu & Palmer, 1994; Leacock & Chodorow, 1998), information-content (Resnik, 1995; Seco et al. , 2004) and text-overlap (Lesk, 1986; Banerjee & Pedersen, 2003) measures, as described in Strube & Ponzetto (2006).
We also used the following resources: the Charniak parser (Charniak, 2000) to carry out the syntactic analysis; the wn::similaritypackage (Pedersen et al. , 2004) to compute the Jiang&Conrath (J&C) distance (Jiang and Conrath, 1997) needed to implement the lexical similarity siml(T,H) as defined in (Corley and Mihalcea, 2005); SVM-lightTK (Moschitti, 2004) to encode the basic tree kernel function, KT, in SVM-light (Joachims, 1999).
David McClosky, Eugene Charniak, and Mark Johnson Brown Laboratory for Linguistic Information Processing (BLLIP) Brown University Providence, RI 02912 {dmcc|ec|mj}@cs.brown.edu Abstract Self-training has been shown capable of improving on state-of-the-art parser performance (McClosky et al., 2006) despite the conventional wisdom on the matter and several studies to the contrary (Charniak, 1997; Steedman et al., 2003).
Research in the field of unsupervised and weakly supervised parsing ranges from various forms of EM training (Pereira and Schabes, 1992; Klein and Manning, 2004; Smith and Eisner, 2004; Smith and Eisner, 2005) over bootstrapping approaches like selftraining (McClosky et al., 2006) to feature-based enhancements of discriminative reranking models (Koo et al., 2008) and the application of semisupervised SVMs (Wang et al., 2008).
Also, on WS-353, our hybrid sense-filtered variants and word-cos-ll obtained a correlation score higher than published results using WordNet-based measures (Jarmasz and Szpakowicz, 2003) (.33 to .35) and Wikipediabased methods (Ponzetto and Strube, 2006) (.19 to .48); and very close to the results obtained by thesaurus-based (Jarmasz and Szpakowicz, 2003) (.55) and LSA-based methods (Finkelstein et al., 2002) (.56).
Translation rules can:  look like phrase pairs with syntax decoration: NPB(NNP(prime) NNP(minister) NNP(keizo) NNP(obuchi)) BUFDFKEUBWAZ  carry extra contextual constraints: VP(VBD(said) x0:SBAR-C) DKx0 (according to this rule, DK can translate to said only if some Chinese sequence to the right ofDK is translated into an SBAR-C)  be non-constituent phrases: VP(VBD(said) SBAR-C(IN(that) x0:S-C)) DKx0 VP(VBD(pointed) PRT(RP(out)) x0:SBAR-C) DXGPx0  contain non-contiguous phrases, effectively phrases with holes: PP(IN(on) NP-C(NPB(DT(the) x0:NNP)) NN(issue)))) GRx0 EVABG6 PP(IN(on) NP-C(NPB(DT(the) NN(issue)) x0:PP)) GRx0 EVEVABABG6  be purely structural (no words): S(x0:NP-C x1:VP)x0 x1  re-order their children: NP-C(NPB(DT(the) x0:NN) PP(IN(of) x1:NP-C)) x1 DFx0 Decoding with this model produces a tree in the target language, bottom-up, by parsing the foreign string using a CYK parser and a binarized rule set (Zhang et al. , 2006).
While this heuristic estimator gives good empirical results, it does not seem to optimize any intuitively reasonable objective function of the (wordaligned) parallel corpus (see e.g., (DeNero et al., 2006)) The mounting number of efforts attacking this problem over the last few years (DeNero et al., 2006; Marcu and Wong, 2002; Birch et al., 2006; Moore and Quirk, 2007; Zhang et al., 2008) exhibits its difficulty.
Given the following SCFG rule:      VP      VB  NP     JJR  ,                VB  NP  will be  JJR we can obtain a set of equivalent binary rules using the synchronous binarization method (Zhang et al., 2006)  as follows:         VP  V1  JJR ,   V1  JJR             V1  VB  V2 ,   VB  V2         V2  NP   ,   NP  will be This binarization is shown with the solid lines as binarization (a) in Figure 1.
V B N P  J J R ( a ) ( b ) V 2 V 1 V 2 ' V 1 ' V P V B N P w ill b e J J R  Figure 1: Two different binarizations (a) and (b) of the same SCFG rule distinguished by the solid lines and dashed lines                         ( W e  h o p e  t h e  s i t u a t i o n  w i l l  b e  b e t t e r  . )           N P        J J R     d e c o d i n g m a t c h  8 7 4  r u l e s m a t c h  6 2  r u l e s c o m p e t i n g  e d g e s :  8 0 1 c o m p e t i n g  e d g e s :  5 7 Figure 2: Edge competitions caused by different binarizations  The edge competition problem for SMT decoding is not addressed in previous work (Zhang et al., 2006; Huang, 2007) in which each SCFG rule is binarized in a fixed way.
Iterative cost reduction algorithm Input: An SCFG  Output: An equivalent binary SCFG    of  1: Function ITERATIVECOSTREDUCTION( ) 2:         0 3:   for each    0do 4:         ( ) =     ,  0 5:   while  (  ) does not converge do 6:        for each      do 7:             [   ]        (  ) 8:            for each     (  ) do 9:                for each        ,     do 10:                              1 11:           (  )   CKYBINARIZATION(  ,  ) 12:                [   ]    (  ) 13:          for each     (  ) do 14:              for each        ,     do 15:                             + 1 16: return   In the iterative cost reduction algorithm, we first obtain an initial binary SCFG  0 using the synchronous binarization method proposed in (Zhang et al., 2006).
Compared with their string-based counterparts, treebased systems offer some attractive features: they are much faster in decoding (linear time vs. cubic time, see (Huang et al., 2006)), do not require a binary-branching grammar as in string-based models (Zhang et al., 2006), and can have separate grammars for parsing and translation, say, a context-free grammar for the former and a tree substitution grammar for the latter (Huang et al., 2006).
Most previous work exploiting unsupervised training data for inferring POS tagging models has focused on semi-supervised methods in the in which the learner is provided with a lexicon specifying the possible tags for each word (Merialdo, 1994; Smith and Eisner, 2005; Goldwater and Griffiths, 2007) or a small number of prototypes for each POS (Haghighi and Klein, 2006).
For this study, the Levenshtein edit-distance score (where a perfect match scores zero) is  Roman Chinese (Pinyin) Alignment Score LEV ashburton ashenbodu |   a   s   h   b   u   r   t   o   n   | |   a   s   h   e   n   b  o  d    u   | 0.67 MLEV ashburton ashenbodu |  a   s   h       b   u   r    t   o   n  | |  a   s   h   e   n   b   o     d   u    | 0.72 MALINE asVburton aseCnpotu |   a   sV    b   <   u   r   t   o   |   n |   a   s   eC  n   p   o     t   u   |   0.48 3 normalized to a similarity score as in (Freeman et al. 2006), where the score ranges from 0 to 1, with 1 being a perfect match.
While transfer learning was proposed more than a decade ago (Thrun, 1996; Caruana, 1997), its application in natural language processing is still a relatively new territory (Blitzer et al., 2006; Daume III, 2007; Jiang and Zhai, 2007a; Arnold et al., 2008; Dredze and Crammer, 2008), and its application in relation extraction is still unexplored.
Letter successor variety (LSV) models (Hafer and Weiss, 1974; Gaussier, 1999; Bernhard, 2005; Bordag, 2005; 669 Keshava and Pitler, 2005; Hammarstrom, 2006; Dasgupta and Ng, 2007; Demberg, 2007) use the hypothesis that there is less certainty when predicting the next character at morpheme boundaries.
However, it seems unrealistic to expect a one-size-fits-all approach to be achieve uniformly high performance across varied languages, and, in fact, it doesnt. Though the system presented in (Dasgupta and Ng, 2007) outperforms the best systems in the 2006 PASCAL challenge for Turkish and Finnish, it still does significantly worse on these languages than English (F-scores of 66.2 and 66.5, compared to 79.4).
Thus, some research has been focused on deriving different word-sense groupings to overcome the finegrained distinctions of WN (Hearst and Schutze, 1993), (Peters et al., 1998), (Mihalcea and Moldovan, 2001), (Agirre and LopezDeLaCalle, 2003), (Navigli, 2006) and (Snow et al., 2007).
Generally, WSD methods use the context of a word for its sense disambiguation, and the context information can come from either annotated/unannotated text or other knowledge resources, such as WordNet (Fellbaum, 1998), SemCor (SemCor, 2008), Open Mind Word Expert (Chklovski and Mihalcea, 2002), eXtended WordNet (Moldovan and Rus, 2001), Wikipedia (Mihalcea, 2007), parallel corpora (Ng, Wang, and Chan, 2003).
They may rely only on this information (e.g., (Turney, 2002; Whitelaw et al., 2005; Riloff and Wiebe, 2003)), or they may combine it with additional information as well (e.g., (Yu and Hatzivassiloglou, 2003; Kim and Hovy, 2004; Bloom et al., 2007; Wilson et al., 2005a)).
With the in-depth study of opinion mining, researchers committed their efforts for more accurate results: the research of sentiment summarization (Philip et al., 2004; Hu et al., KDD 2004), domain transfer problem of the sentiment analysis (Kanayama et al., 2006; Tan et al., 2007; Blitzer et al., 2007; Tan et al., 2008; Andreevskaia et al., 2008; Tan et al., 2009) and finegrained opinion mining (Hatzivassiloglou et al., 2000; Takamura et al., 2007; Bloom et al., 2007; Wang et al., 2008; Titov et al., 2008) are the main branches of the research of opinion mining.
In addition to the widely used BLEU (Papineni et al. , 2002) and NIST (Doddington, 2002) scores, we also evaluate translation quality with the recently proposed Meteor (Banerjee and Lavie, 2005) and four edit-distance style metrics, Word Error Rate (WER), Positionindependent word Error Rate (PER) (Tillmann et al. , 1997), CDER, which allows block reordering (Leusch et al. , 2006), and Translation Edit Rate (TER) (Snover et al. , 2006).
4.2 Approximated BLEU We used the BLEU score (Papineni et al. , 2002) as the loss function computed by: BLEU(E; E) = exp    1N Nsummationdisplay n=1 log pn(E, E)   BP(E, E) (7) where pn() is the n-gram precision of hypothesized translations E ={et}Tt=1 given reference translations E ={et}Tt=1 and BP()1 is a brevity penalty.
EsEn 63.00.9 59.20.9 6.01.4 EnEs 63.80.9 60.51.0 5.21.6 DeEn 71.60.8 69.00.9 3.61.3 EnDe 75.90.8 73.50.9 3.21.2 FrEn 62.90.9 59.21.0 5.91.6 EnFr 63.40.9 60.00.9 5.41.4 bined in a log-linear fashion by adjusting a weight for each of them by means of the MERT (Och, 2003) procedure, optimising the BLEU (Papineni et al., 2002) score obtained on the development partition.
To this purpose, different authors (Papineni et al., 1998; Och and Ney, 2002) propose the use of the so-called log-linear models, where the decision rule is given by the expression y = argmax y Msummationdisplay m=1 mhm(x,y) (3) where hm(x,y) is a score function representing an important feature for the translation of x into y, M is the number of models (or features) and m are the weights of the log-linear combination.
2.1 The BLEU Metric The metric most often used with MERT is BLEU (Papineni et al., 2002), where the score of a candidate c against a reference translation r is: BLEU = BP(len(c),len(r))exp( 4summationdisplay n=1 1 4 logpn), where pn is the n-gram precision2 and BP is a brevity penalty meant to penalize short outputs, to discourage improving precision at the expense of recall.
Demonstrating the inadequacy of such approaches, Al-Onaizan and Papineni (2006) showed that even given the words in the reference translation, and their alignment to the source words, a decoder of this sort charged with merely rearranging them into the correct target-language order could achieve a BLEU score (Papineni et al., 2002) of at best 69%and that only when restricted to keep most words very close to their source positions.
The horizontal axis represents the weight for the outof-domain translation model, and the vertical axis 15% 16% 17% 18% 19% 20% 21% 22% 23% 24% 25% 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Weight for out-of-domain translation model BLEU  sco re 400 K 800 K 1.2 M 1.6 M 2.5 M Figure 2: Results of data selection and linear interpolation (BLEU) represents the automatic metric of translation quality (BLEU score (Papineni et al., 2002) in Fig.
Word Error Rate (WER), which penalizes the edit distance against reference translations (Su et al. , 1992) BLEU: the geometric mean of n-gram precision for the translation results found in reference translations (Papineni et al. , 2002) Translation Accuracy (ACC): subjective evaluation ranks ranging from A to D (A: perfect, B: fair, C: acceptable and D: nonsense), judged blindly by a native speaker (Sumita et al. , 1999) In contrast to WER, higher BLEU and ACC scores indicate better translations.
Many methods for calculating the similarity have been proposed (Niessen et al. , 2000; Akiba et al. , 2001; Papineni et al. , 2002; NIST, 2002; Leusch et al. , 2003; Turian et al. , 2003; Babych and Hartley, 2004; Lin and Och, 2004; Banerjee and Lavie, 2005; Gimenez et al. , 2005).
In our research, 23 scores, namely BLEU (Papineni et al. , 2002) with maximum n-gram lengths of 1, 2, 3, and 4, NIST (NIST, 2002) with maximum n-gram lengths of 1, 2, 3, 4, and 5, GTM (Turian et al. , 2003) with exponents of 1.0, 2.0, and 3.0, METEOR (exact) (Banerjee and Lavie, 2005), WER (Niessen et al. , 2000), PER (Leusch et al. , 2003), and ROUGE (Lin, 2004) with n-gram lengths of 1, 2, 3, and 4 and 4 variants (LCS, S,SU, W-1.2), were used to calculate each similarity S i . Therefore, the value of m in Eq.
In recent years, many researchers have tried to automatically evaluate the quality of MT and improve the performance of automatic MT evaluations (Niessen et al. , 2000; Akiba et al. , 2001; Papineni et al. , 2002; NIST, 2002; Leusch et al. , 2003; Turian et al. , 2003; Babych and Hartley, 2004; Lin and Och, 2004; Banerjee and Lavie, 2005; Gimenez et al. , 2005) because improving the performance of automatic MT evaluation is expected to enable us to use and improve MT systems efficiently.
#Reference: If our player 2, 3, 7 or 5 has the ball and the ball is close to our goal line  PHARAOH++: If player 3 has the ball is in 2 5 the ball is in the area near our goal line  WASP1++: If players 2, 3, 7 and 5 has the ball and the ball is near our goal line  Figure 4: Sample partial system output in the ROBOCUP domain ROBOCUP GEOQUERY BLEU NIST BLEU NIST PHARAOH 0.3247 5.0263 0.2070 3.1478 WASP1 0.4357 5.4486 0.4582 5.9900 PHARAOH++ 0.4336 5.9185 0.5354 6.3637 WASP1++ 0.6022 6.8976 0.5370 6.4808 Table 1: Results of automatic evaluation; bold type indicates the best performing system (or systems) for a given domain-metric pair (p < 0.05) 5.1 Automatic Evaluation Weperformed4runsof10-foldcrossvalidation,and measured the performance of the learned generators using the BLEU score (Papineni et al. , 2002) and the NIST score (Doddington, 2002).
The NIST BLEU-4 is a variant of BLEU (Papineni et al. , 2002) and is computed as a49a51a50 a2a16a52a53a6 a0a9a8a10a0a12a11a54a13a55a15 a26a57a56a33a58a60a59 a43 a61a63a62 a64 a65a67a66a69a68 a28a71a70a46a72a74a73 a65 a6 a0a9a8a10a0a3a11a54a13a19a75a77a76 a6 a0a9a8a10a0a3a11a54a13 (2) where a73 a65 a6 a0a78a8a10a0a3a11a54a13 is the precision of a79 -grams in the hypothesis a0 given the reference a0 a11 and a76 a6 a0a78a8a10a0a3a11a54a13a81a80 a43 is a brevity penalty.
Regressive FLM (rFLM) h(FLM(e,j)) = w1 FLM(e,j)+b Regressive ALM (rALM) h(ALM(e,j)) = w1 ALM(e,j)+b Notice that h() here is supposed to relate FLM or ALM to some independent evaluation metric such as BLEU (Papineni et al. , 2002), not the log likelihood of a translation.
1 Introduction With the introduction of the BLEU metric for machine translation evaluation (Papineni et al, 2002), the advantages of doing automatic evaluation for various NLP applications have become increasingly appreciated: they allow for faster implement-evaluate cycles (by by-passing the human evaluation bottleneck), less variation in evaluation performance due to errors in human assessor judgment, and, not least, the possibility of hill-climbing on such metrics in order to improve system performance (Och 2003).
For comparison purposes, we also computed the value of R 2 for fluency using the BLEU score formula given in (Papineni et al. , 2002), for the 7 systems using the same one reference, and we obtained a similar value, 78.52%; computing the value of R 2 for fluency using the BLEU scores computed with all 4 references available yielded a lower value for R 2, 64.96%, although BLEU scores obtained with multiple references are usually considered more reliable.
For comparison purposes, we also computed the value of R 2 for adequacy using the BLEU score formula given in (Papineni et al. , 2002), for the 7 systems using the same one reference, and we obtain a similar value, 83.91%; computing the value of R 2 for adequacy using the BLEU scores computed with all 4 references available also yielded a lower value for R 2, 62.21%.
2 Recap of BLEU, ROUGE-W and METEOR The most commonly used automatic evaluation metrics, BLEU (Papineni et al. , 2002) and NIST (Doddington, 2002), are based on the assumption that The closer a machine translation is to a promt1: Life is like one nice chocolate in box ref: Life is just like a box of tasty chocolate ref: Life is just like a box of tasty chocolate mt2: Life is of one nice chocolate in box Figure 1: Alignment Example for ROUGE-W fessional human translation, the better it is (Papineni et al. , 2002).