gaof23/citation_sentiment_corpus huggingface dataset:
In order to be able to compare the edit distance with the other metrics, we have used the following formula(Wen et al., 2002)whichnormalisesthe minimum edit distance by the length of the longest questionand transformsit into a similaritymetric: normalisededitdistance = 1 edit dist(q1,q2)max(| q 1 |,| q2 |) Word Ngram Overlap This metric compares the word n-gramsin both questions: ngramoverlap = 1N Nsummationdisplay n=1 | Gn(q1)  Gn(q2) | min(| Gn(q1) |,| Gn(q2) |) where Gn(q) is the set of n-grams of length n in question q and N usually equals 4 (Barzilay and Lee, 2003;Cordeiroet al., 2007).
There exist many different string similarity measures: word overlap (Tomuro and Lytinen, 2004), longest common subsequence (Islamand Inkpen,2007), Levenshteinedit distance (Dolan et al., 2004), word n-gramoverlap (Barzilay and Lee, 2003) etc. Semantic similarity measures are obtained by first computing the semantic similarity of the words containedin the sentencesbeing compared.
(Och and Ney, 2003) invented heuristic symmetriza57 FRENCH/ENGLISH ARABIC/ENGLISH SYSTEM F-MEASURE ( = 0.4) BLEU F-MEASURE ( = 0.1) BLEU GIZA++ 73.5 30.63 75.8 51.55 (FRASER AND MARCU, 2006B) 74.1 31.40 79.1 52.89 LEAF UNSUPERVISED 74.5 72.3 LEAF SEMI-SUPERVISED 76.3 31.86 84.5 54.34 Table 3: Experimental Results tion of the output of a 1-to-N model and a M-to-1 model resulting in a M-to-N alignment, this was extended in (Koehn et al. , 2003).
3.1 Phrase-Based Models According to the translation model presented in (Koehn et al. , 2003), given a source sentence f, the best target translation can be obtained using the following model best e 288 )( )()(maxarg )(maxarg | | e e e eef fee length LM best pp p = = (1) Where the translation model can be decomposed into )( | efp  =  = I i i iii i i II aefpbadef efp 1 1 1 1 ),|()()|( )|(   w (2) Where )|( i i ef is the phrase translation probability.
There have been considerable amount of efforts to improve the reordering model in SMT systems, ranging from the fundamental distance-based distortion model (Och and Ney, 2004; Koehn et al. , 2003), flat reordering model (Wu, 1996; Zens et al. , 2004; Kumar et al. , 2005), to lexicalized reordering model (Tillmann, 2004; Kumar et al. , 2005; Koehn et al. , 2005), hierarchical phrase-based model (Chiang, 2005), and maximum entropy-based phrase reordering model (Xiong et al. , 2006).
They give a probabilistic formation of paraphrasing which naturally falls out of the fact that they use techniques from phrase-based statistical machine translation: e2 = argmax e2:e2negationslash=e1 p(e2|e1) (1) where p(e2|e1) = summationdisplay f p(f|e1)p(e2|f,e1) (2)  summationdisplay f p(f|e1)p(e2|f) (3) Phrase translation probabilities p(f|e1) and p(e2|f) are commonly calculated using maximum likelihood estimation (Koehn et al., 2003): p(f|e) = count(e,f)summationtext f count(e,f) (4) where the counts are collected by enumerating all bilingual phrase pairs that are consistent with the 197 conseguido .opportunitiesequalcreatetofailedhasprojecteuropeanthe oportunidadesdeigualdadlahanoeuropeoproyectoel Figure 1: The interaction of the phrase extraction heuristic with unaligned English words means that the Spanish phrase la igualdad aligns with equal, create equal, and to create equal.
5.1 Experimental setup The baseline model was Hiero with the following baseline features (Chiang, 2005; Chiang, 2007):  two language models  phrase translation probabilities p(f | e) and p(e| f)  lexical weighting in both directions (Koehn et al., 2003)  word penalty  penalties for:  automatically extracted rules  identity rules (translating a word into itself)  two classes of number/name translation rules  glue rules The probability features are base-100 logprobabilities.
Beam-search has been successful in many NLP tasks (Koehn et al., 2003; 562 Inputs: training examples (xi,yi) Initialization: set vectorw = 0 Algorithm: // R training iterations; N examples for t = 1R, i = 1N: zi = argmaxyGEN(xi) (y) vectorw if zi negationslash= yi: vectorw = vectorw + (yi)(zi) Outputs: vectorw Figure 1: The perceptron learning algorithm Collins and Roark, 2004), and can achieve accuracy that is close to exact inference.
Besides relative frequencies, lexical weights (Koehn et al., 2003) are widely used to estimate how well the words in f translate the words in e. To do this, one needs first to estimate a lexical translation probability distribution w(e|f) by relative frequency from the same word alignments in the training corpus: w(e|f) = count(f,e)summationtext e count(f,e) (3) Note that a special source NULL token is added to each source sentence and aligned to each unaligned target word.
1313 E2C C2E Union Heuristic w/ Big 13.37 12.66 14.55 14.28 w/o Big 13.20 12.62 14.53 14.21 Table 3: BLEU-4 scores (test set) of systems based on GIZA++ word alignments 5 6 7 8  BLEU-4 14.27 14.42 14.43 14.45 14.55 Table 4: BLEU-4 scores (test set) of the union alignment, using TTS templates up to a certain size, in terms of the number of leaves in their LHSs 4.1 Baseline Systems GHKM (Galley et al., 2004) is used to generate the baseline TTS templates based on the word alignments computed using GIZA++ and different combination methods, including union and the diagonal growing heuristic (Koehn et al., 2003).
5.3 Baseline System We conducted experiments using different segmenters with a standard log-linear PB-SMT model: GIZA++ implementation of IBM word alignment model 4 (Och and Ney, 2003), the refinement and phrase-extraction heuristics described in (Koehn et al., 2003), minimum-errorrate training (Och, 2003), a 5-gram language model with Kneser-Ney smoothing trained with SRILM (Stolcke, 2002) on the English side of the training data, and Moses (Koehn et al., 2007; Dyer et al., 2008) to translate both single best segmentation and word lattices.
We computed precision, recall and error rate on the entire set of sentence pairs for each data set.5 To evaluate NeurAlign, we used GIZA++ in both directions (E-to-F and F-to-E, where F is either Chinese (C) or Spanish (S)) as input and a refined alignment approach (Och and Ney, 2000) that uses a heuristic combination method called grow-diagfinal (Koehn et al. , 2003) for comparison.
5 Phrase Pair Induction A common approach to phrase-based translation is to extract an inventory of phrase pairs (PPI) from bitext (Koehn et al. , 2003), For example, in the phraseextract algorithm (Och, 2002), a word alignment am1 is generated over the bitext, and all word subsequences ei2i1 and fj2j1 are found that satisfy : am1 : aj  [i1,i2] iff j  [j1,j2] .
We computed precision, recall and error rate on the entire set for each data set.6 For an initial alignment, we used GIZA++ in both directions (E-to-F and F-to-E, where F is either Chinese (C) or Spanish (S)), and also two different combined alignments: intersection of E-to-F and F-to-E; and RA using a heuristic combination approach called grow-diag-final (Koehn et al. , 2003).
Above the phrase level, some models perform no reordering (Zens and Ney 2004; Kumar, Deng, and Byrne 2006), some have a simple distortion model that reorders phrases independently of their content (Koehn, Och, and Marcu 2003; Och and Ney 2004), and some, for example, the Alignment Template System (Och et al. 2004; Thayer et al. 2004), hereafter ATS, and the IBM phrase-based system (Tillmann 2004; Tillmann and Zhang 2005), have phrase-reordering models that add some lexical sensitivity.
Toremedythis situation, we can borrow the probabilistic model of PHARAOH, and define the parsing model as: Pr(d|e(d)) = productdisplay dd w(r(d)) (4) which is the product of the weights of the rules used in a derivation d. The rule weight, w(X  ,), is in turn defined as: P(|)1P(|)2Pw(|)3Pw(|)4 exp(||)5 where P(|) and P(|) are the relative frequencies of  and , and Pw(|) and Pw(|) are 176 the lexical weights (Koehn et al. , 2003).
When we run a phrase-based system, Pharaoh (Koehn et al. , 2003; Koehn, 2004a), on this sentence (using the experimental setup described below), we get the following phrases with translations: (4) [Aozhou] [shi] [yu] [Bei Han] [you] [bangjiao]1 [de shaoshu guojia zhiyi] [Australia] [is] [dipl.
For our experiments we used the following features, analogous to Pharaohs default feature set:  P( | ) and P( | ), the latter of which is not found in the noisy-channel model, but has been previously found to be a helpful feature (Och and Ney, 2002);  the lexical weights Pw( | ) and Pw( | ) (Koehn et al. , 2003), which estimate how well the words in  translate the words in ;2  a phrase penalty exp(1), which allows the model to learn a preference for longer or shorter derivations, analogous to Koehns phrase penalty (Koehn, 2003).
3.4 Lexical Weighting The lexical weight a27 a14a12a91 a29 a92a93a21 of the block a9 a72 a14a12a91 a19a86a92a93a21 is computed similarly to (Koehn et al. , 2003), but the lexical translation probability a27 a14a12a94 a29 a97a100a21 is derived from the block set itself rather than from a word alignment, resulting in a simplified training.
5.1 Pharaoh The baseline system we used for comparison was Pharaoh (Koehn et al. , 2003; Koehn, 2004), a freely available decoder for phrase-based translation models: p(e|f) = p(f|e) pLM(e)LM  pD(e,f)D length(e)W(e) (10) We ran GIZA++ (Och and Ney, 2000) on the training corpus in both directions using its default setting, and then applied the refinement rule diagand described in (Koehn et al. , 2003) to obtain a single many-to-many word alignment for each sentence pair.
h1(eI1,fJ1 ) = log Kproductdisplay k=1 N(z)(T(z), Tk) N(T(z)) h2(eI1,fJ1 ) = log Kproductdisplay k=1 N(z)(T(z), Tk) N(S(z)) h3(eI1,fJ1 ) = log Kproductdisplay k=1 lex(T(z)|S(z))(T(z), Tk) h4(eI1,fJ1 ) = log Kproductdisplay k=1 lex(S(z)|T(z))(T(z), Tk) h5(eI1,fJ1 ) = K h6(eI1,fJ1 ) = log Iproductdisplay i=1 p(ei|ei2,ei1) h7(eI1,fJ1 ) = I 4When computing lexical weighting features (Koehn et al. , 2003), we take only terminals into account.
The translation model used in (Koehn et al. , 2003) is the product of translation probability a34a35a4 a29 a0 a33 a6 a29 a2 a33 a8 and distortion probability a36a37a4a39a38 a33a41a40a43a42a44a33a46a45 a32 a8, a3a5a4a35a29 a0 a30 a32 a6 a29 a2 a30 a32 a8 a10 a30 a47 a33a49a48 a32 a34a35a4 a29 a0a22a33 a6 a29 a2 a33a50a8 a36a51a4a39a38 a33 a40a52a42 a33a53a45 a32 a8 (1) where a38 a33 denotes the start position of the source phrase translated into the a54 -th target phrase, and a42 a33a53a45 a32 denotes the end position of the source phrase translated into the a4a53a54 a40a56a55 a8 -th target phrase.
(Koehn et al. , 2003) used the following distortion model, which simply penalizes nonmonotonic phrase alignments based on the word distance of successively translated source phrases with an appropriate value for the parameter a71, a36a51a4a39a38 a33 a40a52a42 a33a53a45 a32 a8 a10 a71a26a72a73a25a74 a45a62a75 a74a77a76a24a78 a45 a32 a72 (3) a79a17a80a82a81a84a83a85a15a86a88a87a70a89a91a90 languageis a means communication of MG RA RA b1 b2 b3 b4 Figure 1: Phrase alignment and reordering bi-1 bi fi-1 fi ei-1 ei bi-1 bi fi-1 fi ei-1 ei bi-1 bi fi-1 fi ei-1 ei bi-1 bi fi-1 fi ei-1 ei source target target source target target source source d=MA d=MG d=RA d=RG Figure 2: Four types of reordering patterns 3 The Global Phrase Reordering Model Figure 1 shows an example of Japanese-English phrase alignment that consists of four phrase pairs.
Word-based features are used as well, e.g. feature a75 a11a39a99a78a99a18a11 captures word-to-word translation de4On our test set, (Tillmann and Zhang, 2005) reports a BLEU score of a100a63a101a63a102a43a103 and (Ittycheriah and Roukos, 2005) reports a BLEU score of a104a89a103a63a102 a105 . pendencies similar to the use of Model a98 probabilities in (Koehn et al. , 2003).
When evaluated against the state-of-the-art, phrase-based decoder Pharaoh (Koehn, 2004), using the same experimental conditions  translation table trained on the FBIS corpus (7.2M Chinese words and 9.2M English words of parallel text), trigram language model trained on 155M words of English newswire, interpolation weights a65 (Equation 2) trained using discriminative training (Och, 2003) (on the 2002 NIST MT evaluation set), probabilistic beam a90 set to 0.01, histogram beam a58 set to 10  and BLEU (Papineni et al. , 2002) as our metric, the WIDL-NGLM-Aa86 a129 algorithm produces translations that have a BLEU score of 0.2570, while Pharaoh translations have a BLEU score of 0.2635.
Running words 1,864 14,437 Vocabulary size 569 1,081 Table 2: ChineseEnglish corpus statistics (Och, 2003) using Phramer (Olteanu et al. , 2006), a 3-gram language model with Kneser-Ney smoothing trained with SRILM (Stolcke, 2002) on the English side of the training data and Pharaoh (Koehn, 2004) with default settings to decode.
The translation table is obtained as described in (Koehn et al. , 2003), i.e. the alignment tool GIZA++ is run over the training data in both translation directions, and the two alignTest Setting BLEU B1 standard phrase-based SMT 29.22 B2 (B1) + clause splitting 29.13 Table 2: Experiment Baseline Test Setting BLEU BLEU 2-ary 2,3-ary 1 rule 29.77 30.31 2 ME (phrase label) 29.93 30.49 3 ME (left,right) 30.10 30.53 4 ME ((3)+head) 30.24 30.71 5 ME ((3)+phrase label) 30.12 30.30 6 ME ((4)+context) 30.24 30.76 Table 3: Tests on Various Reordering Models The 3rd column comprises the BLEU scores obtained by reordering binary nodes only, the 4th column the scores by reordering both binary and 3-ary nodes.
3 Phrase-Based SMT According to the translation model presented in (Koehn et al. , 2003), given a source sentence f, the best target translation best e can be obtained according to the following model )( )()|(maxarg )|(maxarg e e e eef fee length LM best pp p = = (1) Where the translation model )|( efp can be decomposed into  =  = I i i iii i i II aefpbadef efp 1 1 1 1 ),|()()|( )|(   w (2) Where )|( i i ef and )( 1  ii bad denote phrase translation probability and distortion probability, respectively.
Rules have the form X  e, f, where e and f are phrases containing terminal symbols (words) and possibly co-indexed instances of the nonterminal symbol X.2 Associated with each rule is a set of translation model features, i( f, e); for example, one intuitively natural feature of a rule is the phrase translation (log-)probability ( f, e) = log p(e| f) , directly analogous to the corresponding feature in non-hierarchical phrase-based models like Pharaoh (Koehn et al., 2003).
In addition to this phrase translation probability feature, Hieros feature set includes the inverse phrase translation probability log p( f|e), lexical weights lexwt( f|e) and lexwt(e| f), which are estimates of translation quality based on word-level correspondences (Koehn et al., 2003), and a rule penalty allowing the model to learn a preference for longer or shorter derivations; see (Chiang, 2007) for details.
Given phrase p1 and its paraphrase p2, we compute Score3(p1,p2) by relative frequency (Koehn et al., 2003): Score3(p1,p2) = p(p2|p1) = count(p2,p1)P pprime count(pprime,p1) (7) People may wonder why we do not use the same method on the monolingual parallel and comparable corpora.
1 Introduction Currently, most of the phrase-based statistical machine translation (PBSMT) models (Marcu and Wong, 2002; Koehn et al., 2003) adopt full matching strategy for phrase translation, which means that a phrase pair (tildewidef,tildewidee) can be used for translating a source phrase f, only if tildewidef = f. Due to lack of generalization ability, the full matching strategy has some limitations.
They recover additional latent variables so-called nuisance variablesthat are not of interest to the user.1 For example, though machine translation (MT) seeks to output a string, typical MT systems (Koehn et al., 2003; Chiang, 2007) 1These nuisance variables may be annotated in training data, but it is more common for them to be latent even there, i.e., there is no supervision as to their correct values.
594 2.3 Viterbi Approximation To approximate the intractable decoding problem of (2), most MT systems (Koehn et al., 2003; Chiang, 2007) use a simple Viterbi approximation, y = argmax yT(x) pViterbi(y|x) (4) = argmax yT(x) max dD(x,y) p(y,d|x) (5) = Y parenleftBigg argmax dD(x) p(y,d|x) parenrightBigg (6) Clearly, (5) replaces the sum in (2) with a max.
1LDC2002E18 (4,000 sentences), LDC2002T01, LDC2003E07, LDC2003E14, LDC2004T07, LDC2005T10, LDC2004T08 HK Hansards (500,000 sentences) 2http://www.statmt.org/wmt07/shared-task.html For both the tasks, the word alignment were trained by GIZA++ in two translation directions and refined by grow-diag-final method (Koehn et al., 2003).
Accordingly, in this section we describe a set of experiments which extends the work of (Way and Gough, 2005) by evaluating the Marker-based EBMT system of (Gough & Way, 2004b) against a phrase-based SMT system built using the following components:  Giza++, to extract the word-level correspondences;  The Giza++ word alignments are then refined and used to extract phrasal alignments ((Och & Ney, 2003); or (Koehn et al. , 2003) for a more recent implementation);  Probabilities of the extracted phrases are calculated from relative frequencies;  The resulting phrase translation table is passed to the Pharaoh phrase-based SMT decoder which along with SRI language modelling toolkit5 performs translation.
Table 2: The set of tags used to mark explicit morphemes in English Tag Meaning JJR Adjective, comparative JJS Adjective, superlative NNS Noun, plural POS Possessive ending RBR Adverb, comparative RBS Adverb, superlative VB Verb, base form VBD Verb, past tense VBG Verb, gerund or present participle VBN Verb, past participle VBP Verb, non3rd person singular present VBZ Verb, 3rd person singular present Figure 2: Morpheme alignment between a Turkish and an English sentence 4 Experiments We proceeded with the following sequence of experiments: (1) Baseline: As a baseline system, we used a pure word-based approach and used Pharaoh Training tool (2004), to train on the 22,500 sentences, and decoded using Pharaoh (Koehn et al. , 2003) to obtain translations for a test set of 50 sentences.
On the other hand, models that deal with structures or phrases instead of single words have also been proposed: the syntax translation models are described in (Yamada and Knight, 2001), alignment templates are used in (Och, 2002), and the alignment template approach is re-framed into the so-called phrase based translation (PBT) in (Marcu and Wong, 2002; Zens et al. , 2002; Koehn et al. , 2003; Tomas and Casacuberta, 2003).
In a phrase-based statistical translation (Koehn et al. , 2003), a bilingual text is decomposed as K phrase translation pairs (e1, fa1), (e2, fa2 ),: The input foreign sentence is segmented into phrases f K1, 122 mapped into corresponding English eK1, then, reordered to form the output English sentence according to a phrase alignment index mapping a. In a hierarchical phrase-based translation (Chiang, 2005), translation is modeled after a weighted synchronous-CFG consisting of production rules whose right-hand side is paired (Aho and Ullman, 1969): X  ,, where X is a non-terminal,  and  are strings of terminals and non-terminals.
We use the following features for our rules:  sourceand target-conditioned neg-log lexical weights as described in (Koehn et al. , 2003b)  neg-log relative frequencies: left-handside-conditioned, target-phrase-conditioned, source-phrase-conditioned  Counters: n.o. rule applications, n.o. target words  Flags: IsPurelyLexical (i.e. , contains only terminals), IsPurelyAbstract (i.e. , contains only nonterminals), IsXRule (i.e. , non-syntactical span), IsGlueRule 139  Penalties: rareness penalty exp(1  RuleFrequency); unbalancedness penalty |MeanTargetSourceRatio  n.o. source words n.o. target words| 4 Parsing Our SynCFG rules are equivalent to a probabilistic context-free grammar and decoding is therefore an application of chart parsing.
Baseline Pharaoh with phrases extracted from IBM Model 4 training with maximum phrase length 7 and extraction method diag-growthfinal (Koehn et al. , 2003a)  Lex Phrase-decoder simulation: using only the initial lexical rules from the phrase table, all with LHS X, the Glue rule, and a binary reordering rule with its own reordering-feature  XCat All nonterminals merged into a single X nonterminal: simulation of the system Hiero (Chiang, 2005).
It generates a vector of 5 numeric values for each phrase pair:  phrase translation probability: ( f|e) = count( f, e) count(e),(e| f) = count( f, e) count( f) 2http://www.phramer.org/  Java-based open-source phrase based SMT system 3http://www.isi.edu/licensed-sw/carmel/ 4http://www.speech.sri.com/projects/srilm/ 5http://www.iccs.inf.ed.ac.uk/pkoehn/training.tgz 150  lexical weighting (Koehn et al. , 2003): lex( f|e,a) = nproductdisplay i=1 1 |{j|(i, j)  a}| summationdisplay (i,j)a w(fi|ej) lex(e|f,a) = mproductdisplay j=1 1 |{i|(i, j)  a}| summationdisplay (i,j)a w(ej|fi)  phrase penalty: ( f|e) = e; log(( f|e)) = 1 2.2 Decoding We used the Pharaoh decoder for both the Minimum Error Rate Training (Och, 2003) and test dataset decoding.
Two are conditionalized phrasal models, each EM trained until performance degrades:  C-JPTM3 as described in (Birch et al. , 2006)  Phrasal ITG as described in Section 4.1 Three provide alignments for the surface heuristic:  GIZA++ with grow-diag-final (GDF)  Viterbi Phrasal ITG with and without the noncompositional constraint We use the Pharaoh decoder (Koehn et al. , 2003) with the SMT Shared Task baseline system (Koehn and Monz, 2006).
We use the following features in our induced English-to-English grammar:3 3Hiero also uses lexical weights (Koehn et al. , 2003) in both 122 ??The joint probability of the two English hierarchical paraphrases, conditioned on the nonterminal symbol, as defined by this formula: p(e1, e2|x) = c(X ???e1, e2??summationtext e1prime, e2prime c(X ???e1prime, e2prime??
They are generated from the training corpus via the ?diag-and??method (Koehn et al. , 2003) and smoothed using Kneser-Ney smoothing (Foster et al. , 2006), ??one or several n-gram language model(s) trained with the SRILM toolkit (Stolcke, 2002); in the baseline experiments reported here, we used a trigram model, ??a distortion model which assigns a penalty based on the number of source words which are skipped when generating a new target phrase, ??a word penalty.
2 Architecture of the system The goal of statistical machine translation (SMT) is to produce a target sentence e from a source sentence f. It is today common practice to use phrases as translation units (Koehn et al. , 2003; Och and Ney, 2003) and a log linear framework in order to introduce several models explaining the translation process: e??= argmaxp(e|f) = argmaxe {exp(summationdisplay i ihi(e,f))} (1) The feature functions hi are the system models and the i weights are typically optimized to maximize a scoring function on a development set (Och and Ney, 2002).
Baseline We use the Moses MT system (Koehn et al., 2007) as a baseline and closely follow the example training procedure given for the WMT-07 and WMT-08 shared tasks.4 In particular, we perform word alignment in each direction using GIZA++ (Och and Ney, 2003), apply the grow-diag-finaland heuristic for symmetrization and use a maximum phrase length of 7.
(2003), in which we translate a source-language sentence f into the target-language sentence e that maximizes a linear combination of features and weights:1 e,a = argmax e,a score(e,a,f) (1) = argmax e,a Msummationdisplay m=1 mhm(e,a,f) (2) where a represents the segmentation of e and f into phrases and a correspondence between phrases, and each hm is a R-valued feature with learned weight m. The translation is typically found using beam search (Koehn et al., 2003).
2 Architecture of the system The goal of statistical machine translation (SMT) is to produce a target sentence e from a source sentence f. It is today common practice to use phrases as translation units (Koehn et al., 2003; Och and Ney, 2003) and a log linear framework in order to introduce several models explaining the translation process: e = argmaxp(e|f) = argmaxe {exp(summationdisplay i ihi(e,f))} (1) The feature functions hi are the system models and the i weights are typically optimized to maximize a scoring function on a development set (Och and Ney, 2002).