Venue: NAACL 2021

# Macro-Average: Rare Types Are Important Too

Thamme Gowda

Information Sciences Institute

University of Southern California

tg@isi.edu &Weiqiu You

Dept of Computer and Information Science

University of Pennsylvania

weiqiuy@seas.upenn.edu &Constantine Lignos

Michtom School of Computer Science

Brandeis University

lignos@brandeis.edu &Jonathan May

Information Sciences Institute

University of Southern California

jonmay@isi.edu

###### Abstract

While traditional corpus-level evaluation metrics for machine translation (MT) correlate well with fluency, they struggle to reflect adequacy. Model-based MT metrics trained on segment-level human judgments have emerged as an attractive replacement due to strong correlation results. These models, however, require potentially expensive re-training for new domains and languages. Furthermore, their decisions are inherently non-transparent and appear to reflect unwelcome biases. We explore the simple type-based classifier metric, MacroF\({}_{1}\), and study its applicability to MT evaluation. We find that MacroF\({}_{1}\) is competitive on direct assessment, and outperforms others in indicating downstream cross-lingual information retrieval task performance. Further, we show that MacroF\({}_{1}\) can be used to effectively compare supervised and unsupervised neural machine translation, and reveal significant qualitative differences in the methods' outputs.1

Footnote 1: Tools and analysis are available at [https://github.com/thamegowada/07-nt-eval-macro](https://github.com/thamegowada/07-nt-eval-macro). MT evaluation metrics are at [https://github.com/isi-nlp/saccrebleu/tree/macroavg-naacl21](https://github.com/isi-nlp/saccrebleu/tree/macroavg-naacl21).

## 1 Introduction

Model-based metrics for evaluating machine translation such as BLEURT Sellam et al. (2020), ESIM Mathur et al. (2019), and YiSi Lo (2019) have recently attracted attention due to their superior correlation with human judgments Ma et al. (2019). However, BleuPapineni et al. (2002) remains the most widely used corpus-level MT metric. It correlates reasonably well with human judgments, and moreover is easy to understand and cheap to calculate, requiring only reference translations in the target language. By contrast, model-based metrics require tuning on thousands of examples of human evaluation for every new target language or domain

The source of model-based metrics' (e.g. BLEUR) correlative superiority over model-free metrics (e.g. BLEU) appears to be the former's ability to focus evaluation on _adequacy_, while the latter are overly focused on _fluency_. BLEU and most other generation metrics consider each output _token_ equally. Since natural language is dominated by a few high-count types, an MT model that concentrates on getting its _if_'s, _and_s and _but_s right will benefit from BLEU in the long run more than one that gets its _xylophones_, _peripatetic_s, and _defenestr_s right. Can we derive a metric with the discriminating power of BLEUR that does not share its bias or expense and is as interpretable as BLEU?

As it turns out, the metric may already exist and be in common use. Information extraction and other areas concerned with classification have long used both _micro averaging_, which treats each token equally, and _macro averaging_, which instead treats each _type_ equally, when evaluating. The latter in particular is useful when seeking to avoid results dominated by overly frequent types. In this

\begin{table}
\begin{tabular}{c l c} Reference: & You must be a doctor. \\ Hypothesis: & & must be a doctor. \\  & He & -0.735 \\  & Joe & -0.975 \\  & Sue & -1.043 \\  & She & -1.100 \\ \hline Reference: & It is the greatest country in the world. \\ Hypothesis: & & is the greatest country in the world. \\  & France & -0.022 \\  & America & -0.060 \\  & Russia & -0.161 \\  & Canada & -0.309 \\ \end{tabular}
\end{table}
Table 1: A demonstration of BLEUR’s internal biases; model-free metrics like BLEU would consider each of the errors above to be equally wrong.



work we take a classification-based approach to evaluating machine translation in order to obtain an easy-to-calculate metric that focuses on adequacy as much as BLEURT but does not have the expensive overhead, opacity, or bias of model-based methods.

Our contributions are as follows: We consider MT as a classification task, and thus admit MacroF\({}_{1}\) as a legitimate approach to evaluation (Section 2). We show that MacroF\({}_{1}\) is competitive with other popular methods at tracking human judgments in translation (Section 3.2). We offer an additional justification of MacroF\({}_{1}\) as a performance indicator on adequacy-focused downstream tasks such as cross-lingual information retrieval (Section 3.3). Finally, we demonstrate that MacroF\({}_{1}\) is just as good as the expensive BLEURT at discriminating between structurally different MT approaches in a way Bleu cannot, especially regarding the adequacy of generated text, and provide a novel approach to qualitative analysis of the effect of metrics choice on quantitative evaluation (Section 4).

## 2 NMT as Classification

Neural machine translation (NMT) models are often viewed as pairs of encoder-decoder networks. Viewing NMT as such is useful in practice for implementation; however, such a view is inadequate for theoretical analysis. Gowda and May (2020) provide a high-level view of NMT as two fundamental ML components: an autoregressive and a classifier. Specifically, NMT is viewed as a multi-class classifier that operates on representations from an autoregressor. We may thus consider classifier-based evaluation metrics.

Consider a test corpus, \(T=\{(x^{(i)},h^{(i)},y^{(i)})|i=1,2,3...m\}\) where \(x^{(i)}\), \(h^{(i)}\), and \(y^{(i)}\) are source, system hypothesis, and reference translation, respectively. Let \(x=\{x^{(i)}\lor i\}\) and similar for \(h\) and \(y\). Let \(V_{h},V_{y},V_{hr\gamma y}\), and \(V\) be the vocabulary of \(h\), the vocabulary of \(y\), \(V_{h}\cap V_{y}\), and \(V_{h}\cup V_{y}\), respectively. For each class \(c\in V\),

\[\textsc{Preds}(c) =\sum_{i=1}^{m}C(c,h^{(i)})\] \[\textsc{Refs}(c) =\sum_{i=1}^{m}C(c,y^{(i)})\] \[\textsc{Match}(c) =\sum_{i=1}^{m}min\{C(c,h^{(i)}),C(c,y^{(i)})\}\]

where \(C(c,a)\) counts the number of tokens of type \(c\) in sequence \(a\)(Papinen et al., 2002). For each class \(c\in V_{hr\gamma y}\), precision (\(P_{c}\)), recall (\(R_{c}\)), and \(F_{\beta}\) measure (\(F_{\beta;c}\)) are computed as follows:2

Footnote 2: We consider \(F_{B;c}\) for \(c\notin V_{hr\gamma y}\) to be 0.

\[P_{c} =\frac{\textsc{Match}(c)}{\textsc{Preds}(c)}; R_{c}=\frac{\textsc{Match}(c)}{\textsc{Refs}(c)}\] \[F_{\beta;c} =(1+\beta^{2})\frac{P_{c}\times R_{c}}{\beta^{2}\times P_{c}+R_{ c}}\]

The _macro-average_ consolidates individual performance by averaging by type, while the _micro-average_ averages by token:

\[\textsc{MacroF}_{\beta} =\frac{\sum_{c\in V}F_{\beta;c}}{|V|}\] MicroF\({}_{\beta}\) =\frac{\sum_{c\in V}f(c)\times F_{\beta;c}}{\sum_{c^{\prime}\in V} f(c^{\prime})}\]

where \(f(c)=\textsc{Refs}(c)+k\) for smoothing factor \(k\).3 We scale MacroF\({}_{\beta}\) and MicroF\({}_{\beta}\) values to percentile, similar to Bleu, for the sake of easier readability.

Footnote 3: We use \(k=1\). When \(k\rightarrow\infty\),MicroF\({}_{\beta}\rightarrow\textsc{MacroF}_{\beta}\).

We scale Kendall's rank correlation coefficient, \(\tau\), to compute the association between metrics and human judgments. Correlations with p-values smaller than \(\alpha=0.05\) are considered to be statistically significant.

### Data-to-Text: WebNLG

We use the 2017 WebNLG Challenge dataset (Sinden et al., 2017; Shimorina, 2018)5 to analyze the differences between micro- and macro- averaging. WebNLG is a task of generating English text for sets of triples extracted from DBPedia. Human annotations are available for a sample of 223 records each from nine NLG systems. The human 

judgments provided have three linguistic aspects--fluency, grammar, and semantics6--which enable us to perform a fine grained analysis of our metrics. We compute Kendall's \(\tau\) between metrics and human judgments, which are reported in Table 2.

Footnote 5: [https://github.com/](https://github.com/) 

stronger correlation with the IR metric(s) is more useful than the ones with weaker correlations.
4. Repeat the above steps on many languages to verify the generalizability of findings. An essential resource of this analysis is a dataset with human annotations for computing MT and IR performances. We conduct experiments on two datasets: firstly, on data from the 2020 workshop on _Cross-Language Search and Summarization of Text and Speech_ (CLSSTS) Zavorin et al. (2020), and secondly, on data originally from Europarl, prepared by Lignos et al. (2019) (Europarl).

#### 3.3.1 CLSSTS Datasets

CLSSTS datasets contain queries in English (EN), and documents in many source languages along with their human translations, as well as query-document relevance judgments. We use three source languages: Lithuanian (LT), Pashto (PS), and Bulgarian (BG). The performance of this CLIR task is evaluated using two IR measures: Actual Query Weighted Value (AQWV) and Mean Average Precision (MAP). AQWV9 is derived from Actual Term Weighted Value (ATWV) metric Wegmann et al. (2013). We use a single CLIR system Boschee et al. (2019) with the same IR settings for all MT models in the set, and measure Kendall's \(\tau\) between MT and IR measures. The results, in Table 4, show that MacroF\({}_{1}\) is the strongest indicator of CLIR downstream task performance in five out of six settings. AQWV and MAP have a similar trend in agreement to the MT metrics. ChrF\({}_{1}\) and BLEURT, which are strong contenders when generated text is directly evaluated by humans, do not indicate CLIR task performance as well as MacroF\({}_{1}\), as CLIR tasks require faithful meaning equivalence across the language boundary, and human translators can mistake fluent output for proper translations Callison-Burch et al. (2007).

Footnote 9: [https://www.nist.gov/system/files/documents-/2017/10/26/aqwv_derivation.pdf](https://www.nist.gov/system/files/documents-/2017/10/26/aqwv_derivation.pdf)

#### 3.3.2 Europarl Datasets

We perform a similar analysis to Section 3.3.1 but on another cross-lingual task set up by Lignos et al. (2019) for Czech \(\rightarrow\) English (CS-EN) and German \(\rightarrow\) English (DE-EN), using publicly available data from the Europarl v7 corpus Koehn (2005). This task differs from the CLSSTS task (Section 3.3.1) in several ways. Firstly, MT metrics are computed on test sets from the news domain, whereas IR metrics are from the Europarl domain. The domains are thus intentionally mismatched between MT and IR tests. Secondly, since there are no queries specifically created for the Europarl domain, GOV2 TREC topics 701-850 are used as domain-relevant English queries. And lastly, since there are no query-document relevance human judgments for the chosen query and document sets, the documents retrieved by BM25 Jones et al. (2000) on the English set for each query are treated as relevant documents for computing the performance of the CS-EN and DE-EN CLIR setup. As a result, IR metrics that rely on boolean query-document relevance judgments as ground truth are less informative, and we use Rank-Based Overlap (RBO; \(p=0.98\)) Webber et al. (2010) as our IR metric. We perform our analysis on the same experiments as Lignos et al. (2019).10 NMT models for CS-EN and DE-EN translation are trained using a convolutional NMT architecture Gehring

\begin{table}
\begin{tabular}{c c r r r r r r} Year & Pairs & & * Bleu & Bleu & MacroF\({}_{1}\) & MicroF\({}_{1}\) & ChrF\({}_{1}\) \\ \hline \hline  & & Mean & .751 & .771 & .821 & .818 & .841 \\
2019 & 18 & Median & .782 & .752 & .844 & .844 & .875 \\  & & Wins & 3 & 3 & **6** & 3 & 5 \\ \hline  & & Mean & .858 & .857 & .875 & .873 & .902 \\
2018 & 14 & Median & .868 & .868 & .901 & .879 & .919 \\  & & Wins & 1 & 2 & 3 & 2 & **6** \\ \hline  & & Mean & .752 & .713 & .714 & .742 & .804 \\
2017 & 13 & Median & .758 & .733 & .735 & .728 & .791 \\  & & Wins & 5 & 4 & 2 & 2 & **6** \\ \end{tabular}
\end{table}
Table 3: WMT 2017–19 Metrics task: Mean and median Kendall’ s \(\tau\) between MT metrics and human judgments. Correlations that are not significant at \(\alpha\) = 0.05 are excluded from the calculation of mean, and median, and wins. See Appendix Tables 9, 10, and 11 for full details. * Bleu is pre-computed scores available in the metrics packages. In 2018 and 2019, both MacroF\({}_{1}\) and MicroF\({}_{1}\) outperform Bleu, MacroF\({}_{1}\) outperforms MicroF\({}_{1}\). ChrF\({}_{1}\) has strongest mean and median agreements across the years. Judging based on the number of wins, MacroF\({}_{1}\) has steady progress over the years, and outperforms others in 2019.

 

et al., 2017) implemented in the FAIRSeq Ott et al. (2019) toolkit. For each of CS-EN and DE-EN, a total of \(16\) NMT models that are based on different quantities of training data and BPE hyperparameter values are used. The results in Table 5 show that BLEURT has the highest correlation in both cases. Apart from the trained BLEURTmedian metric, MacroF\({}_{1}\) scores higher than the others on CS-EN, and is competitive on CS-EN. MacroF\({}_{1}\) is not the metric with highest IR task correlation in this setting, unlike in Section 3.3.1, however it is competitive with Bleu and ChrF\({}_{1}\), and thus a safe choice as a downstream task performance indicator.

## 4 Spotting Qualitative Differences Between Supervised and Unsupervised NMT with MacroF\({}_{1}\)

Unsupervised neural machine translation (UNMT) systems trained on massive monolingual data without parallel corpora have made significant progress recently Artetxe et al. (2018); Lample et al. (2018a, b); Conneau and Lample (2019); Song et al. (2019); Liu et al. (2020). In some cases, UNMT yields a Bleu score that is comparable with strong11 supervised neural machine translation (SNMT) systems. In this section we leverage MacroF\({}_{1}\) to investigate differences in the translations from UNMT and SNMT systems that have similar Bleu.

Footnote 11: though not, generally, the strongest

We compare UNMT and SNMT for English \(\leftrightarrow\) German (EN-DE, DE-EN), English \(\leftrightarrow\) French (EN-FR, FR-EN), and English \(\leftrightarrow\) Romanian (EN-RO, RO-EN). All our UNMT models are based on XLM Conneau and Lample (2019), pretrained by Yang (2020). We choose SNMT models with similar Bleu on common test sets by either selecting from systems submitted to previous WMT News Translation shared tasks Bojar et al. (2014, 2016) or by building such systems.12 Specific SNMT models chosen are in the Appendix (Table 12).

Footnote 12: We were unable to find EN-DE and DE-EN systems with comparable Bleu in WMT submissions so we built standard Transformer-base Vaswani et al. (2017) models for these using appropriate quantity of training data to reach the desired Bleu performance. We report EN-RO results with diacritic removed to match the output of UNMT.

Table 6 shows performance for these three language pairs using a variety of metrics. Despite comparable scores in Bleu and only minor differences in MicroF\({}_{1}\) and ChrF\({}_{1}\), SNMT models have consistently higher MacroF\({}_{1}\) and BLEURT than the UNMT models for all six translation directions.

In the following section, we use a pairwise maximum difference discriminator approach to compare corpus-level metrics Bleu and MacroF\({}_{1}\) on a segment level. Qualitatively, we take a closer look at the behavior of the two metrics when comparing a translation with altered meaning to a translation with differing word choices using the metric.

\begin{table}
\begin{tabular}{l l c r r r r r r}  & Domain & IR Score & Bleu & MacroF\({}_{1}\) & MicroF\({}_{1}\) & ChrF\({}_{1}\) & BLEURTmean & BLEURTmedian \\ \hline \hline \multirow{4}{*}{LT-EN} & In & AQWV & .429 & x.363 & **.508** & x.385 & .451 & .420 \\  & & MAP & .495 & .429 & **.575** & .451 & .473 & .486 \\  & & AWWF & x.345 & **.527** & .491 & .491 & .491 & .477 \\  & & MAP & x.273 & x.**455** & .418 & x.418 & x.418 & x.404 \\ \hline \multirow{4}{*}{PS-EN} & In & AQWV & .559 & **.653** & .574 & .581 & .584 & .581 \\  & & MAP & .493 & **.632** & .487 & .494 & .558 & .554 \\  & & AQWV & .589 & **.682** & .593 & .583 & .581 & .571 \\  & & MAP & .519 & **.637** & .523 & .482 & .536 & .526 \\ \hline \multirow{4}{*}{BG-EN} & In & AQWV & x.455 & **.550** & .527 & .382 & x.418 & .418 \\  & & MAP & .491 & **.661** & .564 & .491 & .527 & .527 \\ \cline{1-1}  & & AQWV & x 257 & **.500** & x.330 & x.404 & x.367 & x.367 \\ \cline{1-1}  & & MAP & x.183 & x.**426** & x.257 & x.330 & x.294 & x.294 \\ \end{tabular}
\end{table}
Table 4: CLSSTS CLIR task: Kendall’s \(\tau\) between IR and MT metrics under study. The rows with Domanin=In are where MT and IR scores are computed on the same set of documents, whereas Domain=In+Ext are where IR scores are computed on a larger set of documents that is a superset of segments on which MT scores are computed. **Bold** values are the best correlations achieved in a row-wise setting; values with × are _not_ significant at \(\alpha=0.05\).

\begin{table}
\begin{tabular}{c c c c c c}  & Bleu & MacroF\({}_{1}\) & MicroF\({}_{1}\) & ChrF\({}_{1}\) & BT & BT \\ \hline \hline CS-EN & .850 & .867 & .850 & .850 & **.900** & .867 \\ DE-EN & .900 & .900 & .900 & .912 & **.917** & .900 \\ \end{tabular}
\end{table}
Table 5: Europaral CLIR task: Kendall’s \(\tau\) between MT metrics and RBO. \(\overline{\text{BT}}\) and \(\widehat{\text{BT}}\) are short for BLEURTmean and BLEURTmedian. All correlations are significant at \(\alpha=0.05\).

 



### Pairwise Maximum Difference Discriminator

We consider cases where a metric has a strong opinion of one translation system over another, and analyze whether the opinion is well justified. In order to obtain this analysis, we employ a pairwise segment-level discriminator from within a corpus-level metric, which we call _favoritism_.

We extend the definition of \(T\) from Section 2 to \(T=\{x,h_{S},h_{U},y\}\) where each of \(h_{S}\) and \(h_{U}\) is a separate system's hypothesis set for \(x\).13 Let \(M\) be a corpus-level measure such that \(M(h,y)\in\mathbb{R}\) and a higher value implies better translation quality. \(M(h^{(-i)},y^{(-i)})\) is the corpus-level score obtained by excluding \(h^{(i)}\) and \(y^{(i)}\) from \(h\) and \(y\), respectively. We define the _benefit_ of segment \(i\), \(\delta_{M}(i;h)\):

Footnote 13: The subscripts represent SNMT and UNMT in this case, though the definition is general.

\[\delta_{M}(i;h)=M(h,y)-M(h^{(-i)},y^{(-i)})\]

If \(\delta_{M}(i;h)>0\), then \(i\) is beneficial to \(h\) with respect to \(M\), as the inclusion of \(h^{(i)}\) increases the corpus-level score. We define the _favoritism_ of \(M\) toward \(i\) as \(\delta_{M}(i;h_{S},h_{U})\):

\[\delta_{M}(i;h_{S},h_{U})=\delta_{M}(i;h_{S})-\delta_{M}(i;h_{U}) \tag{1}\]

If \(\delta_{M}(i;h_{S},h_{U})>0\) then \(M\) favors the translation of \(x^{(i)}\) by system \(S\) over that in system \(U\).

Table 7 reflects the results of a manual examination of the ten sentences in the DE-EN test set with greatest magnitude favoritism; complete results are in the Appendix, Tables 15 and 16. Meaning-altering changes such as _'untranslation'_, (wrong) _'time'_, and (wrong) _'translation'_ are marked in _italics_, while changes that do not fundamentally alter the meaning, such as 'synonym,' (different) 'inflection,' and (different) 'word order' are marked in plain text.14

Footnote 14: Some changes, such as ‘word order’ may change meaning; these are italicized or not on a case-by-case basis.

The results indicate that MacroF\({}_{1}\) generally favors SNMT, and with good reasons, as the favored translation does not generally alter sentence meaning, while the disfavored t

\begin{table}
\begin{tabular}{c c r|r r r|r r r|r r r|r r r}  & \multicolumn{2}{c|}{Bleu} & \multicolumn{2}{c|}{MacroF\({}_{1}\)} & \multicolumn{2}{c|}{MicroF\({}_{1}\)} & \multicolumn{2}{c|}{ChGrF\({}_{1}\)} & \multicolumn{2}{c|}{BLEURTEmean} & \multicolumn{2}{c}{BLEURTEmedian} \\  & SN & UN & \(\Delta\) & SN & UN & \(\Delta\) & SN & UN & \(\Delta\) & SN & UN & \(\Delta\) & SN & UN & \(\Delta\) \\ \hline \hline DE-EN & 32.7 & 33.9 & -1.2 & 38.5 & 33.6 & 4.9 & 58.7 & 57.9 & 0.8 & 59.9 & 58.0 & 1.9 & 2.11 & -0.26 & 2.4 & 2.85 & .067 & .22 \\ EN-DE & 24.0 & 24.0 & 0.0 & 24.0 & 23.5 & 0.7 & 47.8 & -0.4 & 53.5 & 52.0 & 1.3 & -1.34 & -2.04 & .07 & -1.12 & .197 & .09 \\ FR-EN & 31.1 & 31.2 & -0.1 & 41.6 & 33.6 & 8.0 & 60.5 & 58.3 & 2.2 & 59.1 & 57.3 & 1.8 & 1.82 & .066 & .17 & .243 & .154 & .09 \\ EN-FR & 25.6 & 27.1 & -1.5 & 31.9 & 27.3 & 4.6 & 53.0 & 52.3 & 0.7 & 56.0 & 57.7 & -1.7 & .14 & .04 & .04 & .02 & .06 & .096 & .063 & .03 \\ RO-EN & 30.8 & 29.6 & 1.2 & 40.3 & 33.0 & 7.3 & 59.8 & 56.5 & 3.3 & 58.0 & 54.7 & 3.3 & .004 & -0.58 & .06 & .045 & .-004 & .04 \\ EN-RO & 31.2 & 31.0 & 0.2 & 34.6 & 31.0 & 3.6 & 55.4 & 53.4 & 2.0 & 59.3 & 56.7 & 2.6 & .030 & -0.46 & .08 & .027 & -0.38 & .07 \\ \end{tabular}
\end{table}
Table 6: For each language direction, UNMT (UN) models have similar Bleu to SNMT (SN) models, and ChrF\({}_{1}\) and MicroF\({}_{1}\) have small differences. However, MacroF\({}_{1}\) scores differ significantly, consistently in favor of SNMT. Both corpus-level interpretations of BLEURT support the trend reflected by MacroF\({}_{1}\), but the value differences are difficult to interpret.

\begin{table}
\begin{tabular}{c l l|l l} \(\delta_{\text{MacRoF${}_{1}$}}\) & Fav & Analysis & \(\delta_{\text{BLEU}}\) & Fav & Analysis \\ \hline \hline .071 S & S: synonym; U: _untranslation, noun_ & .048 S & S: word order; U: word order, _ultranslation_, _ending_ & _?_ \\ .064 S & S: synonym; U: _intranslation_ & .046 S & S: spelling variation; U: synonym, word order, punctuation \\ -.055 U & U: no issues; S: _outranslation_ & .044 S & S: extra determiner; U: paraphrase, synonym, _number, untranslation_ & _?_ \\ .052 S & S: synonym; U: _intranslation, noun_ & .042 S & S: synonym; U: synonym, punctuation, extra adverb \\ -.045 U & U: no issues; S: _outranslation_ & -.039 U & U: no issues; S: _noun_, _verb_ & _!_ \\ .044 S & S: synonym, word order; U: _subject_, _truncation_, word order & -.037 U & U: no issues; S: punctuation \\ .044 S & S: synonym, tense; U: _intranslation_ & -.034 U & U: no issues; S: symbol \\ .043 S & S: inflection, word order; U: _number_ & -.032 U & U: no issues; S: _adjective_, _noun_ & _?_ \\ .041 U & U: _adjective_, _verb_; S: _omitted verb_, _untranslation_ & -.032 U & U: _!_ untransmission_; S: tense, _word order_, _meaning_, active/passive voice \\ .041 S & S: _time_, word order; U: _time_, _ nouns_ & -.031 U & U: _utranstation_; S: word order, synonym, _extra_cont_ & _?_ \\ \end{tabular}
\end{table}
Table 7: Analysis of the ten DE-EN test set segments with the most favoritism in SNMT (S) or UNMT (U), according to MacroF\({}_{1}\) (left) and Bleu (right). Fav is the favored system by metrics. The complete text of the sentences is in the Appendix, Tables 15 and 16.



the other hand, for the ten most favored sentences according to Bleu, four do not contain meaning-altering divergences in the disfavored translation. Importantly, none of the sentences with greatest favoritism according to MacroF\({}_{1}\), all of which having meaning altering changes in the disfavored alternatives, appears in the list for Bleu. This indicates relatively bad judgment on the part of Bleu. One case of good judgment from MacroF\({}_{1}\) and bad judgment from Bleu regarding truncation is shown in Table 8.

From our qualitative examinations, MacroF\({}_{1}\) is better than Bleu at discriminating against untranslations and truncations in UNMT. The case is similar for FR-EN and RO-EN, except that RO-EN has more untranslations for both SNMT and UNMT, possibly due to the smaller training data. Complete tables and annotated sentences are in the Appendix, in Section C.

## 5 Related Work

### MT Metrics

Many metrics have been proposed for MT evaluation, which we broadly categorize into _model-free_ or _model-based_. Model-free metrics compute scores based on translations but have no significant parameters or hyperparameters that must be tuned _a priori_; these include BleuPapineni et al. (2002), NISTDoddington (2002), TERSnover et al. (2006), and ChrF\({}_{1}\)Popovic (2015). Model-based metrics have a significant number of parameters and, sometimes, external resources that must be set prior to use. These include METEORBanerjee and Lavie (2005), BLEURTSellam et al. (2020), YiSiLo (2019), ESIMMathur et al. (2019), and BEERStanojevic and Sima'an (2014). Model-based metrics require significant effort and resources when adapting to a new language or domain, while model-free metrics require only a test set with references.

Mathur et al. (2020) have recently evaluated the utility of popular metrics and recommend the use of either ChrF\({}_{1}\) or a model-based metric instead of Bleu. We compare our MacroF\({}_{1}\) and MicroF\({}_{1}\) metrics with Bleu, ChrF\({}_{1}\), and BLEURTSellam et al. (2020). While Mathur et al. (2020) use Pearson's correlation coefficient (\(r\)) to quantify the correlation between automatic evaluation metrics and human judgements, we instead use Kendall's rank coefficient (\(\tau\)), since \(\tau\) is more robust to outliers than \(r\)Croux and Dehon (2010).

### Rare Words are Important

That natural language word types roughly follow a Zipfian distribution is a well known phenomenonZipf (1949); Powers (1998). The frequent types are mainly so-called "stop words," function words, and other low-information types, while most content words are infrequent types. To counter this natural frequency-based imbalance, statistics such as inverted document frequency (IDF) are commonly used to weigh the _input_ words in applications such as information retrievalJones (1972). In NLG tasks such as MT, where words are the _output_ of a classifier, there has been scant effort to address the imbalance. Doddington (2002) is the only work we know of in which the 'information' of an n-gram is used as its weight, such that rare n-grams attain relatively more importance than in BLEU. We abandon this direction for two reasons: Firstly, as noted in that work, _large amounts of data are required to estimate n-gram statistics_. Secondly, unequal weighing is a bias that is best suited to datasets where the weights are derived from, and such biases often do not generalize to other datasets. Therefore, unlike Doddington (2002), we assign equal weights to all n-gram classes, and in this work we limit our scope to unigrams only.

While Bleu is a precision-oriented measure,

\begin{table}
\begin{tabular}{p{34.1pt}|p{34.1pt}|p{34.1pt}} \hline \(6^{th}\) & \(\delta_{\text{MacoF${}_{1}$}}(h,g_{k},b_{i})\): 0.044, \(\delta_{\text{ 

METEOR Banerjee and Lavie (2005) and CHRF Popovic (2015) include both precision and recall, similar to our methods. However, neither of these measures try to address the natural imbalance of class distribution. BEER Stanojevic and Sima'an (2014) and METEOR Denkowski and Lavie (2011) make an explicit distinction between function and content words; such a distinction inherently captures frequency differences since function words are often frequent and content words are often infrequent types. However, doing so requires the construction of potentially expensive linguistic resources. This work does not make any explicit distinction and uses naturally occurring type counts to effect a similar result.

### F-measure as an Evaluation Metric

F-measure Rijsbergen (1979); Chinchor (1992) is extensively used as an evaluation metric in classification tasks such as part-of-speech tagging, named entity recognition, and sentiment analysis Derczynski (2016). Viewing MT as a multi-class classifier is a relatively new paradigm Gowda and May (2020), and evaluating MT solely as a multi-class classifier as proposed in this work is not an established practice. However, we find that the \(F_{1}\) measure is sometimes used for various analyses when Bleu and others are inadequate: The compare-mt tool Neubig et al. (2019) supports comparison of MT models based on \(F_{1}\) measure of individual types. Gowda and May (2020) use \(F_{1}\) of individual types to uncover frequency-based bias in MT models. Sennrich et al. (2016) use corpus-level _unigram_\(F_{1}\) in addition to Bleu and Chrf, however, corpus-level \(F_{1}\) is computed as MicroF\({}_{1}\). To the best of our knowledge, there is no previous work that clearly formulates the differences between micro- and macro- averages, and justifies the use of MacroF\({}_{1}\) for MT evaluation.

## 6 Discussion and Conclusion

We have evaluated NLG in general and MT specifically as a multi-class classifier, and illustrated the differences between micro- and macro- averages using MicroF\({}_{1}\) and MacroF\({}_{1}\) as examples (Section 2). MacroF\({}_{1}\) captures semantic adequacy better than MicroF\({}_{1}\) (Section 3.1). Bleu, being a micro-averaged measure, served well in an era when generating fluent text was at least as difficult as generating adequate text. Since we are now in an era in which fluency is taken for granted and semantic adequacy is a key discriminating factor, macro-averaged measures such as MacroF\({}_{1}\) are better at judging the generation quality of MT models (Section 3.2). We have found that another popular metric, ChrF\({}_{1}\), also performs well on direct assessment, however, being an implicitly micro-averaged measure, it does not perform as well as MacroF\({}_{1}\) on downstream CLIR tasks (Section 3.3.1). Unlike BLEURT, which is also adequacy-oriented, MacroF\({}_{1}\) is directly interpretable, does not require retuning on expensive human evaluations when changing language or domain, and does not appear to have uncontrollable biases resulting from data effects. It is both easy to understand and to calculate, and is inspectable, enabling fine-grained analysis at the level of individual word types. These attributes make it a useful metric for understanding and addressing the flaws of current models. For instance, we have used MacroF\({}_{1}\) to compare supervised and unsupervised NMT models at the same operating point measured in Bleu, and determined that supervised models have better adequacy than the current unsupervised models (Section 4).

Macro-average is a useful technique for addressing the importance of the long tail of language, and MacroF\({}_{1}\) is our first step in that direction; we anticipate the development of more advanced macro-averaged metrics that take advantage of higher-order and character n-grams in the future.

## 7 Ethical Consideration

Since many ML models including NMT are themselves opaque and known to possess data-induced biases Prates et al. (2019), using opaque and biased evaluation metrics in concurrence makes it even harder to discover and address the flaws in modeling. Hence, we have raised concerns about the opaque nature of the current model-based evaluation metrics, and demonstrated examples displaying unwelcome biases in evaluation. We advocate the use of the MacroF\({}_{1}\) metric, as it is easily interpretable and offers the explanation of score as a composition of individual type performances. In addition, MacroF\({}_{1}\) treats all types equally, and has no parameters that are directly or indirectly estimated from data sets. Unlike MacroF\({}_{1}\), MicroF\({}_{1}\) and other implicitly or explicitly micro-averaged metrics assign lower importance to rare concepts and their associated rare types. The use of microc-averaged metrics in real world evaluation could lead to marginalization of rare types 

_Failure Modes:_ The proposed MacroF\({}_{1}\) metric is not the best measure of fluency of text. Hence we suggest caution while using MacroF\({}_{1}\) to draw fluency related decisions. MacroF\({}_{1}\) is inherently concerned with _words_, and assumes the output language is easily segmentable into word tokens. Using MacroF\({}_{1}\) to evaluate translation into alphabetical languages such as Thai, Lao, and Khmer, that do not use white space to segment words, requires an effective tokenizer. Absent this the method may be ineffective; we have not tested it on languages beyond those listed in Section B.

_Reproducibility:_ Our implementation of MacroF\({}_{1}\) and MicroF\({}_{1}\) has the same user experience as Bleu as implemented in SacreBleu; signatures are provided in Section A. In addition, our implementation is computationally efficient, and has the same (minimal) software and hardware requirements as Bleu. All data for MT and NLG human correlation studies is publicly available and documented. Data for reproducing the IR experiments in Section 3.3.2 is also publicly available and documented. The data for reproducing the IR experiments in Section 3.3.1 is only available to participants in the CLSSTS shared task.

_Climate Impact:_ Our proposed metrics are on par with Bleu and such model-free methods, which consume significantly less energy than most model-based evaluation metrics.

## Acknowledgements

The authors thank Shantanu Agarwal, Joel Barry, and Scott Miller for their help with CLSSTS CLIR experiments, and Daniel Cohen for discussions on IR evaluation methods. This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via AFRL Contract FA8650-17-C-9116. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon.

## References

* May 3, 2018, Conference Track Proceedings_. OpenReview.net.
* Banerjee and Lavie [2005] Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
* Bojar et al. [2014] Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ales Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In _Proceedings of the Ninth Workshop on Statistical Machine Translation_, pages 12-58, Baltimore, Maryland, USA. Association for Computational Linguistics.
* Bojar et al. [2017] Ondrej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the WMT17 metrics shared task. In _Proceedings of the Second Conference on Machine Translation_, pages 489-513, Copenhagen, Denmark. Association for Computational Linguistics.
* Bojar et al. [2016] Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Auerlie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In _Proceedings of the First Conference on Machine Translation_, pages 131-198, Berlin, Germany. Association for Computational Linguistics.
* Boschee et al. [2019] Elizabeth Boschee, Joel Barry, Jayadey Billa, Marjorie Freedman, Thamme Gowda, Constantine Lignons, Chester Palen-Michel, Michael Pust, Bankshem Kayang Khonglah, Srikanth Madhieri, Jonathan May, and Scott Miller. 2019. SARAL: A low-resource cross-lingual domain-focused information retrieval system for effective rapid document triage. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 19-24, Florence, Italy. Association for Computational Linguistics.
* Callison-Burch et al. [2007] Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (Meta-) evaluation of machine translation. In _Proceedings of the Second Workshop on Statistical Machine Translation_, pages 136-158, Prague, Czech Republic. Association for Computational Linguistics 

Nancy Chinchor. 1992. MUC-4 Evaluation Metrics. In _Proceedings of the 4th Conference on Message Understanding_, MUC4 '92, page 22-29, USA. Association for Computational Linguistics.
* Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlche-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_, pages 7059-7069. Curran Associates, Inc.
* Croux and Dehon (2010) Christophe Croux and Catherine Dehon. 2010. Influence functions of the spearman and kendall correlation measures. _Statistical methods & applications_, 19(4):497-515.
* Denkowski and Lavie (2011) Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In _Proceedings of the Sixth Workshop on Statistical Machine Translation_, pages 85-91, Edinburgh, Scotland. Association for Computational Linguistics.
* Derczynski (2016) Leon Derczynski. 2016. Complementarity, F-score, and NLP evaluation. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)_, pages 261-266, Portoroz, Slovenia. European Language Resources Association (ELRA).
* Doddington (2002) George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In _Proceedings of the Second International Conference on Human Language Technology Research_, HLT '02, page 138-145, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
* Gardent et al. (2017) Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrrachini. 2017. Creating training corpora for NLG micro-planners. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 179-188. Association for Computational Linguistics.
* Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_, pages 1243-1252.
* Gowda and May (2020) Thamme Gowda and Jonathan May. 2020. Finding the optimal vocabulary size for neural machine translation. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 3955-3964, Online. Association for Computational Linguistics.
* Grefenstette (2012) Gregory Grefenstette. 2012. _Cross-language information retrieval_, volume 2. Springer Science & Business Media.
* Jones (1972) Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. _Journal of Documentation_, 28:11-21.
* Jones et al. (2000) Karen Sparck Jones, Steve Walker, and Stephen E. Robertson. 2000. A probabilistic model of information retrieval: development and comparative experiments: Part 2. _Information processing & management_, 36(6):809-840.
* Koehn (2005) Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. _Proc. 10th Machine Translation Summit (MT Summit), 2005_, pages 79-86.
* May 3, 2018, Conference Track Proceedings_. OpenReview.net.
* Lample et al. (2018b) Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018b. Phrase-based & neural unsupervised machine translation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 5039-5049, Brussels, Belgium. Association for Computational Linguistics.
* Lignos et al. (2019) Constantine Lignos, Daniel Cohen, Yen-Chieh Lien, Pratik Mehta, W. Bruce Croft, and Scott Miller. 2019. The challenges of optimizing machine translation for low resource cross-language information retrieval. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3497-3502, Hong Kong, China. Association for Computational Linguistics.
* Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation.
* a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In _Proceedings of the Fourth Conference on Machine Translations (Volume 2: Shared Task Papers, Day 1)_, pages 507-513, Florence, Italy. Association for Computational Linguistics.
* Ma et al. (2018) Qingsong Ma, Ondrej Bojar, and Yvette Graham. 2018. Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance. In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_, pages 671-688, Belgium, Brussels. Association for Computational Linguistics.
* Ma et al. (2019) Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In _Proceedings of the Fourth Conference on Machine Translation (Volume 

* _2: Shared Task Papers, Day 1)_, pages 62-90, Florence, Italy. Association for Computational Linguistics.
* Mathur et al. (2019) Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2799-2808, Florence, Italy. Association for Computational Linguistics.
* Mathur et al. (2020) Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4984-4997, Online. Association for Computational Linguistics.
* Neubig et al. (2019) Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, and Xinyi Wang. 2019. compare-mt: A tool for holistic comparison of language generation systems. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_, pages 35-41, Minneapolis, Minnesota. Association for Computational Linguistics.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of NAACL-HLT 2019: Demonstrations_.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Popovic (2015) Maja Popovic. 2015. chrF: character n-gram f-score for automatic MT evaluation. In _Proceedings of the Tenth Workshop on Statistical Machine Translation_, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
* Powers (1998) David M. W. Powers. 1998. Applications and explanations of Zipf's law. In _New Methods in Language Processing and Computational Natural Language Learning_.
* Prates et al. (2019) Marcelo O.R. Prates, Pedro H. Avelar, and Luis C. Lamb. 2019. Assessing gender bias in machine translation: a case study with google translate. _Neural Computing and Applications_, pages 1-19.
* Rujsbergen (1979) C. J. Van Rijsbergen. 1979. _Information Retrieval_, 2nd edition. Butterworth-Heinemann, USA.
* Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7881-7892, Online. Association for Computational Linguistics.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
* Shimorina (2018) Anastasia Shimorina. 2018. Human vs automatic metrics: on the importance of correlation design. _CoRR_, abs/1805.11474.
* Snover et al. (2006) Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In _Proceedings of association for machine translation in the Americas_, volume 200. Cambridge, MA.
* Song et al. (2019) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tiem Yan Liu. 2019. MASS: Masked sequence to sequence pre-training for language generation. In _Proceedings of Machine Learning Research_, volume 97, pages 5926-5936, Long Beach, California, USA. PMLR.
* Stanojevic and Sima'an (2014) Milos Stanojevic and Khalil Sima'an. 2014. BEER: BEfter evaluation as ranking. In _Proceedings of the Ninth Workshop on Statistical Machine Translation_, pages 414-419, Baltimore, Maryland, USA. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in neural information processing systems_, pages 5998-6008.
* Webber et al. (2010) William Webber, Alistair Moffat, and Justin Zobel. 2010. A similarity measure for indefinite rankings. _ACM Transactions on Information Systems (TOIS)_, 28(4):1-38.
* Wegmann et al. (2013) Steven Wegmann, Arlo Faria, Adam Janin, Korbinian Riedhammer, and Nelson Morgan. 2013. The TAO of ATWV: Probing the mysteries of keyword search performance. In _2013 IEEE Workshop on Automatic Speech Recognition and Understanding_, pages 192-197. IEEE.
* Yang (2020) Hans Yang. 2020. XLM-UNMT-Models. [https://github.com/hansxcourse/XLM-UNMT-Models](https://github.com/hansxcourse/XLM-UNMT-Models).
* Zavorin et al. (2020) Ilya Zavorin, Eric Bills, Cassian Corey, Michelle Morrison, Audrey Tong, and Richard Tong. 2020. Corpora for cross-language information retrieval in six less-resourced languages. In _Proceedings of the workshop on Cross-Language Search and Summarization of Text and Speech (CLSSTS2020)_, pages 7-13, Marseille, France. European Language Resources Association.
* Zipf (1949) George Kingsley Zipf. 1949. Human behaviour and the principle of least-effort. Cambridge MA edn. _Addison-Wesley_ 

Metrics Reproducibility

Bleu scores reported in this work are computed with the SacreBleu library and have signature BLEU+case.mixed+lang.<xxx>-yyy+numrefs.1 *smooth.exp+tok.<TOK+version.1.4.13, where <T0K> is zh for Chinese, and 13a for all other languages. MacroF\({}_{1}\) and MicroF\({}_{1}\) use the same tokenizer as Bleu. ChrF\({}_{1}\) is also obtained using SacreBleu and has signature chrf+lang.<xx>-zyy+numhers.6 *space.false *version.1.4.13. BLUERT scores are from the _base_ model of Sellam et al. (2020), which is fine-tuned on WMT Metrics ratings data from 2015-2018. The BLEURT model is retrieved from [https://storage.googleapis.com/bleurt-oss/bleurt-base-128.zip](https://storage.googleapis.com/bleurt-oss/bleurt-base-128.zip).

Macrof\({}_{1}\) and MicroF\({}_{1}\) are computed using our fork of SacreBleu as:

sacrebleu $REF -m macrof microf < $HYP.

## Appendix B Agreement with WMT Human Judgments

Tables 9, 10, and 11 provide \(\tau\) between MT metrics and human judgments on WMT Metrics task 2017-2019. *Bleu is based on pre-computed scores in WMT metrics package, whereas Bleu is based on our recalculation using SacreBleu. Values marked with "are not significant at \(\alpha=0.05\), and hence corresponding rows are excluded from the calculation of mean, median, and standard deviation.

Since MacroF\({}_{1}\) is the only metric that does not achieve statistical significance in the WMT 2019 EN-ZH setting, we carefully inspected it. Human scores for this setting are obtained without looking at the references by bilingual speakers (Ma et al., 2019), but the ZH references are found to have a large number of bracketed EN phrases, especially proper nouns that are rare types. When the text inside these brackets is not generated by an MT system, MacroF\({}_{1}\) naturally penalizes heavily due to the poor recall. Since other metrics assign lower importance to poor recall of such rare types, they achieve relatively better correlation to human scores than MacroF\({}_{1}\). However, since the \(\tau\) values for EN-ZH are relatively lower than the other language pairs, we conclude that poor correlation of MacroF\({}_{1}\) in EN-ZH is due to poor quality references. Some settings did not achieve statistical significance due to a smaller sample set as there were fewer MT systems submitted, e.g. 2017 CSEN.

## Appendix C UNMT and SNMT Models

The UNMT models follow XLM's standard architecture and are trained with 5 million monolingual sentences for each language using a vocabulary size of 60,000. We train SNMT models for EN\(\leftrightarrow\)DE and select models with the most similar (or a slightly lower) BLEU as their UNMT counterparts on newstest2019. The DE-EN model selected is trained with 1 million sentences of parallel data and a vocabulary size of 64,000, and the EN-DE model selected is trained with 250,000 sentences of parallel data and a vocabulary size of 48,000. For EN\(\leftrightarrow\)FR and EN\(\leftrightarrow\)RO, we select SNMT models from submitted systems to WMT shared tasks that have similar or slightly lower BLEU scores to corresponding UNMT models, based on _NewstsTest2014_ for EN\(\leftrightarrow\)FR and _NewstsTest2016_ for EN\(\leftrightarrow\)RO.

Figure 1, which is a visualization of MacroF\({}_{1}\) for SNMT and UNMT models, shows that UNMT is generally better than SNMT on frequent types, however, SNMT outperforms UNMT on the rest leading to a crossover point in MacroF\({}_{1}\) curves. Since MacroF\({}_{1}\) assigns relatively higher weights to infrequent types than in Bleu, SNMT gains higher MacroF\({}_{1}\) than UNMT while both have approximately the same Bleu, as reported in Table 6.

A complete comparison of UNMT vs SNMT in different languages is in Table 12. A manual analysis of the ten sentences with the largest magnitude favoritism according to MacroF\({}_{1}\) and Bleu in

\begin{table}
\begin{tabular}{l r r r r r}  & \multicolumn{2}{c}{*Bleu} & \multicolumn{2}{c}{Blev} & \multicolumn{2}{c}{MacroF\({}_{1}\)} & \multicolumn{2}{c}{MicroF\({}_{1}\)} & \multicolumn{2}{c}{ChrF\({}_{1}\)} \\ \hline \hline DE-CS & .855 & .745 & .964 & .917 & **.982** \\ DE-EN & .571 & .655 & .723 & .695 & **.742** \\ DE-FR & .782 & .881 & **.927** & .844 & .915 \\ EN-CS & .709 & **.954** & .927 & .927 & .908 \\ EN-DE & .540 & .752 & .741 & .773 & **.824** \\ EN-FI & .879 & .818 & .879 & .848 & **.923** \\ EN-GU & .709 & .709 & .600 & **.734** & .709 \\ EN-KK & .491 & .527 & **.685** & .636 & .661 \\ EN-LT & .879 & .848 & **.970** & .939 & .881 \\ EN-RU & .870 & .848 & **.939** & .879 & .930 \\ FI-EN & .788 & .809 & **.909** & .901 & .875 \\ FR-DE & **.822** & .733 & .733 & .764 & .815 \\ GU-EN & .782 & .709 & .855 & .891 & **.945** \\ KK-EN & **.891** & .844 & .796 & .844 & .881 \\ LT-EN & .818 & **.855** & .844 & **.855** & .833 \\ RU-EN & .692 & .729 & .714 & **.780** & .757 \\ ZH-En & .695 & .695 & **.752** & .676 & .715 \\ \hline Median & .782 & .752 & .844 & .844 & .875 \\ Mean & .751 & .771 & .821 & .818 & .841 \\ SD & .124 & .101 & .112 & .093 & .095 \\ \hline EN-ZH & **.606** & **.606** & \textasci{1}.424 & .595 & .594 \\ \hline Wins & 3 & 3 & 6 & 3 & 5 \\ \end{tabular}
\end{table}
Table 9: WMT19 Metrics task: Kendall’s \(\tau\) between metrics and human judgments.

 

the FR-EN and RO-EN test sets is in Table 13 and Table 14. The complete texts of these sentences, their reference translations, and the system translations (including DE-EN mentioned in Sec 4), are shown in Tables 15, 16, 17, 18, 19, and 20.

\begin{table}
\begin{tabular}{l r r r r r}  & *Bleu & Bleu & MacroF\({}_{1}\) & MicroF\({}_{1}\) & ChrF\({}_{1}\) \\ \hline \hline DE-EN & .828 & .845 & .917 & .883 & **.919** \\ EN-DE & .778 & .750 & **.850** & .783 & .848 \\ EN-ET & .868 & .868 & .934 & .906 & **.949** \\ EN-FI & .901 & .848 & .901 & .879 & **.945** \\ EN-RU & .889 & .889 & **.944** & .889 & .930 \\ EN-ZH & .736 & .729 & .685 & **.833** & .827 \\ ET-EN & .884 & .900 & .884 & .878 & **.904** \\ FI-EN & .944 & .944 & .889 & .915 & **.957** \\ RU-EN & .786 & .786 & **.929** & .857 & .869 \\ ZH-EN & .824 & **.872** & .738 & .780 & .820 \\ EN-CS & **.1000** & **.1000** & .949 & **.1000** & .949 \\ \hline Median & .868 & .868 & .901 & .879 & .919 \\ Mean & .858 & .857 & .875 & .873 & .902 \\ SD & .077 & .080 & .087 & .062 & .052 \\ \hline TR-EN & \textasci{{}^{\star}}\).200 & \textasci{{}^{\star}}\).738 & \textasci{{}^{\star}}\).400 & \textasci{{}^{\star}}\).316 & \textasci{{}^{\star}}\).632 \\ EN-TR & \textasci{{}^{\star}}\).571 & \textasci{{}^{\star}}\).400 & \textasci{{}^{\star}}\).837 & \textasci{{}^{\star}}\).571 & **.849** \\ CS-EN & \textasci{{}^{\star}}\).800 & \textasci{{}^{\star}}\).800 & \textasci{{}^{\star}}\).600 & \textasci{{}^{\star}}\).800 & \textasci{{}^{\star}}\).738 \\ \hline Wins & \multicolumn{1}{c}{\(1\)} & \multicolumn{1}{c}{\(2\)} & \multicolumn{1}{c}{\(3\)} & \multicolumn{1}{c}{\(2\)} & \multicolumn{1}{c}{\(6\)} \\ \end{tabular}
\end{table}
Table 10: WMT18 Metrics task: Kendall’s \(\tau\) between metrics and human judgments.

Figure 1: SNMT vs UNMT MacroF\({}_{1}\) on the most frequent 500 types. UNMT outperforms SNMT on frequent types that are weighed heavily by Bleu however, SNMT is generally better than UNMT on rare types; hence, SNMT has a higher MacroF\({}_{1}\).

\begin{table}
\begin{tabular}{l r r r r r}  & *Bleu & Bleu & MacroF\({}_{1}\) & MicroF\({}_{1}\) & ChrF\({}_{1}\) \\ \hline \hline DE-EN & .564 & .564 & .734 & .661 & **.744** \\ EN-CS & .758 & .751 & .767 & .758 & **.878** \\ EN-DE & .714 & **.767** & .562 & .593 & .720 \\ EN-FI & .667 & .697 & .769 & .718 & **.782** \\ EN-RU & .556 & .556 & **.778** & .648 & .669 \\ EN-ZH & **.911** & **.911** & .600 & .854 & .899 \\ LV-EN & **.905** & .714 & **.905** & **.905** & **.905** \\ RU-EN & .778 & .611 & .611 & .722 & **.800** \\ TR-EN & **.911** & .778 & .674 & .733 & .907 \\ ZH-EN & .758 & **.780** & .736 & .824 & .732 \\ \hline Median & .758 & .733 & .735 & .728 & .791 \\ Mean & .752 & .713 & .714 & .742 & .804 \\ SD & .132 & .110 & .103 & .097 & .088 \\ \hline F-EN & **.867** & **.867** & \textasci{{}^{\star}}\).733 & **.867** & **.867** \\ EN-NR & **.857** & .714 & \textasci{{}^{\star}}\).571 & .643 & .849 \\ CS-EN & \textasci{{}^{\star}}\).1000 & \textasci{{}^{\star}}\).1000 & \textasci{{}^{\star}}\).667 & \textasci{{}^{\star}}\).667 & \textasci{{}^{\star}}\).913 \\ \hline Wins & \multicolumn{1}{c}{\(5\)} & \multicolumn{1}{c}{\(4\)} & \multicolumn{1}{c}{\(2\)} & \multicolumn{1}{c}{\(6\)} \\ \end{tabular}
\end{table}
Table 11: WMT17 Metrics task: Kendall’s \(\tau\) between metrics and human judgments.

 



\begin{table}
\begin{tabular}{r l|l|c l} \(\delta_{\text{MacroF}_{1}}\) & Fav & Analysis & \(\delta_{\text{BLEU}}\) & Fav & Analysis \\ \hline \hline
044 S & S: synonym; U: _untranslation_, synonym & -.026 U & U: synonym; S: _omitted adv_, word order \\ -.038 U & U: no issues; S: synonym & .025 S & S: no issues; U: _determiner_, word order \\ -.035 S & S: synonym; U: _untranslation_, synonym & .024 S & S: no issues; U: _repetition_, form \\ -.034 U & U: no issues; S: synonym; word\_order & .021 S & S: _verb_, synonym; U: _untranslation_, _noun_, _time_, synonym \\ -.034 U & U: synonym; S: _word order_, verb\_ref & .021 S & S: synonym; U: synonym \\ -.033 U & U: no issues; S: synonym & -.021 U & U: _omitted NER_; S: synonym, word order \\ .033 S & S: word order; U: _ultranslation_, NER, word & -.021 U & U: _unitranslation_; S: _verb_, word order \\ .032 S & S: synonym; U: _number_, _omitted noun_, _unnallation_, _verb_ & -.021 U & U: synonym; S: _extra preposition_, synonym, word order \\ .030 S & S: _adj_; U: _untranslation_ & .021 S & S: no issues; U: _NER_ \\ .030 S & S: _noum_, synonym; U: _noun_, synonym & -.020 U & U: synonym; S: synonym, word order \\ \end{tabular}
\end{table}
Table 13: Analysis of the ten FR-EN test set segments with the most favoritism in SNMT (S) or UNMT (U), according to MacroF\({}_{1}\) (left) and Bleu (right). Fav is the favored system by metrics. Actual examples are shown in Appendix Tables 17 and 18.




* [10] C.-H. Lee, M.-J. Kim, and J.-P. Klauder, _The Journal of Chemical Physics: Condensed Matter_, vol. 32, no. 4, pp. 579-680, 1996.
* [11] A. G. MacKinnon, D. R. McKenzie, and S. P. McMillan, _Astronomy in Astrophysics_, vol. 3, p. 103, 1996.

[MISSING_PAGE_POST]

 


* [10] S. M. G. K. R. A. W. van der Zweig, and J. H. L. van der Zweig, "The New York: The New York: 



\begin{table}
\begin{tabular}{r l r c} \hline
044 & In me fallair qu en déployer les ac- & All it took was to highlight its mistakes and, in keeping with discline us, dieait confoine aux poffets et aus sousis-prefietis, interprettes au storises. \\ \hline  -035 & Les specialistes disent que les personnes sont systematiquesment contriaries à faireurs aueux, malger un changement dans la tui de a été plus tito dat dans’e name intersistant avactirties de facore giucnique à scientifier himéimir il nüme. & All it took was to highlight its mistakes and, in keeping with aboligies, was entrausted to the effects and sub-prefects, audiorized interpreters. \\ \hline  -038 & Les specialists dissent que les personnes sont systematiquesment contriaries à faireurs aueux, malger un changement dans la tui du e situ de vites plas from forcing anyone to criminate themselves. & Experts say professionals are systematically forced to make their confessions, despite a change in the law aptie a change in the law apti le a este pour ling finre in the year buning difficult paving the authorities for force or anyone to intiminating himself. \\ \hline  -035 & Its sort increases, I’m interseauxite faisant partie du groupe doi sorostamine dans dialogues comme des adolescent ou development esu, an umbrella term for those with atypical chromosomes or apadus (varies or testes) or abidants devrojoped genatilta. & They are interest, intersex, intersex forming part of the dog diseases diagnosed as discouragement, at muchlig temet for those with atypical chromosomes or atypical gpolem possessing chromosomes or organos (voters or insteless), or unusually developed genitalia. & They are interesc, intersex forming part of the dog disease diagnosed as discouragement per topferthology, democons of atypical gpolem possessing chromosomes or grands (ovaries or prostles), or unusually developing sexual organs. & They are intressc, intersex forming part of the dog diseases diagnosed as discouragement for IOP definition & They are intressc, intersex forming part of the dog of diseases diagnosed as discouragement for IOP definition & They are intressc, intersex forming part of the dog of diseases diagnosed as discouragement for IOP definition & They are intressc, intersex forming part of the dog of diseases diagnosed as discouragement for IOP definition & They are intressc, intersex forming part of the dog of diseases diagnosed as discouragement for IOP definition & They are intressc, intersex forming part of the dog of diseases diagnosed as discouragement for IOP definition & They are intressc, intersex forming part of the dog of diseases diagnosed as discouragement for IOP definition & They are intressc, intersex forming part of the dog of diseases diagnosed as discouragement for IOP definition & They are intressc, intersex forming part of the dog of diseases diagnosed as discouragement for IOP definition & They are intressc, intersex forming part of the dog of IOP definition



\begin{table}
\begin{tabular}{r l r}  & Reference \\ \hline
021 & Scott Brown, le capitaine du Celtic Glasgow, av v son appel reject et sera ben suspend pour les deux probains matches de Lieque des champions so con club, contre’t ÉA y i at IC Milan. & But Rep. Bill Shuster (R-Pa.), chairman of the House Transduction Committee, has said he, too, sees it as the most viable long-term alternative. \\ \hline
021 & Les irreductibles du M23, soit quequques centanes de combatants, etaient retranches a prese de 2000 mètes d'altitude sur les collines agricoles de Chanzu, Runyoni and Rhuzi, close to Bunangana and Johna, cloate about 80 km au ordo de Goma, la capitalité de la province du Nord-Kivin. & The idehedle m23, or who are several hundreds in hundred kilometers, were cut off ten nearly 2000 metres above sese as an attitude of almost 2,000 metres in the farmland Runyonyi et huliss of Chanzu, Runyoni and Muzni; close to Bunangana and Johna, cloate about 80 km au ordo de Goma, La capitalite de la product around 80km north of Göma, the capital of North Kivru province \\ \hline
021 & II a midquque que le nouveau tribunal des medias * sera voiquours partial car il’sagit ‘un prolongement du govevernment’ et que les restrictions relatives au contenent etau et la publicne inturial a la place du Kenya dans écéomomomie. \\ \hline
021 & Dans “Les Fous de Benghazi”, bi al avatête pe premier à révéler l’existence d’aun centre de commandenter in the city, the cette ville, bereca ou le raúvelé theybene. & In “Les Fous de Benghazi", he was the first to reveal the existence of a secret CIA commander in this town, eradble of the Libyan revolt. \\ \hline
022 & Le Sená amercain a approouve un projet pilotée de 90 MS T annnee dernüère qui aratiar porte sur envireno 10 000 coûures. & The first nine episodes of Sheriff Callies Wild West swept for available from November 24 on the site watchdiscneyju: minor.com or via its application for phones and tablets. \\ \hline
023 & Leo Spataît Emerson et al. [ENDFOOTNOTE]Footnote 024: Lé president Xi Jinping, who took office in March, has made the fight against corruption a national priority, believing that the phenomenon is a threat to the very existence of the Communist Party. \\ \hline
025 & Less neuf premiers episodes de Sheriff Callie’s Wild West swept for available from November 24 on the site watchdiscneyju: minor.com or via its application for phones and tablets. \\ \hline
026 & Un peu plus tot, sur la route menant à Bunganag, postfortriore avec ! U’onganda, with Uganda, soldiers assisted by civilomas loaded up a mulacherapeutic mount sur un campion demand neurk urchrode than heard new FARDC, to attendee va veron assurer la relieve d’un auture enginion pilonnant les positions du MO23 sur les cellines. \\ \hline
027 & Il V u, zue lac rération, une vi violence faire au corps aimé*, qui agamäe und the body of a loved doges” ent tens per te demps, et to a pile of ashes” in a very short time instead of after a presses of decomposition, which prosces of decomposition that “would accompany the phases of graif”. \\ \hline
028 & Cott Brown, le captiaine du Celtic Glasgow, av v son appel viglet et esra bin suspenndour les devaux pracohains macthedes de Ligue des champions so con club, contre ’T Ajalt et IC Milan. \\ \hline
029 & Les irductibles du M23, soit queques centanes de combatants, etaient retranches a preis de 2000 mètes of altitude sur les collines agricoles de Chanzu, Ruznyoni et holmiés of Chanzu, Ruznyoni and Muzni; close to Bunangana and Johna, two towns of Goma, the capital of Northwest Viv province. \\ \hline
029 & Ifla a midquque que le nouveau tribunal des medias * sera voiquours partical car il’sagit ‘un prolongement du governmentment du government,’ net les resrictions relatives au contenent etau et la publicne inturial a la place du Kenya dans écéomomie commodale. \\ \hline
021 & Dams “Les Fous de Benghazi”, he was the first to reveal the existence of a secret CIA commander in this town, crack of the Libyan revolt. \\ \hline
021 & This eres Rous de Benghazi ”, he was the first to reveal the existence of a secret CIA commander in this town, crack of the Libyan revolt. \\ \hline
0220 & Le Sénat amercain a approouve un projet pilotée de 90 MS T annnee dernüère qui aratiar porte sur envireno 10 000 coûures. \\ \hline \end{tabular}
\end{table}
Table 18: Top 10 segments by \(|\delta_{\mathrm{BLEU}}(i,h_{S},h_{U})|\) on FR-EN.




* [10] David Grimal are o'cariena internaturatura David Ofmai has an international quarter career as a solo violinist, dago sido volitnist, di Sandi carne is a sustiatbeit grateful convente in ultiman jede on e meir patient: what concerts on the main data? Special music of the world and animal si crosehes foreign us with prestigious orchestras such as the Ochsete Paris, chesteine de Paris, Cheschere Patris, Chesteine Pluhrammouque de Radio France, chierse. Ransitional Orchestra National Orchestra, Orleans Naticional Orchestrate National de Lyon, Chamber Cherry Orchstra of Europe, Berlin groupe, Bertiner Symphonauer, New Japan Philamtron, Chocire de Rayon, Catherine Gropperie the New Jap Blulem-Pohrzem, Deursen Salzburg, Jerusalem Synphony Orchestra at Sinfonia Varsu.
* [11] Christian Harrison was born on the American york, the University of Texas at Austin, 2006.
* [12] Michael Plasson, Michael Stinchman, and John Strohl, "Theoretical History of Science," _Journal of Physics: Statistical Mechanics and its Applications_, 39(4):578-690, 1991.
* [13] Peter Evolves, "A new approach to the development of the science and technology for the development of the scientific community," _Nature Reviews_, 100(1):1-2, 1994.
* [14] A. Good portion of the store as it gouver to the massive is devoted to giant rayon of free carriers, where you will find curry where you still find curry whereas, and frozen cirrhosis, whole fish, fash, whole fish, food and space stock, pod carcass, traditionalism-repregure ai commercialypersand foods and more.
* [15] William H. K. M. T. Jones, "Theoretical History of Science," _Proceedings of the Royal Society of London_, 19(1):1-2, 1974.

[MISSING_PAGE_POST]

 


\begin{table}
\begin{tabular}{r l p{142.3pt}} Source & Reference \\ \hline
117 Dwarf Grimal are a certain internam- & David Grunlän has an international scientist for violation solid, dea law- & Dusor Grülan has an international scientist for a solo voluntini. During the per capita as a solution similar people carest in salmi don't bei principal interest of experts on the main classical issues: some me meticlassicia is plain with stages of the world with prostigious orchestra such as Orchestrates who can diversize enclosures which are Ochsier de Paris, Orchèe Philharmonicismenge chetique Paris, Orchèe Philharmonicismenge chetique Paris, Orchèe Philharmonicismenge chetique Paris, Orchèe Rinzel Natihanmanmanman Raioden Unedense, Mexischen Meisnandi and Kachigroze, Tauskie Natural of Christian, Orchèe Shuland Grußer, Menske Universite National of Japan, Chamber Orchestra Company, Lyno Darwin Chuckers and Pierre Strombach, Berlin Sympowrichner, New Japa Phammaher, Obershire of Depart Schrifpacher, The Royal Institute of Defense at Optica, Ed. to Toppe A, Lyon, Marzern Ore-Grieves Zwartan Corporation. Dr Dr. Joseph, Safaray Mezzurean Corporation. Dr. The Salberg Maximum Overflow, The Standing Markering Operators, the Stefan Sysmio University, the Sherman and Stefonia Varoevna university, via subdoctorne der directionof coudest sudvan cum Christoph Echekhot, Michel et al. Chris Eisenbechhold, Michael Kuchlierecht, and Sirwin Magalewski. Dr. Alfred P. Clifford and Richard W. L. Taylor, "A new scientific report on the development of a research unit at the University of California, Berkeley, CA, USA, Helsinki Schiffler, Lawrence Cable, Heinrich Schlurrer, Lucian Schmidt Krittwatt, Mikhail Scustra Institut, Keutzer, Hannamster Krivine, Mishinal Patters, Patrice Rafell Fwheblede de Burgie, Peteros, Radif Frublebeck de Burgie, Peteros, Andrew Nichson, and Peter Fortsov, Andris Nelson, Christian Arming. & David Grunlän has an international videintinist, Pieter Cathu, Heinrich Schiffler, Alexander Procty Research, Edmund Kramer, Michail Scustra Institut, Keutzer, Hannamster Krivine, Mishinal Patters, Patrice Rafell Fwheblede de Burgie, Peteros, Andreis Neesey, and Peter Grosse, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris, and Peter Fortsov, Andris Nelson, and Peter Fortsov, Andris, and Peter Fortsov, Andris, and Peter Fortsov, Andris, and Peter Fortsov, Andris, and Peter Fortsov, Andris, and Peter Fortsov, Andris, and Peter Fortsov, Andris, and Peter Forts, Andris, and Peter Fortsov, Andris, and Peter Forts, Andris, and