SlowGuess/NAACL_2021_OCR huggingface dataset:
Thamme Gowda
Information Sciences Institute
tg@isi.edu
Weiqiu You
Dept of Computer and Information Science
weiqiuy@seas.upenn.edu
Constantine Lignos
Michtom School of Computer Science
lignos@brandeis.edu
Jonathan May
jonmay@isi.edu
While traditional corpus-level evaluation metrics for machine translation (MT) correlate well with fluency, they struggle to reflect adequacy. Model-based MT metrics trained on segment-level human judgments have emerged as an attractive replacement due to strong correlation results. These models, however, require potentially expensive re-training for new domains and languages. Furthermore, their decisions are inherently non-transparent and appear to reflect unwelcome biases. We explore the simple type-based classifier metric,  $\mathrm{MACROF}_1$ , and study its applicability to MT evaluation. We find that  $\mathrm{MACROF}_1$  is competitive on direct assessment, and outperforms others in indicating downstream cross-lingual information retrieval task performance. Further, we show that  $\mathrm{MACROF}_1$  can be used to effectively compare supervised and unsupervised neural machine translation, and reveal significant qualitative differences in the methods' outputs. $^1$
Model-based metrics for evaluating machine translation such as BLEURT (Sellam et al., 2020), ESIM (Mathur et al., 2019), and YiSi (Lo, 2019) have recently attracted attention due to their superior correlation with human judgments (Ma et al., 2019). However, BLEU (Papineni et al., 2002) remains the most widely used corpus-level MT metric. It correlates reasonably well with human judgments, and moreover is easy to understand and cheap to calculate, requiring only reference translations in the target language. By contrast, model-based metrics require tuning on thousands of examples of human evaluation for every new target language or domain
(Sellam et al., 2020). Model-based metric scores are also opaque and can hide undesirable biases, as can be seen in Table 1.
Reference: You must be a doctor.
Hypothesis: must be a doctor.
He -0.735
Joe -0.975
Sue -1.043
She -1.100
Reference: It is the greatest country in the world.
Hypothesis: is the greatest country in the world.
France -0.022
America -0.060
Russia -0.161
Canada -0.309
Table 1: A demonstration of BLEURT's internal biases; model-free metrics like BLEU would consider each of the errors above to be equally wrong.
The source of model-based metrics' (e.g. BLEURT) correlative superiority over model-free metrics (e.g. BLEU) appears to be the former's ability to focus evaluation on adequacy, while the latter are overly focused on fluency. BLEU and most other generation metrics consider each output token equally. Since natural language is dominated by a few high-count types, an MT model that concentrates on getting its ifs, ands and buts right will benefit from BLEU in the long run more than one that gets its xylophones, peripatetics, and defen-estrates right. Can we derive a metric with the discriminating power of BLEURT that does not share its bias or expense and is as interpretable as BLEU?
As it turns out, the metric may already exist and be in common use. Information extraction and other areas concerned with classification have long used both micro averaging, which treats each token equally, and macro averaging, which instead treats each type equally, when evaluating. The latter in particular is useful when seeking to avoid results dominated by overly frequent types. In this
1Tools and analysis are available at https://github. com/thammegowda/007-mt-eval-macro.MT evaluation metrics are at https://github.com/isi-nlp/sacrebleu/tree/ macroavg-naacl21.
Association for Computational Linguistics: Human Language Technologies, pages 1138-1157
work we take a classification-based approach to evaluating machine translation in order to obtain an easy-to-calculate metric that focuses on adequacy as much as BLEURT but does not have the expensive overhead, opacity, or bias of model-based methods.
Our contributions are as follows: We consider MT as a classification task, and thus admit  $\mathrm{MACROF_1}$  as a legitimate approach to evaluation (Section 2). We show that  $\mathrm{MACROF_1}$  is competitive with other popular methods at tracking human judgments in translation (Section 3.2). We offer an additional justification of  $\mathrm{MACROF_1}$  as a performance indicator on adequacy-focused downstream tasks such as cross-lingual information retrieval (Section 3.3). Finally, we demonstrate that  $\mathrm{MACROF_1}$  is just as good as the expensive BLEURT at discriminating between structurally different MT approaches in a way BLEU cannot, especially regarding the adequacy of generated text, and provide a novel approach to qualitative analysis of the effect of metrics choice on quantitative evaluation (Section 4).
2 NMT as Classification
Neural machine translation (NMT) models are often viewed as pairs of encoder-decoder networks. Viewing NMT as such is useful in practice for implementation; however, such a view is inadequate for theoretical analysis. Gowda and May (2020) provide a high-level view of NMT as two fundamental ML components: an autoregressor and a classifier. Specifically, NMT is viewed as a multi-class classifier that operates on representations from an autoregressor. We may thus consider classifier-based evaluation metrics.
Consider a test corpus,  $T = \{(x^{(i)}, h^{(i)}, y^{(i)}) | i = 1, 2, 3 \dots m\}$  where  $x^{(i)}, h^{(i)}$ , and  $y^{(i)}$  are source, system hypothesis, and reference translation, respectively. Let  $x = \{x^{(i)} \forall i\}$  and similar for  $h$  and  $y$ . Let  $V_h, V_y, V_{h \cap y}$ , and  $V$  be the vocabulary of  $h$ , the vocabulary of  $y$ ,  $V_h \cap V_y$ , and  $V_h \cup V_y$ , respectively. For each class  $c \in V$ ,
$$
\begin{array}{l} \operatorname {P R E D S} (c) = \sum_ {i = 1} ^ {m} C (c, h ^ {(i)}) \\ \operatorname {R E F S} (c) = \sum_ {i = 1} ^ {m} C (c, y ^ {(i)}) \\ \operatorname {M A T C H} (c) = \sum_ {i = 1} ^ {m} \min  \left\{C \left(c, h ^ {(i)}\right), C \left(c, y ^ {(i)}\right) \right\} \\ \end{array}
$$
where  $C(c,a)$  counts the number of tokens of type  $c$  in sequence  $a$  (Papineni et al., 2002). For each class  $c \in V_{h \cap y}$ , precision  $(P_c)$ , recall  $(R_c)$ , and  $F_\beta$  measure  $(F_{\beta; c})$  are computed as follows:
$$
P _ {c} = \frac {\operatorname {M A T C H} (c)}{\operatorname {P R E D S} (c)}; \quad R _ {c} = \frac {\operatorname {M A T C H} (c)}{\operatorname {R E F S} (c)}
$$
$$
F _ {\beta ; c} = \left(1 + \beta^ {2}\right) \frac {P _ {c} \times R _ {c}}{\beta^ {2} \times P _ {c} + R _ {c}}
$$
The macro-average consolidates individual performance by averaging by type, while the micro-average averages by token:
$$
\mathrm {M A C R O F} _ {\beta} = \frac {\sum_ {c \in V} F _ {\beta ; c}}{| V |}
$$
$$
\mathrm {M I C R O F} _ {\beta} = \frac {\sum_ {c \in V} f (c) \times F _ {\beta ; c}}{\sum_ {c ^ {\prime} \in V} f \left(c ^ {\prime}\right)}
$$
where  $f(c) = \mathrm{REFS}(c) + k$  for smoothing factor  $k$ .<sup>3</sup> We scale  $\mathrm{MACROF}_{\beta}$  and  $\mathrm{MICROF}_{\beta}$  values to percentile, similar to BLEU, for the sake of easier readability.
3 Justification for  $\mathbf{MACROF}_1$
In the following sections, we verify and justify the utility of  $\mathrm{MACROF}_1$  while also offering a comparison with popular alternatives such as  $\mathrm{MICROF}_1$ , BLEU,  $\mathrm{CHRF}_1$ , and BLEURT. We use Kendall's rank correlation coefficient,  $\tau$ , to compute the association between metrics and human judgments. Correlations with p-values smaller than  $\alpha = 0.05$  are considered to be statistically significant.
3.1 Data-to-Text: WebNLG
We use the 2017 WebNLG Challenge dataset (Gardent et al., 2017; Shimorina, 2018) to analyze the differences between micro- and macro-averaging. WebNLG is a task of generating English text for sets of triples extracted from DBPedia. Human annotations are available for a sample of 223 records each from nine NLG systems. The human
2We consider  $F_{\beta ;c}$  for  $c\notin V_{h\cap \gamma}$  to be 0.
3We use  $k = 1$  .When  $k\to \infty$  ,MICROF  $\beta_{\cdot}\rightarrow$  MACROF.
$^{4}$ BLEU and  $\mathrm{CHRF}_{1}$  scores reported in this work are computed with SACREBLEU; see the Appendix for details. BLEURT scores are from the base model (Sellam et al., 2020). We consider two varieties of averaging to obtain a corpus-level metric from the segment-level BLEURT: mean and median of segment-level scores per corpus.
5https://gitlab.com/webnlg/webnlg-human-evaluation
Name	Fluency & Grammar	Semantics
BLEU	.444	.500
CHRF1	.278	.778
MACROF1	.222	.722
MICROF1	.333	.611
BLEURTmean	.444	.833
BLEURTmedian	.611	.667
Table 2: WebNLG data-to-text task: Kendall's  $\tau$  between system-level MT metric scores and human judgments. Fluency and grammar are correlated identically by all metrics. Values that are not significant at  $\alpha = 0.05$  are indicated by  $\times$ .
judgments provided have three linguistic aspects—fluency, grammar, and semantics<sup>6</sup>—which enable us to perform a fine grained analysis of our metrics. We compute Kendall's  $\tau$  between metrics and human judgments, which are reported in Table 2.
As seen in Table 2, the metrics exhibit much variance in agreements with human judgments. For instance, BLEURTmedian is the best indicator of fluency and grammar, however BLEURTmean is best on semantics. BLEURT, being a model-based measure that is directly trained on human judgments, scores relatively higher than others. Considering the model-free metrics,  $\mathrm{CHRF}_1$  does well on semantics but poorly on fluency and grammar compared to BLEU. Not surprisingly, both  $\mathrm{MICROF_1}$  and  $\mathrm{MACROF_1}$ , which rely solely on unigrams, are poor indicators of fluency and grammar compared to BLEU, however  $\mathrm{MACROF_1}$  is clearly a better indicator of semantics than BLEU. The discrepancy between  $\mathrm{MICROF_1}$  and  $\mathrm{MACROF_1}$  regarding their agreement with fluency, grammar, and semantics is expected: micro-averaging pays more attention to function words (as they are frequent types) that contribute to fluency and grammar whereas macroaveraging pays relatively more attention to the content words that contribute to semantic adequacy.
The take away from this analysis is as follows:  $\mathrm{MACROF}_1$  is a strong indicator of semantic adequacy, however, it is a poor indicator of fluency. We recommend using either  $\mathrm{MACROF}_1$  or  $\mathrm{CHRF}_1$  when semantic adequacy and not fluency is a desired goal.
3.2 Machine Translation: WMT Metrics
In this section, we verify how well the metrics agree with human judgments using Workshop on Machine Translation (WMT) metrics task datasets for 2017-2019 (Bojar et al., 2017; Ma et al., 2018,
2019).<sup>7</sup> We first compute scores from each MT metric, and then calculate the correlation  $\tau$  with human judgments.
As there are many language pairs and translation directions in each year, we report only the mean and median of  $\tau$ , and number of wins per metric for each year in Table 3. We have excluded BLEURT from comparison in this section since the BLEURT models are fine-tuned on the same datasets on which we are evaluating the other methods.  $^8\mathrm{CHRF}_1$  has the strongest mean and median agreement with human judgments across the years. In 2018 and 2019, both  $\mathrm{MACROF_1}$  and  $\mathrm{MICROF_1}$  mean and median agreements outperform BLEU whereas in 2017 BLEU was better than  $\mathrm{MACROF_1}$  and  $\mathrm{MICROF_1}$ .
As seen in Section 3.1,  $\mathrm{MACROF_1}$  weighs towards semantics whereas  $\mathrm{MICROF_1}$  and BLEU weigh towards fluency and grammar. This indicates that recent MT systems are mostly fluent, and adequacy is the key discriminating factor amongst them. BLEU served well in the early era of statistical MT when fluency was a harder objective. Recent advancements in neural MT models such as Transformers (Vaswani et al., 2017) produce fluent outputs, and have brought us to an era where semantic adequacy is the focus.
3.3 Cross-Lingual Information Retrieval
In this section, we determine correlation between MT metrics and downstream cross-lingual information retrieval (CLIR) tasks. CLIR is a kind of information retrieval (IR) task in which documents in one language are retrieved given queries in another (Grefenstette, 2012). A practical solution to CLIR is to translate source documents into the query language using an MT model, then use a monolingual IR system to match queries with translated documents. Correlation between MT and IR metrics is accomplished in the following steps:
1. Build a set of MT models and measure their performance using MT metrics.
2. Using each MT model in the set, translate all source documents to the target language, build an IR model, and measure IR performance on translated documents.
3. For each MT metric, find the correlation between the set of MT scores and their corresponding set of IR scores. The MT metric that has a
<sup>6</sup>Fluency and grammar, which are elicited with nearly identical directions (Gardent et al., 2017), are identically correlated.
$^{7}$ http://www.statmt.org/wmt19/metrics-task.html
<sup>8</sup>https://github.com/google-research/bleurt
Year	Pairs	★BLEU	BLEU	MACROF1	MICROF1	CHRF1
2019	18	Mean	.751	.771	.821	.818	.841
Median	.782	.752	.844	.844	.875
Wins	3	3	6	3	5
2018	14	Mean	.858	.857	.875	.873	.902
Median	.868	.868	.901	.879	.919
Wins	1	2	3	2	6
2017	13	Mean	.752	.713	.714	.742	.804
Median	.758	.733	.735	.728	.791
Wins	5	4	2	2	6
Table 3: WMT 2017–19 Metrics task: Mean and median Kendall's  $\tau$  between MT metrics and human judgments. Correlations that are not significant at  $\alpha = 0.05$  are excluded from the calculation of mean, and median, and wins. See Appendix Tables 9, 10, and 11 for full details.  $\star$ BLEU is pre-computed scores available in the metrics packages. In 2018 and 2019, both  $\mathrm{MACROF}_1$  and  $\mathrm{MICROF}_1$  outperform BLEU,  $\mathrm{MACROF}_1$  outperforms  $\mathrm{MICROF}_1$ .  $\mathrm{CHRF}_1$  has strongest mean and median agreements across the years. Judging based on the number of wins,  $\mathrm{MACROF}_1$  has steady progress over the years, and outperforms others in 2019.
stronger correlation with the IR metric(s) is more useful than the ones with weaker correlations.
4. Repeat the above steps on many languages to verify the generalizability of findings.
An essential resource of this analysis is a dataset with human annotations for computing MT and IR performances. We conduct experiments on two datasets: firstly, on data from the 2020 workshop on Cross-Language Search and Summarization of Text and Speech (CLSSTS) (Zavorin et al., 2020), and secondly, on data originally from Europarl, prepared by Lignos et al. (2019) (Europarl).
3.3.1 CLSSTS Datasets
CLSSTS datasets contain queries in English (EN), and documents in many source languages along with their human translations, as well as query-document relevance judgments. We use three source languages: Lithuanian (LT), Pashto (PS), and Bulgarian (BG). The performance of this CLIR task is evaluated using two IR measures: Actual Query Weighted Value (AOWV) and Mean Average Precision (MAP). AOWV<sup>9</sup> is derived from Actual Term Weighted Value (ATWV) metric (Wegmann et al., 2013).
We use a single CLIR system (Boschee et al., 2019) with the same IR settings for all MT models in the set, and measure Kendall's  $\tau$  between MT and IR measures. The results, in Table 4, show that  $\mathrm{MACROF}_1$  is the strongest indicator of CLIR downstream task performance in five out of six settings. AQWV and MAP have a similar trend in agreement to the MT metrics.  $\mathrm{CHRF}_1$  and BLEURT, which are strong contenders when generated text is directly evaluated by humans, do not indicate
CLIR task performance as well as  $\mathrm{MACROF}_1$ , as CLIR tasks require faithful meaning equivalence across the language boundary, and human translators can mistake fluent output for proper translations (Callison-Burch et al., 2007).
3.3.2 Europarl Datasets
We perform a similar analysis to Section 3.3.1 but on another cross-lingual task set up by Lignos et al. (2019) for Czech  $\rightarrow$  English (CS-EN) and German  $\rightarrow$  English (DE-EN), using publicly available data from the Europarl v7 corpus (Koehn, 2005). This task differs from the CLSSTS task (Section 3.3.1) in several ways. Firstly, MT metrics are computed on test sets from the news domain, whereas IR metrics are from the Europarl domain. The domains are thus intentionally mismatched between MT and IR tests. Secondly, since there are no queries specifically created for the Europarl domain, GOV2 TREC topics 701-850 are used as domain-relevant English queries. And lastly, since there are no query-document relevance human judgments for the chosen query and document sets, the documents retrieved by BM25 (Jones et al., 2000) on the English set for each query are treated as relevant documents for computing the performance of the CS-EN and DE-EN CLIR setup. As a result, IR metrics that rely on boolean query-document relevance judgments as ground truth are less informative, and we use Rank-Based Overlap (RBO;  $p = 0.98$ ) (Webber et al., 2010) as our IR metric.
We perform our analysis on the same experiments as Lignos et al. (2019).<sup>10</sup> NMT models for CS-EN and DE-EN translation are trained using a convolutional NMT architecture (Gehring
<sup>9</sup>https://www.nist.gov/system/files/documents/2017/10/26/aqwv.derivation.pdf
10 https://github.com/ConstantineLignos/ mt-clir-emnlp-2019
Domain	IR Score	BLEU	\( MACROF_1 \)	\( MICROF_1\)	\(CHRF_1\)	BLEURTmean	BLEURTmedian
LT-EN	In	AQWV	.429	\(^\times .363\)	.508	\(^\times .385\)	.451	.420
MAP	.495	.429	.575	.451	.473	.486
In+Ext	AQWV	\(^\times .345\)	.527	.491	.491	.491	.477
MAP	\(^\times .273\)	\(^\times .455\)	\(^\times .418\)	\(^\times .418\)	\(^\times .418\)	\(^\times .404\)
PS-EN	In	AQWV	.559	.653	.574	.581	.584	.581
MAP	.493	.632	.487	.494	.558	.554
In+Ext	AQWV	.589	.682	.593	.583	.581	.571
MAP	.519	.637	.523	.482	.536	.526
BG-EN	In	AQWV	\(^\times .455\)	.550	.527	\(^\times .382\)	\(^\times .418\)	.418
MAP	.491	.661	.564	.491	.527	.527
In+ext	AQWV	\(^\times .257\)	.500	\(^\times .330\)	\(^\times .404\)	\(^\times .367\)	\(^\times .367\)
MAP	\(^\times .183\)	\(^\times .426\)	\(^\times .257\)	\(^\times .330\)	\(^\times .294\)	\(^\times .294\)
Table 4: CLSSTS CLIR task: Kendall's  $\tau$  between IR and MT metrics under study. The rows with Domain=In are where MT and IR scores are computed on the same set of documents, whereas Domain=In+Ext are where IR scores are computed on a larger set of documents that is a superset of segments on which MT scores are computed. Bold values are the best correlations achieved in a row-wise setting; values with  $\times$  are not significant at  $\alpha = 0.05$ .
BLEU	MACROF1	MICROF1	CHRF1	BT	BT
CS-EN	.850	.867	.850	.850	.900	.867
DE-EN	.900	.900	.900	.912	.917	.900
Table 5: Europarl CLIR task: Kendall's  $\tau$  between MT metrics and RBO.  $\overline{\mathrm{BT}}$  and  $\widetilde{\mathrm{BT}}$  are short for BLEURTmean and BLEURTmedian. All correlations are significant at  $\alpha = 0.05$ .
et al., 2017) implemented in the FAIRSeq (Ott et al., 2019) toolkit. For each of CS-EN and DE-EN, a total of 16 NMT models that are based on different quantities of training data and BPE hyperparameter values are used. The results in Table 5 show that BLEURT has the highest correlation in both cases. Apart from the trained BLEURTmedian metric,  $\mathrm{MACROF_1}$  scores higher than the others on CS-EN, and is competitive on CS-EN.  $\mathrm{MACROF_1}$  is not the metric with highest IR task correlation in this setting, unlike in Section 3.3.1, however it is competitive with BLEU and  $\mathrm{CHRF_1}$ , and thus a safe choice as a downstream task performance indicator.
4 Spotting Qualitative Differences Between Supervised and Unsupervised NMT with  $\mathbf{MACROF}_1$
Unsupervised neural machine translation (UNMT) systems trained on massive monolingual data without parallel corpora have made significant progress recently (Artetxe et al., 2018; Lample et al., 2018a,b; Conneau and Lample, 2019; Song et al., 2019; Liu et al., 2020). In some cases, UNMT yields a BLEU score that is comparable with strong $^{11}$  supervised neural machine transla
tion (SNMT) systems. In this section we leverage  $\mathrm{MACROF}_1$  to investigate differences in the translations from UNMT and SNMT systems that have similar BLEU.
We compare UNMT and SNMT for English  $\leftrightarrow$  German (EN-DE, DE-EN), English  $\leftrightarrow$  French (EN-FR, FR-EN), and English  $\leftrightarrow$  Romanian (EN-RO, RO-EN). All our UNMT models are based on XLM (Conneau and Lample, 2019), pretrained by Yang (2020). We choose SNMT models with similar BLEU on common test sets by either selecting from systems submitted to previous WMT News Translation shared tasks (Bojar et al., 2014, 2016) or by building such systems. Specific SNMT models chosen are in the Appendix (Table 12).
Table 6 shows performance for these three language pairs using a variety of metrics. Despite comparable scores in BLEU and only minor differences in  $\mathrm{MICROF_1}$  and  $\mathrm{CHRF_1}$ , SNMT models have consistently higher  $\mathrm{MACROF_1}$  and BLEURT than the UNMT models for all six translation directions.
In the following section, we use a pairwise maximum difference discriminator approach to compare corpus-level metrics BLEU and  $\mathrm{MACROF}_1$  on a segment level. Qualitatively, we take a closer look at the behavior of the two metrics when comparing a translation with altered meaning to a translation with differing word choices using the metric.
<sup>12</sup>We were unable to find EN-DE and DE-EN systems with comparable BLEU in WMT submissions so we built standard Transformer-base (Vaswani et al., 2017) models for these using appropriate quantity of training data to reach the desired BLEU performance. We report EN-RO results with diacritic removed to match the output of UNMT.
though not, generally, the strongest
BLEU	MACROF1	MICROF1	CHRF1	BLEURTmean	BLEURTmedian
SN	UN	Δ	SN	UN	Δ	SN	UN	Δ	SN	UN	Δ	SN	UN	Δ	SN	UN	Δ
DE-EN	32.7	33.9	-1.2	38.5	33.6	4.9	58.7	57.9	0.8	59.9	58.0	1.9	.211	-.026	.24	.285	.067	.22
EN-DE	24.0	24.0	0.0	24.0	23.5	0.5	47.7	48.1	-0.4	53.3	52.0	1.3	-.134	-.204	.07	-.112	-.197	.09
FR-EN	31.1	31.2	-0.1	41.6	33.6	8.0	60.5	58.3	2.2	59.1	57.3	1.8	.182	.066	.17	.243	.154	.09
EN-FR	25.6	27.1	-1.5	31.9	27.3	4.6	53.0	52.3	0.7	56.0	57.7	-1.7	.104	.042	.06	.096	.063	.03
RO-EN	30.8	29.6	1.2	40.3	33.0	7.3	59.8	56.5	3.3	58.0	54.7	3.3	.004	-.058	.06	.045	-.004	.04
EN-RO	31.2	31.0	0.2	34.6	31.0	3.6	55.4	53.4	2.0	59.3	56.7	2.6	.030	-.046	.08	.027	-.038	.07
Table 6: For each language direction, UNMT (UN) models have similar BLEU to SNMT (SN) models, and  $\mathrm{CHRF}_1$  and  $\mathrm{MICROF}_1$  have small differences. However,  $\mathrm{MACROF}_1$  scores differ significantly, consistently in favor of SNMT. Both corpus-level interpretations of BLEURT support the trend reflected by  $\mathrm{MACROF}_1$ , but the value differences are difficult to interpret.
δMACROF1	Fav	Analysis	δBLEU	Fav	Analysis
.071	S	S: synonym; U: untranslation, noun	.048	S	S: word order; U: word order, untranslation, ending
.064	S	S: synonym; U: untranslation	.046	S	S: spelling variation; U: synonym, word order, punctuation
-.055	U	U: no issues; S: untranslation	.044	S	S: extra determiner; U: paraphrase, synonym, number, untranslation
.052	S	S: synonym; U: untranslation, noun	.042	S	S: synonym; U: synonym, punctuation, extra adverb
-.045	U	U: no issues; S: untranslation	-.039	U	U: no issues; S: noun, verb
.044	S	S: synonym, word order; U: subject, truncation, word order	-.037	U	U: no issues; S: punctuation
.044	S	S: synonym, tense; U: untranslation	-.034	U	U: no issues; S: symbol
.043	S	S: inflection, word order; U: number	-.032	U	U: no issues; S: adjective, noun
-.041	U	U: adjective, verb; S: omitted verb, untranslation	-.032	U	U: untranslation; S: tense, word order, meaning, active/passive voice
.041	S	S: time, word order; U: time, nouns	-.031	U	U: untranslation; S: word order, synonym, extra_conj