Dialect-robust Evaluation of Generated Text
Year: 2023

Abstract:
Text generation metrics that are not robust to dialect variation make it impossible to tell how well systems perform for many groups of users, and can even penalize systems for producing text in lower-resource dialects. In this paper, we introduce a suite of methods to assess whether metrics are dialect robust. These methods show that state-of-the-art metrics are not dialect robust: they often prioritize dialect similarity over semantics, preferring outputs that are semantically incorrect over outputs that match the semantics of the reference but contain dialect differences. As a step towards dialect-robust metrics for text generation, we propose NANO, which introduces regional and language information to the metric’s pretraining. NANO significantly improves dialect robustness while preserving the correlation between automated metrics and human ratings. It also enables a more ambitious approach to evaluation, dialect awareness, in which system outputs are scored by both semantic match to the reference and appropriateness in any specified dialect.

heading: 1 Introduction
text: Most natural language generation (NLG) evaluation metrics compare a system output against a human-written reference. References are usually drawn from a relatively narrow range of linguistic styles. They often exclude varieties like Indian English or Iberian Portuguese, which are geographical dialects with millions of speakers. As a result, outputs in dialects that are not represented in the reference may score poorly, discouraging the development of systems to meet the needs of these language communities. Although contemporary metrics such as COMET (Rei et al., 2020) can be reference-free, they still rely on training data and rater pools that do not cover all dialects of interest, leading to a high number of out-of-domain dialects.
∗Work done while Jiao and Tu interning at Google.
The performance of evaluation metrics on these out-of-domain dialects has not been quantified.
We define a dialect-robust evaluation metric as one that produces the same score for system outputs that share the same semantics, but are expressed in different dialects. To understand whether current evaluation metrics are dialect-robust, we propose to quantify the dialect robustness at the dialect feature-level and sentence-level. The analyses measure the dialect-sensitivity of evaluation metrics by comparing semantics-preserving dialect edits to perturbations that change the meaning of sentences.
Through our analyses, we demonstrate that multiple state-of-the-art NLG evaluation metrics are not robust to dialects of Mandarin, English, and Portuguese. In many cases, system outputs that are perturbed so as to differ semantically from the reference score higher than outputs in which the only change is to the dialect. With the goal of increasing the dialect robustness and without performance degradation on standard benchmarks, we propose a training schema NANO. NANO is an unsupervised pretraining step to a metric that distills dialect information of the multilingual pretraining dataset into a model, which we demonstrate leads to improved dialect robustness.
Based on our findings, we lay out research goals toward dialect-inclusive metrics. Moving beyond dialect robustness, we formalize the goal of dialect awareness, in which metrics can be applied to any user-specified language and dialect regardless of the language of the reference or source document.

heading: 2 Dialect Robustness and Awareness
text: Dialects can be regarded as linguistic subdivisions that align with communities of speakers, often grouped by geographical or demographic attributes (Chambers et al., 1998). A classic example is nation-level varieties, such as Brazilian and Iberian Portuguese. Dialects are distinguished from each other by a set of dialect features, which can op-
6010
edits (highlighted in yellow) should not lead to a greater degradation of score than edits that change the underlying semantics (highlighted in underline). BLEURT-20 in the figure assigns higher score to semantically-perturbed sentences than sentences with dialect features, exposing its vulnerability to dialects.
erate at the levels of pronunciation, lexicon, rhetorical devices, and grammar (Whiteman, 2013); one working definition of dialect is as a set of correlated features (Nerbonne, 2009).
Two examples of dialect features are shown in Figure 1. The left side shows the English dialect feature “focus only”, which distinguishes Indian English from other varieties, such as US English (Lange, 2012). The feature changes the surface form but not the underlying semantics. The right panel of Figure 1 shows the Portuguese dialect feature of different lexical choice for the same semantics (“breakfast”), which distinguishes Iberian Portuguese from Brazilian Portuguese. Many dialect features are acceptable in multiple dialects: for example, zero definite article (“∅ main reason is . . . ”)1 is used in Indian English, Singapore English, and several other post-colonial dialects.
Dialect Robustness Consider a translation system that produces Iberian Portuguese outputs. If all the training data for the metric used to evaluate generation quality comes from Brazilian Portuguese, it will likely assign a lower score to Iberian Portuguese outputs, thereby misrepresenting system quality and disincentivizing further development of the more diverse system in favor of one that only produces Brazilian Portuguese. To formalize this intuition, we define dialect robustness in the context of NLG evaluation as:
Definition 2.1 (Dialect robustness). Let y(d) and y(d
′) be two system outputs that are semantically equivalent but written in different dialects. An evaluation metric m : Y → R is dialect robust iff m(y(d)) = m(y(d ′)) for all such (y(d), y(d ′)).2
1https://ewave-atlas.org/parameters/ 62#2/7.0/7.9
2For simplicity we do not include the reference in this definition. A corpus-level reference-based metric could be defined as 1
N ∑ i mi(yi) with mi(yi) = δ(yi, ri), with ri
indicating the reference for example i and δ : Y × Y → R.
This definition is strict: it would not apply to any system that produced even small differences in score between semantically equivalent, regionally distinct outputs. For that reason, we propose a relaxed criterion, which compares the change in the metric induced by dialect to changes induced by semantic perturbations:
Definition 2.2 (ϕ-Dialect robustness). Let y(d) and y(d
′) be two semantically-equivalent system outputs that differ in dialect. Let ϕ : Y → Y∗ be a semantic perturbation function that maps an input to a set of outputs whose semantics are different from the input. An evaluation metric m : Y → R is ϕ-dialect robust if m(y(d), y(d
′)) > m(y(d), ỹ) for all semantically-equivalent (y(d), y(d
′)) and all ỹ ∈ ϕ(y(d)). Dialect Awareness Consider a translation system that is supposed to translate into Brazilian Portuguese but instead produces Iberian Portuguese. In this case, a dialect-robust metric is undesirable because it is unable to detect this mistake. To account for these cases, we define dialect awareness:
Definition 2.3 (Dialect-awareness). Let T be a set of dialect tags. A metric m : Y × T → R is dialect aware iff m(y(d), d) ≥ m(y(d′), d) for all semantically-equivalent input pairs (y(d), y(d
′)) where y(d) is in dialect d ∈ T and y(d′) is in dialect d′ ̸= d. Informally, a metric is dialect aware if, given a dialect identifier and a pair of semanticallyequivalent texts that vary by dialect, it assigns the highest score to the text in the dialect specified by the identifier. Dialect awareness is undefined with respect to inputs that are not semantically equivalent. This means that the definition is agnostic as
Similarly, a corpus-level quality estimation metric could be defined with mi(yi) = δ(yi, xi) with xi indicating the input, such as the source language or passage to be summarized. For the corpus-level metric to be dialect robust (or ϕ-robust), all mi must be dialect robust (or ϕ-robust).
to whether the metric should prioritize matching the target semantics or the target dialect.
Figure 1 illustrates the concepts of dialect robustness and dialect awareness. The top two rows of each panel vary only by dialect; the bottom row shows semantic perturbations of the top row. ϕdialect robustness implies that the top row is scored as more similar to the middle row than to the bottom row. Dialect awareness implies that the quality of the surface form in each row should be highest when paired with the correct dialect label.
Is Semantic Equivalence Realistic? The above definitions presume that it is possible to characterize utterances in different dialects as semantically equivalent. Such characterizations have been criticized as lacking a strong foundation for semantic equivalence, outside the limited case in which the dialect differences are purely phonological (Lavandera, 1978; Romaine, 1981). One such criticism is that a pair of utterances might be semantically equivalent for some communicative purposes, but not for others. To avoid the gray area between dialect differences that change semantics and those that do not, we design perturbations that have a small surface-level impact on the original utterance but a strong effect on its meaning, e.g. by negating the main proposition or changing an important semantic argument. This establishes a necessary condition for dialect robustness: if a metric scores such perturbations more highly than dialect pairs, then it is certainly not dialect robust. Proving that a metric is dialect robust is more difficult, because it requires constructing more subtle semantic perturbations that are harder to distinguish (even conceptually) from dialect variables. Furthermore, from a practical standpoint we cannot evaluate y(d) with respect to all semantic perturbations ỹ ∈ ϕ(y(d)), but the existence of perturbations for which m(y(d), ỹ) > m(y(d), y(d
′)) is enough to disprove dialect robustness.

heading: 3 Existing Metrics
text: To assess the quality of a generated text, most automatic evaluation approaches compare it to a “ground truth” reference, with higher similarity to the reference implying higher-quality output (Celikyilmaz et al., 2020). Similarity can be based on lexical features, as in BLEU (Papineni et al., 2002) and CHRF (Popović, 2015), or distributed repre-
sentations, as in BLEURT (Sellam et al., 2020),3 PRISM (Rei et al., 2020) and YISI (Lo, 2019). When distributed representations are used, they may be unsupervised (Zhang et al., 2020) or finetuned on a corpus of human ratings. In addition to these similarity-based metrics, there are also reference-free metrics for quality estimation (e.g., COMET-EQ; Rei et al., 2020), which we discuss in §5.2. Existing distributed metrics either use the multilingual representation from pretrained models, or create multilingual training data through various augmentation strategies. However, none of them explicitly accounts for dialectal variation.

heading: 4 Testing Dialect Robustness
text: In this section, we describe our methodology for assessing dialect robustness. We first introduce two ways to perturb sentences to get two comparable metrics’ outputs and then describe the statistical tests we use to aggregate the outputs over a corpus.

heading: 4.1 Micro-level Dialect Features
text: Dialect features are local edits that distinguish dialects while avoiding changes to the meaning of the text, as described in §2. Our first robustness assessment uses such features. We start with a base sentence y(base)i taken from a corpus of sentences D. We further assume access to a version of the same sentence in which a dialect feature was introduced, denoted y(dialect)i . Following Definition 2.2, we introduce a semantic perturbation that changes y(base)i to y(perturb)i . Again using English as an example, from the U.S. English base sentence “as recently as April. . . ”, we may produce the Indian English version “recently only in April. . . ” (using the feature focus-only), and the semantic perturbation “as recently as May. . . ”.
Let m(yi, yj) be a metric function that takes a candidate sentence yi and a reference yj as input, and produces a score σ. Given the above defined variations of yi, we define the dialect and perturbed scores as
σ (dialect) m,i = m(y (dialect) i , y (base) i ) (1) σ (perturb) m,i = m(y (perturb) i , y (base) i ). (2)
To satisfy Definition 2.2, σ(dialect)m,i should score higher than σ(perturbation)m,i across the sentences in
3In practice, we use the latest BLEURT-20 (Pu et al., 2021), following the authors’ recommendation in https: //github.com/google-research/bleurt.
the corpus. This implies as a necessary condition that Ei∼D[σ (dialect) m,i ] > Ei∼D[σ (perturb) m,i ].
We consider three perturbation types: deletion, replacement and insertion. Each perturbation aims to change the sentence by only a single word or phrase, so as to induce a strong semantic change with a minimal impact to the surface form. Such perturbations are expected to yield challenging but clear comparisons against dialect variation. There are no standard techniques for introducing semantic perturbations, so we apply few-shot learning by prompting LaMDA (Cohen et al., 2022). For each perturbation type, we provide five exemplars (see Appendix A) and then prompt LaMDA for automatic semantic perturbation given a sentence y (en-base) i . Some sentences are not amenable to all perturbations — for example, some are too short to support deletion — so we choose one perturbation per sentence, with the preference order of replacement, insertion and then deletion.

heading: 4.2 Sentence-level Dialect Rewrites
text: Micro-level dialect features require significant linguistic expertise to identify and have been defined for only a few languages. We thus introduce a less granular method that is based on parallel human translations. Given an English base sentence ENi, we obtain human translations y(j)i and y (k) i in dialects j and k of the target language, e.g., Brazilian and Iberian Portuguese. We can again use the metric m to score the pair, σ(dialect)m,i = m(y (j) i , y (k) i ).
Because we have access to the English base sentence, we can use machine translation to generate a sentence in the target language ENi ==⇒
MT ŷ (j∗) i which we can compare to, yield-
ing σ(MT)m,i = m(y (j) i , ŷ (j∗) i ). Here, j ∗ indicates the locale that we believe is most strongly targeted by the machine translation system (“pt-BR” for Portuguese, “zh-CN” for Mandarin).
Finally, we construct target language perturbations by first perturbing the English source ENi ⇒ ẼNi and then automatically translating the perturbed sentence ẼNi ⇒ ỹ(j∗), yielding σ (perturb) m,i = m(y (j) i , ỹ (j∗) i ). The perturbations are produced by prompting LaMDA with the same exemplars as in §4.1.4
4While it is possible directly perturb the sentences in the target language, using the same English validated few-shot setup scales to more languages at the cost of a more Englishcentric perturbation style.
We expect E[σ(MT)m ] > E[σ(perturb)m ], because both involve machine translation while the latter also involves perturbation to the source. If we have E[σ(perturb)m ] > E[σ(dialect)m ] then metric m strongly disprefers dialect variants, even in favor of inputs that are different in meaning due to the perturbation of the English source.

heading: 4.3 Statistical Methods
text: As a necessary condition for dialect robustness, we test whether the expected scores for dialect rewrites exceed the expected scores for semantic perturbations. A challenge in correctly characterizing the uncertainty of these comparisons is that there is a substantial amount of variance over the original examples. We handle this with two styles of analysis: Mixed-effect Regression For metric m, example i, and condition j ∈ {perturb, dialect, MT}, we model the metric σ(j)m,i via a mixed-effects regression (Baayen, 2012; Speelman et al., 2018),
σ (j) i = θi + ϕj + ϵi,j , (3)
with the subscript m implicit in each term. The first term θi is a random intercept associated with example i, which helps to address the variance across examples; ϕj , the parameter of interest, is a fixed effect associated with the condition j; ϵi,j is a Gaussian error. Because all methods and conditions are applied to all examples, the predictors are uncorrelated. This makes it possible to interpret ϕm,j as an estimate of the expected change in the metric value corresponding to the application of metric m in condition j. By including the θi term, the regression is conceptually equivalent to a pairwise comparison, in the sense that the regression also benefits from the additional power obtained by controlling for per-example variation. Win/loss Analysis and Binomial Test For a coarse-grained evaluation that is more easily comparable across metrics, we count how often each condition j receives a higher score than condition k in a pairwise comparison. When j represents dialect rewrites and k represents semantic perturbations, a high win rate indicates that the metric is more likely to be dialect robust. To measure statistical significance, we apply a one-tailed binomial test, which computes the likelihood of achieving at least n wins on T trials given a null hypothesis win probability 12 . In words, we test against the null hypothesis that for each example, a dialect rewrite
and a semantic perturbation are equally likely to get the higher score.
As discussed in the next section, we perform multiple comparisons per metric, across different conditions and different languages. To adjust the p-values for multiple comparisons, we apply the Bonferroni correction (Dror et al., 2017).

heading: 5 NANO
text: We hypothesize that explicitly encoding dialect information while pretraining a model will lead to improved downstream robustness. To test this hypothesis on learned metrics for text generation, we introduce NANO,5 a model-agnostic pretraining schema with the goal of improving dialect robustness without performance degradation on downstream metric benchmarks.

heading: 5.1 Acceptability Pretraining
text: Given a pretrained model, we add a second pretraining phase to distill dialect information into the model. Specifically, we define the NANO-task as: given an expression y, determine whether it is from a text that has been identified as written in language ℓ and/or dialect region d (e.g., en-IN). Data To construct a training corpus for NANO, we process mC4 (Xue et al., 2021). We split the corpus into sentences and use a Language Identification (LangID) model (Zhang et al., 2018) by Botha et al. (2017) to identify the language and locale information for the sentences.6 Besides LangID output, mC4 provides the URL where a sentence originated from which we extract the region information as an indication of geographic dialect. For Portuguese and Mandarin, we filter an instance if the predicted locale does not agree with the region information from the URL. For other languages, we combine the LangID and region information as a noisy approximation for a dialect of the language in the specific region. For example, if the LangID model predicts that the language is English and the region in the URL indicates India (.in), we treat the instance as en-IN.7 In total, we include ten languages with metric finetuning data evaluated
5The name is motivated by the dialect feature “invariant tag (‘isn’t it’, ‘no’, ‘na’)” (Lange, 2012).
6We use a more current model that is capable of identifying the locale for Mandarin and Portuguese.
7This is an approximation because many dialects do not align with national borders. The development of a datagathering approach for subnational and transnational dialects is an important topic for future work.
during the WMT benchmark with ninety-five language variants following the classification by van Esch et al. (2022).8
Given a sentence, we balance the ratio of sampling a dialect or language tag using a parameter λ. For instance, a sentence with gold dialect tag “ptBR” can be a positive instance for the dialect itself or the general language “pt-any”. At the same time, it can also be a negative instance for other dialect (e.g., “en-IN”) or language (“en-any”). The ratio of positive instances versus negatives instances is always 0.5. For more discussion, see Appendix E. Modeling We use mT5 (Xue et al., 2021) as our base model because the model is pretrained on the mC4 dataset, matching with our corpus choice and ensuring tokenizer compatibility. During pretraining, we transform each sentence into the string candidate: {sentence} language: {language_tag}, where the language_tag can be the dialect or the general language tag. The target label is zero or one, indicating whether the sentence belongs to the language tag. We adapt the encoder-decoder architecture of mT5 for regression by taking the logits of the first decoded token and applying the RMSE loss function between the logits and the label during model training. For more details about training, please see Appendix D.

heading: 5.2 Finetuning
text: Following Pu et al. (2021), we use the test data from the WMT 2015-2019 shared tasks as training data and use the WMT shared task 2020 as test data. There are three possible model specifications: (1) quantifying the semantic similarity between candidate and reference within the same reference, as in BLEURT (Pu et al., 2021) and YISI (Lo, 2019); (2) measuring the similarity between candidate and a cross-lingual reference, as in COMET (Rei et al., 2020); (3) reference-free quality estimation, also performed by COMET. To compare to all models, we finetune on all three settings, using the input formats in Appendix subsection F.1.

heading: 6 Experiments
text: In this section, we demonstrate that existing metrics are not dialect robust by applying our proposed methods and statistical tests to existing corpora in English, Portuguese, and Mandarin (§6.1). We
8Appendix B provides the full list of WMT language variants, which does not cover Portuguese. Our reported results on PT shows NANO’s capability in a zero-shot setting.
show that language-aware pretraining improves the dialect robustness and leads to promising preliminary steps toward dialect-aware metrics (§6.4).
Datasets As described in §4, we consider microlevel and sentence-level dialect rewrites. The micro-level rewrites are based on pairwise data from Demszky et al. (2021), in which each example includes a form containing at least one dialect feature from Indian English and a corresponding “base sentence” in U.S. English. We then apply the semantic perturbation to the base sentence as described in §4.1. For each perturbation type, one of the coauthors manually examined whether the perturbation successfully changes the meaning of the sentence. If all of the three perturbations fail, we exclude the instance from analysis.9
For sentence-level dialect analysis, we use the test set of the FRMT benchmark (Riley et al., 2022). Each instance contains an English sentence and its translations into dialects of the target languages Portuguese and Mandarin. For Portuguese, the two dialects are Brazilian Portuguese (pt-BR) and European Portuguese (pt-PT); for Mandarin, we consider mainland Mandarin and Taiwanese Mandarin, both in simplified script. As described in §4.2, semantic perturbations are obtained by perturbing the English sentences and then translating, using the Google Translate API. Table 8 (Appendix B) shows the number of evaluation examples.

heading: 6.1 Dialect robustness
text: We use the statistical methods reported in §4.3 to test metrics’ sensitivity to dialects.
9For the sentences that have multiple dialect rewritings, we treat each one as an individual data point. When multiple semantic perturbations can be applied, we choose a single one, preferring replacements, then insertions, and then deletions.
Regression Following Equation 3, we use σ (perturb) m,i , σ (dialect) m,i , σ (MT) m,i as conditions and model each metric as a mixed-effects regression. For a dialect-robust metric, we expect ϕdialect > ϕperturb, indicating that dialect rewrites score more highly than semantic perturbations, as required by definition 2.2. The difference ϕdialect − ϕperturb is shown in the Y -axis of Figure 2. We also evaluate ϕMT − ϕperturb as a stress test to measure metrics’ abilities to recognize semantic changes, and to ensure that the semantic perturbations are effective. For all metrics except BLEURT and NANO, ϕdialect − ϕperturb is negative for at least one language, indicating that these metrics are not dialect robust even in the average case. At the same time, all evaluation metrics can distinguish the MT and PERTURB conditions, showing that the issue is specific to dialect and not generally applicable to other paraphrases. Table 2 shows the coefficients before and after using NANO, which improves dialect robustness across all model sizes and languages.
Success Rates In Table 1 we report the success rates of a metric in assigning higher scores to dialect rewrites than to semantic perturbations. BLEURT performs better than other existing evaluation metrics which consistently fail to rank the dialect change above the perturbations. However, no metric correctly ranks the English examples at better than a random chance win rate (0.5), and even BLEURT as the most robust metric only has a 0.59 win rate for PT and ZH. In comparison with BLEURT, NANO achieves a higher win rate when scaled to XL and XXL, marked with in Table 1. The same trend can be observed in the regression analysis, where NANO’s coefficients are positive for all metrics and languages. However, the marginal benefit of NANO over standard fine-
tuning diminishes at scale—while NANO leads to significant improvements at XL scale, it has only a minor effect on the XXL model.

heading: 6.2 Align with Human Judgments
text: Does dialect robustness come at the cost of sacrificing the metrics’ performance on standard benchmark of evaluation metrics? To study this, we evaluate on the test set of WMT 2020. We calculate the segment-level agreement with human ratings and report DaRR (Mathur et al., 2020), a robust variant of Kendall Tau. We follow Pu et al. (2021) and omit ∗-en results because of inconsistencies between benchmark implementations.
Results Table 3 and Table 12 (Appendix F.4) show the performance of existing methods and NANO on WMT 2020 test sets for within the same language and quality estimation settings respectively. In both settings, adding NANO improves the WMT benchmark performance of the mT5XL model compared to the finetuning-only setup. As in the dialect robustness tests, NANO does not help much for the model size XXL and achieves comparable results to finetuning-only settings. Our results are on par with or exceed those of prior metrics, demonstrating that dialect robustness is not in tension with other measures of metric quality.

heading: 6.3 Transfer to Quality Estimation
text: Quality Estimation While we have been focusing the cross-dialect setting within the same language, all the statistical methods can be applied to the cross-language cross-dialect setting, and training with NANO can serve as a quality estimation of the translation quality. Similar to §4.2, given an English base sentence ENi and its translation to two locales (j and k) of a target language. We have
σjm,i = m(ENi, y (j) i ) (4)
σkm,i = m(ENi, y (k) i ). (5)
For a system that produces equally-good quality translations that are in different dialects j and k, we expect E[σjm] ≈ E[σkm] > E[σperturbm ] for a metric that is robust to dialect variations. For the quality estimation, we can also use one dialect (k) as reference and evaluate other conditions (e.g., perturb, MT, dialect j) against dialect k as candidates for evaluation. We can use all statistical methods in §4.3 to understand the difference of outputs from evaluation metrics. Experiment Setup We use the datasets for sentence-level dialect features for the quality estimation with and without references experiments. For quality estimation, we take the English sentences as the source and candidate from each of four conditions: two human-written dialects of target language (e.g., pt-BR), translated outputs to
target language from English and semantic perturbation as the input for the quality estimation. The translated outputs are from the Google Translate API. If a metric is robust to dialects, we expect E[σdialectm ] ≥ E[σMTm ] > E[σperturbm ]. For quality estimation with reference, we keep the same setting as the quality estimation but use one of the two dialects (“zh-CN” for Mandarin and “pt-BR” for Portuguese) as reference. We then use {perturb, MT, the other dialect} as candidates and estimate their quality with regard to the selected dialects. Results We show that success rates of σ(dialect) > σ(perturb) in QE with and without references in Table 4. We show that 1) training with NANO outperforms existing metrics on dialect robustness for both Portuguese and Mandarin; 2) NANO is important to improve dialect robustness with a smaller model size (i.e., mT5XL in our case). The trends are consistent with our findings for the within-language evaluation. Figure 3 shows the coefficients from the regression model and confirms the dialect robustness after training with NANO by assigning higher scores to dialects than semantic perturbations.

heading: 6.4 Dialect Awareness
text: Following Definition 2.3, we test whether it is possible to build metrics that reward outputs in a desired dialect. Because existing metrics do not train
with dialect identifiers, we are only able to test NANO’s dialect awareness, which can serve as a baseline for future works. We use the Mandarin dataset for sentence-level dialect rewrites for our experiments of dialect awareness, because Mandarin is covered during the pretraining of NANO.10 We then score each dialect rewrite against its translation from the English sentence, written as, σjm,i = m(tag, y (MT) i , y (j) i ). The models we use are the ones we trained for dialect robustness tests in Table 1, but we provide specific dialect tags (e.g., zh-CN for Mainland Mandarin) instead of the general language tags (e.g., zh-any) as inputs for inference. During the model inference, we either provide tags that agree or disagree with the candidate sentences. For example, for a candidate sentence in Taiwanese Mandarin, we run inference with both “zh-CN” and “zh-TW”. A dialect-aware metric should assign higher scores for the input with the correct dialect tag.
Results Table 5 shows the results of dialect awareness of NANO. NANOXL assigns higher
10We provide the zero-shot result of dialect awareness of NANO on PT in Appendix G.
scores to the candidates with the correct dialect tag, compared to the finetuning-only setup (−NANOXL). However, at the XXL scale the picture is more mixed: NANOXXL successfully assigns higher scores for zh-CN inputs with zh-CN tag over the zh-TW tag, but it fails on zh-TW inputs. This is compatible with our mixed findings on the impact of NANO on dialect robustness at the XXL scale.

heading: 7 Related Work
text: Most popular NLP datasets and evaluation metrics do not take dialectal variation into consideration. For example, machine translation systems are usually evaluated by whether they match references in the target language, for which the dialect is generally unspecified (Gehrmann et al., 2022). The subcommunity that has attended most to dialect is the VarDial series of workshops, which has featured shared tasks such as dialect classification (Zampieri et al., 2014), translation between dialects (Akhbardeh et al., 2021), and transfer of NLP systems across dialects (Zampieri et al., 2017). Of this prior work, dialect classification is clearly relevant to the criterion of dialect awareness introduced in Definition 2.3 (see also Nerbonne et al., 2011), but our goal is to reward system outputs that match a target dialect rather than to classify the dialect of existing human-written texts. A related line of work has focused on inducing dialect features from corpora (Eisenstein et al., 2010; Jørgensen et al., 2015; Dunn, 2021) and on recognizing dialect features in text (Demszky et al., 2021; Masis et al., 2022). Following the feature-based view of dialect, we use cross-dialectal minimal pairs to measure dialect robustness in §4.1.
On the more specific topic of dialect-aware evaluation, classic approaches focused on the creation of dialect-specific test sets, e.g. for translation to and from Arabic dialects (e.g., Zbib et al., 2012). This idea has been extended to modern multi-task natural language understanding benchmarks by the VALUE project (Ziems et al., 2022), which used transformation rules to convert the GLUE benchmarks (Wang et al., 2018) into African-American English. Our evaluation in §4.2 builds on the FRMT dataset of multi-dialectal translations (Riley et al., 2022) to evaluate metrics for dialect robustness. However, in many cases it is not feasible to produce multi-dialect references or test sets. In these cases, dialect-robust and dialect-aware metrics can provide a more efficient solution, particu-
larly if these capabilities can be achieved through a pretraining step like NANO, which can be transferred to multiple tasks and evaluation settings.
Our work is additionally motivated by several papers that demonstrate the social impact of the failure to consider dialect variation in language technology. For example, literature shows that the out-of-the-box POS taggers (Jørgensen et al., 2015) and language identification and dependency parsing tools (Blodgett et al., 2016) perform poorly on AAVE texts. Other work has demonstrated large racial disparities in the performance of commercial speech recognition systems (DiChristofano et al., 2022; Koenecke et al., 2020). Our results contribute to this line of work by showing that metrics for text generation tend to penalize dialectal variants. We view the design of dialect-robust and dialect-aware metrics like NANO as a step towards making language technology that works more broadly across dialects.

heading: 8 Conclusion and Future Work
text: We introduce and formalize the dialect robustness and dialect awareness in the context of generation evaluation. Grounded by a suite of statistical tests, we find that existing evaluation methods are not robust to dialects. As a first step toward a solution to this problem, we propose NANO as a pretraining strategy. Our experiments demonstrate that NANO offers a size-efficient way to improve both the dialect robustness, shows the preliminary success towards dialect awareness and improves the metric performance of metrics on WMT benchmark.
Due to the limited availability of dialect-parallel corpora, our robustness tests are conducted in thousands of examples for Mandarin and Portuguese and hundreds of examples for English, which is insufficient to capture the full extent of these languages. We encourage future work to develop more resources, including benchmarks and corpora to conduct research on dialects for NLG evaluation. Due to this limitation, our work focuses on dialect robustness and only briefly evaluates dialect awareness. Future works may extend the details and criteria of the dialect-aware NLG evaluation, and we hope our work can serve as a baseline in this new research direction. Our encouraging preliminary results lead us to urge researchers to consider and improve the dialect diversity during pretraining.
Limitations
Besides the limited size of the evaluation corpora and a brevity of the exploration of dialect awareness that we point out as limitations in §8, we again acknowledge the data acquisition strategy as another limitation of our work. Our data acquisition of dialects requires country codes, which exclude many dialects. There is some work on getting dialectal data without country codes: Blodgett et al. (2016) build a dataset of tweets that are likely to include a high density of African-American English by linking geolocated Twitter data with demographic data from the U.S. census. However, this approach is limited to dialects that have strong geographic associations within the United States and which correlate with census demographics like race. Similarly, Abdul-Mageed et al. (2018) build a dataset of city-level Arabic dialects, again relying on Twitter geolocation. An alternative approach that does not rely on geolocation is to translate existing corpora into multiple dialects (e.g., Faisal et al., 2021; Ziems et al., 2022). However, this is labor intensive and therefore difficult to scale up to the amount of data needed for pretraining. We leave to future work the question of how to build largescale corpora for dialects that do not align with easily-identifiable geographical indicators such as national boundaries.

heading: A Example Semantic Perturbations
text: Table 6 shows the task instruction and examples we used to prompt LaMDA for the automatic semantic perturbation on English sentences, for both micro-level (§4.1) and sentence-level (§4.2) studies. During decoding, we use greedy decoding.

heading: B Languages and Variants
text: Table 7 shows the language codes and region codes that we cover during NANO pretraining. We cover 10 WMT languages and 95 language variants, presented as BCP language codes. Although iu is one of the WMT languages, it is not supported by LangID model that we are using and we thus do not include it in our pretraining. Portuguese (PT) is not included because it is not a WMT language. Therefore, all NANO dialect robustness results on PT are fully through zero-shot transfer. We report additional experiments that include Portuguese during pretraining in Appendix E. Our experiments show that pretraining with all languages leads to better dialect robustness on both PT and ZH.

heading: C Metric Implementations
text: We use the official releases of Prism (Thompson and Post, 2020), COMET (Rei et al., 2020) and BLEURT (Pu et al., 2021) in our work. For YiSi, we use an internal implementation. Table 9 presents the supported setups for each model in their latest released versions. Although all metrics could in theory be adapted to different use cases, their existing capabilities restrict the experiments we can run with them. For BLEURT,11 we use the latest checkpoint BLEURT-20. We use Prism12 (m39v1 checkpoint) for quality estimation with and without references. Lastly, there are two models that we use for COMET13. Model wmt21-comet-qe-mqm is for referencefree quality estimation and wmt20-comet-da for reference-based quality estimation. For our experiments, if a language is not supported by the model, we exclude it from the results.

heading: D Training Details and Hyperparameters of NANO
text: Hyperparameters We implement NANO using T5X and SeqIO (Roberts et al., 2022).
11https://github.com/google-research/ bleurt.
12https://github.com/thompsonb/prism. 13https://github.com/Unbabel/COMET
We experimented with the following hyperparemeters during training: learning rate of {1e-3, 1e-4, 1e-5, 3e-5, 5e-5} × sequence length of {512, 1024}. The reported results are based on a learning rate of 1e − 4 and sequence length of 1024. We train for 200,000 steps for pretraining and another 20,000 steps for finetuning. We set the drop out rate as 0.1 and optimizer warm up steps as 10,000. We train with a batch size of 128. Choosing Checkpoints We calculate the Kendall-Tau correlation on the development set every 1000 steps throughout training and choose the checkpoint with the highest correlation as the final checkpoint for evaluation. Compute Time Our models are trained on 64 TPUs, pretraining step normally takes one day to finish across different sizes. While mT5small can be trained within a single day, finetuning mT5XL and mT5XXL takes three and nine days respectively to reach 20,000 steps, but the models converge before they finish training.

heading: E NANO Design Choices
text: Table 10 shows different variations of NANO and their performances. We studied:
• Comparing pretraining on all WMT language variants to only prertaining on zh/pt or zh/pt/en.
• Comparing λ = 1 to λ = 0 and λ = 2, i.e., the balance in pretraining between dialect-tags and language-tags.
• Variations of the ratio of positive vs. negative instances during pretraining. We compare a balanced set to a setup where we have twice as many positive as negative examples.
We gain the following insights: 1) using all WMT languages for pretraining performs better than using partial data; 2) An equal balance between dialect-tags and general language tags (λ = 1) during pretraining improves upon a higher fraction of dialect-tags (λ = 2). However, using only data with general language tags (λ = 0) surprisingly leads to an even better dialect-robustness, although the model will lose its potential for dialect-awareness since it never sees dialect tags; 3) A balanced set of positive and negative instances during pretraining is better than oversampling positive instances.
Following Equation 3, we use σ(perturb)m,i , σ (dialect) m,i , σ (MT) m,i as conditions and model each metric as a mixed-effects regression. Table 11 shows ϕdialect
on English sentences. We include three types of semantic perturbation: replace (highlighted in yellow), delete (highlighted in blue) and insert (highlighted in purple).
with its standard errors against the ϕperturb condition. Take ϕperturb for BLEURT under EN as an example, -0.09 with an error smaller than 0.05 means that semantic perturbation would result in a decrease of 0.09 point for BLEURT compared to the dialect condition, and the result is significant. For a dialect-robust metric, we expect its ϕperturb to be positive. However, this is not always true during our observations. BLEURT performs the best among existing evaluation metrics and all other existing metrics have positive ϕperturb for at least one language of our test data. This indicates that existing evaluation metrics wrongly assign a higher score to semantically-perturbed rewriting than the dialects in at least one of the three languages, suggesting that they should not be used to assess dialects they were not trained for.
F Versatility of NANO
F.1 Input Format We use the following input format to adapt NANO to different use cases.
• For within-language assessment, we format the input as candidate: {sentence} reference: {reference} language: {language_tag}.
• For quality estimation without reference, we format the input as candidate: {sentence} source: {source} language: {language_tag}.
• For quality estimation with reference, we format the input as candidate: {sentence} reference: {reference} source: {source} language: {language_tag}.
The {language_tag} during fine-tuning indicates the language where the candidate sentence comes from, but it is the general language tag (e.g., “enany”) and does not contain the dialect information. We finetune one model for each setting.
F.2 Dialect Robustness We show additional results of coefficients from the regression model across XL and XXL sizes in Figure 2, which shows that training with NANO improves the dialect robustness across both sizes and for all languages. In addition, we compare three pretraining settings: 1) Mandarin and Portuguese only; 2) Mandarin, Portuguese and selected English dialects and 3) ten languages with metric finetuning data evaluated during the WMT benchmark with ninety-five language variants following the classification by van Esch et al. (2022). Table 10 shows the exact numbers for both coefficients and success rates. NANO performs the best with the
full set of languages. NANO improves the success rates under the XL size, but reach comparable results with training without NANO under the XXL size. We suspect the discrepancy between getting a higher coefficients but having nearly the same success rates is because some big increase of score after applying NANO which does not influence the success rates.
F.3 Transfer on Reference-based QE
For a system that produces equally-good quality translations that are in different dialects j and k, we expect E[σjm] ≈ E[σkm] > E[σperturbm ] for a metric that is robust to dialect variations. Quality Estimation with Reference For the quality estimation, we can also use one dialect (k)
as reference and evaluate other conditions (e.g., perturb, MT, dialect j) against dialect k as candidates for evaluation, written as:
σjm,i = m(ENi, y (j) i , y (k) i ) (6)
σ perturb m,i = m(ENi, y (perturb) i , y (k) i ). (7)
For a metric that is robust to dialect variations, we expect E[σjm] > E[σperturbm ]. The candidate can also be y(MT)i . We can use all statistical methods in §4.3 to understand the difference in outputs from evaluation metrics.
We report NANO’s performance on dialect robustness as the reference-based quality estimation in Table 13 and its corresponding WMT performance in Table 14. In the XL setting, NANO improves upon both COMET and the finetuning only setup for the dialect robustness and perfor-
mance on WMT benchmark. However, NANO achieves comparable performances with finetuningonly setting with XXL models. The findings are consistent with our findings for within-language and reference-free quality estimation settings in the main content: NANO provides a size-efficient way for models to improve the dialect robustness and their performance on the WMT metrics benchmark.
F.4 Performance on WMT Tasks
We have shown that NANO is more robust to dialects. Is the robustness at the cost of sacrificing the metrics’ performance on standard benchmark of evaluation metrics? To study this, we evaluate on the test set of WMT 2020.
Metrics We calculate the segment-level agreement with human ratings and report DaRR (Mathur et al., 2020), a robust variant of Kendall Tau. We follow Pu et al. (2021) and omit ∗-en results because of inconsistencies between benchmark implementations.
Results Table 3 and Table 12 show the performance of existing methods and NANO on WMT 2020 test sets for within the same language and quality estimation settings respectively. In both settings, adding NANO improves mT5XL model’s performance on WMT benchmark tasks compared to the finetuning-only setup. As in the dialect robustness tests, NANO does not help much for the model size XXL and achieves comparable results to finetuning-only settings. Moreover, our results are on par with or exceed those of prior metrics, demonstrating that mT5 is an effective base model for developing new metrics.

heading: G Dialect Awareness on PT
text: Table 15 shows the dialect awareness test of NANO on Portuguese. As Portuguese and its language variants are not covered in pretraining, we expect NANO to not perform well in terms of dialect awareness because it has never seen the input dialect tags during training. Table 15 confirms our expectation. We observe that both finetuning-only and pretraining with NANO fail to assign higher scores to candidates with matched input language tags over mismatched dialect tags.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Left blank.
3 A2. Did you discuss any potential risks of your work? Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 5.1
3 B1. Did you cite the creators of artifacts you used? Left blank.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We encourage future work to develop more resources, including benchmarks and corpora to conduct research on dialects for NLG evaluation. Due to this limitation, our work focuses on dialect robustness and only briefly evaluates dialect awareness. Future works may extend the details and criteria of the dialect-aware NLG evaluation, and we hope our work can serve as a baseline in this new research direction. Our encouraging preliminary results lead us to urge researchers to consider and improve the dialect diversity during pretraining. Limitations
Besides the limited size of the evaluation corpora and a brevity of the exploration of dialect awareness that we point out as limitations in §8, we again acknowledge the data acquisition strategy as another limitation of our work. Our data acquisition of dialects requires country codes, which exclude many dialects. There is some work on getting dialectal data without country codes: Blodgett et al. (2016) build a dataset of tweets that are likely to include a high density of African-American English by linking geolocated Twitter data with demographic data from the U. S. census. However, this approach is limited to dialects that have strong geographic associations within the United States and which correlate with census demographics like race. Similarly, Abdul-Mageed et al. (2018) build a dataset of city-level Arabic dialects, again relying on Twitter geolocation. An alternative approach that does not rely on geolocation is to translate existing corpora into multiple dialects (e. g. , Faisal et al. , 2021; Ziems et al. , 2022). However, this is labor intensive and therefore difficult to scale up to the amount of data needed for pretraining. We leave to future work the question of how to build largescale corpora for dialects that do not align with easily-identifiable geographical indicators such as national boundaries