SlowGuess/NAACL_2021_OCR huggingface dataset:
Dorottya Demszky $^{1*}$  Devyani Sharma $^{2}$  Jonathan H. Clark $^{3}$
Vinodkumar Prabhakaran<sup>3</sup> Jacob Eisenstein<sup>3</sup>
$^{1}$ Stanford Linguistics  $^{2}$ Queen Mary University of London  $^{3}$ Google Research
d.sharma@qmul.ac.uk
{jhclark,vinodkpg,jeisenstein}@google.com
Building NLP systems that serve everyone requires accounting for dialect differences. But dialects are not monolithic entities: rather, distinctions between and within dialects are captured by the presence, absence, and frequency of dozens of dialect features in speech and text, such as the deletion of the copula in "He  $\varnothing$  running". In this paper, we introduce the task of dialect feature detection, and present two multitask learning approaches, both based on pretrained transformers. For most dialects, large-scale annotated corpora for these features are unavailable, making it difficult to train recognizers. We train our models on a small number of minimal pairs, building on how linguists typically define dialect features. Evaluation on a test set of 22 dialect features of Indian English demonstrates that these models learn to recognize many features with high accuracy, and that a few minimal pairs can be as effective for training as thousands of labeled examples. We also demonstrate the downstream applicability of dialect feature detection both as a measure of dialect density and as a dialect classifier.
Dialect variation is a pervasive property of language, which must be accounted for if we are to build robust natural language processing (NLP) systems that serve everyone. Linguists do not characterize dialects as simple categories, but rather as collections of correlated features (Nerbonne, 2009), such as the one shown in Figure 1; speakers of any given dialect vary regarding which features they employ, how frequently, and in which contexts. In comparison to approaches that classify speakers or documents across dialects (typically using metadata such as geolocation), the feature-based perspective has several advantages: (1) allowing for fine-grained comparisons of speakers or documents
176. Deletion of copula be: before NPs
Feature area: Agreement
Typical example: He  $\varnothing$  a good teacher.
Figure 1: An example dialect feature from the Electronic World Atlas of Varieties of English (eWAVE).<sup>1</sup>
within dialects, without training on personal metadata; (2) disentangling grammatical constructions that make up the dialect from the content that may be frequently discussed in the dialect; (3) enabling robustness testing of NLP systems across dialect features, helping to ensure adequate performance even on cases other than "high-resource" varieties such as mainstream U.S. English (Blodgett et al., 2016); (4) helping to develop more precise characterizations of dialects, enabling more accurate predictions of variable language use and better interpretations of its social implications (e.g., Craig and Washington, 2002; Van Hofwegen and Wolfram, 2010).
The main challenge for recognizing dialect features computationally is the lack of labeled data. Annotating dialect features requires linguistic expertise and is prohibitively time-consuming given the large number of features and their sparsity. In dialectology, large-scale studies of text are limited to features that can be detected using regular expressions of surface forms and parts-of-speech, e.g., PRP DT for the copula deletion feature in Figure 1; many features cannot be detected with such patterns (e.g. OBJECT FRONTING, EXTRANEUS ARTICLE). Furthermore, part-of-speech tagging is unreliable in many language varieties, such as re
<sup>1</sup>https://ewave-atlas.org. Shapes indicate variety type, e.g. creole, L1, and L2 English varieties.
* Work done while at Google Research.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2315-2338 June 6-11, 2021. ©2021 Association for Computational Linguistics
gional and minority dialects (Jorgensen et al., 2015; Blodgett et al., 2016). As dialect density correlates with social class and economic status (Sahgal and Agnihotri, 1988; Rickford et al., 2015; Grogger et al., 2020), the failure of language technology to cope with dialect differences may create allocational harms that reinforce social hierarchies (Blodgett et al., 2020).
In this paper, we propose and evaluate learning-based approaches to recognize dialect features. We focus on Indian English, given the availability of domain expertise and labeled corpora for evaluation. First, we consider a standard multitask classification approach, in which a pretrained transformer (Vaswani et al., 2017) is fine-tuned to recognize a set of dialect features. The architecture can be trained from two possible sources of supervision: (1) thousands of labeled corpus examples, (2) a small set of minimal pairs, which are hand-crafted examples designed to highlight the key aspects of each dialect feature (as in the "typical example" field of Figure 1). Because most dialects have little or no labeled data, the latter scenario is more realistic for most dialects. We also consider a multitask architecture that learns across multiple features by encoding the feature names, similar to recent work on few-shot or zero-shot multitask learning (Logeswaran et al., 2019; Brown et al., 2020).
In Sections 4 and 5, we discuss empirical evaluations of these models. Our main findings are:
- It is possible to detect individual dialect features: several features can be recognized with reasonably high accuracy. Our best models achieve a macro-AUC of .848 across ten grammatical features for which a large test set is available.
- This performance can be obtained by training on roughly five minimal pairs per feature. Minimal pairs are significantly more effective for training than a comparable number of corpus examples.
- Dialect feature recognizers can be used to rank documents by their density of dialect features, enabling within-dialect density computation for Indian English and accurate classification between Indian and U.S. English.
2 Data and Features of Indian English
We develop methods for detecting 22 dialect features associated with Indian English. Although India has over 125 million English speakers — making it the world's second largest English-speaking
population — there is relatively little NLP research focused on Indian English. Our methods are not designed exclusively for specific properties of Indian English; many of the features that are associated with Indian English are also present in other dialects of English.
We use two sources of data in our study: an annotated corpus (§ 2.1) and a dataset of minimal pairs (§ 2.2). For evaluation, we use corpus annotations exclusively. The features are described in Table 1, and our data is summarized in Table 2.
2.1 Corpus Annotations
The International Corpus of English (ICE; Greenbaum and Nelson, 1996) is a collection of corpora of world varieties of English, organized primarily by the national origin of the speakers/writers. We focus on annotations of spoken dialogs (S1A001 - S1A-090) from the Indian English subcorpus (ICE-India). The ICE-India subcorpus was chosen in part because it is one of the only corpora with large-scale annotations of dialect features. To contrast Indian English with U.S. English (\$ 4), we use the Santa Barbara Corpus of Spoken American English (Du Bois et al., 2000) that constitutes the ICE-USA subcorpus of spoken dialogs.
We work with two main sources of dialect feature annotations in the ICE-India corpus:
Lange features. The first set of annotations come from Claudia Lange (2012), who annotated 10 features in 100 transcripts for an analysis of discourse-driven syntax in Indian English, such as topic marking and fronting. We use half of this data for training (50 transcripts, 9392 utterances), and half for testing (50 transcripts, 9667 utterances).
Extended features. To test a more diverse set of features, we additionally annotated 18 features on a set of 300 turns randomly selected from the conversational subcorpus of ICE-India, $^{2}$  as well as 50 examples randomly selected from a secondary dataset of sociolinguistic interviews (Sharma, 2009) to ensure diverse feature instantiation. We selected our 18 features based on multiple criteria: 1) prevalence in Indian English based on the dialectology literature, 2) coverage in the data (we started out with a larger set of features and removed those with fewer than two occurrences), 3) diversity of linguistic phenomena. The extended
2We manually split turns that were longer than two clauses, resulting in 317 examples.
Feature	Example	Count of Instantiations
Lange (2012)	Our data
ARTICLE OMISSION	(the) chair is black	59
DIRECT OBJECT PRO-DROP	she doesn't like (it)	14
FOCUS itself	he is doing engineering in Delhi itself	24	5
FOCUS only	I was there yesterday only	95	8
HABITUAL PROGRESSIVE	always we are giving receipt	2
STATIVE PROGRESSIVE	he is having a television	3
LACK OF INVERSION IN WH-QUESTIONS	what you are doing?	4
LACK OF AGREEMENT	he do a lot of things	23
LEFT DISLOCATION	my father, he works for a solar company	300	19
MASS NOUNS AS COUNT NOUNS	all the musics are very good	13
NON-INITIAL EXISTENTIAL	every year inflation is there	302	8
OBJECT FRONTING	minimum one month you have to wait	186	14
PP FRONTING WITH REDUCTION	(on the) right side we can see a plate	11
PREPOSITION OMISSION	I went (to) another school	17
INVERSION IN EMBEDDED CLAUSE	I don't know what are they doing	4
INVARIANT TAG (isn't it, no, na)	the children are outside, isn't it?	786	17
EXTRANEOUS ARTICLE	she has a business experience	25
GENERAL EXTENDER and all	then she did her schooling and all	7
COPULA OMISSION	my parents (are) from Gujarat	71
RESUMPTIVE OBJECT PRONOUN	my old life I want to spend it in India	24
RESUMPTIVE SUBJECT PRONOUN	my brother, he lives in California	287
TOPICALIZED NON-ARGUMENT CONSTITUENT	in those years I did not travel	272
Table 1: Features of Indian English used in our evaluations and their counts in the two datasets we study.
Dialect features	Unique annotated examples
Feature set	Count	Corpus ex.	Min. pair ex.
Lange (2012)	10	19059	113
Extended	18	367	208
Table 2: Summary of our labeled data. All corpus examples for the Lange features are from ICE-India; for the Extended feature set, examples are drawn from ICE-India and the Sharma data.
features overlap with those annotated by Lange, yielding a total set of 22 features. Annotations were produced by consensus from the first two authors. To measure interrater agreement, a third author (JE) independently re-annotated  $10\%$  of the examples, with Cohen's  $\kappa = 0.79$  (Cohen, 1960).<sup>3</sup>
2.2 Minimal Pairs
For each of the 22 features in Table 1, we created a small set of minimal pairs. The pairs were created by first designing a short example that demonstrated the feature, and then manipulating the example so that the feature is absent. This "negative" example captures the envelope of variation for the feature, demonstrating a site at which the feature could be applied (Labov, 1972). Consequently,
3Our annotations will be made available at https:// dialectfeatures.page.link/annotations.
negative examples in minimal pairs carry more information than in the typical annotation scenario, where absence of evidence does not usually imply evidence of absence. In our minimal pairs, the negative examples were chosen to be acceptable in standard U.S. and U.K. English, and can thus be viewed as situating dialects against standard varieties. Here are some example minimal pairs:
ARTICLE OMISSION: chair is black  $\rightarrow$  the chair is black
FOCUS only: I was there yesterday  $\underline{\text{only}}\rightarrow I$  was there just yesterday.
NON-INITIAL EXISTENTIAL: every year inflation is there  $\rightarrow$  every year there is inflation.
For most features, each minimal pair contains exactly one positive and one negative example. However, in some cases where more than two variants are available for an example (e.g., for the feature INVARIANT TAG (isn't it, no, na)), we provide multiple positive examples to illustrate different variants. For Lange's set of 10 features, we provide a total of 113 unique examples; for the 18 extended features, we provide a set of 208 unique examples, roughly split equally between positives and negatives. The complete list of minimal pairs is included in Appendix D.
y	x
1	[CLS] article omission [SEP] Chair is black. [SEP]
0	[CLS] article omission [SEP] The chair is black. [SEP]
0	[CLS] article omission [SEP] I was there yesterday only. [SEP]
...	...
1	[CLS] focus only [SEP] I was there yesterday only. [SEP]
0	[CLS] focus only [SEP] I was there just yesterday. [SEP]
0	[CLS] focus only [SEP] Chair is black. [SEP]
...	...
Figure 2: Conversion of minimal pairs to labeled examples for DAMTL, using two minimal pairs.
3 Models and training
We train models to recognize dialect features by fine-tuning the BERT-base uncased transformer architecture (Devlin et al., 2019). We consider two strategies for constructing training data, and two architectures for learning across multiple features.
3.1 Sources of supervision
We consider two possible sources of supervision:
Minimal pairs. We apply a simple procedure to convert minimal pairs into training data for classification. The positive part of each pair is treated as a positive instance for the associated feature, and the negative part is treated as a negative instance. Then, to generate more data, we also include elements of other minimal pairs as examples for each feature: for instance, a positive example of the RESUMPTIVE OBJECT PRONOUN feature would be a negative example for FOCUS only, unless the example happened to contain both features (this was checked manually). In this way, we convert the minimal pairs into roughly 113 examples per feature for Lange's features and roughly 208 examples per feature for the extended features. The total number of unique surface forms is still 113 and 208 respectively. Given the lack of labeled data for most dialects of the world, having existing minimal pairs or collecting a small number of minimal pairs is the most realistic data scenario.
Corpus annotations. When sufficiently dense annotations are available, we can train a classifier
based on these labeled instances. We use 50 of the ICE-India transcripts annotated by Lange, which consists of 9392 labeled examples (utterances) per feature. While we are lucky to have such a large resource for the Indian English dialect, this high-resource data scenario is rare.
3.2 Architectures
We consider two classification architectures:
Multihead. In this architecture, which is standard for multitask classification, we estimate a linear prediction head for each feature, which is simply a vector of weights. This is a multitask architecture, because the vast majority of model parameters from the input through the deep BERT stack remain shared among dialect features. The prediction head is then multiplied by the BERT embedding for the [CLS] token to obtain a score for a feature's applicability to a given instance.
DAMTL. Due to the few-shot nature of our prediction task, we also consider an architecture that attempts to exploit the natural language descriptions of each feature. This is done by concatenating the feature description to each element of the minimal pair. The instance is then labeled for whether the feature is present. This construction is shown in Figure 2. Prediction is performed by learning a single linear prediction head on the [CLS] token. We call this model description-aware multitask learning, or DAMTL.
Model details. Both architectures are built on top of the BERT-base uncased model, which we fine-tune by cross-entropy for 500 epochs (due to the small size of the training data) using the Adam optimizer (Kingma and Ba, 2014), batch size of 32 and a learning rate of  $10^{-5}$ , warmed up over the first 150 epochs. Annotations of dialect features were not used for hyperparameter selection. Instead, the hyperparameters were selected to maximize the discriminability between corpora of Indian and U.S. English, as described in § 5.2. All models trained in less than two hours on a pod of four v2 TPU chips, with the exception of DAMTL on corpus examples, which required up to 18 hours.
3.3 Regular Expressions
In dialectology, regular expression pattern matching is the standard tool for recognizing dialect features (e.g., Nerbonne et al., 2011). For the features
Supervision:Dialect feature	Corpus examples	Minimal pairs
DAMTL	Multihead	DAMTL	Multihead
FOCUS itself*	0.945	0.925	0.974	0.960
FOCUS only*	0.975	0.911	0.994	0.938
INVARIANT TAG	0.991	0.985	0.969	0.925
COPULA OMISSION	0.536	0.641	0.626	0.746
LEFT DISLOCATION	0.855	0.879	0.765	0.885
NON-INITIAL EXISTENTIAL*	0.991	0.992	0.905	0.879
OBJECT FRONTING	0.805	0.809	0.678	0.761
RES. OBJECT PRONOUN	0.595	0.667	0.733	0.825
RES. SUBJECT PRONOUN	0.886	0.887	0.688	0.857
TOPICALIZED NON-ARG. CONST.	0.725	0.727	0.499	0.707
Macro Average	0.830	0.842	0.783	0.848
Table 3: ROC-AUC scores on the Lange feature set, averaged across five random seeds. Asterisk (*) marks features that can be detected with relatively high accuracy ( $>0.85$  ROC-AUC) using regular expressions.
described in Table 1, we were able to design regular expressions for only five. Prior work sometimes relies on patterns that include both surface forms and part-of-speech (e.g., Bohmann, 2019), but part-of-speech cannot necessarily be labeled automatically for non-standard dialects (Jorgensen et al., 2015; Blodgett et al., 2016), so we consider only regular expressions over surface forms.
4 Results on Dialect Feature Detection
In this section, we present results on the detection of individual dialect features. Using the features shown in Table 1, we compare supervision sources (corpus examples versus minimal pairs) and classification architectures (multihead versus DAMTL) as described in § 3. To avoid tuning a threshold for detection, we report area under the ROC curve (ROC-AUC), which has a value of .5 for random guessing and 1 for perfect prediction.
4.1 Results on Lange Data and Features
We first consider the 10 syntactic features from Lange (2012), for which we have large-scale annotated data: the 100 annotated transcripts from the ICE-India corpus are split 50/50 into training and test sets. As shown in Table 3, it is possible to achieve a Macro-AUC approaching .85 overall with multihead predictions on minimal pair examples. This is promising, because it suggests the possibility of recognizing dialect features for which we lack labeled corpus examples – and such low-data
situations are by far the most common data scenario among the dialects of the world.
The multihead architecture outperforms DAMTL on both corpus examples and minimal pairs. In an ablation, we replaced the feature descriptions with non-descriptive identifiers such as "Feature 3". This reduced the Macro-AUC from to .80 with corpus examples, and to .76 with minimal pairs (averaged over five random seeds). We also tried longer feature descriptions, but this did not improve performance.
Unsurprisingly, the lexical features (e.g., FOCUS itself) are easiest to recognize. The more syntactical features (e.g., COPULA OMISSION, RESUMPTIVE OBJECT PRONOUN) are more difficult, although some movement-based features (e.g., LEFT DISLOCATION, RESUMPTIVE SUBJECT PRONOUN) can be recognized accurately.
Qualitative model comparison. We conducted a qualitative comparison of three models: regular expressions and two versions of the multihead model, one trained on corpus examples and another trained on minimal pairs. Table 4 includes illustrative examples for the Lange data and features where models make different predictions. We find that the minimal pair model is better able to account for rare cases (e.g. use of non-focus "only" in Example 1), likely as it was trained on a few carefully selected set of examples illustrating positives and negatives. Both multihead models are able to account for disfluencies and restarts, in contrast to regular expressions (Example 2). Our analysis shows that several model errors are accounted for by difficult examples (Example 3: "is there" followed by "isn't"; Example 6: restart mistaken for left dislocation) or the lack of contextual information available to the model (Example 4 & 7: truncated examples). Please see Appendix B for more details and random samples of model predictions.
Learning from fewer corpus examples. The minimal pair annotations consist of 113 examples; in contrast, there are 9392 labeled corpus examples, requiring far more effort to create. We now consider the situation when the amount of labeled data is reduced, focusing on the Lange features (for which labeled training data is available). As shown in Figure 3, even 5000 labeled corpus examples do not match the performance of training on roughly 5 minimal pairs per feature.
4Features: FOCUS itself, FOCUS only, NON-INITIAL EXISTENTIAL, INVARIANT TAG (isn't it, no, na), and GENERAL EXTENDER and all. Table 7 lists all regular expressions.
Results for area under the precision-recall (AUPR) curve are shown in Appendix C. According to this metric, minimal pairs are less effective than the full training set of corpus examples, on average.
Example	Feature	Gold label	Regex	Multihead Corpus ex.	Min. pair
1	But whereas in Hyderabad they are only stuck with their books and home and work that's all like	FOCUS only	0	1	1	0
2	There is there is a club this humour club oh good and I've chance I had a chance of attending	NON-INITIAL EXISTENTIAL	0	1	0	0
3	New Education Policy is there isn't it?	NON-INITIAL EXISTENTIAL	1	1	0	0
4	I didn't go anywhere no	INvariant TAG (isn't it, no, na)	0	1	1	1
5	In fact my son and daughter they had asked me to buy buy them this thing the sunglasses	LEFT DISLOCATION	1	N/A	1	1
6	His house he is going to college KK diploma electronics	RESUMPTIVE SUBJECT PRONOUN	0	N/A	0	1
7	Which October first I think	COPULA OMISSION	0	N/A	1	0
8	Papers we can't say hard only because they already taught that same	COPULA OMISSION	1	N/A	0	0
9	Just typing work I have to do	OBJECT FRONTING	1	N/A	1	1
10	My post graduation degree I finished it in mid June nineteen eighty-six	RESUMPTIVE OBJECT PRONOUN	1	N/A	0	1
Table 4: Example model predictions from the Lange data and feature set, comparing regular expressions with two versions of the multihead model, one trained on corpus examples and another on minimal pairs. 'Gold label' indicates whether the feature was manually labeled as present in the original Lange data. Green and red indicate correct and incorrect predictions, respectively.
Figure 3: Performance of the multihead model as the number of corpus examples is varied. Box plots are over 10 random data subsets, showing the 25th, 50th, and 75th percentiles; whiskers show the most extreme points within  $\pm 1.5$  times the inter-quartile range.
Corpus examples stratified by feature. One reason that subsampled datasets yield weaker results is that they lack examples for many features. To enable a more direct comparison of corpus examples and minimal pairs, we created a set of "stratified" datasets of corpus examples, such that the number of positive and negative examples for each feature exactly matches the minimal pair data. Averaged over ten such random stratified samples, the multihead model achieves a Macro-AUC of .790  $(\sigma = 0.029)$ , and DAMTL achieves a Macro-AUC of .722  $(\sigma = .020)$ . These results are considerably worse than training on an equivalent number of minimal pairs, where the multihead model achieves a Macro-AUC of .848 and DAMTL achieves a
Macro-AUC of .783. This demonstrates the utility of minimal pairs over corpus examples for learning to recognize dialect features.
4.2 Results on Extended Feature Set