Venue: NAACL 2021

# Learning to Recognize Dialect Features

Dorottya Demszky\({}^{1*}\)  Devyani Sharma\({}^{2}\)   Jonathan H. Clark\({}^{3}\)

Vinodkumar Prabhakaran\({}^{3}\)    Jacob Eisenstein\({}^{3}\)

###### Abstract

Building NLP systems that serve everyone requires accounting for dialect differences. But dialects are not monolithic entities: rather, distinctions between and within dialects are captured by the presence, absence, and frequency of dozens of dialect features in speech and text, such as the deletion of the copula in "He \(\varnothing\) running". In this paper, we introduce the task of dialect feature detection, and present two multitask learning approaches, both based on pre-trained transformers. For most dialects, large-scale annotated corpora for these features are unavailable, making it difficult to train recognizers. We train our models on a small number of minimal pairs, building on how linguists typically define dialect features. Evaluation on a test set of 22 dialect features of Indian English demonstrates that these models learn to recognize many features with high accuracy, and that a few minimal pairs can be as effective for training as thousands of labeled examples. We also demonstrate the downstream applicability of dialect feature detection both as a measure of dialect density and as a dialect classifier.

+
Footnote †: 17]Work done while at Google Research.

Topic example: He O a good teacher.

## 1 Introduction

Dialect variation is a pervasive property of language, which must be accounted for if we are to build robust natural language processing (NLP) systems that serve everyone. Linguists do not characterize dialects as simple categories, but rather as collections of correlated features Nerbonne (2009), such as the one shown in Figure 1; speakers of any given dialect vary regarding which features they employ, how frequently, and in which contexts. In comparison to approaches that classify speakers or documents across dialects (typically using metadata such as geolocation), the feature-based perspective has several advantages: (1) allowing for fine-grained comparisons of speakers or documents _within_ dialects, without training on personal metadata; (2) disentangling grammatical constructions that make up the dialect from the content that may be frequently discussed in the dialect; (3) enabling robustness testing of NLP systems across dialect features, helping to ensure adequate performance even on cases other than "high-resource" varieties such as mainstream U.S. English Blodgett et al. (2016); (4) helping to develop more precise characterizations of dialects, enabling more accurate predictions of variable language use and better interpretations of its social implications e.g., Craig and Washington (2002); Van Hofwegen and Wolfram (2010).

The main challenge for recognizing dialect features computationally is the lack of labeled data. Annotating dialect features requires linguistic expertise and is prohibitively time-consuming given the large number of features and their sparsity. In dialectology, large-scale studies of text are limited to features that can be detected using regular expressions of surface forms and parts-of-speech, e.g., PRP DT for the copula deletion feature in Figure 1; many features cannot be detected with such patterns (e.g. object fronting, extraneous article). Furthermore, part-of-speech tagging is unreliable in many language varieties, such as re

Figure 1: An example dialect feature from the Electronic World Atlas of Varieties of English (eWAVE).1

gional and minority dialects (Jorgensen et al., 2015; Blodgett et al., 2016). As dialect density correlates with social class and economic status Sahgal and Agnihotri (1988); Rickford et al. (2015); Grogger et al. (2020), the failure of language technology to cope with dialect differences may create allocational harms that reinforce social hierarchies Blodgett et al. (2020).

In this paper, we propose and evaluate learning-based approaches to recognize dialect features. We focus on Indian English, given the availability of domain expertise and labeled corpora for evaluation. First, we consider a standard multitask classification approach, in which a pretrained transformer Vaswani et al. (2017) is fine-tuned to recognize a set of dialect features. The architecture can be trained from two possible sources of supervision: (1) thousands of labeled corpus examples, (2) a small set of _minimal pairs_, which are hand-crafted examples designed to highlight the key aspects of each dialect feature (as in the "typical example" field of Figure 1). Because most dialects have little or no labeled data, the latter scenario is more realistic for most dialects. We also consider a multitask architecture that learns across multiple features by encoding the feature names, similar to recent work on few-shot or zero-shot multitask learning Logeswaran et al. (2019); Brown et al. (2020).

In Sections 4 and 5, we discuss empirical evaluations of these models. Our main findings are:

* It is possible to detect individual dialect features: several features can be recognized with reasonably high accuracy. Our best models achieve a macro-AUC of \(.848\) across ten grammatical features for which a large test set is available.
* This performance can be obtained by training on roughly five minimal pairs per feature. Minimal pairs are significantly more effective for training than a comparable number of corpus examples.
* Dialect feature recognizers can be used to rank documents by their density of dialect features, enabling within-dialect density computation for Indian English and accurate classification between Indian and U.S. English.

## 2 Data and Features of Indian English

We develop methods for detecting 22 dialect features associated with Indian English. Although India has over 125 million English speakers -- making it the world's second largest English-speaking population -- there is relatively little NLP research focused on Indian English. Our methods are not designed exclusively for specific properties of Indian English; many of the features that are associated with Indian English are also present in other dialects of English.

We use two sources of data in our study: an annotated corpus (SS 2.1) and a dataset of minimal pairs (SS 2.2). For evaluation, we use corpus annotations exclusively. The features are described in Table 1, and our data is summarized in Table 2.

### Corpus Annotations

The International Corpus of English (ICE; Greenbaum and Nelson, 1996) is a collection of corpora of world varieties of English, organized primarily by the national origin of the speakers/writers. We focus on annotations of spoken dialogs (S1A-001 - S1A-090) from the Indian English subcorpus (ICE-India). The ICE-India subcorpus was chosen in part because it is one of the only corpora with large-scale annotations of dialect features. To contrast Indian English with U.S. English (SS 4), we use the Santa Barbara Corpus of Spoken American English Du Bois et al. (2000) that constitutes the ICE-USA subcorpus of spoken dialogs.

We work with two main sources of dialect feature annotations in the ICE-India corpus:

**Lange features.** The first set of annotations come from Claudia Lange (2012), who annotated 10 features in 100 transcripts for an analysis of discourse-driven syntax in Indian English, such as topic marking and fronting. We use half of this data for training (50 transcripts, 9392 utterances), and half for testing (50 transcripts, 9667 utterances).

**Extended features.** To test a more diverse set of features, we additionally annotated 18 features on a set of 300 turns randomly selected from the conversational subcorpus of ICE-India,2 as well as 50 examples randomly selected from a secondary dataset of sociolinguistic interviews Sharma (2009) to ensure diverse feature instantiation. We selected our 18 features based on multiple criteria: 1) prevalence in Indian English based on the dialectology literature, 2) coverage in the data (we started out with a larger set of features and removed those with fewer than two occurrences), 3) diversity of linguistic phenomena. The extended 

features overlap with those annotated by Lange, yielding a total set of 22 features. Annotations were produced by consensus from the first two authors. To measure interrater agreement, a third author (JE) independently re-annotated 10% of the examples, with Cohen's \(\kappa=0.79\)[Cohen, 1960].3

Footnote 3: Our annotations will be made available at [https://dialectfeatures.page.link/annotations](https://dialectfeatures.page.link/annotations).

### Minimal Pairs

For each of the 22 features in Table 1, we created a small set of minimal pairs. The pairs were created by first designing a short example that demonstrated the feature, and then manipulating the example so that the feature is absent. This "negative" example captures the _envelope of variation_ for the feature, demonstrating a site at which the feature could be applied [Labov, 1972]. Consequently, negative examples in minimal pairs carry more information than in the typical annotation scenario, where absence of evidence does not usually imply evidence of absence. In our minimal pairs, the negative examples were chosen to be acceptable in standard U.S. and U.K. English, and can thus be viewed as situating dialects against standard varieties. Here are some example minimal pairs:

**article omission:**: _chair is black \(\rightarrow\underline{\text{the}}\) chair is black_
**focus only****:**: _I was there yesterday only___\(\to I\)__was there just yesterday._
**non-initial existential****:**: _every year inflation is there \(\rightarrow\) every year there is inflation._

For most features, each minimal pair contains exactly one positive and one negative example. However, in some cases where more than two variants are available for an example (e.g., for the feature invariant tag _(isn't it, no, na)_), we provide multiple positive examples to illustrate different variants. For Lange's set of 10 features, we provide a total of 113 unique examples; for the 18 extended features, we provide a set of 208 unique examples, roughly split equally between positives and negatives. The complete list of minimal pairs is included in Appendix D.

\begin{table}
\begin{tabular}{l l c c} \hline \hline
**Feature** & **Example** & \multicolumn{2}{c}{**Count of Instantiations**} \\ \cline{4-5}  & & Lange (2012) & Our data \\ \hline article omission & _(the) chair is black_ & & 59 \\ direct object pro-drop & _she doesn’t like (it)_ & & 14 \\ focus itself_ & _he is doing engineering in Delhi itself_ & 24 & 5 \\ focus only_ & _I was three yesterday only_ & 95 & 8 \\ habitual progressive & _always we are giving receipt_ & & 2 \\ static progressive & _he is having a television_ & & 3 \\ lack of inversion in WH-questions & _what you are doing?_ & & 4 \\ lack of Agreement & _he do a lot of things_ & & 23 \\ left dislocation & _my father, he works for a solar company_ & 300 & 19 \\ mass nouns as count nouns & _all the musicz any very good_ & & 13 \\ non-initial existential & _every year inflation is there_ & 302 & 8 \\ object fronting & _minimum one month you have to wait_ & 186 & 14 \\ PP fronting with reduction & _(on the) right side we can see a plate_ & & 11 \\ preposition omission & _I went (to) another school_ & & 17 \\ investigation in embedded clause & _I don’t know what are they doing_ & & 4 \\ invariant tag _(isn’t it, no, na)_ & _the children are outside, isn’t it?_ & 786 & 17 \\ extractaneous article & she has a business experience_ & & 25 \\ general extender _and all_ & _then she did her schooling and all_ & & 7 \\ copula omission & _my parents (are) from Gujarat_ & 71 \\ resumptive object pronoun & _my old life I want to spend it in India_ & 24 \\ resumptive subject pronoun & _my brother, he lives in California_ & 287 \\ topicalized non-argument constituent & _in those years I didn travel_ & 272 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Features of Indian English used in our evaluations and their counts in the two datasets we study.

\begin{table}
\begin{tabular}{l l l l} \hline \hline
**Dialect features** & \multicolumn{3}{c}{**Unique annotated examples**} \\ \hline Feature set & Count & Corpus ex. & Min. pair ex. \\ \hline Lange (2012) & 10 & 19059 & 113 \\ Extended & 18 & 367 & 208 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Summary of our labeled data. All corpus examples for the Lange features are from ICE-India; for the Extended feature set, examples are drawn from ICE-India and the Sharma data.

 



## 3 Models and training

We train models to recognize dialect features by fine-tuning the BERT-base uncased transformer architecture Devlin et al. (2019). We consider two strategies for constructing training data, and two architectures for learning across multiple features.

### Sources of supervision

We consider two possible sources of supervision:

**Minimal pairs.** We apply a simple procedure to convert minimal pairs into training data for classification. The positive part of each pair is treated as a positive instance for the associated feature, and the negative part is treated as a negative instance. Then, to generate more data, we also include elements of other minimal pairs as examples for each feature: for instance, a positive example of the resumptive object pronoun feature would be a negative example for focus_only_, unless the example happened to contain both features (this was checked manually). In this way, we convert the minimal pairs into roughly 113 examples per feature for Lange's features and roughly 208 examples per feature for the extended features. The total number of unique surface forms is still 113 and 208 respectively. Given the lack of labeled data for most dialects of the world, having existing minimal pairs or collecting a small number of minimal pairs is the most realistic data scenario.

**Corpus annotations.** When sufficiently dense annotations are available, we can train a classifier based on these labeled instances. We use 50 of the ICE-India transcripts annotated by Lange, which consists of 9392 labeled examples (utterances) per feature. While we are lucky to have such a large resource for the Indian English dialect, this high-resource data scenario is rare.

### Architectures

We consider two classification architectures:

**Multihead.** In this architecture, which is standard for multitask classification, we estimate a linear _prediction head_ for each feature, which is simply a vector of weights. This is a multitask architecture, because the vast majority of model parameters from the input through the deep BERT stack remain shared among dialect features. The prediction head is then multiplied by the BERT embedding for the [cls] token to obtain a score for a feature's applicability to a given instance.

DaMTL.Due to the few-shot nature of our prediction task, we also consider an architecture that attempts to exploit the natural language descriptions of each feature. This is done by concatenating the feature description to each element of the minimal pair. The instance is then labeled for whether the feature is present. This construction is shown in Figure 2. Prediction is performed by learning a single linear prediction head on the [cls] token. We call this model _description-aware multitask learning_, or DaMTL.

Model details.Both architectures are built on top of the BERT-base uncased model, which we fine-tune by cross-entropy for \(500\) epochs (due to the small size of the training data) using the Adam optimizer Kingma and Ba (2014), batch size of \(32\) and a learning rate of \(10^{-5}\), warmed up over the first \(150\) epochs. Annotations of dialect features were not used for hyperparameter selection. Instead, the hyperparameters were selected to maximize the discriminability between corpora of Indian and U.S. English, as described in SS 5.2. All models trained in less than two hours on a pod of four v2 TPU chips, with the exception of DaMTL on corpus examples, which required up to 18 hours.

### Regular Expressions

In dialectology, regular expression pattern matching is the standard tool for recognizing dialect features (e.g., Nerbonne et al., 2011). For the features

Figure 2: Conversion of minimal pairs to labeled examples for DaMTL, using two minimal pairs.



described in Table 1, we were able to design regular expressions for only five.4 Prior work sometimes relies on patterns that include both surface forms and part-of-speech (e.g., Bohmann, 2019), but part-of-speech cannot necessarily be labeled automatically for non-standard dialects (Jorgensen et al., 2015; Blodgett et al., 2016), so we consider only regular expressions over surface forms.

Footnote 4: Features: focus itself, focus only, non-initial existential, invariant tag _(isn’t it, no, na)_, and general extender and all_. Table 7 lists all regular expressions.

## 4 Results on Dialect Feature Detection

In this section, we present results on the detection of individual dialect features. Using the features shown in Table 1, we compare supervision sources (corpus examples versus minimal pairs) and classification architectures (multihead versus DaMTL) as described in SS 3. To avoid tuning a threshold for detection, we report area under the ROC curve (ROC-AUC), which has a value of \(.5\) for random guessing and \(1\) for perfect prediction.5

Footnote 5: Results for area under the precision-recall (AUPR) curve are shown in Appendix C. According to this metric, minimal pairs are less effective than the full training set of corpus examples, on average.

### Results on Lange Data and Features

We first consider the 10 syntactic features from Lange (2012), for which we have large-scale annotated data: the 100 annotated transcripts from the ICE-India corpus are split 50/50 into training and test sets. As shown in Table 3, it is possible to achieve a Macro-AUC approaching .85 overall with multihead predictions on minimal pair examples. This is promising, because it suggests the possibility of recognizing dialect features for which we lack labeled corpus examples - and such low-data situations are by far the most common data scenario among the dialects of the world.

The multihead architecture outperforms DaMTL on both corpus examples and minimal pairs. In an ablation, we replaced the feature descriptions with non-descriptive identifiers such as "Feature 3". This reduced the Macro-AUC from to \(.80\) with corpus examples, and to \(.76\) with minimal pairs (averaged over five random seeds). We also tried longer feature descriptions, but this did not improve performance.

Unsurprisingly, the lexical features (e.g., focus itself) are easiest to recognize. The more syntactical features (e.g., copula omission, resumptive object pronoun) are more difficult, although some movement-based features (e.g., left dislocation, resumptive subject pronoun) can be recognized accurately.

Qualitative model comparison.We conducted a qualitative comparison of three models: regular expressions and two versions of the multihead model, one trained on corpus examples and another trained on minimal pairs. Table 4 includes illustrative examples for the Lange data and features where models make different predictions. We find that the minimal pair model is better able to account for rare cases (e.g. use of non-focus "only" in Example 1), likely as it was trained on a few carefully selected set of examples illustrating positives and negatives. Both multihead models are able to account for disfluencies and restarts, in contrast to regular expressions (Example 2). Our analysis shows that several model errors are accounted for by difficult examples (Example 3: "is there" followed by "isn't"; Example 6: restart mistaken for left dislocation) or the lack of contextual information available to the model (Example 4 & 7: truncated examples). Please see Appendix B for more details and random samples of model predictions.

Learning from fewer corpus examples.The minimal pair annotations consist of 113 examples; in contrast, there are 9392 labeled corpus examples, requiring far more effort to create. We now consider the situation when the amount of labeled data is reduced, focusing on the Lange features (for which labeled training data is available). As shown in Figure 3, even 5000 labeled corpus examples do not match the performance of training on roughly 5 minimal pairs per feature.

\begin{table}
\begin{tabular}{l c c c c} \hline \multicolumn{1}{c}{**Supervision:**} & \multicolumn{2}{c}{**Corpus examples**} & \multicolumn{2}{c}{**Minimal pairs**} \\ Diated feature & DaMTL & Multihead & DaMTL & Multihead \\ \hline Focus _intel*_ & 0.945 & 0.925 & 0.974 & 0.960 \\ Focus only* & 0.975 & 0.911 & 0.994 & 0.938 \\ invaraint Tag & 0.991 & 0.985 & 0.960 & 0.925 \\ Copula Omission & 0.536 & 0.641 & 0.626 & 0.746 \\ Letter Dislocation & 0.855 & 0.879 & 0.765 & 0.885 \\ Non-initial existential* & 0.991 & 0.992 & 0.905 & 0.879 \\ Object Promoting & 0.805 & 0.809 & 0.678 & 0.761 \\ Res. other PROGRAM & 0.595 & 0.667 & 0.733 & 0.825 \\ res. subject pronoun & 0.886 & 0.887 & 0.688 & 0.857 \\ topicalized non-ARG. const. & 0.725 & 0.727 & 0.499 & 0.707 \\
**Macro Average** & 0.830 & 0.842 & 0.783 & 0.848 \\ \hline \end{tabular}
\end{table}
Table 3: ROC-AUC scores on the Lange feature set, averaged across five random seeds. Asterisk (*) marks features that can be detected with relatively high accuracy (\(>0.85\) ROC-AUC) using regular expressions.

 

Corpus examples stratified by feature.One reason that subsampled datasets yield weaker results is that they lack examples for many features. To enable a more direct comparison of corpus examples and minimal pairs, we created a set of "stratified" datasets of corpus examples, such that the number of positive and negative examples for each feature exactly matches the minimal pair data. Averaged over ten such random stratified samples, the multihead model achieves a Macro-AUC of \(.790\) (\(\sigma=0.029\)), and DaMTL achieves a Macro-AUC of \(.722\) (\(\sigma=.020\)). These results are considerably worse than training on an equivalent number of minimal pairs, where the multihead model achieves a Macro-AUC of \(.848\) and DaMTL achieves a Macro-AUC of \(.783\). This demonstrates the utility of minimal pairs over corpus examples for learning to recognize dialect features.

### Results on Extended Feature Set

Next, we consider the extended features, for which we have sufficient annotations for testing but not training (Table 1). Here we compare the DaMTL and multihead models, using minimal pair data in both cases. As shown in Table 5, performance on these features is somewhat lower than on the Lange features, and for several features, at least one of the recognizers does worse than chance: direct object pro-drop, extraneous article, mass nouns as count nouns. These features seem to require deeper syntactic and semantic analysis, which may be difficult to learn from a small number of minimal pairs. On the other extreme, features with a strong lexical signature are recognized with high accuracy: general extented _and all_, Focus _itself_, focus _only_. These three features can also be recognized by regular expressions, as can non-initial existential.6 However, for a number of other features, it is possible to learn a fairly accurate recognizer from just five minimal pairs: article omission, inversion in embedded clause, left dislocation, lack of inversion in whi-questions.

Footnote 6: \band allb, \bitselflb, \bonlyvb, \bis thereb\|{barethere} b

Figure 3: Performance of the multihead model as the number of corpus examples is varied. Box plots are over 10 random data subsets, showing the 25th, 50th, and 75th percentiles; whiskers show the most extreme points within \(\pm 1.5\) times the inter-quartile range.

 



### Summary of Dialect Feature Detection

Many dialect features can be automatically recognized with reasonably high discriminative power, as measured by area under the ROC curve. However, there are also features that are difficult to recognize: particularly, features of omission (such as direct object pro-drop and preposition omission), and the more semantic features such as mass nouns as count nouns. While some features can also be identified through regular expressions (e.g., focus_only_), there are many features that can be learned but cannot be recognized by regular expressions. We now move from individual features to aggregate measures of dialect density.

## 5 Measuring Dialect Density

A dialect density measure (DDM) is an aggregate over multiple dialect features that tracks the vernacularity of a passage of speech or text. Such measures are frequently used in dielectology Van Hofwegen and Wolfram (2010), and are also useful in research on education e.g., Craig and Washington (2002). Recently, a DDM was used to evaluate the performance of speech recognition systems by the density of AAVE features Koenedke et al. (2020). The use of DDMs reflects the reality that speakers construct individual styles drawing on linguistic repertoires such as dialects to varying degrees Benor (2010). This necessitates a more nuanced description for speakers and texts than a discrete dialect category.

Following prior work (e.g., Van Hofwegen and Wolfram (2010) we construct dialect density measures from feature detectors by counting the predicted number of features in each utterance, and dividing by the number of tokens. For the learning-based feature detectors (minimal pairs and corpus examples), we include partial counts from the detection probability; for the regular expression detectors, we simply count the number of matches and dividing by the number of tokens. In addition, we construct a DDM based on a document classifier: we train a classifier to distinguish Indian English from U.S. English, and then use its predictive probability as the DDM. These DDMs are then compared on two tasks: distinguishing Indian and U.S. English, and correlation with the density of expert-annotated features. The classifier is trained by fine-tuning BERT, using a prediction head on the [cls] token.

### Ranking documents by dialect density

One application of dialect feature recognizers is to rank documents based on their dialect density, e.g. to identify challenging cases for evaluating downstream NLP systems, or for dialectology research. We correlate the dialect density against the density of expert-annotated features from Lange (2012), both measured at the transcript-level, and report the Spearman rank-correlation \(\rho\).

As shown in Table 6, the document classifier performs poorly: learning to distinguish Indian and U.S. English offers no information on the density of Indian dialect features, suggesting that the model is attending to other information, such as topics or entities. The feature-based model trained on labeled examples performs best, which is unsurprising because it is trained on the same type of features that it is now asked to predict. Performance is weaker when the model is trained from minimal pairs. Minimal pair training is particularly helpful on rare features, but offers far fewer examples on the high-frequency features, which in turn dominate the DDM scores on test data. Regular expressions perform well on this task, because we happen to have regular expressions for the high-frequency features, and because the precision issues are less problematic in aggregate when the DDM is not applied to non-dialectal transcripts.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Dialet feature** & **DaMTL Multihead** \\ \hline article omission & \(0.581\) & \(0.658\) \\ direct object pro-drop & \(0.493\) & \(0.563\) \\ extraneous article & \(0.546\) & \(0.465\) \\ Focus _itself*_ & \(1.000\) & \(0.949\) \\ Focus only* & \(0.998\) & \(0.775\) \\ habitual progressive & \(0.439\) & \(0.718\) \\ invariant tag & \(0.984\) & \(0.901\) \\ inversion in embedded clause & \(0.719\) & \(0.884\) \\ lack of agreement & \(0.543\) & \(0.674\) \\ Lack or hypertension in whi-questions & \(0.649\) & \(0.660\) \\ left disloccation & \(0.758\) & \(0.820\) \\ mass nouns as count nouns & \(0.443\) & \(0.465\) \\ non-initial existential* & \(0.897\) & \(0.885\) \\ object fronting & \(0.722\) & \(0.789\) \\ preposition omission & \(0.500\) & \(0.648\) \\ PP fronting with reduction & \(0.655\) & \(0.697\) \\ staticy progressive & \(0.645\) & \(0.789\) \\ general extractor _and all_ & \(0.994\) & \(0.991\) \\
**Macro Average** & \(0.698\) & \(0.741\) \\ \hline \hline \end{tabular}
\end{table}
Table 5: ROC-AUC results on the extended feature set, averaged across five random seeds. Because labeled corpus examples are not available for some features, we train only on minimal pairs. Asterisk (*) marks features that can be detected with relatively high accuracy (\(>0.85\) ROC-AUC) using regular expressions.





### Dialect Classification

Another application of dialect feature recognizers is to classify documents or passages by dialect Dunn (2018). This can help to test the performance of downstream models across dialects, assessing dialect transfer loss (e.g., Blodgett et al., 2016), as well as identifying data of interest for manual dialectological research. We formulate a classification problem using the ICE-India and the Santa Barbara Corpus (ICE-USA). Each corpus is divided into equal-size training and test sets. The training corpus was also used for hyperparameter selection for the dialect feature recognition models, as described in SS 3.2.

The dialect classifier was constructed by building on the components from SS 5.1. For the test set, we measure the \(D^{\prime}\) ("D-prime") statistic Macmillan and Creelman (1991),

\[D^{\prime}=\frac{\mu_{\text{IN}}-\mu_{\text{US}}}{\sqrt{\tfrac{1}{2}(\sigma_{ \text{IN}}^{2}+\sigma_{\text{US}}^{2})}}. \tag{1}\]

This statistic, which can be interpreted similarly to a \(Z\)-score, quantifies the extent to which a metric distinguishes between the two populations. We also report classification accuracy; lacking a clear way to set a threshold, for each classifier we balance the number of false positives and false negatives.

As shown in Table 6, both the document classifier and the corpus-based feature detection model (trained on labeled examples) achieve high accuracy at discriminating U.S. and Indian English. The \(D^{\prime}\) discriminability score is higher for the document classifier, which is trained on a cross-entropy objective that encourages making confident predictions. Regular expressions suffer from low precision because they respond to surface cues that may be present in U.S. English, even when the dialect feature is not present (e.g., the word "only", the phrase "is there").

## 6 Related Work

**Dialect classification.** Prior work on dialect in natural language processing has focused on distinguishing between dialects (and closely-related languages). For example, the VarDial 2014 shared task required systems to distinguish between nation-level language varieties, such as British versus U.S. English, as well as closely-related language pairs such as Indonesian versus Malay Zampieri et al. (2014); later evaluation campaigns expanded this set to other varieties Zampieri et al. (2017). In general, participants in these shared tasks have taken a text classification approach; neural architectures have appeared in the more recent editions of these shared tasks, but with a few exceptions (e.g., Bernier-Colborne et al., 2019), they have not outperformed classical techniques such as support vector machines. Our work differs by focusing on a specific set of known dialect features, rather than document-level classification between dialects, which aligns with the linguistic view of dialects as bundles of correlated features Nerbonne (2009) and tracks variable realization of features within dialect usage.

**Discovering and detecting dialect features.** Machine learning feature selection techniques have been employed to discover dialect features from corpora. For example, Dunn (2018, 2019) induces a set of _constructions_ (short sequences of words, parts-of-speech, or constituents) from a "neutral" corpus, and then identifies constructions with distinctive distributions over the geographical subcorpora of the International Corpus of English (ICE). In social media, features of African American Vernacular English (AAVE) can be identified by correlating linguistic frequencies with the aggregate demographic statistics of the geographical areas from which geotagged social media was posted Eisenstein et al. (2011); Stewart (2014); Blodgett et al. (2016). In contrast, we are interested in detecting predefined dialect features from well-validated resources such as dialect atlases.

Along these lines, Jorgensen et al. (2015) and Jones (2015) designed lexical patterns to identify non-standard spellings that match known phonological variables from AAVE (e.g., _sholl_ 'sure'), demonstrating the presence of these variables in social media posts from regions with high propor

\begin{table}
\begin{tabular}{p{113.8pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \hline  & \multicolumn{2}{c}{**Ranking**} & \multicolumn{2}{c}{**Classification**} \\ \cline{2-4}
**Diated density measure** & \(\rho\) & \(D^{\prime}\) & acc. \\ \hline Document classifier & \(-0.17\) & \(14.48\) & \(1\) \\ Multihead, corpus examples & \(0.83\) & \(2.30\) & \(0.95\) \\ Multihead, minimal pairs & \(0.70\) & \(1.85\) & \(0.85\) \\ Regular expressions & \(0.71\) & \(1.61\) & \(0.80\) \\ \hline \hline \end{tabular}
\end{table}
Table 6: Performance of dialect density measures at the tasks of ranking Indian English transcripts by dialect density (quantified by Spearman \(\rho\)) and distinguishing Indian and U.S. English transcripts (quantified by accuracy and \(D^{\prime}\) discriminability).



tions of African Americans. Blodgett et al. (2016) use the same geography-based approach to test for phonological spellings and constructions corresponding to syntactic variables such as habitual _be_; Hovy et al. (2015) show that a syntactic feature of Jutland Danish can be linked to the geographical origin of product reviews. These approaches have focused mainly on features that could be recognized directly from surface forms, or in some cases, from part-of-speech (POS) sequences. In contrast, we show that it is possible to learn to recognize features from examples, enabling the recognition of features for which it is difficult or impossible to craft surface or POS patterns.

Minimal pairs in NLP.A distinguishing aspect of our approach is the use of minimal pairs rather than conventional labeled data. Minimal pairs are well known in natural language processing from the Winograd Schema Levesque et al. (2012), which is traditionally used for evaluation, but Kocijan et al. (2019) show that fine-tuning on a related dataset of minimal pairs can improve performance on the Winograd Schema itself. A similar idea arises in counterfactually-augmented data Kaushik et al. (2019) and contrast sets Gardner et al. (2020), in which annotators are asked to identify the minimal change to an example that is sufficient to alter its label. However, those approaches use counterfactual examples to _augment_ an existing training set, while we propose minimal pairs as a replacement for large-scale labeled data. Minimal pairs have also been used to design controlled experiments and probe neural models' ability to capture various linguistic phenomena Gulordava et al. (2018); Ettinger et al. (2018); Futrell et al. (2019); Gardner et al. (2020); Schuster et al. (2020). Finally, Liang et al. (2020) use contrastive explanations as part of an active learning framework to improve data efficiency. Our work shares the objective of Liang et al. (2020) to improve data efficiency, but is methodologically closer to probing work that uses minimal pairs to represent specific linguistic features.

## 7 Conclusion

We introduce the task of dialect feature detection and demonstrate that it is possible to construct dialect feature recognizers using only a small number of minimal pairs -- in most cases, just five positive and negative examples per feature. This makes it possible to apply computational analysis to the many dialects for which labeled data does not exist. Future work will extend this approach to multiple dialects, focusing on cases in which features are shared across two or more dialects. This lays the groundwork for the creation of dialect-based "checklists" Ribeiro et al. (2020) to assess the performance of NLP systems across the diverse range of linguistic phenomena that may occur in any given language.

## 8 Ethical Considerations

Our objective in building dialect feature recognizers is to aid developers and researchers to effectively benchmark NLP model performance across and within different dialects, and to assist social scientists and dialectologists studying dialect use. The capability to detect dialectal features may enable developers to test for and mitigate any unintentional and undesirable biases in their models towards or against individuals speaking particular dialects. This is especially important because dialect density has been documented to correlate with lower socioeconomic status Sahgal and Agnihotri (1988). However, this technology is not without its risks. As some dialects correlate with ethnicities or countries of origin, there is a potential dual use risk of the technology being used to profile individuals. Dialect features could also be used as predictors in downstream tasks; as with other proxies of demographic information, this could give the appearance of improving accuracy while introducing spurious correlations and imposing disparate impacts on disadvantaged groups. Hence we recommend that developers of this technology consider downstream use cases, including malicious use and misuse, when assessing the social impact of deploying and sharing this technology.

The focus on predefined dialect features can introduce a potential source of bias if the feature set is oriented towards the speech of specific subcommunities within a dialect. However, analogous issues can arise in fully data-driven approaches, in which training corpora may also be biased towards subcommunities of speakers or writers. The feature-based approach has the advantage of making any such bias easier to identify and correct.

Acknowledgments.Thanks to Claudia Lange for sharing her annotations, and for discussion of this research. Thanks to Axel Bohmann for sharing information about his work on recognizing dialect features with regular expressions. Valuable feedback on this research was provided by Jason 

Baldridge, Dan Jurafsky, Slav Petrov, Jason Riesa, Kristina Toutanova, and especially Vera Axelrod. Thanks also to the anonymous reviewers. Devyani Sharma is supported in part by a Google Faculty Research Award.

## References

* Benor (2010) Sarah Bunin Benor. 2010. Ethnolinguistic repertoire: Shifting the analytic focus in language and ethnicity. _Journal of Sociolinguistics_, 14(2):159-183.
* Bernier-Colborne et al. (2019) Gabriel Bernier-Colborne, Cyril Goutte, and Serge Leger. 2019. Improving cuneiform language identification with BERT. In _Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects_, pages 17-25, Ann Arbor, Michigan. Association for Computational Linguistics.
* Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daume III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5454-5476, Online. Association for Computational Linguistics.
* Blodgett et al. (2016) Su Lin Blodgett, Lisa Green, and Brendan O'Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1119-1130.
* Bohmann (2019) Axel Bohmann. 2019. _Variation in English worldwide: Registers and global varieties_. Cambridge University Press, Cambridge.
* Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _arXiv preprint arXiv:2005.14165_.
* Cohen (1960) Jacob Cohen. 1960. A coefficient of agreement for nominal scales. _Educational and psychological measurement_, 20(1):37-46.
* Craig and Washington (2002) Holly K Craig and Julie A Washington. 2002. Oral Language Expectations for African American Preschoolers and Kindergartners. _American Journal of Speech-Language Pathology_, 11(1):59-70.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186.
* Dois et al. (2000) John W Du Bois, Wallace L Chafe, Charles Meyer, Sandra A Thompson, and Nii Martey. 2000. Santa Barbara Corpus of Spoken American English. _CDROML Philadelphia: Linguistic Data Consortium_.
* Dunn (2018) Jonathan Dunn. 2018. Finding variants for construction-based dialectometry: A corpus-based approach to regional cxgs. _Cognitive Linguistics_, 29(2):275-311.
* Dunn (2019) Jonathan Dunn. 2019. Modeling global syntactic variation in English using dialect classification. In _Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects_, pages 42-53, Ann Arbor, Michigan. Association for Computational Linguistics.
* Eisenstein et al. (2011) Jacob Eisenstein, Noah A. Smith, and Eric P. Xing. 2011. Discovering sociolinguistic associations with structured sparsity. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, pages 1365-1374, Portland, Oregon, USA. Association for Computational Linguistics.
* Ettinger et al. (2018) Allysson Ettinger, Ahmed Elgohary, Colin Phillips, and Philip Resnik. 2018. Assessing Composition in Sentence Vector Representations. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 1790-1801.
* Futrell et al. (2019) Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 32-42.
* Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Eizar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaeit, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Alley Zhang, and Ben Zhou. 2020. Evaluating Models' Local Decision Boundaries via Contrast Sets.
* Greenbaum and Nelson (1996) Sidney Greenbaum and Gerald Nelson. 1996. The international corpus of English (ICE) project. _World Englishes_, 15(1):3-15.
* Groggger et al. (2020) Jeffrey Groggger, Andreas Steinmayr, and Joachim Winter. 2020. The wage penalty of regional accents. NBER Working Papers 26719, National Bureau of Economic Research, Inc.
* Gulordava et al. (2018) Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless Green Recurrent Networks Dream Hierarchically. In _Proceedings of the 2018 Conference of the North 

American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1195-1205.
* Hovy et al. (2015) Dirk Hovy, Anders Johannsen, and Anders Sogaard. 2015. User review sites as a resource for large-scale sociolinguistic studies. In _Proceedings of the 24th international conference on World Wide Web_, pages 452-461.
* Jones (2015) Taylor Jones. 2015. Toward a description of african american vernacular english dialect regions using "black twitter". _American Speech_, 90(4):403-440.
* Jorgensen et al. (2015) Anna Jorgensen, Dirk Hovy, and Anders Sogaard. 2015. Challenges of studying and processing dialects in social media. In _Proceedings of the workshop on noisy user-generated text_, pages 9-18.
* Kaushik et al. (2019) Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2019. Learning The Difference That Makes A Difference With Counterfactually-Augmented Data. In _International Conference on Learning Representations_.
* Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Kocijan et al. (2019) Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. 2019. A surprisingly robust trick for the winograd schema challenge. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4837-4842.
* Koenecke et al. (2020) Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R Rickford, Dan Jurafsky, and Sharad Goel. 2020. Racial disparities in automated speech recognition. _Proceedings of the National Academy of Sciences_, 117(14):7684-7689.
* Labov (1972) William Labov. 1972. _Sociolinguistic patterns_. 4. University of Pennsylvania Press.
* Lange (2012) Claudia Lange. 2012. _The syntax of spoken Indian English_. John Benjamins Publishing Company, Amsterdam.
* Levesque et al. (2012) Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In _Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning_. Citeseer.
* Liang et al. (2020) Weixin Liang, James Zou, and Zhou Yu. 2020. Alice: Active learning with contrastive natural language explanations. _arXiv preprint arXiv:2009.10259_.
* Logeswaran et al. (2019) Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-shot entity linking by reading entity descriptions. _arXiv preprint arXiv:1906.07348_.
* Macmillan and Creelman (1991) Neil A Macmillan and C Douglas Creelman. 1991. _Detection theory: A user's guide_. Cambridge University Press.
* Nerbonne (2009) John Nerbonne. 2009. Data-driven dialectology. _Language and Linguistics Compass_, 3(1):175-198.
* Nerbonne et al. (2011) John Nerbonne, Rinke Colen, Charlotte Gooskens, Peter Kleiweg, and Therese Leinonen. 2011. Gabmap-a web application for dialectology. _Dialectologia: revista electronica_, pages 65-89.
* Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In _Proceeding of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4902-4912, Online. Association for Computational Linguistics.
* Rickford et al. (2015) John R. Rickford, Greg J. Duncan, Lisa A. Gennetian, Ray Yun Gou, Rebecca Greene, Lawrence F. Katz, Ronald C. Kessler, Jeffrey R. Kling, Lisa Sanbonmatsu, Andres E. Sanchez-Ordonez, Matthew Sciandra, Ewart Thomas, and Jesus Ludwig. 2015. Neighborhood effects on use of african-american vernacular english. _Proceedings of the National Academy of Sciences_, 112(38):11817-11822.
* Sahgal and Agnihotri (1988) Anju Sahgal and R. K. Agnihotri. 1988. Indian English phonology: A sociolinguistic perspective. _English World-Wide_, 9(1):51-64.
* Schuster et al. (2020) Sebastian Schuster, Yuking Chen, and Judith Degen. 2020. Harnessing the Linguistic Signal to Predict Scalar Inferences. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5387-5403.
* Sharma (2009)Devvani Sharma. 2009. Typological diversity in New Englishes. _English World-Wide_, 30(2):170-195.
* Stewart (2014) Ian Stewart. 2014. Now we stronger than ever: African-American English syntax in Twitter. In _Proceedings of the Student Research Workshop at the 14th Conference of the European Chapter of the Association for Computational Linguistics_, pages 31-37, Gothenburg, Sweden. Association for Computational Linguistics.
* Hofwegen and Wolfram (2010) Janheke Van Hofwegen and Walt Wolfram. 2010. Coming of age in African American English: A longitudinal study. _Journal of Sociolinguistics_, 14(4):427-455.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in neural information processing systems_, pages 5998-6008.
* Zampieri et al. (2017) Marcos Zampieri, Shervin Malmasi, Nikola Ljubesic, Preslav Nakov, Ahmed Ali, Jorg Tiedemann, Yves Scherrer, and Noemi Aepli. 2017. Findings of the VarDial evaluation campaign 2017. In _Proceedings 

of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial)_, pages 1-15, Valencia, Spain. Association for Computational Linguistics.
* Zampieri et al. [2014] Marcos Zampieri, Liling Tan, Nikola Ljubesic, and Jorg Tiedemann. 2014. A report on the DSL shared task 2014. In _Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects_, pages 58-67, Dublin, Ireland. Association for Computational Linguistics and Dublin City University.

 



## Appendix A Regular Expressions

Table 7 shows the regular expressions that we used for the five features, where such patterns were available.

\begin{table}
\begin{tabular}{l l} \hline \hline
**Feature** & **Regular expression** \\ \hline focus itself & \textbackslash{bitself\}b \\ focus only & \textbackslash{bonly\}b \\ non-initial existential & \textbackslash{bitsereb\|\backslash{barethere\}b} \\ invariant tag (_isin’t it, no, na_) & \textbackslash{bin’t itb\|\textbackslash{bis itb\|\textbackslash{bnob}b\|\textbackslash{ bna}b}} \\ general extender and all & \textbackslash{bandall\}b \\ \hline \hline \end{tabular}
\end{table}
Table 7: Regular expressions we used, for the features that such patterns were available.



Sample Outputs

The examples below represent a random sample of the multihead models' outputs for Lange's features, comparing the one that is trained on corpus examples (corpus) to the one that is trained on minimal pairs (minpair). We show true positives (tp), false positives (fp) and false negatives (fn). We randomly sample three examples for each output type (tp, fp, fn) and model (both, corpus only, minpair only).

Our manual inspection shows a few errors in the human annotation by Lange and that certain false positives should be true positives, especially for focus_only_. We highlight such examples in green. Among the rest of the false positives and false negatives, a large proportion of errors can be explained by contextual information that is not available to the models. For example, without context it is ambiguous whether "we possess only" is an example of focus_only_. Inspection of context shows that it is a truncated utterance, representing a standard use of _only_, hence it is correctly characterized as a false positive. Another source of confusion to the model is missing punctuation. For example "Both girls I have never left them alone till now" could be construed as object fronting with resumptive object pronoun. However, in the original context, the example consists of multiple sentences: "Two kids. Both girls. I have never left them alone till now." We removed punctuation from examples, since in many cases automatic ASR models do not produce punctuation either. However, this example demonstrates that punctuation can provide valuable information about clause and phrase boundaries, and should be included if possible.

### Focus itself

[tp:both] We are feeling tired now itself

[tp:both] Coach means they should be coached from when they are in nursery UKG itself

[tp:both] I'm in final year but like they have started from first year itself

[tp:corpus] And she got a chance of operating also during her internship itself nice and because that Coma hospital is for ladies only so she has lot of experience

[tp:minpair] But even if they women is are working as much as a man she is earning the same monthly saving as a man itself

[tp:minpair] You go around say one O'clock and then go for a movie and come back in the evening itself you see you

[fp:minpair] And primarily you know the the issue orders were issued on fifth that is on the election day itself

[fp:minpair] That is to we take on the coughs our human blood itself.

[fp:minpair] Now since you are doing the PGCT now after going back is it possible for you to use simple English in the classroom itself

[fn:both] All the sums were there in the text book itself but still they have not done properly in the exam

[fn:both] And thinking about dissection hall itself they really get scared and that also in the midnight

[fn:both] Means what do you think that the basic itself is not good or now they are getting interest in maths

[fn:corpus] But even if they women is are working as much as a man she is earning the same monthly saving as a man itself

[fn:corpus] You go around say one O'clock and then go for a movie and come back in the evening itself you see you

[fn:minpair] And she got a chance of operating also during her internship itself nice and because that Coma hospital is for ladies only so she has lot of experience

### Focus _only_

[tp:both] All the types only

[tp:both] Hey you sur be like that only

[tp:both] suddenly it will be become perfect only

[tp:corpus] That is I like dressing up I told you at the beginning only

[tp:corpus] Because today only he had come and I've got up today at nine thirty

[tp:corpus] Actually from childhood only I was brought up in the same atmosphere like if Papa still has shifted to another place I would have got the feeling of not having comfortable in a particular language but on the whole I think it doesn't matter exactly how we go about chosing or selecting a language 

[tp:minpair] it was bit it was difficult only

[tp:minpair] I'm one minute I've got it in front of me only

[tp:minpair] He is in our college only

[fp:both] Because we are supposed to perform well there only then

[fp:both] Ho Hollywood Hollywood after Hollywood it seems India only.

[fp:both] No he'll be there in the campus only

[fp:corpus] Oh God there only it's happening so and forget about

[fp:corpus] The thing is that it is rural area

only but the people are from all over india

they are staying here

[fp:corpus] Not much work these days because first week and last week only we've quiet good business

[fp:minpair] Only in India there is manual work

[fp:minpair] Film hits only

[fp:minpair] So Bharati Vidya Bhavan people have such type of persons only

[fn:both] If they be in always that this is there are not improve no improvement only

[fn:both] When we were living when I was living in Kashmir no I was brought up there only and everything is

[fn:both] This is the first phase then in the second phase we have some clinical subjects in which we come in direct contact with the patients but it's on two basis like when we see the patients at the same time we study about the pathology only the pathology and then we learn about some of the drugs which are to be which are used for their treatment

[fn:corpus] No you must put apply science only

[fn:corpus] Actually they are good only

[fn:corpus] it was bit it was difficult only

[fn:minpair] My both the parents are farmers only

[fn:minpair] Because today only he had come and I've got up today at nine thirty

[fn:minpair] That is I like dressing up I told you at the beginning only

### Invariant Tag (_isn't ii, no, na)_

[tp:bothi] Very difficult once the school starts na very difficult

[tp:bothi] I am okay rainy season no

[tp:bothi] Oh yours your head is not reeling any more no ?

[tp:corpus] Kind of but it would be better than an indoor game no

[tp:corpus] We'll ask that person no that Sagar you can tell

[tp:corpus] Nothing at all that's why you got scratching on that day I know that no that's why I asked

[tp:minpair] I'm not fair no

[tp:minpair] Husband no I'll do I'll prepare it

[tp:minpair] He could have agreed no what is that

[fp:bothi] TELCO data hai to kuch problem nai na

[fp:bothi] I think once you have got in you no

[fp:bothi] I didn't go anywhere no

[fp:corpus] Or two hundred rupees that no

[fp:corpus] Know when we go back no I think we'll get a rosy welcome home welcome there

[fp:corpus] I like straight and perspiration then only I feel at home otherwise no

[fp:minpair] No got it repaired

[fp:minpair] No no he is here

[fp:minpair] Okay no but

[fn:bothi] I just go out for tea isn't

[fn:bothi] Hey you you like serious movies is it you like serious movies

[fn:bothi] See no the scene exactly happened you know the other day what happen I was reading baba

[fn:corpus] I'm not fair no

[fn:corpus] I think no

[fn:corpus] Tell me no why you can't tell

[fn:minpair] Yeah then it's first time first time it was new to me no

[fn:minpair] That is the main thing na here that would again the main thing that they don't take at all interest in the their children at all 

[fn:minpair] So culture nahii hai there is I don't follow culture religion nothing na

### Lack of Copula

[fp:corpus] Which October first I think

[fp:corpus] June nineteen eighty-six

[fp:minpair] Construction all before

[fp:minpair] Not in the class

[fp:minpair] The tendency to

[fn:both] you've she said his grandfather still working

[fn:both] Everybody so worried about the exams and studies

[fn:both] Again classes bit too long I feel five O'clock is tiring

### Left Dislocation

[tp:both] This principal she is very particular about it

[tp:both] Vilas and Ramesh they they make noise man

[tp:both] That's why those Muslims they got very angry

[tp:corpus] And med medium class they can't understand soon

[tp:corpus] That will become difficult and common people they don't understand

[tp:corpus] And now the Kukis they refused to pay any more

[tp:minpair] It's because of this some other participant they complained about this and then they started they started this particular

[tp:minpair] We've lot of fun in theatres you know we always take the back seat and all that for this guys distinct one we keep teasing them

[tp:minpair] My post graduation degree I finished it in mid June nineteen eighty-six

[fp:both] But whereas when they really come to know the people they like to help the people

[fp:both] It's actually some of them like to see it really so huge and long and bigger snakes they are in all closed and all there it is nice to see it

[fp:both] But generally the educated people I don't find much variation but in accent there may be a variation

[fp:corpus] Everytime he keeps speaking you know they get irritated and say aram se

[fp:corpus] What happened is they will change programme and the fifty guys they'll just keep quite

[fp:corpus] Whereas Hyderabad the people are more conservative and like they don't like to go out even or at the first move they don't like to talk with people also

[fp:minpair] And the songs now once we hear it afterwards when some other famous songs comes that we forget the last ones

[fp:minpair] But when we approach since it seems they they put lot of conditions yes that you fed up with those people and

[fp:minpair] so that's why we missed we that missed that holiday it being a Sunday

[fn:both] Administration it is all done by Bharati Vidya Bhavan

[fn:both] Oh our Joshi okay II got got him

[fn:both] Yes yes it is true but our constitution makers

[fn:corpus] and he has used the the place where the palace once palace might be there and that portion and the remaining part he built an antenna he has fixed it there at the top

[fn:corpus] Not exactly but Calcutta sweets I think they do have a little flavour and that I haven't got anywhere in India

[fn:corpus] Computer it was in the first semester

[fn:minpair] And med medium class they can't understand soon

[fn:minpair] Shireen she was excellent at that

[fn:minpair] Yeah arti arti students they loiter about in the corridor

### Non-initial Existential _X is / are there_

[tp:both] Libraries are there

[tp:both] only specimen like operated cases like supposing a is there

[tp:both] Problems are there problems are there what

[tp:corpus] to assist there some teachers are there and together we conduct the classes

[tp:corpus] It's there but it's common no 

[TP:corpus] Yeah I think Varalaxmi is there

[FP:both] My husband is there mother is there

[FP:corpus] Come no Shaukat is here Natalie is here even if Savita is not there they two are there na

[FP:corpus] Actually there the thing is that you know for example

[FP:corpus] Any thing is there produced materials which do not require much resource personnel

[FP:minpair] Ph D degree is awarded there

[fn:both] Yeah the royalties too there they're there and we've the king

[fn:both] Okay somebody else's some somebody else is there

[fn:both] In that you know everything is about nature I'll tell you yeah it's very lovely means very nice lovely what but and small children were there in that

[FN:minpair] American and all other capitalist nations were also there

[FN:minpair] Nice movie yaar that song is there no hai apna dil to awara

[FN:minpair] It's not there

### Object Fronting

[tp:both] Just typing work I have to do

[tp:corpus] writing skills there are so many you can teach them

[tp:corpus] Each other and so many things we have learnt

[tp:corpus] My birthday party you arrange

[fp:corpus] Formalities I will come

[fp:corpus] Mar Marxism you were

[ fp:minpair] Other wise we have to

[fn:both] That also I'm not having just I jump jumped jumps I came studies also

[fn:both] Yes Hawa Mahal we heard

[fn:both] About ten to twenty books I'll read that's all

[fn:minpair] Small baby very nice it was

[fn:minpair] But more keen she is

[fn:minpair] And camera handling actually outdoor landscing that landscape shot I have taken and actually the close ups and some parts of your architectural shots of that building Ganesh took my husband took and close ups of the faces my husband and Ganesh took

### Resumptive Object Pronoun

[tp:minpair] and he has used the the place where the palace once palace might be there and that portion and the remaining part he built an antenna he has fixed it there at the top

[tp:minpair] Yeeah also pickles we eat it with this jaggery and lot of butter

[tp:minpair] My post graduation degree I finished it in mid June nineteen eighty-six

[fp:minpair] Having humurous something special I would love it to join it

[fp:minpair] I see a number of people I like them very much

[fp:minpair] Old and ancient things in carving we get it so bautifully

[fn:both] Oh our Joshi okay II got got him

[fn:both] Normaly no we don't overdrawn on account but haan haan whatever is balance you know yeah help them give them suppose cheque books and all we are supposed to keep them yeah two fifty balance

[fn:both] He is in a that's what he was telling me today see I want your draft like draft draft by January by the month of January by the end of January so that II might rectify it and then I will do it I will give it back to you by mid Febraury so that you can get it final draft by by the end of Febraury

[fn:corpus] and he has used the the place where the palace once palace might be there and that portion and the remaining part he built an antenna he has fixed it there at the top

[fn:corpus] Yeah also pickles we eat it with this jaggerly and lot of butter

[fn:corpus] My post graduation degree I finished it in mid June nineteen eighty-six

### Resumptive Subject Pronoun

[tp:corpus] Like those terrorists they wanted us to to accompany them in the revol against India

[tp:corpus] And one more thing another thing how I rectified myself because all almost all of us all my brother and sisters we have read in English medium school 

[Tp:corpus] Dr this Mr V he was totally changed actually because he was the concepts are clear not clear to us

[fp:corpus] There are so many people they can they could shine like anything

[fp:corpus] Kolhapur he had come to Guwahati [fp:corpus] I don't know what he whenever whenever I see those guys they they nicely speak to me

[fp:minpair] His house he is going to college KK diploma electronics

[fn:both] they I thought that another one Patil is there a horrible he is I thought that Patil

[fn:both] Computer it it plays a great role because we are having computers in each field now-a-days

[fn:both] You know that a woman she is a apprehensive about many things

[fn:minpair] Like those terrorists they wanted us to to accompany them in the revol against India

[fn:minpair] Whereas in Hyderabad they still have the old cultures and so many things that even the parents they don't even let the girls talk with the guys

[fn:minpair] And the students who come out with a degree MMSI understand that there is a report that has been received from different firms that the students of BITS Pilani specially MMS candidates they are prepared to soil their hands

### Topicalized Non-argument Constituent

[tp:corpus] for Diwali you went I know that

[tp:corpus] So very long time we have not travelled together

[tp:corpus] Pooja vacation also we used to conduct some classes practical classes

[tp:minpair] In pooja day some important days we stay back

[ fp:corpus] In Jaipur then we have also we have a Birla

[fp:corpus] Like that we

[ fp:corpus] Everytime we have some work to do

[fp:minpair] Aa i i initial periods I did very difficult but I

[fn:both] I mean here in Hyderabad the people are it's okay they are nice

[fn:both] And that old ones again we put them we feel like hearing again

[fn:both] But in drama we'll have to be very different

[fn:corpus] In pooja day some important days we stay back

[fn:minpair] for Diwali you went I know that

[fn:minpair] Pooja vacation also we used to conduct some classes practical classes

[fn:minpair] Sir from Monday onwards I too want to take leave sir for four days because total I have five C Ls so from 

Average Precision Results

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Supervision:** & \multicolumn{3}{c}{**Corpus examples**} & \multicolumn{2}{c}{**Minimal pairs**} \\ Dialect feature & DAMTL & Multihead & DAMTL & Multihead \\ \hline focus _inset/*_ & 0.668 & 0.631 & 0.665 & 0.613 \\ Focus only* & 0.582 & 0.404 & 0.344 & 0.416 \\ InVARIANT Tag & 0.786 & 0.871 & 0.441 & 0.495 \\ copula omission & 0.029 & 0.015 & 0.012 & 0.036 \\ Left DISICATION & 0.425 & 0.383 & 0.149 & 0.232 \\ Non-INITual ENSTITAL* & 0.887 & 0.906 & 0.556 & 0.510 \\ object fronting & 0.238 & 0.202 & 0.031 & 0.083 \\ res. Object pronoun & 0.052 & 0.020 & 0.046 & 0.061 \\ Res. subset pronoun & 0.460 & 0.409 & 0.078 & 0.198 \\ topICALized non-arg. consist & 0.080 & 0.076 & 0.021 & 0.044 \\
**Macro Average** & 0.430 & 0.392 & 0.234 & 0.269 \\ \hline \end{tabular}
\end{table}
Table 8: Average precision for the Lange features. Scores are in the range \([0,1]\), with \(1\) indicating perfect performance. Asterisks mark features that can be recognized with a regular expression.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Dialet feature** & **DaMTL** & **Multihead** \\ \hline article emission & 0.210 & 0.308 \\ direct object pro-drop & 0.044 & 0.057 \\ extraneous article & 0.116 & 0.065 \\ focus itself* & 1.000 & 0.853 \\ Focus only* & 0.859 & 0.274 \\ Habitat progressive & 0.008 & 0.020 \\ Invariant tag & 0.614 & 0.420 \\ inversion in Embedded clause & 0.106 & 0.162 \\ lack of agreement & 0.084 & 0.110 \\ lack of investigation NWI-questions & 0.309 & 0.106 \\ left DISLocation & 0.288 & 0.301 \\ mass nouns as count nouns & 0.045 & 0.034 \\ non-INITIAL existential* & 0.506 & 0.397 \\ object promoting & 0.147 & 0.193 \\ preposition omission & 0.064 & 0.116 \\ pP promoting with reduction & 0.091 & 0.134 \\ static progressive & 0.267 & 0.329 \\ general extender and all & 0.769 & 0.778 \\
**Macro Average** & 0.307 & 0.259 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Average precision for the extended feature set. As described in the main text, corpus training examples are unavailable for these features.

 



## Appendix D Minimal pairs

\begin{tabular}{r l r c} **ID** & **Feature** & **Example** & **Label** \\ \hline

[MISSING_PAGE_POST]

e likes here more & 1 \\
12 & DIRECT object PRO-DROP & he likes it her more & 0 \\
13 & focus _itself_ & So if you're not good at communication you may get filtered & 1 \\  & & at the first level itself & 1 \\
14 & focus _itself_ & So if you're not good at communication you may get filtered & 0 \\  & & at even the first level & \\
15 & focus _itself_ & But I did have some difficulty getting to know people among & 1 \\  & & Indians itself & \\
16 & focus _itself_ & But I did have some difficulty getting to know people among & 0 \\  & & Indians themselves & \\
17 & focus _itself_ & It think you should start going to the gym from now itself. & 1 \\
18 & focus _itself_ & I think you should start going to the gym from now. & 0 \\
19 & focus _itself_ & I did one refresher course in the month of June itself. & 1 \\
10 & focus _itself_ & If do one refresher course in the month of June. & 0 \\
11 & focus _itself_ & He is doing Engineering in Delhi itself. & 1 \\
12 & focus _itself_ & He is doing Engineering in Delhi. & 0 \\
13 & focus _only_ & I'm working very nearby to my house only & 1 \\
14 & focus _only_ & I'm working very near my house & 0 \\
15 & focus _only_ & recently only in April there was a big fight & 1 \\
16 & focus _only_ & as recently as April there was a big fight & 0 \\
17 & focus _only_ & I was there yesterday only & 1 \\
18 & focus _only_ & I was three just yesterday & 0 \\
19 & focus _only_ & She was brought up there and her college was there only & 1 \\
19 & focus _only_ & She was brought up there and her college was there too & 0 \\
20 & focus _only_ & You get on the train and buy the ticket there only & 1 \\
20 & Focus _only_ & You get on the train and buy the ticket there too & 0 \\
21 & HABITUAL PROGRESSive & anybody giving donation, we are giving receipt & 1 \\
22 & HABITUAL PROGRESSIVE & if anybody gives a donation, we give a receipt & 0 \\
22 & HABITUAL PROGRESSive & she is getting nightmares & 1 \\
23 & HABITUAL PROGRESSIVE & they are getting H1B visas to come to the country & 1 \\
23 & HABITUAL PROGRESSive & they get H1B visas to come to the country & 0 \\
24 & HABITUAL PROGRESSive & they are teasing the new children when they join & 1 \\
24 & HABITUAL PROGRESSive & they tease the new children when they join & 0 \\
25 & HABITUAL PROGRESSive & everyone is getting that vaccination in childhood & 1 \\
25 & HABITUAL PROGRESSive & everyone gets that vaccination in childhood & 0 \\
26 & invariant TAG (isn’t it, no, na) & the children are playing outside, isn't it? & 1 \\
26 & invariant TAG (isn’t it, no, na) & the children are playing outside, na? & 1 \\
26 & invariant TAG (isn’t it, no, na) & the children are playing outside, aren't they? & 0 \\
27 & invariant TAG (isn’t it, no, na) & I was very scared to, no? & 1 \\
27 & invariant TAG (isn’t it, no, na) & I was very scared to, na? & 1 \\



[MISSING_PAGE_POST]

 

* 59non-initial existential fire balls are there there are five balls 0
* 60non-initial existential every year inflation is there 1
* 60non-initial existential every year there is inflation 0
* 61objectfronting not so much adjustment i have to make 1
* 61objectfronting i don't have to make so much adjustment 0
* 62objectfronting minimum one month you have to wait 1
* 63objectfronting you have to wait a minimum of one month
* 64objectfronting in fifteen years lot of changes we have seen 1
* 65objectfronting tomorrow this cake you have to try 0
* 66preposition omission I can see some green colour leaves the left side 0
* 67preposition omission I went one year there. 1
* 68preposition omission We don't feel that we should go any other country. 1
* 69preposition omission Those days it was considered a good job. 1
* 69preposition omission In those days it was considered a good job. 0
* 70preposition omission So that time they said okay go and work for a few months. 1
* 71ppromoting with reduction first of all, right side we can see a plate 1
* 71pp frontning with reduction first of all, we can see a plate the right side 0
* 72pr frontning with reduction first of all, we can see a plate on the right side 0
* 73pr frontning with reduction first of all, right side we can see a plate 1
* 74article omission First of all, right side we can see a plate 1
* 75proposition omission first of all, right side we can see a plate 1
* 76preposition omission First of all, we can see a plate the right side 1
* 77pr fronting with reduction First of all, we can see a plate the right side 1
* 78pr frontning with reduction First of all, right side we can see a plate 1
* 79pr frontning with reduction First of all, we can see a plate the right side 1
* 71pr frontning with reduction First of all, right side we can see a plate 1
* 71pr frontning with reduction First of all, we can see a plate the right side 1
* 72pr frontning with reduction First of all, right side we can see a plate 1
* 72pr frontning with reduction First of all, right side we can see a plate 1
* 72pr frontning with reduction First of all, we can see a plate the right side 1
* 72pr frontning with reduction First of all, right side we can see a plate 1

[MISSING_PAGE_POST]

 



[MISSING_PAGE_POST]

* 89copula omission
* 89copula
 

* [104] resumptive subject pronoun
* [104] left dislocation and
* [105] resumptive subject pronoun
* [105] resumptive subject pronoun
* [106] left disloccation
* [107] topicalized non-arg. const
* [108] topicalized non-arg. const
* [109] Topicalized non-arg. const
* [110] PP fronting with reduction

[MISSING_PAGE_POST]