SlowGuess/NAACL_2021_OCR huggingface dataset:
Neil A Macmillan and C Douglas Creelman. 1991. Detection theory: A user's guide. Cambridge University Press.
John Nerbonne. 2009. Data-driven dialectology. Language and Linguistics Compass, 3(1):175-198.
John Nerbonne, Rinke Colen, Charlotte Gooskens, Peter Kleiweg, and Therese Leinonen. 2011. Gabmapa web application for dialectology. Dialectologia: revista electrònica, pages 65-89.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902-4912, Online. Association for Computational Linguistics.
John R. Rickford, Greg J. Duncan, Lisa A. Gennetian, Ray Yun Gou, Rebecca Greene, Lawrence F. Katz, Ronald C. Kessler, Jeffrey R. Kling, Lisa Sanbonmatsu, Andres E. Sanchez-Ordoñez, Matthew Scian-dra, Ewart Thomas, and Jens Ludwig. 2015. Neighborhood effects on use of african-american vernacular english. Proceedings of the National Academy of Sciences, 112(38):11817-11822.
Anju Sahgal and R. K. Agnihotri. 1988. Indian English phonology: A sociolinguistic perspective. English World-Wide, 9(1):51-64.
Sebastian Schuster, Yuxing Chen, and Judith Degen. 2020. Harnessing the Linguistic Signal to Predict Scalar Inferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5387-5403.
Devyani Sharma. 2009. Typological diversity in New Englishes. English World-Wide, 30(2):170-195.
Ian Stewart. 2014. Now we stronger than ever: African-American English syntax in Twitter. In Proceedings of the Student Research Workshop at the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 31-37, Gothenburg, Sweden. Association for Computational Linguistics.
Janneke Van Hofwegen and Walt Wolfram. 2010. Coming of age in African American English: A longitudinal study. Journal of Sociolinguistics, 14(4):427-455.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Marcos Zampieri, Shervin Malmasi, Nikola Ljubesic, Preslav Nakov, Ahmed Ali, Jorg Tiedemann, Yves Scherrer, and Noemi Aepli. 2017. Findings of the VarDial evaluation campaign 2017. In Proceedings
of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), pages 1-15, Valencia, Spain. Association for Computational Linguistics.
Marcos Zampieri, Liling Tan, Nikola Ljubesic, and Jörg Tiedemann. 2014. A report on the DSL shared task 2014. In Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects, pages 58-67, Dublin, Ireland. Association for Computational Linguistics and Dublin City University.
Feature	Regular expression
FOCUS itself	\bitusself\b
FOCUS only	\bonly\b
NON-INITIAL EXISTENTIAL	\bis there\b|\bare there\b
INVARIANT TAG (isn't it, no, na)	\bisn't it\b|\bisis it\b|\bno\b|\bna\b
GENERAL EXTENDER and all	\band all\b
Table 7: Regular expressions we used, for the features that such patterns were available.
A Regular Expressions
Table 7 shows the regular expressions that we used for the five features, where such patterns were available.
B Sample Outputs
The examples below represent a random sample of the multihead models' outputs for Lange's features, comparing the one that is trained on corpus examples (CORPUS) to the one that is trained on minimal pairs (MINPAIR). We show true positives (TP), false positives (FP) and false negatives (FN). We randomly sample three examples for each output type (TP, FP, FN) and model (BOTH, CORPUS only, MINPAIR only).
Our manual inspection shows a few errors in the human annotation by Lange and that certain false positives should be true positives, especially for FOCUS only. We highlight such examples in green. Among the rest of the false positives and false negatives, a large proportion of errors can be explained by contextual information that is not available to the models. For example, without context it is ambiguous whether "we possess only" is an example of FOCUS only. Inspection of context shows that it is a truncated utterance, representing a standard use of only, hence it is correctly characterized as a false positive. Another source of confusion to the model is missing punctuation. For example "Both girls I have never left them alone till now" could be construed as OBJECT FRONTING with RESUMP-TIVE OBJECT PRONOUN. However, in the original context, the example consists of multiple sentences: "Two kids. Both girls. I have never left them alone till now." We removed punctuation from examples, since in many cases automatic ASR models do not produce punctuation either. However, this example demonstrates that punctuation can provide valuable information about clause and phrase boundaries, and should be included if possible.
B.1 Focus itself
[TP:BOTH] We are feeling tired now itself
[TP:BOTH] Coach means they should be coached from when they are in nursery UKG itself
[TP:BOTH] I'm in final year but like they have started from first year itself
[TP:CORPUS] And she got a chance of operating also during her internship itself nice and because that Cama hospital is for ladies only so she has lot of experience
[TP:MINPAIR] But even if they women is are working as much as a man she is earning the same monthly saving as a man itself
[TP:MINPAIR] You go around say one O'clock and then go for a movie and come back in the evening itself you see you
[FP:MINPAIR] And primarily you know the the issue orders were issued on fifth that is on the election day itself
[FP:MINPAIR] That is to we take on the coughs our human blood itself
[FP:MINPAIR] Now since you are doing the PGCT now after going back is it possible for you to use simple English in the classroom itself
[FN: BOTH] All the sums were there in the text book itself but still they have not done properly in the exam
[FN: BOTH] And thinking about dissection hall itself they really get scared and that also in the midnight
[FN: BOTH] Means what do you think that the basic itself is not good or now they are getting interest in maths
[FN:CORPUS] But even if they women is are working as much as a man she is earning the same monthly saving as a man itself
[FN:CORPUS] You go around say one O'clock and then go for a movie and come back in the evening itself you see you
[FN:MINPAIR] And she got a chance of operating also during her internship itself nice and because that Cama hospital is for ladies only so she has lot of experience
B.2 Focus only
[TP:BOTH] All the types only
[TP:BOTH] Hey you sur be like that only
[TP:BOTH] suddenly it will be become perfect only
[TP:CORPUS] That is I like dressing up I told you at the beginning only
[TP:CORPUS] Because today only he had come and I've got up today at nine thirty
[TP:CORPUS] Actually from childhood only I was brought up in the same atmosphere like if Papa still has shifted to another place I would have got the feeling of not having comfortable in a particular language but on the whole I think it doesn't matter exactly how we go about chosing or selecting a language
[TP:MINPAIR] it was bit it was difficult only
[TP:MINPAIR] I'm one minute I've got it in front of me only
[TP:MINPAIR] He is in our college only
[FP:BOTH] Because we are supposed to perform well there only then
[FP:BOTH] Ho Ho Hollywood Hollywood after  
Hollywood it seems India only
[FP:BOTH] No he'll be there in the campus only
[FP:CORPUS] Oh God there only it's happening so and forget about
[FP:CORPUS] The thing is that it is rural area only but the people are from all over india they are staying here
[FP:CORPUS] Not much work these days because first week and last week only we've quiet good business
[FP:MINPAIR] Only in India there is manual work
[FP:MINPAIR] Film hits only
[FP:MINPAIR] So Bharati Vidya Bhavan people have such type of persons only
[FN:BOTH] If they be in always that this is there are not improve no improvement only
[FN:BOTH] When we were living when I was living in Kashmir no I was brought up there only and everything is
[FN: BOTH] This is the first phase then in the second phase we have some clinical subjects in which we come in direct contact with the patients but it's on two basis like when we see the patients at the same time we study about the pathology only the pathology and then we learn about some of the drugs which are to be which are used for their treatment
[FN:CORPUS] No you must put apply science only
[FN:CORPUS] Actually they are good only
[FN:CORPUS] it was bit it was difficult only
[FN:MINPAIR] My both the parents are farmers only
[FN:MINPAIR] Because today only he had come and I've got up today at nine thirty
[FN:MINPAIR] That is I like dressing up I told you at the beginning only
B.3 Invariant Tag (isn't it, no, na)
[TP:BOTH] Very difficult once the school starts na very difficult
[TP:BOTH] I am okay rainy season no
[TP:Both] Oh yours your head is not reeling any more no?
[TP:CORPUS] Kind of but it would be better than an indoor game no
[TP:CORPUS] We'll ask that person no that Sagar you can tell
[TP:CORPUS] Nothing at all that's why you got scratching on that day I know that no that's why I asked
[TP:MINPAIR] I'm not fair no
[TP:MINPAIR] Husband no I'll do I'll prepare it
[TP:MINPAIR] He could have agreed no what is that
[FP:BOTH] TELCO deta hai to kuch problem nahi na
[FP:BOTH] I think once you have got in you no
[FP:BOTH] I didn't go anywhere no
[FP:CORPUS] Or two hundred rupees that no
[FP:CORPUS] Know when we go back no I think we'll get a rosy welcome home welcome there
[FP:CORPUS] I like straight and perspiration then only I feel at home otherwise no
[FP:MINPAIR] No got it repaired
[FP:MINPAIR] No no he is here
[FP:MINPAIR] Okay no but
[FN:BOTH] I just go out for tea isn't
[FN:BOTH] Hey you you like serious movies is it?  
you like serious movies
[FN:BOTH] See no the scene exactly happened you know the other day what happen I was reading baba
[FN:CORPUS] I'm not fair no
[FN:CORPUS] I think no
[FN:CORPUS] Tell me no why you can't tell
[FN:MINPAIR] Yeah then it's first time first time it was new to me no
[FN:MINPAIR] That is the main thing na here that would again the main thing that they don't take at all interest in the their children at all
[FN:MINPAIR] So culture nahi hai there is I don't follow culture religion nothing na
B.4 Lack of Copula
[FP:CORPUS] Which October first I think
[FP:CORPUS] June nineteen eighty-six
[FP:MINPAIR] Construction all before
[FP:MINPAIR] Not in the class
[FP:MINPAIR] The tendency to
[FN: BOTH] you've she said his grandfather still working
[FN: BOTH] Everybody so worried about the exams and studies
[FN:BOTH] Again classes bit too long I feel five O'clock is tiring
B.5 Left Dislocation
[TP: BOTH] This principal she is very particular about it
[TP:BOTH] Vilas and Ramesh they they make noise man
[TP:BOTH] That's why those Muslims they got very angry
[TP:CORPUS] And med medium class they can't understand soon
[TP:CORPUS] That will become difficult and common people they don't understand
[TP:CORPUS] And now the Kukis they refused to pay any more
[TP:MINPAIR] It's because of this some other participant they complained about this and then they started they started this particular
[TP:MINPAIR] We've lot of fun in theatres you know we always take the back seat and all that for this guys distinct one we keep teasing them
[TP:MINPAIR] My post graduation degree I finished it in mid June nineteen eighty-six
[FP:BOTH] But whereas when they really come to know the people they like to help the people
[FP:BOTH] It's actually some of them like to see it really so huge and long and bigger snakes they are in all closed and all there it is nice to see it
[FP:BOTH] But generally the educated people I don't find much variation but in accent there may be a variation
[FP:CORPUS] Everytime he keeps speaking you know they get irritated and say aram se
[FP:CORPUS] What happened is they will change programme and the fifty guys they'll just keep quite
[FP:CORPUS] Whereas Hyderabad the people are more conservative and like they don't like to go out even or at the first move they don't like to talk with people also
[FP:MINPAIR] And the songs now once we hear it afterwards when some other famous songs comes that we forget the last ones
[FP:MINPAIR] But when we approach since it seems they they put lot of conditions yes that you fed up with those people and
[FP:MINPAIR] so that's why we missed we that missed that holiday it being a Sunday
[FN:BOTH] Administration it is all done by Bharati Vidya Bhavan
[FN:Both] Oh our Joshi okay II got got him
[FN:BOTH] Yes yes it is true but our constitution makers
[FN:CORPUS] and he has used the place where the palace once palace might be there and that portion and the remaining part he built an antenna he has fixed it there at the top
[FN:CORPUS] Not exactly but Calcutta sweets I think they do have a little flavour and that I haven't got anywhere in India
[FN:CORPUS] Computer it was in the first semester
[FN:MINPAIR] And med medium class they can't understand soon
[FN:MINPAIR] Shireen she was excellent at that
[FN:MINPAIR] Yeah arti arti students they loiter about in the corridor
B.6 Non-initial Existential  $X$  is / are there
[TP:BOTH] Libraries are there
[TP:BOTH] only specimen like operated cases like supposing a is there
[TP:BOTH] Problems are there problems are there what
[TP:CORPUS] to assist there some teachers are there and together we conduct the classes
[TP:CORPUS] It's there but it's common no
[TP:CORPUS] Yeah I think Varlaxmi is there
[FP:BOTH] My husband is there mother is there
[FP:CORPUS] Come no Shaukat is here Natalie is here even if Savita is not there they two are there na
[FP:CORPUS] Actually there the thing is that you know for example
[FP:CORPUS] Any thing is there produced materials which do not require much resource personnel
[FP:MINPAIR] Ph D degree is awarded there
[FN:BOTH] Yeah the royalties too there they're there and we've the king
[FN:BOTH] Okay somebody else's some somebody else is there
[FN:BOTH] In that you know everything is about nature I'll tell you yeah it's very lovely means very nice lovely what but and small children were there in that
[FN:MINPAIR] American and all other capitalist nations were also there
[FN:MINPAIR] Nice movie yaar that song is there no hai apna dil to awara
[FN:MINPAIR] It's not there
B.7 Object Fronting
[TP:BOTH] Just typing work I have to do
[TP:CORPUS] writing skills there are so many you can teach them
[TP:CORPUS] Each other and so many things we have learnt
[TP:CORPUS] My birthday party you arrange
[FP:CORPUS] Formalities I will come
[FP:CORPUS] Mar Marxism you were
[FP:MINPAIR] Other wise we have to
[FN:BOTH] That also I'm not having just I jump jumped jumped I came studies also
[FN:BOTH] Yes Hawa Mahal we heard
[FN: BOTH] About ten to twenty books I'll read that's all
[FN:MINPAIR] Small baby very nice it was
[FN:MINPAIR] But more keen she is
[FN:MINPAIR] And camera handling actually outdoor landscaping that landscape shot I have taken and actually the close ups and some parts of your
architectural shots of that building Ganesh took my husband took and close ups of the faces my husband and Ganesh took
B.8 Resumptive Object Pronoun
[TP:MINPAIR] and he has used the place where the palace once palace might be there and that portion and the remaining part he built an antenna he has fixed it there at the top
[TP:MINPAIR] Yeah also pickles we eat it with this jaggery and lot of butter
[FP:MINPAIR] Having humorous something special I would love it to join it
[FP:MINPAIR] I see a number of people I like them very much
[FP:MINPAIR] Old and ancient things in carving we get it so beautifully
[FN: BOTH] Normally no we don't overdrawn on account but haan haan whatever is balance you know yeah help them give them suppose cheque books and all we are supposed to keep them yeah two fifty balance
[FN: BOTH] He is in a that's what he was telling me today see I want your draft like draft draft by January by the month of January by the end of January so that II might rectify it and then I will do it I will give it back to you by mid Febraury so that you can get it final draft by by the end of Febraury
[FN:CORPUS] Yeah also pickles we eat it with this jaggery and lot of butter
[FN:CORPUS] My post graduation degree I finished it in mid June nineteen eighty-six
B.9 Resumptive Subject Pronoun
[TP:CORPUS] Like those terrorists they wanted us to accompany them in the revolt against India
[TP:CORPUS] And one more thing another thing how I rectified myself because all almost all all of us all my brother and sisters we have read in English medium school
[TP:CORPUS] Dr this Mr V he was totally changed actually because he was the concepts are clear not clear to us
[FP:CORPUS] There are so many people they can they could shine like anything
[FP:CORPUS] Kolhapur he had come to Guwahati
[FP:CORPUS] I don't know what he whenever whenever I see those guys they they nicely speak to me
[FP:MINPAIR] His house he is going to college KK diploma electronics
[FN:BOTH] they I thought that another one Patil is there a horrible he is I thought that Patil
[FN:BOTH] Computer it it plays a great role because we are having computers in each field now-a-days
[FN:BOTH] You know that a woman she is a apprehensive about many things
[FN:MINPAIR] Like those terrorists they wanted us to accompany them in the revolt against India
[FN:MINPAIR] Whereas in Hyderabad they still have the old cultures and so many things that even the parents they don't even let the girls talk with the guys
[FN:MINPAIR] And the students who come out with a degree MMSI understand that there is a report that has been received from different firms that the students of BITS Pilani specially MMS candidates they are prepared to soil their hands
B.10 Topicalized Non-argument Constituent
[TP:CORPUS] for Diwali you went I know that
[TP:CORPUS] So very long time we have not travelled together
[TP:CORPUS] Pooja vacation also we used to conduct some classes practical classes
[TP:MINPAIR] In pooja day some important days we stay back
[FP:CORPUS] In Jaipur then we have also we have a Birla
[FP:CORPUS] Like that we
[FP:CORPUS] Everytime we have some work to do
[FP:MINPAIR] Aa i i initial periods I did very difficult but I
[FN:BOTH] I mean here in Hyderabad the people are it's okay they are nice
[FN: BOTH] And that old ones again we put them we feel like hearing again
[FN:BOTH] But in drama we'll have to be very different
[FN:CORPUS] In pooja day some important days we stay back
[FN:MINPAIR] for Diwali you went I know that
[FN:MINPAIR] Pooja vacation also we used to conduct some classes practical classes
[FN:MINPAIR] Sir from Monday onwards I too want to take leave sir for four days because total I have five C Ls so from
C Average Precision Results
Supervision:
Dialect feature	Corpus examples	Minimal pairs
DAMTL	Multihead	DAMTL	Multihead
FOCUS itself*	0.668	0.631	0.665	0.613
FOCUS only*	0.582	0.404	0.344	0.416
INvariant TAG	0.876	0.871	0.441	0.495
COPULA OMISSION	0.029	0.015	0.012	0.036
LEFT DISLOCATION	0.425	0.383	0.149	0.232
NON-INITIAL EXISTENTIAL*	0.887	0.906	0.556	0.510
OBJECT FRONTING	0.238	0.202	0.031	0.083
RES. OBJECT PRONOUN	0.052	0.020	0.046	0.061
RES. SUBJECT PRONOUN	0.460	0.409	0.078	0.198
TOPICALIZED NON-ARG. CONST.	0.080	0.076	0.021	0.044
Macro Average	0.430	0.392	0.234	0.269
Table 8: Average precision for the Lange features. Scores are in the range [0, 1], with 1 indicating perfect performance. Asterisks mark features that can be recognized with a regular expression.
Dialect feature	DAMTL	Multihead
ARTICLE OMISSION	0.210	0.308
DIRECT OBJECT PRO-DROP	0.044	0.057
EXTRANEIOUS ARTICLE	0.116	0.065
FOCUS itself*	1.000	0.853
FOCUS only*	0.859	0.274
HABITUAL PROGRESSIVE	0.008	0.020
INvariant TAG	0.614	0.420
INVERSION IN EMBEDDED CLAUSE	0.106	0.162
LACK OF AGREEMENT	0.084	0.110
LACK OF INVERSION IN WH-QUESTIONS	0.309	0.106
LEFT DISLOCATION	0.288	0.301
MASS NOUNS AS COUNT NOUNS	0.045	0.034
NON-INITIAL EXISTENTIAL*	0.506	0.397
OBJECT FRONTING	0.147	0.193
PREPOSITION OMISSION	0.064	0.116
PP FRONTING WITH REDUCTION	0.091	0.134
STATIVE PROGRESSIVE	0.267	0.329
GENERAL EXTENDER and all	0.769	0.778
Macro Average	0.307	0.259
Table 9: Average precision for the extended feature set. As described in the main text, corpus training examples are unavailable for these features.
D Minimal pairs
27 INVARIANT TAG (isn't it, no, na)
28 INVARIANT TAG (isn't it, no, na)
28 INVARIANT TAG (isn't it, no, na)
28 INVARIANT TAG (isn't it, no, na)
29 INVARIANT TAG (isn't it, no, na)
29 INVARIANT TAG (isn't it, no, na)
29 INVARIANT TAG (isn't it, no, na)
30 INVARIANT TAG (isn't it, no, na)
30 INVARIANT TAG (isn't it, no, na)
30 INVARIANT TAG (isn't it, no, na)
30 INVARIANT TAG (isn't it, no, na)
31 INVERSION IN EMBEDDED CLAUSE
31 INVERSION IN EMBEDDED CLAUSE
32 INVERSION IN EMBEDDED CLAUSE
32 INVERSION IN EMBEDDED CLAUSE
33 INVERSION IN EMBEDDED CLAUSE
33 INVERSION IN EMBEDDED CLAUSE
34 INVERSION IN EMBEDDED CLAUSE
34 INVERSION IN EMBEDDED CLAUSE
35 INVERSION IN EMBEDDED CLAUSE
35 INVERSION IN EMBEDDED CLAUSE
36 LACK OF AGREEMENT
36 LACK OF AGREEMENT
37 LACK OF AGREEMENT
37 LACK OF AGREEMENT
38 LACK OF AGREEMENT
38 LACK OF AGREEMENT
39 LACK OF AGREEMENT
39 LACK OF AGREEMENT
40 LACK OF AGREEMENT
40 LACK OF AGREEMENT
41 LACK OF INV. IN WH-QUESTIONS
41 LACK OF INV. IN WH-QUESTIONS
42 LACK OF INV. IN WH-QUESTIONS
42 LACK OF INV. IN WH-QUESTIONS
43 LACK OF INV. IN WH-QUESTIONS
43 LACK OF INV. IN WH-QUESTIONS
44 LACK OF INV. IN WH-QUESTIONS
44 LACK OF INV. IN WH-QUESTIONS
45 LACK OF INV. IN WH-QUESTIONS
45 LACK OF INV. IN WH-QUESTIONS
46 LEFT DISLOCATION
46 LEFT DISLOCATION
47 LEFT DISLOCATION
47 LEFT DISLOCATION
48 LEFT DISLOCATION
48 LEFT DISLOCATION
49 LEFT DISLOCATION
49 LEFT DISLOCATION
50 LEFT DISLOCATION
50 LEFT DISLOCATION
51 MASS NOUNS AS COUNT NOUNS
51 MASS NOUNS AS COUNT NOUNS
52 MASS NOUNS AS COUNT NOUNS
52 MASS NOUNS AS COUNT NOUNS
53 MASS NOUNS AS COUNT NOUNS
53 MASS NOUNS AS COUNT NOUNS
54 MASS NOUNS AS COUNT NOUNS
54 MASS NOUNS AS COUNT NOUNS
55 MASS NOUNS AS COUNT NOUNS
55 MASS NOUNS AS COUNT NOUNS
56 NON-INITIAL EXISTENTIAL
56 NON-INITIAL EXISTENTIAL
57 NON-INITIAL EXISTENTIAL
57 NON-INITIAL EXISTENTIAL
58 NON-INITIAL EXISTENTIAL
58 NON-INITIAL EXISTENTIAL