ELQA: A Corpus of Metalinguistic Questions and Answers about English
Year: 2023

Abstract:
We present ELQA, a corpus of questions and answers in and about the English language. Collected from two online forums, the >70k questions (from English learners and others) cover wide-ranging topics including grammar, meaning, fluency, and etymology. The answers include descriptions of general properties of English vocabulary and grammar as well as explanations about specific (correct and incorrect) usage examples. Unlike most NLP datasets, this corpus is metalinguistic—it consists of language about language. As such, it can facilitate investigations of the metalinguistic capabilities of NLU models, as well as educational applications in the language learning domain. To study this, we define a free-form question answering task on our dataset and conduct evaluations on multiple LLMs (Large Language Models) to analyze their capacity to generate metalinguistic answers.

heading: 1 Introduction
text: Language is so powerful that it can be reflected back on itself. Statements like “In informal usage, a steep learning curve means something that is difficult (and takes much effort) to learn” or “In some cases, an adjective has both -ic and -ical forms, with no difference in meaning” expressly concern linguistic inventories, structures, and behaviors. In other words, they are metalinguistic—they use language to discuss language (cf. Wilson, 2013). They may concern a particular instance of language use, or properties of a language or speaker in general; either way, they are metalinguistic in making linguistic phenomena the subject matter of a linguistic utterance. For the rest of this paper, the term metalanguage is used for natural language text in which natural language is also the subject matter.
While NLP models have become powerful at predicting text in many settings, it remains to be seen whether such capability extends to metalanguage— where linguistic strings are not being deployed to
contribute to the discourse with their normal denotations, but rather, are treated as entities with linguistic properties (e.g., grammar, meaning). One way this can be explored is in a question answering framework, which requires suitable datasets, ideally based on questions that are realistic and paired with high-quality answers.
In this paper, we present a corpus of metalinguistic questions and answers about English. The corpus is collected and carefully processed from two Stack Exchange forum sites: English Language & Usage (ENG) and English Language Learners (ELL). It covers more than 70k questions on numerous topics about English such as grammar, meaning, fluency, and etymology along with answers. Our corpus, ELQA (English Language Questions and Answers), can serve as a tool to facilitate metalinguistic studies. Moreover, since questions in ELQA cover a variety of topics in English, it can be used in the educational and English language learning domains.
As the first case study of ELQA, we investigate the performance of current state-of-the-art NLP technology on free-form question answering in the English language domain. Additionally, we explore the possibility of building NLP models that can directly answer questions from language learners. We process a subset of ELQA and make it appropriate for this task. Then, we report on the results of both automatic and human evaluations using different experimental settings of T51 and GPT-32 models. Although most of these models achieve high ratings for well-formedness, the validity of their answers is significantly lower than that of human-authored answers, indicating that this type of metalinguistic QA task is challenging even for large language models.
Our main contributions are: 1) we release the
1https://github.com/google-research/t5x 2https://openai.com/blog/gpt-3-apps
2031
first publicly available metalinguistic QA dataset,3 focused on the English language; 2) we present a taxonomy of questions in the corpus along with analysis; and 3) we investigate to what extent LLMs are able to articulate appropriate generalizations about language in response to these questions.

heading: 2 Related Work
text: Stack Exchange is a network of numerous CQA sites (originally and most famously, Stack Over-
3https://github.com/shabnam-b/ELQA
flow) built on a common platform. Stack Exchange forums have been featured in a number of previous datasets (Yao et al., 2013; Hoogeveen et al., 2015; Ahmad et al., 2018; Penha et al., 2019; Campos et al., 2020; Kumar and Black, 2020; Rogers et al., 2023), including the English site (our ENG) along with others such as Ask Ubuntu, Android, Gaming and WordPress (dos Santos et al., 2015; Nakov et al., 2017). We focus on ENG and ELL as they concern the English language itself; we show that these datasets cover a wide range of metalinguistic questions.
Our use of these forums contrasts with previous work on metalanguage in corpora, which annotated and quantified mentions (Anderson et al., 2004; Wilson, 2010, 2011, 2012, 2017), but did not consider entire questions and answers about language. Taylor (2015) studied metalanguage in online forums, but with a focus on the usage of metalinguistic expressions of mock politeness. More recently, Bogetic (2021) published the first corpus of contemporary Slovene, Croatian and Serbian media metalanguage texts.
So far, metalanguage has not been a focus in the QA domain—ours is the first publicly available English metalinguistic QA dataset. Most QA tasks are set up to have a question and a reference document, where the objective is to find the answer based on the document (Fan et al., 2019; Kwiatkowski et al., 2019). In this paper, we explored a type of “closed-book” question answering task (Roberts et al., 2020; Khashabi et al., 2021). To the best of our knowledge, this task has not been explored to date within the realm of English language questions
that require significant generalization and adaptation rather than looking up facts.

heading: 3 Constructing the Dataset
text: We collect our data from two sites on Stack Exchange: English Language & Usage (ENG)4 and English Language Learners (ELL).5 Sample screenshots of the site are shown in Figure 1. The Stack Exchange data is publicly released under the CCBY-SA 3.0 license. We preprocessed the data until 2021-12-06 collected from the Internet Archive6 to be suitable for NLP studies and release it as ELQA. Additionally, some cleanup (e.g., removing posts marked as “spam” or “offensive”) was done. Fields for each entry (question) include the title, body, user bio (if available), score (which is calculated based on up-votes and down-votes by other users), tags (user-assigned, related to the area/topic of the question), favorite count, and a list of answers. Textual content (body and user bio) is provided in two formats: HTML and plain text without HTML tags.
We release two versions of ELQA based on different preprocessing steps. In ELQA-large, we keep questions as long as they don’t include any images (<img> HTML tag) and have an answer with a score of at least 2 (meaning at least two people other than the user posting the answer found it helpful). For ELQA-small, we applied further filtering to ensure that the data has the least amount of noise: a) questions should have a score of at least
4https://english.stackexchange.com/ 5https://ell.stackexchange.com/ 6https://archive.org/
2 (ensuring questions are clear and coherent), b) question has an answer with a score higher than 3 and c) there are no hyperlinks in at least one of the high-rated answers. The last step reduces noise and facilitates a fair comparison for the closed-book question-answering task (§4) with model-generated answers, as models cannot be expected to have access to the web to suggest valid URLs compared to humans who would search the web for appropriate resources to include in their answers.
For quality assurance, we also did a human annotation on ELQA-small. Two of the authors annotated 250 question and answer pairs for the following: 1) Is the question answerable? and 2) Does the answer fully address the question? We found 99.2% of the questions answerable and 91.8% of the answers acceptable.
Table 1 contains overall statistics on both versions. Figure 2 shows the distribution of the 10 most common tags in each of the sites. Since users assign these tags to their questions (0 to multiple), similar or near-duplicate tags are common within the collection. Some form more general and more fine-grained variants, e.g. ‘meaning’ and ‘meaningin-context’. In addition to available user-assigned tags, we manually inspected a large subset of the data to identify salient types of questions. These are defined below and illustrated in Table 2. We then labeled 100 random questions to get a rough estimate of their frequencies (two annotators annotated these 100 samples and they agreed on 92% of cases in an overlapping subset). • Fluency (≈38% of questions): Usually asking about a particular sentence, comparison of multiple sentences, and/or probing how an expression should be used in general. The user wants to know if X is correct, or to decide between multiple choices, which one is correct. “Correct” could mean grammatical, most natural/idiomatic, stylistically appropriate, conveying the intended meaning, etc. In Qs where options are provided by the user, there are cases in which 1) none of the choices are correct, 2) multiple choices are correct, and 3) only one is correct. • Form to Meaning (Interpretation) (≈19% of questions): Questions such as “What does X mean?” (of an expression in general, or an encountered passage) or “What’s the difference in meaning between X and Y?”. • Meaning to Form (Encoding) (≈20% of questions): In these questions, the user gives some
explanation/definition and asks for the term or for form to express it. • Grammatical Analysis (≈11% of questions): Questions about parts of speech and other aspects of syntactic analysis. (e.g. “Is this a verb or an adjective?”; “Can an article ever go after the noun it modifies?”). Note that Fluency questions may mention grammatical terminology, but the grammatical categories are not the focus. • Other (≈10% of questions): Any other type of question not listed above. This includes questions about pronunciation, etymology, etc.
As can be seen from the examples in Table 2, it is common for questions and answers to contain example usages, often visually distinguished with Markdown formatting (such as blockquotes, bullets, and italics) which we retain in the processed corpus markup. Examples can be incorporated into a post in a variety of ways—e.g., asking for an interpretation of one usage, as in the Form to Meaning example in Table 2, or contrasting multiple usages such as in the following question:
Did VS Have done What is difference between the following statements: Did you tell your parents yet? Have you told your parents yet? Haven’t you told your parents yet? Are these questions correct? why do we use one over another in some cases? What is the difference in meaning?
Usage examples provided in a question may be instances that the author encountered “in the wild” (such as in a novel or film), or in a grammar book or dictionary, or they may have been constructed
by the user. Answers sometimes include examples found through a corpus search.

heading: 4 English Language Question Answering
text: Large language models can produce output that is fluent and (at times) informationally adequate when presented with factual questions about entities in the world (Roberts et al., 2020). But how do such models perform when asked questions about the language itself? In this section, we investigate the free-form English language question answering task.
This task has the potential to benefit educational applications for language learners. Research on NLP for educational purposes has investigated tasks such as automated grammatical error correction (Dale et al., 2012; Ng et al., 2014; Bryant et al., 2019; Wang et al., 2021, inter alia), question and quiz generation for language learning (Sakaguchi et al., 2013; Chinkina and Meurers, 2017; Marrese-Taylor et al., 2018; Vachev et al., 2021), and automated essay scoring (Burstein, 2003; Farag et al., 2018, inter alia). Nevertheless, an application that has not been taken up by the educational NLP community is free-form question answering about language. Second language learners possess a degree of metalinguistic awareness about the language they are learning, and often turn to teachers or more advanced speakers with explicit questions about vocabulary, grammar, and usage. Community Question Answering (CQA) websites such as Stack Exchange have sites for language learners’ questions and answers. These sites require consid-
erable effort by volunteers, and learners may have to wait for an answer—if an answer is provided at all. In fact, looking at the data from 2021-12-06 for ENG and ELL, 9% of questions have no answers.

heading: 4.1 Data
text: We randomly divided ELQA-small into train/test/dev splits. This resulted in 21,175 Q&A pairs in the train split and 3,107 Q&A pairs in each of the dev and test splits. Answers in these splits have a score of at least 4. If there are multiple high-rated answers to a question, we include all of them for training. Some of these questions can be answered by looking at a dictionary or vocabulary list for descriptions. But many of them are explanations in relation to particular instances of language use and require significant reasoning rather than looking up facts. Thus in this setup, we do not have any external context/reference available at evaluation time, i.e. this is a closed-book QA task.
The input for the task is Title: [Q title] <sep> Body: [Q body]. We use the HTML version of ELQA for this task since metalinguistic mentions are usually distinguished via formatting (e.g., blockquotes, bullets) and the ultimate goal is a system that humans can easily use to get answers to their language-related questions.

heading: 4.2 Setup
text: We use T5 (Raffel et al., 2020; Roberts et al., 2022) and GPT-3 (Brown et al., 2020) as our models since they have been shown to be strong baselines in other QA domains. We believe the questions in ELQA offer new challenges for the QA task since they require different types of knowledge/understanding to be able to generate answers. Addition-
ally, these questions contain noise (grammatical errors) and cases of textual metalanguage which is likely harder to comprehend for a model.
We fine-tune T5-l and T5-xxl for this task.7 We saved multiple checkpoints during fine-tuning and evaluated them with the interpolation of BLEU (Papineni et al., 2002), BERTScore (Zhang et al., 2020) and ROUGE (Lin, 2004) on the dev set to choose the best-performing one (checkpoint at 75k updates, hyperparameters available in Table 8 in the Appendix).
With GPT-3 we used text-davinci-003 and experimented with both fine-tuning (FT) on 100 and 1000 samples and a few-shot (FS) setting in which the model is given a few demonstrations of the questions and answers at inference time as conditioning, but no weights are updated (Radford et al., 2019). In the FS setting, we show the model four Q&A pairs since we wanted the model to see different question types but there were also limits on the input length. To select these 4 pairs, we randomly created 5 different sets of Q&A pairs, evaluated on a subset of dev, and chose the best-performing set for the experiments (dev results available in Appendix, Table 9).

heading: 4.3 Results

heading: 4.3.1 Automatic Evaluation
text: Results are shown in Table 3. GPT-3 FS outperforms all other methods in all metrics with a large margin except for BLEU Score. We also observed that using GPT-3 in a few-shot setup worked much better than the fine-tuned version. Looking at some of the model-generated answers, we noticed that the fine-tuned model tends to generate longer an-
7This took 5 days with v3-8 TPU (provided by Google)
swers containing redundant text. We observed improvements when we used 1000 samples instead of 100 for fine-tuning and hence, fine-tuning on larger data might result in better performance, however, we only experimented with 100 and 1000 samples in this paper due to having limited resources.
Based on Table 3, T5-xxl seems to perform similarly to GPT-3 FT-1000. However, a small manual evaluation showed otherwise (GPT-3 FT-1000 answers were slightly better). Furthermore, we observe that the scores for even the best system are very low, but manual evaluations showed that the GPT-3 FS generates fairly good answers in many cases. Due to these observations and also given the well-known limitations of automatic metrics for evaluating generation tasks (Kasai et al., 2022; Celikyilmaz et al., 2020; Bhakthavatsalam et al., 2021), we believe conducting human evaluation for deeper analysis is necessary for this task.
In Table 4, we show results for each site to see if one is more challenging than the other. Overall, models perform slightly better on ELL based on automatic metrics—but we see in the next section (Table 5) that there isn’t really a meaningful difference between the sites when humans evaluate the answers.

heading: 4.3.2 Human Evaluation
text: Human evaluators were presented with the question title and body, and then asked to rate 5 answers: a top-rated human-provided answer, a low-rated human-provided answer, and answers generated by 3 of our best models: GPT-3 FS, GPT3 FT-1000, T5-xxl.
They were asked to give ratings (via a slider widget, on a 1–5 integer scale—the higher, the better) for two criteria (C1 & C2):8
1. Does the answer look grammatically/ structurally like a good answer (ignoring whether it answers the question)? 2. Is the information in this answer a valid response to the question (ignoring formatting/ stylistic issues)?
The first criterion aims to get a score for fluency and coherence and the second one for correctness and completeness.
We collected ratings for a set of 75 questions (375 different answers). Each question with its set of answers was evaluated by at least 2 raters, and then the average score was calculated based on their responses.9 We also report the average z-score which is calculated over each annotator’s raw score distribution for each metric, intended to normalize interannotator variation in how the scale is interpreted for each of the two metrics (details in Appendix B).
The results of this study are shown in Table 5. Overall, answers generated by GPT-3 FS have a small gap with human answers in both C1 and C2.10 GPT-3 FT-1000 comes next, with less accurate answers containing redundant text and hallucinations. The smallest model, T5-xxl, ranks last.
Rankings based on human evaluations are available in Table 6. These results are also indicating that model-generated answers are fluent in most cases, but they are not as good as human answers
8The survey interface is illustrated in Figure 5 of Appendix C.
9Evaluators consisted of 6 English native speakers who are senior NLP researchers and graduate students. The answer source was hidden and the order (5 answers) was randomized. Annotations took about 90 minutes on average.
10We selected half of the human evaluation samples from very recent posts (June 2021 until January 2023) on Stack Exchange and compared the results with older posts. The models’ scores were comparable or better on the more recent data, so we didn’t see evidence of models having an advantage due to the fact that they may have been trained on some of the data available on the web. For reference, human evaluation scores for recent data were Avg. C1=4.82, Avg. C2=3.83 and for older data, Avg. C1=4.86, Avg. C2=3.61.
when correctness/completeness is considered (GPT3 FS is ranked first or as good as a top-rated human answer in only 45% of cases).
For each criterion and Top-rated human, Lowrated human and GPT-3 FS, histograms of the average score of the two annotators are plotted in Figure 3. We can observe that GPT-3 FS and Lowrated human have very similar numbers of highscoring answers (human evaluation scores), but the number of low-scoring human answers drops off gradually as quality decreases, while the distribution is more spread out for GPT-3 FS. I.e., the model has some moderately bad answers as well as some good ones, whereas Low-rated human answers cluster more on the upper end of the scale.
C1 (fluency/coherence). All models generated fairly fluent and well-structured answers. We even notice that GPT-3 FS scores higher in wellformedness than human answers. We looked at those samples and we believe there are two main reasons for this: 1) Some human answers were very long, containing multiple different quotes from different sources. On average, our evaluators preferred the structure of answers from GPT-3 FS, which took the form of a short paragraph addressing the question. 2) Some human answers have a more casual/conversational nature. On the
other hand, GPT-3 FS generated more authoritativesounding, teacher-like answers with complete sentences, which were preferred by our evaluators in some cases. C2 (correctness/completeness). On average, models are worse on this metric, though sometimes they did produce acceptable answers (perhaps because variants of the question are commonly discussed on the web).
One challenge in this domain is that questions, particularly from ELL, may not be fully fluent. In the Fluency example from Table 7 we see that there are some grammatical errors in the question that are unrelated to the topic of the question. In addition, the questioner uses incorrect terminology, mentioning verb tense in the post title even though the question is actually about subject-verb agreement with respect to number (as can be inferred from the examples). While the good human response correctly flags the incorrect terminology and answers the underlying question, GPT-3 models give irrelevant answers about tense.
Another correctness failure from GPT-3 FS can be seen in the following pair involving a Meaningto-Form question:
What is the analog equivalent of ’digitally’? In electronics, we can solve a problem digitally or analoguely ? That doesn’t sound right, but
surely there must be a word I can use? Update: some definitions: digitally: Relating to or being a service that provides information expressed in discrete numerical form. analoguely: Relating to or being a service that provides information expressed in continuous, non-discrete form. (This is my made up definition of a word I made up). analogously: Similar or alike in such a way as to permit the drawing of an analogy. As you can see, the word analogously doesn’t mean the opposite of digitally, it means something else. Answer: The analog equivalent of ’digitally’ is ’analogously’. Analogously means "in a similar way" or "by comparison". It is used to describe a situation where two things are similar in some way, but not necessarily opposite. For example, you could say that a digital signal is analogous to an analog signal, because they are both signals, but they are different in terms of how they are represented.
We see that the model begins by ignoring the user’s explanation that analogously does not have the desired meaning, and suggests it anyway. The rest of the model’s answer gives a (valid) general definition of analogously and clumsily attempts to apply it to the two kinds of signals, which is not responsive to the question. It may be the case that models particularly struggle with questions for which there is no straightforward solution (in this case, no adverb derived from analogue has the user’s intended meaning).

heading: 4.4 Discussion

heading: 4.4.1 Metrics
text: Our human evaluations found that the high-rated human answers are the best, and GPT-3 FS is roughly on par with the low-rated human answer, while other models are worse.
As noted in §4.3.1, we did not find the automatic metrics very informative. We believe this is due to the long and free-form nature of this task: concepts can be explained in different ways and answers can include different examples which make automatic
evaluations very challenging. For example, for the question Why is this sentence: "Additional nine features were added. . . " incorrect? a human answer is:
It has to do with the order of the adjectives. For example, consider this sentence: <blockquote> Happy nine men walk into a bar. </blockquote> Both nine and happy are adjectives, but we are really intending nine to describe the happy men, not happy describing the nine men. I don’t ....
Model-generated answers mostly mirror the examples in the question. But answers given by humans mention new examples—which are not in the question (and therefore unlikely to come up in other answers). Thus a good answer may mention superficially off-topic terms metalinguistically, which presents evaluation challenges unlike those of summarization or MT, for example (Reiter, 2018; Celikyilmaz et al., 2020). For QA tasks in metalinguistic domains such as ours, caution is warranted when applying automatic metrics that are unaware of the use/mention distinction. Human evaluation is likely to be more illuminating. We release our full human evaluation data to enable detailed comparisons in future research.

heading: 4.4.2 Qualitative Analysis
text: We examined behavioral patterns for the best model, GPT-3 FS, organized by question type, with observations as follows. Interpretation and Encoding: These are the categories where the model performs the best, especially if the question has a straightforward answer (for example, there aren’t several possible answers based on context). Topics in these categories often center on definitions of vocabulary and idioms. These questions may involve less reasoning to answer, and models may be leveraging content from the online dictionaries and thesauri in their web training data.
Grammatical Analysis: When asked about general concepts and rules of grammar, the modelgenerated answers are usually acceptable:
-ed and -ing adjectives What is the difference? <blockquote> I am interesting in mathematics. </blockquote> versus <blockquote> I am interested in mathematics. </blockquote> Murphy intermediate grammar Unit 97 tells me that I cannot use the former. Extended and general explanations will be appreciated. Answer: The difference between -ed and -ing adjectives is that the -ed adjectives describe a feeling or emotion that is felt by the subject, while the -ing adjectives describe a quality of the subject itself. In the example sentence, interested is the correct word to use because it describes the feeling of interest that the subject has in mathematics. Interesting would be incorrect because it describes the quality of the subject itself, not the feeling of the subject. For example, if you were to say "This book is interesting," you would be describing the book itself, not the feeling you have towards the book. On the other hand, if you were to say "I am interested in this book," you would be describing the feeling you have towards the book.
However, when questions become more specific to examples, sometimes the model struggles:
What does "in" refer to? <blockquote> A symbiotic relationship is an interaction between two or more species in which one species lives in or on another species. Answer: It refers to the other species. The symbiotic species lives in the other species.
The example in the question uses the word in twice. Rather than addressing the first token, which the questioner has bolded, the answer addresses the second one (denoting the physical relationship between one species and another). Fluency: The model correctly answers many of the fluency questions. Others require reasoning and understanding of context, and the model struggles to generate a correct answer. For example:
Returning for the nth time Suppose I went to the supermarket three times. Is my third trip considered my "second time returning there" or my "third time returning there?" Thanks. (I know this question sounds silly, but English is my second language...) Answer: Your third trip to the supermarket would be considered your third time returning there. The phrase returning for the nth time is used to refer to the nth time you have returned to a place, not the n-1th time.
We also observe that when the correct usage of a case is not mentioned in the question itself, the model sometimes suggests incorrect corrections and wrong explanations:
Why is this usage of "to" incorrect? I am in communication with a friend who is learn-
ing English. I noticed her using "to" in the wrong context, but I’m unable to explain which rule she is breaking. <blockquote> I can’t access to Gmail. </blockquote> Answer: In this sentence, the word to is incorrect because it is being used as a preposition. The correct word to use in this context is for. <blockquote> I can’t access for Gmail. </blockquote> The word to is used as a preposition when it is followed by a verb, such as go, come, run, etc. For example: <blockquote> I am going to the store. </blockquote> However, when to is followed by a noun, it is usually incorrect. In this case, the correct word to use is for. <blockquote> I need a gift for my friend.</blockquote>

heading: 4.4.3 Outlook
text: Overall, we were surprised by the quality of many of the answers from GPT-3 FS: many would likely have received high user ratings if submitted as answers on the site. At the same time, the model is not to the point where we would want to trust its answers without human review. We believe that answer confidence estimation—so that users can be shown only the best model-generated answers—is an important direction for using learner QA models in practice (Jiang et al., 2021).

heading: 5 Conclusion
text: We presented ELQA, a dataset containing metalinguistic questions and answers about the English language. We provided analysis and a taxonomy of the data, along with experiments on free-form answer generation and investigated the extent to which language models can articulate their generalizations about language. Since many of the questions in ELQA were asked by language learners, it forms a potentially useful and so far untapped resource for educational NLP purposes and metalinguistic question answering. We release the dataset to enable further studies of this task.
Ethics Statement
We have released a processed version of an already public online forum dataset, in a manner consistent with the terms of the license, which require attribution of all posts (§3). The models we have presented are intended only as baselines for future research, not for deployment. Models should be carefully stress-tested for undesirable heuristics/ biases before deployment. Systems for the generation task, in particular, would risk misleading language learners with plausible but incorrect answers, so it is important to not deploy a generation
system until it is approximately as reliable as existing non-automated alternatives, and to present the output with caveats. Potential biases reflecting the demographics of authors represented in the training data (in terms of native language, level of English proficiency, etc.) also need to be considered if models are deployed for different target populations.
Limitations
One limitation of our dataset, ELQA, is that the corpus only contains questions in English and about English. However, Stack Exchange has sites with questions about other languages and our main data extraction scripts are general enough that they can be used to create corpora for other sites on Stack Exchange. Of course, language-specific processing steps, quality assurance and analysis must be applied before releasing such data.
Most importantly, the models we have presented here are intended only as baselines for future research, not for deployment. Potential biases reflecting the demographics of authors represented in the training data (in terms of native language, level of English proficiency, etc.) also need to be considered if models are deployed for different target populations. Moreover, many of these types of questions are found on the web, and a lot of the same topics are brought up by many users, so a model’s ability to generate correct answers cannot necessarily be attributed to abstract reasoning.

heading: Acknowledgements
text: We thank the anonymous reviewers for their insightful comments. We thank Daniel Khashabi for helpful discussions and feedback. This research was supported in part by NSF award IIS-2144881.

heading: A Data Credits
text: The Stack Exchange license requires that any Internet use of the content should include a hyperlink directly to the original question and the profile of the authors. Below are URLs for all the examples used in this paper. The post URL incorporates the post title.
• https://ell.stackexchange.com/questions/12/dates -and-times-on-in-at (Q by bytebuster, A by waiwai933) • https://ell.stackexchange.com/questions/146633/o n-my-own-way-vs-in-my-own-way (Q by bavyan-yaldo) • https://ell.stackexchange.com/questions/19684/wo ndering-what-get-by-means-in-this-context (Q by nima) • https://english.stackexchange.com/questions/7489 6/grammatically-correct-synonym-for-level-of-c atastrophicness? (Q by solvingPuzzles) • https://english.stackexchange.com/questions/1343 52/should-i-modify-a-gerund-using-an-adjective -or-an-adverb (Q by worawit-tepsan) • https://english.stackexchange.com/questions/22 2567/what-is-the-etymology-of-physician (Q by casvaart) • https://ell.stackexchange.com/questions/185516/d id-vs-have-done (Q by learner) • https://english.stackexchange.com/questions/1628 24/what-is-the-analog-equivalent-of-digitally (Q by rocketmagnet, first A by AllisonAshley, second A by Hot Licks) • https://ell.stackexchange.com/questions/13749/ex plain-the-verb-tense-in-all-that-glitters-is-n ot-gold (Q by Chinmay235, first A by J.R., second A by sajad) • https://english.stackexchange.com/questions/1628 24/what-is-the-analog-equivalent-of-digitally (Q by Rocketmagnet) • https://english.stackexchange.com/questions/20 3518/why-is-this-sentence-additional-nine-fea tures-were-added-incorrect (Q by user95069), A by Nick2253 • https://english.stackexchange.com/questions/4938 4/ed-and-ing-adjectives (Q by itun) • https://ell.stackexchange.com/questions/87725/wh at-does-in-refer-to (Q by Anfi) • https://english.stackexchange.com/questions/1029 96/returning-for-the-nth-time (Q by AlicornTwilightisaTroll) • https://english.stackexchange.com/questions/5533 1/why-is-this-usage-of-to-incorrect (Q by Ademos) • https://ell.stackexchange.com/questions/87725/wh at-does-in-refer-to (Q by Anfi) • https://ell.stackexchange.com/questions/322637/h e-is-more-than-a-friend-is (Q by Loviii, first A by MarcInManhattan, second A by Kirt) • https://english.stackexchange.com/questions/25 8060/verb-for-doing-something-unknowingly (Q by Daniel Bramhall , first A by chasly - supports Monica , second A by talrnu) • https://ell.stackexchange.com/questions/322580/k now-someone-in-detail (Q by Simo Ita)

heading: B On our use of z-scores
text: In our human evaluation, raters were presented with a question and five candidate answers and asked to
rate each on a scale from 1 to 5 for each of our two criteria (C1 and C2).
Our main goal is to compare the quality of the answers across 5 conditions (3 systems, 2 posts from the site). Raters may have different interpretations of the absolute scales—for example, some raters could be more generous than others overall in terms of the numerical rating, even if they agree on the ranking of systems.
There are several possible ways to factor out this bias. One way is to compute standard scores, a.k.a. z-scores, for each annotator’s distribution of responses on each criterion. Consider C1: from the ratings of an annotator a we have the empirical distribution
pC1a (yC1i,a ∣ xi) where i indexes the items (answers, of which multiple ones may belong to the same question), and likewise for C2. For each of these distributions we fit a normal distribution by computing mean and standard deviation. For an absolute rating yC1i,a , its zscore zC1i,a is its number of standard deviations above the mean rating for that annotator on that metric (a negative z-score indicates it is below the mean). Averaging the z-scores for a particular condition, we can see whether annotators tended to rate outputs in that condition with higher or lower scores than the other outputs they saw in the sample. Note that the z-score computation ignores the grouping of answers from different conditions into questions, so it is not directly measuring annotators’ rankings of candidate answers to a particular question.

heading: C Further Details
text: ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
3 A2. Did you discuss any potential risks of your work? 6
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: 3, Appendix
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 3

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? 3

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3.1, 4.1
C 3 Did you run computational experiments? 4.2

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 4.2, appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4.2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 4

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? 4, appendix

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 4

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? 3,4
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: 4 (footnote)

heading: Future Work
text: The models we have presented are intended only as baselines for future research, not for deployment. Models should be carefully stress-tested for undesirable heuristics/ biases before deployment. Systems for the generation task, in particular, would risk misleading language learners with plausible but incorrect answers, so it is important to not deploy a generation
system until it is approximately as reliable as existing non-automated alternatives, and to present the output with caveats. Potential biases reflecting the demographics of authors represented in the training data (in terms of native language, level of English proficiency, etc. ) also need to be considered if models are deployed for different target populations. Limitations
One limitation of our dataset, ELQA, is that the corpus only contains questions in English and about English. However, Stack Exchange has sites with questions about other languages and our main data extraction scripts are general enough that they can be used to create corpora for other sites on Stack Exchange. Of course, language-specific processing steps, quality assurance and analysis must be applied before releasing such data. Most importantly, the models we have presented here are intended only as baselines for future research, not for deployment. Potential biases reflecting the demographics of authors represented in the training data (in terms of native language, level of English proficiency, etc. ) also need to be considered if models are deployed for different target populations. Moreover, many of these types of questions are found on the web, and a lot of the same topics are brought up by many users, so a model’s ability to generate correct answers cannot necessarily be attributed to abstract reasoning