SlowGuess/NAACL_2022_OCR huggingface dataset:
Zijie Zeng, Xinyu Li, Dragan Gasevic and Guanliang Chen*
Centre for Learning Analytics, Monash University
Melbourne, Victoria, Australia
{Zijie.Zeng, Xinyu.Li, Dragan.Gasevic, Guanliang.Chen} @monash.edu
Deep Learning (DL) techniques have been increasingly adopted for Automatic Text Scoring in education. However, these techniques often suffer from their inabilities to explain and justify how a prediction is made, which, unavoidably, decreases their trustworthiness and hinders educators from embracing them in practice. This study aimed to investigate whether (and to what extent) DL-based graders align with human graders regarding the important words they identify when marking short answer questions. To this end, we first conducted a user study to ask human graders to manually annotate important words in assessing answer quality and then measured the overlap between these human-annotated words and those identified by DL-based graders (i.e., those receiving large attention weights). Furthermore, we ran a randomized controlled experiment to explore the impact of highlighting important words detected by DL-based graders on human grading. The results showed that: (i) DL-based graders, to a certain degree, displayed alignment with human graders no matter whether DL-based graders and human graders agreed on the quality of an answer; and (ii) it is possible to facilitate human grading by highlighting those DL-detected important words, though further investigations are necessary to understand how human graders exploit such highlighted words.
Automatic Text Scoring refers to the task of applying computational techniques to score written text based on certain grading criteria (Alikaniotis et al., 2016). Since its inception (Page, 1966), Automatic Text Scoring has been actively investigated and applied to assist educators in scoring student-written text, e.g., short answer questions and essays, which are often referred to as Automated Short Answer Scoring (ASAS) (Brew and Leacock, 2013) and Essay Scoring (Rodriguez et al., 2019). Driven by the
great success of Deep Learning (DL) techniques in various NLP tasks, researchers have endeavored to apply them to construct ASAS systems in recent years (Xia et al., 2020; Sung et al., 2019b,a), some of which displayed performance comparable to human graders. For instance, (Xia et al., 2020) showed that the average performance of an attention-based bidirectional LSTM model could be up to 0.71 (measured by the metric Quadratic Weighted Kappa) in the ASAS competition organized by the Hewlett Foundation, which can be deemed as achieving a substantial agreement with human graders.
Though being effective, DL-based ASAS systems have been widely plagued by the inability to explain how the quality of an answer is graded. The lack of understanding the underlying working mechanism of these systems, beyond question, may stop educators from adopting them in teaching practice as there remain concerns that the use of such ASAS systems might unintentionally encourage students to produce formulaic writings, i.e., writing that is often lengthy and involve complex words, but not much quality content (Wilson et al., 2021; Chen and Cheng, 2008; Wang et al., 2013). Inspired by the research efforts in the broader NLP communities, i.e., those focusing on dissecting complex deep neural net architectures and explaining how they work (Serrano and Smith, 2019; Wegreffe and Pinter, 2019; Jain and Wallace, 2019; Xie et al., 2020; Sun et al., 2021), in the study presented in this paper we aimed to gain a better understanding of how DL-based ASAS systems work. Specifically, we investigated (i) the alignment between DL-based graders and human graders in terms of the words they think are important in the task of Short Answer Scoring and (ii) whether the important words identified by DL-based graders can be of use to human graders in the marking process. Formally, this study was guided by the following two Research Questions:
Human Language Technologies, pages 191 - 205
RQ1 To what extent do DL-based graders align with human graders regarding the words that are important in assessing answer quality?
RQ2 Can the important words identified by DL-based graders be used to facilitate human graders to perform marking?
We conducted two user studies to answer RQ1 and RQ2. For RQ1, with the dataset provided by the Hewlett Foundation, we constructed relatively simple but effective BERT-based graders (i.e., coupling BERT with a single classification layer for prediction), whose performances were comparable to those reported in recent studies (Xia et al., 2020; Surya et al., 2019). To locate words that were essential in determining answer quality, we extracted weights allocated to different input words in the self-attention layers of BERT. Then, we ran a user study to ask human graders to not only score an answer but also annotate the words they believed were important in contributing to or hurting the answer quality. We measured the alignment between BERT-based graders and human graders with the aid of the Jaccard coefficient. Building upon the results of RQ1, we further implemented a randomized controlled trial to investigate whether displaying the important words identified by BERT-based graders can help human graders improve marking accuracy and efficiency to answer RQ2.
In summary, our work contributes to the research on Automated Text Scoring with the following main findings: (i) text spans contained in an answer that increase the answer quality, compared to those decreasing answer quality, are more likely to be identified by human graders; (ii) there exists a certain level of alignment between DL-based graders and human graders regarding the words they think are important in assessing answer quality no matter whether they agree on the quality score of an answer; and (iii) the important words detected by DL-based graders can be potentially used to facilitate human grading, though more research efforts are required to understand how these words are to be utilized by human graders.
2.1 Automated Short Answer Scoring
As a sub-branch of Automatic Text Scoring, ASAS aims to leverage statistical and machine learning techniques to assess the quality of short answers authored by students in education (Burrows et al.,
2015; Xia et al., 2020). Given its important role played in supporting educators to scale up their teaching practices (e.g., to meet the need of marking up to tens of thousands of answers submitted by students and provide informative feedback in a Massive Open Online Course (Pappano, 2012) in a relatively short amount of time), ASAS has been drawing attention from researchers since its inception (Page, 1966). Typically, ASAS can be tackled as either a classification problem (Xia et al., 2020) or a regression problem (Sahu and Bhowmick, 2020). As surveyed in (Bonthu et al., 2021), the approaches used to tackle ASAS often fall into two categories. One is based on traditional machine learning techniques such as SVM (Gleize and Grau, 2013; Mohler et al., 2011; Higgins et al., 2014), K-means (Sorour et al., 2015), Linear Regression (Nau et al., 2017; Heilman and Madnani, 2015; Higgins et al., 2014), and Random Forests (Higgins et al., 2014; Ramachandran et al., 2015; Ishioka and Kameda, 2017), all of which heavily rely on the input of manually-crafted features. For example, Sultan et al. (2016) devised a set of features which were based on a lexical similarity (i.e., similarities between words identified by a paraphrase database (Ganitkevitch et al., 2013)) and monolingual alignment (Sultan et al., 2014), and input the designed features to a ridge regression model to obtain the score of an answer. The other category is based on DL techniques, such as Bi-LSTM (Xia et al., 2020; Kim et al., 2018) and BERT (Sung et al., 2019b), which, in contrast to traditional machine learning approaches, often demonstrate superior performance without the need to engineer human-crafted features. For instance, Sung et al. (2019b) proposed a fine-tuned BERT model for short answer scoring, which outperformed human experts in classifying short answers collected in the subject of psychology.
It has been documented that the use of certain Automated Text Scoring systems in education tends to promote formulaic writing among students, i.e., producing lengthy and complex but not quality content (Wilson et al., 2021; Chen and Cheng, 2008; Wang et al., 2013). As a result, there remain concerns about the ability of these ASAS systems in supporting teachers and instructors (Wilson et al., 2021). To alleviate this issue, some studies were proposed to investigate how ASAS systems work to reach a decision (Higgins et al., 2014), which mainly focused on the ASAS systems powered by
traditional machine learning approaches. As an example, Higgins et al. (2014) demonstrated that including syntactically-informed features could boost the predictive performance of an ASAS model, enhancing the model's ability to identify high-quality responses written by students. To our knowledge, it remains largely unexplored the interpretation ability and reliability of ASAS systems powered by up-to-date DL techniques. An exception is presented by (Manabe and Hagiwara, 2021), in which a toolkit named EXPATS is introduced to enable people to visualize not only models based on traditional machine learning techniques but also those based on DL techniques as well as the predictions produced by these models. Our work distinguished itself from previous studies by collecting human-annotated data to inspect how an answer was evaluated by an ASAS system, i.e., comparing the overlap between the important answer words identified by DL-based graders and human graders, to shed light on the extent to which DL-based graders acted like human graders in the marking process.
2.2 Interpretability of Deep Learning Models in NLP
The interpretability and explainability of a predictive model have been widely acknowledged as an essential factor in helping human users understand the validity of a prediction and decide whether to adopt the model for practical use (Jacovi and Goldberg, 2020; Belinkov et al., 2020). In this strand of research, one common method is called test-based (Li et al., 2015; Jain and Wallace, 2019; Sun et al., 2020; Lei et al., 2016), which interprets a prediction by identifying relevant parts of input data that drive the prediction, e.g., words contained in a long sentence that play major roles in determining the overall sentimental polarity of the sentence. When it comes to the application of DL-models equipped with attention mechanism (Bahdanau et al., 2014) for NLP tasks, researchers often regard the weights assigned by the attention layer to different parts of input text as indicators of their importance to the model prediction (Mohankumar et al., 2020; Yang et al., 2016; Wang et al., 2016). However, it remains disputable to use attention weights to measure the importance of input text (Wiegreffe and Pinter, 2019; Serrano and Smith, 2019; Jain and Wallace, 2019). For instance, by manipulating attention weights in well-trained models to analyze their influences upon predictions in text classifi-
cation, Serrano and Smith (2019) concluded that attention weights can only noisily predict the overall importance of different input text to a model prediction and thus should not be regarded as an optional measure for strict importance ranking. On the contrary, Wegreffe and Pinter (2019) claimed that the feasibility of attention-as-explanation depends on the concrete definition of explanation. In line with this claim, when inspecting a Transformer-based model for non-factoid question answering, Bolotova et al. (2020) extracted the self-attention weights assigned to words in an answer to measure their importance. Similarly, Zou and Ding (2021) analyzed the self-attention weights in three Transformer-based models to investigate whether these models displayed human-like attention in the task of goal-directed reading comprehension. Similar to these studies, we treated the self-attention weights in a BERT-based model as proxies to reveal the importance of different input text, but in the task of ASAS.
2.3 Human Grading
It has been documented that human grading can be affected by various factors, e.g., students' ethnicity (Hinnerich et al., 2011; Van Ewijk, 2011) and gender (Protivínský and Münich, 2018), and the order of answers to be graded Yen et al. (2020). For instance, Protivínský and Münich (2018) showed that teachers' grading was biased towards female students in subjects of mathematics and the native language (Czech), and the observed grading difference was likely due to the different levels of non-cognitive skills (e.g., engagement in the classroom) displayed by female and male students. In a different vein, Yen et al. (2020) demonstrated that human graders spent much less time if they were presented with answers sorted according to their similarities with the marking rubric. Previous research on text reading showed that highlighting can facilitate people to remember and comprehend reading materials (Fowler and Barker, 1974; Lorch, 1989; Lorch et al., 1995; Dodson et al., 2017; Silvers and Kreiner, 1997). Inspired by this, we were interested in investigating whether it can facilitate human graders to score answer quality by highlighting words that DL-based graders identified as important in ASAS.
This study was approved by the Human Research Ethics Committee at Monash University (Project ID 30074). In the following, we first describe the task and dataset based on which we examined the alignment between DL-based graders and human graders, followed by the construction of the DL-based graders. Then, we detailed the setup of the two user studies we implemented to answer RQ1 and RQ2.
3.1 Task, Dataset, and DL-based Graders
This study focused on the ASAS challenge organized by the Hewlett Foundation in Kaggle<sup>1</sup>, whose dataset contains over 17,000 answers written by students of Grade 10 in the US to 10 different question prompts. The subjects of these question prompts include such subjects as science, biology, and English. Notice that each of the question prompts and the corresponding collected answers have their own unique characteristics, e.g., different marking rubrics, scoring scale (some are on [0,2] and the others are on [0,3]), whether relevant source material was provided, and the average length of answers (ranging from 40 to 60 words). It is worth noting that each of the answers contained in the original dataset was double-scored, i.e., being rated by two independent human graders, and we denoted these scores as Ground-truth Score1 and Ground-truth Score2, respectively, in this study. As specified in the challenge, Ground-truth Score1 is the final score an answer received and also the score that a model should aim to predict. As for Ground-truth Score2, it can be used as a measure of reliability. For instance, researchers can measure the agreements (i) between a model and Ground-truth Score1 and (ii) between Ground-truth Score1 and Ground-truth Score2, and then calculate the difference between the two agreements to gain a rough understanding of the gap between the constructed model and human graders.
In line with previous studies (Xia et al., 2020; Surya et al., 2019; Sung et al., 2019b), we tackled ASAS as a classification problem, i.e., classifying answers to different quality groups. Inspired by the great success of BERT (Devlin et al., 2018) in various NLP tasks, we also used it to construct DL-based graders to score an answer automatically. The model structure is relatively simple,
i.e., we only coupled BERT with a single classification layer for prediction and then adapted the model to capture the unique characteristics of this task by fine-tuning on the graded answers. Given the unique characteristics of the different question prompts contained in the dataset, we decided to build a BERT-based grader for each of the question prompts. For each question prompt, we randomly split the answers in the ratio of 8:1:1 as training, validation, and testing sets. The details of the construction process are provided in Appendix A. As suggested by the challenge requirement, we used the metric Quadratic Weighted Kappa (QWK) to measure the performance of DL-based graders, which ranged from 0.660 to 0.891 for the 10 question prompts. As our goal was to investigate that, when a DL-based model was able to achieve a substantial level of performance in assessing answer quality, whether and to what extent it aligned with human graders in the marking process. Thus, we chose only the three question prompts (i.e., Prompt 1, 5, and 6, which were all graded on a scale of [0, 3]) in which DL-based graders achieved the best prediction performance (i.e., with QWK 0.831, 0.860, and 0.891, respectively) for the two user studies, as described below.
3.2 Study One
For RQ1, we designed a study to collect answer annotations from human graders, i.e., important words or text spans that human graders believe to increase or decrease the quality of an answer.
Participants. For Study 1, we recruited a total of 20 participants (7 females, 13 males), all of whom had received at least a master's degree, were proficient in English, and were employed by Monash University. In particular, all the participants had certain years of prior teaching experience, i.e., 13 participants less than 3 years, four had  $3\sim 5$  years, and three more than 5 years of experience. All participants were informed of the purpose of this study (and also Study 2) and signed consent forms before participating in the studies.
Study Setup. For each question prompt, we randomly selected five answers from each quality level (i.e., scores in the scale of [0, 3]) from the testing data set, which resulted in a total of 60 answers. In particular, we developed a grading system to allow participants to not only score an answer but also annotate the words or text spans that they thought important in determining the quality levels. When an-
<sup>1</sup>https://www.kaggle.com/c/asap-sas
notating an answer, the participants were required to annotate not only the text spans that increasing answer quality but also those decreasing answer quality, which were correspondingly denoted as Positive and Negative text spans in later analysis. We provided the screenshots of the grading system in Appendix C. Each of the 20 participants was required to attend a 90-minute session to grade 30 answers so that we collected a total of 600 assessment scores and annotations from our participants. Every answer was graded by 10 participants. In particular, we assigned the selected answers to the participants in a way that each participant was required to score answers of every quality level. After completion, we compensated each participant with a gift card worth \( \\( {75} \) AUD for their time (i.e., \( \$ {50} \) AUD per hour,which is comparable to the hourly rate for people with a master's degree in Australia).
Procedure. To ensure the quality of the collected data, we expected the participants to undertake adequate training to understand how they should mark before moving to the actual answer scoring and annotation. Therefore, the grading system we developed provided two modes for participants, i.e., Practice for pre-task training and Actual Task for actual data collection. Only after finishing the activities scheduled in Practice, the participants were allowed to start the Actual Task. Both Practice and Actual Task required a participant to evaluate an answer by following the steps described below. The main difference lied in the sources from which the presented answers were selected, i.e., validation set for Practice and testing set for Actual Task.
(1) Material Reading. A participant was asked to read a prompt, an article relevant to the prompt (if available in the original dataset), marking rubric, and exemplar answers with scores assigned by human graders (i.e., Ground-truth Score1).
(2) Pre-task questionnaire. The participant was asked to indicate their familiarity, interestingness, and perceived difficulty of the question prompt by answering three questions on a rating scale of [1, 5], which were provided in the Appendix B.
(3) Answer assessment. An answer was presented for the participant to rate its quality. Note that the marking rubric was displayed along with the to-be-graded answer to facilitate the participant to mark.
(4) Answer annotation. The participant was instructed to highlight words and text spans that they identified as important in determining answer quality.
Grading Alignment Measurement. Here, we introduce how we detected the words that DL-based graders paid attention to in the marking process and further measured the overlap between these words and those annotated by the human participants in Study 1. Similar to the work by Bolotova et al. (2020), we extracted the words by calculating weights assigned to each token in the self-attention layers of the adopted BERT model. Specifically, for a token  $t$  contained in an answer  $\text{Ans}$ , its attention weight is calculated as follows:
$$
A t n \_ S c o r e (t) = \frac {\sum_ {i \in A n s} w _ {i \rightarrow t}}{| A n s |} \tag {1}
$$
Here,  $|Ans|$  denotes the length of  $Ans$ , and  $w_{i\rightarrow t}$  denotes the attention weight assigned by token  $i$  to token  $t$ , which can be retrieved and calculated in the attention layers of BERT. In more detail, there were 12 attention heads in the adopted BERT model; we first averaged the weights contained in these attention heads and then retrieved  $w_{i\rightarrow t}$  in the averaged attention head. In brief, for a token in an answer, we summed up the attention weights from each token contained in the answer to the target token and then normalized the sum by the answer length to obtain the importance of the target token. Then, we could select the top K words with the largest attention weights as the set of important words identified by BERT-based graders. As suggested by Bolotova et al. (2020), we determined the value of K with the aid of a linear regression model, which took the length of an answer as the only input to predict the number of important words it should output. The regression model was trained based on the human-annotated data we collected in Study 1. Then, we followed the approach adopted by Bolotova et al. (2020) and measured their grading alignment by calculating the Jaccard coefficient between the two sets of important words identified by a human grader and a BERT-based grader. We also followed Qu et al. (2019) to exclude stop words from the set of important words before calculating the Jaccard coefficient. The stop words were detected with the aid of NLTK toolkit<sup>2</sup>. We first calculate the Jaccard coefficients over all
$^{2}$ https://www.nltk.org/
< answer, human grader > data pairs collected in this study and then averaged them as the final measure of the alignment between BERT-based graders and the human graders.
3.3 Study Two
For RQ2, we conducted a randomized controlled experiment to investigate whether human grading could be facilitated by observing highlighted important words detected by DL-based graders.
Participants. We randomly invited 10 out of the 20 participants who had taken part in Study 1 for Study 2. As for the prior teaching experience, seven out of the 10 participants were with less than 3 years, two were between 3 to 5 years, and one with more than 5 years of experience.