SlowGuess/NAACL_2022_OCR huggingface dataset:
Ying-Jian Wang, Hui-Fang Shang, and Paul Briody. 2013. Exploring the impact of using automated writing evaluation in english as a foreign language university students' writing. Computer Assisted Language Learning, 26(3):234-257.
SarahWiegreffe and Yuval Pinter.2019. Attention is not not explanation. arXiv preprint arXiv:1908.04626.
Joshua Wilson, Cristina Ahrendt, Emily A Fudge, Alexandria Raiche, Gaysha Beard, and Charles MacArthur. 2021. Elementary teachers' perceptions of automated feedback and automated scoring: Transforming the teaching and learning of writing using automated writing evaluation. Computers & Education, 168:104208.
Linzhong Xia, Mingxiang Guan, Jun Liu, Xuemei Cao, and Dean Luo. 2020. Attention-based bidirectional long short-term memory neural network for short answer scoring. In International Conference on Machine Learning and Intelligent Communications, pages 104-112. Springer.
Ning Xie, Gabrielle Ras, Marcel van Gerven, and Derek Doran. 2020. Explainable deep learning: A field guide for the uninitiated. arXiv preprint arXiv:2004.14545.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 1480-1489.
Michael Yen, Sergey Karayev, and Eric Wang. 2020. Analysis of grading times of short answer questions. In Proceedings of the Seventh ACM Conference on Learning@ Scale, pages 365-368.
Jiajie Zou and Nai Ding. 2021. Deep neural networks evolve human-like attention distribution during goal-directed reading comprehension.
A BERT-based Grader Construction
We constructed our DL-based graders based on the well-known pre-trained language model BERT (Devlin et al., 2018). Similar to (Sung et al., 2019b), we implemented the BERT-based graders by adopting the pre-trained bert-base-cased model (12 layers, 768 neurons in each hidden layer, 12 attention heads, and a total of 110M parameters) with a sequence classification layer on top provided by the Python package Transformers<sup>3</sup>. Then, we fine-tuned the whole model on the training set of each question prompt. It should be noted that only the student-authored answers were used as input to train the model. During model training, the batch size was set to 16, and we selected the number of epoch from  $\{1, 2, 3, 4, 5\}$  and the learning rate from  $\{1e - 5, 5e - 5, 1e - 4\}$ . For the optimizer, we used Adam with decoupled weight decay (i.e., AdamW). To determine the best values for the hyperparameters mentioned above, we trained the models with different combinations of parameter values and selected the best model based on its performance on the validation set for each question prompt. The run time for fine-tuning each BERT-based grader is roughly 1.5 hours (5 epochs, with NVIDIA Tesla P4 GPU). Table 4 details the parameter choices for each constructed model and their corresponding performance (measured by QWK). By comparing QWK1 and QWK2 reported in Table 4, the difference between which can be regarded as the gap between human graders and the constructed BERT-based graders, we can conclude that the BERT-based graders achieved a substantial level of predictive performance in most of the question prompts. Note that the codes for constructing the models can be accessed via https://github.com/douglashiwo/AttentionAlignmentASAS.
B Pre-/Post-task Questionnaires
In Study 1, before scoring answers to a question prompt, the participants were required to answer the following three questions to indicate their interest, familiarity, and perceived difficulty of the question prompt on a rating scale of [1, 5]:
- To what extent are you interested in the prompt topic? (with 1 being not interested at all and 5 being very interested)
Table 4: Performances of the constructed BERT-based graders in different question prompts. QWK1 denotes the QWK between the Ground-truth Score1 and Ground-truth Score2 provided in the original dataset, while QWK2 denotes the QWK between the Ground-truth Score1 and the predictions generated by our BERT-based graders.
Prompt ID	QWK1	QWK2	Learning rate	Epoch
1	0.950	0.860	5e-5	4
2	0.900	0.716	1e-5	5
3	0.681	0.291	1e-5	5
4	0.683	0.729	5e-5	5
5	0.962	0.891	5e-5	5
6	0.952	0.831	5e-5	5
7	0.959	0.731	5e-5	3
8	0.866	0.660	5e-5	2
9	0.782	0.730	5e-5	3
10	0.887	0.719	1e-5	4
- To what extent are you familiar with the prompt topic? (with 1 being not familiar at all and 5 being very familiar)
- How would you describe the difficulty level of the prompt topic to students of Grade 10 in the US (i.e., the second year in high school)? (with 1 being very easy and 5 being very difficult)
In Study 2, the participants in the Experimental group were required to answer the following two questions to share their opinions regarding the highlighted important words detected by BERT-based graders on a rating scale of [1, 5]:
- To what extent do you think the text highlighted by our automatic model grader are actually "important" in determining the quality of an answer? (with 1 being not important at all and 5 being very important)
- To what extent do you think the text highlighted by our automatic model grader helps you grade answers? (with 1 being not helpful at all and 5 being very helpful)
C The Developed Grading System
Figure 1 shows the screen of the developed grading system in which participants scored an answer in Study 1. Figure 2 shows the screen of the developed grading system in which participants annotated an answer after assigning a score value to the answer in Study 1. Figure 3 shows the screen of the
developed grading system in which participants in the Experimental group scored an answer in Study 2; the screen that participants in the Control group is similar to Figure 3 but without any answer words being highlighted.
Marking Rubric
Essay answers to the prompt above are scored on a four-point scale (0-3) using a holistic scoring method. Please refer to the following marking rubric to grade the essay answers.
Here are some key points that correct essay answers are likely to contain:
- You need to know how much vinegar was used in each container.
- You need to know what type of vinegar was used in each container.
- You need to know what materials to test.
- You need to know what size/surface area of materials should be used.
- You need to know how long each sample was rinsed in distilled water.
- You need to know what drying method to use.
- You need to know what size/type of container to use.
- Other acceptable responses.
Below is an answer given to the prompt question and your task is to score it on a four-point scale (0-3):
In order to replicate this experiment, I would need to know additional information such as the four different samples that they used (because I could have chosen metal, carbboard and many other sample materials that they didn't use and would get different results. Also I would also need to know the amount of vinegar to pour because this can caute a major change. Lastly, they might want to tell where to sit the samples while they dry for 30 minutes because if they are sitting in room temp. or by a light source makes a difference too.
Figure 1: The screen of the developed grading system for a participant to score an answer in Study 1:  $①$  shows the marking rubric to participants;  $②$  shows the instruction and an answer to be graded;  $③$  shows the score scale (i.e., [1, 3]); and  $④$  shows a participant's overall progress in the study (i.e., how many answers the participant has graded and how many more needs to be graded). After clicking the Next button, the participant will enter the next screen to annotate important words contained in the same answer.
Figure 2: The screen of the developed grading system for a participant to annotate an answer in Study 1:  $①$  gives annotation instruction to a participant;  $②$  displays the same answer that the participant scored in the previous screen; and  $③$  allows a participant to remove to all existing annotations and start the annotation from scratch again, and the participant is informed that annotation is not mandatory if she believes that there is nothing contributing/hurting the quality of the answer.
Figure 3: The screen of the developed grading system for a participant in the Experimental group to score an answer in Study 2:  $①$  shows the marking rubric to participants;  $②$  shows the instruction and an answer to be graded;  $③$  shows the score scale (i.e., [1, 3]);  $④$  shows a participant's overall progress in the study (i.e., how many answers the participant has graded and how many more needs to be graded); and particularly,  $⑤$  highlights the important words detected by BERT-based graders developed in our work. After clicking the Next button, the participant will be directed to assess the next answer.
In order to replicate this experiment , I would need to knowadditional information such as the four different samples thatthey used ( because I could have choosen metal , carboard andmany other sample materials that they didn ' t use and would getdifferent results . Also I would also need to know the amount ofvinegar to pour because this can caute a major change . Lastly ,they might want to tell where to sit the samples while they dry for30 minutes because if they are sitting in room temp . or by a lightsource makes a difference too .