SlowGuess/NAACL_2022_OCR huggingface dataset:
Study Setup. The 10 participants were randomly assigned to two groups, i.e., 5 in the control group and 5 in the experimental group. Only the participants in the experimental group were displayed with the highlighted words detected by BERT-based graders. We randomly selected 36 answers from the testing data for participants to mark. It is worth noting that these answers were unseen to the participants in Study 1, and these answers covered all quality levels in all question prompts. Each participant was required to attend a 45-minute session to assess a total of 24 answers. Each answer was assessed by the same number of participants from the control group and the experimental group. Similar to Study 1, every participant was required to score answers of all quality levels. After completion, each participant received a gift card worth $40 AUD to compensate for their time.
Procedure. The procedure of Study 2 only consisted of two main steps from the procedure of Study 1 and the participants did not need to receive pre-task training again, as described below.
(1) Material Reading. Similar to Study 1, we presented participants with all relevant materials to help them get familiar with the task requirement.
(2) Answer Assessment. In this step, the participants in the control group were displayed with answers without any highlighted words. As for the answers presented to participants in the experimental group, the important words contained in these answers, i.e., those detected by applying DL-based graders as described in Section 3.2, were highlighted.
It is noteworthy that the participants were informed about (i) the nature of these highlighted words (i.e., important words detected by DL-based graders) and (ii) the reliability of these highlighted words (i.e., some might be helpful for assessing answer quality while the others are not). We used the linear regression constructed in Section 3.2 to determine the number of words that should be highlighted to a participant.
(3) Post-task questionnaire. This step was only for the participants in the experimental group after completing the whole study. The participants were asked to answer two questions on a rating scale of [1, 5] with regard to the usefulness of the highlighted words for marking.
With the completion of the study, we compared the grading performance of the participants in the two groups from two perspectives. One is accuracy, which was calculated as the QWK between the participant-provided scores and the ground-truth scores. Here, we regarded that a participant-provided score matched the ground-truth data if it matched either Ground-truth Score1 or Ground-truth Score2 in the original dataset; otherwise, we simply regarded Ground-truth Score1 as the ground-truth. The other is efficiency, which was calculated as the average amount of time that participants spent in scoring answers. The amount of time that a participant used in assessing an answer was computed as the time difference between the moment when the participant entered the screen of the grading system to assess the answer and the moment when the participant clicked to move to grade the next answer.
Recall that all answers used in this study were assessed on a grading scale of [0, 3]. For the following analysis, we aggregated and denoted the answers whose Ground-truth Score1 is 0 or 1 in the original dataset as Low quality, and those of score 2 or 3 as High quality.
4.1 Results on RQ1
Table 1 details the fractions of answers with different types of annotations, i.e., positive/negative text spans that increased/decreased answer quality. We observed that  $89.3\%$  answers were annotated by our
Table 1: The fraction of answers received different types annotations from the participants in Study 1.
Quality Level	Low	High	ALL
% Answers with positive annotations	53.7	95.3	74.5
% Answers with negative annotations	52.3	32.7	42.5
% Answers with annotations	81.3	97.3	89.3
participants, among which  $74.5\%$  received positive annotations while only  $42.5\%$  received negative annotations. This indicates that text spans contributing to answer quality were more likely to be determined and identified by human graders than those lowering answer quality. This is corroborated by the results in both high-quality and low-quality answer categories. For instance, the fraction of answers with positive and negative annotations were  $95.3\%$  and  $32.7\%$ , respectively. We can make similar observations in low-quality answers, but the difference was only  $1.4\%$ .
Table 2 describes the alignment between BERT-based graders and human graders involved in Study 1, i.e., the Jaccard coefficients between the two sets of the important words identified by them when performing marking. As a baseline for comparison, we selected important words based on randomly-assigned weights and measured their overlap with the important words annotated by human graders (as detailed in the column RANDOM). By comparing the results of RANDOM and ALL, we can conclude that there existed some agreement between BERT-based graders and human graders. When considering both positive and negative annotations, the alignment reached the value of 0.252 (about  $69\%$  more than that of randomly-annotated important words, i.e., 0.149). Also, we noticed that the alignment in high-quality answers was more than that in lower-quality answers (0.275 vs. 0.224). When considering only positive/negative annotations for measurement, there was a higher level of alignment in positive annotations than negative annotations (0.251 vs. 0.174). This suggests that DL-based graders and human graders were more likely to agree with each other regarding what makes a good answer than what makes a bad answer. This finding is related to what we observed in Table 1, i.e., human graders made more positive annotations than negative annotations in Study 1. These findings together imply that, to a certain degree, there do exist similarities between the BERT-based graders and human graders in the grading process.
It should be noted that (i) only 40 out of the 60 answers used in Study 1 were correctly assessed by BERT-based graders and (ii) only 342 out of the 600 (about  $57\%$ ) collected assessment scores provided by our participants were in agreement with the ground-truth (i.e., matching to either Ground-truth Score1 or Ground-truth Score2 in the original dataset). To further investigate grading alignment in different conditions (e.g., BERT-based graders and human graders simultaneously failed to assess the quality of an answer), we further divided the collected data into two groups for analysis, i.e., G1 consisting of answers for which human graders and BERT-based graders simultaneously delivered correct/incorrect answer scores and G2 consisting of the other data for which either human graders or BERT-based graders failed to produce correct assessments, as detailed in Table 2. The G1 alignment was similar to those calculated by taking all answers into account (i.e., the ALL column). To our surprise, there also exists an overall alignment of 0.248 between BERT-based graders and human graders in G2. This means, even in the situations where human graders were able to correctly assess the quality of an answer but BERT-based graders failed to do so (or the opposite), there is some overlap between the human-annotated and DL-detected important words. This implies that, in certain cases, even human graders and DL-based graders disagreed on the score value of an answer, they did reach a partial agreement on the words that were important for assessing the answer's quality.
4.2 Results on RQ2
It should be pointed out that 25 out of the 36 selected answers for our participants to grade in Study 2 were accurately scored by BERT-based graders. The reasons we also included the inaccurately-scored answers are two-fold. Firstly, our ultimate goal is to use important words located by BERT-based graders to facilitate human grading in practice. As a DL-based grader is unlikely to derive correct predictions all the time in real-world scenarios, it is very likely that some of the DL-identified plausibly-important words will be presented to human graders in certain cases. Secondly, as demonstrated in Study 1, when BERT-based graders disagreed with human graders regarding an answer's quality, they still had moderate overlap between the important words they identified. Thus, we were interested in investigating whether human graders
Table 2: The overlap between human graders' different types of annotations (i.e., Positive or Negative) with the important words detected by BERT-based graders. The column ALL includes all collected data for analysis; G1 includes only the answers that were simultaneously correctly/incorrectly assessed by the two types of graders, and G2 includes the other data in which either BERT-based graders or human graders did not match to the ground-truth scores. Paired t-tests were applied to examine the difference between Random and ALL regarding the same type of annotation while unpaired t-tests were adopted to examine the difference between Random and (G1, G2) regarding the same type of annotation. Significant results were marked with either \*\* (if  $p < 0.001$ ) or  $*$  (if  $p < 0.01$ ).
RANDOM	ALL	G1	G2
Annotation type	Pos	Neg	Pos & Neg	Pos	Neg	Pos & Neg	Pos	Neg	Pos & Neg	Pos	Neg	Pos & Neg
Low quality	0.128	0.123	0.136	** 0.235	** 0.176	** 0.224	** 0.268	* 0.176	** 0.233	** 0.205	0.178	** 0.210
High quality	0.155	0.116	0.160	** 0.261	** 0.170	** 0.275	** 0.256	* 0.173	** 0.273	** 0.267	0.165	** 0.278
Average	0.146	0.120	0.149	** 0.251	** 0.174	** 0.252	** 0.260	** 0.174	** 0.254	** 0.241	* 0.172	** 0.248
Table 3: The participants' grading performance in Study 2. The difference between the two groups regarding their average time spent grading one answer (measured in seconds) was tested to be significant with paired t-test  $(p < 0.01)$ .
Metrics	Control	Experimental
QWK	0.71	0.74
Avg. time per answer	42.17	54.83
could be facilitated by observing (not necessarily correct) important words identified by BERT-based graders. The grading performances of the participants are given in Table 3, i.e., QWK for accuracy and the average amount of time spent in scoring an answer for efficiency. The results showed that, by highlighting important words detected by BERT-based graders, we could facilitate human grading in terms of grading accuracy. However, contrary to our expectation, the Experimental participants spent roughly 12 more seconds in scoring an answer than the Control participants. This is probably because, as being informed that the highlighted words were identified by a DL-based model and not necessarily useful in discerning answer quality, the participants allocated more time in scrutinizing the answer text to judge its quality. This was partially supported by their responses to questions in the post-task questionnaire, i.e., an average rating of only 3.0 (out of 5.0) was reported regarding whether the participants considered the highlighted words as actually important, but an average rating of 3.8 (out of 5.0) was reported regarding the usefulness of the highlighted words for marking.
5 Discussions and Conclusion
Given the increasing popularity of DL-based Automatic Text Scoring systems in education, this work investigated the interpretability of these sys
tems regarding their alignment with human graders in the task of ASAS. Through two well-designed user studies, we demonstrated that (i) there exists certain alignment between DL-based graders and human graders in terms of the important words they identified for assessing answer quality; and (ii) it is possible to utilize the important words detected by DL-based graders to facilitate human grading.
Though several interesting findings have been enabled, it is not the time yet to advocate that we should incorporate such DL-identified important words into ASAS systems to facilitate human grading due to the following limitations in our study. Firstly, as observed in Study 1, DL-based graders displayed certain agreement with human graders, even in the circumstances of being unable to correctly assess the quality of an answer. It remains unknown what types of words (or concepts) DL-based graders agree/disagree on with human graders. For example, would it be possible that both types of graders tend to hold similar misconceptions (e.g., identifying the same type of plausibly-correct words)? This calls for future research to characterize and analyze the different types of important answer spans identified by DL-based graders and human graders under different circumstances. By doing this, we can potentially derive more insights on how to further improve the prediction performance of DL-based graders. Secondly, and more importantly, it remains unknown the sense-making process the human graders follow to make use of the DL-detected important words. As showed in Study 2, the human graders in the Experimental group neither agreed nor disagreed that the DL-highlighted words were important, but they considered the DL-highlighted words were of value and delivered better grading accuracy (at the cost of a lower grading efficiency). Would it be the case that the highlighted words provoke more
in-depth thinking and understanding of an answer among human graders and thus enabling them to better assess the answer quality? This motivates us to conduct further studies to investigate human graders' marking process in using DL-highlighted words in the future.
Dimitrios Alikaniotis, Helen Yannakoudakis, and Marek Rei. 2016. Automatic text scoring using neural networks. arXiv preprint arXiv:1606.04289.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Yonatan Belinkov, Sebastian Gehrmann, and Ellie Pavlick. 2020. Interpretability and analysis in neural nlp. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 1-5.
Valeria Bolotova, Vladislav Blinov, Yukun Zheng, W Bruce Croft, Falk Scholer, and Mark Sanderson. 2020. Do people and neural nets pay attention to the same words: Studying eye-tracking data for non-factoid qa evaluation. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 85-94.
Sridevi Bonthu, S Rama Sree, and MHM Krishna Prasad. 2021. Automated short answer grading using deep learning: A survey. In International Cross-Domain Conference for Machine Learning and Knowledge Extraction, pages 61-78. Springer.
Chris Brew and Claudia Leacock. 2013. Automated short answer scoring: Principles and prospects. In Handbook of Automated Essay Evaluation, pages 158-174. Routledge.
Steven Burrows, Iryna Gurevych, and Benno Stein. 2015. The eras and trends of automatic short answer grading. International Journal of Artificial Intelligence in Education, 25(1):60-117.
Chi-Fen Emily Chen and Wei-Yuan Eugene Cheng Cheng. 2008. Beyond the design of automated writing evaluation: Pedagogical practices and perceived learning effectiveness in efl writing classes. Language Learning & Technology, 12(2):94-112.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Samuel Dodson, Luanne Freund, and Rick Kopak. 2017. Do highlights affect comprehension? lessons from a user study. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval, pages 381-384.
Robert L Fowler and Anne S Barker. 1974. Effectiveness of highlighting for retention of text material. Journal of Applied Psychology, 59(3):358.
Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. *Ppdb: The paraphrase database*. In *Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 758-764.
Martin Gleize and Brigitte Grau. 2013. Limsiiles: Basic english substitution for student answer assessment at semeval 2013. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), volume 2, pages 598-602.
Michael Heilman and Nitin Madnani. 2015. The impact of training data on automated short answer scoring performance. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 81-85.
Derrick Higgins, Chris Brew, Michael Heilman, Ramon Ziai, Lei Chen, Aoife Cahill, Michael Flor, Nitin Madnani, Joel Tetreault, Daniel Blanchard, et al. 2014. Is getting the right answer just about choosing the right words? the role of syntactically-informed features in short answer scoring. arXiv preprint arXiv:1403.0801.
Björn Tyrefors Hinnerich, Erik Hoglin, Magnus Johannesson, et al. 2011. Ethnic discrimination in high school grading: Evidence from a field experiment. Stockholm School of Economics Working Paper Series in Economic and Finance, (733).
Tsunenori Ishioka and Masayuki Kameda. 2017. Overwritable automated japanese short-answer scoring and support system. In Proceedings of the International Conference on Web Intelligence, pages 50-56.
Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198-4205, Online. Association for Computational Linguistics.
Sarthak Jain and Byron C Wallace. 2019. Attention is not explanation. arXiv preprint arXiv:1902.10186.
Byung-Hak Kim, Ethan Vizitei, and Varun Ganapathi. 2018. Gritnet: Student performance prediction with deep learning.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. arXiv preprint arXiv:1606.04155.
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2015. Visualizing and understanding neural models in nlp. arXiv preprint arXiv:1506.01066.
Elizabeth Puzgles Lorch, Madeline A Klusewitz, et al. 1995. Effects of typographical cues on reading and recall of text. Contemporary educational psychology, 20(1):51-64.
Robert F Lorch. 1989. Text-signaling devices and their effects on reading and memory processes. Educational psychology review, 1(3):209-234.
Hitoshi Manabe and Masato Hagiwara. 2021. Expats: A toolkit for explainable automated text scoring. arXiv preprint arXiv:2104.03364.
Akash Kumar Mohankumar, Preksha Nema, Sharan Narasimhan, Mitesh M Khapra, Balaji Vasan Srinivasan, and Balaraman Ravindran. 2020. Towards transparent and explainable attention models. arXiv preprint arXiv:2004.14243.
Michael Mohler, Razvan Bunescu, and Rada Mihalcea. 2011. Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 752-762.
Jonathan Nau, Aluizio Haendchen Filho, and Guilherme Passero. 2017. Evaluating semantic analysis methods for short answer grading using linear regression. *Sciences*, 3(2):437-450.
Ellis B Page. 1966. The imminence of... grading essays by computer. The Phi Delta Kappan, 47(5):238-243.
Laura Pappano. 2012. The year of the mooc. *The New York Times*, 2(12):2012.
Tomáš Protivínský and Daniel Münich. 2018. Gender bias in teachers' grading: What is in the grade. Studies in Educational Evaluation, 59:141-149.
Chen Qu, Liu Yang, W Bruce Croft, Falk Scholer, and Yongfeng Zhang. 2019. Answer interaction in non-factoid question answering systems. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval, pages 249-253.
Lakshmi Ramachandran, Jian Cheng, and Peter Foltz. 2015. Identifying patterns for short answer scoring using graph-based lexico-semantic text matching. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 97-106.
Pedro Uria Rodriguez, Amir Jafari, and Christopher M Ormerod. 2019. Language models and automated essay scoring. arXiv preprint arXiv:1909.09482.
Archana Sahu and Plaban Kumar Bhowmick. 2020. Feature engineering and ensemble-based approach for improving automatic short-answer grading performance. IEEE Transactions on Learning Technologies, 13(1):77-90.
Sofia Serrano and Noah A Smith. 2019. Is attention interpretable? arXiv preprint arXiv:1906.03731.
Vicki L Silvers and David S Kreiner. 1997. The effects of pre-existing inappropriate highlighting on reading comprehension. Literacy Research and Instruction, 36(3):217-223.
Shaymaa E Sorour, Tsunenori Mine, Kazumasasa Goda, and Sachio Hirokawa. 2015. A predictive model to evaluate student performance. Journal of Information Processing, 23(2):192-201.
Md Arafat Sultan, Steven Bethard, and Tamara Sumner. 2014. Back to basics for monolingual alignment: Exploiting word similarity and contextual evidence. Transactions of the Association for Computational Linguistics, 2:219-230.
Md Arafat Sultan, Cristobal Salazar, and Tamara Sumner. 2016. Fast and easy short answer grading with high accuracy. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1070-1075.
Xiaofei Sun, Diyi Yang, Xiaoya Li, Tianwei Zhang, Yuxian Meng, Qiu Han, Guoyin Wang, Eduard Hovy, and Jiwei Li. 2021. Interpreting deep learning models in natural language processing: A review. arXiv preprint arXiv:2110.10470.
Zijun Sun, Chun Fan, Qinghong Han, Xiaofei Sun, Yuxian Meng, Fei Wu, and Jiwei Li. 2020. Self-explaining structures improve nlp models. arXiv preprint arXiv:2012.01786.
Chul Sung, Tejas Dhamecha, Swarnadeep Saha, Tengfei Ma, Vinay Reddy, and Rishi Arora. 2019a. Pre-training bert on domain resources for short answer grading. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6071-6075.
Chul Sung, Tejas Indulal Dhamecha, and Nirmal Mukhi. 2019b. Improving short answer grading using transformer-based pre-training. In International Conference on Artificial Intelligence in Education, pages 469-481. Springer.
K Surya, Ekansh Gayakwad, and MK Nallakaruppan. 2019. Deep learning for short answer scoring. Int. J. Recent Technol. Eng, 7(6):1712-1715.
Reyn Van Ewijk. 2011. Same work, lower grade? student ethnicity and teachers' subjective assessments. Economics of Education Review, 30(5):1045-1058.
Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016. Attention-based LSTM for aspect-level sentiment classification. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 606-615.