SlowGuess/NAACL_2022_OCR huggingface dataset:
An interesting question for analysis is "What if we score the traits as the primary task?" In order to do that, we changed our system to make scoring one of the trait as the primary task, and scoring the rest of the traits as well as the essay overall, as auxiliary tasks. The comparison of these results are shown in Figure 3 (in the MTL* column). We see that our MTL* system outperforms the STL-based system on scoring individual traits, although it would take a lot longer time to train (as it would be equivalent to running the MTL system between 4 to 6 times).
In order to know which trait is most important for each essay set, we run a series of ablation tests. For each essay set, we ablate one essay trait at a time before scoring the essay. Table 5 reports the results of the ablation test. The values in the table correspond to the drop in performance in scoring the essay holistically. We find that the Content is the most important essay trait for 3 of the essay sets. Prompt Adherence and Word Choice are the most important traits for 2 of the essay sets where they are scored.
7.5 Error Analysis
As we have seen, the MTL model generally helps over the STL model when it comes to holistic essay scoring, especially if there is no well-defined rule (Example: Holistic Score = Sum of trait scores) for scoring the essay holistically.
A possible scenario where STL could help over MTL is if the holistic score is a well-defined function of the trait scores AND the STL system can predict the trait scores with a good deal of accuracy. The essay sets corresponding to Prompts 7 & 8 are two such essay sets, where the overall score is a function of the individual trait scores. To verify this, we ran the experiments in a pipelined manner - first scoring the essay traits, then calculating the holistic score using the predicted trait scores and comparing it with the gold standard holistic scores. We found no difference in QWK for Prompt 7 (a QWK of 0.796 vs. 0.795), but a much lesser performance with Prompt 8 (a QWK of 0.684 vs. 0.699) as compared to our MTL-based system. One of the main reasons for this is due to the poor performance in predicting the trait scores as single tasks.
7.6 Runtime Analysis
We also ran experiments to see how much resources and time our approaches will take. Table 6 gives the total training time (in hours). The total training time is the total time taken to train our system to score the essay holistically as well as all the traits in that essay set for all 100 epochs. We also report the speed-up when using the MTL approach as compared to the STL approach. From our results, we observe a 2.30 to 3.70 speed-up in using the MTL models as compared to using the STL models. The BERT-STL experiments ran for about 5 days (113 hours).
We also report the average number of training
Essay Trait	Prompt 1	Prompt 2	Prompt 3	Prompt 4	Prompt 5	Prompt 6	Prompt 7	Prompt 8
Content	0.0148	0.0092	0.0064	0.0074	0.0030	0.0128	0.0102	0.0102
Organization	0.0122	0.0088	—	—	—	—	0.0090	0.0052
Word Choice	0.0080	0.0164	—	—	—	—	—	0.0264
Sentence Fluency	0.0086	0.0036	—	—	—	—	—	0.0196
Conventions	0.0090	0.0018	—	—	—	—	0.0076	0.0056
Prompt Adherence	—	—	0.0282	0.0112	0.0026	0.0044	—	—
Language	—	—	0.0080	0.0108	0.0088	0.0030	—	—
Narrativity	—	—	0.0092	0.0050	0.0124	0.0062	—	—
Style	—	—	—	—	—	—	0.0030	—
Voice	—	—	—	—	—	—	—	0.0094
Table 5: Results of the ablation tests. The numbers show the drop in performance when we ablate each of the essay traits from each of the essay sets (Prompt 1 to 8). The most important features in each essay set are written in boldface and underlined. Cells with a — in them mean that the essay trait was not present in that essay set.
System	STL Time	MTL Time	Speed-Up
LSTM	24.62 hours	10.45 hours	2.30
BiLSTM	40.98 hours	11.32 hours	3.70
Table 6: Total training time for each system for all prompts, traits and folds, using our neural network systems.
System	Average	Range
STL-LSTM	326K	326K
STL-BiLSTM	436K	436K
MTL-LSTM	891K	829K - 1.08M
MTL-BiLSTM	1.5M	1.38M - 1.85M
Table 7: Average and range of training parameters per essay set for each system.
parameters per system in Table 7. For the STL systems, the number of trainable parameters is the same irrespective of essay set. For the MTL systems, the models, the number of training parameters varies based on the number of essay traits in the essay set. Prompts 3 to 7, which have only 4 traits, have about 1.38 million training parameters. On the other hand, Prompt 8, which has 6 essay traits, has over 1.85 million training parameters.
All our experiments were run on an Nvidia GeForce GTX 1080 Ti Graphics Card with 12GB of GPU memory, using Python version 3.5.2, Keras version 2.2.4 and Tensorflow version  $1.14^{8}$ .
7.7 Comparison with Transformer Models
Most modern NLP systems have started to use attention-based transformer networks and large pretrained language models. Yang et al. (2020), Cao et al. (2020), and Uto et al. (2020) use the BERT-
base-uncased (Devlin et al., 2019) pre-trained language model to perofrm automatic essay grading achieving QWKs in the range of 0.79 to 0.805. However, BERT has about 110 million parameters (compared to our largest model with just under 2 million parameters). Another limiting factor with using BERT is the fact that we can only input 512 tokens. This is a problem, especially for Prompt 8, where the average essay length is about 650 words. Mayfield and Black (2020) describe some of the other limitations of using BERT for scoring essays.
In this paper, we described an approach to use multi-task learning to automatically score essays and their traits. We achieve this by concatenating a representation of the essay with the trait scores - predicted as an auxiliary task. We compared our results with single-task learning models as well. We found out that the MTL system with the Bi-Directional LSTM outperforms the STL-based systems and has results comparable with a baseline BERT-STL system. We then ran an ablation test and found out which essay trait was important for the corresponding essay sets. We also report our system's performance, which shows a 2.30 to 3.70 speed-up of using the multi-task learning system, compared to using a single task learning system.
An exciting avenue of future work is using trait scoring to aid in providing text feedback to the writer, like showing where the low score for the trait originates, similar to Hellman et al. (2020) (for content scoring), rather than a trait-specific score only. We also plan to investigate using ALBERT (Lan et al., 2020), in lieu of the essay stack, to grade essays and their traits simultaneously. We also plan to explore how to extend our approach
8We have uploaded the rest of the requirements along with the code
in generalizing our system, training it on essays written in response to one set of source prompts, and tested it on essays written for another prompt.
The authors would like to thank the anonymous reviewers of the ACL Rolling Review for their comments in helping improve this work.
The authors would also like to acknowledge the different research grants in the various host institutions, namely:
- The Young Faculty Research Fellowship (YFRF) Award supported by the Visveshvaraya Ph.D. Scheme for Electronics and IT, Ministry of Electronics and Information Technology (MeitY), Government of India, being implemented by Digital India Corporation (formerly Media Lab Asia).
- The Presidency University Faculty Seed Grant Award.
for funding us in carrying out our research.
Yue Cao, Hanqi Jin, Xiaojun Wan, and Zhiwei Yu. 2020. Domain-adaptive neural automated essay scoring. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1011-1020.
Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41-75.
Jieun Chae and Ani Nenkova. 2009. Predicting the fluency of text with shallow structural features: Case studies of machine translation and human-written text. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pages 139-147, Athens, Greece. Association for Computational Linguistics.
Jacob Cohen. 1968. Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493-2537.
Mădalina Cozma, Andrei Butnaru, and Radu Tudor Ionescu. 2018. Automated essay scoring with string kernels and word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 503-509, Melbourne, Australia. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Fei Dong and Yue Zhang. 2016. Automatic features for essay scoring – an empirical study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1072–1077, Austin, Texas. Association for Computational Linguistics.
Fei Dong, Yue Zhang, and Jie Yang. 2017. Attention-based recurrent convolutional neural network for automatic essay scoring. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 153-162, Vancouver, Canada. Association for Computational Linguistics.
Scott Hellman, William Murray, Adam Wiemerslage, Mark Rosenstein, Peter Foltz, Lee Becker, and Marcia Derr. 2020. Multiple instance learning for content feedback localization without annotation. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 30-40, Seattle, WA, USA  $\rightarrow$  Online. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. *Neural computation*, 9(8):1735-1780.
Zixuan Ke, Hrishikesh Inamdar, Hui Lin, and Vincent Ng. 2019. Give me more feedback II: Annotating thesis strength and related attributes in student essays. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3994-4004, Florence, Italy. Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations.
Sandeep Mathias and Pushpak Bhattacharyya. 2018a. ASAP++: Enriching the ASAP automated essay grading dataset with essay attribute scores. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Sandeep Mathias and Pushpak Bhattacharyya. 2018b. Thank "goodness"! a way to measure style in student essays. In Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications, pages 35-41, Melbourne, Australia. Association for Computational Linguistics.
Sandeep Mathias and Pushpak Bhattacharyya. 2020. Can neural networks automatically score essay traits? In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 85-91, Seattle, WA, USA (Online). Association for Computational Linguistics.
Sandeep Mathias, Diptesh Kanojia, Kevin Patel, Samarth Agrawal, Abhijit Mishra, and Pushpak Bhattacharyya. 2018. Eyes are the windows to the soul: Predicting the rating of text quality using gaze behaviour. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2352-2362, Melbourne, Australia. Association for Computational Linguistics.
Sandeep Mathias, Rudra Murthy, Diptesh Kanojia, Abhijit Mishra, and Pushpak Bhattacharyya. 2020. Happy are those who grade without seeing: A multitask learning approach to grade essays using gaze behaviour. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 858-872, Suzhou, China. Association for Computational Linguistics.
Elijah Mayfield and Alan W Black. 2020. Should you fine-tune BERT for automated essay scoring? In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 151-162, Seattle, WA, USA (Online). Association for Computational Linguistics.
Ellis B Page. 1966. The imminence of... grading essays by computer. The Phi Delta Kappan, 47(5):238-243.
Isaac Persing, Alan Davis, and Vincent Ng. 2010. Modeling organization in student essays. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 229-239, Cambridge, MA. Association for Computational Linguistics.
Isaac Persing and Vincent Ng. 2013. Modeling thesis clarity in student essays. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 260-269, Sofia, Bulgaria. Association for Computational Linguistics.
Isaac Persing and Vincent Ng. 2014. Modeling prompt adherence in student essays. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1534-1543, Baltimore, Maryland. Association for Computational Linguistics.
Isaac Persing and Vincent Ng. 2015. Modeling argument strength in student essays. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 543-552, Beijing, China. Association for Computational Linguistics.
Isaac Persing and Vincent Ng. 2016. Modeling stance in student essays. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2174-2184, Berlin, Germany. Association for Computational Linguistics.
Peter Phandi, Kian Ming A. Chai, and Hwee Tou Ng. 2015. Flexible domain adaptation for automated essay scoring using correlated linear regression. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 431-439, Lisbon, Portugal. Association for Computational Linguistics.
Taraka Rama and Sowmya Vajjala. 2021. Are pretrained text representations useful for multilingual and multi-dimensional language proficiency modeling?
Robert Ridley, Liang He, Xin-yu Dai, Shujian Huang, and Jiajun Chen. 2021. Automated cross-prompt scoring of essay traits. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13745-13753.
Mark D Shermis and Jill Burstein. 2013. Handbook of automated essay evaluation: Current applications and new directions. Routledge.
Swapna Somasundaran, Jill Burstein, and Martin Chodorow. 2014. Lexical chaining for measuring discourse coherence quality in test-taker essays. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 950-961, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
Swapna Somasundaran, Michael Flor, Martin Chodorow, Hillary Molloy, Binod Gyawali, and Laura McCulla. 2018. Towards evaluating narrative quality in student writing. Transactions of the Association for Computational Linguistics, 6:91-106.
Wei Song, Ziyao Song, Lizhen Liu, and Ruiji Fu. 2020. Hierarchical multi-task learning for organization evaluation of argumentative student essays. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3875-3881.
Kaveh Taghipour. 2017. Robust trait-specific essay scoring using neural networks and density estimators. Ph.D. thesis, National University of Singapore (Singapore).
Kaveh Taghipour and Hwee Tou Ng. 2016. A neural approach to automated essay scoring. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1882-1891, Austin, Texas. Association for Computational Linguistics.
Yi Tay, Minh Phan, Luu Anh Tuan, and Siu Cheung Hui. 2018. Skipflow: Incorporating neural coherence features for end-to-end automatic text scoring. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), pages 5948-5955.
Masaki Uto, Yikuan Xie, and Maomi Ueno. 2020. Neural automated essay scoring incorporating handcrafted features. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6077-6088, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Ruosong Yang, Jiannong Cao, Zhiyuan Wen, Youzheng Wu, and Xiaodong He. 2020. Enhancing automated essay scoring performance via fine-tuning pre-trained language models with combination of regression and ranking. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1560-1569, Online. Association for Computational Linguistics.
Torsten Zesch, Michael Wojatzki, and Dirk Scholten-Akoun. 2015. Task-independent features for automated essay grading. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 224-232, Denver, Colorado. Association for Computational Linguistics.
Haoran Zhang and Diane Litman. 2018. Co-attention based neural network for source-dependent essay scoring. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 399–409, New Orleans, Louisiana. Association for Computational Linguistics.
Haoran Zhang and Diane Litman. 2020. Automated topical component extraction using neural network attention scores from source-based essay scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8569-8584, Online. Association for Computational Linguistics.