SlowGuess/NAACL_2022_OCR huggingface dataset:
For the proposed model architecture which is depicted in Figure 1, the BERT model in the left part are shared by the document-scale and token-scale essay representations, and the other BERT model in the right part are shared by all segment-scale essay representations. We use the "bert-base-uncased" which includes 12 transformer layers and the hidden size is 768. In the training stage, we freeze all the layers in the BERT models except the last layer, which is more task related than other layers. The Longformer model used in our work is "longformer-base-4096". For the MR loss, we set  $b$  to 0. The weights  $\alpha$ ,  $\beta$  and  $\gamma$  are tuned according to the performance on develop set. We use Adam optimizer (Kingma and Ba, 2015) to fine-tune model parameters in an end-to-end fashion with learning rate of 6e-5,  $\beta 1 = 0.9$ ,  $\beta 2 = 0.999$ ,  $L2$  weight decay of 0.005. The coefficient weight  $\alpha$  in R-Drop is 9. We set the batch size to 32. We use dropout in the training stage and the drop rate is set to 0.1. We train all the models for 80 epochs, and select the best model according the performance on the develop set. We use a greedy search method to find the best combination of segment scales, which is shown in detail in Appendix A. Following (Cao et al., 2020), we perform the significance test for our models.
For each prompt, we use all the essays from other prompts in ASAP data set.
ID	Models	P1	P2	P3	P4	P5	P6	P7	P8	Average
1	EASE(SVR) (Phandi et al., 2015)	0.781	0.621	0.630	0.749	0.782	0.771	0.727	0.534	0.699
2	EASE(BLRR) (Phandi et al., 2015)	0.761	0.606	0.621	0.742	0.784	0.775	0.730	0.617	0.705
3	CNN(10 runs) + LSTM(10 runs) (Taghipour and Ng, 2016)	0.821	0.688	0.694	0.805	0.807	0.819	0.808	0.644	0.761
4	Hierarchical LSTM-CNN-Attention (Dong et al., 2017)	0.822	0.682	0.672	0.814*	0.803	0.811	0.801	0.705	0.764
5	SKIPFLOW LSTM(Bilinear) (Tay et al., 2018)	0.830	0.678	0.677	0.778	0.795	0.807	0.790	0.670	0.753
6	SKIPFLOW LSTM(Tensor) (Tay et al., 2018)	0.832	0.684	0.695	0.788	0.815	0.810	0.800	0.697	0.764
7	Dilated LSTM With RL (Wang et al., 2018)	0.776	0.659	0.688	0.778	0.805	0.791	0.760	0.545	0.724
8	HA-LSTM+SST+DAT (Cao et al., 2020)	0.836	0.730	0.732	0.822	0.835	0.832*	0.821	0.718	0.790
9	BERT+SST+DAT (Cao et al., 2020)	0.824	0.699	0.726	0.859	0.822	0.828	0.840	0.726	0.791*
10	R²BERT (Yang et al., 2020)	0.817	0.719	0.698	0.845	0.841	0.847	0.839	0.744	0.794*
11	BERT-DOC-TOK-SEG	0.836	0.695	0.700	0.815	0.812	0.816	0.838	0.744	0.782
12	Tran-BERT-MS-ML-R	0.834	0.716	0.714	0.812	0.813	0.836	0.839	0.766	0.791*
Table 2: Experiment results of all models in terms of QWK on ASAP. The name of our implemented models are in bold. The bold number is the best performance for each prompt. The best 3 average QWK are annotated with * .
ID	Models	P1	P2	P8	Average
8	HA-LSTM+SST+DAT	0.836	0.730	0.718	0.761
9	HA-BERT+SST+DAT	0.824	0.699	0.726	0.750
10	R2BERT	0.817	0.719	0.744	0.760
12	Tran-BERT-MS-ML-R	0.834	0.716	0.766	0.772
Table 2 shows the performance of baseline models and our proposed models with joint learning of multi-scale essay representation. Table 3 shows the results of our model and the state-of-the-art models on essays in prompt 1, 2 and 8, whose WordPiece length are longer than 510. We summarize some findings from the experiment results.
- Our model 12 almost obtains the published state-of-the-art for neural approaches. For the prompts 1,2 and 8, whose WordPiece length are longer than 510, we improve the result from 0.761 to 0.772. As Longformer is good at encoding long text, we also use it to encode essays of prompt 1, 2 and 8 directly but the performance is poor compared to the methods in Table 3. The results demonstrate the effectiveness of the proposed framework for encoding and scoring essays. We further re-implement BERT $^2$  proposed by (Yang et al., 2020), and our implementation of BERT $^2$  is not as well-performing as the published result. Though (Uto et al., 2020) obtain a much better result(QWK 0.801), our method performs much better than their system with only neural features(QWK 0.730), which demonstrates the strong essay encoding ability of our neural approach.
- Compared to the models 4 and 6, our model 11 uses multi-scale features to encode essays instead of LSTM based models, and we use the same regression loss to optimize the model. Our model simply changes the representation way and significantly improves the result from 0.764 to 0.782, which demonstrates the strong encoding ability armed by multi-scale representation for long text. Before that, the conventional way of using BERT can not surpass the performance of models 4 and 6.
4.5 Further analysis
Multi-scale Representation We further analyze the effectiveness of employing each scale essay representation to the joint learning process.
Table 3: Experiment results of our model and the state-of-the-art models on ASAP long essays (WordPiece length are longer than 510). The name of our implemented model is in bold.
Models	Average QWK
BERT-DOC	0.760
BERT-TOK	0.764
BERT-DOC-TOK	0.768
BERT-DOC-TOK-SEG	0.782
Table 4: Performance of different feature scale models on ASAP data set.
Models	RMSE
BERT-DOC	0.742
BERT-TOK	0.760
BERT-DOC-TOK	0.691
BERT-DOC-TOK-SEG	0.607
Table 5: Performance of different feature scale models on CRP data set. The evaluation metric is RMSE. Lower numbers are better.
Table 4 and Table 5 show the performance of our models to represent essays on different feature scales, which are trained with MSE loss and without transfer learning. Table 4 shows the performance on ASAP data set while Table 5 shows the performance on CRP data set. The improvement of
BERT-DOC-TOK-SEG over BERT-DOC, BERT-TOK, BERT-DOC-TOK are significant  $(p < 0.0001)$  on CRP data set, and are significant  $(p < 0.0001)$  in most cases on ASAP data set. Results on both table indicate the similar findings.
- Combining the features from document-scale and token-scale, BERT-DOC-TOK outperforms the models BERT-DOC and BERT-TOK, which only use one scale features. This demonstrates that our proposed framework can benefit from multi-scale essay representation even with only two scales.
- By additionally incorporating multiple segment-scale features, BERT-DOC-TOK-SEG performs much better than BERT-DOC-TOK. This demonstrates the effectiveness and generalization ability of our multi-scale essay representation on multiple tasks.
Models	Average QWK
Longformer-DOC	0.746
Longformer-DOC-TOK-SEG	0.771
Reasons for Effectiveness of Multi-scale Representation Though the experiment shows the effectiveness of multi-scale representation, we further explore the reason. We could doubt that the effectiveness comes from supporting long sequences, not the multi-scale itself. As Longformer is good at dealing with long texts, we compare the results between Longformer-DOC and Longformer-DOC-TOK-SEG. The results of the significance test show that the improvement of Longformer-DOC-TOK-SEG over Longformer-DOC are significant  $(p < 0.0001)$  in most cases. Performance of the two models are shown in Table 6, and we get the following findings.
- Though Longformer-DOC supports long sequences encoding, it performs poor, which indicates us that supporting long sequence ability is not enough for a good essay scoring system.
- Longformer-DOC-TOK-SEG outperforms Longformer-DOC significantly, which indicates the effectiveness of our model comes from encoding essays by multi-scale
features, not only comes from the ability to deal with long texts.
These results are consistent with our intuition that our approach takes into account different level features of essays and predict the scores more accurately. We consider it caused by that multi-scale features are not effectively constructed in the representation layer of pre-trained model due to the lack of data for fine-tuning in the AES task. Therefore, we need to explicitly model the multi-scale information of the essay data and combine it with the powerful linguistic knowledge of pre-trained model.
Table 6: Performance of multi-scale Longformer models on ASAP data set.
Models	Average
BERT-DOC-TOK-SEG	0.782
Tran-BERT-MS	0.788
Tran-BERT-MS-ML	0.790
Tran-BERT-MS-ML-R	0.791
Table 7: Experiment results for transfer learning with multiple loss functions and R-Drop.
Transfer Learning with Multiple Losses and R-Drop We further explore the effectiveness of pretraining with adding multiple loss functions and employing R-Drop. As is shown in table 7, by incorporating the pre-training stage which learns the knowledge from out-of-domain data, Tran-BERT-MS model improves the result from 0.782 to 0.788 compared to BERT-DOC-TOK-SEG model. The model Tran-BERT-MS-ML which jointly learns with multiple loss functions further improves the performance from 0.788 to 0.790. We consider it due to the reason that MR brings ranking information and SIM takes into account the overall score distribution information. Diverse losses bring different but positive influence on the optimization direction and act as an enabler. By employing R-Drop, Tran-BERT-MS-ML-R improves the QWK slightly, which comes from the fact that R-Drop plays a regularization role.
In this paper, we propose a novel multi-scale essay representation approach based on pre-trained language model, and employ multiple losses and transfer learning for AES task. We almost obtain the state-of-the-art result among deep learning models. In addition, we show multi-scale representation has a significant advantage when dealing with long texts.
One of the future directions could be exploring soft multi-scale representation. Introducing linguistic knowledge to segment at a more reasonable scale may bring further improvement.
Dimitrios Alikaniotis, Helen Yannakoudakis, and Marek Rei. 2016. Automatic text scoring using neural networks. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 715-725.
Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater® v.2. The Journal of Technology, Learning, and Assessment, 4(3).
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. In arXiv: Computation and Language.
Yue Cao, Hanqi Jin, Xiaojun Wan, and Zhiwei Yu. 2020. Domain-adaptive neural automated essay scoring. In SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information, pages 1011-1020.
Hongbo Chen and Ben He. 2013. Automated essay scoring by maximizing human-machine agreement. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1741-1752.
J Cohen. 1968. Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213-220.
Mădalina Cozma, Andrei M. Butnaru, and Radu Tudor Ionescu. 2018. Automated essay scoring with string kernels and word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
Tirthankar Dasgupta, Abir Naskar, Lipika Dey, and Rupsa Saha. 2018. Augmenting textual qualitative features in deep convolution recurrent neural network for automatic essay scoring. In Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications, pages 93-102.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186.
Fei Dong and Yue Zhang. 2016. Automatic features for essay scoring – an empirical study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1072–1077.
Fei Dong, Yue Zhang, and Jie Yang. 2017. Attention-based recurrent convolutional neural network for automatic essay scoring. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 153-162.
Youmna Farag, Helen Yannakoudakis, and Ted Briscoe. 2018. Neural automated essay scoring and coherence modeling for adversarially crafted input. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technology, pages 263-271.
Cancan Jin, Ben He, Kai Hui, and Le Sun. 2018. Tdnn: A two-stage deep neural network for prompt-independent automated essay scoring. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 1088-1097.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference for Learning Representations.
Leah S. Larkey. 1998. Automatic essay grading using text categorization techniques. In SIGIR '98 Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, pages 90-95.
Xiaobo Liang, Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, and TieYan Liu. 2021. R-drop: Regularized dropout for neural networks. In Advances in Neural Information Processing Systems, pages 10890-10905.
Sandeep Mathias, Rudra Murthy, Diptesh Kanojia, Abhijit Mishra, and Pushpak Bhattacharyya. 2020. Happy are those who grade without seeing: A multitask learning approach to grade essays using gaze behaviour. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 858-872.
Elijah Mayfield and Alan W Black. 2020. Should you fine-tune bert for automated essay scoring? In Proceedings of the 15th Workshop on Innovative Use of NLP for Building Educational Applications, pages 151-162.
Panitan Muangkammuen and Fumiyo Fukumoto. 2020. Multi-task learning for automated essay scoring with sentiment analysis. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 116-123.
Andriy Mulyar, Elliot Schumacher, Masoud Rouhizadeh, and Mark Dredze. 2019. Phenotyping of clinical notes with improved document classification models using contextualized neural
language models. In 33rd Conference on Neural Information Processing Systems (NeurIPS).
Peter Phandi, Kian Ming A. Chai, and Hwee Tou Ng. 2015. Flexible domain adaptation for automated essay scoring using correlated linear regression. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 431-439.
Robert Ridley, Liang He, Xinyu Dai, Shujian Huang, and Jiajun Chen. 2021. Automated cross-prompt scoring of essay traits. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 13745-13753.
Pedro Uria Rodriguez, Amir Jafari, and Christopher M. Ormerod. 2019. Language models and automated essay scoring. In arXiv: Computation and Language.
Lawrence M. Rudner and Tahung Liang. 2002. Automated essay scoring using bayes' theorem. The Journal of Technology, Learning, and Assessment, 1(2):3-21.
Wei Song, Kai Zhang, Ruiji Fu, Lizhen Liu, Ting Liu, and Miaomiao Cheng. 2020. Multi-stage pre-training for automated chinese essay scoring. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6723-6733.
Kaveh Taghipour and Hwee Tou Ng. 2016. A neural approach to automated essay scoring. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1882-1891.
Yi Tay, Minh C. Phan, Luu Anh Tuan, and Siu Cheung Hui. 2018. Skipflow: incorporating neural coherence features for end-to-end automatic text scoring. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 5948-5955.
Masaki Uto, Yikuan Xie, and Maomi Ueno. 2020. Neural automated essay scoring incorporating handcrafted features. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6077-6088.
Yucheng Wang, Zhongyu Wei, Yaqian Zhou, and Xuanjing Huang. 2018. Automatic essay scoring incorporating rating schema via reinforcement learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 791-797.
Ruosong Yang, Jiannong Cao, Zhiyuan Wen, Youzheng Wu, and Xiaodong He. 2020. Enhancing automated essay scoring performance via fine-tuning pre-trained language models with combination of regression and ranking. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1560-1569.
Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading esol texts. In HLT '11 Proceedings of the
49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 180-189.
All the segment-scales we explore range from 10 to 190. The interval between two neighbor scales is 20. As the combination number of all segment-scales is exponential, we use a greedy search method to find the best combination.
1. Initialize the segment-scale value set  $R$  as the document-scale and token-scale.
2. Experiment the combination of each segment-scale with the token-scale and document-scale essay representation, and compute the average QWK on develop set for all segment-scales, which is denoted as  $QWK_{ave}$ . The scale with higher QWK compared to  $QWK_{ave}$  is added to the candidate scale list  $L$  and the scales in  $L$  are sorted according to their QWK values from large to small.
3. For each  $i$  from 1 to  $|L|$ , we perform experiments on the combination of the first  $i$  segment-scales in  $L$  with the token-scale and document-scale. The combination segment-scales with the best performance on develop set are added to the segment-scale value set  $R$