SlowGuess/NAACL_2022_OCR huggingface dataset:
Yongjie Wang<sup>1</sup> Chuan Wang<sup>1</sup> Ruobing Li<sup>1</sup> Hui Lin<sup>1,2</sup>
$^{1}$ LAIX Inc.
{yongjie.wang, chuan.wang, ruobing.li, hui.lin}@liulishuo.com
In recent years, pre-trained models have become dominant in most natural language processing (NLP) tasks. However, in the area of Automated Essay Scoring (AES), pre-trained models such as BERT have not been properly used to outperform other deep learning models such as LSTM. In this paper, we introduce a novel multi-scale essay representation for BERT that can be jointly learned. We also employ multiple losses and transfer learning from out-of-domain essays to further improve the performance. Experiment results show that our approach derives much benefit from joint learning of multi-scale essay representation and obtains almost the state-of-the-art result among all deep learning models in the  $\mathrm{ASAP}^1$  task. Our multi-scale essay representation also generalizes well to CommonLit Readability Prize  $(\mathrm{CRP}^2)$  data set, which suggests that the novel text representation proposed in this paper may be a new and effective choice for long-text tasks.
AES is a valuable task, which can promote the development of automated assessment and help teachers reduce the heavy burden of assessment. With the rise of online education in recent years, more and more researchers begin to pay attention to this field.
AES systems typically consist of two modules, which are essay representation and essay scoring modules. The essay representation module extracts features to represent an essay and the essay scoring module rates the essay with the extracted features.
When a teacher rates an essay, the scores are often affected by multiple signals from different granularity levels, such as token level, sentence level, paragraph level and etc. For example, the
features may include the numbers of words, the essay structure, the master degree of vocabulary and syntactic complexity, etc. These features come from different scales of the essay. This inspires us to extract multi-scale features from the essays which represent multi-level characteristics of the essays.
Most of the deep neural networks AES systems use LSTM or CNN. Some researchers (Uto et al., 2020; Rodriguez et al., 2019; Mayfield and Black, 2020) attempt to use BERT (Devlin et al., 2019) in their AES systems but fail to outperform other deep neural networks methods (Dong et al., 2017; Tay et al., 2018). We believe previous approaches using BERT for AES suffer from at least three limitations. First, the pre-trained models are usually trained on sentence-level, but fail to learn enough knowledge of essays. Second, the AES training data is usually quite limited for direct fine-tuning of the pre-trained models in order to learn better representation of essays. Last but not least, mean squared error (MSE) is commonly used in the AES task as the loss function. However, the distribution of the sample population and the sorting properties between samples are also important issues to be considered when designing the loss functions as they imitate the psychological process of teachers rating essays. Different optimizations can also bring diversity to the final overall score distribution and contribute to the effectiveness of ensemble learning.
To address the aforementioned issues and limitations, we introduce joint learning of multi-scale essay representation into the AES task with BERT, which outperforms the state-of-the-art deep learning models based on LSTM (Dong et al., 2017; Tay et al., 2018). We propose to explicitly model more effective representations by extracting multiscale features as well as leveraging the knowledge learned from numerous sentence data. As the training data is limited, we also employ transfer learn
1https://www.kaggle.com/c/asap-aes
$^{2}$ https://www.kaggle.com/c/
commonlitreadabilityprize/data
Human Language Technologies, pages 3416 - 3425
ing from out-of-domain essays which is inspired by (Song et al., 2020). To introduce the diversity of essay scoring distribution, we combine two other loss functions with MSE. When training our model with multiple losses and transfer learning using R-Drop (Liang et al., 2021), we almost achieve the state-of-the-art result among all deep learning models. The source code of prediction module with a trained model for ASAP's prompt 8 is publicly available<sup>3</sup>.
In summary, the contribution of this work is as follows:
- We propose a novel essay scoring approach to jointly learn multi-scale essay representation with BERT, which significantly improve the result compared to traditionally using pretrained language models.
- Our method shows significant advantages in long text tasks and obtains almost the state-of-the-art result among all deep learning models in the ASAP task.
- We introduce two new loss functions which are inspired by the mental process of teacher rating essays, and employ transfer learning from out-of-domain essays with R-Drop (Liang et al., 2021), which further improves the performance for rating essays.
The dominant approaches in AES can be grouped into three categories: traditional AES, deep neural networks AES and pre-training AES.
- Traditional AES usually uses regression or ranking systems with complicated handcrafted features to rate an essay (Larkey, 1998; Rudner and Liang, 2002; Attali and Burstein, 2006; Yannakoudakis et al., 2011; Chen and He, 2013; Phandi et al., 2015; Cozma et al., 2018). These handcrafted features are based on the prior knowledge of linguists. Therefore they can achieve good performance even with small amounts of data.
- Deep Neural Networks AES has made great progress and achieved comparable results with traditional AES recently (Taghipour and Ng, 2016; Dong and Zhang, 2016; Dong et al.,
2017; Alikaniotis et al., 2016; Wang et al., 2018; Tay et al., 2018; Farag et al., 2018; Song et al., 2020; Ridley et al., 2021; Muangkammuen and Fukumoto, 2020; Mathias et al., 2020). While the handcrafted features are complicated to implement and careful manual design makes these features less portable, deep neural networks such as LSTM or CNN can automatically discover and learn complex features of essays, which makes AES an end-to-end task. Saving much time to design features, deep neural networks can transfer well among different AES tasks. By combining traditional and deep neural network approaches, AES can even obtain a better result, which benefits from both representations (Jin et al., 2018; Dasgupta et al., 2018; Uto et al., 2020). However, ensemble way still needs handcrafted features which cost numerous energy of researchers.
- Pre-training AES uses the pre-trained language model as the initial essay representation module and fine-tune the model on the essay training set. Though the pre-trained methods have achieved the state-of-the-art performance in most NLP tasks, most of them (Uto et al., 2020; Rodriguez et al., 2019; Mayfield and Black, 2020) fail to show an advantage over other deep learning methods (Dong et al., 2017; Tay et al., 2018) in AES task. As far as we know, the work from Cao et al. (2020) and Yang et al. (2020) are the only two pretraining approaches which surpass the other deep learning methods. Their improvement mainly comes from the training optimization. Cao et al. (2020) employ two self-supervised tasks and domain adversarial training, while Yang et al. (2020) combine regression and ranking to train their model.
The AES task is defined as following:
Given an essay with  $n$  words  $X = \{x_{i}\}_{i=1}^{n}$ , we need to output one score  $y$  as a result of measuring the level of this essay.
Quadratic weighted Kappa (QWK) (Cohen, 1968) metric is commonly used to evaluate AES systems by researchers, which measures the agreement between the scoring results of two raters.
3https://github.com/lingochamp/Multi-Scale-BERT-AES
3.2 Multi-scale Essay Representation
We obtain the multi-scale essay representation from three scales: token-scale, segment-scale and document-scale.
Token-scale and Document-scale Input We apply one pre-trained BERT (Devlin et al., 2019) model for token-scale and document-scale essay representations. The BERT tokenizer is used to split the essay into a token sequence  $T_{1} = [t_{1}, t_{2}, \dots, t_{n}]$ , where  $t_{i}$  is the  $i$ th token and  $n$  is the number of the tokens in the essay. The token we mentioned in this paper all refer to WordPiece, which is obtained by the subword tokenization algorithm used for BERT. We construct a new sequence  $T_{2}$  from  $T_{1}$  as following.  $L$  is set to 510, which is the max sequence length supported by BERT except the token [CLS] and [SEP].
$$
T _ {2} = \left\{ \begin{array}{l l} {[ \mathrm {C L S} ] + [ t _ {1}, t _ {2},.., t _ {L} ] + [ \mathrm {S E P} ]} & n > L \\ {[ \mathrm {C L S} ] + T _ {1} + [ \mathrm {S E P} ]} & n = L \\ {[ \mathrm {C L S} ] + T _ {1} + [ \mathrm {P A D} ] * (L - n) + [ \mathrm {S E P} ]} & n <   L \end{array} \right.
$$
The final input representation are the sum of the token embeddings, the segmentation embeddings and the position embeddings. A detailed description can be found in the work of BERT (Devlin et al., 2019).
Document-scale The document-scale representation is obtained by the  $[CLS]$  output of the BERT model. As the  $[CLS]$  output aggregates the whole sequence representation, it attempts to extract the essay information from the most global granularity.
Token-scale As the BERT model is pre-trained by Masked Language Modeling (Devlin et al., 2019), the sequence outputs can capture the context information to represent each token. An essay often consists of hundreds of tokens, thus RNN is not the proper choice to combine all the token information due to the gradients vanishing problem. Instead, we utilize a max-pooling operation to all the sequence outputs and obtain the combined token-scale essay representation. Specifically, the max-pooling layer generates a  $d$ -dimensional vector  $W = [w_{1}, w_{2}, \dots, w_{j}, \dots, w_{d}]$  and the element  $w_{j}$  is computed as below:
$$
w _ {j} = m a x \{h _ {1, j}, h _ {2, j}, \dots , h _ {n, j} \}
$$
where  $d$  is the hidden size of the BERT model. As we use the pre-trained BERT model bert-base-uncased, the hidden size  $d$  is 768. All the  $n$  sequence outputs of the BERT model are annotated as  $[h_1,h_2,\dots ,h_i,\dots ,h_n]$ , where  $h_i$  is a  $d$ -dimensional
vector  $[h_{i,1}, h_{i,2}, \ldots, h_{i,d}]$  representing the  $i$ th sequence output, and  $h_{i,j}$  is the  $j$ th element in  $h_i$ .
Segment-scale Assuming the segment-scale value set is  $K = [k_{1}, k_{2}, \dots, k_{i}, \dots, k_{S}]$ , where  $S$  is the number of segment scales we want to explore, and  $k_{i}$  is the  $i$ th segment-scale in  $K$ . Given a token sequence  $T_{1} = [t_{1}, t_{2}, \dots, t_{n}]$  for an essay, we obtain the segment-scale essay representation corresponding to scale  $k_{i}$  as follows:
1. We define  $n_p$  as the maximum number of tokens corresponding to each essay prompt  $p$ . We truncate the token sequence to  $n_p$  tokens if the essay length is longer than  $n_p$ , otherwise we pad [PAD] to the sequence to reach the length  $n_p$ .
2. Divide the token sequence into  $m = \lceil n_p / k_i\rceil$  segments and each segment is of length  $k_{i}$  except for the last segment, which is similar to the work of (Mulyar et al., 2019).
3. Input each of the  $m$  segment tokens into the BERT model, and get  $m$  segment representation vectors from the  $[CLS]$  output.
4. Use an LSTM model to process the sequence of  $m$  segment representations, followed by attention pooling operation on the hidden states of the LSTM output to obtain the segment-scale essay representation corresponding to scale  $k_{i}$ .
The LSTM cell units process the sequence of segment representations and generate the hidden states as follows:
$$
\begin{array}{l} i _ {t} = \sigma \left(Q _ {i} \cdot s _ {t} + U _ {i} \cdot h _ {t - 1} + b _ {i}\right) \\ f _ {t} = \sigma \left(Q _ {f} \cdot s _ {t} + U _ {f} \cdot h _ {t - 1} + b _ {f}\right) \\ \hat {c} _ {t} = \tanh  (Q _ {c} \cdot s _ {t} + U _ {c} \cdot h _ {t - 1} + b _ {c}) \\ c _ {t} = i _ {t} \circ \hat {c} _ {t} + f _ {t} \circ c _ {t - 1} \\ o _ {t} = \sigma \left(Q _ {o} \cdot s _ {t} + U _ {o} \cdot h _ {t - 1} + b _ {o}\right) \\ h _ {t} = o _ {t} \circ \tanh  \left(c _ {t}\right) \\ \end{array}
$$
where  $s_t$  is the  $t$ th segment representation from BERT [CLS] output and  $h_t$  is the  $t$ th hidden state generated from LSTM.  $Q_i, Q_f, Q_c, Q_o, U_i, U_f, U_c$  and  $U_o$  are weight matrices, and  $b_i, b_f, b_c,$  and  $b_o$  are bias vectors.
The attention pooling operation we use is similar to the work of (Dong et al., 2017), which is defined as follows:
$$
\begin{array}{l} \hat {\alpha} _ {t} = \tanh  \left(Q _ {a} \cdot h _ {t} + b _ {a}\right) \\ \alpha_ {t} = \frac {e ^ {q _ {a} \cdot \hat {\alpha} _ {t}}}{\sum_ {j} e ^ {q _ {a} \cdot \hat {\alpha} _ {j}}} \\ o = \sum_ {t} \alpha_ {t} \cdot h _ {t} \\ \end{array}
$$
4https://huggingface.co/ bert-base-uncased
$o$  is the segment-scale essay representation corresponding to the scale  $k_{i}$ .  $\alpha_{t}$  is the attention weight for hidden state  $h_t$ .  $Q_{a}, b_{a}, q_{a}$  are the weight matrix, bias and weight vector respectively.
The model architecture is depicted in Figure 1.
We apply one BERT model to obtain the document-scale and token-scale essay representation. The concatenation of them is input into a dense regression layer which predicts the score corresponding to the document-scale and token-scale. For each segment-scale  $k$  with number of segments  $m$ , we apply another BERT model to get  $m$  CLS outputs, and apply an LSTM model followed by an attention layer to get the segment-scale representation. We input the segment-scale representation into another dense regression layer to get the score corresponding to segment-scale  $k$ . The final score is obtained by adding the scores of all  $S$  segment-scales and the score of the document-scale and token-scale, which is illustrated as below:
$$
\begin{array}{l} y = \sum_ {k} y _ {k} + y _ {d o c, t o k} \\ y _ {k} = \tilde {W} _ {s e g} \cdot o _ {k} + b _ {s e g} \\ y _ {d o c, t o k} = \tilde {W} _ {d o c, t o k} \cdot H _ {d o c, t o k} + b _ {d o c, t o k} \\ H _ {d o c, t o k} = w _ {d o c} \bigoplus W \\ \end{array}
$$
$y_{k}$  is the predicted score corresponding to segment-scale  $k$ .  $y_{doc, tok}$  is the predicted score corresponding to the document-scale and token-scale.  $\hat{W}_{seg}$  and  $b_{seg}$  are weight matrix and bias for segment-scale respectively.  $W_{doc, tok}$  and  $b_{doc, tok}$  are weight matrix and bias for document and token-scales,  $o_{k}$  is the segment-scale essay representation with the scale  $k$ .  $w_{doc}$  is the document-scale essay representation.  $W$  is the token-scale essay representation.  $H_{doc, tok}$  is the concatenation of document-scale and token-scale essay representations.
3.4 Loss Function
We use three loss functions to train the model.
MSE measures the average value of square errors between predicted scores and labels, which is defined as below:
$$
M S E (y, \hat {y}) = \frac {1}{N} \sum_ {i} (y _ {i} - \hat {y} _ {i}) ^ {2}
$$
where  $y_{i}$  and  $\hat{y}_i$  are the predicted score and the label for the  $i$ th essay respectively,  $N$  is the number of the essays.
Similarity (SIM) measures whether two vectors are similar or dissimilar by using cosine function.
A teacher takes into account the overall level distribution of all the students when rating an essay. Following such intuition, we introduce the SIM loss to the AES task. In each training step, we take the predicted scores of the essays in the batch as the predicted vector  $y$ , and the labels as the label vector  $\hat{y}$ . The SIM loss awards the similar vector pairs to make the model think more about the correlation among the batch of essays. The SIM loss is defined as below:
$$
\begin{array}{l} S I M (y, \hat {y}) = 1 - c o s (y, \hat {y}) \\ y = \left[ y _ {1}, y _ {2}, \dots , y _ {N} \right] \\ \hat {y} = [ \hat {y} _ {1}, \hat {y} _ {2}, \dots , \hat {y} _ {N} ] \\ \end{array}
$$
where  $y_{i}$  and  $\hat{y}_i$  are the predicted score and label for the ith essay respectively,  $N$  is the number of the essays.
Margin Ranking (MR) measures the ranking orders for each essay pair in the batch. We intuitively introduce MR loss because the sorting property between essays is a key factor to scoring. For each batch of essays, we first enumerate all the essay pairs, and then compute the MR loss as follows. The MR loss attempts to make the model penalize wrong order.
$$
\begin{array}{l} M R (y, \hat {y}) = \frac {1}{\hat {N}} \sum_ {i, j} \max  \left(0, - r _ {i, j} \left(y _ {i} - y _ {j}\right) + b\right) \\ r _ {i, j} = \left\{ \begin{array}{l l} 1 & \hat {y} _ {i} > \hat {y} _ {j} \\ - 1 & \hat {y} _ {i} <   \hat {y} _ {j} \\ - s g n (y _ {i} - y _ {j}) & \hat {y} _ {i} = \hat {y} _ {j} \end{array} \right. \\ \end{array}
$$
$y_{i}$  and  $\hat{y}_i$  are the predicted score and label for the  $i$ th essay respectively.  $\hat{N}$  is the number of the essay pairs.  $b$  is a hyper parameter, which is set to 0 in our experiment. For each sample pair  $(i,j)$ , when the label  $\hat{y}_i$  is larger than  $\hat{y}_j$ , the predicted result  $y_{i}$  should be larger than  $y_{j}$ , otherwise, the pair contributes  $y_{j} - y_{i}$  to the loss. When  $\hat{y}_i$  is equal to  $\hat{y}_j$ , the loss is actually  $|y_i - y_j|$ .
The combined loss is described as below:
$$
\begin{array}{c} L o s s _ {t o t a l} (y, \hat {y}) = \alpha M S E (y, \hat {y}) + \beta M R (y, \hat {y}) + \\ \gamma S I M (y, \hat {y}). \end{array}
$$
$\alpha, \beta, \gamma$  are weight parameters which are tuned according to the performance on develop set.
4.1 Data and Evaluation
ASAP data set is widely used in the AES task, which contains eight different prompts. A detailed description can be seen in Table 1. For each prompt, the WordPiece length indicates the smallest number which is bigger than the length of  $90\%$  of the
Figure 1: The proposed automated essay scoring architecture based on multi-scale essay representation. The left part illustrates the document-scale and token-scale essay representation and scoring module, and the right part illustrates  $S$  segment-scale essay representations and scoring modules.
essays in terms of WordPiece number. We evaluate the scoring performance using QWK on ASAP data set, which is the official metric in the ASAP competition. Following previous work, we adopt 5-fold cross validation with 60/20/20 split for train, develop and test sets.
CRP data set provides 2834 excerpts from several time periods and reading ease scores which range from -3.68 to 1.72. The average length of the excerpts is 175 and the WordPiece length is 252. We also use 5-fold cross validation with 60/20/20 split for train, develop and test sets on CRP data set. As the RMSE metric is used in the CRP competition, we also use it to evaluate our system in ease score prediction task.
Prompt	Essays	Avg length	Score Range	WordPiece length
1	1783	350	2-12	649
2	1800	350	1-6	704
3	1726	150	0-3	219
4	1772	150	0-3	203
5	1805	150	0-4	258
6	1800	150	0-4	289
7	1569	250	0-30	371
8	723	650	0-60	1077
Table 1: Statistics of ASAP data set.
The baseline models for comparison are described as follows.
EASE is the best open-source system that participated in the ASAP competition and ranked the third place among 154 participants. EASE uses regression techniques with handcrafted features. Results of EASE with the settings of Support Vector Regression (SVR) and Bayesian Linear Ridge Regression (BLRR) are reported in (Phandi et al., 2015).
CNN+RNN Various deep neural networks based on CNN and RNN for AES are studied by (Taghipour and Ng, 2016). They combine CNN ensembles and LSTM ensembles over 10 runs and get the best result in their experiment.
Hierarchical LSTM-CNN-Attention (Dong et al., 2017) builds a hierarchical sentence-document model, which uses CNN to encode sentences and LSTM to encode texts. The attention mechanism is used to automatically determine the relative weights of words and sentences in generating sentence representations and text representations respectively. They obtain the state-of-the-art result among all neural models without pretraining.
5http://github.com/edm/ease
SKIPFLOW (Tay et al., 2018) proposes to use SKIPFLOW mechanism to model the relationships between snapshots of the hidden representations of an LSTM. The work of (Tay et al., 2018) also obtains the state-of-the-art result among all neural models without pre-training.
Dilated LSTM with Reinforcement Learning (Wang et al., 2018) proposes a method using a dilated LSTM network in a reinforcement learning framework. They attempt to directly optimize the model using the QWK metric which considers the rating schema.
HA-LSTM+SST+DAT and BERT+SST+DAT (Cao et al., 2020) propose to use two self-supervised tasks and a domain adversarial training technique to optimize their training, which is the first work to use pre-trained language model to outperform LSTM based methods. They experiment with both hierarchical LSTM model and BERT in their work, which are HA-LSTM+SST+DAT and BERT +SST + DAT respectively.
$\mathbf{BERT}^2$  (Yang et al., 2020) combines regression and ranking to fine-tune BERT model which also outperforms LSTM based methods and even obtains the new state-of-the-art.
To compare with the baseline models and further study the effectiveness of multi-scale essay representations, losses and transfer learning, we conduct the following experiments.
Multi-scale Models. These models are optimized with MSE loss, and BERT-DOC represents essays with document-scale features based on BERT. BERT-TOK represents essays with token-scale features based on BERT. BERT-DOC-TOK represents essays with both document-scale and token-scale features based on BERT. BERT-DOC-TOK-SEG represents essays with document-scale, token-scale, and multiple segment-scale features based on BERT. Longformer (Beltagy et al., 2020) is an extension for transformers with an attention mechanism that scales linearly with sequence length, making it easy to process long documents. We conduct experiments to show that our multi-scale features also works with Longformer and can further improve the performance in long text tasks. Longformer-DOC-TOK-SEG uses document-scale, token-scale, and multiple segment-scale features to represent essays, but based on Longformer instead of BERT. Longformer-DOC
represents essays with document-scale features based on Longformer.
Models with Transfer Learning. To transfer learn from the out-of-domain essays  $^{6}$ , we additionally employ a pre-training stage, which is similar to the work of (Song et al., 2020). In this stage, we scale all the labels of essays from out-of-domain data into range 0-1 and pre-train the model on them with MSE loss. After the pre-training stage, we continue to fine-tune the model on in-domain essays. Tran-BERT-MS has the same modules as BERT-DOC-TOK-SEG with pre-training on out-of-domain data. MS means multiple scale features.
Models with Multiple Losses. Based on Tran-BERT-MS model, we explore the performance of adding multiple loss functions. Tran-BERT-MS-ML additionally employs MR loss and SIM loss. ML means multiple losses. Tran-BERT-MS-ML-R incorporates R-Drop strategy (Liang et al., 2021) in training based on Tran-BERT-MS-ML model.