SlowGuess/NAACL_2022_OCR huggingface dataset:
$\clubsuit$  Rahul Kumar,  $\clubsuit$ Sandeep Mathias,  $\diamond$ Sriparna Saha,  $\heartsuit$ Pushpak Bhattacharyya
\*Department of Computer Science and Engineering, IIT Kanpur
$\spadesuit$ Department of Computer Science and Engineering, Presidency University, Bangalore
$\diamond$ Department of Computer Science and Engineering, IIT Patna
$^{\diamond}$ Department of Computer Science and Engineering, IIT Bombay
$\clubsuit$ rahul@iitk.ac.in,  $\clubsuit$ sandeepalbert@presidencyuniversity.in,
$\diamondsuit$ sriparna@iitp.ac.in,  $\diamondsuit$ pb@cse.iitb.ac.in
Most research in the area of automatic essay grading (AEG) is geared towards scoring the essay holistically while there has also been little work done on scoring individual essay traits. In this paper, we describe a way to score essays using a multi-task learning (MTL) approach, where scoring the essay holistically is the primary task, and scoring the essay traits is the auxiliary task. We compare our results with a single-task learning (STL) approach, using both LSTMs and BiLSTMs. To find out which traits work best for different types of essays, we conduct ablation tests for each of the essay traits. We also report the runtime and number of training parameters for each system. We find that MTL-based BiLSTM system gives the best results for scoring the essay holistically, as well as performing well on scoring the essay traits. The MTL systems also give a speed-up of between 2.30 to 3.70 times the speed of the STL system, when it comes to scoring the essay and all the traits.
An essay is a piece of text that is written in response to a topic, called a prompt (Mathias and Bhattacharyya, 2020). Qualitative evaluation of the essay consumes a lot of time and resources. Hence, in 1966, Page proposed a method of automatically scoring essays using computers (Page, 1966), giving rise to the domain of Automatic Essay Grading.
Essay traits are different aspects of the essay that can aid in explaining the score assigned to the essay. Examples of essay traits include content (how much information is present in the essay) (Page, 1966), organization (how well the essay is structured) (Persing et al., 2010), style (how well written the essay is) (Page, 1966), prompt adherence (how
much the essay stays on topic for the essay prompt) (Persing and Ng, 2014), etc.
Most of the research work done in the field of AEG is geared toward scoring the essay holistically, rather than studying the importance of essay traits in the overall essay score. In this paper, we ask the question:
"Can we use information learnt from scoring essay traits to score an essay holistically?"
In our paper, not only do we score essays holistically, but we also describe how to score essay traits simultaneously in a multi-task learning framework. Scoring essay traits is essential as it could help in explaining why the essay was scored the way it was, as well as providing valuable insights to the writer about what aspects of the essay were well-written and what the writer needs to improve.
Multi-task learning is a machine learning technique where we use information from multiple auxiliary tasks to perform a primary task (Caruana, 1997). In our experiments, scoring the individual essay traits is the auxiliary task, and scoring the essay holistically is the primary task. In addition to this, we also study the impact of scoring an essay trait as the primary task while the other traits and overall essay score are auxiliary tasks.
Contributions. In this paper, we describe a way to simultaneously score essay traits and the essay itself using multi-task learning. We evaluate our system against different types of essays and essay traits. We also share our code and the data for reproducibility and further research<sup>1</sup>.
1The code and data are at the following location: https://github.com/ASAP-AEG/ MTL-Essay-Traits-Scoring
Human Language Technologies, pages 1485 - 1495
Most of the work done in the area of automatic essay grading is in the area of holistic AEG - where we provide a single score for the entire essay based on its quality. However, for writers of an essay, a holistic score alone would not be enough. Providing trait-specific scores will tell the writer which aspects of the essay need improvement.
In our dataset, we observe that writers of good essays usually have a lot of content, appropriate word choice, very few errors, etc. Essays that are poorly written often lack one or more of these qualities (i.e. they are either too short, have lots of errors, etc.). We, therefore, observe a high correlation between individual trait scores and the overall essay score (Pearson correlation trait scores and overall essay score  $>0.7$  across all essay sets in our dataset). Hence, we believe that using essay trait scores will benefit in scoring the essay holistically, as their scores will provide more relevant information to the AEG system.
3.1 Holistic Essay Grading
Holistic essay grading involves assigning an overall score for an essay (Mathias and Bhattacharyya, 2020). The first AEG system was designed by Page (1966). In the decade of the 2000s there were a lot of AEG systems which were developed commercially (see Sermis and Burstein (2013) for more details).
After the release of Kaggle's Automatic Student Assessment Prize's (ASAP) Automatic Essay Grading (AEG) dataset in  $2012^{2}$ , there has been a lot of research on holistic essay grading. Initial approaches, such as those of Phandi et al. (2015) and Zesch et al. (2015) used feature engineering techniques and domain adaptation in scoring the essays. More recent papers look at using a number of deep learning approaches, such as LSTMs (Taghipour and Ng, 2016; Tay et al., 2018) and CNNs (Dong and Zhang, 2016) or both (Dong et al., 2017; Zhang and Litman, 2018, 2020). Zhang and Litman (2020) describe a way to extract important information, called topical components, from a source-dependent response<sup>3</sup>.
3.2 Trait-specific Essay Grading
In the last decade or so, there has been some work done in scoring essay traits such as sentence fluency (Chae and Nenkova, 2009), organization (Persing et al., 2010; Taghipour, 2017; Mathias et al., 2018; Song et al., 2020), thesis clarity (Persing and Ng, 2013; Ke et al., 2019) coherence (Somasundaran et al., 2014; Mathias et al., 2018), prompt adherence (Persing and Ng, 2014), argument strength (Persing and Ng, 2015; Taghipour, 2017), stance (Persing and Ng, 2016), style (Mathias and Bhattacharyya, 2018b) and narrative quality (Somasundaran et al., 2018). None of the above work, however, uses trait information to score the essay holistically.
There has also been work on scoring multiple essay traits (Taghipour, 2017; Mathias and Bhattacharyya, 2018a, 2020). (Rama and Vajjala, 2021) discuss solutions across multiple languages (German, Czech and Italian). Mathias and Bhattacharyya (2020) describes work on the use of neural networks for scoring essay traits. Our work combines the scores of essay traits for holistic essay grading. We focus on using trait-specific essay grading to improve the performance of an automatic essay grading system. We also show how using multi-task learning - simultaneously scoring both the essay and its traits - we are able to speed up the training of our system without too much of a loss in scoring the essay traits. (Ridley et al., 2021) describe a multi-task learning approach to grade essays and their traits using a neural network. Our system differs from theirs with respect to the shared layers and trait-specific layers. While Ridley et al. (2021) share the embedding and word-specific layers (to get sentence representations), we share only the embedding layer.
3.3 Multi-task Learning
Multitask Learning was proposed by Caruana (1997) where the argument was that training signals from related tasks could help in a better generalization of the model. Collobert et al. (2011) successfully demonstrated how tasks like Part-of-Speech tagging, chunking and Named Entity Recognition can help each other when trained jointly using deep neural networks. Song et al. (2020) described a multi-task learning approach to score organization in essays, where the auxiliary tasks were classifying the sentences and paragraphs, and the primary task was scoring the essay's organization. Cao et al.
$^{2}$ https://www.kaggle.com/c/asap-aes
3We define what a source-dependent response is in the Dataset Section (i.e. Section 5).
(2020) also use a domain adaptive MTL approach to grade essays, where their auxiliary tasks are sentence reordering, noise identification, as well as domain adversarial training. However, they also use all the other essay sets as part of their training, whereas we use only the essays present in the respective essay set for training.
4 System Architecture
4.1 STL Essay Grading Stack
For scoring the essays, we use essay grading stacks. Each stack is used for scoring a single essay trait. The architecture of the stack is based on the architecture of the holistic essay grading system proposed by Dong et al. (2017). The essay grading stack takes the essay as input (split into tokens and sentences) and returns the score of the essay / essay trait as the output. Figure 1 shows the architecture for the essay grading stack.
For each essay, we first split the essay into tokens and sentences. This is given as an input to the essay grading stack. In the word embedding layer, we look up the word embeddings of each token. Just like Taghipour and Ng (2016), Dong et al. (2017), Tay et al. (2018), Mathias and Bhattacharyya (2020) and Mathias et al. (2020), we use the most frequent 4000 words of the training data as the vocabulary with all other words mapping to a special unknown token. This is done mainly to capture out-of-vocabulary words, as well words that generally don't belong in the topic<sup>4</sup>. If the vocabulary size is too small, then a number of words will be marked as spelling errors. On the other hand, if the vocabulary size is too large, a lot of spurious words would also be learnt as important ones.
This sequence of word embeddings is then sent to the next layer - the 1 dimension CNN layer - to get local information from nearby words. The output of this layer is aggregated using attention pooling to get the sentence representation of the sentence. This is done for all sentences in the essay.
Each of the sentence representations are then sent through a recurrent layer. We experiment on two different types of recurrent layers - a unidirectional LSTM (Hochreiter and Schmidhuber, 1997) and bidirectional LSTM (BiLSTM) - as the recurrent layer. The outputs of the recurrent layer are
pooled using attention pooling to get the representation for the essay. This essay representation is then sent through a fully-connected Dense layer with a sigmoid activation function to score the essay either holistically or a particular essay trait. For our experiments, we minimize the mean squared error loss.
Prior to input, we scale the scores to the range of [0, 1] using min-max normalization. The output of the sigmoid function is a scalar in the range of [0, 1] which is rescaled back up to a score in the original score range and rounded off to get the score for the essay. This essay stack is used for the scoring of the single-task learning (STL) models.
4.2 MTL Model
The architecture of our MTL model for an essay of  $M$  traits is shown in Figure 2. Here, the word embedding layer is shared across all the tasks. In the multi-task learning framework, each stack is used to learn an essay representation for each essay trait. In a similar manner, the essay representation for the overall score is learnt and it is concatenated with the predicted trait scores before being sent to a Dense layer with a sigmoid activation function to score the essay holistically. For calculating each score - both overall and trait scores - we use the mean squared error loss function. We experimented with multiple weights for the loss function for the essay trait scoring task, but settled on uniform weights for all the traits and the overall scoring task. This is done because we want to get accurate predictions of the traits scores which are used for predicting the overall score.
5 Dataset Used
For our experiments, we use the Automated Student's Assessment Prize (ASAP) Automatic Essay Grading (AEG) dataset. The dataset has a total of 8 essay sets - where each essay set has a number of essays written in response to the same essay prompt. In total, there are nearly 13,000 English essays in the dataset, written by American high school students from classes 7 to 10.
Table 1 gives the properties of each of the essay sets in our dataset. It reports the overall essay scoring range, traits scoring, average word count, number of traits, number of essays and essay type.
We use the overall scores directly from the ASAP AEG dataset. Since the original dataset only provided trait-specific scores for Prompts 7
4 Using the word "relay" when writing about something you like instead of "really". "Relay" could be a valid word in the context of athletics, but not in the context of an argumentative essay on libraries! These valid words are learnt from the training data.
Figure 1: Essay stack architecture. This is the architecture for the Single-Task Learning systems.
Essay Set	Score Range	Trait Sc. Range	Word Count	No. of Traits	No. of Essays	Essay Type
Prompt 1	2-12	1-6	350	5	1783	Argumentative / Persuasive
Prompt 2	1-6	1-6	350	5	1800	Argumentative / Persuasive
Prompt 3	0-3	0-3	100	4	1726	Source-Dependent Response
Prompt 4	0-3	0-3	100	4	1772	Source-Dependent Response
Prompt 5	0-4	0-4	125	4	1805	Source-Dependent Response
Prompt 6	0-4	0-4	150	4	1800	Source-Dependent Response
Prompt 7	0-30	0-6	300	4	1569	Narrative / Descriptive
Prompt 8	0-60	0-12	600	6	723	Narrative / Descriptive
Total	0-60	0-12	100-600	4-6	12978	-
Table 1: Properties of the different essay sets in the ASAP AEG dataset we used in our experiments. Average word count numbers are rounded up to the nearest multiple of 25.
& 8, we use the trait-specific scores provided by Mathias and Bhattacharyya (2018a).
Depending on the type of prompt for the essay set, each essay set has a different set of traits. Argumentative / Persuasive essays are essays which the writer is prompted to take a stand on a topic and argue for their stance. These essay sets have traits like content, organization, word choice, sentence fluency, and conventions. Source-dependent responses (Zhang and Litman, 2018) are essays where the writer reads a piece of text and answers a question based on the text that they just read<sup>5</sup>. These essay sets have traits like content, prompt adherence (Persing and Ng, 2014), language and
narrativity (Somasundaran et al., 2018). Narrative / Descriptive essays are essays where the writer has to narrate a story or incident or anecdote. They have traits like content, organization, style, conventions, voice, word choice, and sentence fluency $^{6}$ . Table 2 lists the different essay traits for each essay set.
6.1 Evaluation Metric
We use Cohen's Kappa with quadratic weights (Cohen, 1968) (QWK) as the evaluation metric. This is done for the following reasons. Firstly, the final scores predicted by the system are distinct numbers/grades, rather than continuous values; so we cannot use the Pearson Correlation Coefficient or Mean Squared Error. Secondly, evaluation metrics
Neither the original ASAP dataset, nor Mathias and Bhattacharyya (2018a) have scored narrativity for the narrative essays.
5 A sample prompt is "Based on the excerpt, describe the obstacles the builders of the Empire State Building faced in attempting to allow dirigibles to dock there. Support your answer with relevant and specific information from the excerpt." It involves the writer reading the excerpt from The Empire State Building by Marcia Amidon Lusted before writing the essay.
Figure 2: Architecture of our MTL system showing an input essay with  $M$  traits being scored, with the overall score and each trait's essay grading stack.
Essay Set	Trait 1	Trait 2	Trait 3	Trait 4	Trait 5	Trait 6
Prompt 1	Content	Organization	Word Choice	Sentence Fluency	Conventions	N/A
Prompt 2	Content	Organization	Word Choice	Sentence Fluency	Conventions	N/A
Prompt 3	Content	Prompt Adherence	Language	Narrativity	N/A	N/A
Prompt 4	Content	Prompt Adherence	Language	Narrativity	N/A	N/A
Prompt 5	Content	Prompt Adherence	Language	Narrativity	N/A	N/A
Prompt 6	Content	Prompt Adherence	Language	Narrativity	N/A	N/A
Prompt 7	Content	Organization	Style	Conventions	N/A	N/A
Prompt 8	Content	Organization	Voice	Word Choice	Sentence Fluency	Conventions
like F-Score and accuracy do not take into account chance agreements. For example, if we are to grade every essay with the mean score or most frequent score, we would get F-Score and accuracy as high as  $60\%$  or more, whereas the Kappa score will be 0! Thirdly, the fact that the scores given are ordered (i.e.  $0 < 1 < 2 < 3\ldots$ ) means that we need to use weighted Kappa to capture the distance between the actual and predicted scores. Between linear weighted Kappa and QWK, we choose QWK because it rewards matches and punishes mismatches more distinctly than linear weighted Kappa.
6.2 Evaluation Method
We evaluate our experiments using five-fold cross validation. We use the same data splits as used by Taghipour and Ng (2016). To avoid overfitting, we choose the model which gives the best result on the validation set for evaluating on the test set, and we report the mean value of all 5 folds.
Table 2: Traits that are present in each essay set in our dataset. The trait scores are taken from the original ASAP dataset, as well as from ASAP++ (Mathias and Bhattacharyya, 2018a).
Layer	Param. Name	Param. Value
Embedding	Embedding Dim.	50
Embeddings	GloVe
Word CNN	Window Size	5
Filters	100
Sentence LSTM	Hidden Units	100
Epochs	100
Batch Size	100
Dropout Rate	0.5
Initial Learning Rate	0.001
Momentum	0.9
Optimizer	RMSProp
Table 3: Neural network hyper-parameters for each layer, showing the hyper-parameter name and its corresponding value.
6.3 Experiment Configuration
Table 3 gives the different hyperparameters used in our systems. For the sake of uniformity, we use these hyperparameters irrespective of the network configuration (STL vs MTL, or LSTM vs BiLSTM).
Essay Set	Kernel	STL-LSTM	STL-BiLSTM	MTL-LSTM	MTL-BiLSTM	BERT-STL
Prompt 1	0.804	0.813	0.818	0.830*	0.831*	0.800
Prompt 2	0.687	0.660	0.658	0.667	0.689*★	0.679
Prompt 3	0.704	0.661	0.653	0.644	0.687*★	0.679
Prompt 4	0.743	0.790	0.780	0.786	0.798	0.822
Prompt 5	0.799	0.798	0.789	0.782	0.800	0.803
Prompt 6	0.753	0.807	0.803	0.806	0.813*★	0.797
Prompt 7	0.698	0.792	0.786	0.791	0.795★	0.827
Prompt 8	0.552	0.678	0.697	0.679	0.699*★	0.725
Mean QWK	0.717	0.750	0.748	0.748	0.764*★	0.767
Table 4: Results of our experiments for scoring the essays holistically. Figures in boldface represent the best results per essay set. * represents a statistically significant improvement using the MTL systems over the STL-LSTM system.  $\star$  represents a statistically significant improvement of using the MTL-BiLSTM system over the MTL-LSTM system.
To evaluate the performance of our systems in scoring the essay overall, we use 4 different configurations - STL-LSTM, STL-BiLSTM, MTL-LSTM, and MTL-BiLSTM. In addition to the above systems, we also compare our approach with a state-of-the-art string kernel system designed by Cozma et al. (2018), using the same splits for training, testing, and validation<sup>7</sup>, as well as a baseline transformer-based implementation (BERT-STL), using the BERT-base-uncased model. We run this baseline model for 100 epochs and a batch size of 30, all other hyperparameters remaining default.
We also study the effect of using our system to grade an essay trait as the primary task, and score the other traits and the essay overall as auxiliary tasks (MTL*).
In the STL configurations, we train our system to predict a single score at a time- either the overall essay score or the score for any of the essay traits. In the MTL configurations, our system learns to score the essay and all its traits simultaneously. The LSTM configurations use only a forward direction LSTM, while the BiLSTM configurations use a bidirectional (i.e. forward and reverse) LSTM.
In this section, we report our results and analyze them for different experiments.
7.1 Performance on Holistic Essay Scoring
Table 4 gives the QWK scores of each of our systems as they score each essay set holistically. The different systems used are the Single Task Learning
(STL) (only scoring the essay overall) and Multitask Learning (MTL) (scoring the essay and the traits simultaneously). The first column lists out the different essay sets (Prompts 1 to 8). The next three columns report results for STL using both LSTM and BiLSTM, as well as results using the string kernel-based approach of Cozma et al. (2018). The next two columns report results for the MTL systems using both LSTM and BiLSTM. The last column shows the results using the baseline BERT-STL system.
From the table, we see that the MTL-BiLSTM performs the best of all the non-transformer systems (almost as good as the results of our BERT-STL system). In order to see if the improvements observed are statistically significant, we run the Paired T-Test for each of the essay sets and compare the results using a p-value of  $p < 0.05$ .
7.2 Performance on Scoring Essay Traits
We also look at how our system performs in the auxiliary tasks - namely scoring the different essay traits. Figure 3 gives the results of our experiments in scoring the essay traits, using the String Kernel (HISK) (Cozma et al., 2018), CNN-LSTM (STL) (Dong et al., 2017), and Our Systems (MTL and MTL*). We use the same evaluation method, which we used for scoring essay traits, with the same data splits. For the STL systems, we train them for every essay trait individually. MTL* is the results of using our system to score essay trait as the primary task, and score the other traits and the essay overall as auxiliary tasks.
We compare the results with that of our MTL-BiLSTM system, which was trained to score the essay traits as auxiliary tasks. Figure 3 gives the
${}^{7}$  Cozma et al. (2018) do not provide their folds,so we run their system on our training/validation/test split,as given by Taghipour and Ng (2016).
Figure 3: Results of the performance of scoring traits for different systems. The different systems are Histogram Intersection String Kernel (HISK) using  $\nu$  support vector regressor (Cozma et al., 2018), LSTM-CNN Single-task Learning System (STL) (Dong et al., 2017), our system (MTL) where the traits are auxiliary tasks, and our system (MTL*) where the traits are the primary task. NOTE that the y-axis starts from a QWK of 0.45. This was done to highlight the difference in the performance of each system for each trait.
results of our experiments. From the figure, we see that, while the STL-LSTM system is able to outperform our MTL-BiLSTM system, the MTL* system (where the traits are primary tasks) outperforms the STL-LSTM system. While the STL system optimizes for scoring only a trait, the MTL* system learns information from other traits to score the given essay trait.
One of the reasons for the different trait performance depends on how easy (or difficult) it is to score the individual trait, as well which all essay traits have the particular trait. For example, prompt adherence has a higher average QWK than the other traits because it is present mainly in the source-dependent essays (which have a mean QWK over  $1\%$  higher than the mean QWK across all essay sets). Similarly, Voice has the lowest QWK mainly because it is present only in Prompt # 8, which has a very low holistic QWK.
7.3 Scoring Traits as the Primary Task