SlowGuess/ACL_2020_OCR huggingface dataset:
Haoran Zhang
Pittsburgh, PA 15260
colinzhang@cs.pitt.edu
Diane Litman
Department of Computer Science & LRDC
litman@cs.pitt.edu
While automated essay scoring (AES) can reliably grade essays at scale, automated writing evaluation (AWE) additionally provides formative feedback to guide essay revision. However, a neural AES typically does not provide useful feature representations for supporting AWE. This paper presents a method for linking AWE and neural AES, by extracting Topical Components (TCs) representing evidence from a source text using the intermediate output of attention layers. We evaluate performance using a feature-based AES requiring TCs. Results show that performance is comparable whether using automatically or manually constructed TCs for 1) representing essays as rubric-based features, 2) grading essays.
Automated essay scoring (AES) systems reliably grade essays at scale, while automated writing evaluation (AWE) systems additionally provide formative feedback to guide revision. Although neural networks currently generate state-of-the-art AES results (Alikaniotis et al., 2016; Taghipour and Ng, 2016; Dong et al., 2017; Farag et al., 2018; Jin et al., 2018; Li et al., 2018; Tay et al., 2018; Zhang and Litman, 2018), non-neural AES create feature representations more easily useable by AWE (Roscoe et al., 2014; Foltz and Rosenstein, 2015; Crossley and McNamara, 2016; Woods et al., 2017; Madnani et al., 2018; Zhang et al., 2019). We believe that neural AES can also provide useful information for creating feature representations, e.g., by exploiting information in the intermediate layers.
Our work focuses on a particular source-based essay writing task called the response-to-text assessment (RTA) (Correnti et al., 2013). Recently, an RTA AWE system (Zhang et al., 2019) was built by extracting rubric-based features related to the
use of Topical Components (TCs) in an essay. However, manual expert effort was first required to create the TCs. For each source, the TCs consist of a comprehensive list of topics related to evidence which include: 1) important words indicating the set of evidence topics in the source, and 2) phrases representing specific examples for each topic that students need to find and use in their essays.
To eliminate this expert effort, we propose a method for using the interpretable output of the attention layers of a neural AES for source-based essay writing, with the goal of extracting TCs. We evaluate this method by using the extracted TCs to support feature-based AES for two RTA source texts. Our results show that 1) the feature-based AES with TCs manually created by humans is matched by our neural method for generating TCs, and 2) the values of the rubric-based essay features based on automatic TCs are highly correlated with human Evidence scores.
Three recent AWE systems have used non-neural AES to provide rubric-specific feedback. Woods et al. (2017) developed an influence estimation process that used a logistic regression AES to identify sentences needing feedback. Shibani et al. (2019) presented a web-based tool that provides formative feedback on rhetorical moves in writing. Zhang et al. (2019) used features created for a random forest AES to select feedback messages, although human effort was first needed to create TCs from a source text. We automatically extract TCs using neural AES, thereby eliminating this expert effort.
Others have also proposed methods for preprocessing source information external to an essay. Content importance models for AES predict the parts of a source text that students should include when writing a summary (Klebanov et al.,
8569
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8569-8584 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Source Excerpt: Today, Yala Sub-District Hospital has medicine, free of charge, for all of the most common diseases. Water is connected to the hospital, which also has a generator for electricity. Bed nets are used in every sleeping site in Sauri...
Essay Prompt: The author provided one specific example of how the quality of life can be improved by the Millennium Villages Project in Sauri, Kenya. Based on the article, did the author provide a convincing argument that winning the fight against poverty is achievable in our lifetime? Explain why or why not with 3-4 examples from the text to support your answer.
Essay: In my opinion I think that they will achieve it in lifetime. During the years threw 2004 and 2008 they made progress. People didn't have the money to buy the stuff in 2004. The hospital was packed with patients and they didn't have alot of treatment in 2004. In 2008 it changed the hospital had medicine, free of charge, and for all the common dieases. Water was connected to the hospital and has a generator for electricity. Everybody has net in their site. The hunger crisis has been addressed with fertilizer and seeds, as well as the tools needed to maintain the food. The school has no fees and they serve lunch. To me thats sounds like it is going achieve it in the lifetime.
Table 1: A source excerpt for the  $RTA_{MVP}$  prompt and an essay with score of 3.
Prompt	RTAMVP	RTASpace
Score 1	852	538
(29%)	(26%)
Score 2	1197	789
(40%)	(38%)
Score 3	616	512
(21%)	(25%)
Score 4	305	237
(10%)	(11%)
Total	2970	2076
Table 2: The Evidence score distribution of RTA.
2014). Methods for extracting important keywords or keyphrases also exist, both supervised (unlike our approach) (Meng et al., 2017; Mahata et al., 2018; Florescu and Jin, 2018) and unsupervised (Florescu and Caragea, 2017). Rahimi and Litman (2016) developed a TC extraction LDA model (Blei et al., 2003). While the LDA model considers all words equally, our model takes essay scores into account by using attention to represent word importance. Both the unsupervised keyword and LDA models will serve as baselines in our experiments.
In the computer vision area, attention cropped images have been used for further image classification or object detection (Cao et al., 2015; Yuxin et al., 2018; Ebrahimpour et al., 2019). In the NLP area, Lei et al. (2016) proposed to use a generator to find candidate rationale and these are passed through the encoder for prediction. Our work is similar in spirit to this type of work.
3 RTA Corpus and Prior AES Systems
The essays in our corpus were written by students in grades 4 to 8 in response to two RTA source texts (Correnti et al., 2013):  $RTA_{MVP}$  (2970 essays) and  $RTASpace$  (2076 essays). Table 1 shows an excerpt from  $RTA_{MVP}$ , the associated essay writing prompt, and a student essay. The bolding in the source indicates evidence examples that ex
perts manually labeled as important for students to discuss (i.e., TC phrases). Evidence usage in each essay was manually scored on a scale of 1 to 4 (low to high). The distribution of Evidence scores is shown in Table 2. The essay in Table 1 received a score of 3, with the bolding indicating phrases semantically related to the TCs from the source text.
To date, two approaches to AES have been proposed for the RTA:  $AES_{\text{rubric}}$  and  $AES_{\text{neural}}$ . To support the needs of AWE,  $AES_{\text{rubric}}$  (Zhang and Litman, 2017) used a traditional supervised learning framework where rubric-motivated features were extracted from every essay before model training - Number of Pieces of Evidence (NPE) $^{1}$ , Concentration (CON), Specificity (SPC) $^{2}$ , Word Count (WOC). The two aspects of TCs introduced in Section 1 (topic words, specific example phrases) were used during feature extraction.
Motivated by improving stand-alone AES performance (i.e., when an interpretable model was not needed for subsequent AWE), Zhang and Litman (2018) developed  $AES_{neural}$ , a hierarchical neural model with the co-attention mechanism in the sentence level to capture the relationship between the essay and the source. Neither feature engineering nor TC creation were needed before training.
4 Attention-Based TC Extraction:  $TC_{attn}$
In this section we propose a method for extracting TCs based on the  $AES_{neural}$  attention level outputs. Since the self-attention and co-attention mechanisms were designed to capture sentence and phrase importance, we hypothesize that the attention scores can help determine if a sentence or
An integer feature based on the list of topic words for each topic.
2A vector of integer values indicating the number of specific example phrases (semantically) mentioned in the essay per topic.
8570
No.	Sentences	attn sent	attn phrase
1	People didn't have the money to buy the stuff in 2004.	0.00420	0.23372
2	The hunger crisis has been addressed with fertilizer and seeds, as well as the tools needed to maintain the food.	0.08709	0.62848
3	The school has no fees and they serve lunch.	0.10686	0.63369
phrase has important source-related information.
To provide intuition, Table 3 shows examples sentences from the student essay in Table 1. Bolded are phrases with the highest self-attention score within the sentence. Italics are specific example phrases that refer to the manually constructed TCs for the source. Attn sent is the text to essay attention score that measures which essay sentences have the closest meaning to a source sentence. Attn phrase is the self-attention score of the bolded phrase that measures phrase importance. A sentence with a high attention score tends to include at least one specific example phrase, and vice versa. The phrase with the highest attention score tends to include at least one specific example phrase if the sentence has a high attention score.
Based on these observations, we first extract the output of two layers from the neural network: 1) the  $\text{att}n_{sent}$  of each sentence, and 2) the output of the convolutional layer as the representation of the phrase with the highest  $\text{att}n_{phrase}$  in each sentence (denoted by  $\text{conn}_{phrase}$ ). We also extract the plain text of the phrase with the highest  $\text{att}n_{phrase}$  in each sentence (denoted by  $\text{text}_{phrase}$ ). Then, our  $TC_{attn}$  method uses the extracted information in 3 main steps: 1) filtering out  $\text{text}_{phrase}$  from sentences with low  $\text{att}n_{sent}$ , 2) clustering all remaining  $\text{text}_{phrase}$  based on  $\text{conn}_{phrase}$ , and 3) generating TCs from clusters.
The first filtering step keeps all  $text_{phrase}$  where the original sentences have  $attn_{sent}$  higher than a threshold. The intuition is that lower  $attn_{sent}$  indicates less source-related information.
The second step clusters these  $text_{phrase}$  based on their corresponding representations  $cnn_{phrase}$ . We use k-medoids to cluster  $text_{phrase}$  into  $M$  clusters, where  $M$  is the number of topics in the source text. Then, for  $text_{phrase}$  in each topic cluster, we use k-medoids to cluster them into  $N$  clusters, where  $N$  is the number of the specific example phrases we want to extract from each topic. The outputs of this step are  $M*N$  clusters.
The third step uses the topic and example clus-
Table 3: Example attention scores of essay sentences.
Layer	Parameter Name	Value
Embedding	Embedding dimension	50
Word-CNN	Kernel size	5
Number of filters	100
Sent-LSTM	Hidden units	100
Modeling	Hidden units	100
Dropout	Dropout rate	0.5
Others	Epochs	100
Batch size	100
Initial learning rate	0.001
Momentum	0.9
Table 4: Hyper-parameters for neural training.
Figure 1: An overview of four TC extraction systems.
tering to extract TCs. As noted earlier, TCs include two parts: topic words, and specific example phrases. Since our method is data-driven and students introduce their vocabulary into the corpus, essay text is noisy. To make the TC output cleaner, we filter out words that are not in the source text. To obtain topic words, we combine all  $text_{phrase}$  from each topic cluster to calculate the word frequency per topic. To make topics unique, we assign each word to the topic cluster in which it has the highest normalized word frequency. We then include the top  $K_{topic}$  words based on their frequency in each topic cluster. To obtain example phrases, we combine all  $text_{phrase}$  from each example cluster to calculate the word frequency per example, then include the top  $K_{example}$  words based on their frequency in each example cluster.
5 Experimental Setup and Results
Figure 1 shows an overview of four TC extraction methods to be evaluated.  $TC_{manual}$  (upper bound) uses a human expert to extract TCs from a source text.  $TC_{attn}$  is our proposed method and automatically extracts TCs using both a source text and student essays.  $TC_{lda}$  (Rahimi and Litman, 2016) (baseline) builds on LDA to extract TCs from student essays only, while  $TC_{pr}$  (baseline) builds on PositionRank (Florescu and Caragea, 2017) to instead extract TCs from only the source text.
Since PositionRank is not designed for TC ex
Prompt	Component	Parameter	\(TC_{Ida}\)	\(TC_{pr}\)	\(TC_{attn}\)
\(RTA_{MVP}\)	Topic Words	Number of Topics	9	19	16
Number of Words	30	20	25
Example Phrases	Number of Topics	20	1	18
Number of Phrases	15	20	15
\(RTA_{Space}\)	Topic Words	Number of Topics	15	20	10
Number of Words	10	10	20
Example Phrases	Number of Topics	10	1	9
Number of Phrases	20	50	20
traction, we needed to further process its output to create  $TC_{pr}$ . To extract topic words, we extract all keywords from the output. Next, we map each word to a higher dimension with word embedding. Lastly, we cluster all keywords using k-medoids into  $PR_{topic}$  topics. To extract example phrases, we put them into only one topic and remove all redundant example phrases if they are subsets of other example phrases.
We configure experiments to test two hypotheses: H1) the  $AES_{\text{rubric}}$  model for scoring Evidence (Zhang and Litman, 2017) will perform comparably when extracting features using either  $TC_{\text{attn}}$  or  $TC_{\text{manual}}$ , and will perform worse when using  $TC_{\text{lda}}$  or  $TC_{\text{pr}}$ ; H2) the correlation between the human Evidence score and the feature values (NPE and sum of SPC features)<sup>3</sup> will be comparable when extracted using  $TC_{\text{attn}}$  and  $TC_{\text{manual}}$ , and will be stronger than when using  $TC_{\text{lda}}$  and  $TC_{\text{pr}}$ . The experiment for H1 tests the impact of using our proposed TC extraction method on the downstream  $AES_{\text{rubric}}$  task, while the H2 experiment examines the impact on the essay representation itself.
Following Zhang and Litman (2017), we stratify essay corpora:  $40\%$  for training word embeddings and extracting TCs,  $20\%$  for selecting the best embedding and parameters, and  $40\%$  for testing. We use the hyper-parameters from Zhang and Litman (2018) for neural training as shown in Table 4. Table 5 shows all other parameters selected using the development set.
Results for H1. H1 is supported by the results in Table 6, which compares the Quadratic Weighted Kappa (QWK) between human and  $AES_{rubric}$  Evidence scores (values 1-4) when  $AES_{rubric}$  uses  $TC_{manual}$  versus each of the automatic methods.  $TC_{attn}$  always yields better performance, and even significantly better than  $TC_{manual}$ .
Results for H2. The results in Table 7 support H2.  $TC_{attn}$  outperforms the two automated base
Table 5: Parameters for different models.
Prompt	TCmanual (1)	TClda (2)	TCpr (3)	TCattn (4)
RTAMVP	0.643 (2,3)	0.614 (3)	0.525	0.648 (1,2,3)
RTASpace	0.609 (3)	0.615 (3)	0.559	0.622 (1,3)
Table 6: The performance (QWK) of  $AES_{\text{rubric}}$  using different TC extraction methods for feature creation. The numbers in the parentheses show the model numbers over which the current model performs significantly better  $(p < 0.05)$ . The best results between automated methods in each row are in bold.
Prompt	Feature	TCmanual	TCIda	TCpr	TCattn
RTAMVP	NPE	0.542	0.482	0.587	0.639
SPC (sum)	0.689	0.585	0.365	0.679
RTASpace	NPE	0.484	0.513	0.494	0.625
SPC (sum)	0.601	0.574	0.533	0.598
Table 7: Pearson's r comparing feature values computed using each TC extraction method with human (gold-standard) Evidence essay scores. All correlation values are significant  $(p\leq 0.05)$ . The best results between automated methods in each row are in bold.
lines, and for NPE even yields stronger correlations than the manual TC method.
Qualitative Analysis. The manually-created topic words for  $RTAMVP$  represent 4 topics, which are "hospital", "malaria", "farming" and "school". Although Table 5 shows that the automated list has more topics for topic words and might have broken one topic into separate topics, a good automated list should have more topics related to the 4 topics above. We manually assign a topic for each of the topic words from the different automated methods.  $TC_{lda}$  has 4 related topics out of 9 (44.44%),  $TC_{pr}$  has 6 related topics out of 19 (31.58%), and  $TC_{attn}$  has 10 related topics out of 16 (62.50%). Obviously,  $TC_{attn}$  preserves more related topics than our baselines.
Moving to the second aspect of TCs (specific example phrases), Table 8 shows the first 10 specific example phrases for a manually-created category that introduces the changes made by the MVP project<sup>5</sup>. This category is a mixture of different topics because it talks about the "hospital", "malaria", "school", and "farming" at the same time.  $TC_{attn}$  has overlap with  $TC_{manual}$  on different topics. However,  $TC_{lda}$  mainly talks about "hospital", because the nature of the LDA model doesn't allow mixing specific example phrases about different topics in one category. Unfortunately,  $TC_{pr}$
4All Topic Words generated by different models can be found in the Appendix A.1.
5All Specific Example Phrases generated by different models can be found in the Appendix A.2.
These features are extracted based on TCs.
8572
TCmanual	TClda	TCpr	TCattn
progress just four years	running water electricity	brighter future hannah	electricity running water irrigation set
medicine most common diseases	water connected hospital generator electricity	millennium villages project	poor showed treatment school supplies
water connected hospital	patients afford	unpaved dirt road	farmers could crops afford bed
hospital generator electricity	rooms packed patients probably	bar sauri primary school	electricity hospital
bed nets used every sleeping site	share beds	future hannah	better fertilizer medicine enough also
hunger crisis addressed fertilizer seeds	recieve treatment	sauri primary school	rooms packed patients
tools needed maintain food supply	doctoral officer running hospital	villages project	food fertilizer crops get supply
no school fees	doctors clinical	villages project	five net costs 5
school attendance rate way up	water fertilizer knowledge	village leaders	nets net bed free
kids go school now	receive treatment	dirt road	running water supplies schools almost
...	...	...	...
Table 8: Specific example phrases for the  $RTA_{MVP}$  progress topic.
does not include any overlapped specific phrase in the first 10 items; they all refer to some general example phrases from the beginning of the source article. Although there are some related specific example phrases in the full list, they are mainly about school. This is because the PositionRank algorithm tends to assign higher scores to words that appear early in the text.
This paper proposes  $TC_{attn}$ , a method for using the attention scores in a neural AES model to automatically extract the Topical Components of a source text. Evaluations show the potential of  $TC_{attn}$  for eliminating expert effort without degrading  $AES_{rubric}$  performance or the feature representations themselves.  $TC_{attn}$  outperforms baselines and generates comparable or even better results than a manual approach.
Although  $TC_{attn}$  outperforms all baselines and requires no human effort on TC extraction, annotation of essay evidence scores is still needed. This leads to an interesting future investigation direction, which is training the  $AES_{neural}$  using the gold standard that can be extracted automatically.
One of our next steps is to investigate the impact of TC extraction methods on a corresponding AWE system (Zhang et al., 2019), which uses the feature values produced by  $AES_{rubric}$  to generate formative feedback to guide essay revision.
Currently, the  $TC_{lda}$  are trained on student essays, while the  $TC_{pr}$  only works on the source article. However,  $TC_{attn}$  uses both student essays and the source article for TC generation. It might be hard to say that the superior performance of  $TC_{attn}$  is due to the neural architecture and attention scores rather than the richer training resources. Therefore, a comparison between  $TC_{attn}$  and a model that uses both student essays and the source article is needed.
We would like to show our appreciation to every member of the RTA group for sharing their pearls of wisdom with us. We are also immensely grateful to all members of the PETAL group and reviewers for their comments on an earlier version of the paper.
The research reported here was supported, in whole or in part, by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A160245 to the University of Pittsburgh. The opinions expressed are those of the authors and do not represent the views of the Institute or the U.S. Department of Education.
Dimitrios Alikaniotis, Helen Yannakoudakis, and Marek Rei. 2016. Automatic text scoring using neural networks. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 715-725.
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993-1022.
Chunshui Cao, Xianming Liu, Yi Yang, Yinan Yu, Jiang Wang, Zilei Wang, Yongzhen Huang, Liang Wang, Chang Huang, Wei Xu, et al. 2015. Look and think twice: Capturing top-down visual attention with feedback convolutional neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 2956-2964.
Richard Correnti, Lindsay Clare Matsumura, Laura Hamilton, and Elaine Wang. 2013. Assessing students' skills at writing analytically in response to texts. *The Elementary School Journal*, 114(2):142-177.
Scott A Crossley and Danielle S McNamara. 2016. Adaptive educational technologies for literacy instruction. Routledge.
8573