Aggregating Multiple Heuristic Signals as Supervision for Unsupervised Automated Essay Scoring
Year: 2023

Abstract:
Automated Essay Scoring (AES) aims to evaluate the quality score for input essays. In this work, we propose a novel unsupervised AES approach ULRA, which does not require groundtruth scores of essays for training. The core idea of our ULRA is to use multiple heuristic quality signals as the pseudo-groundtruth, and then train a neural AES model by learning from the aggregation of these quality signals. To aggregate these inconsistent quality signals into a unified supervision, we view the AES task as a ranking problem, and design a special Deep Pairwise Rank Aggregation (DPRA) loss for training. In the DPRA loss, we set a learnable confidence weight for each signal to address the conflicts among signals, and train the neural AES model in a pairwise way to disentangle the cascade effect among partialorder pairs. Experiments on eight prompts of ASPA dataset show that ULRA achieves the state-of-the-art performance compared with previous unsupervised methods in terms of both transductive and inductive settings. Further, our approach achieves comparable performance with many existing domain-adapted supervised models, showing the effectiveness of ULRA. The code is available at https: //github.com/tenvence/ulra.

heading: 1 Introduction
text: Automated Essay Scoring (AES) that aims to score the writing quality of essays without human intervention, is an important application of natural language processing in education. State-of-the-art AES models are typically trained in a supervised way with large labeled corpora, comprising essays and their groundtruth quality scores (Cozma et al., 2018; Ke and Ng, 2019; Kumar et al., 2022; Wang et al., 2022). However, collecting labeled essays is time-consuming and labor-intensive, especially for essays written specific to new prompts and when there is no professional scoring staff available.
∗ Corresponding author.
Unsupervised AES can get rid of the requirement of groundtruth scores for training, and thus has significant potential in both scientific research and practical applications. Its importance can be summarized in three key aspects: 1) Unsupervised AES models can handle special scenarios that lack labeling resource, such as the absence of professional scoring staff, the need for rapid essay scoring without timely labeled data, or the cold start scoring of an AES system without historical labeled data; 2) Unsupervised AES models can serve as pseudo-label generators or validators for essay scoring based on semi-supervised learning, few-shot learning, or transfer learning; 3) In practical writing tests, unsupervised AES models can rapidly provide a preliminary decision-making basis for scoring staff prior to scoring.
Early work tackles unsupervised AES by using the clustering method (Chen et al., 2010). To solve the problem of unordered clusters (i.e., cannot map clusters to ordinal scores), Chen et al. (2010) propose to use a heuristic quality signal the number of unique term in essay as the initial score of each essay, and then iteratively propagate the scores to other essays in the same cluster. However, such unsupervised clustering process is uncontrollable (i.e., there is no guarantee that clusters are generated towards to essay quality). Recently, researchers propose to use a heuristic quality signal word count as the weak supervision to train a neural AES model (Zhang and Litman, 2021). However, they demonstrate that directly regressing the predicted score to a real-valued quality signal (i.e., word count) leads to poor performance.
The above unsupervised AES methods provide a good idea that heuristic quality signals can be used as an alternative of groundtruth scores for model training, but have two major drawbacks. 1) Signal values are too noisy to be directly regressed to. Considering that the quality signal and groundtruth score may have completely different values but sim-
13999
ilar partial orders, it is better to utilize the partial orders in signal rather than the values. 2) Single signal is too weak to provide good supervision. Since a single quality signal cannot comprehensively describe the quality of essay, more quality signals should be introduced to bring stronger and more robust crowdsourcing-like supervision.
To this end, we propose a novel framework for Unsupervised AES by Learning from Rank Aggregation (ULRA). The core idea of our ULRA is to introduce multiple heuristic quality signals as the pseudo-groundtruth, and then train a neural AES model by learning from the aggregation of these quality signals. Specifically, our ULRA contains a Heuristic Essay Ranking (HER) module which views each signal as a ranking metric to generate multiple rank lists, and a Deep Pairwise Rank Aggregation (DPRA) module which can aggregate the inconsistent quality signals for model training. In the HER module, we introduce three types of classic quality signals for essay ranking. In the DPRA module, we set a learnable confidence weight for each signal to address the conflicts among signals, and train the neural AES model in a pairwise way to disentangle the cascade effect among partial-order pairs. We conduct experiments on eight prompts of ASAP dataset, which demonstrate that our proposed ULRA significantly outperforms previous unsupervised methods, and can even achieve comparable performance to many existing domain-adapted supervised methods.

heading: 2 Related Work
text: Automated Essay Scoring. Early supervised AES systems are developed with handcrafted features (Attali and Burstein, 2004; Phandi et al., 2015; Yannakoudakis et al., 2011). While with the development of deep learning, most of recent AES methods try to use neural model to solve the problem (Taghipour and Ng, 2016; Alikaniotis et al., 2016; Dong et al., 2017; Nadeem et al., 2019; Tay et al., 2018; Wang et al., 2018; Liu et al., 2019; Uto et al., 2020). These methods are effective but labor-intensive for labeling. To reduce the reliance on labels, the generic method (Attali et al., 2010), cross-prompt methods (Cao et al., 2020; Dong et al., 2017; Jin et al., 2018), and one-shot method (Jiang et al., 2021) are proposed. For unsupervised AES setting, Chen et al. (2010) propose a voting algorithm that iteratively updates the scores by the heuristic quality signals. Zhang and Litman
(2021) try to directly regress the predicted score to a real-valued quality signal, but achieve poor performance, because of the weak information of the quality feature. Overall, the effective unsupervised AES methods have not been widely explored.
Rank Aggregation. Rank aggregation (RA) is to aggregate multiple ranked list (i.e. base rankers) into one single list (i.e. aggregated ranker), which is intended to be more reliable than the base rankers (Deng et al., 2014). Many prior work have been proposed to effectively and efficiently solve the RA problem. These works can be roughly divided into three categories, which are the permutation-based methods (Mallows, 1957; Qin et al., 2010), the matrix factorization methods (Gleich and Lim, 2011), and the score-based probabilistic methods (Bradley and Terry, 1952; Luce, 2012; Pfeiffer et al., 2012; Thurstone, 1927; Chen et al., 2013). The scorebased methods are gradually developed, which usually predicts a score for each object based on the base rankers, and obtains the aggregated ranker based on the scores. Bradley-Terry (BT) model (Bradley and Terry, 1952) is an early work that is instructive for the following researches. It is proposed to model a probabilistic relationship between objects, according to the achieved scores, which is suitable for pairwise comparison. Then, Thurstone model (Thurstone, 1927) extends the BT model by assuming that the score for each object has a Gaussian distribution. Another important extension is Crowd-BT (Chen et al., 2013) model, which assigns a learnable weight for each base ranker, and optimizes the scores and the weights by active learning and online learning. Crowd-BT achieves competitive performance on the task of reading difficulty ranking, and directly inspires our work.

heading: 3 Problem Definition
text: We firstly introduce some notation and formalize the unsupervised AES problem. Let X = {xi}Ni=1 be a set of essays which are written to a prompt, and Y = {1, 2, · · · , L} be the pre-defined scores. For unsupervised AES, we are given a set of unlabeled essays D = {xi}NDi=1 ⊆ X for model training. The purpose of unsupervised AES is to train a model Fθ with parameters θ to predict the score of each essay xi ∈ T ⊆ X into the score set Y , by
ŷi = Fθ(xi;D) ∈ Y . (1)
In this paper, the model Fθ is a neural AES model, and we consider two settings of unsuper-
Final ScoreScoring Strategy
vised AES, transductive and inductive. For the transductive setting, the test set is just the training set T = D. For the inductive setting, the test set does not intersect the training set T ∩ D = ∅.

heading: 4 The ULRA Framework

heading: 4.1 An Overview of ULRA
text: Our ULRA framework involves two stages, model training and model inference. As shown in Figure 1, in the model training stage, the ULRA framework contains two modules: 1) Heuristic Essay Ranking module, which can generate partial-order pairs by ranking essays according to heuristic quality signals, and 2) Deep Pairwise Rank Aggregation module, which trains a neural AES model by aggregating the partial-order pairs derived from multiple quality signals into a unified supervision. In the model inference stage, considering that the essay scores predicted by the neural AES model may have a different range from the pre-defined score set Y , we propose a Scoring Strategy to transform the predicted scores given by the neural AES model into the range of the pre-defined score set. In addition, it should be noted that Figure 1 only shows the case of transductive setting, while for inductive setting, the trained neural AES model can be directly used to score unseen essays.

heading: 4.2 Heuristic Essay Ranking
text: As illustrated in Figure 1, the heuristic essay ranking module contains three components: quality signals, essay ranking, and partial-order pairs generation. Among them, multiple classic quality signals are introduced to describe the quality of essays from different aspects. Each quality signal can then be used to rank essays according to signal values and generate a rank list. Finally, each rank list can be transformed into many partial-order pairs for later model training.
Quality Signals. The quality signals are important to the ULRA framework, since it provides all supervision for model training. To obtain high-quality supervision, we investigate a lot of studies based on handcrafted quality signals (Lagakis and Demetriadis, 2021; Chen and He, 2013; Uto et al., 2020; Phandi et al., 2015). Considering that in practical unsupervised AES, there is no labeled data available as standard to select the most relevant signals, we just employ a set of quality signals designed in a classic work (Chen and He, 2013), which contains three aspects of signals, i.e., surface, preposition, and readability. In experiments, we find that not all of these signals are highly correlated with the groundtruth score. Moreover, for many of these signals, they may be highly correlated with the groundtruth score in one prompt but less correlated
in other prompts. Such quality signals with unstable supervision pose great challenges for the robustness of model training.
Essay Ranking. Compared with calculating the quality score of an essay based on its quality signals, it is easier to judge the relative quality of two essays based on their quality signals. Therefore, for each quality signal, we only reserve the partial-order relationship among essays by ranking the essays. Specifically, we rank the essays in a batch-wise way. Let B = {xi}NBi=1 denote an essay batch with NB essays, and R = {rk}Kk=1 denote K heuristic quality signals. As shown in Figure 1, we can calculate each quality signal rk on each essay xi to get a signal value gi,k = rk(xi), which can then be used for essay ranking. For the k-th quality signal, we can rank all essays xi based on their signal values gi,k to get a partial-order rank list xp1 ≻k · · · ≻k xpNB , where ≻k denotes the partial-order relation defined by the k-th quality signal and pj denotes the index of j-th essays in the rank list. Finally, we can get K rank lists {xp1 ≻k · · · ≻k xpNB} K k=1. Partial-Order Pairs Generation. Considering that only part of the partial-order information in each rank list is correct, we transform each rank list into a set of partial-order pairs, which allows the incorrect partial-order pairs to be corrected by other rank lists. Specifically, for a batch B, each rank list can be transformed into a total of NB(NB − 1)/2 partial-order pairs. Then, we can use a binary matrix with size of NB(NB − 1)/2×K to record the transformed partial-order pairs, that is,
M = [ 1xi≻kxj ]NB(NB−1) 2 ×K , (2)
where i ̸= j and 1 is an indicator function. M reflects the partial-order relationship between two essays in terms of different heuristic quality signals, and will be used as the supervision information to train the neural AES model in the next module.

heading: 4.3 Deep Pairwise Rank Aggregation
text: This module mainly deals with how to address the inconsistent partial-order supervision from multiple quality signals, so that the neural AES model can learn how to judge the partial-order relationship of essay quality. To address this problem, we design a deep pairwise rank aggregation loss, which set a learnable confidence weight for each signal to measure the importance of each signal.
Neural AES Model. We denote the neural AES model as Fθ(·) with learnable parameters θ. By feeding an essay xi into the model, we can get the predicted score si = Fθ(xi) ∈ R (not the final score) for the essay xi. The neural AES model consists of two components, an essay encoder which maps the essay into an essay embedding, and a fully-connected (fc) layer which maps the embedding into a predicted score.
Confidence Weight. Considering that two signals may provide opposite partial order for an essay pair, we expect to measure which one is more trustworthy. Therefore, we set a learnable confidence weight ηk for the k-th quality signal to measure its confidence. The learnable weight ηk can be defined as the probability that the partial-order information in the k-th rank list agrees with the groundtruth score. Inspired by Crowd-BT (Chen et al., 2013), we formalize ηk as
ηk ≡ P (xi ≻k xj | xi ≻ xj) , (3)
where xi ≻k xj and xi ≻ xj denote the partialorder relationship between two essays in the k-th rank list and the ground truth score, respectively. In ULRA, ηk is generated by applying sigmoid function on learnable parameters W = {c1 · · · , cK},
ηk = sigmoid(ck) ∈ [0, 1] , (4)
where ck is a learnable parameter and is optimized with model parameter θ together.
Deep Pairwise Rank Aggregation Loss. Based on the partial-order pairs derived from multiple signals and the confidence weight corresponding to each signal, we can define a special Deep Pairwise Rank Aggregation (DPRA) loss for model training.
Specifically, given an essay pair (xi, xj), we can use the neural AES model to get their predicted scores si and sj , respectively. Inspired by the Bradley-Terry model for paired comparisons (Bradley and Terry, 1952), we can define the predicted probability of xi ≻ xj as
P (xi ≻ xj) = sigmoid (si − sj) . (5)
If si ≫ sj , P(xi ≻ xj) tends to be 1; If si ≪ sj , P(xi ≻ xj) tends to be 0; While si = sj , P(xi ≻ xj) = 0.5.
To further get the predicted probability of the partial-order pair xi ≻k xj generated by the k-th
Algorithm 1: The Scoring Process of ULRA Input: A set of unlabeled essays D ⊆ X Output: The final score of each essay in T ⊆ X ▷ Training: learning from rank aggregation; Initialize a neural AES model; for each epoch do
for each batch of D do ▷ Heuristic Essay Ranking; Extract quality signals for each essay; Rank essays based on each signal; Generate partial-order pairs from rank lists; ▷ Deep Pairwise Rank Aggregation; for each partial-order pair do
Predict probability of partial-order pair; Calculate the pairwise loss by Eq.7;
end Optimize the neural AES model by Eq.8;
end end ▷ Inference: scoring essays by trained neural model; for each essay in T do
▷ Here T can just be D; Predict score of essay by the neural AES model; end Get final scores for all essays by the scoring strategy; return all final scores;
quality signal, we can make use of the confidence weights and apply the law of total probability as
P(xi ≻k xj) =P(xi ≻k xj | xi ≻ xj) · P(xi ≻ xj) + P(xi ≻k xj | xi ≺ xj) · P(xi ≺ xj) =P(xi ≻k xj | xi ≻ xj) · P(xi ≻ xj) + P(xj ≺k xi | xj ≻ xi) · P(xi ≺ xj) =ηk · P(xi ≻ xj) + (1− ηk) · P(xi ≺ xj) . (6)
Here, P(xi ≻k xj) is the predicted probability of xi ≻k xj , the label of which is 1xi≻kxj . Then, the loss function for si, sj , and ηk can be formulated as a negative log likelihood loss, which is
L(si, sj , ηk) = −1xi≻kxj log P(xi ≻k xj) . (7)
For each essay batch B, a set of NB(NB − 1)/2 partial-order pairs are obtained, which is denoted as SB. Based on the supervision of M, the loss function can be formulated as
L = ∑
(xi,xj)∈SB
K∑
k=1
−M(i,j),k · log P(xi ≻k xj) .
(8)

heading: 4.4 Scoring Strategy
text: Considering that the range of predicted score is not constrained during training process, the predicted score can be any real number. Therefore, we should
Prompt Numbers of Essays Genre Average Length Score Range

heading: 1 1783 ARG 350 [2, 12]

heading: CH .414 .399 .244 .409 .267 .212 .225 .413

heading: W .437 .432 .261 .401 .300 .208 .253 .330

heading: CO .140 .195 .115 .180 .111 .121 .073 .144

heading: UW .314 .375 .394 .471 .344 .323 .294 .221
text: cast the predicted score si of an essay xi into the range of the pre-defined score set Y = {1, · · · , L} to get the final scores ŷi ∈ Y . Here, we can get the final score ŷi of xi by min-max transformation[ (L− 1) si−min(s1,··· ,sN )max(s1,··· ,sN )−min(s1,··· ,sN ) ] + 1.

heading: 5 Experiments

heading: 5.1 Dataset and Evaluation Metric
text: Experiments are conducted on the Automated Student Assessment Prize1 (ASAP) dataset, which is widely used for the AES task. A total of 12,978 essays in ASAP are divided into 8 different sets, each of which corresponds to a prompt. The statistics of the dataset is shown in Table 1.
Quadratic Weighted Kappa (QWK) is adopted to be the evaluation metric. Specifically, given a score set Y = {1, · · · , L}, QWK is calculated to measure the automated predicted scores (Rater A)
1https://www.kaggle.com/c/asap-aes
and the resolved human scores (Rater B),
κ = 1− ∑
i,j wi,j ·Oi,j∑ i,j wi,j · Ei,j
(9)
where wi,j = (i − j)2/(L − 1)2 is the difference between scores of the raters, O is an L-by-L histogram matrix, Oi,j is the number of essays that received a score i by Rater A and a score j by Rater B, and E is the normalized outer product between each rater’s histogram vector of scores.

heading: 5.2 Implementation Details
text: Quality Signals Setting. The employed 20 quality signals, which are commonly used in some earlier work (Chen and He, 2013; Uto et al., 2020; Phandi et al., 2015), fall into following three categories:
• Surface Signals: character number (CH), word number (W), commas number (CO), and number
of unique words (UW);
• Preposition Signals: number of noun-plural words (NNP), number of determiner words (DT), number of noun-singular words (NN), number of adverb words (RB), number of adjective words (JJ), and number of preposition/subordinatingconjunction words (IN);
• Readability Signals: Gunning Fog (GF) index (Gunning, 1969), SMOG index (Mc Laughlin, 1969), RIX (Anderson, 1983), Dale-Chall (DC) index (Dale and Chall, 1948), wordtype number (WT), sentence number (S), number of long words (LW), number of complex words (CW), number of non-basic words (NBW), and number of difficult words (DW).
In Table 2, we demonstrate QWK between each signal and the groundtruth of each prompt. It indicates that single quality signal carry noisy partial-order
information of groundtruth scores, which results in poor performance.
Dataset Setting. For the transductive setting, the model is trained on the entire dataset (w/o labels), and is tested on the entire dataset, which means that all test essays have been seen during training. For the inductive setting, the dataset (w/o labels) is divided into the training set, the validation set, and the test set in a ratio of 6:2:2, which means that all test essays have not been seen during training. Due to the unsupervised setting, the validation set is useless and therefore discarded for ULRA.
Model Setting. The model is implemented by PyTroch (Paszke et al., 2019) and Higgingface Transformers (Wolf et al., 2020) libraries. BERT (Devlin et al., 2019) with pretained parameters bert-base-uncased is adopted as the essay encoder, whose hidden size is 712. The essay embedding is achieved by mean-pooling the token embeddings of BERT output. The fc layer of the neural AES model maps the essay embeddings into scalars. Each confidence weight ηk is initialized as 0.9.
Training. AdamW (Loshchilov and Hutter, 2017) is adopted as the optimizer, whose weight decay is set as 5e-4. The learning rates for the neural AES model and all ηk are 5e-5 and 5e-2, respectively. The batch size is set as 32. Our model is trained for 30 epochs, and the model which achieves minimum loss is selected to report the results.

heading: 5.3 Comparison Methods
text: We mainly compare our method with previous unsupervised AES methods, Signal Clustering (Chen et al., 2010) and Signal Regression (Zhang and Litman, 2021). Considering that they only employ one
quality signal as supervision, we extend them by introducing the 20 signals we used into their method. Four variants are tested: (1) averaged signal as supervision, (2) averaged output as prediction, (3) aggregated signal as supervision, and (4) aggregated output as prediction. Here, aggregated means that multiple rank lists are aggregated into one rank lists based on a rank aggregation algorithm (Chen et al., 2013). We also list two additional baselines, which directly apply the mean and maximum of the 20 quality signals as the predicted scores, respectively. Moreover, we list the performance of several stateof-the-art supervised methods (including general supervised, cross-prompt, and one-shot).

heading: 5.4 Performance Comparison
text: We can find that ULRA outperforms all unsupervised methods with a large improvement, and achieves the average QWK of 0.615 and 0.614 under transductive (Table 3) and inductive (Table 4) settings, respectively. It indicates that ULRA can perform well on both seen and unseen essay sets.
Compared with the cross-prompt and one-shot methods, we can find that ULRA achieves competitive performance, which is only 0.047 and 0.073 lower than that of the cross-prompt and one-shot methods, respectively. By observing the general supervised methods, we can find that the performance of ULRA is still much lower than theirs, due to the lack of strong supervision. But on some prompts, ULRA achieves comparable performance with the handcrafted features-based supervised method BLRR (e.g., prompt 1 and 3).
By observing the variants of two unsupervised methods, we can find that both unsupervised methods achieve improvements after introducing 20
quality signals. Among the four variants, two aggregated variants outperform two averaged variants. It indicates that the aggregation operation is better than the averaging operation, no matter as supervision or as prediction.

heading: 5.5 Ablation Study
text: We firstly study the effect of confidence weight ηk and neural model on the performance. As shown in Table 5, by replacing learnable ηk with fixed ηk = 1, the performance drops a lot. It indicates that the learnable ηk can address conflicts among inconsistent signals. The performance also drops a lot when using the non-pretrained encoder, or directly setting the essay scores si as learnable parameters. It indicates that a good essay encoder can make full use of the textual information of essays to improve scoring performance.
We then study the effect of signals on the performance by removing some types of signals from supervision. As show in Table 5, the performance drops by about 0.02 after removing one type, and continues to drop after further removing another one. It indicates that all three types of quality signals are useful for model training.
We finally study the effect of using the four vari-
ants on the performance. As shown in Table 5, using the same 20 signals, aggregating signals during training (i.e., ULRA) is superior to aggregating before training (i.e., averaged/aggregated signal as supervision) or during inference (i.e., averaged/aggregated output as prediction).

heading: 5.6 Model Analysis
text: Effect of More Unlabeled Essays. We study the impact of varying the number of unlabeled essays on the performance of ULRA, i.e., whether our ULRA requires numerous unlabeled essays to achieve a good enough performance. To this end, we vary the ratio of essays for training from 0.2 to 1.0 step by 0.2. As shown in Figure 2(a), we can find that the lines show a trend that goes up first and then keeps stable after the ratio about 0.6. It indicates that about 60% of unlabeled essays are sufficient to train a good enough ULRA model.
Effect of More Training Pairs. We study the impact of varying the number of training pairs on the performance of ULRA, i.e., whether our ULRA framework can benefit from more training pairs. To this end, we vary the batch size from 2 to 32, so that the number of training pairs in a batch will accordingly vary from 1 to 496. As shown in Figure 2(b), we can find that all the lines show a trend that goes up. It indicates that a larger number of training pairs can lead to better performance.
Effect of Weak Signals. We study the impact of weak signals (i.e., low correlation with groundtruth
scores) on the performance of ULRA. To this end, we add additional 0 to 80 weak signal(s) to the 20 quality signals. Note that the details can be seen in Appendix A. As shown in Figure 2(c), we can find that almost all lines show an overall downward trend. It indicates that weak signals will weaken the supervision and thus reduce the model performance. However, when between 0 and 10, all lines do not go down too much, and some lines even go up (e.g. prompt 1, 3, and 4). It indicates that ULRA is robust to weak signals if the weak signals are not yet dominant the signal set.
Effect of More Signals. We study the impact of the number of employed signals on the performance of ULRA, assuming that these signals have similar correlations with the groundtruth scores. To this end, we conduct experiments based on the bestN quality signals and the worst-N quality signals, according to QWKs with the groundtruth scores which are shown in Table 2. By varying N from 1 to 10, as shown in Figure 3, we can find that all the lines of best-N and most lines of worst-N show an upward trend. It indicates that more signals can often lead to better performance. By comparing the lines of best-N (with blue color) and worst-N (with red color), we can find that in most prompts, the performance differences between best-N and worst-N decrease with the growth of N . This may be because that more signals can help better address conflicts among signals, and therefore achieve more robust performance.
Effect of Confidence Weights. We study the impact of the learnable confident weights in ULRA. To this end, we calculate the Spearman’s correlation coefficient the learned confidence weights and the corresponding QWKs listed in Table 2 under each prompt. As shown in Table 6, we can find that they are highly correlated under both transductive and inductive settings, which indicate that the learned confidence weights can indeed reflect the
confidences of quality signals.
Effect of Different Scoring Strategies. We study the impact of different scoring strategies on the performance of ULRA. To this end, we test other four scoring strategies, which conduct score transformation based on predefined distributions. These distributions include groundtruth distribution (G), normal distribution (N), triangle distribution (T), and uniform distribution (U). Note that the details can be seen in Appendix B. As shown in Table 7, we can find that our scoring strategy outperforms all strategies except for the strategy based on groundtruth distribution. It indicates that our scoring strategy can adaptively learn the score distribution and ULRA can achieve better performance once the distribution of groundtruth score is known.
Groundtruth as Signal. In our ULRA framework, the applied 20 heuristic quality signals contain the information about score ranking, but also noise. We want to explore how the model would perform if the input signal contains no noise at all. Therefore, we conduct experiments by feeding the groundtruth scores as the signal in our ULRA. As shown in Table 8, the average QWK of our ULRA is 0.220 and 0.049 lower than which applies the groundtruth scores under the transductive and inductive settings. It indicates that the signals with less noise can help model to achieve better performance.

heading: 6 Conclusion
text: In this paper, we aim to perform essay scoring under the unsupervised setting. To this end, we propose a novel ULRA framework to train a neural AES model by aggregating the partial-order knowledge contained in multiple heuristic quality signals. To address the conflicts among different signals and get a unified supervision, we design a deep pairwise rank aggregation loss for model training. Experimental results demonstrate the effectiveness of ULRA for unsupervised essay scoring.
Limitations
Although our ULRA outperforms all unsupervised baseline methods, there are still some limitations.
The first limitation is that there is still a gap between the performance of our unsupervised method and that of some supervised methods. Although our ULRA can complete the AES task without label annotations, it is still worth exploring an unsupervised AES method whose performance is comparable to the state-of-the-art supervised method.
The second limitation is that the essay encoder which adopted in our ULRA (i.e., BERT) is pretrained on the English-based corpora, and the essays for training is also written by English. Thus, our ULRA works mostly for English, which means a well-trained ULRA model may fail to perform well on the essays written by other languages. An unsupervised AES system which supports multiple languages needs to be further explored.
The third limitation is that it requires about 25G GPU memory for training, which may fail on devices with small GPU memory. A possible solution is to set a smaller batch size, but this may take longer time. However, the evaluation process only requires about 2G GPU memory, which can run in most of GPU devices, or even CPU devices.

heading: Acknowledgements
text: This work is supported by National Natural Science Foundation of China under Grant Nos. 61972192, 62172208, 61906085, 41972111. This work is partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization.

heading: A Details of Weak Signals
text: In Section 5.6, we study the impact of using weak signals on the performance of our ULRA. We add additional 0 to 80 weak signals into the set of employed 20 quality signals. These 80 weak signals include: (1) mean of characters per word, (2) variance of characters per word, (3) mean of word per sentence, (4) variance of word per sentence, (5) mean of NNP per sentence, (6) number of PRP, (7) mean of PRP per sentence, (8) number of NNS, (9) mean of NNS per sentence, (10) number of VBZ, (11) mean of VBZ per sentence, (12) mean of DT per sentence, (13) mean of NN per sentence, (14) mean of RB per sentence, (15) number of POS, (16) mean of POS per sentence, (17) number of TO, (18) mean of TO per sentence, (19) number of VB, (20) mean of VB per sentence, (21) mean of JJ per sentence, (22) number of VBP, (23) mean of VBP per sentence, (24) mean of IN per sentence, (25) number of CC, (26) mean of CC per sentence,
(27) number of VBG, (28) mean of VBG per sentence, (29) number of VBN, (30) mean of VBN per sentence, (31) number of WP, (32) mean of WP per sentence, (33) number of MD, (34) mean of MD per sentence, (35) number of WRB, (36) mean of WRB per sentence, (37) number of VBD, (38) mean of VBD per sentence, (39) number of NNPS, (40) mean of NNPS per sentence, (41) number of CD, (42) mean of CD per sentence, (43) number of JJR, (44) mean of JJR per sentence, (45) number of JJS, (46) mean of JJS per sentence, (47) number of RBR, (48) mean of RBR per sentence, (49) number of RBS, (50) mean of RBS per sentence, (51) number of RP, (52) mean of RP per sentence, (53) number of EX, (54) mean of EX per sentence, (55) number of WDT, (56) mean of WDT per sentence, (57) number of UH, (58) mean of UH per sentence, (59) number of PDT, (60) mean of PDT per sentence, (61) number of LS, (62) mean of LS per sentence,(63) mean of clause per sentence, (64) mean of clause length, (65) number of maximum of clause per sentence, (66) mean of tree depth of sentences, (67) mean of average leaf depth of sentences, (68) number of error words, (69) ratio of stop words, (70) ratio of positive sentiment, (71) ratio of negative sentiment, (72) ratio of neutral sentiment, (73) ratio of compound sentiment, (74) Kincaid index, (75) ARI index, (76) Coleman-Liau index, (77) Flesch Reading Ease index, (78) LIX, (79) sentence beginnings with pronoun, and (80) sentence beginnings with preposition.
Through comparing Table 9 and Table 2, we can find that the 80 weak signals are less correlated with the groundtruth scores, compared with the 20 quality signals used in ULRA.

heading: B Details of Different Scoring Strategies
text: In Section 5.6, we study the impact of different scoring strategies on the performance of our ULRA. We test other four scoring strategies, which conduct score transformations based on predefined distributions (i.e., groundtruth distribution, normal
distribution, triangle distribution, and uniform distribution).
For the scoring strategy based on groundtruth distribution, we denote the distribution of the groundtruth labels in X as {a1, · · · , aL}, where ai is the number of essays with the score i ∈ Y . We fist rank all essays in X according to their predicted scores {si}Ni=1, and get a rank list of essays {xr1 , · · · , xrN }, where ri is the rank index. Finally, for each essay xri , if its ranking index ri satisfies ∑t−1 j=1 aj < ri ≤ ∑t j=1 aj for t ∈ Y , the corresponding final score ŷri is set as t. For the scoring strategy based on normal distribution, we first rank all essays in X according to their predicted scores {si}Ni=1, and get a rank list of essays {xr1 , · · · , xrN }, where ri is the rank index. Next, we use the normal distribution N (L−12 , 1) to calculate the proportion2 of samples in i-th final score to the total number of samples after the score transformation for all i ∈ Y , which is
ϕi = exp
( − ( i− 1− L− 1
2
)2 /2 ) , (10)
Φi = ϕi/
L∑
j=1
ϕj , (11)
where Φi is the proportion of samples in i-th final score to the total number of samples. Then, the sample number in i-th final score after the score transformation is
Ψi = ⌊NΦi⌋ , (12)
where ⌊·⌋ is the floor function. Finally, for each essay xri , if its ranking index ri satisfies ∑t−1 j=0Ψj <
ri ≤ ∑t
j=0Ψj for t ∈ Y , the corresponding final score ŷri is set as t. Note that we additionally define Φ0 = 0.
For the scoring strategy based on triangle distribution, we first rank all essays in X according to their predicted scores {si}Ni=1, and get a rank list of essays {xr1 , · · · , xrN }, where ri is the rank index. Then, we use the triangle distribution to calculate the proportion of samples in i-th final score to the total number of samples after the score transformation for all i ∈ Y , which is
ϕi = − ∣∣∣∣i− 1− L− 1 2 ∣∣∣∣+ L+ 1 2 . (13)
2Since the calculation is a proportion (e.g., Equation 11), the term 1/ √ 2π is removed for simplicity in Equation 10.
Same as the scoring strategy based on normal distribution, the sample number in i-th final score after the score transformation is
Ψi = ⌊Nϕi/ L∑
j=1
ϕj⌋ . (14)
Finally, for each essay xri , if its ranking index ri satisfies ∑t−1 j=0Ψj < ri ≤ ∑t j=0Ψj for t ∈ Y , the corresponding final score ŷri is set as t. Note that we additionally define Ψ0 = 0.
For the scoring strategy based on uniform distribution, we fist rank all essays in X according to their predicted scores {si}Ni=1, and get a rank list of essays {xr1 , · · · , xrN }, where ri is the rank index. The final score ŷri of xri is set as ⌊ LN ri⌋+ 1.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section Limitations
7 A2. Did you discuss any potential risks of your work? Our work provides methodological contributions that do not have direct boarder impacts. Although our work might indirectly lead to future researches and applications, it is premature to predict their positive or negative impacts.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 5.1
3 B1. Did you cite the creators of artifacts you used? Section 5.1
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section 5.1
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 5.1
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 5.1
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 5.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 5.1
C 3 Did you run computational experiments? Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 5.2 & Section Limitations
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5.2
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5.2
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Although our work might indirectly lead to future researches and applications, it is premature to predict their positive or negative impacts. 3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank. B 3 Did you use or create scientific artifacts? Section 5. 1
3 B1. Did you cite the creators of artifacts you used? Section 5. 1
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section 5. 1
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 5. 1
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 5. 1
3 B5. Did you provide documentation of the artifacts, e. g. , coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc. ? Section 5. 1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 5. 1
C 3 Did you run computational experiments? Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e. g. , GPU hours), and computing infrastructure used? Section 5. 2 & Section Limitations
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance. 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5. 2
3 C3. Did you report descriptive statistics about your results (e. g. , error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5. 2
3 C4. If you used existing packages (e. g. , for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e. g. , NLTK, Spacy, ROUGE, etc. )? Section 5. 2
D 7 Did you use human annotators (e. g. , crowdworkers) or research with human participants? Left blank. D1. Did you report the full text of instructions given to participants, including e. g. , screenshots, disclaimers of any risks to participants or annotators, etc. ? No response. D2. Did you report information about how you recruited (e. g. , crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e. g. , country of residence)? No response. D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response. D4