PMAES: Prompt-mapping Contrastive Learning for Cross-prompt Automated Essay Scoring
Year: 2023

Abstract:
Current cross-prompt automated essay scoring (AES) is a challenging task due to the large discrepancies between different prompts, such as different genres and expressions. The main goal of current cross-prompt AES systems is to learn enough shared features between the source and target prompts to grade well on the target prompt. However, because the features are captured based on the original prompt representation, they may be limited by being extracted directly between essays. In fact, when the representations of two prompts are more similar, we can gain more shared features between them. Based on this motivation, in this paper, we propose a learning strategy called "prompt-mapping" to learn about more consistent representations of source and target prompts. In this way, we can obtain more shared features between the two prompts and use them to better represent the essays for the target prompt. Experimental results on the ASAP++ dataset demonstrate the effectiveness of our method. We also design experiments in different settings to show that our method can be applied in different scenarios. Our code is available at https://github.com/gdufsnlp/PMAES.

heading: 1 Introduction
text: Automated Essay Scoring (AES) aims to evaluate the quality of essays automatically. Compared with human grading process, a robust AES system can not only reduce the work of teachers, but also improve the consistency of grading (Hearst, 2000;Weigle, 2002) and make it broadly available to language learners.
AES has been studied for many years. Early studies focus more on handcrafted features, such as lexical features (Rudner and Liang, 2002;Attali and Burstein, 2006;Yannakoudakis et al., 2011). With the rise of deep learning, many studies based on
∗ Corresponding author.
neural networks for prompt-specific settings have been proposed and achieved better results (Dong et al., 2017; Tay et al., 2018; Liao et al., 2021;Xie et al., 2022). These studies follow the same setting, that is, both rated training essays and unrated test essays belong to the same prompt.
Another type of work is cross-prompt AES. In this setting, labeled training essays are from source prompts and unlabeled test essays are from a different target prompt. Existing studies mainly focus on obtaining sufficient shared features between source and target prompts to grade the target prompt essays effectively. Some of them obtain shared features by extracting handcrafted features (Phandi et al., 2015; Ridley et al., 2020; Ridley et al., 2021) while others learn shared features by optimizing additional training objectives, such as the multi-task learning (Cummins et al., 2016), two-stage strategy (Jin et al., 2018; Li et al., 2020) and self-supervised learning task (Cao et al., 2020). Although these methods can effectively capture shared features between different prompts, we argue that these features are captured based on the original representations of the essays from source and target prompts. It may be limited by directly extracting the shared features among them.
Intuitively, when the representations of the essays from the source and target prompts are more consistent, they can share more knowledge between
1489
them. To this end, we propose a prompt representation learning framework for cross-prompt AES (PMAES) in which we design a prompt-mapping contrastive learning strategy to effectively learn about more consistent representations of source and target prompts. To do this, we design a mapping operation to project each essay from the source prompt to the target prompt and get its mapping representation specific to the target prompt. For each essay on the source prompt (let’s say rs), we first determine how similar it is to all the essays in the target prompt by their original representations (e.g., by taking the dot product with the inverse matrix of the representations of all the essays in the target prompt) as the weights of the rs to each target essay. Then, we employ a learnable parameter matrix (specifically, a prompt-mapping matrix) to acquire the weighted representation of the source prompt essay projected on the target prompt to express the mapping representation of the source essay rs (let’s say r̂s). These source essay representations and source mapping representations are treated as the source-to-target mapping pairs (rs, r̂s). By decreasing the distance between the essays in these mapping pairs, we may gradually reduce the discrepancy between the source and target prompts and finally make the representations of the two prompt essays more consistent. It is worth noting that the above description is about mapping from source to target. Naturally, we also perform target-to-source prompt mapping operations to further learn a more consistent representations of the two prompts, which will be described in Section 3.4.
As demonstrated in Figure 1, given the original essay representations of a source and a target prompt (which we marked in green and red, respectively), there are very few shared features between them under the original representations (which we marked in yellow). When we train the model using our proposed prompt-mapping approach, the representations of the two prompts may become more similar, which enables more shared features across the two prompts. We show them in Figure 1(b) and Figure 1(c). As the shared features increase, we can get more accurate representations of target prompt essays and grade them more accurately. To summarize, the main contributions of our work are as follows:
1) To the best of our knowledge, this is the first attempt to explore the learning of consistent rep-
resentations of different prompts by introducing a prompt-mapping learning strategy in order to obtain more shared features between the source and target prompts.
2) We conduct comprehensive experiments on the ASAP++ dataset, and the results show that our approach outperforms the state-of-the-art model on both single-overall and multi-attribute scoring tasks. Also, the prompt consistency experiments show that our method can make source and target prompts much more similar to each other.
3) We further design three types of source-target settings. The results show that our approach can be adapted to multiple scenarios.

heading: 2 Related Work

heading: 2.1 Prompt-specific AES
text: Prompt-specific AES aims to train and test essays on the same prompt. Early studies (Rudner and Liang, 2002;Attali and Burstein, 2006; Mohler and Mihalcea, 2009; Persing and Ng, 2013; Sakaguchi et al., 2015; Sultan et al., 2016) rate essays by extracting handcrafted features to train a machine learning model. Recently, with the rise of deep learning, a growing number of studies (Taghipour and Ng, 2016; Dong and Zhang, 2016; Dong et al., 2017; Dasgupta et al., 2018; Li et al., 2018; Tay et al., 2018; Uto et al., 2020; Hussein et al., 2020; Ma et al., 2021; Liao et al., 2021; Wang et al., 2022; Xie et al., 2022) propose scoring models based on neural networks and achieve promising results.

heading: 2.2 Cross-prompt AES
text: Cross-prompt AES aims to train models from labeled source prompt essays and rate target prompt essays. Phandi et al. (2015) train the Bayesian linear ridge regression algorithm from the source prompt using manual features, then test it directly on the target prompt. Cummins et al. (2016) adopt multi-task learning to address the problem of prompt adaptation. Jin et al. (2018) propose a twostage approach for the problem of cross-prompt AES. In the first stage, they train a RankSVM on prompt-independent features to obtain pseudolabels for target prompt essays. In the second stage, a neural network model learns more promptdependent features in the pseudo-labeled essays. Li et al. (2020) also adopts a two-stage approach to train a model to learn common knowledge and provide pseudo labels for target prompt essays in the first stage, then use a Siamese framework to
learn more prompt-dependent features in the second stage. Cao et al. (2020) train sentence reordering and noise identification tasks with adversarial training to improve the domain adaptability of the model. Ridley et al. (2020) utilize the handcrafted features to provide prompt agnostic information and achieve good results. Ridley et al. (2021) expand this prompt-agnostic information for multiattribute scoring tasks.

heading: 2.3 Contrastive Learning
text: Contrastive learning is an unsupervised learning method originally used in computer vision (Hadsell et al., 2006). The main idea is to gradually bring the anchor and its positive samples closer together in a shared semantic space while distinguishing the anchor from other samples, such as the work of Chen et al. (2020). Recently, contrastive learning has shown satisfactory results in textual representation learning. Data augmentation is a general strategy for obtaining positive samples, such as translation (Han et al., 2022), synonym replacement (Wang et al., 2021), word repetition (Wu et al., 2022) or textual representation perturbation (Gao et al., 2021; Yan et al., 2021).

heading: 3 Our Approach
text: The whole architecture of our approach is shown in Figure 2. It contains three components: shared encoder, scorer and prompt-mapping contrastive learning. The shared encoder provides a shared representation for the other two components, the scorer is used to predict the score, and the promptmapping contrastive learning is used to maximize the consistency of source and target prompts.

heading: 3.1 Task Definition
text: Given source prompt data Ds = {(xsi , ysi )}Pi=1 and target prompt data Dt = {xti}Qi=1, where x s/t i is the i-th essay in source/target prompt, P and Q are the number of essays in the source and target prompts. For single-overall scoring task, ysi is the overall score of source prompt essay xsi , and for multiattribute scoring task, ysi = {ys1i , ys2i , ..., ysKi } is the set of attribute scores, and ys1i is the overall score. The task of our approach is to train a model with Ds and Dt as inputs and output the score of all target prompt essays. The complete algorithm is shown in Algorithm 1.
Algorithm 1: Procedure of our approach
Input: {(xsi , ysi )}Pi=1, {xti}Qi=1 Output: shared encoder F , scorer G
1 Calculate Is and It using Eq. 14; 2 for sampling mini-batch do 3 rsi = F(xsi ), rti = F(xti); 4 Calculate r̂si and r̂ t i using Eq. 15; 5 Calculate Ls→t and Lt→s using Eq. 16 and Eq. 17 ; 6 Lpm = Ls→t + Lt→s; 7 if single-overall scoring task then 8 Calculate zsi using Eq. 5; 9 Calculate ŷsi using Eq. 6;
10 Calculate Laes_so using Eq. 7; 11 if epoch=1 then 12 Update F and G minimizing Laes_so;

heading: 13 else
text: 14 Update F and G minimizing
Lpm and Laes_so;
15 if multi-attribute scoring task then 16 Calculate {zski }Kk=1 using Eq. 8; 17 Calculate {ŷski }Kk=1 using Eq. 9; 18 Calculate Laes_ma using Eq. 10; 19 Calculate Lcor using Eq. 13; 20 Update F and G minimizing
Laes_ma, Lpm and Lcor;

heading: 3.2 Shared Encoder
text: To better encode essays, we use the hierarchical structure proposed by Dong et al. (2017) as a shared encoder, in which the sentence-level representation is extracted by CNN and attention pooling from words, and LSTM and another attention pooling are used to capture essay-level representation from all sentences. In this paper, as with Ridley et al. (2021), we use POS embedding1 to represent the essay text due to their ability to obtain better generalized representations. Suppose each essay is composed of n sentences, and each sentence contains m words. We use wi to denote the POS embedding of each word for convenience. Then, the sentence-level representation is captured by CNN with attention pooling:
ci = CNN([wi : wi+l−1]), i = 1, 2, ...,m (1)
st = attention([c1 : cm]) (2) 1We use the NLTK (http://www.nltk.org)
where l is the kernel size of CNN, ci is the output of the convolution operation applied to i-th POS embedding, and st is the representation of t-th sentence.
The essay-level representation is captured by LSTM with another attention pooling:
ht = LSTM(st−1, st), t = 1, 2, ..., n (3)
r = attention([h1 : hn]) (4)
where ht is the output of LSTM at the t-th time step, and r is the final essay representation.

heading: 3.3 Scorer
text: In this paper, we evaluate our approach both on single-overall scoring task and multi-attribute scoring task. Therefore, we have two types of scorers, corresponding to two forms of loss function. We also use the same handcrafted features as Ridley et al. (2021), denoted as f.

heading: 3.3.1 Single-overall Scorer
text: For single-overall scoring task, firstly, we concatenate the essay representation r and handcrafted features f, denoted as [r; f]. Then, feeding it into a tanh dense layer to get z. Finally, another dense layer with sigmoid activation is applied to predict the overall score ŷ. The corresponding equations are as follows (Eq. 5 and Eq. 6):
z = tanh(Wz[r; f] + bz) (5)
ŷ = σ(Wyz + by) (6)
where Wz and Wy are the trainable weight matrices, bz and by are the bias vectors, σ is the sigmoid function. We use mean squared error (MSE) as the loss function, defined as follows:
Laes_so = 1
N
N∑
i
(ŷi − yi)2 (7)
where N is the number of essays in a batch.

heading: 3.3.2 Multi-attribute Scorer
text: For multi-attribute scoring task, we first input the essay representation r into a specific relu dense layer to get the representation zk of the k-th attribute. Then, concatenating zk with f and feeding into a specific sigmoid dense layer to predict the kth attribute score ŷk. The corresponding equations are as follows (Eq. 8 and Eq. 9):
zk = relu(W kz r + b k z) (8)
ŷj = σ(W ky [z k; f] + bky) (9)
where W kz and W k y are the trainable weight matrices , bkz and b k y are the bias vectors. Suppose the total number of attributes is K, the multi-attribute scoring loss is defined as follows:
Laes_ma = 1
NK
N∑
i
K∑
k
(ŷki − yki )2 (10)
It should be noted that not all essays have all attributes (as shown in Table 5). So we use the mask mechanism proposed by Ridley et al. (2021)
to account for the attributes without gold scores when calculating the loss.
maskki = { 1, if yki ∈ yi 0, otherwise
(11)
yi = yi ⊗ maski , ŷi = ŷi ⊗ maski (12) In addition, we believe that when predicting one attribute score, the other attributes can provide useful information for it. Therefore, we propose an inter-attribute correlation loss Lcor.
Lcor = 1
K
N∑
i
K∑
k
− log( K∑
j,j ̸=k g(zki , z j i )) (13)
where g(zki , z j i ) = exp(cos(z k i , z j i )/ρ), cos(·) is the cosine similarity function, and ρ is a hyperparameter. The goal of Lcor is to maximize the mutual information among all attributes.

heading: 3.4 Prompt-mapping Contrastive Learning
text: In order to capture more shared features between the source and target prompts, we propose a prompt-mapping contrastive learning strategy to learn about more consistent representations of source and target prompts. For convenience, let’s take the source-to-target prompt mapping as an example to describe our method in detail. The targetto-source prompt mapping is the same operation.
Firstly, we use shared encoder F to encode all source and target prompt essays in training data to obtain the source prompt representation Is ∈ RP∗u and the target prompt representation It ∈ RQ∗u (as shown in Eq. 14), where u is the number of LSTM hidden units, P and Q are the number of source and target prompt essays.
Is = F({xsi}Pi=1), It = F({xti}Qi=1) (14)
Next, we will obtain source-to-target mapping pairs. First, we take each source essay representation, let’s say rsi , to dot product with I ⊤ t , where I⊤t ∈ Ru∗Q is the transpose of It, which is used to obtain how similar it is to all the essays in the target prompt as the weights of the rsi to each target prompt essay. After that, we use a learnable parameter matrix Ws ∈ RQ∗u to acquire the weighted representations of the source prompt essays projected on the target prompt to express the source mapping representation r̂si , as shown in Eq. 15. In this way, rsi and r̂ s i can form the source-to-target mapping pair (rsi , r̂ s i ).
Similarly, for the target-to-source mapping pairs, r̂ti can be obtained by using r t i , I ⊤ s ∈ Ru∗P and Wt ∈ RP∗u, and finally get the target-to-source mapping pair (rti , r̂ t i).
r̂si = Ws · (rsi ⊗ I⊤t ), r̂ti = Wt · (rti ⊗ I⊤s ) (15)
where ⊗ is the dot product operation. Finally, we take the mapping pairs (rsi , r̂ s i ) and (rti , r̂ t i) as the positive pairs. For the selection of negative samples, we follow the work of SimCLR (Chen et al., 2020) which takes the other samples in the same batch as the negative samples. The contrastive learning loss functions of mapping from source to target and from target to source are defined as follows:
Ls→t = Ns∑
i
− log f(r s i , r̂ s i )
Ns∑ j f(rsi , r s j ) + f(r s i , r̂ s j )
(16)
Lt→s = Nt∑
i
− log f(r t i , r̂ t i)
Nt∑ j f(rti , r t j) + f(r t i , r̂ t j)
(17)
where f(a, b) = exp(cos(a, b)/τ), cos(·) is cosine similarity function, τ is temperature hyperparameter, Ns and Nt are the batch size of source prompt essays and target prompt essays. The prompt-mapping contrastive learning loss is defined as:
Lpm = Ls→t + Lt→s (18) The total loss of single-overall scoring task is:
Lso = Laes_so + λ1Lpm (19)
The total loss of multi-attribute scoring task is:
Lma = Laes_ma + λ1Lpm + λ2Lcor (20)
where λ1 and λ2 are weighted hyper-parameters.

heading: 4 Experiments

heading: 4.1 Datasets and Evaluation Metrics
text: We conduct the experiments on the ASAP++ (Mathias and Bhattacharyya, 2018) dataset, which is an extension of the ASAP2 dataset. Each essay has an overall score and multiple attribute scores. The statistics are provided in Appendix A.
2https://www.kaggle.com/c/asap-aes/data
We use Quadratic Weighted Kappa (QWK) as the evaluation metric to measure the consistency between the real scores and the predicted scores, which is the general evaluation metric in AES tasks (Jin et al., 2018;Li et al., 2020;Ridley et al., 2021).

heading: 4.2 Implementation Details
text: We use the same data partition as the current stateof-the-art model (Ridley et al., 2021), that is for each prompt as target prompt, then the rest of prompts are set to be source prompt. For example, assume the target prompt is P8, then the source prompt consists of P1∼P7. We use labeled source prompt essays and unlabeled target prompt essays as training data, and the same unlabeled target prompt essays as test data. The validation data is from labeled source prompt essays.
We use the same handcrafted features proposed by (Ridley et al., 2020) in single-overall and multiattribute scoring task, including features of Lengthbased, Readability, Text Complexity, Text Variation and Sentiment. We use the length of the longest essay in the dataset as the padding length to ensure
that the essay information can be retained as much as possible. We use 50-dimension POS embedding as input and train all models for 50 epochs. We report the average results across five random seeds. More details are provided in Appendix B.

heading: 4.3 Baseline Models
text: We compare with the existing models on singleoverall scoring task and multi-attribute scoring task. For single-overall scoring task, we use Hi att (Dong et al., 2017) and PAES (Ridley et al., 2020) as baseline models, which are both the singleoverall scoring models. For multi-attribute scoring task, we use Hi att (Dong et al., 2017), AES aug (Hussein et al., 2020), PAES (Ridley et al., 2020), CTS no att (Ridley et al., 2021) and the current state-of-the-art model CTS (Ridley et al., 2021) as the comparison models. The details of baseline models are described as follow:
(1) Hi att: Dong et al. (2017) propose a hierarchical structure with attention pooling for singleoverall scoring task, which scores essays by extracting the sentence- and essay-level features.
(2) AES aug: Hussein et al. (2020) convert the model proposed by Taghipour and Ng (2016) into a multi-task architecture, which can be used to rate the multi-attribute scores at the same time.
(3) PAES: Ridley et al. (2020) apply a neural model with handcrafted features for single-overall scoring.
(4) CTS: Ridley et al. (2021) propose the first model for the cross-prompt multi-attribute scoring task, in which they develop a trait-attention mechanism to establish interactions between different attributes.
(5) CTS no att: This model (Ridley et al., 2021) has the same shared- and private-layers as CTS, and removes the trait-attention mechanism.

heading: 5 Results and Analysis

heading: 5.1 Main Results
text: We report the main results on single-overall scoring task and multi-attribute scoring task.
For single-overall scoring task, we use Hi att and PAES as baseline models, which are both singleoverall scoring models. As shown in Table 1, compared with Hi att and PAES, PMAES achieves the best results, improving the average QWK score by 21.3% and 2.9%, respectively, which proves the effectiveness of our approach on this task.
For multi-attribute scoring task, following Rid-
ley et al. (2021), we report the results from two dimensions. For the average QWK score across all attributes for each prompt (Table 1), we can see that our approach achieves 0.566 average QWK score, which outperforms all baseline models. For the average QWK score across all prompts for each attribute (Table 2), PMAES not only achieves the state-of-the-art average performance but also gets best performance on all prompts, which shows the significant improvement of PMAES for this task. Based on the above results, we can see that PMAES is suitable for both grading a single overall score and multiple attribute scores.
Meanwhile, we discover that PMAES fails to perform well in P4 and P6 as target prompts. Through analysis, we find that essays in P4 and P6 are source-dependent types and were written by 10th graders. Their writing requirements are relatively difficult. P4 requires students to write a response to figure out the source author’s thoughts, while P6 requires students to summarize academic excerpts. We believe that P4 and P6 share a few features with other prompts. In this case, the way our method maps P4/P6 and the source prompt to each other may lead to a low-scoring performance.

heading: 5.2 Ablation Studies
text: We conduct the ablation experiments both on single-overall scoring task and multi-attribute scor-
ing task, which are shown in Table 3 and Table 4. For single-overall scoring task, as shown in Table 3, we can see that if training model without Lpm, the average QWK score drops by 4.2%, and the QWK scores of the majority of prompts also drop significantly. Especially in P1 and P2, the QWK scores drop by 15.6% and 12.3%. It proves that our proposed prompt-mapping contrastive learning is effective in this task.
For multi-attribute scoring task, we also show the results from two dimensions. Firstly, as shown in Table 3, it can be seen that the average QWK score drops by 0.6% after removing Lcor and by 1.3% after removing Lpm, which demonstrates that both Lpm and Lcor contribute to improve the scoring performance, and Lpm contributes more. When we remove these two components (w/o Lpm & Lcor), the average QWK score drops by 1.7%. This shows that Lpm and Lcor can promote each other and further improve the scoring performance. Secondly, for the dimension of the average QWK score across all prompts for each attribute, we show the results in Table 4. The average QWK score drops by 0.7% after removing Lcor, by 1.4% after removing Lpm and by 1.8% after removing both components. It further demonstrates the effectiveness of our model. We also can see that when we remove both of them, the QWK scores drop on almost all attributes. Especially on Organization, after removing Lpm and Lcor, the QWK score drops significantly (by 4.9%).
Based on the above results, it can be found that our proposed approach can effectively improve the model scoring performance in the single-overall scoring task and the multi-attribute scoring task.

heading: 5.3 Analysis of Prompt Consistency
text: To further investigate the effectiveness of promptmapping contrastive learning on prompt consistency, we present our analysis using two methods: 1) Measuring the distance between source and target prompts using the Maximum Mean Discrepancy (MMD, Gretton et al., 2012). 2) Visualizing the essay representations of source and target prompts by using t-SNE (Van der Maaten and Hinton, 2008) to observe the degree of the consistency of prompts.

heading: 5.3.1 MMD for Prompt Consistency
text: Maximum Mean Discrepancy (MMD) is a kernelbased method that measures the distance between two matrices based on their respective mean embeddings. Inspired by previous work (Thota and
Leontidis, 2021; Yue et al., 2022), we quantify the degree of consistency by calculating the MMD distance between the source and target prompt essay representation matrices. A smaller distance indicates a greater degree of consistency between the source and target prompts, whereas a larger distance indicates a lesser degree of congruence. More details are provided in Appendix C

heading: 5.3.2 Visualization for Prompt Consistency
text: We use the t-SNE (Van der Maaten and Hinton, 2008) toolkit to visualize the representations of all essays on source and target prompts in training data to demonstrate prompt representations, which are generated by shared encoder under random initialization (original), training with PMAES w/o Lpm and PMAES, respectively.
Firstly, as shown in Figure 3(a) and Figure 3(b), we show the visualization results of source and target prompt essay representations with P1 and P2 as target prompts. Taking Figure 3(a) for example, we can see that a clear discrepancy exists in the original representations of source prompt (green) and target prompt (red). After training with PMAES w/o Lpm, the prompt representations become more discrete, while prompt representations generated by PMAES are undoubtedly more consistent and close to each other. The same phenomenon occurs in Figure 3(b).
Secondly, to further show how the prompt representations change as the number of training epochs increases, we visualize the essay representations generated by the epochs 0 (original), 4, 14, 34 and 50 during training w/o Lpm and PMAES with P1 as the target prompt. As shown in Table 4, the top row shows the results of training with PMAES w/o Lpm, and the bottom row shows the results of training with PMAES. The results show that the representations generated by these two models are relatively divergent at the beginning of training. As
the training epochs increase, PMAES makes the prompt representations gradually consistent, while PMAES w/o Lpm makes them gradually discrete.
Based on the results of MMD and visualization analysis, it can be seen that w/o Lpm not only fails to maintain the consistency of source and target prompts, but also damages it. In contrast, our approach can significantly make these two prompts more consistent to improve scoring performance.

heading: 5.4 Results of Different Source-target Settings
text: Most of the current cross-prompt AES studies train on multiple prompts (source prompt) and test on a single prompt (target prompt), namely the manyto-one setting, which is the general setting in crossprompt AES and is shown in Section 5.1. To verify the performance of our approach in many practical settings, we conduct comprehensive experiments for different source-target settings. More details are provided in Appendix D.

heading: 6 Conclusions
text: In this paper, we propose a new method for cross-prompt AES that aims to capture more shared features between the source and target prompts. Specifically, we design prompt-mapping contrastive learning to decrease the distance between the mapping pairs from source-to-target and target-to-source simultaneously and finally make the representations of the two prompts more consistent. Experimental results demonstrate that our approach achieves the state-of-the-art on both singleoverall scoring task and multi-attribute scoring task. We further design experiments for three sourcetarget settings, which proves that our approach can be adapted to multiple scenarios.
Limitations
Our approach achieves promising results in crossprompt AES by enhancing the consistency between source and target prompts. We believe that this idea can also be used to other cross-domain or domain adaptation tasks. In addition, as can be seen from Table 1, our approach fails to perform well in some cases. We think that forcing the representations of two prompts to be closer during model training may result in more errors when the prompts’ grading rubrics, writing genres, and writing requirements are quite different. Therefore, there are two possible directions can be explored for future research: 1) More fine-grained shared features can be extracted to improve scoring performance. 2) Scoreaware information can be integrated into model to improve source and target prompts consistency.

heading: Acknowledgements
text: This work is supported by the National Natural Science Foundation of China [grant number: 61976062].

heading: A Statistics of Datasets
text: The ASAP++ dataset includes 12,978 English writings in response to eight prompts. Table 5 displays the statistics for both ASAP and ASAP++.
B Implementation Details
The implementation details of our model are presented as follows:
For single-overall scoring task, we optimize only the Laes_so in the first epoch, which is used to initialize the model weights, and optimize the Laes_so and Lpm in the rest epochs. We set the kernel size as 3, the number of filters as 100 for CNN and the number of hidden units as 50 for LSTM. We use
Adam (Kingma and Ba, 2015) as the optimizer with the learning rate = 0.0001, τ = 0.1 and λ1 = 0.5. We use the model with the highest QWK score in the development set to evaluate the test set.
For multi-attribute scoring task, the detailed parameters are as follows: the kernel size is 5, the number of filters is 100 for CNN and the number of hidden units is 100 for LSTM. The optimizer is RMSprop (Dauphin et al., 2015) with the learning rate = 0.001, τ = 0.001, ρ = 0.1, λ1 = 0.5 and the λ2 = 0.1. We take the model with the highest average QWK score of all attributes in the development set to evaluate the test set.

heading: C MMD for Prompt Consistency
text: The MMD distance can be calculated by the following equation:
MMD = ∥∥∥∥∥∥ 1 P P∑
i=1
φ(rsi )− 1
Q
Q∑
j=1
φ(rtj) ∥∥∥∥∥∥ 2
H
(21)
where φ(·) denotes the function that is used to map the original variable to the Reproducing Kernel Hilbert Space (RKHS), P and Q are the number of source and target prompt essays in the training data, rsi and r t j are the representation of source and target prompt essays.
We take the essay representation matrices of source and target prompts generated by shared encoder to calculate the MMD distance. In order to better show the effectiveness of our proposed prompt-mapping contrastive learning in improving the consistency of source and target prompts, we use the shared encoding layer representations obtained at three settings: random initialization (original), training PMAES without Lpm (w/o Lpm), and training with PMAES. We show the results in Table 6. As can be seen, compared with PMAES, w/o Lpm leads to an increase in MMD distance, which indicates that the prompt consistency is bro-

heading: P1 - 0.526 | 0.598 0.457 | 0.552 0.533 | 0.560 0.557 | 0.673 0.423 | 0.517 0.701 | 0.733 0.344 | 0.405 0.506 | 0.577

heading: P2 0.354 | 0.450 - 0.192 | 0.426 0.325 | 0.485 0.210 | 0.434 0.144 | 0.269 0.222 | 0.451 0.488 | 0.552 0.276 | 0.438

heading: P3 0.428 | 0.780 0.222 | 0.620 - 0.652 | 0.658 0.772 | 0.747 0.613 | 0.626 0.576 | 0.709 0.087 | 0.297 0.479 | 0.634

heading: P4 0.436 | 0.742 0.220 | 0.542 0.639 | 0.656 - 0.745 | 0.735 0.635 | 0.629 0.601 | 0.532 0.153 | 0.348 0.490 | 0.598

heading: P5 0.540 | 0.742 0.323 | 0.570 0.563 | 0.621 0.614 | 0.628 - 0.598 | 0.608 0.634 | 0.641 0.141 | 0.271 0.488 | 0.583

heading: P6 0.655 | 0.592 0.438 | 0.558 0.396 | 0.505 0.406 | 0.575 0.448 | 0.535 - 0.477 | 0.407 0.320 | 0.565 0.449 | 0.534

heading: P7 0.666 | 0.667 0.500 | 0.612 0.490 | 0.507 0.457 | 0.534 0.535 | 0.509 0.396 | 0.346 - 0.427 | 0.562 0.496 | 0.534

heading: P8 0.408 | 0.416 0.313 | 0.466 0.404 | 0.441 0.459 | 0.502 0.062 | 0.155 0.029 | 0.099 0.390 | 0.497 - 0.295 | 0.368

heading: P1 - 0.371 | 0.483 0.477 | 0.553 0.529 | 0.531 0.608 | 0.659 0.470 | 0.513 0.736 | 0.731 0.362 | 0.421 0.507 | 0.556

heading: P2 0.516 | 0.598 - 0.200 | 0.420 0.316 | 0.497 0.239 | 0.400 0.121 | 0.273 0.217 | 0.460 0.516 | 0.549 0.304 | 0.457

heading: P3 0.458 | 0.782 0.382 | 0.519 - 0.656 | 0.657 0.758 | 0.759 0.597 | 0.633 0.599 | 0.716 0.088 | 0.265 0.506 | 0.619

heading: P4 0.513 | 0.717 0.309 | 0.482 0.591 | 0.638 - 0.749 | 0.742 0.604 | 0.616 0.598 | 0.531 0.164 | 0.346 0.504 | 0.582

heading: P5 0.424 | 0.750 0.275 | 0.606 0.583 | 0.627 0.608 | 0.637 - 0.599 | 0.612 0.601 | 0.555 0.113 | 0.325 0.458 | 0.588

heading: P6 0.665 | 0.719 0.454 | 0.534 0.386 | 0.579 0.466 | 0.621 0.459 | 0.609 - 0.466 | 0.503 0.334 | 0.374 0.461 | 0.563

heading: P7 0.633 | 0.660 0.461 | 0.607 0.485 | 0.452 0.460 | 0.505 0.510 | 0.512 0.463 | 0.343 - 0.428 | 0.574 0.491 | 0.522

heading: P8 0.405 | 0.452 0.447 | 0.217 0.308 | 0.385 0.246 | 0.486 0.198 | 0.172 0.077 | 0.192 0.423 | 0.451 - 0.301 | 0.336
text: ken. In contrast, PMAES can significantly reduce the MMD distance, which indicates that our approach is effective in improving prompt consistency. These results prove that our approach can effectively improve the consistency of source and target prompts.

heading: D Results of Different Source-target Settings
text: We argue that there are different situations may exist in practical settings. For example, source prompt and target prompt are all containing multiple prompts (namely many-to-many), source prompt contains only one prompt and target prompt contains multiple prompts (namely one-to-many), or source prompt and target prompt both contain only one prompt (namely one-to-one). To this end, we conduct comprehensive experiments for these settings to verify the performance of our approach in multiple scenarios.
D.1 Results of Many-to-many Setting
The experimental results of the many-to-many setting are shown in Table 7. For convenience, we design 6 source-target pairs for this setting. Since each prompt has its own score range, we calculate the QWK score for each prompt separately, and report the average QWK score of all prompts in target prompt. As shown in Table 7, PMAES outperform w/o Lpm in all source-target pairs with the QWK scores increase by 11.1%, 26.6%, 6.6%,
8.9%, 1.0% and 11.6%. The results demonstrate that our approach is suitable for many-to-many setting.
D.2 Results of One-to-many Setting Table 8 (top subtable) shows the experimental results of the One-to-many setting. Same as manyto-many setting, we also calculate the QWK score for each target prompt individually. In this setting, source prompt contains only one prompt, and target prompt consists of the remaining 7 prompts. Compared with w/o Lpm, the average QWK scores of PMAES increase by 7.1%, 16.2%, 15.5%, 10.8%, 9.5%, 8.5%, 3.8%, 7.3%, respectively. This proves that our approach is also remarkable in one-tomany setting.
D.3 Results of One-to-one Setting The experimental results of the one-to-one setting are shown in Table 8 (bottom subtable). For each prompt, we take each of the remaining 7 prompt as its corresponding target prompt to construct one-toone source-target pairs. We use "a|b" form to represent the performance of without using and using prompt-mapping contrastive learning, where "a" denotes QWK score of w/o Lpm and "b" denotes QWK score of PMAES. It can be observed that PMAES outperforms PMAES w/o Lpm in most source-target pairs. The average QWK score for each prompt as the source prompt are all improved, it can be demonstrated that our approach is stable and effective in one-to-one setting.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section after Conclusion
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section1
7 A4. Have you used AI writing assistants when working on this paper? No using AI writing assistants.
B 3 Did you use or create scientific artifacts? Section 4.1

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.1
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix A
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.1
C 7 Did you run computational experiments? Left blank.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.2

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3.2 and Section 4.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Therefore, there are two possible directions can be explored for future research: 1) More fine-grained shared features can be extracted to improve scoring performance. 2) Scoreaware information can be integrated into model to improve source and target prompts consistency