Distilling Calibrated Knowledge for Stance Detection
Year: 2023

Abstract:
Stance detection aims to determine the position of an author toward a target and provides insights into people’s views on controversial topics such as marijuana legalization. Despite recent progress in this task, most existing approaches use hard labels (one-hot vectors) during training, which ignores meaningful signals among categories offered by soft labels. In this work, we explore knowledge distillation for stance detection and present a comprehensive analysis. Our contributions are: 1) we propose to use knowledge distillation over multiple generations in which a student is taken as a new teacher to transfer knowledge to a new fresh student; 2) we propose a novel dynamic temperature scaling for knowledge distillation to calibrate teacher predictions in each generation step. Extensive results on three stance detection datasets show that knowledge distillation benefits stance detection and a teacher is able to transfer knowledge to a student more smoothly via calibrated guiding signals. We publicly release our code to facilitate future research.1

heading: 1 Introduction
text: The stance detection task aims to identify the position of a user toward a specific target (Mohammad et al., 2016b; Küçük and Can, 2020; AlDayel and Magdy, 2020). The target is usually a controversial topic (Stab et al., 2018; Glandt et al., 2021), a public figure (Sobhani et al., 2017; Li et al., 2021a) or a claim that could be a rumor’s post (Derczynski et al., 2017; Gorrell et al., 2019). For example, for the sentence in Table 1, we can infer that the author is against the marijuana legalization implied by the presence of the words “illegal drugs” and “disproportionate share of violence”. Even though impressive progress has been made in stance detection, most previous works rely on onehot annotation labels in which meaningful signals
1https://github.com/chuchun8/CKD
among different categories are ignored during training. Knowledge distillation (KD) (Hinton et al., 2015) transfers knowledge from a teacher model to a student model by training the student model to imitate the teacher’s prediction logits (which we call soft labels). Recent work has started to investigate knowledge distillation in the context of stance detection. Li et al. (2021b) evaluated knowledge distillation on stance detection datasets and proposed an adaptive knowledge distillation method (AKD) that applies less temperature scaling to the samples with larger confidence obtained from teacher predictions. However, the improvement brought by AKD could be limited if the teacher model is poorly calibrated (over-confident in its predictions).
Model miscalibration can be widely observed in modern neural networks (Guo et al., 2017; Yang and Song, 2021; Guo et al., 2021). If a model is well calibrated, then the probability associated with the predicted label should reflect its ground-truth correctness. According to our empirical observations, teacher models of AKD trained on stance detection datasets are not well-calibrated, producing peaked distributions of confidence in stance detection. Yang et al. (2019) showed that the teacher that provides less peaked training signals makes it possible for the student to learn better signals from interclass similarity and can potentially reduce overfitting. Therefore, we associate the performance of knowledge distillation with the calibration of teacher models in stance detection and propose to further improve the task performance by calibrating teacher predictions. Specifically, we recalibrate
6316
teacher predictions using a post-processing method called temperature scaling (Platt, 1999; Guo et al., 2017) and the student model is trained based on both hard labels and calibrated teacher predictions.
Further, born-again networks (Furlanello et al., 2018), in which the teacher and student models have identical model architectures, have achieved additional improvements with multiple students generations. At each consecutive step, a student model is taken as a new teacher to transfer knowledge to a new fresh student model. In this paper, we explore the born-again networks for stance detection and propose to calibrate teacher predictions in each generation. Extensive experiments on stance detection datasets show that a teacher can transfer knowledge to a student more smoothly via calibrated soft labels generated by the teacher and training student models over multiple generations helps improve the task performance.
Our contributions are summarized as follows:
• We investigate knowledge distillation in generations for stance detection and observe performance gains over multiple generations.
• We explore the connection between knowledge distillation and calibration in stance detection and propose a Calibration-based Knowledge Distillation method (which we call CKD) that dynamically updates the temperature used in knowledge distillation in each generation.
• Our CKD consistently outperforms strong baselines of stance detection, indicating that transferring knowledge from a well-calibrated teacher is more beneficial to the stance detection task.

heading: 2 Related Work
text: Previous works for stance detection mainly focus on the in-target setting (Mohammad et al., 2016b; Du et al., 2017; Sobhani et al., 2017; Siddiqua et al., 2019; Li and Caragea, 2019, 2021a) where the test target has always been seen in the training stage. Recently, cross-target stance detection (Augenstein et al., 2016; Xu et al., 2018; Zhang et al., 2020; Liang et al., 2021) and zero-shot stance detection (Allaway and McKeown, 2020; Allaway et al., 2021; Liu et al., 2021a; Liang et al., 2022a,b; Li et al., 2023) have also attracted a lot of attention. In this paper, we mainly focus on the in-target stance detection.
An ad-hoc training strategy that trains one model for one target has been widely used in previous works (Mohammad et al., 2017; Du et al., 2017; Sun et al., 2018; Wei et al., 2018; Li and Caragea, 2019, 2021b). However, it only considers one target during training and thus it fails to exploit the potential of all training data. Recent works (Schiller et al., 2021; Li and Caragea, 2021a) have shown that multi-target training that trains one model on all targets of a dataset can benefit the stance detection task. In our work, we adopt this multitarget training strategy and conduct extensive experiments in the multi-target training setting on three stance detection datasets (Stab et al., 2018; Glandt et al., 2021; Li et al., 2021a).
Knowledge distillation, initially proposed by Hinton et al. (2015), has been widely used in natural language processing to distill external knowledge into a model (Kim and Rush, 2016; Sun et al., 2019; Aguilar et al., 2020; Tong et al., 2020; Currey et al., 2020; Jiao et al., 2020; Hosseini and Caragea, 2021; Zhao and Caragea, 2021). Interestingly, despite that recent works (Furlanello et al., 2018; Clark et al., 2019; Yang et al., 2019; Mobahi et al., 2020; Liu et al., 2021b) have made impressive progress in knowledge distillation in which the teacher and student have the same model architecture, not much attention has been paid to using knowledge distillation for stance detection. One exception is the work by Li et al. (2021b) who proposed an adaptive knowledge distillation method (AKD) that applies instance-specific temperature scaling to the teacher predictions in one generation. In contrast to this prior work which uses only a single generation step for knowledge distillation, we explore born-again networks on three stance datasets and observe further improvements in performance of stance detection models when they are trained over multiple generations.
Motivated by recent advances in calibration of neural networks (Guo et al., 2017; Desai and Durrett, 2020; Guo et al., 2021; Park and Caragea, 2022a,b; Hosseini and Caragea, 2022), we hypothesize that the miscalibration of teacher predictions has a negative impact on the student model and further test this hypothesis by calibrating teacher’s predictions in born-again networks. In this work, we conclude that calibrating teacher’s predictions in each generation benefits the student model by providing smoother supervision signals in stance detection task.

heading: 3 Approach

heading: 3.1 Knowledge Distillation
text: Suppose a given training stance dataset of size n is Dtr = {(xi, yi)}ni=1, where xi includes an input sentence and a corresponding target and yi is the hard label. The goal of stance detection is to predict the stance label given an input sentence and a target.
Standard supervised learning aims to minimize the cross-entropy loss of training data:
LCE = ∑
(xi,yi)∈Dtr lCE(p(xi), yi) (1)
where lCE(.) represents the cross-entropy, p(xi) = σ(z(xi)) denotes softmax predictions of the model. In knowledge distillation, a student model is trained based on two signals: the hard labels and the teacher estimates, which are known as soft labels. Knowledge distillation (Hinton et al., 2015) transfers knowledge from a teacher model to a student model by minimizing LKD, which is the sum of the cross-entropy loss between the student predictions and hard labels and the distance loss between the predictions of the student and those of the teacher:
LKD = (1− λ)LCE + λLKL (2) LKL = ∑
xi∈Dtr lKL(p
τ s(xi), p τ t (xi)) (3)
where LKL is Kullback-Leibler (KL) divergence loss, and pτt (x) = σ(zt(x)/τ) and p τ s(x) = σ(zs(x)/τ) denote the softmax outputs of the teacher and student models (respectively) with temperature scaling τ , where τ is the fixed temperature used to scale the model predictions, σ(.) is the softmax function, zt(x) and zs(x) denote the output logits of the teacher and student models respectively; λ is a coefficient to balance the importance of the two loss functions and it could be optimized using teacher annealing (Clark et al., 2019)
that dynamically mixes the teacher prediction with the ground-truth label during training. The student model learns more from teacher predictions at the early training stage, and mostly relies on the ground-truth labels at the end of training.
Model performance can be further improved with multiple students generations (Furlanello et al., 2018) in which a student is taken as a new teacher to transfer knowledge to a new student. In this paper, we explore the knowledge distillation over multiple generations on stance detection (see Figure 1 for training over multiple generations).

heading: 3.2 Calibration of Neural Networks
text: Miscalibration can be widely observed in modern neural networks (Guo et al., 2017; Desai and Durrett, 2020). The prediction confidence of a wellcalibrated model is expected to reflect its classification accuracy. For example, given 100 model predictions, each with confidence of 0.7, we expect 70 of them to be correctly classified. One notion of miscalibration is Expected Calibration Error (ECE) (Naeini et al., 2015) that represents the difference in expectation between confidence and accuracy. ECE can be defined as:
ECE = E P̂ [|P(Ŷ = Y |P̂ = p)− p|] (4)
where Y and Ŷ denote the ground-truth label and the predicted label, respectively, P̂ is the output probability associated with the predicted label and p ∈ (0, 1) is a given confidence. As Eq. (4) cannot be estimated using a finite number of samples if P̂ is a continuous random variable, empirical approximations (Guo et al., 2017) are usually adopted by partitioning predictions into M bins of equal size and computing the weighted average:
ECE =
M∑
m=1
|Bm| n |acc(Bm)− conf(Bm)| (5)
where Bm denotes the set of samples whose prediction confidence falls into the interval (m−1M , m M ], M is the bin size, acc(Bm) and conf(Bm) are accuracy and average confidence of Bm, respectively.
Temperature scaling (Guo et al., 2017; Desai and Durrett, 2020) is an effective post-processing calibration method that produces calibrated probabilities. Given the logit vector z(x), the new confidence prediction is σ(z(x)/τ) after temperature scaling with temperature τ being usually fixed in knowledge distillation.

heading: 3.3 Calibrated Knowledge Distillation
text: In this paper, we propose an improved knowledge distillation method called CKD and show that transferring knowledge from a well-calibrated teacher improves the performance of students for stance detection.
The overall training procedure of CKD is shown in Algorithm 1. In the first step, a teacher model is trained from hard labels of stance detection dataset. In the second step, we do temperature scaling of teacher predictions but in our proposed approach, instead of using a fixed temperature τ that shows the best performance on the validation set, we dynamically choose the temperature τ that minimizes the ECE in each generation step (see Step 2 in Algorithm 1). Note that the selection of temperature in each generation is time-efficient as we simply need to divide the softmax teacher outputs by potential temperature values and then compute the ECE. Then, in each consecutive step, a new student model is initialized with an identical model architecture and trained based on both hard labels and calibrated teacher predictions. We expect the calibrated teacher predictions to help students learn better inter-class similarity and achieve better performance. The algorithm is also illustrated in Figure 1.
To summarize, CKD is different from previous adaptive knowledge distillation (Li et al., 2021b) of stance detection in the following aspects:
• We empirically connect the temperature of knowledge distillation with the calibration of distilled networks in CKD. • The temperature scaling hyperparameter τ is dynamically updated in each generation step.

heading: 4 Experimental Settings
text: In this section, we first describe the stance detection datasets used for evaluation and introduce the evaluation metrics. Then, we describe several stance
Algorithm 1: Calibrated-Knowledge Distillation with Dynamic Temp. Scaling
Require :Train set Dtr = {(xi, yi)}ni=1 Val set Dval
1 Train the first teacher model by minimizing the cross-entropy loss of Dtr
LCE = ∑
xi∈Dtr lCE(p(xi), yi)
2 Do temperature scaling of teacher predictions with updated τ of temperature scaling that minimizes the ECE of teacher predictions on the Dval 3 Train a student model by minimizing the sum of cross-entropy loss and the KL-divergence loss of Dtr
LKD = (1− λ)LCE + λLKL
4 Iterative training: Use the student as a new teacher and go back to step 2.
detection baselines that are used to be compared with our proposed method.

heading: 4.1 Datasets
text: Three stance detection datasets of diverse domains are used to evaluate the performance of the proposed method. Train, validation and test sets are as provided by the authors. Examples from these datasets are shown in Table 2 and summary statistics of these datasets are shown in Tables 3, 4, 5. Details of these datasets are described as follows.
AM AM (Stab et al., 2018) is an argument mining dataset containing eight topics: “Abortion”, “Cloning”, “Death Penalty”, “Gun Control”, “Marijuana Legalization”, “Minimum Wage”, “Nuclear Energy” and “School Uniforms”. The dataset is annotated for detecting whether an argument is in support of, neutral or opposed to a given topic.
COVID-19 COVID-19 (Glandt et al., 2021) is a stance detection dataset collected during COVID19 pandemic, which contains four targets “Face Mask”, “Fauci”, “Stay at Home Orders” and “School Closures”. The dataset is annotated for detecting whether the author is in favor of, neutral or against these topics.
P-Stance P-Stance (Li et al., 2021a) is a stance detection dataset collected during the 2020 U.S. presidential election, which contains three public figures “Donald Trump”, “Joe Biden” and “Bernie Sanders”. The dataset is annotated for detecting
whether the author is in favor of or against these presidential candidates during the election.

heading: 4.2 Evaluation Metrics
text: Similar to Mohammad et al. (2016a), macroaverage of F1-score (Fmacro) and micro-average of F1-score (Fmicro) are adopted to evaluate the performance of models. Favg is first calculated by averaging the F1-scores of label “Favor” and “Against”.2 We calculate the Favg for each target
2Note that we calculate the Favg and Fmicro by averaging the F1-scores of label “Favor”, “Against” and “Neutral” for COVID-19 dataset to be consistent with their results.
and Fmacro is calculated by averaging the Favg across all targets for each dataset. Further, we obtain Fmicro by averaging the F1-scores of “Favor” and “Against” across all targets for each dataset.

heading: 4.3 Baseline Methods
text: We run experiments with the following strong baselines of stance detection:
FNN: Feed-forward networks that take texts as inputs without considering the target information.
BiCE (Augenstein et al., 2016): A BiLSTM model that uses conditional encoding for stance detection. The topic is first encoded by a BiLSTM, whose hidden representations are then utilized to initialize a second BiLSTM with texts as inputs.
TAN (Du et al., 2017): TAN is an attentionbased LSTM model that learns the correlation between target and text representations.
CrossNet (Xu et al., 2018): CrossNet improves the BiCE by introducing a self-attention layer.
BERT (Devlin et al., 2019): A pre-trained language model that jointly encodes target and text, and predicts the stance by appending a linear layer to the hidden representation of the [CLS] token. We fine-tune BERT on the stance detection task.
TGA (Allaway and McKeown, 2020): A BERTbased model that exploits topic-grouped attention.
BERT-KE (Kawintiranon and Singh, 2021): A BERT model that is pre-trained with knowledge enhanced masked language modeling.
AKD (Li et al., 2021b): An adaptive knowledge distillation method that uses instance-specific temperature for stance detection.
In addition, we compare the CKD with the following knowledge distillation ablation methods:
KD-1: A vanilla knowledge distillation method without temperature scaling (τ = 1). The student has the same model architecture as the teacher (which is also known as self-distillation (Furlanello
et al., 2018; Zhang and Sabuncu, 2020)). KD-T: A knowledge distillation method with temperature τ . τ is chosen from {2, 3.5, 5} on the validation set. The student has the same model architecture as the teacher.
LSR (Szegedy et al., 2016): A label smoothing regularization technique used to encourage the base model to be less confident in making predictions.
The proposed method is listed as follows: CKD: A knowledge distillation method with temperature τ . Unlike KD-T where τ is fixed, in our proposed method τ is dynamically chosen from 1 to 5 with a step size of 0.01 to minimize the ECE in each generation (step 2 of Algorithm 1). The pretrained BERTweet (Nguyen et al., 2020) is used as teacher and student models for CKD and knowledge distillation methods. The student has the same model architecture as the teacher. We use the default bin size of 10 to compute ECE and we report the performance of using different bin sizes in the next section.
Note that unlike CKD, temperature τ of KD-T is chosen out of three options because it is impractical for KD-T to select the temperature in the range of 1 to 5 with step size 0.01, which requires to repeat the whole training procedure hundreds of times. Since temperature scaling will not change the prediction label, we can only tune the temperature according to the classification performance of student model on the validation set for KD-T, which is one of weaknesses of previous born-again networks. In this paper, we propose to select the temperature according to the ECE, which can be simply calculated with each teacher model alone (no training involved in temperature selection step) and thus it is much more time-efficient.
We adopt the teacher annealing (Clark et al., 2019) for all knowledge distillation methods in our experiments. We performed all experiments on a single NVIDIA A5000 GPU. We implement our model in PyTorch framework (Paszke et al., 2019) using HuggingFace Transformers library (Wolf et al., 2020). More details of the hyperparameters are described in Appendix A.

heading: 5 Results
text: In this section, we first compare the proposed CKD with strong baselines of stance detection. Then we present an ablation study to show that dynamically updating the temperature helps improve the performance in generations. In addition, we compare
the performance of different bin sizes of our proposed CKD on stance detection datasets. Next, we compare our CKD with the best baseline AKD and visualize the relationship between the confidence and the evaluation metric (Fmicro) in the form of reliability diagrams. At last, we discuss the performance of our CKD and other KD baselines in ECE on stance detection datasets.
Main results Table 6 shows performance comparisons of CKD with the stance detection baselines on all three stance detection datasets. We can observe that our proposed CKD performs best in overall. Specifically, the best student model of CKD outperforms the best-performing baseline AKD by 1.86%, 2.10% and 0.36% in Fmicro on COVID-19, AM, and P-Stance datasets, respectively. A more detailed comparison between our CKD and AKD is presented later in this section. Note that CKD shows less improvements on PStance dataset. One explanation is that P-Stance has much larger train set for targets and the effect of knowledge distillation diminishes with increasing the size of train set—a fact that was observed before (Zhang and Sabuncu, 2020).
Ablation study Table 7 shows the comparison results of our proposed method with the ablation methods of fixed temperature or no temperature (τ = 1) mentioned above on all three stance detection datasets. We train born-again networks sequentially with multiple generations. Gen0 and Gen1 columns show the performance of the first teacher (i.e., the BERTweet model in Table 6) and student, respectively. Then the first student is taken as a new teacher to transfer knowledge to the second student and results of the second student are shown
in Gen2 column. We can observe that both CKD and baselines achieve superior performance over the first teacher model (Gen0) on COVID-19 and AM, demonstrating the effectiveness of born-again networks on stance detection. Moreover, our proposed CKD consistently outperforms the distillation baselines on almost all stance detection datasets in different generations, which indicates that dynamically updating τ according to ECE as compared with a fixed τ yields better performance and a well-calibrated teacher can help train a better student.
Bin size We evaluate the proposed CKD using different bin sizes. Table 8 shows performance comparisons of CKD using different bin sizes on stance datasets. We can see that CKD models with different bin sizes outperform the best distillation baseline KD-T in most cases, demonstrating the ef-
fectiveness of the proposed method. Moreover, we observe that the default bin size of 10 achieves the best performance on all stance datasets in overall.
CKD vs. AKD First, we compare CKD with AKD in terms of Fmicro and Fmacro in Table 9. AKD (Li et al., 2021b) adopts an instance-specific temperature scaling strategy and is trained in only one generation. Here, we further extend AKD over multiple generations. First, we see that AKD achieves better performance with more generations, reinforcing the claim that training student models in multiple generations can improve the task performance. Second, our CKD shows superior performance over AKD on most generations for each dataset, which indicates that a well-calibrated teacher contributes more to stance detection task in generations.
Second, we compare reliability diagrams of our CKD and AKD on COVID-19, AM, and P-Stance datasets in Figures 2 and 3. Reliability diagrams are a visual representation of model calibration (Niculescu-Mizil and Caruana, 2005; Guo et al., 2017). The prediction space is usually discretized
into ten bins and the mean predicted value (i.e., confidence) is plotted against the expected sample accuracy in each bin. If a model is perfectly calibrated, then acc(Bm) will be equal to conf(Bm) for all m ∈ {1, ...,M}. We can see that AKD is over-confident in its predictions on all datasets, with large gap areas in Figure 3. We then can observe that our CKD produces more calibrated predictions on COVID-19, AM and P-Stance datasets, verifying the calibration effects of temperature scaling. Based on these observations, we conclude that a student learns better from calibrated teacher predictions that provide less peaked supervision signals in each generation.
ECE results In order to better understand the role of calibration in knowledge distillation in generations, we show the comparison of experimental results (ECE) on three stance detection datasets in Table 10. For each model, we report the performance of the teacher that helps train the best student on each dataset. We can observe that our proposed CKD shows the best performance, achieving the lowest ECE on each dataset. Moreover, the best student of our CKD achieves the highest microaveraged F1 on all datasets, as shown in Table 7, which indicates that calibrating teacher predictions can benefit the stance detection task.

heading: 6 Conclusion
text: In this paper, we study the problem of knowledge distillation in generations on stance detection. We show that knowledge distillation in multiple generations can be beneficial to stance detection. Moreover, based on the existing works, we provide a new perspective that a well-calibrated teacher can benefit the student by providing smoother training signals and make it possible for the student to learn from inter-class similarity. Our proposed CKD produces calibrated teacher predictions by dynamically updating the temperature used for scaling in each generation. Experimental results show that our proposed method consistently outperforms the best-performing baseline on different stance detection datasets. Future work includes extending our proposed method to a broader range of NLP tasks such as emotion classification.

heading: 7 Limitations
text: One limitation of our method is that it requires multiple generations to achieve the best performance on stance detection datasets. While the best student model significantly outperforms strong baselines, it takes longer training time and requires extra memory for the teacher model. This is a common limitation for knowledge distillation in generations. Another limitation of our method is that the improvements brought by knowledge distillation saturate after a few generations, which can be also observed in previous work. We will explore how to improve the performance saturation in the future.

heading: 8 Ethical Considerations
text: Beyond the proposed method that helps correctly identify the stance towards specific targets, it is very important to consider the ethical implications of stance detection systems. Since stance detection systems could automatically collect and aggregate the topical stance for a specific target, these systems may have significant impact on decision-making. Algorithms are not perfect, and thus a potential harm is that these systems may make incorrect predictions and further mislead the decision-making. Researchers should be aware of potential harms from the misuse of stance detection systems, and should respect people’s privacy during the data collection.

heading: Acknowledgments
text: We thank the National Science Foundation for support from grants IIS-1912887, IIS-2107487, and ITE-2137846 which supported the research and the computation in this study. We also thank our reviewers for their insightful feedback and comments.

heading: A Hyperparameters
text: For each non-BERT model, AdamW optimizer (Loshchilov and Hutter, 2019) is used with a learning rate of 1e-3, and gradient clipping if the norm of the gradients exceeds 1. Each model is trained for 30 epochs, with a mini-batch size of 128 in each iteration. A dropout of 0.5 is used after the embedding layer. The dimension of feed-forward
layers is 300 and the hidden dimension of LSTM is 300 for TAN, BiCE and CrossNet.
For each BERT-based model, AdamW optimizer is used with a learning rate of 2e-5, and gradient clipping if the norm of the gradients exceeds 1. Each model is fine-tuned for 5 epochs, with a minibatch size of 32.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitation Section after the Conclusion.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethical Considerations Section after the Conclusion.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 4.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We discussed our usage of baseline models and previous datasets in Section 4.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.
C 3 Did you run computational experiments? Section 5.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We will explore how to improve the performance saturation in the future