Revisit Few-shot Intent Classification with PLMs: Direct Fine-tuning vs. Continual Pre-training
Year: 2023

Abstract:
We consider the task of few-shot intent detection, which involves training a deep learning model to classify utterances based on their underlying intents using only a small amount of labeled data. The current approach to address this problem is through continual pretraining, i.e., fine-tuning pre-trained language models (PLMs) on external resources (e.g., conversational corpora, public intent detection datasets, or natural language understanding datasets) before using them as utterance encoders for training an intent classifier. In this paper, we show that continual pre-training may not be essential, since the overfitting problem of PLMs on this task may not be as serious as expected. Specifically, we find that directly fine-tuning PLMs on only a handful of labeled examples already yields decent results compared to methods that employ continual pre-training, and the performance gap diminishes rapidly as the number of labeled data increases. To maximize the utilization of the limited available data, we propose a context augmentation method and leverage sequential self-distillation to boost performance. Comprehensive experiments on real-world benchmarks show that given only two or more labeled samples per class, direct fine-tuning outperforms many strong baselines that utilize external data sources for continual pre-training. The code can be found at https://github.com/ hdzhang-code/DFTPlus.

heading: 1 Introduction
text: Intent detection is a critical module in task-oriented dialogue systems. The target is to classify utterances according to user intents. Recent progress in intent detection highly relies on deep models and datasets with well-crafted annotations. Using largescale models or datasets has been recognized as a de facto recipe for many tasks in natural language
∗ Corresponding author.
processing (NLP) including intent detection . However, large training datasets are often not available due to the cost of labeling. Therefore, few-shot intent detection, which aims to train a classifier with only a few labeled examples, has attracted considerable attention in recent years (Dopierre et al., 2021; Zhang et al., 2022; Mi et al., 2022).
The main obstacle for few-shot learning is commonly believed to be overfitting, i.e. the model trained with only a few examples tends to overfit to the training data and perform much worse on test data (Vinyals et al., 2016; Zhang et al., 2022). To alleviate the problem, the mainstream approach is to transfer knowledge from external resources such as another labeled dataset, which has been widely used for few-shot image classification (FeiFei et al., 2006; Snell et al., 2017) and few-shot intent detection (Yu et al., 2018; Geng et al., 2019; Nguyen et al., 2020).
Since recently emerged large-scale pre-trained language models (PLMs) have achieved great success in various NLP tasks, most recent few-shot intent detection methods propose to fine-tune PLMs on external resources before applying them on the target task, which is known as continual pretraining (Gururangan et al., 2020; Ye et al., 2021), as illustrated in Fig 1. The external resources utilized for continual pre-training include conversational corpus (Wu et al., 2020a; Mehri et al., 2020; Vulić et al., 2021), natural language understanding datasets (Zhang et al., 2020a), public intent detection datasets (Zhang et al., 2021a; Yu et al., 2021), and paraphrase corpus (Ma et al., 2022). While these methods have achieved state-of-the-
11105
art results, the use of external training corpora induces extra data processing effort (e.g., SBERTParaphrase Ma et al. (2022) uses 83 million sentence pairs from 12 datasets) as well as model bias (e.g., the trained model may be biased to the intent classes used in continual pre-training) (Xia et al., 2020b, 2021a; Nguyen et al., 2020).
It is commonly believed that directly fine-tuning PLMs with a small amount of data may generate unacceptable variance (Lee et al., 2020; Dodge et al., 2020). However, it has been recently found that the instability may be caused by incorrect use of optimizer and insufficient training (Mosbach et al., 2021; Zhang et al., 2020c). Further, some studies (Hao et al., 2019; Li et al., 2019) have revealed that in sentiment analysis and paraphase detection tasks, when directly fine-tuned with a small dataset, PLMs sush as BERT (Devlin et al., 2019) demonstrate a certain level of resilience to overfitting. Therefore, a thorough investigation is needed to explore the direct fine-tuning of PLMs for few-shot intent detection. In this work, we make the following contributions:
• We take an empirical investigation into the overfitting issue when directly fine-tuning PLMs on few-shot intent detection tasks. Our study suggests that overfitting may not be a significant concern, since the test performance improves rapidly as the size of training data increases. Further, the model’s performance does not degrade as training continues. It implies that early stopping is not necessary, which is often employed to prevent overfitting
in few-shot learning and requires an additional set of labeled data for validation.
• We find that direct fine-tuning (DFT) already yields decent results compared with continual pre-training methods. We further devise a DFT++ framework to fully exploit the given few labeled data and boost the performance. DFT++ introduces a novel context augmentation mechanism by using a generative PLM to generate contextually relevant unlabeled data to enable better adaptation to target data distribution, as well as a sequential self-distillation mechanism to exploit the multi-view structure in data. A comprehensive evaluation shows that DFT++ outperforms state-of-the-art continual pre-training methods with only the few labeled data provided for the task, without resorting to external training corpora.

heading: 2 Direct Fine-tuning
text: We investigate a straightforward approach for fewshot intent detection – directly fine-tuning (DFT) PLMs with the few-shot data at hand. However, it is a common belief that such a process may lead to severe overfitting. Before going into detail, we first formally define the problem.

heading: 2.1 Problem Formulation
text: Few-shot intent detection aims to train an intent classifier with only a small labeled dataset D = {(xi, yi)}N , where N is the dataset size, xi denotes the ith utterance, and yi is the label. The number of
samples per label is typically less than 10. We follow the standard practice (Sun et al., 2019; Zhang et al., 2021a) to apply a linear classifier on top of the utterance representations:
p(y|hi) = softmax (Whi + b) ∈ RL, (1)
where hi ∈ Rd is the representation of the ith utterance in D, W ∈ RL×d and b ∈ RL are the parameters of the linear layer, and L is the number of classes. We use the representation of the [CLS] token as the utterance embedding hi. The model parameters θ = {ϕ,W,b}, with ϕ being the parameters of the PLM, are trained on D. We use a cross-entropy loss Lce (·) to learn the model parameters:
θ = argmin θ
Lce (D; θ) . (2)
Unlike the popular approach of continual pretraining (Zhang et al., 2020a, 2022, 2021b), DFT fine-tunes PLMs directly on the few-shot data, which may experience overfitting, leading to suboptimal performance. To examine this issue, we conduct the following experiments.

heading: 2.2 Experiments
text: Datasets We utilize four large-scale practical datasets. HINT3 (Arora et al., 2020b) is created from live chatbots with 51 intents. BANKING77 (Casanueva et al., 2020) is a fine-trained dataset focusing on banking services, containing 77 intents. MCID (Arora et al., 2020a) is a crosslingual dataset for “Covid-19” with 16 intents, and we use the English version only. HWU64 (Liu et al., 2019a) is a large-scale multi-domain dataset with 64 intents. The statistics of the datasets are given in Table 1. To simulate few-shot scenarios, we randomly sample K samples per label from the training set of each dataset to form the dataset D.
Baselines To evaluate DFT, we compare it against IsoIntentBERT (Zhang et al., 2022), a competitive baseline applying continual pre-training
with public intent detection datasets. We follow the original work to pre-train BERT on OOS (Larson et al., 2019), a multi-domain public intent detection dataset containing diverse semantics, and then perform in-task fine-tuning on the small dataset D.
Results and Findings We plot the learning curves of DFT in Fig. 3, where the following observations can be drawn. First, comparing the results in 1-shot and 5-shot scenarios, the test performance of DFT improves drastically as the number of labeled examples rises from 1 to 5, leading to a fast reduction in the performance gap between the training and test performance. Second, the test performance does not deteriorate as the training progresses, and the learning curves exhibit a flat trend. These observations are consistent across various datasets and different models (BERT and
RoBERTa), including both 1-shot and 5-shot1 scenarios. The observations also align with previous findings in sentiment analysis (Li et al., 2019) and paraphrase detection (Hao et al., 2019) tasks.
The flat learning curves indicate that early stopping is not necessary, which is often used to prevent overfitting and requires an additional set of labeled data. This is important for practitioners because model selection has been identified as a roadblock for true few-shot learning (Perez et al., 2021), where the labeled data is so limited that it is not worth setting aside a portion of it for early stopping. On the other hand, the rapidly reduced performance gap between DFT and IsoIntentBERT (Fig. 4) casts doubt on the necessity of continual pre-training. Thus, we raise an intriguing question:
• With only the given few labeled data, is it possible to achieve comparable or better performance than continual pre-training methods?
Our attempt to answer the question leads to DFT++, a framework designed to fully exploit the given few labeled data, which provides an affirmative answer.

heading: 3 Push the Limit of Direct Fine-Tuning
text: To push the limit of few-shot intent detection with only a few labeled data at hand and without using any external training corpora, DFT++ introduces two mechanisms, as shown in Fig. 2. The first is a novel context augmentation mechanism, wherein the few data are used to prompt a generative PLM to generate contextually relevant unlabeled utterances to better model target data distribution. The second is a sequential self-distillation mechanism.

heading: 3.1 Context Augmentation
text: Unlike continual pre-training methods that leverage external training corpora, we use the few data to solicit knowledge from generative PLMs. An intuitive way is data augmentation, which prompts the model to generate new utterances with the given intent class. However, as suggested by Sahu et al. (2022) and our analysis (Section 3.4), data augmentation for intent detection with tens of intent classes is challenging. Hence, we propose to exploit contextual relevance in an unsupervised manner instead. Specifically, for each intent class, we compose the few data into a prompt and then feed
1We observe the same patterns when further increasing the shot number beyond 5.
it to GPT-J (Wang and Komatsuzaki, 2021), a powerful generative PLM, to generate novel unlabeled utterances. Fig. 5 gives an example of the prompt and generated results. The generated unlabeled data is combined with the given utterances in D to compose a corpus Daug = {xi}i, which can be used for masked language modeling (MLM). Hence, the model parameters θ are learned by simultaneously minimizing both the cross-entropy loss Lce and the MLM loss Lmlm:
θ = argmin θ
( Lce(D; θ) + λLmlm(Daug; θ) ) ,
(3) where λ is a balancing parameter.
Notice that there is a critical difference between the proposed context augmentation and conventional data augmentation methods. Context augmentation generates contextually relevant data (i.e., utterances with similar context to the given input but not necessarily belong to the same label class), and we use the generated data in an unsupervised manner via MLM. In contrast, conventional data augmentation methods generate new utterances with the same label as the given utterance and utilize them in a supervised manner.

heading: 3.2 Sequential Self-distillation
text: To further boost performance, we employ selfdistillation (Mobahi et al., 2020; Allen-Zhu and Li, 2020) (Fig. 2). The knowledge in the learned model is distilled into another model with the same
architecture by matching their output logits2:
θk = argmin θk
KL (
f (D; θk) t , f (D; θk−1) t
) , (4)
where KL(·) is the Kullback-Leibler (KL) divergence, f(·) is the output logit of the model, and t is the temperature parameter. We adopt the bornagain strategy (Furlanello et al., 2018) to iteratively distill the model into a sequence of generations. Hence, the model at kth generation with parameters θk is distilled to match the (k − 1)th generation with parameters θk−1.
Self-distillation can provably improve model performance if the data has a multi-view structure, i.e., the data has multiple features (views) to help identify its class (Allen-Zhu and Li, 2020). Such structures naturally exist in utterances. For instance, given the following utterance of label “travel alert”,
“How safe is visiting Canada this week”,
both “safe” and “visiting” indicate the intent label, and it is likely that the model learns only one of them because a single feature may be sufficient to discriminate the above utterance from others with different labels, especially with limited training data. Sequential self-distillation can help to learn both features, as shown in Allen-Zhu and Li (2020).

heading: 3.3 Experiments
text: We evaluate DFT++ on the same benchmarks used to evaluate DFT. We compare DFT++ with stateof-the-art continual pre-training methods. Since early stopping is not necessary, as demonstrated in subsection 2.2, we combine the validation and test sets for a more comprehensive evaluation.
Baselines We compare with the following baselines. TOD-BERT (Wu et al., 2020a) conducts continual pre-training on dialogue corpus with MLM and response objectives. DNNC-NLI (Zhang et al., 2020b) and SE-NLI (Ma et al., 2022) employ NLI datasets. DNNC-NLI is equipped with a BERT-style pair-wise similarity model and a nearest neighbor classifier. SE-NLI employs sentence encoder (Reimers and Gurevych, 2019) with siamese and triplet architecture to learn the semantic similarity. DNNC-Intent, CPFT (Zhang et al., 2021b), IntentBERT (Zhang et al., 2021a) and IsoIntentBERT (Zhang et al., 2022) use external
2We have also tried to add a cross-entropy term (Tian et al., 2020), but find it hurts the performance.
intent detection datasets. DNNC-Intent shares the same model structure as DNNC-NLI. CPFT adopts contrastive learning and MLM. IntentBERT employs standard supervised pre-training, based on which IsoIntentBERT introduces isotropization to improve performance. SE-Paraphrase (Ma et al., 2022) exploits paraphrase corpus, using the same model architecture for sentence encode as SE-NLI.
For all the baselines, we download the publicly released model if available. Otherwise, we follow the original work’s guidelines to perform continual pre-training. Next, we perform standard fine-tuning similar to DFT, using hyperparameters searched within the same range as our method, with three exceptions: DNNC-NLI, DNNC-Intent, and CPFT. For these methods, we use the original design and training configuration for in-task fine-tuning.
In addition, we compare DFT++ against CINS (Mi et al., 2022), the most recent promptbased method. CINS addresses intent detection by converting it into a cloze-filling problem through a carefully designed prompt template. Similar to our method, CINS directly fine-tunes PLMs on a limited amount of data.
Our method We evaluate our method and the baselines based on two popular PLMs: BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019b). The representation of the token [CLS] is used as the utterance embedding. For a fair comparison, we select the hyper-parameters with the same validation data as used by the baselines, i.e., we follow IsoIntentBERT to use a portion of the OOS dataset as the validation data. The best hyper-parameters and the parameter range are given in the appendix.
Main results We first examine the performance using a moderately small amount of data, specifically 5-shot and 10-shot scenarios. The results are summarized in Table 2. Remarkably, DFT++ performs comparably to a diverse set of baselines that leverage external resources, despite the fact that it solely utilizes the limited few-shot data available. The superiority of DFT++ can be attributed to the effective utilization of context augmentation and sequential self-distillation, both of which demonstrate improved results when applied independently in most cases. We notice that DFT++ performs better when using the stronger base model RoBERTa. As shown in Table 2b, DFT++ outperforms all the baselines in most cases. Moreover, as shown in Table 3, in most cases, DFT++ also outperforms
CINS, the most recent prompt-based method, despite that CINS employs T5-base (Raffel et al., 2020) with 220 million parameters, which is almost twice the size of our base model.
To study the impact of the number of labeled data on performance, we reduce the number to only 1 sample per label and present the results in Fig. 6. We experiment with BANKING77, a chal-
lenging fine-grained dataset. When using BERT, we observe that DFT++ begins to outperform the baselines at a crossing point of 4. When using RoBERTa, the crossing point is even smaller, at 2, which is quite surprising. We have also observed similar phenomena on other datasets, as detailed in the appendix. The observations confirm our claim that the overfitting issue in directly fine-tuning PLMs for few-shot intent detection may not be as severe as initially presumed. The performance disadvantage resulting from overfitting can be effectively alleviated by leveraging other techniques to exploit the limited available data, even without resorting to the continual pre-training approach. However, in scenarios with an extremely small number of labeled data, the transferred knowledge from continual pre-training still provides significantly better performance compared to DFT++.

heading: 3.4 Analysis
text: Comparison between contextual augmentation and conventional data augmentation methods
We compare our proposed context augmentation with the following conventional data augmentation methods. Easy Data Augmentation (EDA) (Wei and Zou, 2019) modifies a small number of utterances, e.g., through word swapping, to generate new augmented instances. Back-translation (BT) (Edunov et al., 2018) translates an utterance into another language and then translates it back3. PromDA (Wang et al., 2022) and SuperGen (Meng et al., 2022) are recent data augmentation methods leveraging generative PLMs. GPT-J-DA (Sahu et al., 2022) exploits the data generated by GPT-J in a supervised manner. The results in Table 4 show context augmentation is more robust against data shift. Note that SuperGen is designed for coarsegrained tasks with only two or three labels, such as sentiment classification. As a result, it may not scale effectively to intent detection tasks that involve a larger number of intents, typically ranging in the tens. The comparison between context augmentation and GPT-J-DA highlights the superiority of unsupervised exploitation of the generated data. The inconsistent effectiveness of GPT-J-DA is also reported by Sahu et al. (2022).
3We use French as the intermediate language, and utilize T5-base (Raffel et al., 2020) and opus-mt-fr-en (Tiedemann and Thottingal, 2020) for translation.
Quality of context augmentation To demonstrate the quality of the data generated by context augmentation, we provide some good and bad examples of generated utterances in Table 5. It is observed that GPT-J is able to generate grammatically fluent utterances that exhibit a high level of contextual relevance to the input utterances, which are utilized by DFT++ to better model the target data distribution. On the other hand, as also observed in Sahu et al. (2022), some of the generated utterances deviate from the original label and, therefore, are not suitable for data augmentation. However, DFT++ mitigates this issue by focusing solely on leveraging contextual relevance, resulting in improved robustness against data shift (Table 4).
Complementarity of continual pre-training and DFT++ Continual pre-training and DFT++ mitigate overfitting from different aspects. The former leverages external data, while the latter maximizes the utilization of the limited available data. Hence, it is likely that they are complementary. To support this claim, we present empirical results demonstrating their complementarity in Table 6. It is observed that when combined with DFT++, the two competitive methods, IsoIntentBERT and SE-Paraphrase, both demonstrate improved performance.
Impact of hyper-parameters We study the impact of two key hyper-parameters, the size of the generated data and the number of self-distillation generations. As visualized in Fig. 7a, a positive correlation is found between the performance and the size of the augmented data. The performance saturates after the data size per label reaches 50. It is noted that when only the given data are used for MLM, i.e., the generated data size is 0, MLM has an adversarial effect probably due to overfitting on the few given data. Such negative effect is successfully alleviated by context augmentation. As for self-distillation generations, we find that multiple generations of self-distillation are necessary to achieve better performance. In the appendix, we further analyze the impact of of the temperature parameters of GPT-J and self-distillation.
Comparison with alternative context augmentation methods We have also studied alternative context augmentation methods. The first one is Easy Data Augmentation (EDA) (Wei and Zou, 2019) with random synonym replacement, insertion, swap, and deletion. The second approach involves manually collecting a domain-specific corpus. We conduct experiments on BANKING77,
since it focuses on a single domain, making it convenient to collect the corpus. We extract web pages from Wikipedia4 with keywords that are closely relevant to “Banking”, such as “Bank” and “Credit card”. The keywords can be found in the appendix. As shown by Table 7, our GPT-J-based context augmentation outperforms the alternatives. We attribute the superiority to the grammatical fluency achieved by leveraging the generative power of GPT-J, which is typically compromised by EDA. Additionally, the high degree of semantic relevance observed in our approach is rarely guaranteed in the noisy corpus collected from Wikipedia.

heading: 4 Related Works
text: Few-shot Intent Detection Before the era of PLMs, the study of few-shot intent detection focuses on model architecture (Geng et al., 2019; Xia et al., 2020a; Nguyen et al., 2020). Recently, finetuning PLMs has become the mainstream methodology. Zhang et al. (2020b) fine-tune pair-wise encoder on natural language inference (NLI) tasks. Zhang et al. (2021b) fine-tune PLMs in a con-
4https://en.wikipedia.org
trastive manner. Zhang et al. (2021a) leverage public intent detection dataset, which is further improved by isotropization (Zhang et al., 2022). Other settings are also studied, including semi-supervised learning (Dopierre et al., 2020, 2021) and incremental learning (Xia et al., 2021b). Unlike the mainstream strategy, our method does not require continual pre-training on extra resources.
Continual Pre-training of PLMs Continual pretraining of PLMs is helpful (Gururangan et al., 2020; Ye et al., 2021; Luo et al., 2021). For dialogue understanding, many works leverage conversational corpus to perform continual pre-training. Li et al. (2020) conducts continual pre-training with a dialogue-adaptive pre-training objective and a synthesized in-domain corpus. Wu et al. (2020b) further pre-trains BERT with dialogue corpora through masked language modeling and contrastive loss. Henderson et al. (2020) use Reddit conversational corpus to pre-train a dual-encoder model. Vulić et al. (2021) adopts adaptive conversational fine-tuning on a dialogue corpus.
PLM-based Data Augmentation Rosenbaum et al. (2022) fine-tune PLMs to generate data for
intent detection and slot tagging. Jolly et al. (2020) develop novel sampling strategies to improve the generated utterances. Kumar et al. (2022) pre-train a token insertion PLM for utterances generation. However, these methods require slot values, which are assumed unavailable in this work. Papangelis et al. (2021) fine-tune PLMs with reinforcement learning, but our augmentation method adopts offthe-shelf PLM without further training. The closest work to ours is Sahu et al. (2022), which utilizes off-the-shelf PLMs for data augmentation. However, our method focuses solely on leveraging contextual relevance to achieve improved robustness. PLM-based data augmentation has been explored for other tasks, e.g. sentiment classification (Yoo et al., 2021; Wang et al., 2022; Chen and Liu, 2022) and natural language inference (Meng et al., 2022; Ye et al., 2022). However, these approaches may fail to scale to intent detection tasks with tens of intent classes, as shown by Sahu et al. (2022) and our experiments.

heading: 5 Conclusions and Limitations
text: We revisit few-shot intent detection with PLMs by comparing two approaches: direct fine-tuning and continual pre-training. We show that the overfitting issue may not be as significant as commonly believed. In most cases, our proposed framework, DFT++, demonstrates superior performance compared to mainstream continual pre-training methods that rely on external training corpora.
One limitation of DFT++ is the computational overhead caused by generative PLMs. Additionally, our current approach includes all utterances generated by the PLM, even those that might lack contextual relevance or contain noise. These issues are left for future exploration.

heading: Acknowledgments
text: We would like to thank the anonymous reviewers for their helpful comments. This research was partially supported by the grant of HK ITF ITS/359/21FP.

heading: A Appendix
text: Hyper-parameters We determine the hyperparameters by grid search. The best hyperparameters and the search range are summarized in Table 8 and Table 9, respectively. The grid search is performed with OOS dataset. Specifically, we follow IsoIntentBERT to use the two domains “Travel” and “Kitchen dining” as the validation set. To guarantee a fair comparison, the same validation set is also employed for all the baselines.
Implementation details We use Python, PyTorch library and Hugging Face library 5 to implement the model. We adopt bert-base-uncased and roberta-base with around 110 million parameters. We use AdamW as the optimizer. We use different learning rates for PLMs and the linear classifier, determined by grid-search. The parameter for weight decay is set to 1e− 3. We employ a linear scheduler with the warm-up proportion of 5%. We fine-tune the model for 200 epochs to guarantee convergence. The experiments are conducted with Nvidia RTX 3090 GPUs. We repeat all experiments for 5 times, reporting the averaged accuracy and standard deviation.
Impact of the number of labeled data on performance We provide the full results in Fig. 9. It is
5https://github.com/huggingface/transformers
observed that DFT++ outperforms many competitive methods fine-tuned on extra data even when the number of labeled data is small.
Keywords used to collect the corpus for an alternative context augmentation method As introduced in subsection 3.4, one alternative context augmentation method involves manually collecting a domain-specific corpus. We experiment with BANKING77. To collect an external corpus, we extract web pages from Wikipedia6 with keywords closely related to “Banking”, such as “Bank” and “Credit card”. The adopted keywords are summarized in Table 10.
Analysis of hyper-parameters We show the impact of the temperature parameter of GPT-J and self-distillation in Fig. 8. The temperature parameter of GPT-J controls the diversity of the generated context. A higher temperature makes the generated text more diverse. As shown in the figure, the best performance is reached when the diversity is moderate.For self-distillation, both small and large temperatures can produce good results.
6https://en.wikipedia.org
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: These issues are left for future exploration