Unsupervised Distractor Generation via Large Language Model Distilling and Counterfactual Contrastive Decoding (2024)

Abstract: Within the context of reading comprehension, the task of Distractor Generation (DG) aims to generate several incorrect options to confuse readers. Traditional supervised methods for DG rely heavily on expensive human-annotated distractor labels. In this paper, we propose an unsupervised DG framework, leveraging Large Language Models (LLMs) as cost-effective annotators to enhance the DG capability of smaller student models. Specially, to perform knowledge distilling, we propose a dual task training strategy that integrates pseudo distractors from LLMs and the original answer information as the objective targets with a twostage training process. Moreover, we devise a counterfactual contrastive decoding mechanism for increasing the distracting capability of the DG model. Experiments show that our unsupervised generation method with Bart-base greatly surpasses GPT-3.5-turbo performance with only 200× fewer model parameters. Our proposed unsupervised DG method offers a cost-effective framework for practical reading comprehension applications, without the need of laborious distractor annotation and costly large-size models.

heading: 1 Introduction
text: Reading comprehension assessment holds significant importance in the educational field. Typically, a reading comprehension sample consists of four components: passage, question, answer and multiple distractors. In recent years, while the clozestyle Distractor Generation (DG) task has received wide interest (Ren and Zhu, 2021; Chiang et al., 2022; Wang et al., 2023), the DG task with complete long sentences is less addressed, primarily due to a scarcity of available supervised data.
Limited by the expensive annotation cost, there are just a few reading comprehension datasets for DG from examination scene (Lai et al., 2017; Sun
* Corresponding author.
et al., 2019). Methods on these single-sourcing datasets (Gao et al., 2019; Zhou et al., 2020) all face challenges of insufficient generalization in real-world applications. On the other hand, unsupervised generation methods remain inaccessible considering the difficulty brought by the reading comprehension context.
Recently, LLMs like GPT (OpenAI, 2023) and LLaMa (Touvron et al., 2023) have demonstrated powerful ability as automatic annotators to label training data (Arora et al., 2022; Gilardi et al., 2023). LLMs have been successfully applied in various fields of NLP, including multi-choice questionanswering task (Bitew et al., 2023; Nasution, 2023; Doughty et al., 2024). However, compared to previous fine-tuned methods, mainstream LLMs often fail to achieve a satisfactory performance on DG, as illustrated in Figure 1. Additionally, deploying LLMs in real world applications is challenging due to their substantial computational resource requirements and the closed-source model parameters.
To meet the high need of DG for real applications, where there are no distractor labelling data and only limited computational resource, we propose an unsupervised DG framework with a small model. We adopt the distilling paradigm to en-
827
hance the smaller student model’s generation capability with pseudo labels from LLMs (Smith et al., 2022; Arora et al., 2022), regarding LLMs as data annotators to assist the training of the smaller model (Wang et al., 2021). Recognizing the suboptimal performance of LLMs on DG, we propose a dual task training strategy by integrating both pseudo distractors and golden answers as training targets. Furthermore, we devise a two-stage training framework to reduce the negative impact caused by the conflict semantics presented in answers and distractors. Note that we do not use the reference distractors for the unsupervised setting.
The utilization of answer information as training target, although improves the model’s performance in generation quality, makes harm to the counterfactual capability of the generative model. To address this issue, we introduce contrastive decoding (Li et al., 2023) into the inference process of DG. Specifically, we penalize factual text patterns favored by the answer generation module while encourage counterfactual results generated by the distractor generation agent. Additionally, we apply plausibility constraint to restrict the effect of contrastive decoding for more stable generation results. Note that we do not leverage LLMs during inference to ensure an easy deployment in real applications.
We conduct experiments on RACE (Lai et al., 2017) and Dream (Sun et al., 2019). As illustrated in Figure 1, our unsupervised method with Bartbase significantly outperforms zero-shot LLMs with 200× fewer model parameters. Experimental results also show that our proposed counterfactual contrastive decoding method greatly improves the distracting capability of the generation model. Moreover, we leverage GPT-4 (OpenAI, 2023) to evaluate the generated distractors, demonstrating that our method obtains a better performance than GPT-3.5-turbo both in generation quality and distracting level.
To sum up, our contributions are as follows:
• We propose an unsupervised DG framework with dual task training, integrating the original answer information and pseudo distractors generated by LLMs.
• We devise a new counterfactual contrastive decoding method to improve the distracting level of generated outputs. The proposed optimization can be transferred to other counterfactual generation tasks.
• Our method greatly outperforms teacher LLMs across various evaluation metrics. Our method provides a valuable approach for constructing reading comprehension data in diverse real-world applications, eliminating the need for costly human-annotated data and large-size models.

heading: 2 Related Work

heading: 2.1 Distractor Generation
text: Previous researches on distractor generation in reading comprehension mostly concentrate on designing attention framework based on end-to-end models (Gao et al., 2019). Co-attention (Zhou et al., 2020) and reforming modules (Qiu et al., 2020) among passage, question and answer are proposed to extract key information about question-related and counterfactual details. Besides, Mixture of Expert is utilized in some works to ensure the quality and diversity of the generation results (Qiu et al., 2020; Qu et al., 2023).
Recently, DG studies with LLMs mostly focus on knowledge-based distractor generation in education field. Bitew et al. (2023) explores questionsimilarity based example selection method to enhance LLMs’ DG performance in in-context learning. Doughty et al. (2024) designs complex prompt to generate multi-choice question answering data for Python programming learning. Nasution (2023) asks ChatGPT to construct multi-choice data with the input of biology subject for biology learning.

heading: 2.2 LLM Knowledge Distillation
text: LLMs have been widely applied to generate pseudo labels to reduce the labeling cost in unsupervised situation. Recent works have proved the effectiveness of LLMs in annotating accuracy compared to human-annotated results in various NLP tasks (Gilardi et al., 2023; He et al., 2023). Related works have already been applied in question answering (Saad-Falcon et al., 2023), information retrieval (Bonifacio et al., 2022), text summarization (Wang et al., 2021) and common sense reasoning (Whitehouse et al., 2023) , covering both NLU and NLG fields.
Similar unsupervised methods are still underexplored in DG task, though previous works suffer from the expensive labeling cost. Addressing this gap is the main target of our paper.

heading: 3 Method

heading: 3.1 Task Definition
text: The reading comprehension data generally consists of four components: passage, question, answer and multiple distractors. In this context, the DG model regards the triplet of passage(p), question(q) and answer(a) as input and generates results with a probability of pd:
pd = p(di|p, q, a, d<i) (1) In this work, we propose an unsupervised framework for DG task, where the reference distractors in the training data are unseen but only the passage(p), question(q) and answer(a) are provided. In this way, we would liberate experts from the laborious work of distractor annotation.
An overview of our unsupervised framework is displayed in Figure 2. We first apply LLM as a teacher model to generation pseudo distractors. Next, we train a smaller student model with both pseudo labels and answers as generating targets through a two-stage training process.

heading: 3.2 Generating Distractors with LLMs
text: Previous works on DG task (Gao et al., 2019; Zhou et al., 2020) mostly depend on human-annotated data. The common-used dataset like RACE (Lai et al., 2017) is sourced from the educational domain and annotated by professional teachers, exhibiting a high quality but expensive cost. The powerful generation capability of LLMs presents a chance for DG task in reading comprehension to overcome the problem of limited data.
Instead of utilizing human-annotated distractors, we obtain pseudo distractors from a LLM teacher with the input passage, question and answer. To save space, we display the prompts in Appendix A.1.
To guarantee the distracting level of the generated pseudo distractors, we filter out results that exhibit high similarity to the answer by calculating the BLEU-4 score. All pseudo outputs with a BLEU-4 score greater than 30 are dropped, along with their associated question-answer pairs.

heading: 3.3 Dual Task Training with Student Models
text: The pseudo distractors from LLMs (denoted as d′) can be directly distilled to a student model as supervised signals during training. However, employing this straightforward augmentation method may not lead to satisfactory results because of the suboptimal performance of LLM in DG task.
As a pair of dual tasks, the answer generation task exhibits similarities with the DG task. Both tasks require a comprehensive understanding on the input passage and deep analysis on questionrelated contexts. To this end, we introduce answer generation into our unsupervised distractor training process as an auxiliary task.
To take advantage of both pseudo distractor generation and answer generation, we devise a twostage training procedure for the smaller student model. Firstly, the model treats [p, q, d′] as input and generates a, addressing answer generation:
pa = p(ai|p, q, d′, a<i) (2) Secondly, we just interchange answers and
pseudo distractors, applying [p, q, a] as input and d′ as output, serving as distractor generation:
pd = p(d ′ i|p, q, a, d′<i) (3)
In the experiment, we prepend task-specific tokens to distinguish these two tasks. For distractor generation, we replace the decoder start token (often eos token) with the special token [DIS] and [ANS] for answer generation.
We denote the model training in answer generation as Ma, and distractor generation as Md. In the two-stage training procedure, Md is initialized with the parameters of Ma. Both models apply the cross-entropy (CE) loss for training.

heading: 3.4 Contrastive Decoding
text: The introduction of answer generation in dual task training raises up a challenge on the distracting level of the generated result, as the model may generate correct content. Inspired by the contrastive decoding method (Li et al., 2023), we propose a novel decoding strategy called counterfactual contrastive decoding (CCD) to solve this issue.

heading: 3.4.1 Counterfactual Contrastive Decoding
text: Generally speaking, CCD rewards counterfactual text patterns favoured by the distractor generation model while penalizes factual text patterns generated by the answer generation model. To obtain the output probabilities of both models, we propose a two-stage inference process:
Stage 1 Perform the inference process with p, q, a as input on DG model Md to generate one distractor result, referring as dinter.
Stage 2 Regard Md as the expert model and Ma as the amateur model, and apply contrastive decoding as Equation 4:
CCD-scorei = log pd(yi|p, q, a, y<i)
pa(yi|p, q, dinter, y<i) (4)
where y is the output sequence of Md. Please note that different from the training process, dinter replaces the pseudo distractor d′ in the input sequence of Ma, thereby avoiding the dependence on LLM during inference.
Taking into account the high similarity between Md and Ma models, some common text patterns with high probability in both models will become hard to be generated while the generation probability of some implausible tokens will greatly improve.
To address this issue, we optimize the counterfactual contrastive decoding as:
CCD-score = log-softmax(logit′d)
logit′d = logitd ∗ f(logitd, logita) logita = Ma(p, q, dinter, y<i) logitd = Md(p, q, a, y<i) (5)
The scaling function f is calculated as:
f(x, y) = exp(sgn(x) ∗ (σ(x− y t )− 0.5)) (6)
where σ is the Sigmoid function, sgn(∗) is the Signum function. t is a hyper-parameter to control the scaling degree and avoid the saturation of the sigmoid function.
As the final value of CCD-score mostly depends on the absolute value of logitd, the implausible tokens with low generation probability are still hard to be generated after the logit scaling. On the other hand, the scaling factor is determined by the difference between logitd and logita. Notably, it will be close to 1 when the logitd is approximately equal to logita, maintaining the high probability of tokens that both models exhibit high confidence on.

heading: 3.4.2 Plausibility Constraint
text: To improve the stability of the decoding results, we introduce a plausibility constraint method to restrict the effect of counterfactual contrastive decoding. For distractor generation, we propose a rank-based plausibility constraint:
CCD-score = { Equation 5 if xi ∈ Vadj log-softmax(logitd) otherwise
(7)
Vadj = {yi ∈ V : rank(pd(yi|y<i)) > r} (8)
We rank the probability across the vocabulary. For tokens with extremely high probability, we will fix their logits and only adjust tokens ranked after r-th, where r is a hyper-parameter. Compared to the value-based plausibility constraint method proposed in Li et al. (2023), our rank-based method is more stable and controllable.

heading: 4 Experimental Setup

heading: 4.1 Student Model Training
text: We choose Bart-base as the student model , which has only 139M parameters and can be easily deployed in real applications. We train the student model with 5-epoch answer generation and 10- epoch distractor generation in the two-stage training process. In both stages we set the maximum learning rate, batch size and warmup ratio to 10−5, 48 and 0.1. During inference, we adopt Jaccard Distance for generating 3 different results with the beam size as 20, and other detail settings can be referred to Zhou et al. (2020). r and t are set to 15 and 2 in experiment.

heading: 4.2 Applying LLMs
text: We conduct experiments on two mainstream series LLMs: LLaMa-2 and GPT-3.5. For LLaMa-2, we apply LLaMa-2-13B-chat implemented by HuggingFace. For GPT-3.5, we apply GPT-3.5-turbo with the official API.
LLM distractor generation for distilling We apply the prompt shown in Appendix A.1 to generate one pseudo distractor for each input p, q and a. For the sake of reproductivity, we turn off the sampling and set the temperature to zero.
LLM zero-shot inference To explore the performance of LLM, we apply LLMs to generate three distractors with the prompt displayed in Appendix A.2. Due to the high cost of applying LLMs with a large beam size, we do not ask LLMs to return 20 results for filtering like student model does.

heading: 4.3 Datasets
text: We conduct extensive experiments on RACE (Lai et al., 2017) and Dream (Sun et al., 2019), which are collected from the English exams of middle and high schools in China. The detailed statistics of the cleaned dataset are shown in Table 1.
We drop the human-annotated distractors in training set and utilize these datasets as unsupervised data. LLMs are applied to generate one pseudo distractor for each question-answer pair in the training set for student model training. The generated results will be filtered with BLEU score as mentioned in Section 3.2.

heading: 4.4 Automatic Evaluation Metrics
text: We apply BLEU1, Rouge2 and BertScore3 to evaluate the generation quality, and Distinct (Li et al., 2016) to evaluate the generation diversity.
As an important evaluation aspect, the distracting level of generated distractors has not received sufficient attention in previous works. Therefore, we propose a new automatic evaluation metric called Faithful Score to measure the distracting level of the generated results.
Faithful Score Based on RACE and Dream, we follow Jiang et al. (2020) and train an Alberta model on the machine reading comprehension task. This model aims to judge whether a given candidate is a correct answer to the corresponding passage-question pair and return a classification score ranging [0, 100]. The DG task prefers models that generate distracting results with low Faithful Score.
We conduct evaluation on the test set of RACE and Dream, and results are shown in Table 2. The Faithful Score values on reference answers are between 75 and 80 for both datasets and about 10 on distractors. The experiment results prove the excellent discrimination ability of this metric.

heading: 5 Results and Analysis

heading: 5.1 Main Results
text: The experiment results on RACE and Dream are shown in Table 3 and Table 4. For RACE, we display the results on previous works including
1https://pypi.org/project/sacrebleu/ 2https://pypi.org/project/pyrouge/ 3https://github.com/huggingface/evaluate
HSA (Gao et al., 2019), HCA (Zhou et al., 2020), EDGE (Qiu et al., 2020), HMD-Net (Maurya and Desarkar, 2020), MSG-Net (Xie et al., 2022), DGMoE (Qu et al., 2023). The results of fine-tuned Bart-base and LLMs are also displayed for comparison. For our full model, we apply GPT-3.5-turbo to generate pseudo distractors. We mainly evaluate the results from three aspects: quality, diversity and distracting level.
Performance on the generation quality Both GPT-3.5-turbo and LLaMa-2-13B-chat just manage to obtain half of the SOTA BLEU-4 score on two datasets. Compared to GPT-3.5-turbo, LLaMa2-13B-chat is more unstable. On the more challenging dataset RACE, LLaMa-2-13B-chat achieves far inferior performance on BertScore (70.03) and Rouge-L (19.48) than other methods.
Our proposed unsupervised method greatly outperforms LLMs on the generation quality. In terms of BLEU-4, we achieve 1.76 and 3.82 points improvements on RACE and Dream compared to GPT-3.5-turbo. And our model has achieved an approximate performance on BertScore and RougeL compared to the fully fine-tuned SOTA result. As for BLEU-4, there still exists an obvious gap between the unsupervised and fine-tuned results.
Performance on the generation diversity We apply Distinct-1 and Distinct-2 to measure the generation diversity. On both RACE and Dream, our method achieves a close performance to SOTA. Among all these methods, LLaMa-2-13B-chat achieves greatest diversity performance. However, this may be due to the high randomness of its result that improves the generation diversity while makes harm to the generation quality (Fang and Jiang, 2022).
Performance on the distracting level We apply Faithful Score to evaluate the distracting level of the generated results. Our proposed method outperforms GPT-3.5 and LLaMa-2-13B-chat on both datasets and performs closely to the fully fine-tuned method. On RACE, our method is just 0.98 points lower than fine-tuned method, and on Dream our method outperforms 3.37 points.

heading: 5.2 Effect of Dual Task Training
text: We conduct experiments on RACE to investigate the impact of the dual task training based on GPT3.5-turbo. We train the student model in four different settings: with only pseudo distractors (Pseudo Label); with only answer targets (Answer Label); with the mixed data of pseudo distractors and answers (Mixed Data) and two-stage training process
(Two Stage). In all these settings, we do not apply our proposed counterfactual contrastive decoding during inference.
The results are shown in Table 6. Compared to pseudo-distractor-only method, results of answeronly method have a slight improvement on BLEU4 and a tiny decline on Distinct-1. However, the Faithful Score of pseudo-distractor-only method outperforms that of answer-only method significantly, with a decline of 18.0 points. As for the dual task training with mixed data, the incorporation of answer information brings significant improvement on the generation quality. Nevertheless, it raises up a negative impact on the result’s distracting level, as evidenced by the Faithful Score value.
The two-stage training process proves to be effective to address the above limitation. We observe a decrease of 5.09 points in Faithful Score, accompanied by only a slight 0.27 points reduction in BLEU-4.

heading: 5.3 Effect of Counterfactual Contrastive Decoding
text: Further, we explore the effect of counterfactual contrastive decoding based on GPT-3.5-turbo. We
apply the DG model with two-stage training as the base model. The results on RACE dataset are shown in Table 5.
Despite a decline on the generation quality, CCD contributes to an improvement on model’s counterfactual generation capability, with a 1.74 points decline on Faithful Score. Besides, plausibility constraint successfully enhances the stability of the generated results, further reducing Faithful Score by 0.21 points and achieving a great trade-off between the generation quality and distracting level.

heading: 5.4 Performance with Low-Resource Setting
text: Our unsupervised method leverages answer information to enhance the generation quality. However, obtaining a sufficient number of annotated answer labels still requires investment, even if cheaper than distractors. In this section, we simulate the low-resource scenario of real-world applications on RACE dataset, and investigate the performance of both supervised and our unsupervised methods. The results are shown in Figure 3. The experiments of our unsupervised method are conducted based on GPT-3.5-turbo.
As the data ratio of the training set decreases, the generation quality of the supervised method declines continuously. In contrast, our unsupervised method maintains a stable BLEU-4 score until the data ratio decreases to 1% (about 400 samples) and demonstrates comparable performance to the supervised method in low-resource situation.
With the decrease of the unannotated data num-
ber, the second stage training for distractor generation becomes insufficient, resulting in a heightened impact from the first stage answer generation training on the final results. This leads to a faster increase in the Faithful Score of unsupervised method compared to supervised one.
Besides, we conduct experiments to explore LLMs’ performance on few-shot settings. The results in Table 7 indicate that adding 5 randomly selected demonstrations to the prompt only increases the BLEU-4 score by 0.3 points for LLaMa-2-13Bchat.

heading: 5.5 Evaluation from GPT-4
text: N-gram based quality metrics like BLEU are not completely consistent with the actual performance in generation task. Thereby we apply GPT-4 as a professional evaluator to measure the performance of different methods from two aspects: the quality and distracting level of the generated results.
Concretely, we apply GPT-4 to compare two distractors generated by different methods. GPT-4 is asked to return "Win", "Lose" or "Tie" according to the comparison result. The order of two input distractors will be randomly shuffled in the input prompt for a fair comparison. We randomly select 1000 samples from the test set of RACE for evaluation. The prompt for GPT-4 evaluation is proposed in Section A.3. The results are shown in Table 8.
For the generation quality, GPT-4 favors results from the supervised method more, and our method’s performance is slightly better than GPT3.5-turbo. As for the distracting level, our method performs significantly better than the other two methods, demonstrating the effectiveness of our counterfactual contrastive decoding strategy.

heading: 5.6 Performance of Different LLMs
text: Further, to explore the impact of different LLMs on the performance, we adopt GPT-3.5-turbo and LLaMa-2-13B as different teacher models, and the results are shown in Table 9.
Generally, the generation diversity and distracting level of the generated results are not significantly related to the selection of teacher model. However, the unsupervised method based on GPT3.5-turbo outperforms LLaMa-2-13B by an average of 1.40 points in BLEU-4 score, which is positively correlated with the zero-shot performance of LLMs.

heading: 5.7 Case Study
text: Table 10 illustrates two examples of the generated distractors by three models. All three models produce fluent results without grammar errors. However, in case 1, the results from GPT-3.5-turbo suffers a problem of low diversity. And in case 2, both fine-tuned Bart-base and GPT-3.5-turbo generate correct answers like ’how the fathers raise their children’, ’what kind of food the fathers and their children eat’ and ’how the fathers survive in the desert without cooking skills’. This mistake is not observed in our unsupervised results with counterfactual contrastive decoding.

heading: 6 Conclusion
text: In this paper, we propose an unsupervised distractor generation method. We apply Large Language Models as distractor labelers and construct dual task training process to enhance the student model’s generation capability. Moreover, we optimize contrastive generation method in counterfactual generation context. Experiment results indicate that our method generally outperforms LLM-based approaches and is comparable to fully fine-tuned results in low-resource situations. Given the absence of human-annotated distractor dataset, our work can make contribution to building solid reading comprehension data in more future scenarios.

heading: Acknowledgements
text: This work is supported by the National Natural Science Foundation of China (62076008) and the Key Project of Natural Science Foundation of China (61936012).
Limitations
First, there still remains an obvious gap between our proposed unsupervised method and supervised method in DG task, especially for the quality of the generated results. Second, we only experiment on two English dataset from reading comprehension data due to the lack of high-quality testing data. The performance of our method on other application scenarios requires further exploration. Third, we utilize human-annotated answer labels in our unsupervised method, which brings some cost of manual annotation. Fourth, while recent works succeed to enhance student models with rationales generated by LLMs in various NLU tasks, we fail to introduce these methods into DG task. We analyze that for NLG tasks, the rationales generated by LLMs rather complicate the generation process and put negative impact on the model’s performance.
Related work can be further explored in the future. Last, due to limited time, we do not explore the performance on more LLMs and student models with different parameter scales.

heading: A Prompt for LLMs
text: In this section we display the prompts applied to different LLMs.
A.1 Prompt for generating pseudo distractors We generate pseudo distractors by LLMs for the training of student model. The instructions are shown in Table 11
A.2 Prompt for zero-shot inference When applying LLMs for zero-shot inference in distractor generation, we ask LLMs to generate three different results in one generation process. The prompt are shown in Table 12
A.3 Prompt for GPT-4 evaluation We compare the quality and distracting level of results from two different models with GPT-4, and the prompts are shown in Table 13.

heading: Future Work
text: Related work can be further explored in the future. Last, due to limited time, we do not explore the performance on more LLMs and student models with different parameter scales