Multistage Collaborative Knowledge Distillation from a Large Language Model for Semi-Supervised Sequence Generation (2024)

Abstract: We study semi-supervised sequence generation tasks, where the few labeled examples are too scarce to finetune a model, and meanwhile, fewshot prompted large language models (LLMs) exhibit room for improvement. In this paper, we present the discovery that a student model distilled from a few-shot prompted LLM can commonly generalize better than its teacher to unseen examples on such tasks. We find that the student is able to learn a general pattern from the high-quality pseudolabels produced by the teacher during knowledge distillation (KD), and favorably not a general pattern from the low-quality pseudolabels. Leveraging this discovery, we propose a new method, Multistage Collaborative Knowledge Distillation from an LLM (MCKD), for these tasks. MCKD first few-shot prompts an LLM to produce pseudolabels for unlabeled data. Then at each stage of an iterative KD process, a new pair of students is trained on disjoint partitions of the pseudolabeled data, and produces new and improved pseudolabels for their unseen partitions. We conduct extensive experiments on four syntactic and semantic parsing datasets and show the effectiveness of MCKD for lowresource semi-supervised sequence generation. On CRAFT biomedical parsing, for example, 3-stage MCKD with 50 labeled examples outperforms an LLM teacher and vanilla KD by 7.5% and 3.7% parsing F1, respectively, and matches the performance of supervised finetuning with 500 labeled examples. 1

heading: 1 Introduction
text: Low-resource tasks are common in real life, including within specialized domains, since data annotation often requires expert knowledge and incurs significant costs (Verspoor et al., 2012). Semisupervised learning has been proposed as a solution
∗Equal contribution 1Our code is made available
github.com/andotalao24/Multistage-CollaborativeKnowledge-Distillation.
when abundant unlabeled data are available (Blum and Mitchell, 1998; McClosky et al., 2006; Han et al., 2018; Kontonis et al., 2023). In a typical application, a model is trained on limited labeled data and produces pseudolabels for unlabeled data (McClosky et al., 2008; Amini et al., 2022). The pseudolabeled data are then filtered according to confidence thresholds and used to train a new model. In more extreme few-shot cases, labeled data are too scarce to finetune a model to begin with. Large language models (LLMs) offer a useful mechanism for synthesizing pseudolabels, thanks to their remarkable ability to learn in context from only a handful of demonstrations (Wang et al., 2021b; Yoo et al., 2021; Ding et al., 2022; Wang et al., 2023). Smaller and faster models can then be trained for these tasks using knowledge distillation (KD) from LLMs.
In this paper, we study a challenging semisupervised sequence generation setting where labeled data are too few to finetune a model and few-shot prompted LLMs exhibit room for improvement. These will happen when the task is both expensive to annotate and under-represented in the pretraining of off-the-shelf LLMs. For example, it took 80 annotators around 2.5 years to parse 20k sentences of biomedical text in the CRAFT corpus (Verspoor et al., 2012). Meanwhile, pretrained LLMs do not always excel at tasks in specialized domains (Kung et al., 2022; Singhal et al., 2022) and tasks that involve specialized structured outputs, e.g., semantic or syntactic parsing. The overarching research question of this paper is whether LLMs can still be leveraged in such scenarios to develop strong prediction models.
To this end, we examine knowledge distillation (KD) from a few-shot prompted LLM to a much smaller model. We discover that the student can commonly outperform its LLM teacher on unseen evaluation data from such tasks. Our analysis reveals that the student can learn a general pattern from high-quality pseudolabels from the teacher,
14201
while helpfully failing to capture a general pattern for the low-quality ones due to their noisy nature. This discovery is encouraging because it opens up the possibility for leveraging the KD students as teachers for further distillation.
Leveraging the discovery, we propose a novel method, Multistage Collaborative KD from an LLM (MCKD), for semi-supervised sequence generation. MCKD first collects pseudolabels for a large amount of unlabeled data from a few-shot prompted LLM, bootstraping a multistage KD process. At each KD stage, a new pair of students is trained on distinct partitions of the pseudolabeled data and asked to produce pseudolabels for the data that they have not been trained on. This process improves upon vanilla KD through two core mechanisms: (1) cross-partition labeling at each KD stage, which splits all pseudolabeled data into two mutually exclusive partitions to train a collaborative pair of students and leverages their strong generalization capabilities to relabel the data, and (2) multistage KD, whereby student models continue to generalize better and produce higher quality pseudolabels than their teachers over multiple KD stages.
MCKD outperforms the LLM teacher and the relevant KD baselines in our evaluation, and is competitive with supervised finetuning with many more labeled data. On CRAFT constituency parsing, for example, MCKD with 50 labeled examples outperforms the LLM and vanilla KD by 7.5% and 3.7% F1, respectively, and matches supervised finetuning with 500 examples. On ATIS semantic parsing, MCKD with 50 labeled examples outperforms the LLM and vanilla KD by 7.2% and 2.3% F1, matching supervised finetuning with >750 examples.
The following is a summary of our contributions:
• We study if LLMs can be leveraged to train fast and accurate student models for semi-supervised sequence generation tasks where the LLM itself exhibits room for improvement when prompted with in-context learning examples. We find that KD students can often achieve better generalization than their LLM teachers (Section 5.1). • We propose MCKD, a novel KD-based solution for such tasks. Data partitioning and crosspartition labeling enable MCKD to gradually improve the quality of pseudolabels over multiple stages of distillation, yielding increasingly better generations of students (Section 2). • MCKD substantially outperforms prompted
LLM, finetuned, and KD baselines on multiple low-resource tasks (Section 3-4). • Further analyses show that: (a) MCKD students correct many of the teacher’s errors, and (b) MCKD scales well with increasing amounts of available unlabeled data (Section 5.2-5.3).
2 MCKD: Multistage Collaborative Knowledge Distillation from an LLM

heading: 2.1 Problem setup
text: Semi-supervised sequence-to-sequence generation tasks consist of few labeled examples Dlabeled = {xi, yi}Nlabeledi=1 and many unlabeled examples Dunlabeled = {xi}Nunlabeledi=1 for training, where xi and yi are all sequences. We focus on a scenario in which (1) Dlabeled is not large enough for training a capable prediction network via direct supervised finetuning, and (2) LLM few-shot prompting with demonstrations sampled from Dlabeled exhibits room for improvement. This is a challenging but prevalent scenario that occurs when a task, such as parsing, is both expensive to annotate and underrepresented in LLM pretraining.

heading: 2.2 Method
text: Figure 1 illustrates the MCKD algorithm. We detail the steps below.
LLM Prompting. We first sample examples from the labeled dataset Dlabeled and then few-shot prompt an LLM with those to produce pseudolabels for unlabeled data Dunlabeled.
Intermediate KD stages. Since we focus on tasks where few-shot prompted LLMs exhibit room for improvement, we consider the possibility that models finetuned on a sufficient amount of data, even if pseudolabeled by an LLM teacher or a model distilled from it, might be able to learn the task better. In Section 5.1, we will show empirical evidence that this conjecture is indeed true and analyze this phenomenon. We will present the discovery that, during KD from a few-shot prompted LLM or an intermediate model distilled from the LLM, (1) the student performance improves and approaches 100% F1 on teacher pseudolabels, and (2), meanwhile, the student performance on held-out evaluation data improves and surpasses that of the teacher.
To leverage the above discovery in MCKD, we perform data partitioning and propose crosspartition labeling with a collaborative pair of
B denote the same partitions but with teacher-generated pseudolabels. (2) At the i-th
intermediate KD stage, students siA and s i B are trained on previously pseudolabeled data D (x,ŷi−1) A and D (x,ŷi−1) B , respectively, and leveraged to label the other partitions DunlabeledB and DunlabeledA and produce D (x,ŷi) B and D (x,ŷi) A , which will be used to train the next-stage student(s). (3) In the final KD stage, a single final student sn is trained on both latest pseudolabeled partitions D(x,ŷn−1)A and D (x,ŷn−1) B .
student models at each intermediate KD stage. Concretely, we partition unlabeled data Dunlabeled randomly and evenly into two distillation sets DunlabeledA and DunlabeledB . Then, at each intermediate KD stage, we (1) train a collaborative pair of students using the mutually exclusive data partitions with pseudolabels generated by model(s) from the previous stage, which can be the LLM teacher or a pair of distilled previous-stage students, and (2) let each current-stage student produce pseudolabels for the partition that it has not been trained on, in order to gradually improve the quality of pseudolabels for Dunlabeled over stages.
Why do we need cross-partition labeling? Cross-partition labeling is key to the operation of MCKD. Since the student model almost perfectly fits its training data, i.e., pseudolabels produced by the previous-stage model(s), during KD (Section 5.1), letting the student label its own training partition would reproduce nearly the same pseudolabels that it was trained on. Using those pseudolabels to train a next-stage student can only be expected to yield very little improvement, if any. We overcome this problem by partitioning the unlabeled data so that each student is asked to label a partition that it was not trained on.
The final KD stage. In the last distillation stage, a single student model is trained on the entire Dunlabeled with the latest pseudolabels.

heading: 2.3 KD mechanism
text: For sequence-level knowledge distillation, the teacher can provide multiple types of supervision
signals for the student, such as, token-level logits, greedy beams (hard pseudolabels), and beams that are close to the ground truth. Kim and Rush (2016) have observed that greedy beams are typically the most effective, although the other two may marginally further increase the student performance. In this work, we adopt greedy beams generated by the teacher for KD, following the literature on KD from LLMs (Wang et al., 2021b; Yoo et al., 2021; Ding et al., 2022; Li et al., 2023; Gilardi et al., 2023). MCKD can thus be used to distill both open-source and closed-source LLMs where only generated beams are provided.

heading: 3 Experimental Setup

heading: 3.1 Data and evaluation
text: Tasks and datasets. We perform experiments on two sequence prediction tasks: constituency parsing and task-oriented semantic parsing, where the outputs are structured sequences and a generalpurpose LLM pretrained to generate natural language text has to undergo distribution shifts. For constituency parsing, we use the Penn Treebank (PTB) dataset (Marcus et al., 1993) from the news domain and the Colorado Richly Annotated FullText (CRAFT) corpus from the biomedical domain (Verspoor et al., 2012). Semi-supervised learning for biomedical tasks is especially valuable since data annotation usually requires expertise and is formidably expensive.2 For task-oriented semantic parsing, we use the Airline Travel Information
2For example, it took 80 annotators around 2.5 years to parse the 20k sentences in the CRAFT corpus.
Systems (ATIS) dataset (Tur et al., 2010) and the Snips dataset.3 These datasets have been used for joint intent classification and slot filling. We focus on the latter that is a sequence generation task, letting models input natural langauge queries and output sequences of per-word slot labels. More details about the datasets are shown in Appendix A.
Simulating low-resource scenarios. For the constituency parsing datasets, PTB and CRAFT, we randomly sample 250 examples from the training set as the pool of labeled data. In different experiments, we randomly sample 50 or 250 examples from this pool to be the labeled set Dlabeled. For the task-oriented semantic parsing datasets, ATIS and Snips, we randomly sample 50 examples as the labeled set Dlabeled since the datasets are smaller.
We use the remaining training examples, excluding the 250 above for constituency parsing and 50 for semantic parsing, as the pool of unlabeled data. For the main results (Section 4), we randomly sample 20k unlabeled examples for PTB and use the entire unlabeled pool for CRAFT, ATIS, and Snips as the set Dunlabeled. We randomly and evenly partition Dunlabeled into DunlabeledA and DunlabeledB . For further analyses (Section 5), the amount of unlabeled data randomly sampled from the pool will be described in the respective contexts.
Evaluation. We compute token-level F1-score for each example and average over examples to measure model performance. For constituency parsing, we follow common practice to use bracketing but discard constituent tags in training and evaluation, and remove punctuation before computing parsing F1.4 We minimally post-process model outputs, balancing brackets and fixing accidental word segmentation.5 For semantic parsing, we use the evaluation function from Goo et al. (2018).6

heading: 3.2 Model, learning, and prediction
text: The LLM teacher. We use GPT-3.5 Turbo7, a model supported by the ChatGPT API, to collect
3https://github.com/snipsco/nlu-benchmark/ tree/master/2017-06-custom-intent-engines
4Decisions associated with punctuation are often arbitrary in ground truth parse trees and not informative of model performance.
5We do not constrain model generation. Early during training, T5 fails to consistently output well-formed parse trees. This is less of an issue after the first few epochs.
6https://github.com/MiuLab/SlotGated-SLU/blob/ master/utils.py
7https://platform.openai.com/docs/models
initial pseudolabels for Dunlabeled. We access the model through the OpenAI API. We randomly and uniformly sample 30 labeled parsing examples from Dlabeled to include in the prompt, since 50 or 250 examples will exceed the allowed maximum context length. Details of prompt design can be found in Appendix B.
The student model. We finetune T5-Base (220M parameters; Raffel et al., 2020) as the student model using the Huggingface transformers library (Wolf et al., 2020). Our assumed low-resource semi-supervised setting implies that the amount of available labeled data is small and a reliably large validation set does not exist. Inspired by classic unsupervised algorithms such as K-Means, we train each student until a convergence criteria is met — we train the student until the training accuracy on teacher pseudolabels changes by less than 0.1% for three consecutive epochs. The other hyperparameters are learning rate 3× 10−4, batch size 32, and 21 maximum training epochs.8 Generation is performed via greedy decoding. Results are from single runs that take less than 6 hours on a single RTX 6000 GPU unless otherwise stated.

heading: 3.3 Baselines
text: We compare MCKD with the few-shot prompted LLM, T5-Base finetuned using few-shot labeled data Dlabeled, and the following KD baselines.
Vanilla KD. We employ sequence-level knowledge distillation (Kim and Rush, 2016) where a student is trained on hard labels generated by GPT3.5 Turbo for the unlabeled dataset Dunlabeled.
KD + SD w/ filtering. We first train a student with teacher pseudolabels on the entire unlabeled dataset Dunlabeled through vanilla KD. Then we apply self-distillation (SD): letting the student predict pseudolabels for Dunlabeled, filtering them by keeping r% of the data where the student has the highest average token log-probabilities, and retraining a student on the filtered pseudolabeled data. Confidence-based filtering is a common method to optimize learning from pseudolabels (Lang et al., 2022; Wang et al., 2021a; Mohananey et al., 2020;
8Oracle experiments on PTB show that the validation performance doesn’t change significantly given a wide range of these hyperparameters, and thus we use a default set in all experiments. In scenarios where a labeled validation set is available, further tuning may prove even more beneficial due to optimization behavior such as double descent (Opper et al., 1989; LeCun et al., 1991; Maddox et al., 2020).
Vinyals et al., 2015; McClosky et al., 2006). We note that KD + SD without filtering is similar to vanilla KD, since the student pseudolabels on Dunlabeled can be expected to be almost identical with teacher pseudolabels. We do not report a baseline that similarly filters LLM pseudolabels before we train a student, since the GPT-3.5 Turbo API access does not provide logits.

heading: 4 Main Results
text: Baseline performance. As shown in Table 1, GPT-3.5 Turbo’s few-shot performance is reasonably good but has room for improvement. Due to the maximum input length for GPT-3.5 Turbo, we always sample 30 in-context demonstrations. In most cases, sampling them from a set of 50 or 250 labeled data Dlabeled does not make a big difference.
Supervised finetuning of T5-Base on Dlabeled has the worst performance. This matches our assumption that the amount of available labeled data is too small to directly finetune a model. On both datasets for constituency parsing, 50 labeled examples completely fail to effectively finetune T5-Base to generate well-formed parse trees.
Vanilla KD yields students that outperform the teacher across all datasets. KD + SD with confidence-based filtering is shown to have limited performance improvement over vanilla KD and is subject to the selection of threshold for highconfidence labels for each case. For instance, keeping the 25% most confident pseudolabels gives the best performance for PTB with 50 labeled data, while keeping 75% performs the best for CRAFT with 50 labeled data. Selecting a proper threshold thus can be resource consuming and difficult in practice especially in low-data regimes with no labeled validation data.
MCKD performance. Our MCKD approach substantially outperforms the few-shot prompted GPT-3.5 Turbo and KD baselines in all settings. For example, on CRAFT with 50 gold labeled training examples, 2-stage MCKD outperforms GPT-3.5 Turbo by 6.9% F1 and Vanilla KD by 3.1% F1. 3-stage MCKD further improves results, outperforming GPT-3.5 Turbo by 7.5% and Vanilla KD by 3.7%.
3-stage MCKD performs better than 2-stage MCKD consistently, but requires additional training time. The 4th-stage KD has marginal benefits in some settings. In practice, 2-stage MCKD is a
250 500 750 #Examples
60
65
70
75
80
F1 (%
)
SFT_PTB MCKD_PTB
SFT_CRAFT MCKD_CRAFT
(a) Constituency parsing. The performance of MCKD using 50 annotations (shown in horizontal lines) is competitive with supervised finetuning (SFT) using 500 or more annotations on PTB and CRAFT.
250 500 750 4000 #Examples
70
75
80
85
90
95
100
F1 (%
)
SFT_Snips MCKD_Snips
SFT_ATIS MCKD_ATIS
(b) Task-oriented semantic parsing. The performance of MCKD using 50 annotations (shown in horizontal lines) outperforms supervised finetuning (SFT) with 750 annotations on ATIS and Snips.
Figure 2: The test F1 scores of supervised finetuned (SFT) models increase with more gold annotated data. MCKD needs much less labeled data to match the performance of SFT.
good starting point and practitioners can increase the number of stages based on available resources.
Comparison with direct supervised finetuning. For constituency parsing, 3-stage MCKD using 50 labeled examples can match the performance of supervised finetuning (SFT) of T5-Base using 500 labeled examples on PTB, and outperform SFT using 500 labeled examples on CRAFT (Figure 2a). For task-oriented semantic parsing, 3-stage MCKD with 50 labeled examples can substantially outperform SFT on 750 labeled examples on both ATIS and Snips (Figure 2b). These results demonstrate the label-efficiency of MCKD and its effectiveness in semi-supervised sequence generation tasks where annotated data are scarce.

heading: 5 Analysis
text: This section presents additional analyses that motivate the design of MCKD and explore more deeply
into why it performs well. All analyses are conducted on PTB data. For simplicity and without loss of generality, we examine 2-stage KD with a single student per stage; the analyses can be generalized to multiple students and more stages which are advantageous in more applied settings.
In each experiment, we sample subsets of DunlabeledA and DunlabeledB (see Section 3.1) and denote them as DA and DB . We vary the sizes of DA and DB across experiments to study the effect of the distillation set size on performance. We train a stage-1 student s1 using DA with pseudolabels from an LLM teacher t and evaluate it on DB against ground truth labels (assumed to be available only for analysis purposes and not during the training in MCKD). Then we train a stage-2 student s2 using DB with pseudolabels from s1 and evaluate it on the test set.

heading: 5.1 A student can generalize better than its teacher
text: To understand the generalization properties of a prompted LLM teacher and its KD students, we first train a student s1 on 4k data DA pseudolabeled by the LLM teacher t. We observe that the student s1 (1) approaches 100% F1-score on the pseudolabels of the teacher t, and (2) outperforms the teacher t on 4k held-out data DB (Figure 3). The mean training F1 of s1 on DA examples plateaus around 97%, while the median F1 is 100% as shown in the box plots, indicating that s1 almost perfectly learns the pseudolabels of t, including potentially noisy ones. However, this does not weaken the generalization of s1, as its performance continues to improve on DB gold labels, eventually outperforming teacher t by an absolute ∼ 13%. This observation holds for various distillation set
sizes (see Appendix D). To also examine if student s1 can provide better distillation signal than the LLM teacher t, we perform the following experiment: First, we train a stage-2 student sB2 on 4k data DB pseudolabeled by s1. We then train a similar student sB1 on DB pseudolabeled by t. Finally, we evaluate sB1 and s B 2 on the unseen test set. Our results show that the F1score of sB2 (72.5%) surpasses that of s B 1 (71.3%), confirming that the pseudolabels of student s1 can train a better model than the LLM psuedolabels.
Memorization vs. generalization. Why is the generalization of our student s1 robust to the presence of noisy teacher pseudolabels? Feldman (2020) shows that the signal in a training example can either be internalized by a trained model as an instance of the general distribution that it learns or memorized separately. To find out if s1 learns high and low-quality pseudolabels of teacher t differently in our context, we design the following experiment: We first identify the examples in the distillation set DA where s1 achieves a 100% F1
on pseudolabels of teacher t, i.e., its training labels. From these well-learned training examples, we randomly sample 100 clean ones on which pseudolabels of t have a 96.3% average F1 with respect to the ground truth labels, and denote them as Dclean. We also sample 100 noisy examples where pseudolabels of t have only a 31.8% average F1 with respect to ground truth, and denote them as Dnoisy. We then finetune a new T5-base student s′1 on the remaining data DA − (Dclean ∪ Dnoisy) and evaluate it on Dclean and Dnoisy against pseudolabels of teacher t.
As shown in Figure 4, s′1 obtains a high teacherstudent F1 on Dclean but a low teacher-student F1 on Dnoisy. The original student s1, with Dclean and Dnoisy in its training data, was able to learn both sets well with a 97% teacher-student F1 in Figure 3. When both are removed from training (of s′1), Dclean still resembles the general distribution learned by s′1 from the remaining training data, while Dnoisy is now out-of-distribution. The s′1 results above thus confirm that our stage-1 MCKD student learns a general distribution over clean teacher pseudolabels while memorizing the noisy ones in a more instance-specific way. This explains why the generalization of the student is not affected by its learning of teacher errors in Figure 3.

heading: 5.2 The student corrects the teacher’s errors
text: We have observed the student s1 trained on 4k instances of DA pseudolabeled by the LLM teacher t to outperform its teacher t on the held-out set DB . To understand on which held-out examples student s1 is outperforming the teacher t, we split DB evenly into two halves: the sentences on which t achieves the highest F1 (Dhigh) and those on
which it has the lowest F1 (Dlow). As illustrated in Figure 5, the performance of s1 improves on both halves given more pseudolabeled training data DA. Given a large enough DA, the performance of s1 approaches that of t on Dhigh and is better on Dlow. In other words, the student is able to correct the teacher’s mistakes while almost completely retaining its true knowledge.

heading: 5.3 Large distillation sets are useful
text: Given an equal amount of labeled data Dlabeled, we investigate in this section if the stage-2 MCKD student s2 can benefit from growing amounts of unlabeled data Dunlabeled. First, Figure 6 shows that the performance of the stage-1 student s1 on the held-out set DB of 4k instances improves as the distillation set DA grows larger. With a DA of 2k or more instances, s1 outperforms the few-shot prompted LLM teacher t on DB .
A stage-2 student s2 is then trained on DB with pseudolabels from s1 and evaluated on the test set. We examine the effect of the sizes of distillation sets DA and DB on the final test performance of s2 in Figure 7. When the distillation set DA for s1 is too small, increasing the size of DB by generating more pseudolabels with s1 does not substantially improve performance for s2, likely due to the pseudolabels being too noisy. Learning from an increasing amount of such noisy labels may even worsen the test set performance of s2; for instance, with a DA of 2k instances, enlarging DB from 4k to 8k instances reduces the test F1 of s2 from 71.3 to 69.0. On the other hand, when DB is too small, making DA larger alone does not improve the test set performance of s2 either, despite higher-quality pseudolabels in DB from a better-trained s1. Overall, the above results indicate that adequately large
distillation sets are needed at every stage of MCKD to optimize the performance of the final student. In the original, more streamlined implementation in Section 2, we employ a pair of students at each stage to pseudolabel examples, so that the final student can take advantage of the most data available.

heading: 6 Related Work
text: Knowledge distillation (KD), apart from being used for model compression (Tang et al., 2019; Jiao et al., 2020; Bucila et al., 2006; Sun et al., 2020), has shown effectiveness in semi-supervised learning (Iliopoulos et al., 2022), where a teacher model annotates unlabeled data to train a student model. We study low-resource semi-supervised sequence generation with KD. We use hard pseudolabels from teachers to train student models,
following common practice in sequence-level knowledge distillation (Kim and Rush, 2016), including scenarios when the teacher is an LLM (Ding et al., 2022; Yoo et al., 2021; Wang et al., 2021b; Gilardi et al., 2023; Shridhar et al., 2022; Ho et al., 2022; Li et al., 2023).
Self-distillation (SD) through training the model with its self-generated pseudolabels is often valuable in enhancing generalization, but finding reliable confidence thresholds to filter the pseudolabels and ensure their quality is often tricky (Furlanello et al., 2018; Liu et al., 2021). Instead, we improve the pseudolabel quality over stages by crosspartition labeling, taking advantage of the better generalization abilities of models over stages.
Some past works also leverage several student models to improve the distillation performance. Zhang et al. (2018) and Guo et al. (2020) introduce several student models that learn from each other’s predicted logits. Song and Chai (2018) and Zhu et al. (2018) introduce extra branches of the student model (e.g., high-level layers or the classifier heads) to gather their multiple views as additional information for student training, which is shown helpful to improve the generalization. These works are mainly evaluated on image classification and do not analyze distillation from LLMs.
Co-training (Blum and Mitchell, 1998; Han et al., 2018; Lang et al., 2022) is relevant to our work since it involves using separate models to generate improved pseudolabels. In co-training, different learners are typically initialized with different views of the data. Then they provide confident predictions to each other for additional training. Wei et al. (2021) provides theoretical explanation on why generated pseudolables can be more credible than original labels for training. In contrast, our approach begins with an LLM teacher and trains better student models over stages.

heading: 7 Conclusion
text: Semi-supervised sequence generation in specialized low-resource settings can be challenging for NLP systems. We present the discovery that student models distilled from a few-shot prompted LLM can often correct the teacher’s errors, thus showing superior generalization, on such tasks. Exploiting this phenomenon along with strategies such as data partitioning and cross-partition labeling over multiple stages, we then propose MCKD, a novel solution that utilizes knowledge distillation
(KD) from an LLM, whose performance can be only at a mediocre level, to train much smaller yet high-accuracy students. MCKD essentially leverages progressively better models to train future generations with even better performance. Extensive empirical evaluation verifies the effectiveness and label efficiency of MCKD.

heading: 8 Limitations
text: We have studied the scenario of semi-supervised sequence generation where labeled data are too scarce to train a model while a few-shot prompted LLM exhibits room for improvement. We therefore use an LLM as the initial teacher for knowledge distillation (KD) and follow common practice in sequence-level KD by using hard labels for KD. Future work may explore the effectiveness of the proposed multistage KD with cross-partition labeling (1) from a non-LLM teacher and (2) using soft logits or other mechanisms for KD.
The GPT-3.5 Turbo teacher model in our experiments is a company hosted service and many details are missing about how it was implemented and which data it was trained on. Relatively more information is available about T5-based. For instance, it was pretrained on large amounts of news data; this may explain why its performance on PTB is relatively higher than that on CRAFT.

heading: A Dataset Details
text: In Table 2, we show examples of the contituency and semantic parsing tasks that we work with. We use the training and testing sets described in Section 3.1. We assume a low-resource scenario where a labeled validation set is unavailable; we find default parameters that are not tuned on datasetspecific validation sets are effective across datasets. For PTB we adopt the standard data splits.
Constituency parsing. The standard split for the test set of PTB contains 2,416 examples. For training, we randomly sample 20,000 examples as unlabeled training data. CRAFT contains 21,121 examples, but does not have standard splits; we randomly sample 50% examples as the training set and 40% examples as the test set. The remaining could be used for validation, although we assume a low-resource scenario where a labeled validation set is unavailable. For CRAFT, the test set contains 8,448 examples, while the unlabeled training set contains 10,560 examples.
Semantic parsing. For ATIS, the training set contains 4,478 utterances and the test set contains 893 utterances. There are 120 slot labels, i.e., labels for words. For Snips, the training set contains 13,084 utterances and the test set contains 700 utterances. There are 72 slot labels. We adopt dataset splits by Goo et al. (2018).

heading: B Prompting an LLM Teacher for Pseudolabel Generation
text: We employ the standard random selection approach (Brown et al., 2020) to choose in-context exemplars from available labeled data to prompt the LLM. The prompt format is shown in Table 3.
For task-oriented semantic parsing, we use the following instruction for ATIS and Snips respectively: “Predict tag for each word in the sentence. ‘O’ for unrelated word. Focus on words related to the flight information, such as locations, time and airline.”; “Predict tag for each word in the sentence. ‘O’ for unrelated word. Focus on words related to the music, restuarant, weather, book, playlist and searching.” We also preprocess the output of incontext exemplars to interleave each predicted tag with its according slot word so as to improve the prompting performance for LLMs. For instance, for the example shown in Table 2, the output when prompting LLM is formatted as “list O the O fares O of O US B-airline_name Air B-airline_name
flights from O Boston B-fromloc.city_name to O Philadelphia B-toloc.city_name”.

heading: C Statistical Significance of Results
text: We present the statistical significance of the results in Table 1. On each dataset, for the setting that uses 50 labeled examples, we perform a paired t-test to compare (1) the distribution of per-example F1 scores of predictions by the best KD baseline and (2) that of predictions by the 2-stage MCKD. Under the null hypothesis, the distribution of the perexample F1 scores achieved by the strongest baseline model is the same as the per-example F1 scores achieved by the 2-stage MCKD. In Table 4, we observe that the p-values are all far below the commonly used 0.05 threshold. Therefore, we reject the null hypotheses and conclude that the 2-stage MCKD is significantly better than the strongest baselines in the mentioned dataset settings.

heading: D Relation Between Training Fidelity and Generalization Ability
text: As discussed in Section 5.1, we find high training fidelity of a student model is generally correlated with better performance on held-out data, despite the unreliability of teacher labels. This section investigates whether the observation still holds with different scales of the distillation set, especially when the size is substantially smaller. Results are shown in Figure 8. Similar patterns across different scales of distillation sets can be observed. Even if the distillation set only contains 250 teacher pseudolabels, the observation holds true.

heading: Future Work
text: Future work may explore the effectiveness of the proposed multistage KD with cross-partition labeling (1) from a non-LLM teacher and (2) using soft logits or other mechanisms for KD. The GPT-3. 5 Turbo teacher model in our experiments is a company hosted service and many details are missing about how it was implemented and which data it was trained on. Relatively more information is available about T5-based. For instance, it was pretrained on large amounts of news data; this may explain why its performance on PTB is relatively higher than that on CRAFT