DisCo: Distilled Student Models Co-training for Semi-supervised Text Mining
Authors: Weifeng Jiang
Qianren Mao
Chenghua Lin
Jianxin Li
Ting Deng
Weiyi Yang
Zheng Wang
Venue: EMNLP 2023 Main (2023)
Keywords: co-training
semi-supervised learning
knowledge distillation
TL;DR: We present DisCo, a framework of co-training distilled students with limited labelled data, which is used for targeting the lightweight models for semi-supervised text mining.

Abstract: Many text mining models are constructed by fine-tuning a large deep pre-trained language model (PLM) in downstream tasks.
However, a significant challenge that arises nowadays is how to maintain performance when we use a lightweight model with limited labeled samples. 
We present DisCo, a semi-supervised learning (SSL) framework for fine-tuning a cohort of small student models generated from a large PLM using knowledge distillation.
Our key insight is to share complementary knowledge among distilled student cohorts to promote their SSL effectiveness. 
DisCo employs a novel co-training technique to optimize a cohort of multiple small student models by promoting knowledge sharing among students under diversified views: model views produced by different distillation strategies and data views produced by various input augmentations. 
We evaluate DisCo on both semi-supervised text classification and extractive summarization tasks.
Experimental results show that DisCo can produce student models that are $7.6\times$ smaller and $4.8 \times$ faster in inference than the baseline PLMs while maintaining comparable performance.
We also show that DisCo-generated student models outperform the similar-sized models elaborately tuned in distinct tasks.

DisCo: Distilled Student Models Co-training for Semi-supervised Text Mining
Weifeng Jiang Qianren Mao", " Chenghua Lin', Jianxin Li", Ting Deng', Weiyi Yang Zheng Wang'
SCSE, Nanyang Technological University, Singapore.
Zhongguaneun Laboratory, Beijing, P.R.China.
Department of Computer Science, University of Manchester, U.K.
School of Computer Science and Engineering, Beihang University, Beijing, P.R.China.
The School of Computing, University of Leeds, U.K.
$2200770e.nte itt..en..uuugu....
maoqrOzgclab.edr eed.e...laa.eu...r..... edu.cn
Abstract
al., 2021) and edge (Tambe al., 2021) devices. Therefore, maintaining the generalization ability of the reduced-sized model is crucial and feasi- ble (Sun et al., 2019; Sanh et al., 2019; Jiao et al., 2020; Wang et al., 2020).
Many text mining models are constructed by fine-tuning large deep pre-trained language model (PLM) in downstream tasks. However, significant challenge nowadays is maintaining performance when
lightweight model with limited labelled samples. We present DisCo, semi-supervised learning (SSL) frame- work for fine-tuning cohort of small student models generated from a large PI.M using knowledge distillation. Our key insight is to share complementary knowledge among dis- tilled student cohorts to promote their SSL ef- fectiveness. DiSCo employs novel co-training technique optimize a cohort of multiple small student models by promoting knowledge shar- ing among students under diversified views: model views produced by different distillation strategies and data views produced by various input augmentations. We evaluate DisCo both semi-supervised text classification and ex- tractive summarization tasks. Experimental re- sults show that DisCo can produce student mod- els that are 7.6x smaller and 4.8x faster infer- ence than the haseline PLMMs while maintaining comparable performance. We also show that DisCo-generated student models outperform the similar-sized models elaborately tuned in distinct tasks.
Semi-supervised learning (SSL) emerges as a practical paradigm to improve model generaliza- tion by leveraging both limited labelled data and extensive unlabeled data (Rasmus et al., 2015; Lee et al., 2013; Tarvainen and Valpola, 2017; Miy- ato et al., 2019; Berthelot et al., 2019; Sohn et al., 2020; Fan al., 2023; Zhang et al., 2021; Berth- elot et al., 2022; Zheng al., 2022: Yang et al., 2023). While promising, combining SSL with a reduced-size model derived from PLMs still neces- sitates a well-defined learning strategy to achieve improved downstream performances (Wang et al.. 2022a). This necessity arises because these shal- low networks typically have lower capacity, and the scarcity of labeled data further curtails the model's optimization abilities. Besides, major hurdle is lack of labelled data samples a particular prob- lem for text mining tasks because the labelling text labour-intensive and error-prone (Gururangan et al., 2019; Chen et al., 2020; Xie et al., 2020; Lee et al.. 2021; Xu et al., 2022: Zhao et al., 2023).
This paper thus targets using SSL to leverage distilled PLMs in a situation where only limited labelled data is available and fast model inference is needed on resource- resource-constrained devices. To this end, we use the well-established teacher-student knowledge distillation technique to construct small student models from teacher PL.M and then fine- tune them in the downstream SSL tasks. We aim to improve the effectiveness of fine-tuning small student models for text-mining tasks with limited labelled samples.
1 Introduction
Large pre-trained language models (PLMs), such BERT (Devlin et al., 2019), GPT-3 (Brown et al., 2020), play a crucial role in the development of nat- ural language processing applications, where one prominent training regime is to fine-tune the large and expensive PL.Ms for the downstream tasks of interest (Jiao et al., 2020).
Minimizing the model size and accelerating the model inference are desired for systems with lim- ited computation resources, such as mobile (Liu
We present DisCo, a novel co-training approach aimed enhancing the SSL performances by using distilled small models and few labelled data. The student models in the DisCo acquire complemen-
Internship achievements at Zhongguancun Laboratory. Qianren Mao is the corresponding author.

---

tary information from multiple views, thereby im- proving the generalization ability despite the small model size and limited labelled samples. we intro- duce two types of view diversities for co-training: i) model view diversity, which leverages diversi- fied initializations for student models in the cohort, ii) data view diversity, which incorporates varied noisy samples for student models in the cohort. Specificallyy, the model view diversity is generated by different task-agnostic knowledge distillations from the teacher model. The data view diversity is achieved through various embedding-basee based data augmentations to the input instances.
2
Methodology
2.1 Overview of DisCo
DiSCo jointly trains distilled student cohorts to im- prove model effectiveness in a complementary way from diversified views. As working example, we explain how use a dual-student DisCo train two kinds of student models (see Figure 1). Ex- tension to more students is straightforward (see section 2.3). To this end, DisCo introduces two ini- tialization views during the co-training process: (i) model views which are different student model vari- ants distilled from the teacher model, and (ii) data views which are different data augmented instances produced by the training input.
Intuitively, DisCo with the model view encour- ages the student models to learn from each other interactively and maintain reciprocal collabora- tion. The student cohort with the model views increases each participating model's 's posterior en- tropy (Chaudhari et al., 2017; Pereyra et al., 2017; Zhang al., 2018), helping them converge to flatter minimum with better generalization. At the same time, DisCo with the data views regularizes student predictions to invariant when applying noises to input examples. Doing so improves the models' robustness on diverse noisy samples gen- erated from the same instance. This, in turn, helps the models to obtain missing inductive biases on learning behaviour, i.e., adding more inductive bi- ases to the models can lessen their variance (Xie al., 2020; Lovering al., 2021).
In DisCo, two kinds of compressed students (rep- resented by two different colours in Figure (()) are generated by the same teacher. This process al- lows us to pre-encode the model view specifically for DisCo. Additionally, we duplicate copies of single student model to receive supervised and unsupervised data individually. In the supervised learning phase, DisCo optimizes two students ing labelled samples. In the unsupervised learning phase, each student model concurrently shares the parameters with its corresponding duplicate, which is trained by supervised learning. The subsequent consistency training loss then optimizes the stu- dents using unlabeled samples.
For an ablation comparison of DisCo, we intro- duce the variant of DisCo only equipped with the model view, shown in Figure 1(b). In this vari- ant, labelled and unlabeled data are duplicated and would be fed the students directly. DisCo and its variant ensure reciprocal collaboration among the distilled students and can enhance the generaliza- tion ability of the student cohort by the consistency constraint. In this section, we introduce DisCo from two aspects: knowledge distillation and the co- co-training strategy.
We have implemented working prototype of DisCo' and applied it to text classification and ex- tractive summarization tasks. We show that by co- training just two student models, DisCo can deliver faster inference while maintaining the performance level of the large PLM. Specifically, DisCo can produce a student model that is 7.6x smaller (4- layer TinyBERT) with 4.8x faster inference time by achieving superior ROUGE performance in extractive summarization than the source teacher model (12-layer BERT). It also achieves a better comparable text classification performance com- pared to the previous oate-of--hheann (SOTA) SSL methods with 12-layer BERT while maintaining a lightweight architecture with only 6-layer Tiny- BERT. We also show that DisCo substantially out- performs other SSL baselines by delivering higher accuracy when using the same student models in model size.
2.2 Student Model Generation
Our current implementation uses knowledge dis- tillation to generate small-sized models from a PLM. Like the task-agnostic distillation of Tiny- BERT² (Jiao et al., 2020), we use the original BERT without fine-tuning as the teacher model to generate the student models (In most cases, two stu- dent models at least are generated in our implemen- tation). The task-agnostic distillation method is
'Cnde and data are available at: https://github.. Li teSSLHub/DDscCo
httpp:: http:://gtttubbcccce cwt/cccm//////eeeee homm/huawei-noahy Pretrained Eaetrained-Langugee Model/ooeeeeeeee /ttp///eeee/ateeeiin/tttt

---

Teacher
Teacher
RD
kD
4
fx
Student
logits
Student
logits
Data
share
share
Augmentationn
Student
logits
Student
logits
Labeled
Unlabeled x²
Labeled (3.9)
Unlabaled xx
Ey
03.9
Student
logits
Student
logits
share
share
Augmentation
Student
logits -
Student
logits -
L
(a) model-view (V) data-view (V)
(b) model-view (V/) & data-view (X)
Figure I: The training architecture of DisCo (a) and the ablation variant (b). v refers to 'DO USE' the and x is 'DO NOT USE" LL, is supervised loss and L. is unsupervised. 'KD' is an abbreviation for knowledge distillation.
convenient for using any teacher network directly.
2.2.2 Data View Encoding
We use large-scale general-domain corpus of
We use different data augmentation strategies at the token embedding layer to create different data views from the input samples. Our intuition is that advanced data augmentation
WikiText- 1033 released by Merity al. (2017) the training data of the distillation. The stu- dent mimics the teacher's behaviour through the representation distillation from BERT layers: (i)
introduce inductive biases since they are based on ran- dom sampling at the token embedding layer with minimal semantic impact (Xic et al., 2020; Wu et al., 2020; Yan et al., 2021; Gao et al., 2021). Inspired by ConSERT (Yan et al., 2021) and Sim- CSE (Gao et al., 2021), we adopt convenient data augmentation methods: adversarial attack (Kurakin et al., 2017), token shuffling (Lee al., 2020), cut- off (Shen et al., 2020) and dropout (Hinton et al., 2012), described as follows.
the output of the embedding layer, (ii) the hidden states, and (ii)) attention matrices.
2.2.1 Model View Encoding
To ensure the grouped students present different view of the teacher, we distil different BERT layers from the same teacher. Model view encoding diver- sifies the individual student by leveraging different knowledge of the teacher. We propose two different strategies for the knowledge distillation process: (i) Separated-layer KD (SKD): the student learns from the alternate k-layer of the teacher. For instance, [3, 6, 9, 12) the 4 alternate layers of BERT. (ii) Connected- layer KD (CKD): the student leamss from the continuous K-layer of the teacher. For example, (1, 4) are the continuous 4 layers of BERT. In the case of dual-student DisCo, the two students with two kinds of knowledge distillation strategies are represented as AA and SBK The co-training framework will encourage the distinct individual model to teach each other in complementary man- underlying model view initialization.
Adversarial Attack (AD). We implement it with Smoothness-Inducing Adversarial Regularization (SIAR)' (Jiang et al., 2020), which encourages the model's output not to change too much when a small perturbation is injected to the input.
Token Shufiling (TS). This strategy is slightly sim- ilar to Lee et al. (2020) and Yan et al. (2021), and implement it by passing the shuffled position IDs the embedding layer while keeping the order of the token IDs unchanged.
Cutoff (CO). This method randomly erases some tokens for token cutoff in the embedding matrix.
With consistency constraints, our co-training framework can obtain valid inductive biases on model views, enabling student peers teach each other and to generalize unseen data. Apart from the model views, we also introduce data views pro- duced by various data augmentations of inputs to expand the inductive biases.
Dropout (DO), As same as in BERT, this scheme randomly drops elements by a specific probability
and sets their values to zero.
DISCo incorporates two forms of data view dur- ing co-training: a HARD FORM and SOFT FORM. Taking dual-student networks for example, we use
"The adversarial perturhed embeddings are generated in the AD strategy by maximizing the supervised loss.
"https://hugginggaee co wt.ddatasets//iiiitt......

---

Datasets
Label Type
#Classes
#Labeled
#Unlabelled
#Dev
#Test
287.227
Mail
Extractive Sentences
3
10/100/11000
13,368
11,490
-10/-1000-1,000
Agnews
News Topic
4
x10/ x 30/ X 100
20,000
8,000
7.600
Answer
Q&A Topic
10
x10/ x 30/ X 100
50,000
20,000
59,727
DBpedia
Wikipedia Topic
14
x10/ x 30/ x 100
70,000
28,000
70,000
Dataset statisties and dataset split of the semi-supervised extractive summarization dataset and several typical semi-sut supervised text classification datasets, in which 'x' means the number of data class.
Table I:
different data augmentation approaches, such as AD and DO, implement the HARD FORM data view. Regarding the SOFT FORM data view, we ap- ply the same data augmentation approach, includ- ing AD with two rounds of random initialization to ensure distinct views. In DisCo, each student ob- tains perturbation differences through the various combinations of the HARD FORM and SOFT FORM.
Square Error (MSE) encourage the two students to predict similarly:
= (3)
(4)
Overall Training Objective. Finally, we combine supervised cross-entropy loss with unsupervised consistency loss and train the model by minimizing the joint loss:
2.2.3 Co-training Framework
Formally, we are provided with a semi-supervised dataset D, D = S S U. S = {(t,)) is labelled data, where (i,9) will be used for two kinds of students identically. u = (x") is unlabeled data, and two copies made for two kinds of students identically. For X D, let (X) and (X) denote the two data views of data X. A pair of models ASK = fa and B f8) are two distilled student models which we treat as the model view of dual- student DisCo. Student fa only uses (X),, and Student fa uses (x()).
=s= Lin Lyy + n), d. (Lunu + Laga). (5)
where u(t,n) min( 1). It represents the ramp- up weight starting from zero, gradually increas- ing along with linear curve during the initial n training steps. d the hyperparameter balancing supervised and unsupervised learning.
2.3 Co-training of Multi-student Peers
By training collaboratively with the cohort of students fA and fa, the co-training optimization ob- jective allows them share the complementary information, which improves the generalization ability of network.
So far, our discussion has been focused on training two students. DisCo can be naturally extended to support not only two students in the student co- hort but more student networks. Given K networks
Supervised Student Cohort Optimization. For supervised parts, we use the categorical Cross- Entropy (CE) loss function for optimizing student Sa and student fB, respectively. They are trained with the labeled data (t. 9) sampled from
1 O2, .... Ox 2 2), the objective function for optimising all Oz, (1 (1sk s K), becomes:
K
Lo =
Lyg + p(t,n) Lmma), (6)
Ls (E)
(1)
K
Lsp CE)(f(((((())
(2)
Lu K
Σ ()))) (7) 1=1.
Unsupervised Student Cohort Optimization. In standard co-training, multiple classifiers pected to provide consistent predictions unla- beled data x' U.
Equation (5), is now a particular case of (6) with = 2. With more than networks in the cohort, learning strategy for each student of DisCo takes the ensemble of other K-1 student peers to provide mimicry targets. Namely, each student learns from all other students in the cohort individually.
The consistency cost of the unlabeled data x" is computed from the two student output log- its: 2x(dx(x")) and <x(()(xx))) We use the Mean

---

Agnews
Aaswer
DBpedia
Models
P
Avg
10
30
200
10
30
200
10
30
200
BERTBASE
109.48M
81.00
84.32
87.24
60.10
1.
69.28
96.59
98.21
98.79
82.18
UDA
109.48M
84.70
86.89
88.56
64.28
67.70
69.71
98.13
98.67
98.85
84.17
TinyBERT
66,96M
71.45
82.46
87.59
52.84
60.59
68.71
96.89
98.16
98.65
79.70
UDAmmauuet
66,96M
73.90
85.
87,54
57.14
62.86
67.93
97.41
97.87
98.26
81.79
DisCo (84))
66,96M
74.38
86.39
88. 70
57.62
64.04
69.57
98.50
98.45
98.57
82.02
DisCo (8
66.96M
77.45
86.93
88.82
59.10
66.58
69.75
98.57
98.61
98.73
82.73
TinyBERT
14.35M
69.67
78.35
85.12
42.66
53.63
61.89
89.65
96.88
97.58
75.05
UDA unyuurrr
14.35M
69.60
77.56
83,60
40.69
55.43
63.34
88.50
93.63
95,98
74.26
DisCo (8
14.35M
76.90
85.39
87.82
51.48
62.36
68.10
94,02
98.13
98.56
80.31
DisCo (S
14.35M
77.36
85.55
87.95
51.31
62.93
68.24
94.79
98.
98.63
80.54
FLiText
9.60M
67.14
77.12
82.12
48.30
57.01
63.09
89.26
94.04
97.01
75.01
DisCo (842)
8.90M
70.61
81.87
86.08
48.41
57.84
64.04
89.67
96.06
97.58
76.90
DisCo (8
8.90M
75.05
82.16
86.38
51.05
58.83
65.63
89.55
96.14
97.70
78.05
Table 2: Text classification performance (Ace (9)) on typical semi semi-supervvsed text classification tasks. P is the number model parameters. The best results are in-bold.
3 Experiments
Appendix A.2. We run each experiment with three random seeds and report the mean performance on test data and the experiments are conducted on single NVIDIA Tesla V100 32GB GPU.
Datasets. We evaluate DisCo extractive sum- marization and text classification tasks, shown in Table For extractive summarization, we use the CNN/DailyMail (Hermann et al., 2015) dataset, training the model with 10/100/1000 labeled ex- amples. Regarding text classification, we evalu- ate on semi-supervised datasets: Agnews (Zhang et al., 2015) for News Topic classification, Ya- hoo!Answers (Chang et al., 2008) for Q&A topic classification, and DBpedia (Mendes et al., 2012) for WikiPedia topic classification. The models are trained with 10/30/200 labeled data per class and 5000 unlabeled data per class. Further details on the evaluation methodology are in Appendix A.3.
Competing Baselines. For text classification tasks, we compare DisCo with: (i) supervised baselines, BERT and default TinyBERT (Jiao et al., 2020), (ii) semi-supervised UDA (Xie et al., 2020) and FLiText (Liu et al., 2021). We also compare with other prominent SSL text classification meth- ods and report their results the Unified SSL Benchmark (USB) (Wang et al., 2022a). Most of these SSL methods work well on computer vision (CV) tasks, and Wang al. (2022a) generalize them to NLP tasks by integrating a 12-layer BERT, More detailed introductions are given in Appendix A.4. For extractive summarization tasks, we com- pare: (i) supervised basline, BERTSUM (Liu and Lapata, 2019), (ii) two SOTA cemi-supervised ex- tractive summarization methods, UDASUM and CPSUM (Wang et al., 2022b), (iii) three unsuper- vised techniques, LEAD-3, TextRank (Mihalcea and Tarau, 2004) and LexRank (Erkan and Radev, 2004). We use the open-source releases of the com- peting baselines.
Implementation Details. The main experimental results presented in this paper come from the best model view and data view we found among multi- ple combinations of view encoding strategies. Tak- example, present the results of AK and SBK as the model-view being a combination of SKD (alternate K-layer) and CKD (continuous K- K-layer). The data view the SOFT FORM of two different AD initialization. Specifically, DisCo (S^6) uses CKD model-view of BERT layers [1, 2, 3, 4, 5, 6/ and SOFT FORM data- view of AD. DisCo (6))) uses SKD midel-view of BERT layers (2, 4, 6, 8, 10, 12) and SOFT FORM data-view of AD. DisCo (SA+ and SB4) use similar combinations the DisCo (SA6 and SB6). DisCo A2) uses CKD with [1, 2) BERT layers and DisCo B2) uses CKD with (11, 12) BERT layers. The de- tails of DISCo's hyperparameter are presented in
ing
DisCo
4 Experimental Results
4.1 Evaluation on Text Classification
As shown in Table the two students produced by DisCo with 6-layer distilled BERT AS and 5B6) consistently outperform TinyBERT and UDATHByERR in all text classification tasks. More- over, one student of our stul-sttdent 6-layer DisCo

---

Table 3: Text classification performance (Ace (%)) of other prominent SSL text classification models and all results reported by the Unified SSL Benchmark (USB) (Wang al., 2022a). D refers to datasets, Lm the number of the BERT layers used by models and od is labeled data per class.
Table 4: ROUGE F1 performance of the extractive sum- marization. I0=100 refers the labeled data per class. SSL baselines (CPSUM and UDASUM) use the same unlabeled data as DisCo has used.
CNN/DailyMail
Models
Ld
R-1
R-2
R-L
D
Models
L
L
Acc
ORACLE
100
48.35
26.28
44.61
[7 []-model (Rasauus al., 2015)
50
86.56
LEAD-3
100
40.04
17.21
36.14
P-Laheling (Lee et 2013)
50
87.01
TextRank
100
33.84
13.11
23.98
Agnews
MeanTeacher (Tarvainen und Valpola, 2017)
50
86.77
LexRank
100
34.63
12.72
21.25
PCM (Xa et 2022)
12
30
88.42
BERTSUM
109.48M
100
38.58
15.97
34.79
MixText (Chen 2020)
30
87,40
CPSUM
109.48M
100
38.10
15.90
34.39
DisCo (ours)
30
86.93
UDASUM
109.48M
100
38.58
15.87
34.78
AdaMatch [Betholot 2022)
200
69.18
Tiny TERSSERS
14.35M
100
39.83
17.24
35.98
YahoolAnswer
CRMactch (Fan al., 3023)
200
69.38
Tiny BERTSUMU
BERSSERS
14,35M
14.35M
100
100
39.88
40.06
17.14
17.32
36.00
36.18
SimMatch (Zheng et al., 2022)
69.36
UDASUM BiyBBarr
14.35M
100
40.11
17.43
36.23
FlexMatch (Zhong 2021)
12
200
68.58
UDASUM MinyBaarps
14,35M
100
39.90
17.25
36.05
VAT (Miyato et 2019)
200
68.47
UDASUM UDASUMT1GSEERRT TiGBERRT
14.35M
100
40.11
17.34
36.19
Teahhe (Tarvainen Valpols, 2017)
200
66.57
DisCo (SM)
14.35M
100
40.39
17.57
36.47
DisCo (ours)
6
200
69.75
DisCo (SU4)
14.35M
100
40.41
17.59
36.50
PCM (Xu 2022)
10
98.70
DBpedia
Mixtext (Chen 2020)
10
98.39
12
VAT (Miyato al,, 2019)
10
98,
48.40
Table 5: Model efficiency about the model size and inference speedup on single NVIDIA Tesla V100 32GB TPU.T refers the speedup extractive summarization models trained with 100 labeled data. T (ms) illustrates the speedup of text classification models trained with Agnews 200 labeled data per class.
DisCo (ours)
10
98.57
outperforms the 12-layer supervised BERTBASE by 0.55% average improvement in accuracy. These results suggest that DisCo provides simple but effective way to improve the generalization ability of small networks by training collaboratively with a cohort of other networks.
Models
T Tis(ms)
Models
T Tmc(m))
BERTSUM CPSUM
12.66
BERT
12.94
12.66
TinyBERT
2.86
TEnBERTSUM
2.64
UDATIBBBRTT
2.86
Compared with the FLiText, DisCo i the average classification accuracy by 1.9% while using a student model with 0.7M fewer parame- ters than FLiText. FLiText relies heavily on back- translation models for generating augmented data, similar to UDA. Unfortunately, this strategy fails to eliminate error propagation introduced by the back-translation model and requires additional data pre-processing. Besides, FLiText consists of two training stages and needs supervised optimization in both stages, increasing training costs and exter- nal supervised settings.
UDASUM UDDSUMMT,RRRRT Bny
2.64
FLiText
1.56
improves
DiSCo (SM or SB4)
2.64
DisCo (S or S
1.72
based DisCo achieves better performance than all 12-layer BERT-based SSL benchmarks. These re- sults demonstrate that our model has superiority certain scenarios of the lightweight model architec- ture and limited manual annotation.
4.2 Evaluation on Extractive Summarization
Table 3 shows results when comparing DisCo to other prominent SSL methods which are in- tegrated with 12-layer BERT. We take the re- sults from the source publication Unified SSL Benchmark (USB) (Wang et al., 2022a) for these baselines. However, most of them perform worse than DIsCo's students only with a 6-layer BERT using same labeled data. In the case of Ya- Ano!Ar text classification, our 6-layer BERT-
For the semi-supervised extractive summarization tasks, our dual- DISCo outperforms all base- lines in Table 4. Despite using smaller-sized, 4-yayer model, DisCo performs better than the 12- layer BERTSUM, UDA, and CPSUM. The results show that our methods can reduce the cost of su- pervision in extractive summarization tasks. Other ROUGE results with 10 or 1000 labeled examples are presented in Appendix A.5.

---

Table 6: Text classification performance (Ace (%)) of
Table 7: Performance comparison between DisCo and a single student model with AD augmentation. The 'Sin- gleStudent' is the better-performen model among the two students within the DisCo framework.
DisCo with multiple student peers. The students (SA2,
S S SD2) are distilled from layers (1, (3, 4), (9, 10) and {11, 12) of the teacher BASE. respec- tively. The first four students adopt HARD FORM data views which are AD, DO, TS, and CO, respectively. The last four students adopt SOFT FORM data view with dif- ferent DO initialization. Better results than dual-student DisCo Table 2 is in-bold.
Models
Agmews
YasoolAnswer
DBpedia
SingleStudent"
10
73.52
55,43
93.65
DisCo (S*6)
10
74.38
57.62
98.50
DisCo (S
10
77.45
59.10
98.57
SingleStudent
10
75.49
47.57
89.30
Models
Ld
Agnews
YasoolAnswer
DBpedia
DisCo (SM)
10
76.90
51,48
94.02
DisCo (S*
200
87.58
66.74
98.23
DisCo (St4)
10
77.36
51.31
94.79
DisCo (S#3)
200
87.41
66.28
98.33
SingleStudent
10
68.79
48,87
77.26
DisCo )
200
87.83
65.63
97.69
DisCo (SA2)
10
70.61
48.41
89.67
DisCo (S02)
200
87.59
65.87
98.34
DisCo (S
10
75.05
51.05
89.55
DisCo (S*3)
200
86.99
65.71
98.10
DisCo (St))
200
86.71
64.01
98.18
DisCo (S
200
86.79
63.96
98.12
DisCo (S"2)
200
86.63
63.83
98.01
(multi-students) in the co-training process yields a considerable performance enhancement compared to less populous student group (dual-students).
4.3
Model Efficiency
As shown in Table 5, compared with the teacher BERT all 4-layer student models give faster inference time by speeding up the inference by 4.80x-7.52x for the two tasks. FL.iText slightly faster than the smaller model generated DisCo. This is because FLiText uses a convolutional net- work while our student models use BERT with multi-head self aelf-attention. The lower computational complexity of convolutional networks However, despite the FLiText having more parameters, it gives worse performance (about 3.04% accuracy defects on average), as shown in Table 2.
4.4.2 Effect of using Multi-View Strategy
As shown in Table 8, DisCo composed of the stu- dent networks distilled from the teacher is obvi- ously superior to DisCo composed of two randomly initialized student networks, which verifies the ad- vantage of our model view settings. DisCo, the data view of SOFT FORM and HARD FORM brings the best effect, namely combinations of DO and AD encoded data view. Other data views with combi- nations of TS and CO yielded sub-optimal effects, which are presented in Appendix A.5. Under the same model view, DisCo integrating with the SOFT FORM data view is slightly better than the one using HARD FORM data view. The observations indicate adversarial perturbations are more useful for dual- student DISCo. Modelling the invariance of the internal noise in the sentences can thus improve the model's 's robustness.
4.4
Ablation Studies
4.4.1
Effect of using Multi-student Peers
Having examined the dual-studem DisCo in prior experiments, our next focus explore the scala- bility of DisCo by introducing more students in the cohort. As the results are shown in Table 6, we can see that the performance of every single student improves with an extension to four students in the DisCo cohort, which demonstrates that the gener- alization ability of students is enhanced when they learn together with increasing numbers of peers.
Further, we plot the training loss contour of DisCo and its ablation model Figure 2. Both models have a fair benign landscape dominated by a region with convex contours in the centre and no dramatic non-convexity. We observe that the optima obtained by training with the model view and the data view are flatter than those obtained only with a model view. flat landscape implies that the small perturbations of the model param- eters cannot hurt the final performance seriously, while a chaotic landscape is more sensitive to subtle changes (Li et al., 2018).
Besides, the results in Table have validated the necessity of co-training with multiple students. is evident that a greater number of student peers
'The ID-CNN requires O(k x x d) operations used by FLiText. In contrast, the multi-head self-attention mechanism of BERT requires O(n² x n x d operations, where a is the sequence length, is the representation dimension, & is the kernel size of convolutions.

---

Table 8: The impact of incorporating multi-view encoding for the dual-student DisCo. The HARD data-view is created using dropout (DO) and adversarial attack (AD). The SOFT view employs adversarial attack (AD) with varying initialization. The model-view (V) refers to that students are trained from scratch without any teacher knowledge.
CNV/Daily Mail w. 100
Agnews w.
10
YaboolAnseer
10
model view
data view
54,
SM, RR-LL
SM, ACCC
ACC
SM, ACC
5tt ACC
X
36.74
36.69
14.15
14.12
32.91
32.86
37.51
36.76
22.23
21.62
SMyS
39.96
39,93
17.23
17.23
36.07
36.06
73.18
73.62
52.56
52.95
SMigM
40.06
40.09
17.30
17.33
36,18
36.19
74.06
73.51
51.44
50.16
X
HARD
17.36
36.26
53.80
SM
40.28
40.24
17.46
17.46
36.37
36.33
77.45
77.77
56,22
35.43
SM
40.16
40.13
17.39
17.37
36.26
36.23
77.28
76.20
51.66
51.77
X
SOFT
SM
40.32
40.31
17.52
17.52
36.41
36.40
77.18
77.45
55.76
55.44
275
0.23
23
0.75
0.75
35
225
232
3.0
0.0
35
-279
-1.7
-05
(a) model-viow (f/)& data-view (b) model-view (f/) & data-view ) (c) model-viow (r)) data-view (X) (d) model-viow (f)) data-view
Figure 2: 2D visualization of the loss surface contour of DisCo (w. model view and data view) and its ablation variant (w. model view). Subfigures (a) and (b) are the text classification tasks for Agnews dataset with 10 labeled data per class. Subfigures (c) and (d) are extractive summarization tasks with 100 labeled data.
Table 9: Performance comparison (Ace (%) of the back- translation (BT) and Adversarial Attack (AD) augmenta- tion methods within UDA and FLiText frameworks.
augmentation strategy, a technique distinctly dif- ferent from the token embedding level data aug- mentation employed in DisCo framework. To ensure balanced comparison, substituted the back translation approach with our augmenta- tion method for UDA/FLiText. The outcomes of this modification are portrayed Table 9.
Models
Aug
L
Agnews
YahoorAnswer
DBpeida
BT
10
73.90
57.14
97.41
UDA Boy
AD
10
61.20
52.29
88.76
BT
10
67.14
48.30
89.26
FLiText
AD
10
65.15
48.06
85.17
These results underscore that regardless of the data augmentation strategy implemented, the per- formance of both UDA and FLiTeet falls short com- pared to DisCo framework. This substantiates our claim that our co-training framework is superior in distilling knowledge encapsulated in unsuper- vised data. Furthermore, the performance across most tasks experiences decline after the augmen- tation technique alteration. As stipulated (Xie et al., 2020), the UDA/FLiTex framework necessi- tates that augmented data maintain 'similar seman- tic meanings' thereby making back-translation a more suitable for UDA/FLiText, compared to the AD augmentation incorporated.
4.5
Discussion
Single Student with AD Augmentation To demonstrate the necessity of multi-student co- training, compare the single-student model without co-training with AD data augmentations. Naturally, the single model exclusively uses super- vised data, missing out on leveraging unsupervised data. A noteworthy performance decline is ob- served in Table 7 and most differently sized models in DBpedia suffer noticeable performance drops. These results validate the DisCo framework's 's effi- cacy under co-training optimization.
4.5.1
Conclusion
4.5.2 UDA/FLiText with AD Augmentation
In this paper, we present DisCo, framework of co- training distilled students with limited labelled data, which is used for targeting the lightweight
the preceding analysis detailed in Table 2, UDA/FLiText utilized back translation as their data

---

models for semi-supervised text mining. DisCo leverages model views and data views to improve the model's 's effectiveness. We evaluate DisCo by applying it to text classification and extractive sum- marization tasks and comparing with a diverse set of baselines. Experimental results show that DisCo substantially achieves better performance across scenarios using lightweight SSL models.
8-13 2014, Montreal, Quebee, Canada, pages 2654- 2662.
David Berthelot, Nicholas Carlini, lan J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. 2019. Mixmatch: A holistic approach to semi- supervised learning, In Advances in Neural Infor- mation Processing Systems 32: Annual Conference on Neural Information Processing Systems, NewrIPS, pages 5050-5060.
6 Limitations
David Berthelot, Rehecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alexey Kurakin. 2022. Adamatch: A unified approach to semi-supervised learning and domain adaptation. In The Tenth Inter- national Conference on Learning Representations, ICLR 2022. OpenReview.net.
Naturally, there is room for further work and im- provement, and we discuss few points here. In this paper, we apply DisCo to BERT-based stu- dent models created from the BERT-based teacher model. would be useful to evaluate if our ap- proach can generalize to other model architectures like TexICNN (Kim, 2014) and MLP-Mixer (Tol- stikhin et al., 2021). It would also be interesting to extend our work to utilize the inherent knowledge of other language models (e.g.. RoBERTa (Liu et al., 2019), GPT (Radford al., 2018; Radford et al.; Brown et al., 2020), T5 (Raffel et al., 2020)).
Avrim Blum and Tom M. Mitchell. 1998. Combining labeled and unlabeled data with co co-training. In Pro- ceedings of the Eleventh Annual Conference on Com- putational Learning Theory, COLT, pages 92-100. ACM.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz I Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Pro- cessing Systems 2020, NeurIPS.
Another limitation of our framework settings is the uniform number of BERT layers in all distilled student models. To address this, students in DisCo can be enhanced by introducing architectural diver- sity, such as varying the number of layers. Previous studies (Mirzadeh et al., 2020; Son et al., 2021) have demonstrated that a larger-size student, acting as an assistant network, can effectively simulate the teacher and narrow the gap between the student and the teacher. We acknowledge these limitations and plan to address them in future work.
Ming-Wes Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance of semantic rep- resentation: Dataless classification, In Proceedings of the Twenty-Thin AAAI Conference on Artificial Intelligence, AAAI, pages 830-835. AAAI Press.
7 Ethical Statement
The authors declare that we have no conflicts of interest. Informed consent is obtained from all in- dividual participants involved in the study. This article does not contain any studies involving hu- man participants performed by any authors.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jen- nifer T. Chayes, Levent Sagun, and Riccardo Zecchina. 2017. Entropy-sgd: Biasing gradient de- scent into wide valleys. In 5th International Confer- ence on Learning Representations, ICLR. OpenRe- view.net.
Acknowledgements
Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mix- text: Linguistically sequistically-iffomeedd informed interpolation of hidden space for semi-supervised text classification. In Pro- ceedings of the 58th Annual Meeting of the Associ- ation.for Computational Linguistics, ACL, (Volume 1: Long Papers), pages 2147-2157. Association for Computational Linguistics.
This work is supported in part by the National Natu- ral Science Foundation of China (No.U20B2053).
References
Jimmy Ba and Rich Caruana. 2014. Do deep nets really need to be deep? In Advances in Neural Information Processing Systems 27: Annual Conference on Neu- ral Information Processing Systems 2014, December
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of

---

the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACI-HILT, Volume I (Long and Short Papers), pages 4171-4186, Association for Compu- tational Linguistics.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang. and Qun Liu. 2020. Tinybert: Distilling BERT for natural language under- standing. In Findings of the Association for Compu- tational Linguistics: EMNLP, volume EMNLP 2020 of Findings of ACL., pages 4163-4174, Association for Computational Linguisties.
Günes Erkan and Dragomir R. Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text sum- marization. J. Artif. Intell. Res., 22:457 479..97.
Taehyeon Kim, Jaehoon Oh, Nakyil Kim, Sangwook Cho, and Se- Young Yun. 2021. Comparing kullback- leibler divergence and mean squared error loss in knowledge distillation. In Proceedings of the Thir- tieth International Joint Conference on Artificial In- relligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 2628- 6628-2635. ij- cai.org.
Yue Fan, Anna Kukleva, Dengxin Dai, and Bernt Schiele. 2023. Revisiting consistency regularization for semi-supervised learning. Int. J. Comput. Vis., 13163):626-643.
Tommaso Furlanello, Zachary Lipton, Michael Tschan- nen, Laurent Itti, and Anima Anandkumar. 2018. Born again neural networks. In Proceedings of the 35th International Conference on Machine Learn- ing. volume 80 of Proceedings of Machine Learning Research, pages 1607-1616. PMLR.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference Empirical Methods in Natural Lan- guage Processing, EMNLP, pages 1746-1751. Asso- ciation for Computational Linguistics.
Tianyu Gao. Xingeheng Yao. and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. EMNLP, pages 6894-6910. Association for Compu- tational Linguistics.
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. 2017. Adversarial examples in the physical world. In 5th International Conference on Learning Represen- tations, ICLR, Workshop Track Proceedings. OpenRe- view.net.
Suchin Gururangan, Tam Dang, Dallas Card, and Noah A. Smith. 2019. Variational pretraining for semi-supervised text classification. In Proceedings of the 57th Conference of the Association for Compu- lational Linguistics, ACL. (Volume 1: Long Papers), pages 5880-5894, Association for Computational Linguistics.
Dong-Hyun Lee et al. 2013. Pseudo-label: The simple and efficient sumi-supervised learning method for deep neural networks. Workshop on challenges in representation learning, ICML,, volume 3, page 896.
Haejun Lee, Drew A. Hudson, Kangwook Lee, and Christopher D. Manning. 2020. SIM: learning discourse language representation with sentence un- shuffling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 1551-1562. Association for Compu- tational Linguistics.
Karl Moritz Hermann, Tomás Kociský, Edward Grefen- stette. Lasse Espeholt, Will Kay, Mustafa Suleyman. and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28: Annual Conference on Neu- ral Information Processing Systems, NeurlPS, pages 1693-1701.
Ju Hyoung Lee, Sang-Ki Ko, and Yo-Sub Han. 2021. Salnet: Semi-supervised few-shot text classifica- tion with iientioonbbsed lexicon construction. In Thirty-Fifth AAAI Conference on Artificial Intelli- gence, AAAI, pages 13189-13197. AAAI Press.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co- adaptation of feature detectors. arXiv preprint arXiv:1207.05a.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. 2018 Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems 31: Annual Conference on Neu- ral Information Processing Systems, NeurlPS, pages 6391-6401.
Geoffrey E. Hinton, Oriol Vinyals. and Jeffrey Dean. 2015. Distilling the knowledge in neural network. NIPS Deep Learning and Representation Learning Workshop.
Chin-Yew Lin and Eduard H. Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Human Language Technology Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics, HLT-NAACL. Association for Computational Linguistics.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xi- aodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. SMART: robust and efficient fine-tuning for pre- trained natural language models through principled regularized optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL, (Volume I: Long Papers), pages 2177-2190 Association for Computational Linguis- tics.
Chen Liu, Mengchao Zhang, Zhibing Fu, Panpan Hou, and Yu Li. 2021. Flitext: A faster and lighter semi- supervised text classification with convolution net- works. In Proceedings of the 2027 Conference on

---

Empirical Methods in Natural Language Processing, EMNLP. pages 2481-2491. Association for Compu- tational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, et al. 2018. Improving language under- standing by generative pre-training.
Yang Liu and Mirella Lapata. 2019. Text summariza-
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language mod- els are unsupervised multitask learners. OpenAl blog. 1(8).
tion with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing, EMMNL--JCNIE pages 3728-3738, Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21 (140):1-67.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. 2015. Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems 28: Annual Confer- ence on Neural Information Processing Systems 2015, NeuralPS, pages 3546-3554.
Charles Lovering, Rohan Jha, Tal Linzen, and Ellic Pavlick. 2021. Predicting inductive biases of pre- trained models. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event. Austria, May 3-7, 2021. OpenReview.net.
Mehdi Sajjadi, Mchran Javanmardi, and Tolga Tasdizen. 2016. Regularization with stochastic transformations and perturbations for deep supe-supe learning. In Advances in Neural Information Processing Sys- tems 29; Annual Conference on Neural Information Processing Systems, NeurIPS, pages 1163-1171.
Pablo N Mendes, Max Jakob, and Christian Bizer. 2012. DBpedia: A multilingual cross-domain knowledge base. European Language Resources Association (ELRA).
Stephen Merity, Caiming Xiong. James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. In 5th International Conference on Learning Representations, ICLR, ICLR,Confereet Conference Track Proceed- ings. OpenReview.net.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.0110S.
Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. 2020. A simple but tough-to-beat data augmentation approach for natural language un- derstanding and generation. CORR, abs/2009.1388 138888
Rada Mihalcea and Paul Tarau. 2004. Textrank: Bring- ing order into text. In Proceedings of the 2004 Con- ference on Empirical Methods in Natural Language Processing, EMNLP, pages 411- Association for Computational Linguistics.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. 2020. Fix- match: Simplifying semi-supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurIPS.
Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. 2020. Improved knowledge distilla- tion via teacher assistant. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI, pages 5191-5198. AAAI Press.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2019. Virtual adversarial training: A regularization method for supervised and semi- supervised learning. IEEE Trans. Pattern Anal. Mach. Intell., 1179)11979
Wonchul Son, Jaemin Na, Junyong Choi, and Wonjun Hwang. 2021. Densely guided knowledge distillation using multiple teacher assistants. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV, pages 9375-9384, IEEE.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. 2017. Regularizing neural networks by penalizing confident output distri- butions. In 5th International Conference on Learning Representations, ICLR. OpenReview.net.
Siqi Sun. Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model com- pression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Nat- ural Language Processing, EMNLP-IJCNLPP. pages 4322-4331. Association for Computational Linguis- tics.
Siyuan Qiao, Wei Shen, Zhishuai Zhang, Bo Wang, and Alan L. Yuille. 2018 Deep co- co-training for semi- supervised image recognition. In Proceedings of the 15th European Conference Computer Vision, ECCV, volume 11219 of Lecture Notes in Computer Science, pages 142-159. Springer.
Thierry Tambe, Coleman Hooper, Lillian Pentecost, Tianyu Jia, En-Yu Yang, Marco Donato, Victor Sanh, Paul N. Whatmough, Alexander M. Rush,

---

David Brooks, and YeoY Wei. 2021. Edge- bert: Sentence-level energy optimizations for latency- aware multi-task NLP inference. In MICRO '21: 54th Anmual IEEEJACM International Symposium on Microarchitecturee Virtual Event, Greece, October 18-22, 2021, pages 830-844 8400-84..44 ACM.
on Neural Information Processing Systems, NeurIPS, pages 6256-6286
Hai-Ming Xu, Lingqiao Liu, and Ebsan Abbasnejad. 2022. Progressive class semantic matching for semi- supervised text classification. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. NAACL, pages 3003- 3013. Association for Computational Linguistics.
Antti Tarvainen and Harri Valpola. 2017. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning re- sults. In 5th International Conference on Learning Representations, ICLR, Workshop Track Proceedings. OpenReview.net.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. Consert: A con- trastive framework for supe-supe sentence repre- sentation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing, ACL/JCNLP, ( (Vol- ume I: Long Papers), pages 5065-5075 Association for Computational Linguistics.
Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jes- sica Yung, Andreas Steiner. Daniel Keysers, Jakob Uszkoreit, Mario Lucic. and Alexey Dosovitskiy. 2021. Mlp-mixer: An all-mip architecture for vi- sion. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Infor- mation Processing Systems 2021, NeurIPS, pages 24261-24272.
Weiyi Yang, Richong Zhang. Junfan Chen, Lihong Wang, and Jacin Kim. 2023. Prototype-guided pscudo labeling for cemi-supervised text classifica- tion. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), pages Canada. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural In- formation Processing Systems, NeurlPS, pages 5998- 6008.
16382, Toronto.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang. Manabu Okumura, and Takahiro Shi- nozaki. 2021. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Ad- vances in Neural Information Processing Systems 34: Annual Conference on Neural Information Process- ing Systems, NeurIPS, pages 18408-18419.
Wenhui Wang. Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems, NeurlPS.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In Advances in Neural Information Pro- cessing Systems 28: Annual Conference on Neural Information Processing Systems, NeurlPS, pages 649- 657.
Yidong Wang. Hao Chen, Yue Fan, SUN Wang, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang, Zhi Zhou, Lan-Zhe Guo, et al. 2022a. Ush: A unified semi-supervised learning benchmark for classifica- tion. In Thitty-sixth Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track, NewrlPS.
Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. 2018. Deep mutual learning. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pages 4320-4328. Computer Vision Foundation / IEEE Computer Society.
Yiming Wang, Qianren Mao, Junnan Liu, Weifeng Jiang, Hongdong Zhu, and Jianxin Li. 2022b. Noise- injected consistency training and entropy-constraine pseudo labeling for semi-sure supervisee extractive sum- marization. In Proceedings of the 29th International Conference on Computational Linguistics, COLING, pages 6447-6456. International Committee on Com- putational Linguistics.
Kun Zhao, Bohao Yang, Chenghua Lin, Wenge Rong. Aline Villavicencio. and Xiaohui Cui. 2023. Eval- uating open-domain dialogues in latent space with next sentence prediction and mutual information. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers). pages 662-574, Toronto, Canada.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020. CLEAR: con- trastive learning for sentence representation. CoRR, Abs/2012.15546
Mingkai Zheng, Shan You, Lang Huang, Fei Wang. Chen Qian, and Chang Xu. 2022. Simmatch: Semi- supervised learning with similarity matching. In IEEE/CVF Conference on Computer Vision and Pat- tem Recognition, CVPR, pages 14451-14461. IEEE.
Qizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong, and Quoc Le. 2020. Unsupervised data augmentation for consistency training. In Advances Neural Infor- mation Processing Systems 33: Annual Conference

---

A Appendix
agreement sufficient unlabeled examples, the co-training framework finally achieves two accu- rate classifiers on each view semi-supervised manner (Qiao et al., 2018)
A.1 Background and Related Work
Knowledge Distillation (KD). The KD (Hinton et al., 2015) is one of the promising ways to trans- fer from powerful large network or ensemble to a small network meet the low-memory or fast execution requirements. BANs (Furlanello al., 2018) sequentially distill the teacher model into multiple generations of student models with iden- tical architecture to achieve better performance. BERT-PKD (Sun et al., 2019) distills patiently from multiple intermediate layers of the teacher model at the fine-tuning stage. DistilBERT (Sanh al., 2019) and MiniLM (Wang al., 2020) lever- age knowledge distillation during the pre-training stage. TinyBERT (Jiao et al., 2020) sets a two- stage knowledge distillation procedure that con- tains general-domain and tasks-specific distillation in Transformer (Vaswani et al., 2017). Despite their success, they may encounter difficulties affecting the sub-optimal performance in language under- standing tasks due to the trade-off between model compression and performance loss.
A.2 Hyperparameters
The BERT as the teacher model, has a total of 109M parameters (the number of layers N= 12, the hidden size 768, the forward size d' = 3072 and the head number h= 12). We used the BERT tok- enizer to tokenize the text. The source text's max sentence length is 512 for extractive summarization and 256 for text classification. For extractive sum- marization, we select the top 3 sentences according to the average length the Oracle human-written summaries. We use the default dropout settings in our distilled BERT architecture. The ratio of to- ken cutoff is set to 0.2, suggested in (Yan et al., 2021; Shen et al., 2020). The ratio of dropout is set to 0.1. Adam optimizer with B1 0.9, B2 0.999 is used for fine-tuning, We the learning rate le-4 for extractive summarization and 5e-3 for text classification, in which the learning rate warm-up is 20% of the total steps. The for balancing su- pervised and unsupervised learning is set to I in all our experiments. The supervised batch size is set to 4, and the unsupervised batch size is 32 for the summarization task (16 for the classification task) in our experiments.
Semi-supervised Learning (SSL). The majority of SSL algorithms are primarily concentrated in the field of computer vision, including Pseudo La- beling (Lee et al., 2013), Mcan Teacher (Tarvainen and Valpola, 2017), VAT (Miyato et al., 2019), Mix- Match (Berthelot et al., 2019), FixMatch (Sohn al., 2020), CRMatch (Fan et al., 2023), Flex- Match (Zhang et al., 2021), AdaMatch (Berthelot et al., 2022), and SimMatch (Zheng et al., 2022), all of which exploit unlabeled data by encouraging invariant predictions to input perturbations (Saj- jadi et al., 2016). The success of semi-supervised learning methods in the visual area motivates re- search in the NLP community. Typical techniques include VAMPIRE (Gururangam et al., 2019), Mix- Text (Chen al., 2020) and UDA (Xie et al., 2020). Under the low-density separation assump- tion, these SSL methods perform better than their sully-supervised counterparts while using only a fraction of labelled samples.
A.3 Evaluation Methodology
Extractive summarization quality is evaluated with ROUGE (Lin and Hovy, 2003). We report the full-length FI-based ROUGE-1, ROUGE-2, and ROUGE-L (R-1, R-2, and R-L), and these ROUGE scores are computed using 10UGE-1.5.55 5.pl script?. We report the accuracy (denoted as Acc) results in the text classification tasks.
A.4 Baselines Details
For the text classification task, TinyBERT (Jiao et al., 2020) a compressed model implemented by 6-layer or 4-layer BERTBASE- For cumi-super methods, we use the released code to train the UDA, which includes ready-made 12-layer BERTBASB. 6- layer, or 4-layer TinyBERT. FLiText (Liu et al.. 2021) is a lightweight and fast semi-supervised learning framework for the text classification task. FLiText consists of two training stages. It first
Co- To-Trinnig. It is classic award-winning method for semi-supervised learning paradigm, training (or more) deep neural networks com- plementary views (i.e., data view from different sources that describe the same instances) (Blum and Mitchell, 1998). By minimizing the error on limited labelled examples and maximizing the
https://gi http://ggttttthheeee wtw.hhuw...oooooogggeaeee reseerrhreeerrrrereeeeee thub. http::://gghhiu..cmm/mmeeepppppp

---

DisCo (5*6)
DisCo (S
Agnews 10
Yahoo! w.
DBpedia
w.
10
S
S86
S
S
S
S86
0xD:1.2.3.4.5,6
02D:7.8.9110.11.22
80.71
81.05
56.61
55.08
98.04
98.12
SKD:2,4,6.8.10.12
CKD.1.2.3.4,5,6
74.45
74.38
59.10
57.62
98.57
98.50
SKD:2,4,6.8.10,12
SKD:1,3,4,5,799111
79.67
80.13
56.50
56.73
97.84
97.79
Table 10: Text classification performance (Ace (%)) comparison with different combinations of model views in dual-student DisCo (6-layer TinyBERT as the students). SKD denotes the esparated-layer knowledge distillation and CKD denotes connected-laye knowledge distillation.
Table 11: ROUGE performance of models using 10 1000 labelled CNN/Daily Mail examples.
front of every sentence to represent each sen- tence and use their last hidden states to classify whether the sentence belongs to the summary. The SOTA semi-supervised extractive summarization model, CPSUM (Wang al., 2022b), combines the noise-injected consistency training and the entropy-
CNN/DailyMai
Models
T.,
L
R-1
R-2
R-L
CPSUM UDASUM
12
10
39.00
16.64
35.23
12
10
39.03
16.49
35.21
UDASUM Thny BERTMM
4
10
38.67
16.62
35.23
constrained pscudo labelling with the BERT BASE
IDASUMMTI.FFRR
4
10
38.78
16.38
35.00
encoder. We also integrate the encoder of CPSUM with a slighter TinyBERT. It should be noted that
DisCo (SM)
4
10
39.20
16.51
35.34
DisCo ($B4)
4
10
38.88
16.61
35.17
the ORACLE, system is upper bound of the ex- tractive summarization.
CPSUM
12
1000
40.42
17.62
36.59
UDASUM
12
1000
40.29
17.65
36.54
UDASUM TIyBERTSS
4
1000
39.99
17.43
36.20
UDASUM IDDSSUMT,BEERRS BiyBERTO
4
1000
40.22
17.54
36.34
A.5
Performance under Few-labels Settings
DisCo (8*)
4
1000
40,49
17.65
36.57
The form using differently labelled data in Table 2 indicates that there is large performance gap be- tween the 12-layer models and 4-layer models with only 10 labelled data due the dramatic reduction in model size.
DisCo ($84)
4
1000
40.49
17.64
36.56
trains a large inspirer model (BERT) and then opti- mizes target network (TexICNN).
However, as shown in Table 11, the extrac- tive summarization tasks, DisCo works particularly well than the 12-layer models in the scenario of 100 labelled examples. The extractive summarization task is to classify every single sentence within a document, and the two views effectively encourage invariant prediction for unlabeled points' pertur- bations. DisCo achieves superior performance, shown in Table 11, whether it uses only 10 or 1000 labelled data extractive summarization. The su- periority of DisCo with 4-layer BERT is more evi- dent when processing 10 labelled extractive sum- marization, compared to CPSUM and UDASUM with 1 12-layer BERT. The results also indicate that method can be suitable for extreme cases that suffer from severe data scarcity problems.
Other SSL algorithms integrated with BERT are implemented in a unified supivsuper learning benchmark (USB) (Wang et al., 2022a) for classi- fication, including Mean Teacher (Tarvainen and Valpola, 2017), VAT (Miyato et al., 2019), Fix- Match (Sohn et al., 2020), CRMatch (Fan et al., 2023), AdaMatch (Berthelot al., 2022), and SimMatch (Zheng al., 2022), all utilizing un- labeled data for invariant predictions. We report their text classification results in the USB bench- mark testing. PCM (Xu et al., 2022) is a com- plex multi-submodule combination SSL model with three components, a K-way classifier, the class semantic representation, and class-sentence matching classifier. MixText (Chen al., 2020) is a regularization-basedd SSL model with an interpolation-basee based augmentation technique. Both PCM and MixText use a 12-layer BERT as the backbone model.
A.6 More Different Model-view Analysis
We further investigate the performance of different model view combinations in dual-students DisCo. As described in section 2.2.1, the model view en- coding has two forms: Separated-layer KD (SKD) and Connected-layer KD (CKD). Results of DisCo equipped with different model-view variants on the
For extractive summarization, we extend Tiny- BERT and UDA for classifying every sentence, termed as UDASUM and TERSBERTSUM. Specif- ically, multiple [CLS] symbols are inserted in

---

Agnews
Yahoo!Answer
DBpeida
Models
10
200
30
200
DisCo (SM)
76.90
85.39
87.82
51.48
62.36
68.10
94.02
98.13 13
98.56
DisCo (5*+)
77.36
85,55
87.95
51.31
62.93
68.24
94.79
98.14
98.63
Model Averaging Ensemble
77.45
85.60
88.04
52.23
63.17
68.49
94.75
98.20
98.66
Table 12: Comparison of 4-layer DisCo with model averaging ensemble.
classification tasks are summarized in Table 10. Although all three combinations of model views achieve improvement (compared to results in Ta- ble 2), the combinations of CKD and SKD for two students perform slightly better than other combi- nations. According to Sun et al. (2019), distilling across alternate layers in knowledge distillation captures more diverse representations, while dis- tilling along connected layers tends to capture relatively homogeneous representations. By com- bining these two distinct strategies of model view encoding, DisCo acquires additional inductive bias for each student the cohort, resulting in improved performance on downstream tasks.
CO2
75A
DO,
AD,
40.30
CO2
40.08
40.10
40.19
40.22
T5a
40.03
40.07
400.15
DO2
40.00
39.98
40.08
-40.10
-40.05
ADg
40.21
40.22
40.24
40.31
-40.00
DisCo (SM)
CO
75A
DO,
ADA
CO2
40.08
40.07
40.
40.
40.30
75.
40.06
40.06
40.12
40.18
40.20
A.7 More Different Data-view Analysis
40. 15
DOg
39.99
39.97
40.09
40.28
Figure 3, visualize the effect of DisCo (4- layer) integrating different data views encoding methods in the summarization task. We find that: DisCo integrating with the adversarial attack (AD) obtains superior performances, especially when data view is the adversarial attack in a SOFT FORM (ADA. ADB). DisCo with HARD FORM data views like AA DOB) or (DOA, ADB) get sub-optimal ef- fectiveness. This suggests that more advanced data augmentation methods pave the way for more refined data view.
-40.55
ADr
40.21
40.21
40.29
40.32
(b) DisCo (S**)
Figure 3: The performance visualization of the dual- student DisCo with data view using different combi- nations of data augmentation strategies. The row indi- cates the 1st data sertt--uumetattinnbaeeee ing strategy, while the column indicates the 2nd data- augmentation-bast based data view encoding strategy. The results of iual-students DisCo with 4-layer TinyBERT being students are evaluated on the CNN/DailyMai with 100 labelled data.
data view encod-
A.8 Model Ensembling for Multiple Students
Model ensembling is an effective strategy, often yielding superior performance compared to individ- ual models. As shown in Table 12, using simple model averaging for the 4-layer student model from Table 2 resulted in enhanced performance.
A.9 Selection of MSE or KL Loss
In our framework, we use the MSE loss to align the logits of the students. However, besides using MSE loss, employing Kullback-Leible (KL) diver- gence maintain consistency between the student predictions also widely chosen approach. We prefer the MSE loss in our framework because the student can learn better without suffering from the information loss that occurs when passing through logits to probability space (Ba and Caruana, 2014).
However, the core focus of our research to ascertain the potential of a single model within our framework. Training requires two or more student models, but only one is essential for inference. Hav- ing multiple students during training ensures per- formance comparable the teacher model, while selecting one student for inference upholds com- putational efficiency. Diving deeper into ensemble techniques to further amplify performance wasn't our primary objective.
As shown Table 13, the 4-layer DisCo with MSE loss performs better in the majority cases. However, when labeled data is extremely limited

---

Agnews
Yahoo!Answer
DBpeida
Models
30
10
200
200
DisCo (S**) MSE
76.90
85.39
87.82
51.48
62.36
68.10
94.02
98.13
98.56
DisCo (S*+) + MSE
77.36
85,55
87.95
51.31
62.93
68.24
94.79
98.14
98.63
DisCo (S**) + KL
76.46
83.76
87.20
52.90
61.81
67.17
95.63
97.72
98.38
DisCo (Se1) +
77.31
83.94
87.17
53.68
63.21
67.68
96.14
97.83
98.41
Table 13: Comparison between MSE loss and KL divergence in 4-layer DisCo.
(c.g.. 10 per class), KL divergence may surpass MSE in performance. This be attributed the noisy predictions produced by the student model, its performance not optimal because of the limited labeled data. KL divergence enforces label matching, thereby reducing issues resulting from corrupted knowledge transferred from another stu- dent model (Kim et al., 2021).
A.10 Details in Loss Landscape Visualization
Our loss visualization approach adheres to the 'fil- ter normalization' method (Li et al., 2018). For each setting, select the pepprrrforiing student checkpoint based on its validation results. Sub- sequently, we generate two random vectors and normalize them using parameters specific to each model. Ultimately, using the same training data and augmentation techniques, we plot the training loss landscape following the two normalized directions.


Reviews:
[{"review_id": "vTC7P67R51", "reviewer": "Reviewer_QmSR", "summary": "", "questions": "Questions about method.\n\n* Why not use logits for the initial KD from the teacher step in Disco? This seemed to work pretty well for Distilbert, which retained 97% GLUE accuracy with 40% smaller model from bert-base. Albeit they used more supervised training data.\n\nQuestions about results.\n\n* It seems somewhat awkward that Disco produces multiple students. Is the recommendation that we should only keep the better performing student based on some validation accuracy? Did you consider ensembling the students through model averaging or aggregating predictions or a MoE-like approach? That being said, utilizing multiple students at test time could lead to more resource requirements.\n\n* How many random seeds for SingleStudent are used in Table 7? For Disco, it is almost as if there are two random seeds, so it seems it would be more fair if multiple seeds of SingleStudent were used as well. Also, did you consider ways for single student to leverage the unlabeled data?\n\n* It would be helpful to have more details about the loss visualization. For example, how many data are used to draw the landscape? Is it using the same 10 labeled data from training?\n\n* The loss landscape is visually appealing but I am not convinced it is that one is \"much flatter\" than the other.\n\n* (low priority) More should be said about CNN/DailyMail when using 10/1000 in the main text. Otherwise perhaps remove mentions of the 10/1000 settings in the main text altogether.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer QmSR,\n\nWe would like to thank you for all the meticulous and constructive feedback and positive comments. We have elaborated on our responses as follows:\n\n> **Q1: Why not use logits for the initial KD from the teacher step in Disco?**\n\n- As outlined in Section 2.2, our choice for the teacher step KD was to adopt a **task-agnostic KD**, following the approach used by TinyBERT [1]. \n\n    The primary goal of this task-agnostic KD is to offer a good and general initialization for the compact student model tailored for downstream tasks. This eliminates the need to repeatedly fine-tune the large teacher model and then distil it to the student across different datasets, ensuring computational efficiency.\n\n- Given that the prediction head in the task-agnostic KD is irrelevant to the downstream tasks, distilling logits from the teacher step was deemed non-essential for our methodology. \n\n> **Q2: Concern about \"Disco produces multiple students\"**\n\n- Training requires two or more student models, but **only one is essential for inference**. Having multiple students during training ensures performance comparable to the teacher model, while selecting one student for inference maintains computational efficiency.\n\n- Selecting a single model for final inference based on validation performance is commonly adopted in methods involving multiple produced models, such as mutual learning [2] and collaborative learning [3].\n\n- Model ensembling is indeed an effective strategy (as shown in the subsequent table). However, our primary focus is to determine **the potential of a singular model** within our framework (a single student for inference already obtains noticeable performances). Exploring ensemble techniques that can seamlessly integrate various perspectives of the student model presents a promising direction for future work.\n\n- Moreover, as demonstrated in Table 2 and Table 4 and Table 6, regardless of whether we pick the student model based on validation performance or heuristic methods, its results consistently outperform baselines, ensuring robust final performance.\n\n***Table 1: Comparison of 4-layer DisCo with Model Averaging Ensemble***\n| Models                   |           | Agnews    |           |           | Yahoo     |           |           | dbpedia   |           |\n|--------------------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n|                          | 10        | 30        | 200       | 10        | 30        | 200       | 10        | 30        | 200       |\n| DisCo ($\\rm S^{A4}$ )    | 76.90     | 85.39     | 87.82     | 51.48     | 62.36     | 68.10     | 94.02     | 98.13     | 98.56     |\n| DisCo ($\\rm S^{B4}$ )    | 77.36     | 85.55     | 87.95     | 51.31     | 62.93     | 68.24     | **94.79** | 98.14     | 98.63     |\n| Model Averaging Ensemble | **77.45** | **85.60** | **88.04** | **52.23** | **63.17** | **68.49** | 94.75     | **98.20** | **98.66** |\n\n> **Q3: How many random seeds for SingleStudent**\n\n- For SingleStudent, we **consistently used three random seeds**, ensuring uniformity in our experimental process. The model and dataset settings from the 6-layer DisCo in Table 7 are the same as those in Table 2.\n\n- Incorporating unsupervised data directly during SingleStudent fine-tuning presents challenges. SSL techniques like consistency constraints or mix-up are typically employed to utilize unsupervised data. \n\n- (can be discussed) Furthermore, the dual-student DisCo, without the model view, can essentially be perceived as co-training SingleStudent alongside its duplicate, making optimal use of unlabeled data. Performance comparisons are shown in Table 8.\n\n> **Q4, 5: Concern about 'loss visualization'**\n\n- More details about loss visualization will be added to the appendix in the final version. We follow the \"filter normalization\" method [4]. We select the top-performing student checkpoint and use **the same training data and augmentation techniques** for plotting. \n\n- (can be discussed) Reviewers' perspectives are expected to be rigorous. As for the notion of a flatter loss landscape, we are open to engaging in a thorough discussion. \n\n    Employing the \"filter normalization\" method ensures our results remain unaffected by the scale of the model parameters. Therefore, achieving a distinctly \"flatter\" result can prove challenging given identical model architecture and training data.\n\n    In light of this, we acknowledge that our statement might be a \"strong claim\". We will modify the claim in the final version. \n\n---\n\nIn addition, thank you for highlighting the concerns regarding the presentation and the omission of certain references. We appreciate your feedback and will make the necessary revisions (e.g., \"strong claims\") in the final version to address these issues.\n\nBest wishes,\n\nAuthors of Paper5607\n\n**Reference:**\n\n[1] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for Natural Language Understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2020.\n\n[2] Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep Mutual Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.\n\n[3] Siyuan Qiao, Wei Shen, Zhishuai Zhang, Bo Wang, and Alan Yuille. Deep Co-Training for Semi-Supervised Image Recognition. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.\n\n[4] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the Loss Landscape of Neural Nets. In Advances in Neural Information Processing Systems, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Reason 1. The paper presents promising positive results for KD with BERT when there is small amount of labeled data but lots of unlabeled data. The approach is straightforward and depends on a combination of logit matching and layer matching. Many people would be interested in this approach as KD is often required to deploy NLP models. NOTE: Results are promising although perhaps not definitive that this is a new sota method for the semi-supervised setting.\n\nReason 2. The extensive analysis, results, and ablations will definitely be useful for researchers using this and similar KD work. Using more than two students is particularly interesting.\n\nReason 3. More specifically, the comparison between model and data-views will help researchers understand the important conceptual components when designing KD pipelines.", "reasons_to_reject": "Reason 1. Although the two student approach is convenient, it remains unclear if two students are necessary. For example, finetuning the initial teacher then providing pseudo labels for the student seems like an obvious baseline --- I think this baseline was only included for Agnews where it does outperform Disco. I have more question about single student in the question section.\n\nReason 2. The Disco approach has some awkwardness, since the output is two student models --- if we do not have much validation data, then it might be hard to pick the better model among the two. Perhaps more statistics could be reported about performance or a protocol for combining models. This would make the findings more actionable.\n\nReason 3. It's unclear if the tested datasets are \"low resource\". Perhaps datasets that are more likely represent real world semi-supervised data would be more informative.\n\nReason 4. Sometimes the text is a bit strong wrt claims. For instance, I am not sure table 6 is showing anything \"unequivocally\" about the Disco performance.", "questions_to_authors": "Questions about method.\n\n* Why not use logits for the initial KD from the teacher step in Disco? This seemed to work pretty well for Distilbert, which retained 97% GLUE accuracy with 40% smaller model from bert-base. Albeit they used more supervised training data.\n\nQuestions about results.\n\n* It seems somewhat awkward that Disco produces multiple students. Is the recommendation that we should only keep the better performing student based on some validation accuracy? Did you consider ensembling the students through model averaging or aggregating predictions or a MoE-like approach? That being said, utilizing multiple students at test time could lead to more resource requirements.\n\n* How many random seeds for SingleStudent are used in Table 7? For Disco, it is almost as if there are two random seeds, so it seems it would be more fair if multiple seeds of SingleStudent were used as well. Also, did you consider ways for single student to leverage the unlabeled data?\n\n* It would be helpful to have more details about the loss visualization. For example, how many data are used to draw the landscape? Is it using the same 10 labeled data from training?\n\n* The loss landscape is visually appealing but I am not convinced it is that one is \"much flatter\" than the other.\n\n* (low priority) More should be said about CNN/DailyMail when using 10/1000 in the main text. Otherwise perhaps remove mentions of the 10/1000 settings in the main text altogether.", "paper_topic_and_main_contributions": "The paper presents a technique for knowledge distillation in the semi-supervised setting for BERT on multiple datasets. The approach jointly trains two students. By logit matching between the students, the setup enables effective use of the unlabeled data. The results are encouraging that this is a helpful KD recipe when there are only a small amount of labeled data. The paper includes extensive experiments, analysis, and details that should be helpful for replication and future work.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Furlanello et al Born Again Neural Networks is probably relevant as it is example where student can improve over teacher."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Style\n\n* (very low priority) I found the blue bold to be distracting and confusing at times. It is a personal choice, but I think the paper is easier to read with the blue numbers removed. Especially since sometimes blue bold text is not the best.\n* (very low priority) I could also do without the average in Table 1. It's confusing to average 10/30/200 together."}}, {"review_id": "CmPivyJNkI", "reviewer": "Reviewer_vrX3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer vrX3,\n\nThank you for your valuable feedback on our work. We appreciate your positive feedback and understand the concerns you have raised regarding some of the framework details, experiments, and presentation. Please find our detailed responses below:\n\n> **C1: Concern about \"use MSE or KL loss\"**\n- We apologize for the oversight. Equations 3, 4, and 7 contain typo errors. We actually **employ the MSE loss in the logits of the model**, not in the prediction. These will be corrected in the final version.\n- Our choice of loss function **aligns with previous SSL works**. In PI-Model and Temporal ensembling [1], the MSE loss was found to produce \"better results than cross-entropy loss\". Multiple SSL works, such as Mean Teacher [2] and Dual Student [3], follow this loss function.\n\n- The comparison between MSE and KL loss has been a topic of extensive discussion and research [4]. Given your concerns, we recognize the importance of this comparison and will endeavor to include it in the appendix of the final version.\n\n> **C2: Concern about 'model selection for inference'**\n- The subsequent table further illustrates the validation performance for the models presented in Table 2.  Given that all student model architectures are identical, **a model with better validation performance typically exhibits improved test results**.\n \n- Additionally, as shown in Tables 2, 4, and 6, regardless of the selection method—validation performance, heuristic methods, or even random—the student model consistently outperforms all baselines, ensuring robust final performance.\n\n- Choosing a single model for final inference based on validation performance is a common practice in methods involving multiple produced models, like mutual learning [5] and collaborative learning [6].   \n\n ***Table 1: Performance Comparison of DisCo on Validation and Test Sets***\n\n| Models                | dataset |           | Agnews    |           |           | Yahoo     |           |           | dbpedia   |           |\n| --------------------- | ------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |\n|                       |         | 10        | 30        | 200       | 10        | 30        | 200       | 10        | 30        | 200       |\n| DisCo ($\\rm S^{A6}$ ) | Val     | 74.69     | 86.68     | 89.24     | 57.42     | 65.97     | 69.53     | 98.6      | 98.53     | 98.61     |\n|                       | Test    | 74.38     | 86.39     | 88.70     | 57.62     | 66.04     | 69.57     | 98.5      | 98.45     | 98.57     |\n| DisCo ($\\rm S^{B6}$ ) | Val     | **77.40** | **87.14** | **89.45** | **58.48** | **66.47** | **69.87** | **98.63** | **98.60** | **98.77** |\n|                       | Test    | **77.45** | **86.93** | **88.82** | **59.10** | **66.58** | **69.75** | **98.57** | **98.61** | **98.73** |\n| DisCo ($\\rm S^{A4}$ ) | Val     | 76.44     | 85.50     | **88.16** | 50.98     | 62.37     | 67.97     | 94.05     | **98.14** | **98.63** |\n|                       | Test    | 76.90     | 85.39     | 87.82     | **51.48** | 62.36     | 68.10     | 94.02     | 98.13     | 98.56     |\n| DisCo ($\\rm S^{B4}$ ) | Val     | **77.41** | **85.68** | 88.11     | **51.25** | **63.24** | **68.42** | **94.78** | 98.11     | 98.62     |\n|                       | Test    | **77.36** | **85.55** | **87.95** | 51.31     | **62.93** | **68.24** | **94.79** | **98.14** | **98.63** |\n| DisCo ($\\rm S^{A2}$ ) | Val     | 70.47     | 82.10     | 86.31     | 48.10     | 57.59     | 64.20     | 89.78     | 96.05     | 97.66     |\n|                       | Test    | 70.61     | 81.87     | 86.08     | 48.41     | 57.84     | 64.04     | **89.67** | 96.06     | 97.58     |\n| DisCo ($\\rm S^{B2}$ ) | Val     | **74.30** | **82.24** | **86.74** | **50.95** | **58.36** | **65.69** | **89.80** | **96.25** | **97.82** |\n|                       | Test    | **75.05** | **82.16** | **86.38** | **51.05** | **58.83** | **65.63** | 89.55     | **96.14** | **97.70** |\n\n---\n\nIn addition, thank you for highlighting the issues with the tables and presentation. We will streamline and clarify them in the final version of the paper.\n\nBest wishes,\n\nAuthors of Paper5607\n\n**Reference:**\n\n[1] Samuli Laine and Timo Aila. Temporal Ensembling for Semi-Supervised Learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.\n\n[2] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings, 2017.\n\n[3] Zhanghan Ke, Daoye Wang, Qiong Yan, Jimmy S. J. Ren, and Rynson W. H. Lau. Dual Student: Breaking the Limits of the Teacher in Semi-Supervised Learning. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, 2019.\n\n[4] Taehyeon Kim, Jaehoon Oh, Nakyil Kim, Sangwook Cho, and Se-Young Yun. Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, 2021.\n\n[5] Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep Mutual Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.\n\n[6] Siyuan Qiao, Wei Shen, Zhishuai Zhang, Bo Wang, and Alan Yuille. Deep Co-Training for Semi-Supervised Image Recognition. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors provide comprehensive evaluation results, demonstrating the robustness of performance on the classification task.\n2. The proposed training schema can bring insights for audience working on knowledge distillation.", "reasons_to_reject": "1. Some technique choice in this paper is not suitable. For example, the authors use MSE to measure the difference between two probability distribution (the predictions on two models). On probability distribution difference, KL divergence or cross-entropy are more reasonable choice. \n2. The variance between student models seems large according to table 2. This would raise a realistic change in model selection after distillation. Without accessing test set, we would not know which model performs better. The authors need to at least provide student model selected on validation set for each entry to give more realistic comparison.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents DisCo, a method to distill large language models into multiple small models with co-training for semi-supervised learning. DisCo tries to distill one large model into multiple small models by providing different distillation objectives, data transformations and distributions to build a committee of student models. During this process, DisCo also employs consistency training loss among students enable co-training among students. From the empirical results, the student models can outperform baseline models with similar size.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The multiple gigantic tables are a bit hard to follow and read. I suggest to subtract some entries and add the full table in appendix."}, "extra_text": {}}, {"review_id": "MN7Fkq31Zy", "reviewer": "Reviewer_VCv8", "summary": "", "questions": "A: L 330 ff: How did you come up with “the best model view and data view we found”? Was there any hyperparameter tuning? If yes: budget, search space, etc. (Appendix A2, which is referenced, does not provide any information)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer VCv8,\n\nWe would like to express our sincere gratitude for strongly supporting our contributions! Below, we provide a comprehensive response to each of your comments and concerns:\n\n> **C1: Concern about \"the source code for future release\"**\n- Thank you for your feedback. All our datasets and portions of our code related to the model architecture can be found in the Supplementary Materials. We will release the full code after the blind review, as providing links during this period is restricted.\n\n> **C2: Concern about \"explanation of SOFT FORM and HARD FORM\"**\n- The details related to \"SOFT FORM\" and \"HARD FORM\" are elaborated from lines 241 to 251.\n\n- Specifically, there are various data augmentation strategies, e.g., Adversarial Attack (AD), Cut Off (CO), and Dropout (DO). The \"HARD FORM\" combines **different strategies** for distinct data views, such as AD with DO. In contrast, \"SOFT FORM\" uses the **same strategy** across data views, like AD with AD. With the same strategy, different initializations can also produce varied data views.\n\n- Thank you for pointing it out. We will adjust the explanation in the final version to make it clearer.\n\n> **Q1 How did you come up with \"the best model view and data view we found\"?** \n- The specifics regarding selecting the best model view and data view are detailed in Appendix A5 and Appendix A6, respectively.\n \n- Regarding the best model view, we referred to established knowledge distillation methods [1,2,3] and **heuristically selected several plausible layer mapping strategies**. The selection of the best model view was then made based on the comparative performance of these strategies.\n \n- As for the best data view, we considered **pair-wise combinations of the four different data augmentation strategies discussed in the paper**. The optimal combination was determined based on the performance of these combinations.\n\n- Your concerned details will be introduced in the Appendix in our final paper.\n\n---\n\nIn conclusion, we once again thank you for your insightful critique and direction. Your feedback is instrumental in refining our paper.\n\nBest wishes,\n\nAuthors of Paper5607\n\n**Reference:**\n\n[1] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for Natural Language Understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2020.\n\n[2] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. In Advances in Neural Information Processing Systems, 2020.\n\n[3] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In CoRR, 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors present a very convincing collection of ablations and thorough analyses of their models, extending it from two to multiple models, considering different options of data augmentations\n- An effective SSL method allowing for leveraging unlabeled data is important in the current situation of ever-increasing amounts of unlabeled data\n", "reasons_to_reject": "- The authors just state “1Code and data available at: [url redacted for blind review].” which is not sufficient imho; they could append it or make it available via an anonymous link. Further, I checked the pre-print of the paper (I still don’t know any of the authors, so no objections regarding anonymity), and not even there they provide a link to the code. So I am wondering what reasons they have to withhold access to their code.\n- The explanation of “SOFT FORM” and “HARD FORM” is not really clear in the paper\n", "questions_to_authors": "A: L 330 ff: How did you come up with “the best model view and data view we found”? Was there any hyperparameter tuning? If yes: budget, search space, etc. (Appendix A2, which is referenced, does not provide any information)?", "paper_topic_and_main_contributions": "The authors propose a strategy for distilling and ensembling models in a semi-supervised setting, allowing for (a) faster inference in (b) smaller models, which (c) exhibit superior performance to comparable approaches. They evaluate their proposed modeling strategy on a set of benchmark data sets for classification and summarization and show superior performance to other small models as well as competitive performance to the full models.", "excitement": 4, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- I think there is a type in Figure 1 (left), where the outcome of the bottom one of the two blue students is denoted with “\\bar y” instead of “\\hat y”.\n- l. 352: “Competiting” should be “Competing”?\n- l. 360: Acronym CV not defined?\n- l. 122: SOTA used w/o being defined; l. 366 “state-of-the-art” used\n"}}]

Author rebuttal:

Meta-review: The paper introduces DisCo, a semi-supervised learning framework designed for fine-tuning a set of smaller models derived from a pre-trained language model via knowledge distillation. The primary innovation is in the co-training technique, which encourages knowledge sharing amongst the student models based on varied views produced by differing distillation strategies and input augmentations. 

Strengths:
The paper's innovative approach with KD in scenarios with limited labeled data was seen as promising, particularly its results and methodology (QmSR). The results are found to be promising, especially on the classification task, and the proposed training schema is found to provide new insights for those in the knowledge distillation domain (vrX3). The introduction of an effective SSL method is deemed crucial (Reviewer VCv8).

Weaknesses:
Concerns were raised regarding the two-student approach, with questions surrounding its necessity and the complications it introduces in model selection due to the potential variance between student models (QmSR, vrX3). The datasets used for testing were scrutinized for their authenticity in representing "low resource" scenarios, and some claims, such as those in table 6, were seen as possibly overstated (QmSR). Some design choices like MSE were questioned and variance of some of the results seemed too large (vrX3).
Decision: Accept-Main