Venue: NAACL 2024

# Self-Regulated Data-Free Knowledge Amalgamation for Text Classification

Prashanth Vijayaraghavan

IBM Research

San Jose CA 95120

prashanthv@ibm.com Hongzhi Wang

IBM Research

San Jose CA 95120

hongzhiw@us.ibm.com Luyao Shi

IBM Research

San Jose CA 95120

hongzhiw@us.ibm.com Liyan Degan

IBM Research

San Jose CA 95120

Ehsan Degah

IBM Research

San Jose CA 95120

eehgha@us.ibm.com

###### Abstract

Recently, there has been a growing availability of pre-trained text models on various model repositories. These models greatly reduce the cost of training new models from scratch as they can be fine-tuned for specific tasks or trained on large datasets. However, these datasets may not be publicly accessible due to the privacy, security, or intellectual property issues. In this paper, we aim to develop a lightweight student network that can learn from multiple teacher models without accessing their original training data. Hence, we investigate Data-Free Knowledge Amalgamation (Dfka), a knowledge-transfer task that combines insights from multiple pre-trained teacher models and transfers them effectively to a compact student network. To accomplish this, we propose StrataNet, a modeling framework comprising: (a) a steerable data generator that produces text data tailored to each teacher and (b) an amalgamation module that implements a self-regulative strategy using confidence estimates from the teachers' different layers to selectively integrate their knowledge and train a versatile student. We evaluate our method on three benchmark text classification datasets with varying labels or domains. Empirically, we demonstrate that the student model learned using our StratAet outperforms several baselines significantly under data-driven and data-free constraints.

## 1 Introduction

Recent NLP advancements have yielded numerous pre-trained models, often achieving state-of-the-art performance across various tasks. These models are publicly available to promote reproducibility and further research. To facilitate knowledge transfer from pre-trained teacher models, Hinton et al. (2015) pioneered Knowledge Distillation (KD), utilizing soft target labels to train light-weight student models effectively. Subsequently, diverse KD approaches have been successfully applied in different domains. Traditionally, KD relies on using original training data to guide the student model's learning from a task-specific teacher model. However, this approach has limitations, often involving learning from a single teacher model Sanh et al. (2019); Liu et al. (2020) or a task-specific ensemble of teachers Fukuda et al. (2017); Tian et al. (2019).

Unlike traditional KD, where teachers focus on the same task, knowledge amalgamation (KA) techniques Luo et al. (2019); Shen et al. (2019) enable learning in a student network by integrating knowledge from multiple teachers with diverse expertise. These methods enhance the student model's classification abilities across a wider range of labels. While KA techniques are well-established in Computer Vision, their exploration in NLP literature is limited. Li et al. (2022) utilized Monte-Carlo Dropout to estimate model uncertainty for merging knowledge from different pre-trained teacher models. However, these techniques often require access to unlabeled data from the original training set used by the pre-trained models Luo et al. (2019); Shen et al. (2019); Li et al. (2021); Vongkulbhisal et al. (2021);

Figure 1: Given a set of pre-trained teacher models (Teacher Models 1 & 2), each with distinct expertise, the goal is to train a student model capable of amalgamating their knowledge, mastering prediction across all specialized classes of the teachers.



2019) to train a versatile student model. Unfortunately, the original training data and annotations are often unavailable due to various issues. Moreover, the diverse expertise of teacher models may lead to uncertain states and probabilities when handling input sequences outside their domains. These challenges hinder the application of KA methods in broader domains. To address this, we explore a practical knowledge-transfer task called Data-Free Knowledge Amalgamation (Dfka). Figure 1 provides an overview of this task, aiming to enhance the student model's capabilities by integrating knowledge from multiple pre-trained teachers without access to the original training data.

To achieve our goal, we introduce StrataNet1, a knowledge amalgamation framework with: (i) a flexible generation module creating pseudo text data for each pre-trained teacher network, and (ii) an amalgamation module enabling self-regulated integration of teachers' knowledge during student model training. Integration is guided by a teacher-specific out-of-distribution (OOD) score, assessing the reliability of intermediate and output states of every pre-trained teacher model.

Footnote 1: Short for Selective Transformer based Self-RegulATiveAmalgamation NETwork

**Contributions**: (1) Introduction of StrataNet, a pioneering data-free knowledge amalgamation (DFKA) method for lightweight student model training without accessing original training data. (2) Proposal of a block-wise amalgamation strategy for integrating knowledge from multiple heterogeneous (or homogeneous) teacher model layers into the student model. (3) Demonstration of superior performance by our StrataNet-trained student model compared to various baselines across three benchmark text datasets: AG News, OhSumed Abstracts, and 5 Abstracts Group.

## 2 Related Work

In this section, we explore the relevant literature concerning knowledge distillation (KD) and amalgamation. KD is a technique aimed at transferring knowledge from a large teacher network to a student model, offering benefits across various NLP tasks and facilitating model compression. These tasks encompass question answering Izacard and Grave (2020); Yang et al. (2020), multi-modal summarization Zhang et al. (2022), and neural machine translation Tan et al. (2019); Wang et al. (2021); Zhou et al. (2019), among others. Notable approaches such as DistilBERT Sanh et al. (2019) and TinyBERT Jiao et al. (2019) primarily focus on compressing models, maintaining the student architecture identical to that of the teacher model (i.e., homogeneous setting). Fewer models, like those by Tang et al. Tang et al. (2019, 2019), train a heterogeneous student model. While KD has found widespread application in NLP, data-free knowledge distillation (DFKD) remains relatively underexplored compared to its application in computer vision. Recent studies Melas-Kyriazi et al. (2020); Ma et al. (2020, 2022) have delved into training compressed student models under data-free settings using techniques such as training data augmentation, plug & play embedding guessing, and reinforced topic promter.

In contrast to the singular teacher model approach in KD, knowledge amalgamation (KA) involves training a versatile student model by amalgamating insights from multiple pre-trained teacher models. Li et al. Li et al. (2022) utilized Monte Carlo Dropout to estimate model uncertainty and perform classification on the union of label sets from different teacher models. Although these methods do not rely on human-annotated labels, they leverage input text from the original training data. Jin et al. Jin et al. (2022) proposed a parameter space merging method for dualess knowledge fusion, assuming an impractical uniformity in model architectures across input and merged models. Differing from the aforementioned approaches, our method, StrataNet, introduces a framework for data-free knowledge amalgamation (DFKA) in text, representing a pioneering exploration in NLP literature involving multiple heterogeneous teacher networks.

## 3 Problem Setup

Given \(K\) pre-trained teacher models \({\cal T}=\{{\cal T}_{i}\}^{K}_{i=1}\), each with \(L_{{\cal T}_{i}}\)-layers and its own domain of expertise, i.e., performing a \(c_{i}\)-class classification task with few overlapping or disjoint set of labels \({\cal Y}_{i}=\{y_{i}^{j}\}_{j=1}^{c_{i}}\), our goal is to train a lightweight student model \({\cal S}\) with \(L_{{\cal S}}\)-layers such that it can compute predictions over the union of all the label sets, \({\cal Y}=\bigcup_{i=1}^{K}{\cal Y}_{i}\) and \(L_{{\cal S}}\leq\min(\{L_{{\cal T}_{i}}\}_{i=1}^{K})\).

## 4 Proposed Approach

### Overview

In this section, we outline our framework, StrataNet, designed to train a lightweight student model using multiple teachers under data-free 

constraints. We address the following factors: (a) lack of training data, (b) existence of specialized teachers with non-overlapping or partially overlapping label sets, and (c) need to integrate knowledge from diverse teachers. Our StrataNet consists of two main components. The first, \({\cal G}_{i}\), is a teacher-specific steerable data generator. It guides a base pre-trained language model, \({\cal P}\), to generate tailored text for each teacher, \({\cal T}_{i}\), overcoming data scarcity by creating pseudo-data samples. The second component, the amalgamation module, serves two functions. It evaluates each teacher's confidence in predicting within their expertise and employs blockwise integration with a selective transformer to fuse knowledge from multiple teachers. Utilizing confidence scores, this approach appropriately weights representations from different teacher models, effectively managing diverse teacher architectures.

### Steerable Data Generator

To overcome the challenge of unavailability of the original training data for teacher models, we utilize a conditional text generation method that generates pseudo-data samples specifically tailored to the label set of the teacher \({\cal T}_{i}\). Given a teacher model \({\cal T}_{i}\) and any class label \(c\in{\cal Y}_{i}\), a steerable text generator, \({\cal G}_{i}\), produces a class-controlled text \(x\) of length \(N\) as follows: \(P(x_{1:N}|c)=\prod_{t=1}^{N}P(x_{t}|x_{1:t-1},c)\)

For each teacher \({\cal T}_{i}\), our steerable text generator produces pseudo-data samples \({\cal D}_{i}^{p}=({\cal X}_{i}^{p},\hat{\cal Y}_{i}^{p})\) by applying an inference-time controllable generation method to steer an unconditional language model towards the desired class label relevant to a specific teacher. The generation process entails guiding a base pre-trained language model (PLM), denoted as \({\cal P}\), using a post-processing module. By adjusting the parameters during the decoding phase, the generator exhibits varying degrees of class control over the text sampled from the chosen base PLM.

Based on a recent study by Gu et al. (2022), we adopt a variant of the weighted decoding method to generate class-conditional text using a pre-trained unconditional language model, denoted as \({\cal P}\). In this approach, we model the generation process by incorporating a Bayesian factorization as follows:

\[P(x_{t}|x_{1:t-1},c)\propto P(x_{t}|x_{1:t-1})P(c|x_{1:t})^{\gamma} \tag{1}\]

Here, \(\gamma\) represents a hyperparameter for control strength. The first term corresponds to the output probabilities generated by the chosen PLM, while the second term relies on the teacher model to estimate the likelihood of the generated text (up to the current time step \(t\)) being classified under the class label \(c\). During the sampling process, the value of \(\gamma\) regulates the influence of the teacher model.

One challenge in this approach is the computational complexity of teacher-guided sequence sampling. To compute the second term in Equation 1,

Figure 2: Illustration of our StrataNet framework.

 

we need to estimate the class probability \(P(c|x_{1:t})\), requiring evaluation of \(P(c|x_{1:t-1},x_{t})\) for every token in the vocabulary \(\mathcal{V}\) at the \(t^{th}\) timestep. To reduce inference time, we exclude low-probability tokens and prioritize a subset for teacher guidance. Tokens with low probability \(P(x_{t}|x_{1:t-1})\) from the PLM are discarded, even if the teacher model assigns high weights \(P(c|x_{1:t})\). Consequently, we exclusively use the top-m tokens with higher probabilities, guided by the teacher model's weights. Subsequently, we employ a top-k sampling strategy, where \(k<m\ll|\mathcal{V}|\)). Our experiments indicate that setting \(m=100\) is notably effective. Table 1 displays sample generations produced using the teacher-guided generation module. In this experiment, we trained two teachers on the AG News and OhSumed label sets. Subsequently, we generated pseudo-data samples for the "Sports" category under the guidance of the teacher trained on the AG News dataset and for the "Cardiovascular Diseases" category under the guidance of the teacher trained on the OhSumed dataset. Notably, we observe that the generated text is not only fluent but also relevant to the respective category of interest.

### Block-wise Amalgamation Module

We introduce an amalgamation module that estimates out-of-distribution (OOD) scores for each teacher using a Teacher-specific OOD Estimator. It integrates selective informative states from relevant teachers based on these scores using a Selective Transformer (ST-amalg), transferring them to a student in a blockwise manner to accommodate varying sizes of teacher models.

### Teacher-specific OOD Estimator

Due to diverse label sets \(\{\mathcal{Y}_{i}\}_{i=1}^{K}\) in pre-trained teacher models \(\{\mathcal{T}\}_{i=1}^{K}\), any input text from an unseen category for a specific teacher is considered out-of-distribution (OOD). Extracted features from that teacher's intermediate layer may not be sufficient for effective knowledge transfer to the student model. Studies indicate: (a) Transformer-based models encode transferable features in various intermediate layers Liu et al. (2019); Rogers et al. (2021), and (b) final layers, especially in models like BERT, are highly task-specific Kovaleva et al. (2019); Rogers et al. (2021). Considering these, we propose layer-wise teacher-specific lightweight OOD estimators, explained below.

#### 4.4.1 OOD Score Computation

For an input text \(x\in\mathcal{X}_{i}\) with label \(y\in\hat{\mathcal{Y}}_{i}\), a transformer-based pre-trained teacher model \(\mathcal{T}_{i}\) produces contextual token-level latent embeddings at each layer \(l\in L_{\mathcal{T}_{i}}\). These are averaged into a single latent representation \(h_{i}^{l}\in\mathcal{R}^{d_{i}}\), where \(d_{i}\) is the dimensions of the latent representations. To compute an OOD score for any new input \(x_{new}\), we use a Mahalanobis distance (md) based OOD detection technique. For an in-distribution (id) dataset with \(c_{i}\)-labels associated with \(\mathcal{T}_{i}\), the md technique fits \(c_{i}\)-class conditional Gaussian distributions \(\mathcal{N}(\mu_{y},\Sigma)\) to each of the \(c_{i}\) Id classes based on training latent representations \(h_{i}^{l}\). However, Ren et al. (2021) proposed a Relative Mahalanobis distance (Rmd) that outperforms Md in OOD detection for both near and far-OOD scenarios by calculating the distance between class-conditional Gaussians and a single background Gaussian using data from all classes. For an input \(x_{new}\) with the latent representation \(\hat{h}_{i}^{l}\) at layer \(l\), Rmd is given by:

\[\text{Rmd}_{y}(\hat{h}_{i}^{l})=\text{M}_{y}(\hat{h}_{i}^{l})- \text{M}_{bg}(\hat{h}_{i}^{l})\] (2) \[\text{M}_{by}(\hat{h}_{i}^{l})=(\hat{h}_{i}^{l}-\mu_{y})^{T} \Sigma^{-1}(\hat{h}_{i}^{l}-\mu_{y})\] (3) \[\mathcal{C}_{i}^{l}(\hat{h}_{i}^{l})=-min_{y}\{\text{R}\text{M}_{ y}(\hat{h}_{i}^{l})\}\] (4) 

where \({\cal C}_{i}^{l}\) refers to the confidence score of \(x_{new}\) being in-domain for \({\cal T}_{i}\) based on representation at layer \(l\), \(\mu_{y}\) is a class-conditional mean vectors and \(\Sigma\) is the covariance matrix, \(\mbox{Md}_{bg}\) indicates Mahalanobis distance of \(h_{i}^{l}\) to the background distribution fitted to the entire training data usually. The Rmd score acts as a contrastive measure indicating the sample's proximity to both the training and background domains. Higher scores indicate greater out-of-distribution characteristics, resulting in lower Id confidence scores, \({\cal C}_{i}^{l}\). Alternatively, in some cases, intermediate layers can be partitioned into \(B\) blocks. Applying a similar procedure as described in Equations (2)-(4), confidence scores can be calculated for each block. Here, \(\hat{h}_{i}^{b}\) represents the latent representation for block \(b\), obtained by mean pooling over the layer representations within that block. We use a held-out subset of pseudo-data samples, \(\hat{\mathcal{D}}_{i}^{p}\), generated for each teacher \({\cal T}_{i}\).

### Selective Transformer-based Block-wise Amalgamation ST-amalg

To transfer knowledge from diverse, larger teachers to a lightweight student model, we align intermediate representations in a block-wise manner, accommodating the varying number of layers between them. Each teacher network \({\cal T}_{i}\) may have a different number of grouped layers. We compute confidence-aware block-wise intermediate representations, \(z_{i}^{b}\), using the confidence score at each block \(b\) for each teacher. Inspired by the literature on multimodal analysis Urooj et al. (2020); Vijayaraghavan and Roy (2023); Lin et al. (2022), we consider the intermediate latent vectors from \(K\) teachers, denoted as \(\{z_{i}^{b}\}_{i=1}^{K}\), as a token sequence fed into a Transformer layer. We introduce a learnable special token \([AMALG]\), similar to \([CLS]\), to integrate confidence-enriched representations from teachers into a final block-level amalgamated representation, denoted as \(\hat{z}_{\cal T}^{b}\). Therefore, we refer to this layer as the Selective Transformer-based amalgamation layer (ST-amalg). Formally,

\[z_{i}^{b}=f(h_{i}^{b})+g({\cal C}_{i}^{b}) \tag{5}\] \[\hat{z}_{\cal T}^{b}=\textsc{ST-amalg}(\{z_{i}^{b}\}_{i=1}^{K}) \tag{6}\]

where \(f,g\) are linear layers to enrich the block-level embeddings.

## 5 Training Objectives & Details

To amalgamate knowledge at intermediate layers, we compute L2-normalized distance between the student's projected block-level representation and the corresponding teachers' amalgamated embedding. Formally,

\[\begin{split}{\cal L}_{\rm AMAL}=\sum_{b=1}^{B}{\cal L}_{\rm ANAL }^{b}\\ s.t.~{}~{}~{}{\cal L}_{\rm ANAL}^{b}=||\hat{z}_{\cal S}^{b}- \hat{z}_{\cal T}^{b}||_{2}^{2}\end{split} \tag{7}\]

For the output prediction layer, we compute the KL divergence loss based on confidence weighted combination of Teacher models and the temperature \(\tau\) as: \({\cal L}_{out}=KL(\hat{\cal T}(x),\hat{\cal S},\tau)\).

### Training details

Given steerable data generators \(\{{\cal G}_{i}\}_{i=1}^{K}\) tied to teachers \(\{{\cal T}_{i}\}_{i=1}^{K}\), we produce a student training transfer set, denoted as \({\cal D}^{p}\), by combining the pseudo-data samples generated for all the labels associated with each teacher. Next, we divide the intermediate layers into \(B\)-blocks such that the number of layers in each block may vary according to the number of layers in the teacher model. In our experiments, the teacher models (Teacher 1 and Teacher 2) are based on Bert-base-uncased Devlin et al. (2018), and we set \(B\) to the number of intermediate layers in the compressed student model \({\cal S}\), i.e., \(\textsc{Bert}_{6}\). We then compute the number of layers within each block for each teacher accordingly. A subset of pseudo-data samples generated for each teacher \({\cal T}_{i}\), represented as \(\hat{\mathcal{D}}_{i}^{p}\), is used compute the layer-wise distribution statistics for OOD estimation. Finally, we use the student training transfer set \({\cal D}^{p}\) to train the student model by: (a) computing the confidence of teachers' block-wise features in predicting each input text, (b) amalgamating the confidence-enriched representations from teachers and (c) optimizing the weighted sum of intermediate (\({\cal L}_{\rm ANAL}\)) and output prediction layer (\({\cal L}_{out}\)) losses, expressed as:

\[{\cal L}=\lambda\cdot{\cal L}_{\rm ANAL}+(1-\lambda)\cdot{\cal L}_{out} \tag{8}\]

## 6 Experiments

Our experiments address the following research questions: **(RQ1)** How does our model compare to baseline approaches for knowledge distillation in both data-driven and data-free scenarios? **(RQ2)** What is the individual impact of each component in our model on overall performance? **(RQ3)** How does our model fare when multiple heterogeneous teachers are utilized? 



### Datasets

We evaluate our approach using the following benchmark datasets: (a) **AG News2**[30]: It consists of news articles grouped into four major classes--World, Sports, Business, and Sci/Tech. (b) **5 Abstract Group3**[19]: This dataset contains academic paper abstracts from five different domains--business, AI, sociology, transport, and law. (c) **Ohsumed4**[6]: It comprises medical abstracts specifically related to cardiovascular diseases. We focus on single-label text categorization and exclude documents that belong to multiple categories. The data statistics for these benchmark datasets are presented in Table 2.

Footnote 2: [http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html](http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html)

Footnote 3: [https://github.com/qianliu0708/5AbstractsGroup](https://github.com/qianliu0708/5AbstractsGroup)

Footnote 4: [https://disi.unitn.it/moschitti/corpora.htm](https://disi.unitn.it/moschitti/corpora.htm)

### Baselines

We conduct a comparative analysis of our proposed model with data-driven and data-free baselines. Here is a summary of the baselines:

**Teacher Models**, which are used to predict individually. We assign zero probabilities to classes outside the expertise of each teacher. **Ensemble**, which concatenates the output logits from all the teachers to obtains predictions over all the labels \({\cal Y}\). **MUKA-Hard/Soft**[19], which is a data-driven KA method that uses Monte-Carlo Dropout based model uncertainty to guide the student training. **Vanilla KA**[10] (R/CD): which aims to mimic the soft targets produced by the logits combination of all teacher models using KL-divergence. In a data-free scenario, we consider two settings: (i) Random Text (R): The student model is trained on text sequences constructed using randomly selected words from the vocabulary of the pre-trained teacher models; and (ii) Cross-Domain Texts (CD): The student model is trained on cross-domain text corpora like Wikitext-103. **As-Dfd**[20], which is a data-free knowledge distillation approach. We modify this model for the Dfka scenario by crafting pseudo-embeddings for each teacher as specified in their original study and train a student model using self-supervision and KL-divergence. **StrataNet**, which is our complete Dfka model that generates pseudo-data samples and leverages the produced data for knowledge amalgamation.

### Metrics

To be comparable with prior studies, we compute the classification accuracy across various datasets. In particular, we report the mean and standard deviations of the accuracy over three runs in SS7.

## 7 Results and Discussion

**Overall Performance** The evaluation results are presented in Table 3, providing a summary of our findings. To ensure a fair comparison, our baselines incorporate cross-domain data (CD), similar to our model that utilizes a resource like PLM. Additionally, we implement a variation of the data-free knowledge distillation method [20] for Dfka. Compared to all the baselines, our StrataNet model demonstrates significant improvement over other Dfka baselines across various text classification datasets. Notably, our com

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Datasets** & **\#Classes** & **\#Train** & **\#Valid** & **\#Test** \\ \hline AG News & 4 & 108,000 & 12,000 & 7,600 \\ \hline
5 Abstracts Group & 5 & 4,770 & 530 & 1,000 \\ \hline OhSumed & 23 & 3,021 & 336 & 4,043 \\ \hline \end{tabular}
\end{table}
Table 2: Data Statistics of benchmark text classification datasets.

\begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Models**} & **AG** & **5Abstracts** & **Oroup** & **OnSumed** \\ \cline{2-4}  & & & \\ Supervised & 94.6 & 90.7 & 70.5 \\ \hline \multicolumn{4}{c}{**Data–Driven Methods**} \\ \hline Teacher 1* & 49.9 & 42.0 & 36.2 \\ Teacher 2* & 47.5 & 51.5 & 38.18 \\ Ensemble* & 59.8 & 62.3 & 45.48 \\  & 87.0 & 79.0 \\ MUKA-Hard* & \((\pm 0.40)\) & \((\pm 0.82)\) & — \\  & 87.1 & 79.3 \\ MUKA-Soft* & \((\pm 0.19)\) & \((\pm 0.85)\) & — \\ \hline \multicolumn{4}{c}{**Data-Free Methods**} \\ \hline Teacher 1 & 45.8 & 41.75 & 32.8 \\ Teacher 2 & 46.9 & 46.88 & 35.6 \\ Ensemble & 55.86 & 53.67 & 41.94 \\ \hline Vanilla KA & 58.9 & 56.27 & 47.33 \\ (R) & \((\pm 3.19)\) & \((\pm 2.76)\) & \((\pm 4.41)\) \\ \hline Vanilla KA & 62.43 & 61.55 & 50.91 \\ (CD) & \((\pm 2.62)\) & \((\pm 0.91)\) & \((\pm 2.8)\) \\ \hline As-Dfd & 74.89 & 69.83 & 56.08 \\  & \((\pm 0.89)\) & \((\pm 1.06)\) & \((\pm 1.6)\) \\ \hline
* & **88.76*
* & **83.6*
* & **65.92*
* \\  & \((\pm 0.19)\) & \((\pm 0.28)\) & \((\pm 0.41)\) \\ \hline \end{tabular}
\end{table}
Table 3: Evaluation results on benchmark text classification dataset averaged over 3 runs. Our method achieve statistically significant improvements over the closest baselines (\(p<0.01\)). Bold face indicates the best results and
* refers to results from prior literature.


pact student model trained under data-free settings shows an approximately 4% increase in performance compared to the best-performing data-driven model in certain cases. We intuit that the knowledge from the intermediate layers are beneficial for the performance improvement.

### Ablation Studies (RQ2)

#### 7.1.1 Effect of Rmd

In order to measure the effect of Rmd (explained in SS4.4), we replace the OOD score computation using other methods including: (a) embedding-based Mahalanobis distance (Md) and (b) maximum softmax probability (MSP) at the final layer. Figure 3(A) shows how modifying the OOD score has a significant impact on the overall performance of the model. RMD OOD score helps achieve the best performance of our model.

#### 7.1.2 Impact of ST-amalg

To evaluate the contribution of ST-amalg, we introduce two variants: (a) \({\tt StratanNet}_{mul}\):simply multiply the block-level confidence score with the teacher embeddings instead of the embedding enrichment (as in Equation 5, (b) \({\tt StratanNet}_{noST}\): remove ST-amalg and use a linear layer on top of confidence weighted sum of teachers' latent vectors in Equation 6. Figure 3(B) shows that both the variants lead to significant performance degradation, asserting their value to the overall model performance. This validates our intuition that the embedding enrichment and ST-amalg serve as critical components to select the important block-level features from different teacher models 5.

Footnote 5: Additional experiments on using LLM like Llama-2 for the data generation module in Appendix B.2

### Effect of Multiple Heterogeneous Teachers (RQ3)

To demonstrate our model's ability to generalize across multiple heterogeneous teachers, we explore scenarios with three (1 BERT-base, 1 RoBerta-base, and 1 ALBERT) and four (1 BERT-base, 2 RoBerta-base, and 1 ALBERT) teachers, each with different architectures. Results are shown in Figure 3(C). While baseline KA methods struggle with increased teacher diversity, our approach consistently improves accuracy and maintains performance with more teachers. These findings underscore the robustness and effectiveness of our method across diverse experimental setups.

## 8 Conclusion

In this study, we introduce Data-Free Knowledge Amalgamation (DFKA), a method to train a lightweight student network from diverse teacher models without their original training data. Our framework, StrataNet, employs a steerable data generator and an amalgamation module for effective knowledge transfer. Experimental results on text datasets demonstrate the superiority of StrataNet over various baselines, both in data-driven and data-free scenarios. Ablation studies highlight the importance of different model components. This work opens avenues for efficient knowledge transfer in text classification, offering practical solutions for resource-constrained environments.

## Limitations

While our StrataNet model outperforms existing baselines, it has certain limitations. The steerable generation module, which guides text genera

Figure 3: (A) Impact of different OOD scores – Rmd, Md & MSP, (B) Impact of ST-amalg, (C) Effect of Multiple Heterogeneous teachers on OhSumed dataset.

 

tion for specific classes, may not consistently produce accurate class-specific text. Moreover, it may not capture the full diversity of complex training datasets. Further research is needed to investigate and improve the generation module. Additionally, there is potential to expand knowledge amalgamation to tasks beyond text classification, which warrants future research.

## Ethics Statement

Our StrataNet model focuses on improving the performance of Dfka and does not introduce new ethical concerns compared to other KD/KA methods. However, we want to acknowledge two key risks here: (a) data-free knowledge amalgamation strategies can potentially be used as a precursor to model extraction attacks, compromising the confidentiality of blackbox models, as demonstrated in Truong et al. (2021), and (b) model compression itself may introduce biases, as suggested by Hooker et al. (2020). It is important to address these risks, which are not specific to our method but are common in data-free model compression techniques, in future research.

## References

* J. J. Chung, E. Kamar, and S. Amershi (2023)Increasing diversity while maintaining accuracy: text data generation with large language models and human interventions. arXiv preprint arXiv:2306.04140. Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2018)Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. Cited by: SS1.
* T. Fukuda, M. Suzuki, G. Kurata, S. Thomas, J. Cui, and B. Ramabhadran (2017)Efficient knowledge distillation from an ensemble of teachers. In Interspeech, pp. 3697-3701. Cited by: SS1.
* Y. Gu, X. Feng, S. Ma, J. Wu, H. Gong, and B. Qin (2022)Improving controllable text generation with position-aware weighted decoding. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 3449-3467. Cited by: SS1.
* G. Hinton, O. Vinyals, and J. Dean (2015)Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Cited by: SS1.
* S. Hooker, N. Moorosi, C. Clark, S. Bengio, and E. Denton (2020)Characterising bias in compressed models. arXiv preprint arXiv:2010.03058. Cited by: SS1.
* G. Izacard and E. Grave (2020)Distilling knowledge from reader to retriever for question answering. arXiv preprint arXiv:2012.04584. Cited by: SS1.
* X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu (2019)Tinybert: distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351. Cited by: SS1.
* X. Jin, X. Ren, D. Preotiuc-Pietro, and P. Cheng (2022)Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09849. Cited by: SS1.
* T. Joachims (1998)Text categorization with support vector machines: learning with many relevant features. In European conference on machine learning, pp. 137-142. Cited by: SS1.
* O. Kovaleva, A. Romanov, A. Rogers, and A. Rumshisky (2019)Revelating the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 4365-4374. External Links: Link, Document Cited by: SS1.
* L. Li, Y. Lin, X. Ren, Guangxiang Zhao, P. Li, J. Zhou, and X. Sun (2021)Model uncertainty-aware knowledge amalgamation for pre-trained language models. arXiv preprint arXiv:2112.07327. Cited by: SS1.
* L. Li, Y. Lin, X. Ren, G. Zhao, P. Li, J. Zhou, and X. Sun (2022)From mimicking to integrating: knowledge integration for pretrained language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, pp. 6391-6402. External Links: Link, Document Cited by: SS1.
* F. Lin, S. Wu, Y. Ma, and S. Tian (2022)Full-scale selective transformer for semantic segmentation. In Proceedings of the Asian Conference on Computer Vision (ACCV), pp. 2663-2679. Cited by: SS1.
* N. F. Liu, M. Gardner, Y. Belinkov, M. E. Peters, and N. A. Smith (2019)Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 1073-1094. External Links: Link, Document Cited by: SS1.
* Qian Liu, H. Huang, Y. Gao, X. Wei, and R. Geng (2017)Leveraging pattern associations for word embedding models. In Database Systems for Advanced Applications: 22nd International Conference, DASFAA 2017, Suzhou, China, March 27-30, 2017, Proceedings, Part I 22, pp. 423-438. Cited by: SS1.

 

Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. 2020. FastBERT: a self-distilling BERT with adaptive inference time. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6035-6044, Online. Association for Computational Linguistics.
* Luo et al. (2022) Renqian Luo, Laii Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. 2022. Biogpt: generative pre-trained transformer for biomedical text generation and mining. _Briefings in Bioinformatics_, 23(6).
* Luo et al. (2019) Sihui Luo, Xinchao Wang, Gongfan Fang, Yao Hu, Dapeng Tao, and Mingli Song. 2019. Knowledge amalgamation from heterogeneous networks by common feature learning. _arXiv preprint arXiv:1906.10546_.
* Ma et al. (2020) Xinyin Ma, Yongliang Shen, Gongfan Fang, Chen Chen, Chenghao Jia, and Weiming Lu. 2020. Adversarial self-supervised data-free distillation for text classification. _arXiv preprint arXiv:2010.04883_.
* Ma et al. (2022) Xinyin Ma, Xinchao Wang, Gongfan Fang, Yongliang Shen, and Weiming Lu. 2022. Prompting to distill: Boosting data-free knowledge distillation via reinforced prompt. _arXiv preprint arXiv:2205.07523_.
* Melas-Kyriazi et al. (2020) Luke Melas-Kyriazi, George Han, and Celine Liang. 2020. Generation-distillation for efficient natural language understanding in low-data settings. _arXiv preprint arXiv:2002.00733_.
* Ren et al. (2021) Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. 2021. A simple fix to mahalanobis distance for improving near-ood detection. _arXiv preprint arXiv:2106.09022_.
* Rogers et al. (2021) Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2021. A primer in bertology: What we know about how Bert works. _Transactions of the Association for Computational Linguistics_, 8:842-866.
* Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. _arXiv preprint arXiv:1910.01108_.
* Shen et al. (2019) Chengchao Shen, Xinchao Wang, Jie Song, Li Sun, and Mingli Song. 2019. Amalgamating knowledge towards comprehensive classification. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 33, pages 3068-3075.
* Tan et al. (2019) Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Ti-Yan Liu. 2019. Multilingual neural machine translation with knowledge distillation. _arXiv preprint arXiv:1902.10461_.
* Tang et al. (2019a) Raphael Tang, Yao Lu, and Jimmy Lin. 2019a. Natural language generation for effective knowledge distillation. In _Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019)_, pages 202-208.
* Tang et al. (2019b) Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019b. Distilling task-specific knowledge from bert into simple neural networks. _arXiv preprint arXiv:1903.12136_.
* Tian et al. (2019) Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2019. Contrastive representation distillation. _arXiv preprint arXiv:1910.10699_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Truong et al. (2021) Jean-Baptiste Truong, Pratyush Maini, Robert J Walls, and Nicolas Papernot. 2021. Data-free model extraction. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 4771-4780.
* Urooj et al. (2020) Aisha Urooj, Amir Mazaheri, Niels Da vitoria lobo, and Mubarak Shah. 2020. MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 4648-4660, Online. Association for Computational Linguistics.
* Vijayakabavan and Roy (2023) Prashanth Vijayakabavan and Deb Roy. 2023. M-sense: Modeling narrative structure in short personal narratives using protagonist's mental representations. _arXiv preprint arXiv:2302.09418_.
* Vongkulbhisal et al. (2019) Jayakorn Vongkulbhisal, Phongtharin Vinayavekhin, and Marco Visentini-Scarzanella. 2019. Unifying heterogeneous classifiers with distillation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 3175-3184.
* Wang et al. (2021) Fusheng Wang, Jianhao Yan, Fanding Meng, and Jie Zhou. 2021. Selective knowledge distillation for neural machine translation. _arXiv preprint arXiv:2105.12967_.
* Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. _arXiv preprint arXiv:1910.03771_.
* Yang et al. (2020) Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. 2020. Model compression with two-stage multi-teacher knowledge distillation for web question answering system. In _Proceedings of the 13th International Conference on Web Search and Data Mining_, pages 690-698.
* Zhang et al. (2015) Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In _NIPS_ 

Zhengkun Zhang, Xiaojun Meng, Ysheng Wang, Xin Jiang, Qun Liu, and Zhenglu Yang. 2022. Unimis: A unified framework for multimodal summarization with knowledge distillation. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 36, pages 11757-11764.
* Zhou et al. (2019) Chunting Zhou, Graham Neubig, and Jiatao Gu. 2019. Understanding knowledge distillation in non-autoregressive machine translation. _arXiv preprint arXiv:1911.02727_.

 

Implementation Details

We base our StrataNet implementation on PyTorch6, Huggingface Wolf et al. (2019) and PyTorch Lightning7. We tune our model hyperparameters using grid-search. For the generation module, we sample a maximum of 128 tokens. The top 200 tokens were selected using the nucleus sampling method with a sampling threshold of \(p=0.9\). For Ohsumed dataset, we used BioGPT Luo et al. (2022) in order to tailor the data generation process to the domain of interest. Trained on large-scale PubMed abstracts, BioGPT is a specialized Transformer language model designed for generating and mining biomedical text. In our experiments, we use a compressed Bert model with 6 layers, referred to as Bert\({}_{6}\), as our student model. Table 4 shows the tuned hyperparameters used by both the generation and distillation component of our StrataNet model. Our method trains a compressed student model (e.g., Bert\({}_{6}\)) using a confidence score that selectively amalgamates the knowledge from intermediate and output layers of multiple teachers.

Footnote 5: [https://pytorch.org/](https://pytorch.org/)

Footnote 6: [https://www.pytorchlightning.ai/](https://www.pytorchlightning.ai/)

## Appendix B Ablation Studies

### Effect of heterogeneous teachers and student model layers

In Section SS6, we conducted experiments using a compressed Bert\({}_{6}\) model, and the results demonstrated no significant performance degradation. To delve deeper, we run additional experiments involving \(\{6,4\}\)-layer student models with different teacher configurations: a homogeneous setting (\(\mathcal{T}_{1},\mathcal{T}_{2}\): Bert\({}_{large}\)) and a heterogeneous setting (\(\mathcal{T}_{1}\): Bert\({}_{large}\), \(\mathcal{T}_{2}\): RoBert\({}_{large}\)). The evaluations on the AG News dataset reveal the poor performance of the data-free baseline As-Dfd with compressed layers, highlighting the challenges of the heterogeneous setting. However, our StrataNet framework demonstrates consistent and robust performance under both configurations, even with higher compression.

**Importance of Intermediate Layers:** We conduct a sensitivity analysis by varying \(\lambda\) in the loss function, which is associated with the knowledge from intermediate layers. Figure 4 presents the effects of different \(\lambda\) values on the AG News and 5 Abstracts Group datasets. We find that the model performs best with \(\lambda\sim 0.65\), indicating the relatively higher importance of intermediate layers for improving performance. This finding aligns with prior studies Liu et al. (2019); Rogers et al. (2021), which have observed that Transformer-based models often encode transferable features in their intermediate layers.

### Impact of Steerable Data Generation

We evaluate the impact of the Steerable Data generation module through Llm\({}_{Manual}\), involving manual prompting of an LLM like Llama-2 Touvron et al. (2023) using task-specific prompts and em

Figure 4: Effect of modifying \(\lambda\).

 

ploying diversification techniques (DTs) like sampling variations and temperature adjustments as described in Chung et al. (2023). Figure 5 shows no significant performance improvement with a more potent Llama-2 model. While relying solely on manual prompting may lack dataset diversity, diversification techniques enhance performance but might introduce irrelevant tokens, impacting overall generation accuracy. Details of the manually designed prompts are given below.

#### b.2.1 Manually-designed Prompts

Table 6 show samples of the manually designed prompts to the Llama-2 model.

#### b.2.2 Generation Parameters

For diversification, we use different temperature setting while we sample tokens. We used five temperature values \(\rho\in\{0.3,0.5,0.7,0.9,1.3\}\). Furthermore, we also experimented with different sampling techniques. For nucleus sampling, we varied the top-\(p\) between \(\{0.65,0.95\}\). For top-k sampling, we chose \(k\in\{10,25,35,50,75\}\).

\begin{table}
\begin{tabular}{l|c} \hline \hline
**Datasets** & **Manual Prompts** \\ \hline AG News & Generate a [Category] news ¡article/story¿ DBPedia & Generate a document about [Category] \\ IMDb & Generate a [Category] movie review \\ SST-2 & Generate a [Category] sentence \\ OhSumed & Generate an abstract about [Category] \\ \hline \hline \end{tabular}
\end{table}
Table 6: Samples of dataset-specific manually designed prompts provided as input to the Llama-2 (llama-2-70bchat) model.

Figure 5: Effect of Steerable Data Generation. Llama-2 with manually designed prompts doesn’t outperform our generation module.