Query-based Knowledge Transfer for Heterogeneous Learning Environments

Authors: Norah Alballa, Wenxuan Zhang, Ziquan Liu, Ahmed M. Abdelmoniem, Mohamed Elhoseiny, Marco Canini
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Collaborative Learning, Knowledge Distillation, Query-based Knowledge Transfer.

Abstract:
Decentralized collaborative learning under data heterogeneity and privacy constraints has rapidly advanced. However, existing solutions like federated learning,  ensembles, and transfer learning,  often fail to adequately serve the unique needs of clients, especially when local data representation is limited.  
To address this issue, we propose a novel framework called Query-based Knowledge Transfer (QKT) that enables tailored knowledge acquisition to fulfill specific client needs without direct data exchange. 
It employs a data-free masking strategy to facilitate the communication-efficient query-focused knowledge transformation while refining task-specific parameters to mitigate knowledge interference and forgetting. Our experiments, conducted on both standard and clinical benchmarks, show that QKT significantly outperforms existing collaborative learning methods by an average of 20.91% points in single-class query settings and an average of 14.32% points in multi-class query scenarios.
Further analysis and ablation studies reveal that QKT effectively balances the learning of new and existing knowledge, showing strong potential for its application in decentralized learning.

Decision: accept (poster)
Reviewer scores: 6, 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730570758205

thread_type:
official_review

<thread_object>
creator: reviewer_5p86

<content>
summary:
The authors propose a framework called Query-based Knowledge Transfer (QKT). This framework uses a data-free masking strategy that facilitates more effective knowledge distillation, addressing statistically heterogeneous environments in Federated Learning.

strengths:
-The paper is well-motivated. Figures 1 and 2 makes it easy to understand the strong motivations of this paper. The writing is also clear.
- The idea of using noise to identify expert models is interesting, as it avoids dodgy reliances of "public data" or "synthetic data" in federated learning context, where such data may be scarce.

weaknesses:
-While the framework proposed seems general, it is not clear why only image-related tasks are considered.
- The accuracies on simple datasets in Table 2 seem surprisingly low, for the baseline approaches.
- The authors seem to have chosen some relatively general baselines to compare against, neglecting several Federated learning + knowledge distillation papers; such as Zhu, Z., Hong, J., & Zhou, J. (2021, July). Data-free knowledge distillation for heterogeneous federated learning. In International conference on machine learning (pp. 12878-12889). PMLR. 
- the entire idea seems overly reliant on the overconfidence concept (Guo et al., 2017). This implies that a careful tuning of \lambda in (3) is needed. The experiments seem to not mention what happens when a client may be an expert in multiple classes at once, or when multiple clients are experts in a single class

questions:
-how will the framework extend to other modalities?
- the authors claim that QKT completes learning in a single round. Is there convergence guarantees for this?
- using noise to check what expert a model is could also reveal the client's data distribution, which is itself a privacy intrusion
</content>
</thread_object>

</thread>
<thread>
creation date:
1730727164455

thread_type:
official_review

<thread_object>
creator: reviewer_2vf7

<content>
summary:
The paper introduces Query-based Knowledge Transfer (QKT) to learn query class in a collaborative learning setting while preventing interference from irrelevant information and mitigating catastrophic forgetting. It achieves this by implementing a two-phase distillation process that involves query-focused learning and classification head refinement. 
Experimental results demonstrate that QKT outperforms traditional federated learning and other collaborative methods in single- and multi-class query tasks.

strengths:
- The variant QKT Light can reduce communication overhead by requiring only one communication round, making it well-suited for communication-limited settings.
- The authors demonstrate the adaptability of the proposed method across diverse datasets and tasks, including medical and standard image classification benchmarks.

weaknesses:
- The practical applications or user cases motivating this approach for query-based knowledge transfer are not clearly defined, especially given that personalized federated learning can also handle similar scenarios.
- The novelty of the paper's contributions is somewhat limited, as knowledge distillation and classification head refinement are established methods in personalized FL​  [1,2,3,4]
- Table 2 lacks comparisons with advanced personalized FL baselines after 2022 such as those in [5,6].
- Table 1 also lacks a baseline of standalone local fine-tuning on the query class, which is the simplest way to learn the query class without any communication. How effective is this baseline?
- QKT's performance is highly sensitive to the hyperparameters (e.g., $\lambda$ in the mask shown in Table 4), which could be challenging to optimize in real-world scenarios.


reference: 
- [1] Data-Free Knowledge Distillation for Heterogeneous Federated Learning  ICML 2021
- [2] On bridging genertic and personalized federated learning for image classification. ICLR 2022
- [3] Think locally, act globally: Federated learning with local and global representations. 2020
- [4] Exploiting shared representations for personalized federated learning ICML 2021
- [5] Test-time robust personalization for federated learning. ICLR, 2023.
- [6] PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees CVPR 2024

questions:
- It is unclear why learning irrelevant classes is inherently problematic, as shown in Figure 1; incorporating such classes could potentially enhance the model’s ability to handle out-of-distribution data in test time [5].
- In Phase 1 (lines 219-221), it seems to be assumed that clients are aware of the irrelevant classes for defining the threshold $\tau$. Does it mean that each client knows the complete set of available classes [C]? This might raise questions about the method's applicability in real-world decentralized settings where it could be hard to know what other clients’ classes are.


post-rebuttal: Thanks to the authors for addressing my concerns about motivation and baselines. I raised the score accordingly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730866609666

thread_type:
official_review

<thread_object>
creator: reviewer_pbtb

<content>
summary:
The paper introduces a framework that addresses the challenges arising from knowledge interference and catastrophic forgetting in FL due to the collaboration mechanisms that transfer knowledge between clients through weights. This is achieved by a data-free masking strategy that masks "irrelevant" knowledge from peer clients and a two-phase training process to balance knowledge transfer and retention.

strengths:
1. The problem defined in the paper is interesting and the problems of knowledge interference and catastrophic forgetting make sense.
2. The paper is easy to understand.

weaknesses:
1. Some of the claims like "QKT performs knowledge transfer in a single round, which is significantly more efficient than traditional centralized and peer-to-peer FL approaches that involve multiple communication rounds.", do not seem well justified since the goal of FL is joint collaborative training vs goal of this work is to distill knowledge from well trained peers. 
2. The algorithmic details are not well presented - for example, how the teacher models' are trained and how their predictions are sent to the student model to obtain loss in Eq 4 are missing. 
3. Some algorithmic details like how communication rounds = 100 were set for other approaches and what synthetic datasets were used are also missing.

questions:
1. How would the methods compare in more homogeneous data settings? And how does the quality of the teacher model impact the performance? Is there a criterion to eliminate teacher models apart from the class based criterion. 
2. Can you elaborate on what practical use case would this method be helpful in? And how does this work compare with body of work in knowledge distillation from multiple teachers in centralized settings?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731058733767

thread_type:
official_review

<thread_object>
creator: reviewer_mlpa

<content>
summary:
This paper addresses a challenge where the models are learned from heterogeneous environments. In specific, different clients can train their models with their local data. To this end, the authors introduce QKT to enhance the local model (a.k.a. student model) by distilling the knowledge from other pretrained models (a.k.a. teacher models). The solution consists of two phases: Feature Extractor Enhancement and Classification Head Refinement. The first phase filters out the irrelevant teacher models and uses them to facilitate the performance of the student model, and the second phase takes a further step to train the classifier. Extensive experiments demonstrate the effectiveness of the proposed method.

strengths:
1. This paper proposes a new decentralized collaborative learning approach to enhance the performance of a client. 
2. This work is well-motivated by identifying the limitations of knowledge distillation. 
3. The work is verified on various datasets and demonstrates superior performance over other baselines.

weaknesses:
1. The authors should discuss the recent development of ensemble learning rather than (personalized) federated learning. The proposed QKT relies on a number of pretrained teacher models and looks pretty similar to ensemble learning settings. 
2. The proposed work requires pretraining for numerous teacher models. I cannot find the pretraining details in the paper. The authors should discuss this issue. Moreover, since these models store on a client, it leads to a significant storage burden if the client does not have sufficient storage/computation resources.

questions:
See **Weaknesses**.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731200037805

thread_type:
official_review

<thread_object>
creator: reviewer_ycup

<content>
summary:
The authors propose the Query-based Knowledge Transfer (QKT) method where the learning process consists of two stages: 1) the feature extractor enhancement stage, where the model is optimized to approximate the prediction of the teacher model regarding the query classes, and 2) the classification head refinement stage, where the initialized classification head is reused and trained to maintain the accuracy of the original tasks. The method is empirically shown to be effective regarding the query-class accuracy gain and the local-class forgetting.

strengths:
Originality: Although knowledge distillation is well-known, the authors additionally propose a masking strategy for query-oriented learning, and add a second stage in which they reuse the classification head to maintain performance.

Significance: The method is empirically shown to be effective both in improving the query-class performance and maintaining the local-class performance.

Strengths:
1. The masking strategy is innovative and motivated based on the observation made by (Guo et al 2017).
2. The experiments are comprehensive with multiple popular benchmarks included. Experimental results show strong improvement across datasets, even with the light version of QKT.
3. The effects of various factors are presented.

weaknesses:
1. It is unclear what Figure 2 is showing. The background color is inconsistent. Besides, it is not obvious that the QKT improves the boundaries.
2. As the losses in both stages are both equation 4, it is not clear why the second stage helps to maintain the performance. Could the author provide some intuitions behind the strategy?
3. While the performances of different methods are reported, the time of convergence and the computational costs in communication are missing. It would be beneficial for authors to show the speed and computational costs of QKT compared to the baselines. The convergence can be slow due to the synthetic dataset generation and query-class identification of dynamic models. The datasets shown are mostly small toy datasets. It is unclear whether the method scales.
4. There is no experiment showing how the performance changes during stages 1 and 2. It could be insightful for the authors to show the dynamic performance within the stages.
5. The performance is evaluated with 10 clients. Could the author show how the baselines and the QKT method perform with an increasing amount of clients?

questions:
As listed in the weakness.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper studies knowledge transfer under heterogeneity.  Different clients train local models with their local data, and knowledge is distilled across clients using pretrained models as teachers. Irrelevant teacher models are filtered out via masking in distillation (without data exchange, through an appropriate query submission) and the head of the student is refined to avoid catastrophic forgetting. Experiments show good performance w.r.t. predictive power in new tasks and forgetting.

Reviewers raised several concerns regarding scaling with the number of clients, as well as w.r.t. comparison to other competitors, the impact of heterogeneity, and many more, that the authors have addressed during the rebuttal stage.