One-Shot Federated Distillation Using Monoclass Teachers: A Study of Knowledge Fragmentation and Out-of-Distribution Supervision
Authors: Cedric Maron
Virginie Fresse
ORZALESI
Venue: Accepted by TMLR ()

Abstract: The performance of machine learning models critically depends on the quality and diversity of training data. However, privacy, legal, and proprietary concerns often limit direct data sharing. Many organizations possess high-quality data for specific classes and may wish to share the knowledge derived from it without revealing the data or engaging in collaborative training. While federated learning (FL) enables distributed model training, it typically assumes mutual benefit, requires repeated communication, and produces a shared global model. Another paradigm, knowledge distillation (KD), allows a student model to learn from teacher predictions. We propose a one-shot federated distillation method in which a single client learns from monoclass teacher models trained independently by multiple providers. Each provider shares its model once, and the client combines these with unlabeled data to distill a multiclass student model—aggregating knowledge from disjoint, class-specific sources. This unidirectional, asymmetric setup poses a key challenge: out-of-distribution (OOD) supervision, where monoclass teachers often mispredict unseen inputs, leading to noisy signals for the student. The main contribution of this work is a systematic study of knowledge fragmentation in one-shot federated distillation with monoclass teachers. We evaluate five configurations with varying class coverage per provider and show that increasing fragmentation intensifies OOD supervision, degrading student performance. Experiments on MNIST, FashionMNIST, and CIFAR-10 confirm that fragmentation consistently reduces student accuracy. To mitigate this, we discuss three strategies: (1) exposing teachers to diverse off-class examples, (2) penalizing overconfidence, and (3) using contrastive learning to sharpen feature boundaries.

Published in Transactions on Machine Learning Research (08/2025)
One-Shot Federated Distillation Using Monoclass Teach- ers: A Study of Knowledge Fragmentation and Out-of- Distribution Supervision
Cedric Maron Segula Technologies, France
cedric. madric.maronneeegul...o
Virginic Fresse
Hubert Curien Laboratory, Université Jean Monnet, Saint- Siinn-,t France
Mathieu Orzalesi
muthieeu.orzalesi@ssspoo.
Segula Technologies, France
Reviewed on OpenReview: https: // openreview. net/ forum? id= ENdm5BW7aF
Abstract
The performance of machine learning models critically depends the quality and diversity of training data. However, privacy, legal, and proprietary concerns often limit direct data sharing. Many organizations possess high-quality data for specific classes and may wish to share the knowledge derived from it without revealing the data or engaging in collaborative training. While federated learning (FL) enables distributed model training, it typically assumes mutual benefit, requires repeated communication, and produces a shared global model. Another paradigm, knowledge distillation (KD), allows student model to learn from teacher predictions. We propose a one-shot federated distillation method in which single client learns from monoclass teacher models trained independently by multiple providers. Each provider shares its model once, and the client combines these with unlabeled data to distill a multiclass student model-agrregating knowledge from disjoint, class-specific sources. This unidirectional, asymmetric setup poses a key challenge: out-of-distribution (OOD) supervision, where monoclass teachers often mispredict unseen inputs, leading to noisy signals for the student. The main contribution of this work is a systematic study of knowledge fragmentation in onc-shot federated distillation with monoclass teachers. We evaluate five configurations with varying class coverage per provider and show that increasing fragmentation intensifies OOD supervision, degrading student performance. Experiments on MNIST, FashionMNIST, and CIFAR-10 confirm that fragmentation consistently reduces student accuracy. To mitigate this, we discuss three strategies: (1) exposing teachers to diverse off-class examples, (2) penalizing overconfidence, and (3) using contrastive learning to sharpen fcature boundarics.
1 Introduction
Modern machine learning models rely heavily diverse, high-quality training data. Yel. in real-world scenarios, relevant data is often siloed across organizations, each holding large datasets focused on specific classes or domains. Privacy, legal, and proprictary constraints frequently prohibit raw data sharing. Many organizations are open to sharing learned knowledge, either freely or commercially, particularly when they hold more data higher quality than others. This creates an opportunity for under-resourced clients to benefit from distributed, class-specific expertise without requiring direct collaboration.
Federated learning (FL) offers decentralized solution for training models across distributed data sources without sharing raw data. However, typical FL protocols require multiple rounds of communication, syn-

---

Published in Transactions on Machine Learning Research (08/2025)
chronized participation, and shared training objectives assumptions that break down in asymmetric scenarios where a single client wishes to learn from independent providers who have no incentive to collaborate or receive updates.
Knowledge distillation (KD) provides another paradigm for model training, enabling unidirectional knowl- edge transfer: a student model learns from the predictions of pretrained teachers, potentially using data different from what the teachers were trained on. KD is especially attractive in settings where privacy, scalability, and minimal coordination are critical, as it allows knowledge sharing without revealing raw data requiring ongoing interaction.
Building on this idea, we propose a one-shot federated distillation framework in which a client learns from monoclass teacher models trained independently by different providers. Each teacher, trained on privale, class-specific data, is shared once and remains passive thereafter. The client aggregates these predictions with its own unlabclod data to train a multi-class student model. This approach is scalable, srivacy-rresering, and coordination-free but introduces key challenge: out-of-distribution (OOD) supervision.
Because each teacher is trained on single class, it often produces confident but incorrect predictions inputs from unseen classes. Aggregating such outputs introduces noise that can impair the generalization ability of the student model.
This paper presents a systematic study of knowledge fragmentation in this one-shot distillation setting. We define five configurations with varying class coverage per teacher and empirically evaluate the impact of fragmentation on the performance of the student. Experiments on MNIST, FashionMNIST, and CIFAR- 10 show that increasing fragmentation consistently degrades accuracy. To address this, discuss three mitigation strategies: (1) exposing teachers to diverse off-class examples, (2) penalizing overconfidence in the student loss, and (3) applying contrastive learning to sharpen feature boundaries.
Our findings highlight the risks of OOD supervision in fragmented settings and propose practical strategies to improve robustness in privacy-constrained, asymmetric knowledge transfer scenarios.
The structure of the paper as follows. The paper begins by reviewing related work on knowledge distilla- tion, federated learning, and modular learning architectures. This is followed by detailed prosentation of the proposed client-serve framework, including the training of monoclass teacher models and the distillation procedure used to construct the student model. The experimental setup is then described, with emphasis on the design of knowledge fragmentation configurations and the evaluation protocol across multiple bench- mark datascts. Subsequently, the impact of fragmentation on student model performance is analyzed, and the key limitations introduced by out-of-distribution supervision are discussed. Several mitigat stralegies are then discussed. The paper concludes with a summary of findings and a discussion of directions for future work.
2
Related Work
This work lies at the intersection of knowledge distillation, expert specialization, and decentralized learning. Relevant literature is reviewed across three areas: (1) knowledge distillation in single and multi-teacher settings; (2) federated and decentralized learning frameworks; and (3) modular and mixture-of-expprrte ar- chitectures for specialization. Prior methods analyzed in terms of communication structure, supervision format, and assumptions about label space coverage. Particular attention is given to approaches requiring full label access and the challenges introduced by fragmented class supervision. In contrast to these approaches, the method studied here operates under extreme supervision fragmentation, using one shot, unidirectional communication where only the weights of independently trained monoclass experts, each trained on a sin- gle class, are shared. The primary goal is to analyze how such fragmentation impacts the effectiveness of dist illiaiiaonn and the resulting generalization of the student model.
2

---

Published in Transactions on Machine Learning Research (08/2025)
2.1 Knowledge Distillation
Knowlodge distillation (KD) trains a compact student model to mimic the predictions of a larger teacher. The Ioundational work by Hinton et al. (2015) introduced soft largets as supervisory signals, capturing richer information than hard labels. A broad survey of KD) techniques is provided by Gou et al. (2021).
Multi-teacher KD extends this idca by aggregating predictions from multiple teachers. Fukuda et al. (017
explored strategies like switched and augmented training to integrate teacher outputs. Zhang et al. 2022a
proposed confidence-aware weighting scheme, assigning higher influence to more reliable teachers. These methods improve supervision but typically assume centralized training and a shared label space.
Frivacy-preserving knowledge transfer methods like the Private Aggregation of Teacher Ensembles (PATE) framework (Papernot al., 2016) mitigate data-sharing risks by training teachers on disjoint private datasets. The ensemble remains inaccessible to the student; instead, a separate student model learns from public data labeled via noisy, differentially private aggregation of teacher predictions. While PATE offers strong privacy guarantees, it assumes sufficient label coverage across teachers and requires carefully calibrated aggregation mechanism.
These approaches demonstrate that supervision and training can be decoupled. Yet in decentralized envi- ronments with extreme supervision fragmentation, where each teacher observes only a single class, standard KD techniques become infcasible. Their reliance on shared or overlapping class coverage, bidircctional com- munication, or joint prediction spaces limits their applicability. The next section examines federated and decentralized paradigms that attempt to relax these assumptions.
2.2 Federated Learning and Decentralized Distillation
Federated learning enables distributed model training without sharing raw data. Classical methods such as McMahan et al. (2017) require synchronized communication rounds and assume shared label spaces across clients. Personalized FL methods (Fallah et al., 2020 Li et al., 2021) introduce client-specific adaptation while maintaining a global initialization, but still rely on centralized coordination and iterative updates.
To reduce communication, several FL variants adopt knowledge distillation, exchanging logits instcad of model parameters. For example, FEDMD (Li & Wang, 2019) aligns local models using shared public
dataset, FEDKD (Wu et al.) 2022) performs round- logit exchanges, and FEDDF (Lin et al. 2020 ensembles local models at central server. While reducing parameter transfer, these approaches still require auxiliary data, coordination, and label-space alignment.
One-shot and limited-round methods have emerged to further reduce overhead. FROKT [Li et al. 2020) performs one-shot logit transfer on shared public data, FEDISCA (Kang et al.) (023) combines image syn- thesis with model adaptation, and DS-FL fItahara et al. 2021 removes the server via peer-to-peer updates. However, these methods still rely on shared data, task alignment, or multiple participants interacting during training. Several recent methods explicitly address the missing-class problem in federated learning. FE- DRS Li & Zhan 2021) introduces a restricted softmax mechanism that avoids updating classifier weights for unseen classes during local training, mitigating issues from label distribution skew. FEDMR (Hu et al. 2022) proposes model recombination, where layers from locally trained models are shuffled and recombined by the server to preserve local specialization and enhance global generalization. FEDLC (Zhang et al, 2022b) calibrates local logits before applying softmax, correcting for class imbalance and reducing overfitting to locally dominant classes. FEDGELA (Fan et al. 2023) addresses partially class-disjoint data by fixing global classifier as a simplex Equiangular Tight Frame (ETF) and adapting it locally, ensuring both global consistency and local adaptability. While these methods improve robustness under heterogeneous label distri- butions, they rely on repeated communication, coordinated updates, or centralized aggregation. In contrast, our method performs one-shot distillation using independently trained monoclass exports, transmitted once to the client without coordination, label sharing, or iterative updates. This enables decentralized learning under extreme fragmentation, relaxing many assumptions made in prior federated learning work.
3

---

Published in Transactions on Machine Learning Research (08/2025)
2.3 Mixture of Experts and Modular Architectures
Mixture of Experts (MoE) architectures consist of multiple specialized subnetworks, called experts, where only a subset activated for each input (Jacobs et al. 1991). A gating mechanism dynamically selects
or weights relevant experts, enabling scalable model capacity through sparse computation. Large scale
implementations such as GShard (Lepikhin et al., 2020 and Switch Transformers (Fodus et al. demonstrated strong performance on language and vision tasks by leveraging expert routing.
2022 have
MoE models are typically trained end-to-end using global loss, with both the experts and gating mechanism co-optimized over a shared dataset and unified label space. Extensions such as Modular Networks and Task Routing [Rosenbaum et al. 2019) improve modularity and interpretability, but continue to rely on centralized training, sharod data, and coordinated optimization.
While the proposed method also employs specialized models, it differs fundamentally in structure and training assumptions. Experts are trained independently on private, class-specific data, without access to other classes or coordination during training. No gating mechanism or global objective is used. Instead, the softmaxed putputs of all monoclass teachers are aggregated to supervise the training of a multi-class student model via a one-shot distillation process. This approach enables modular knowledge transfer in decentralized settings, without requiring shared data or collaborative training.
2.4 Summary
This work explores a distillation setting where each provider trains a monoclass teacher on private data, without coordination label sharing. These frozen models are sent to client, which distills global classifier in single step, without joint training, public datasets, or bidirectional communication.
Unlike prior work in multi-teacher distillation, modular architectures, and federated learning, this setup introduces extreme label fragmentation, where supervision is narrowly scoped and docentralized. Existing methods do not address the generalization challenges introduced by such fragmentation. This gap is ad- dressed through a systematic analysis of how label space fragmentation impacts student generalization, and by introducing one-shot distillation strategy tailored to decentralized, privacy-sensitiv environments.
3 Methodology
Problem Setting
In many practical applications, valuable labeled data distributed across multiple independent providers. However, privacy, legal, or proprictary constraints often prevent these providers from sharing raw data. While traditional approaches such as federated learning enable collaborative model training without data sharing, they often require frequent bidirectional communication and coordinated optimization, which may not be feasible in all enviromments.
This work considers decentralized and asynchronous knowledge-sharing scenario. In this setting, a lightweight client aims to learn a classifier over a target set of classes by interacting with a set of providers. Each provider willing share trained models but not raw data. Specifically, each provider independently trains and shares set of monoclass teacher models, each specialized in recognizing single class. These teachers encapsulate local knowledge and are transferred to the client as frozen models. The client then uses the outputs of these teachers to supervise a student model through unidirectional knowledge distillation.
This setup supports privacy preservation by avoiding raw data exposure while allowing lightweight clients benefit from distributed expertiso. It also climinates the need for coordinated model updates or gra- or asynchronous
dient exchange across providers, enabling operation in low-trust, environments.
4

---

Published in Transactions on Machine Learning Research (08/2025)
Overview of the Approach
The approach consists of the following components (illustrated in Figure D:
(1) set of providers holding local datasets; (2) monoclass teachers trained to recognize individual classes while rejecting unfamiliar inputs as "other"; (3) a client-side mochanism to request and aggregate teacher outputs for desired classification task; and (4) a student network trained to mimic the aggregated teacher signal in an unsupervised manner.
T
Monocless
Provider
Provider 2
Provider 3
S
Student
T1
Tt
Dalasel
Private
Private data
Private data
data
Tn
Tn
Tn
Class request
Distillation
Client
Client data
S
T
Figure 1: System overview of the proposed decentralized distillation approach. Each provider trains set of monoclass teacher models (T) on private local data without sharing raw inputs. The client requests teachers for specific classes and receives frozen models. These teacher models are then used to supervise a student model (S) on unlabeled client data through unidirectional knowledge distillation. The legend shows the roles of cach component: datasets (cylinders), monoclass teachers (T), student model (S); class request (solid arrow), and distillation flow (dashed arrow).
Providers and Class Assignment
Let there P providers. Each provider E (l....,P) holds data corresponding to a subset of classes:
Cp (1,2..... K}.
where K is the total number of classes. in the datasets of multiple providers. selection and aggregation.
The class subsets Cp may overlap, meaning a given class k can appear This redundancy can contribute to increased robustness in teacher
Monoclass Teacher Training
To share knowledge without exposing raw data, each provider trains a scparate monoclass teachor for each supported class. Each teacher is trained to perform binary classification: distinguishing its main class from an aggregated other class composed of out-of-scope samples. Formally, for an input æ, a teacher for class k from provider p outputs:
Ammaii (r): the logit corresponding to the main class.
(p) *kother (x)): the logit corresponding to the "other" class.

---

Published in Transactions on Machine Learning Research (08/2025)
This design enables teachers to encapsulate provider-specific expertise while remaining agnostic to classes they have not seen.
Client Request and Teacher Selection
A client defines set of requested classes:
P CR S Upp. p=l
with cardinality [Cxl C. For each class CR, let
Px (plkEE
denote the set of providers offering class k. When multiple providers offer the same class, the client selects single teacher to use, following simple criterion:
T(k) min Pk.
i.e., the provider with the smallest index. This selection strategy can be extended in future work using confidence scores or quality metrics.
Teacher Output Aggregation
Each selected teacher T(k) produces its main logit 2k, (a) for input æ. The client aggregates these logits into unified prediction vector:
Zagz (x) ( (Zagg.Aa (I),, Zagg,bz (x), ..... Zagg ko (x)) RC,
where begg,b(2)) (z) =
2 main (x) and CR {ki, ......, kc). For simplicity, only the main logits are used for
(T)))
aggregation in this work, but the "other" logits may be incorporated in future extensions via weighted fusion or uncertainly modeling.
The full aggregation and distillation process, including logit selection, softmax normalization, and student supervision, illustrated in Figure 2
Student Model and Distillation Loss
The student network receives the same input x and produces logit vector:
zs x
(2 (x), (s), .... zg,ke (x)) E R²C
Since the client operates in an unlabeled setting, the student is trained match the Seacher-provider aggregated logits using mean squared error (MSE) loss:
LMsE(
Σ
(1) (
kECn
equivalently,
LMSE(1) = [| (x))1
This distillation approach allows the student to learn from soft signals derived from independently trained and fragmented sources without requiring access to labels or provider data.
Reuse and Contributions.
This approach reuses standard components of knowledge distillation, including the use of soft predictions and MSE-based student supervision. However, the key novelty lies in the one-shot federated setup using monoclass teacher models, the unlabeled aggregation of fragmented outputs, and the analysis of how such fragmentation impacts student learning. Unlike traditional distillation or federated learning setups, our method assumes collaboration between providers and access to ground-truth labels the client side.

---

Published in Transactions on Machine Learning Research (08/2025)
Client image
Cat teacher
Bird teacher
Dog teacher
Monoclass teachers
Other
Other
Other
Raw teacher logits
Cat
Bird
Dog
Teachers logits
Other
Other
Other
Cat
Bird
Dog
Student
after
softmax
Aggregated
Bird
Doog
teachers output
Student logits
Distillation Loss (MSE)
Figure 2: Illustration of the distillation process using monoclass teachers. A shared client image is passed all relevant monoclass teachers (c.g., Cat, Bird, Dog). Each teacher produces logits for its main class and "other" class. After applying softmax, the main logits are aggregated into a unified prediction vector. This vector supervises the student model via mean squared error (MSF) loss.

---

Published in Transactions on Machine Learning Research (08/2025)
4 Experiments
The goal of the experiments is to evaluate the effectiveness of the proposed knowledge-sharing method in distilling accurate and generalizable models from Iragmented sources of expertise. In many real-world scenarios, knowledge providers may be specialized in only specific subset of classes. Due to privacy resource constraints, these providers can typically share only teacher models trained on their local, limited data.
The proposed method addresses this challenge by aggregating outputs from specialized teachers. These teachers may come from different providers, each exposed to a different subset of classes. Their predictions are used to train single unified student model. A central difficulty lies in integrating this fragmented knowledge. Predictions may be inconsistent, uncalibrated, biased due to the varying and limited training scopes of individual providers.
4.1 Motivation
This work focuses scenarios with high knowledge fragmentation, where each provider contributes only or few monoclass teachers. Such settings reflect practical constraints in domains like healtheare, industry, or edge-device learning. In these contexts, privacy, bandwidth, or computational limilations often prevent centralized training.
The proposed approach is intended for deployment. Both teacher and student models are compact and suitable for resource-constrained environments, such mobile devices or IoT platforms. Demonstrating effective generalization under these constraints is central objective.
4.2
Experimental Objectives
The experiments designed to answer the following questions:
How well can a student learn from fragmented supervision?
How does fragmentation intensity affect generalization?
Do stronger monoclass teachers improve aggregation quality?
How close can the student get to fully supervised performance?
4.3 Experimental Setup
Evaluation is conducted on three standard image classification benchmarks: MNIST, FashionMNIST,, and CIFAR-10, representing increasing levels of difficulty. Each dataset is split into training and test sets. From the training set, 80% is used to train teacher models, and 20% reserved as unlabeled data for the client (student). Only the outputs from the teachers are available to the student during training.
Model Architecture. Both leacher and student models use the same compact convolutional architec- ture. Designed for deployment in low-resource environments, it comprises fewer than 100K parameters while maintaining strong representational capacity. The architecture fcatures two convolutional blocks with batch normalization and pooling, followed by lightweight. fully connected head. This shared design ensures consis- tency in capacity and learning dynamics across both roles. The complete layer configuration is summarized in Table [
Teacher and Student Sctup. Each teacher trained as a binary classifier to distinguish one target class from all others. Its output logits correspond the assigned class ("main', label 0) and the rest ('other', label 1). The student is standard multiclass classifier trained to predict across all classes.
8

---

Published in Transactions on Machine Learning Research (08/2025)
Layer
Type
Output Shape
Details
Input
(1 or 3, 32, 32)
Depends on dataset
Conv1
Conv2D
(8, 32, 32)
8 filters, kernel size 3, padding 1
BN1
BatchNorm22
(8, 32, 32)
ReLU + MaxPool
Activation + Pool
(8, 16, 16)
Pool size 2x2
Dropout1
Dropout
(8, 16, 16)
p= 0.3
Conv2
Conv2D
(16, 16, 16)
16 filters, kernel size 3, padding 1
BN2
BatchNorm2D
(16, 16, 16)
ReLU MaxPool
Activation + Pool
(16, 8, 8)
Pool size 2x2
Dropout2
Dropout
(16, 8)
p 0.3
Flatten
(1024)
Flatten from 16x8x8
FC1
Linear
(16)
Followed by LayerNorm and ReLU
Dropout3
Dropout
(16)
p= 0.3
FC2
Linear
(10)
Number of output classes
Table 1: Architecture of the model used for both teachers and student.
Teacher Output Aggregation. inference, each Leacher's logits are passed through a softmax over its two outputs. We extract the softmax probability corresponding to the "main" class and concatenate these across all teachers. This produces a pseudo-logit vector of dimension N (number of teachers), used as a soft target to train the student using mean squared error (MSE) loss.
Training Protocol. Teachers are trained for 200 epochs with cross-entropy loss, without class weighting, The student is trained for 200 epochs with MSE loss on aggregated soft outputs. Both use the Adam optimizer with a learning rate 0.001. Batch size is 2048 for teachers and 512 for the student.
Supervised Client Baseline. As reference, the architecture is trained using labels from the client split. This provides a supervised baseline for comparison with the distillation-based student.
4.4 Fragmentation Configurations
Five fragmentation configurations are defined to simulate varying degrees of knowledge distribution across providers. Fach provider (i.e., teacher) trained on specific subset of classes. As fragmentation increases, the number of providers grows and each teacher's view of the full class distribution becomes more limited.
Setup
Providers
Classes Provider
Class Assignments
10
10,1,2,3,,,5,66,,888, [0,1,2,3,4), [5,6,7.8,9]
2
5
C
4
4
[0,1,2,3], [3,4,5,6], 8697, [9,0,1,2] [0,1,2], [3,4,5], [6,7,1] [8,9,2] [0,1], [2,3], [4,5], [6,7], [8,9]
4
3
E
5
2
Table 2: Fragmentation setups showing the number of providers, classes per provider, and class assignments.
Sctup A represents centralized training with full class access. Sctup corresponds to the maximum frag- mentation, where each provider sees only two disjoint classes. Intermediate setups simulate increasing specialization and overlap.

---

Published in Transactions on Machine Learning Research (08/2025)
4.5
Evaluation Metrics
The method is evaluated using the following metrics:
Binary Teacher Accuracy: Each teacher is trained to detect one specific class. It performs a binary classification task. The output is 0 for the main class and for the "other" classes The reported accuracy is the average of the accuracy on the main class and the "other" class.
Predictions are obtained from the concatenated 'main class" probabilities across all teachers. A confusion matrix is computed to assess the quality of the aggre- gated ensemble output prior to student training.
Aggregation Confusion Matrix:
Student Accuracy: The student top-1 accuracy on the test set measures how well it generalizes from teacher outputs without label supervision.
Supervised Client Accuracy: Accuracy of the same model trained with labeled client data. This serves performance upper bound under the same data budget.
4.6 Summary
These experiments quantify the trade-ofls between model specialization and generalization under fragmented supervision. By distilling knowledge from multiple narrow teacher models into a unified student, the fea- sibility of label-free generalization in deploymeent-constraimne settings demonstrated. The experimental setup isolates the effects of fragmentation and enables evaluation of how aggregation quality impacts student performance.
5 Analysis of Knowledge Fragmentation Impact
The effect of knowledge fragmentation is evaluated by measuring teacher accuracy, aggregation quality, and student generalization across increasing levels of fragmentation. Five fragmentation setups are tested on MNIST, FashionMNIST, and CIFAR-10. These datasets differ in complexity and class similarily, providing broader view of how fragmentation impacts performance.
5.1 MNIST: Low Complexity Domain
In MNIST (Figure 3h, teacher models retain high accuracy across all fragmentation levels. Performance decreases slightly, from 99.27% in the full-data setup to 79.88% in the most fragmented configuration. This shows that local learning remains strong even with limited class coverage. In contrast, aggregation accuracy drops more sharply, from 99.12% to 42.39%, indicating that outputs from disjoint teachers become harder to combine as class overlap decreases.
Student performance closely follows the decline in aggregation quality. Accuracy drops from 98.46%, near the full-data. baseline of 98.36%, to 44.47%. The student model fails to generalize when trained inconsistent and fragmented aggregated outputs. These results suggest that, in low-complexity settings, fragmentation primarily degrades the aggregation step.
Figure Ahowss progressive degradation in prediction coherence. Setup nearly diagonal, indicating strong alignment. Setup introduces slight confusion (e.g., vs. 5), while setup shows increased confusion for shared digits like 3, 7, and In setups D and F, matrices become noisier- digit 1 remains clear, but digit is misclassified in setup D despite multiple appearances. By setup E, the class structure collapses, and aggregation fails to unify teacher outputs.
5.2 FashionMNIST: Medium Complexity Domain
In FashionMNIST (Figure 5 teacher accuracy remains relatively high, decreasing from 95.43% 74.34% across setups. Aggregation accuracy, however, shows more severe drop drop-from 92.86% in the full setup to
10

---

Published in Transactions on Machine Learning Research (08/2025)
44.88% in the most fragmented case. This decline reflects the difficulty of merging outputs when classes are visually similar and rely shared context for disambiguation.
Student accuracy is affected in the same way. Performance decreases from 89.14% to 44.47%. In earlier setups, the student performs near the baseline of 89.31%, but generalization fails when aggregation becomes unreliable. These results suggest that in mediun-complexity tasks, fragmentation amplifies inconsistency among teachers, especially for fine-grained class distinctions.
In Figure [ setup A shows strong class separability, with predictions aligned along the diagonal. Setup B introduces confusion among visually similar classes like shirts (6), coats (4), pullovers (2), and T-shirts (0). In setup C, this confusion intensifies, particularly for coats and T-shirts, due to inconsistent predictions from overlapping groups. Setups D and show widespread misclassifications, with shirts and pullovers frequently confused, and structure breaking down entirely the disjoint setting.
5.3 CIFAR-10: High Complexity Domain
In CIFAR-10 (Figure 7 teacher accuracy starts at 80.72% and falls to 66.22% as fragmentation increases. Aggregation suffers a stronger docline, dropping from 74.28% to 29.46%. Visually similar classes (c.g., cats vs. dogs, trucks vs. cars) contribute high ambiguity across teachers. When context lost, aggregation consistency is severely reduced.
Student accuracy starts at 61.18%, near the bascline (61.08%), but drops sharply to 29.94%. Fragmentation disrupts not only the integration of local predictions but also the formation of global class boundaries by the student. Aggregated outputs lose coherence, and learning becomes unstable.
In Figure setup shows relatively good class separability. In setup B, misclassifications increase sharply across the boundary, particularly between classes 3-5, 0 and 8 and 9. This indicates aggregation challenges across disjoint providers. In sctup C confusion persists among shared animal classes (e.g., 3, 5, 6), suggesting misaligned knowledge despite partial redundancy. In setup D, narrow overlaps lead to severe ambiguity for classes like cats (3), deer (4), and birds (2), which are spread across multiple groups. Finally, setup E, with disjoint class pairs, exhibits the most severe degradation.
11

---

Published in Transactions on Machine Learning Research (08/2025)
100
(98.36%) Teacher Avg Binary Ace
80
Softmax Aggregation Acc
Accuracy
40
20
I[0. 10..1P.3144.11..11.00 71181911 116
001.2344.......
[10. 101.21B4411.. 122940
110.12
Sctup
Figure 3: Comparison of teacher, aggregation, and student accuracy across fragmentation setups for MNIST.
0
3339
0
0
1
0
1331
5
0
0
5
0
3337
0
0
5
0
0
2
3175
0
1
3
5
893
5
18
150
117
0
4
1145
2
0
25
0
2207
52
338
49
Lood
2015
3
0
3
2
1060
0
5
0
2063
11
1
0
2
1370
1
0
6-
0
0
1168
0
2
6
19
4
2
1140
1
0
3
12399
1230
49
340
10
19
1
3
1182
9-
127
1047
24
534
14
580
Labal
Predicted
Label
(a) Setup A: Full access
(b) Setup B: Split halves
(c) Setup C: Overlapping groups
D01122
22
19
101
10
1319
13
438
334
162
1
203
34
3
0
1
32
12
14
37
10
2044
10
19
42
37
0
160
20
42
4
261
196
240
Thue Lobel
131
186
375
386
25
209
517
46
84
642
0
122
35
0
91
1
0
122
873
0
214
30
0
7
0
999
180
28
118
40
241
133
8
132
433
220
Predicted
Pradicted Label
(d) Setup D: Narrow overlaps
(e) Setup E: Disjoint pairs
Figure 4: Confusion matrices of aggregated teacher outputs for MNIST across fragmentation setups.
12

---

Published in Transactions on Machine Learning Research (08/2025)
100
(89.31%)
leacher Aug Binary Ace
80
Softmex Aggregation Acc
Student Acc
Accuracy (%)
20
[1o. 001.2/8.4.9M02211.0000 11/8.9211
110.1.23445...30
0O. 00. 00AM233.4..166.11.000
[10.40
Setup
Figure 5: Comparison of teacher, aggregation, and student accuracy across fragmentation setups for Fash- jonMNIST.
100
1
0
1281
2
13
3
0
2175
3
0
2
19
0
1057
12
f1
0
41
0
0
23
1040
6
32
89
0
3
0
510
33
289
1
370
1
0
21
:123
33
0
1100
15
37
13
1152
Label
1.9
2
2
1180
2
165
29
682
299
6-129
72
35
57
0
8999
7
0
368
180
37
351
258
6
30
317
68
56
1
71.
1
1
L170
17
3
1167
85
1097
13
0
0
1160
23
1162
0
1130
Predicted Label
Predicted
Predicted Label
(a) Setup A: Full access
(b) Setup B: Split halves
(c) Setup C: Overlapping groups
1022
33
9
0
948
24
64
156
1165
72
3
0
1001
0
46
40
19
407
150
29
527
11
25
6
156
140
0
35
41
0
825
True Labre
18
7
1143
0
935
23
120
24
97
1043
5
2
974
63
68
64
6
5
1116
6
1
0
0
952
0
246
4
8
18
3
3160
754
12
19
29
227
196
Prodicted Label
Predicted Label
(d) Setup Narrow overlaps
(e) Setup Disjoint pairs
Figure 6: Confusion matrices of aggregated teacher outputs for FashionMNIST across fragmentation setups.
13

---

Published in Transactions on Machine Learning Research (08/2025)
100
(61.08%)
Teacher Avg Binary Acc
80
Softmex Aggregation Acc
Student Acc
Accuracy (%)
20
[1o. 001.213..5.M...1110000 11/8..1111
110. 110.1.234466..333
ro) 10122300 12-3113330 21123313..4..6666....33 7.1.7.....111........
1O. 00. N0.M22344..11..110000
Setup
Figure 7: Comparison of teacher, aggregation, and student accuracy across fragmentation setups for CIFAR- 10.
10
067
2
19
22
53
786
5
12
20
3
12
761
3
7
42
55
85
57
0
614
35
114
51
59
25
29
8
24
395
41
64
140
159
64
83
18
29
519
4/
6
42
54
152
12/
1/
450
80
230
17
32
433
116
23
9
23
15
55
57
18
11
115
42
667
36
283
654
32
36
2
9
81
154
2
290
16
383
35
10
5
43
39
45
36
812
8
6-
21
75
37
33
805
13
5
9
133
80
3
5
727
13
13
45
49
15
30
71
11
16
16
14
34
03
6
30
797
200
34
24
564
369
16
24
461
Label
Predicted Label
(a) Setup A: Full access
(b) Setup B: Split halves
(c) Setup C: Overlapping groups
607
16
21
22
20
153
61
660
81
24
20
43
11
655
80
681
213
84
241
233
242
13
75
23
27
119
210
10
51
257
227
42
155
133
41
26
357
244
1
4
33
Tru Labre
13
314
255
261
28
270
103
27
128
41
348
3
7
44
345
118
20
36
787
35
22
12
82
22
38
21
896
97
172
203
76
66
83
195
22
5
1
87
85
67
14
3
17
761
12
505
120
40
9
62
114
2
83
10
Prodicted Label
Predicted Label
(d) Setup Narrow overlaps
(e) Setup Disjoint pairs
Figure 8: Confusion matrices of aggregated teacher outputs for CIFAR-10 across fragmentation setups.
14

---

Published in Transactions on Machine Learning Research (08/2025)
5.4 Discussion
The results highlight a consistent and significant impact of knowledge fragmentation across all datasets. As fragmentation increases, student performance declines sharply, despite relatively stable teacher accuracy. This drop is primarily driven by degraded aggregation quality.
In MNIST, a low-complexity dataset with clearly separable classes, teacher accuracy remains high across all sctups. Aggregation performance declines more rapidly, especially with disjoint or narrowly overlapping subsets. Still, the task structure allows some generalization, and student accuracy stays moderate until extreme fragmentation.
MNSS shows more subtle inter-class differences. Teachers remain locally accurate, but aggregation becomes less reliable as fragmentation increases. Visual similarity between classes such shirts, coats, and pullovers leads to confusion when global context is lost. Student performance drops earlier and more sharply than in MNIST.
CIFAR-10 shows the strongest sensitivity. Even with full data, aggregation and student accuracy are lower due visual complexity and semantic overlap between classes. Fragmentation quickly leads to inconsistent aggregated outputs, and student learning collapses under moderate-to-higgh fragmentation. The teacher ensemble fails to provide coherent supervision.
Across datasets, fragmentation produces noisy aggregated outputs. This is not caused by poor teacher accuracy on their assigned classes, but by their inability to correctly handle out-of-scope inputs. The bottleneck lies in the quality of the outputs from the teachers, which directly impact aggregation. As a result, the student receives inconsistent supervision and fails to generalize. Mitigating this requires reducing noise at the teacher level, particularly how unscen classes are trcated.
While privacy motivates our decentralized and communication-free design, it is not empirically evaluated in this work. Our approach assumess white-box access to teacher models and does not incorporate formal privacy-preserving mechanisms. Future work could explore integrating differential privacy (DP), secure enclaves, or cryptographic aggregation to strengt protection against privacy leakage.
As the number of classes increases, we expect fragmentation effects to intensify. Larger label spaces increase the likelihood of both semantic overlap between classes and severe supervision gaps across experts. such settings, monoclass teachers are more likely produce conflicting or overconfident predictions on out- of-scope examples, further degrading aggregation. While current experiments are limited to ten-class datasets, extending this approach to more fine-grained or large-scale classification tasks (e.g., CIFAR-100, ImageNet) remains an important direction for future work. Mitigation strategies such as hierarchical class grouping, broader "other" class exposure, or adaptive aggregation will likely become even more critical in such scttings.
6 Improving Teacher Generalization under Fragmented Supervision
In highly fragmented settings, each teacher is exposed to limited subset of the label space. Without explicit mechanisms to distinguish between in-class and out-of-class inputs, teachers often produce overconfident or inconsistent outputs on unfamiliar samples. These errors compound during aggregation, reducing supervision quality and harming generalization.
To address this, several strategies can be adopted improve teacher robustness and reduce noise in their outputs, particularly when encountering out-of-scope inputs.
1. Expose Teachers to Rich and Varied "Other" Distribution To improve the ability of teachers to reject inputs outside their assigned classes, training should include a diverse set of examples grouped inlo a generic 'other' class label. These samples should span a wide range of shapes, textures, and semantics, encouraging teachers to build robust and discriminative boundaries. Unlike random negatives, this curated "other" distribution increases coverage of the input space and helps reduce overconfident misclassification unfamiliar samples. Such data can be sourced from unassigned training classes external datasets.
15

---

Published in Transactions on Machine Learning Research (08/2025)
2. Shape the Loss to Improve Robustness on "Other" Samples To increase robustness, the loss function can be adapted to handle the diverse and ambiguous nature of "other" samples more effectively. Beyond standard classification loss, additional terms can be used to encourage uncertainty, penalize overcon- fidence, or emphasize margin separation between assigned and non-assigned classes. This tailored objective helps the teacher produce more slable outputs when facing unfamiliar dala, reducing misclassification and improving the quality of aggregation.
3. Apply Contrastive Learning to Enhance Representations Contrastive learning can be used within each teacher to improve feature separation between assigned classes and "other" samples. By pulling together representations of in-class examples and pushing away those from the "other" category, the teacher learns more discriminative and robust embeddings. This leads to better class boundaries and reduces con- fusion when facing unfamiliar inputs, ultimately supporting more consistent aggregation.
These strategies aim to mitigale the core challenges of fragmented supervision by making teachers more aware of what lies outside their expertise. In doing so, they produce softer, more calibrated outputs unfamiliar inputs-ultimatee enabling the student to learn from more stable and informative aggregated signal.
7 Conclusion
We studied the impact of knowledge fragmentation on one-shot foderated distillation using monoclass teach- ers. Through systematic ex experiments on MNIST, FashionMNIST, and CIFAR-10, we showed that increasing class fragmentation significantly degrades student performance, with effects intensifying as visual complexity grows.
Our analysis revealed that fragmented supervision introduces ambiguity and inconsistency in the aggregated predictions. Although individual teachers perform well on their assigned classes, their lack of exposure to diverse inputs lcads to overconfident and unreliable outputs on unfamiliar examples, ultimately impairing the distillation process.
To mitigate these effects, we discussed several strategies, including exposing teachers a broader "other' class during training, penalizing overconfidence, and enhancing feature separation through contrastive learning, These approaches aim to enhance the robustness of teachers and the stability of aggregation under fragmented supervision.
Overall, our findings highlight critical limitations in naively combining narrow experts for knowledge dis- tillation. In future work, we plan to further refine mitigation techniques and investigate more adaptive aggregation methods to support knowledge transfer in highly fragmented, privacy-sensitive environments.
Broader Impact
one-shot distillation method that enables decentralized model train- ing using monoclass teacher models. While the method avoids data sharing and reduces coordination overhead, it may lead lo overconfident predictions on out-of-distribution inpuls when teachers are trained narrow class subsets. These risks are particularly relevant in sensitive applications such as healthcare or surveillance. Future applications should incorporate uncertainty calibration or robust fallback strategies to prevent misuse.
This work proposes
References
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in neural information processing systems, 33:3557-3563 2020.
16

---

Published in Transactions on Machine Learning Research (08/2025)
Ziqing Fan, Jiangchao Yao, Bo Han, Ya Zhang, Yanfeng Wang, et al. Federated learning with bilateral curation for partially class-disjoint data. Advances in Newral Information Processing Systems, 36:32006- 32019, 2023.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(100): 2022.
Takashi Fukuda, Masayuki Suzuki, Gakuto Kurata, Samuel Thomas, Jia Cui, and Bhuvana Ramabhadran. Efficient knowledge distillation from an ensemble of teachers. In Interspeech, pp. 3697-3701, 2017.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Compuler Vision, 129(6):1789 1819, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0255 2015.
Ming Hu, Zhihao Yue, Zhiwei Ling, Xian Wei, and Mingsong Chen. Fedmr: Fedreated learning via model recombination. arXiv preprint arXiv:2208.00 07677, 2022.
Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, and Koji Yamamoto. Distillation- based semi-supervised federated learning for commuicatioon-fffiicnn collaborative training with non-iid private data. IEEE Transactions on Mobile Compuling, 22(1):191 205, 2021.
Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 87, 1991.
Myeongkyun Kang, Philip Chikontwe, Soopil Kim, Kyong Hwan Jin, Ehsan Adeli, Kilian M Pohl, and Sang Hyun Park. One-shot federated learning on medical data using knowledge distillation with image synthesis and client model adaptation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 521-531. Springer, 2023.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint Sii:2006.16666 2020.
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXie preprinl arXiv:1910.08: 035811 2019.
Qinbin Li, Bingsheng He, and Dawn Song. Practical one-shot federated learning for cross-silo setting. arXiv preprint r:Xiu:2010010011 2020.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International conference on machine learning, pp. 6357-6368. PMLR, 2021.
Xin-Chun Li and De-Chuan Zhan, Fedrs: Federatod learning with restricted softmax for label distribution non-iid data. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp. 995-1005, 2021.
Tao Lin, Lingjing Kong, Sebastian Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in foderated learning. Advances in neural information processing systems, 33:2351-2363, 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Commmicationn-ffiiiontt learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273-1283 PMLR, 2017.
Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, lan Goodfellow, and Kunal Talwar. Semi-supervised knowledge transfer for deep learning from private training data. arXiv preprint arXiv:1610.0575,, 2016.
Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. Routing networks and the challenges of modular and compositional computation. Xix preprint Xrxi::1904 12774, 2019.
17

---

Published in Transactions on Machine Learning Research (08/2025)
Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongteng Huang, and Xing Xic. Communication-cffieeeni feder- ated learning via knowledge distillation. Nature commumications, 13(1):2032, 2022.
Hailin Zhang, Defang Chen, and Can Wang. Confidence-awaree multi-teacher knowledge distillation. In TCASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (TCASSP): pp. 4498 4502. IEEE, 2022a.
Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learning with label distribution skew via logits calibration. In International Conference on Machine Learning, pp. 26311 1-2629.. PMLR, 2022b.
18


Reviews:
[{"review_id": "MG11W8Xxcs", "reviewer": "Reviewer_oT3w", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The structure is unclear, and the idea of simple teacher selection is underdeveloped.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This draft has confusing motivation. For example, it states, “However, privacy, legal, and proprietary concerns often limit direct data sharing.” Additionally, “While federated learning (FL) enables distributed model training, it typically assumes mutual benefit, requires repeated communication, and produces a shared global model.” It is strange that FL is introduced with \"However\" since it also has privacy and sharing data issues.  Then, it suddenly shifts to “one-shot federated distillation.” Later, the focus changes to the OOD setting. I cannot continue my review.\n\nThe authors are required to state their question clearly before the review process.\nThe quality of this draft clearly lowers the standards of TMLR.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "It would benefit from a more coherent organization and a deeper exploration of the idea.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fACq3uDJGm", "reviewer": "Reviewer_fPw7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n+ This paper explores an interesting topic on a one-shot federated distillation method. The paper addresses the important and timely problem of privacy-preserving knowledge transfer and model distillation in federated and decentralized environments. \n+ The authors show some experimental studies on the proposed method across multiple datasets.\n\n\n### Weakness\n- The primary concern is that the core innovative contribution is neither clearly stated nor sufficiently differentiated from prior literature. The related work discussion is general. It remains unclear which particular direction the authors follow. Moreover, this paper lacks comparative baselines. Thus, it is unclear whether the proposed method has advantages compared with the existing methods.\n- Although the methodology in section 3 explained each step of the proposed method. It is unclear which part is new and which part is from the current literature.\n- Comparative experiments against existing baselines are lacking. It is important to compare the proposed method with state-of-the-art methods, such as  [1]. \n- Additionally, the proposed idea is motivated by improving privacy. However, the experimental part lacks the attacking tests or privacy verification. That is, the claimed privacy properties are not empirically supported or discussed in terms of formal privacy guarantees (e.g., differential privacy, membership inference resistance).\n\n[1] Federated Distillation: A Survey. Lin Li, Jianping Gou, Baosheng Yu, Lan Du, Zhang Yiand Dacheng Tao.\n\n- The discussions on \"improving teacher generalization\" (Section 6) are general. More experimental results to support the discussion are expected.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on the challenge of knowledge transfer in distributed machine learning environments where data sharing is constrained by privacy, legal, or proprietary concerns.  The authors consider the knowledge distillation (KD), which allows a student model to learn from\nteacher predictions.\n\nSpecifically, the authors propose a one-shot federated distillation framework using monoclass teacher models. In this setup, each provider independently trains a set of teacher models, each specialized in a single class, and shares the frozen model weights once with a client. The client then distills a global multi-class student using only the predictions of these monoclass teachers on unlabeled local data.\n\nExperiments on MNIST, FashionMNIST, and CIFAR-10 show that increasing fragmentation consistently degrades accuracy. To address this, the authors explore three mitigation strategies: (1) exposing teachers to diverse off-class examples, (2) penalizing overconfidence in the student loss, and (3) applying contrastive learning to sharpen feature boundaries.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Clearly state what is new compared with prior works in federated and knowledge distillation (particularly in relation to the \"one-shot\" and \"fragmented\" settings). Outline concrete distinctions from survey-cited methods.\n- Include experimental results and quantitative comparisons with established baselines (e.g., classical federated distillation, multi-teacher KD, or recent privacy-preserving distillation protocols such as those reviewed in [1]).\n- Explicit Attribution: For each methodological component, specify if it is adopted from existing literature, or describe succinctly how it departs from precedent.\n- To support the privacy motivation, add empirical or theoretical analyses such as privacy risk/attack experiments, ablation of privacy mechanisms, or mention of compatibilities with formal frameworks (e.g., differential privacy).\n- Elaborate on the discussion in Section 6 with concrete evidence.", "broader_impact_concerns": "The research motivation is particularly relevant in sensitive applications such as healthcare or surveillance.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HSVOxnhmTR", "reviewer": "Reviewer_9Jpp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) Well motivated. The missing class problem among federated clients is a common and critical challenge in the development of federated learning algorithms. The author tries to solve this problem through one-shot distillation.\n2) Clear presentation. The contents in the paper are clear and easy to follow.\n3) Performance shown in Figure5, and Figure 7 seems good.\n\nWeaknesses:\n1) The paper lacks sufficient reference citations. For example, the entire introduction section has no references to support their claims. Additionally, the missing class problem is a widely studied topic, but the paper does not cite relevant works in this area. To name a few:\n* [ICML]FedAWS: Federated learning with only positive labels\n* [KDD]FedRS: Fedrs: Federated learning with restricted softmax for label distribution non-iid data\n* [TMLR]FedMR: Federated learning under partially class-disjoint data via manifold reshaping\n* [ICML]FedLC: Federated learning with label distribution skew via logits calibration\n* [NeurIPS]FedGELA: Federated learning with bilateral curation for partially class-disjoint data\n\n2) I am concerned about the applicability of this method to scenarios with a larger number of classes. It would be helpful to evaluate the method on datasets like TinyImageNet, which contains 100 or more classes, to assess its scalability and performance.\n3) The mention of \"modular learning architectures\" in the last paragraph of the introduction seems unclear. It is not explained why this approach is introduced or how it helps address the problem. Providing a clearer rationale and connection to the problem would improve understanding.\n4) There appears to be an issue with the manuscript's flow between Sections 5.3 and 5.4.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a one-shot federated distillation method where a client learns a multi-class student model by aggregating knowledge from monoclass teacher models provided by multiple data sources. A key challenge is out-of-distribution (OOD) supervision, where monoclass teachers mispredict unseen inputs, leading to noisy signals that degrade student performance. Experiments on MNIST, FashionMNIST, and CIFAR-10 show that increased knowledge fragmentation reduces student accuracy. To mitigate this, the authors propose three strategies: exposing teachers to diverse off-class examples, penalizing overconfidence, and using contrastive learning to sharpen feature boundaries.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) The manuscript needs to include appropriate citations for references throughout, particularly in the introduction and related work sections. \n2) The effectiveness of the proposed method should be verified with scenarios involving more classes.\n3) As for other minor requested changes, please see weaknesses part.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)