Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
Authors: Guan Huang
Tao Shu
Venue: ICML 2025 poster (2025)
Keywords: Low Earth Orbit Satellites
Model Personalization
One-Shot Federated Learning
Communication Constraints
Primary area: applications
TL;DR: We introduce Federated Oriented Learning (FOL), a one-shot personalized federated learning framework that improves local models under communication constraints through model alignment, ensemble refinement, and knowledge distillation.

Abstract: Personalized Federated Learning (PFL) has become a promising learning paradigm, enabling the training of high-quality personalized models through multiple communication rounds between clients and a central server. However, directly applying traditional PFL in real-world environments where communication is expensive, limited, or infeasible is challenging, as seen in Low Earth Orbit (LEO) satellite constellations, which face severe communication constraints due to their high mobility, limited contact windows. To address these issues, we introduce Federated Oriented Learning (FOL), a novel four-stage one-shot PFL algorithm designed to enhance local model performance by leveraging neighboring models within stringent communication constraints. FOL comprises model pretraining, model collection, model alignment (via fine-tuning, pruning, post fine-tuning, and ensemble refinement), and knowledge distillation stages. We establish two theoretical guarantees on empirical risk discrepancy between student and teacher models and the convergence of the distillation process.  Extensive experiments on datasets Wildfire, Hurricane, CIFAR-10, CIFAR-100, and SVHN demonstrate that FOL consistently outperforms state-of-the-art one-shot Federated Learning (OFL) methods; for example, it achieves accuracy improvements of up to 39.24\% over the baselines on the Wildfire dataset.

Recently, Personalized Federated Learning (PFL) has been proposed an effective means to address the sub-optimal performance of the global model produced by federated learning (FL) (Li al., 2020b; Yang al., 2019; Huang Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
Guan Huang'
Tao Shu
Abstract
et al., 2024) when the model is used at local clients with het- erogeneous (i.e., non-IlD) datasets (Huang et al., 2023; Ye et al., 2023). Benefiting from the shared knowledge of the global model, PFL adapts it to form personalized local mod- els at each client, which perform better than both the global model and those local models trained independently on each client's data. (Tan et al., 2022a; Deng et al., 2020; Collins et al., 2021). Existing works in PFL can be broadly catego- rized into two groups: (1) Parameter decoupling approaches, which divide the model into shared and personalized com- ponents. The shared backbone (feature extractor) captures generalizable knowledge across clients, while the personal- ized head (classifier) is specifically adapted each client's unique data distribution (Liang al., 2020; Oh et al., 2021: Tan et al., 2022b; Su et al., 2023a; Wu al., 2024). (2) Optimization- basedd approaches, which employ advanced optimization techniques, such as regularization-be -based meth- ods, dynamic aggregation, and second-order optimization, to constrain local models and adjust aggregation weights, thereby balancing global collaboration with local personal- ization (T Dinh al., 2020; Li al., 2021; Luo et al., 2023; Liu al., 2023a; Yang et al., 2024c).
Personalized Federated Learning (PFL) has be- come a promising learning paradigm, enabling the training of high-quality personalized models through multiple communication rounds between clients and a central server. However, directly ap- plying traditional PFL in real-world environments where communication is expensive, limited, in- feasible is challenging, as seen in Low Earth Orbit (LEO) satellite constellations, which face severe communication constraints due to their high mo- bility, limited contact windows. To address these issues, we introduce Federated Oriented Learning (FOL), novel four-stage one-shot PFL algorithm designed to enhance local model performance by leveraging neighboring models within strin- gent communication constraints. FOL comprises model pretraining, model collection, model align- ment (via fine-tuning, pruning, post fine-tuning, and ensemble refinement). and knowledge distil- lation stages. We establish two theoretical guar- antees empirical risk discrepancy between stu- dent and teacher models and the convergence of the distillation process. Extensive experiments on datasets Wildfire, Hurricane, CIFAR-10, CIFAR- 100, and SVHN demonstrate that FOL consis- tently outperforms state-of-the-art one-shot Fed- erated Learning (OFL) methods; for example, achieves accuracy improvements of up to 39.24% over the baselines on the Wildfire dataset.
However, as today's 's model becomes larger and larger (e.g., the number of parameters of a state-of-the-art transformer- based models ranges from tens to hundreds of billions (De- vlin et al., 2019; Dosovitskiy et al., 2020; Huang & Shu, 2024), notable limitation of existing PFL methods is their dependence on multiple communication rounds update models while in each round the amount of data that needs to be communicated is massive. While iterative communi- cation is essential for progressively refining personalized models and improving accuracy, it becomes impracticable in many real-world scenarios where communication oppor- tunities are expensive, limited, or infeasible. For instance, in Low Earth Orbit (LEO) satellite constellations, such as Starlink (SpaceX, 2020), OneWeb (OneWeb, 2021), and Kuiper (Amazon, 2022), satellites are strategically deployed in different orbits for distinct observational tasks (c.g.. some predominantly monitor polar regions, while others focus on oceans or landmasses) (Doe & Smith, 2019). This special- ized deployment makes multi-round communication with the same peer satellites untenable due to their high mobility and limited contact windows (Brown & Green, 2021). A
1. Introduction
Department of CSSE, Auburn University, Auburn, AL, 36849, USA. Correspondence to: Tao Shu assuuauburr.cdu>..
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
similar challenge is observed in autonomous vehicles oper- ating in isolated regions with limited connectivity (Lee & Park, 2020). These vehicles may only intermittently connect to each other, making frequent communication impractical. Other real-world examples include mobile devices in ar- cas with poor network coverage (Taylor & Nguyen. 2018) and Internet of Things (IoT) devices with sporadic internet access (Kim & Chen, 2019). Therefore, how develop one-shot PFL (or OPFL in short) methods that can deliver personalized models with single-round communication (ide- ally, when two clients meet), so as to ensure superior and robust performance across non-IlD clients' data remains a critical challenge for practical PFL implementation.
For example, it is not uncommon for client to keep collect- ing new data during its operation, and periodically use its updated dataset to improve its personalized local model (see satellite-based example in (Walden et al., 2020; Maskey & Cho, 2020)). The reliance of personalized models on the global model in this scenario requires periodic one-shot FL to update the global model, and hence the one-shot benefit essentially vanishes with time. Even worse, for those ap- plications of an ad-hoc nature, e.g., an autonomous vehicle network, there does not exist an central parameter server, so none of the existing OFL.-based methods is even feasible. Clearly, an OPFL method that does not rely on global model is highly desirable.
To address the challenges of communication constraints in federated learning, one-shot Federated Learning (OFL) has emerged as promising paradigm, enabling the training of a global model with a single communication round between clients and the central server (Konečný et al., 2016; McMa- han et al., 2017; Guha et al., 2019; Li et al., 2020a; Dai et al., 2024). Existing OFL methods primarily focus on two key strategies: ensemble refinement and synthesized data en- hancement. Ensemble refinement focuses on improving the aggregation of local models into a better generalized global ensemble across clients by leveraging techniques such as parameter alignment, neuron matching, and weighted model combination (Su et al., 2023b; Liu et al., 2023b; Tang et al., 2024; Dai et al., 2024). On the other hand, synthesized data enhancement strategies aim to create synthesized data on the server side to facilitate global model training without ac- cessing raw client data. This can involve dataset distillation, generative-basee models, or fata-free synthesis approaches (Zhou et al., 2020; Zhang et al., 2022a; Kang et al., 2023; Heinbaugh et al., 2023; Yang et al., 2024a;b).
This paper proposes FOL. (Federated Oriented Learning), novel distributed OPFL method that allows client to continuously improve its local model by learning from each of its neighbors through one-shot communication of their local models (e.g., in a LEO satellite network, this occurs when two satellites fly to each other's proximity, at which point communication cost via a direct sater-satellite link is lowest). While neighboring clients could have very diverse local datasets and hence very different local models, the tuning and pruning components in FOL allows a client to extract the most relevant portion of a neighbor's 's knowledge that is best aligned with the client's 's local model (and hence the word "oriented" in the name of the method), and then the ensemble component in FOL allows the client to inte- grate that portion into its local model, gaining strengthened representability for the specific local task of the client. In each round the client performs such knowledge extraction and integration for the top-K neighbors that have the best knowledge alignment with the client, and hence strengthens the diversity of the client's 's local model. Benefiting from the shared knowledge between neighbors, the client's local model performs better than an isolatedly trained one over data unseen in its local training dataset. Two theoretical bounds on empirical risk discrepancy and convergence of the proposed FOL method are established. Extensive ex- periments were conducted on datasets Wildfire, Hurricane, CIFAR- 10, CIFAR-100, and SVHN to demonstrate the ef- fectiveness of FOL. Our experiment results verify that FOL consistently outperform counterparts, achieving accuracy improvements of up to 39.24% on Wildfire dataset.
At first glance, one might tend to assume that it is triv- ial/straightforward to obtain an OPFL model by directly fine-tuning the global model produced by OFL on each client's local dataset. However, already shown in the large body of PFL literature, this belief does not hold in practice, especially for clients of highly diverse datasets. This is because the global model produced by an FL algo- rithm typically lacks the adaptive modules necessary for effective local adaptation. Without additional adaptation steps that explicitly designed for personalization, a direct fine-tuning approach tends to preserve the model's initial global biases instead of aligning it with each client's spe- cific data distribution and needs. As result, local model fine-tuned from the global model over local dataset could perform arbitrarily poorly (Zhang et al., 2022b; Collins et al., 2022; Song et al., 2024).
2. Problem Statement
Given an image classification task, denote a set of n clients as K. = 21, ,n). Each client k E KC maintains a
private dataset D* = {(x,, ((.,,))))[*) y where I; is the input feature, y1 is the corresponding label, and |D* is the dataset size. The goal of OPFL. is to learn personalized local model for each client by collecting a set of neighboring models through shot communication process (each
Additionally, some practical considerations in the aforemen- tioned worldworld applications may also limit the applica- bility of building personalized models over global model.
2

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
pair of neighboring clients can exchange model parameters only once) and adapting these collected models its own data distribution, For each model collection round e {I..... E let of- and k e(0)+ be the parameters of client k's local model at the beginning of the round and those of its updated local model after the learning, respectively, where of:)- = 0%- 0(--1)-, i.e., it is carried over from the resulting model of the previous round. During round e, client k collects models from its neighboring clients to form a model collection set (6 - where Q is predefined number of other peers' models that each client can totally hold. Client k then updates its own local model by integrating the collected models {or (e)
from its peer clients. Each client can maintain most Q peer models locally at any time. Once client k has accu- mulated a total of Q peer models, we denote this set as the model collection set (d) 1
(0)))1 Each received neighboring
model of is then fine-tuned, pruned, and post fine-tuned on client k's local training dataset Di. resulting in an adapted model (5 Afterward, these adapted neighbor models {0 10
j- and the client's current local model, de- noted by 0%.)= are evaluated on a local validation set Dé. The top- performing models are then selected based on their validation accuracy, forming the set {s{() (e) y.K These selected models are then combined into an ensemble by optimizing a weight vector Wk that reflects their relative contributions. Finally, knowledge distillation is employed to distill the ensemble's s collective knowledge into an up- dated personalized model e( This updated local model of+)+ is then carried over as client k's initial model for the next round, i.e., o(e+1)- = o The procedure of FOL is given in Algorithm 1.
1=11 with its current model ofc- g(e-1)+ using its private dataset DK_ The updated personalized model o/+)+ is obtained by solving:
1
of arg min
Σ (EL.U)EDR
oQ ), Mi),
0
[D4]
(1)
where (11;; 0) is kth client's prediction function that out- puts the logits of 2, given parameter 0. f is the cross entropy loss function. The updated model o(()= is then used for subsequent rounds tasks.
3.2. Base Models Alignment
In contrast to OPFL, which explicitly addresses client-level personalization through local adaptation and/or refinement of received models, traditional OFL aims to learn a single global parameter As that serves all clients uniformly. This approach can be formalized as:
The collected models from neighboring clients may exhibit significant discrepancies with client k's local data distribu- tion. Such misalignments can lead to suboptimal ensemble performance and ineffective knowledge distillation. There- fore, it is crucial to ensure that the received neighboring models are well-aligned with the target client's local data distribution to maximize the benefits of subsequent ensem- ble and distillation steps. To address the aforementioned challenges, we propose four-stage process.
0s arg min és
Σ
((s((z;0s), vi), (2)
[Ds
(ED.))))DD
where Ds represents public/sytthesized data on the server side, and I's denotes the global model architecture.
Fine- TimenTuning. Upon receiving { 19 client fine-tunes each of these models on its local training dataset D re- spectively to better align them with its local data distribution.
3. Methodology
3.1. Architecture Overview
This fine-tuning process can be formalized as:
The proposed FOL. framework is multi-stage solution meticulously designed to solve the OPFL optimization prob- lem articulated in Eq. (1). FOL systematically integrates a client's local model with a set of neighboring models through a sequence of coordinated stages, which ensures the derivation of an optimal personalized model ofe)+ tailored to each client's 's unique data distribution. Specifically, FOL encompasses the following key stages:
(e) $ J-rh arg min
Σ
DE Praain
(4)
where á is initialized by d (6))
A Alignment-Aware Structured Pruning. While fine-tuning helps to adapt neighbor's 's model to local data, structured pruning can further ensures the model removes irrelevant or redundant filters/neurons for client k's tasks (He & Xiao, 2023: Dery et al., 2024). In heterogeneous settings, neigh- boring models may have distinct architectures and special- ized feature extractors that, if preserved appropriately, could benefit client & in its future ensemble and distillation steps. However, traditional structured pruning methods (An et al., 2024; Sun & Shi, 2024) typically remove filters based solely on their local importance, without considering their align- ment with the target model. This can lead to two key issues:
Initially, each client & E K = 21, n) trains an local model of on its local training dataset D This training process can be formalized as: I Σ {(fa(zu; 0.).)) v.). (3)
0(1)= + arg min #8
After the initial preparation stage, each client starts to share its current local model with its peers. In each model collec-
tion round e E (1...., F), every client & collects models
3

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
Algorithm 1 Federated Oriented Learning (FOL)
inition of shmed is given in Appendix B.I). unsbarred (i+k) \ Lsturedd be the layers unique to the neighbor model I all
1: Input: Set of clients K, number of model collection rounds F, model collection size Q, top-K selection size K, and local datasets Dim and D
that have no direct counterpart in of))_ For each shared layer 1 E I shared, let w and W denote the weight matrices of o Jー and o ((e)= respectively. We introduce a gating vector 0y = [0,,1,... O1,m,] for each layer 1, where my is the number of filters (or neurons) in layer l. Each gat- ing parameter on, E [0. 1] controls the retention or pruning of the i-th filter or neuron. For unshared layers u E Lunsbared- similarly define gating vector au to prune filters in that layer, but there is no alignment term because these layers do not correspond to any component in the local model. Hence, (e)) the pruned model Φ josk is obtained as the minimizer of the following objective function:
2: for = to F do
KC do 4: // Collect up to Q neighhor models Q) 6: Initialize an empty list S. to store scores
3: for each client
5: 1 j-1 + CollecNNeighbors(k.
7: for j = 1 to Q do
8:
Fine-Tune neighbor model on client I:
+ FineTTune( (e) Di
9:
k
10:
Alignment-Baeed Structured Pruning
+- Alignment j-tk o(e)= Dr mmiin)
11:
12:
Post-Pruning Fine-Turee Tuning
min (oc cochaatccteaattaa (or cot
Σ
(Ug(a of (ar): (a.B..s)
P (PrSWEEP
13:
PostFineTunet jj j- D (rain)
14:
Evaluate on validation set
Thss Loss Local Uata.
score, (c) (0)=)))) <- Evaluate 0b (c) j- D
15:
k
Append (scorek (c) (0 (o) jook) to (c)
Σ
Wiy Σ
2
16:
17:
end for
(1. (.
18:
// Evaluate local model o/²)
Aligiment Regahroononn (Shardd Layers
score (c) (((( f- Evaluate (8(0)) ((0)= Di
19:
Σ
a loowwbe w
Valorral
Append (score (c) (o(-) o to s(a)
20:
16 (c
21:
// Select top-k K candidates by validation score,
Grouged sco Shanal Layax
with cosine similarity as tie-breaker
& w
22:
{s]
Tim1 (Sp K,
Turalaaed
2.1
UC (entues(hhs)
CB = cosine similarity(0)
Group-Lasoo for Unstarred Layers
23:
I/ Optimize ensemble weights
(5)
24:
Wi k f OptimizeWeights// J D
(e)
Ji=1
where Ayp and y are hyperparameters controlling the strength of the alignment regularization and the structured pruning, respectively. ||-1|2.1 represents the group-lasso norm( Vogt & Roth, 2010). C denotes element-wise multiplication.
25:
// Distill knowledge into updated local model
gle)+ +- Distill(0).)) (0))), w(e), D )
26:
27:
end for
28:
end for Output: 10( JeEK
Post-Pruning Fine- TunniT After the structured pruning, Φب may experience slight drop in
29:
the pruned model i(e) performance due to the removal of certain filters or neurons.
To recover any potential accuracy loss, we perform post fine-tuning:
(1) over-pruning, where excessive removal of filters causes the received model to lose diverse feature extractors that could have been beneficial, and (2) excessive mismatch, where the retained filters extract features that are too differ- ent from those of the local model, making ensemble process ineffective due to highly inconsistent predictions.
arg min
Σ
(1)(0;)))3)) (6)
D Prain
(#..):)EDE
where ø is initialized by Ōz
To address this, we incorporate an alignment regularization (c) term that guides the pruning process to retain filters in o oz
Evaluation of Validation Accuracy. For each pruned and post fine-tuned model Φfe), and the current local model o())-
that are coherent with o(+)- thereby ensuring structured adaptation rather than an arbitrary reduction of parameters.
we compute the validation accuracy on the local validation set Di: a The validation accuracy is defined as:
Let: L (-+t) be the set of all layers in the neighbor model (() I.shared C all (j+k) be the subset of layers that struc- turally match (i.e., have the same dimensionality and func- tion type) with the local model 0(()= (The detailed def-
score &. (e) = D
Σ
((arg max ( 0) = y:),
where 1(-) is the indicator function.
In scenarios where multiple models have the same validation

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
score, we employ cosine similarity to determine the best- aligned models relative to the local model e(-)_ The cosine similarity between two models 0 and 02 is computed as:
A prevalent approach to obtaining a global model is to di- rectly average the parameters of individual models using FedAvg (McMahan et al., 2017). However, this method can be ineffective in scenarios with non-IID data distributions or significant heterogeneity in client model architectures (Li et al., 2024). Previous studies, such as (Zhang et al., 2022a), have attempted to address client models heterogeneity by employing simple or weighted averaging strategies, such as or weights proportional assigning uniform weights w n However, as highlighted to client data sizes w Ekm '* by (Wang et al., 2023), these averaging techniques are often ineffective in highly non-III settings. To enhance ensem- ble performance, recent methods (Gong et al., 2021: Diao et al., 2023) typically require modifying each client's local training phase or transmitting additional information (e.g., intermediate gradients, model updates, or feature represen- tations). While these approaches can be effective in many settings, they are often impractical in OFL settings due in- creased communication overhead. In this study, we propose constructing a more effective weighted ensemble to address both client data and model heterogeneity, ensuring robust performance without incurring additional communication costs or altering local training processes.
(01,02) cosine similarity(@1, (01,02) | 10-10-10211
(8)
where (01.02) denotes the dot product of the parameter vectors of models 0 and 02. ||-|| denotes the Euclidean norm.
To ensure that only the most effective models contribute to the ensemble, client & then selects the top-K performing models based on these scores: K ) = Top (ofe) 3 U U {8,
(o) U (scoref. (e) (o(+ )), K),
where CB specifies that in the event of tied scores, mod- els are further ranked using their cosine similarity to the local model g(0)= and pseudocode of TopK(.) is given in Algorithm 2.
Algorithm/ 2 Top-K K Model Selection
Require: (of 1Q
j= k 0())-
D K
Ensure:
(0,²) ===1
I: Initialize an empty list M
2: for each model (f)) in (of())
(e) (e) j= do
(e)
Optimal Weighted Ensemble. To better accommodate heterogeneous data distributions and models, we introduce a weighted ensemble approach. Let {s denote the set of selected base models for client l in round e. Our ensemble model A (c) is formulated as:
3: Compute score (0 (o) using Eq. equation 7
(e) 4: Append jo) score (e)
(o ((x))) to M
5: end for
6: Compute score (e) (o(o))) using Eq. equation 7
7: Append (o(*) score, (e) & (ol") to M
A ooo (1;
18/
(
Σ
K
(10)
8: Sort M in descending order by validation score
d=1
9: // Determine cutoff score for Tie-break
where is input sample (e.g., an image), and w (c) ( {wf Ji=1 is a weight vector representing each model's relative contribution. Since the ensemble operation is ap- plied at the logit layer, this reweighting mechanism can be seamlessly integrated into settings with heterogeneous or homogeneous model architectures.
10: Let = score of the K-th entry in M
11: // Split into Above and Tie group
12: My + {u E M| score (c) (v) > 5*}
13: M= (u E M score & (v)= }
14: // Start with all Above models
15: T T+M. TMs
To determine the optimal weights, client k refines the weight vector w by solving the following optimization problem using its local training set Dk i
16: // Tie-break only the cutoff group
(e)
17: for each model v E MM= do
18: Compute cosine similarity between each model in
the group and g(m) using Eq. equation 8
arg min
Σ
E(Awy (mn (0) (610 H.)).. v.). (11)
19: end for
20: Sort the Tie group in descending order based on cosine
3.4. Regularization- Based Knowledge Distillation
similarity
The ensemble outcome (a.k.a. the teacher model) is K- times larger than 0(e)- in its size and therefore cannot be directly carried over to the next round. Instead, ideally, we need a smaller student model that captures the knowledge of the teacher mother and meanwhile is comparable to of.) in its structure and size.
21: Append the first (K - (T]) entries of M= to T
22: (o/)) K + T
(e) K
23: Output: {8) Fi=1
3.3. Enhancing Ensemble Model Quality
The ensemble model functions as "teacher" by aggregat- ing knowledge from all selected base models {s] ())))) ]
To address this, we propose a regularization- -based knowl- edge distillation (KD) mechanism that (i) matches the local
5

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
model's logits to those of the weighted ensemble (teacher) and (ii) constrains the student's parameters remain close to the pre-distillation model o(=) thereby preserving client- specific features and limiting the norm of weight updates.
((V. is L-Lipschitz continuous is widely adopted in ML literature facilitate theoretical analyses and derive perfor- mance bounds (Mao et al., 2023; Safaryan et al., 2023).
B. Convergence of Distillation. Without loss of generality, we analyze a single model-collection round e and omit the superscripts + and for 0%² to streamline notation. We denote 0% as the model parameters at local training epoch r E {0, ,,,,,,,,1] Let LKD. (0), as defined in Eq. (12).rep- resent the local distillation objective for client k, which measures the discrepancy between the student's output and the teacher's output. During each local epoch r, the client updates its parameters 0% by:
This distillation gives birth to o())+ by minimizing the fol-
lowing KL-bbsedd distillation loss:
KL soottmu (25)
(x
Σ
(12)
Delans 0 A Al0
soltmax
1.
T
where (T.(-l)) denotes the Kullback-Leible divergence be- tween the teacher's output probability vector (softmax out- puts) and the student's probability vector. The temperature parameter T > 0 controls the smoothness of the softmax distributions applied to the logits. À 2 0 weights the penalty on deviations of (/-)= from o(0)-
0% = 0% ry WVLKD,,O% (D,
(16)
where 1 > is the learning rate, and encapsulates any randomness (c.g., from mini-bater sampling).
To make the convergence analysis mathematically tractable, we follow standard practice (Regatti et al., 2022; Deng et al., 2024) and impose the following assumptions:
3.5. Theoretical Analysis
A Assumption 1.
The functions (.D, for k E [K] are -.-smooth. For all 0, 0' E Re (where p denotes the total number of model parameters),
A. Bound on the Empirical Risk Discrepancy Between Student and Teacher Models. Without loss of generality, we analyze single model collection round and omit the superscripts + and for o( to streamline notation. We Assumption 2. The function & is j-strongly convex; i.e., there exists μ >> 0 such that for all 0, H' E RP:
[|VL - VLKD,.(0')L) Lg 10 0'1l
(17)
denote of-) as the final updltedd local model for client is at round e. The teacher ensemble's 's empirical risk on D defined as:
Irain is
(xD,)00)= 2 ( + (O)) 0) + 0 olp. (18)
Assumption 3. The expectation of the squared norm of the stochastic gradients is bounded; that is, for all 0 E RP,
(13) After solving the above distillation objective Eq. (12), the resulting student model o( has an empirical risk defined as:
R( ((A., (()) = (ana)))))
[AAA to (mu (st
1 so]
V x (0, E) & [[)xk,,(0)]{²t'',
(19)
where o² >> 0 is a constant bounding the additional variance arising from stochastic gradient estimates.
R = E( [(f((Iz;)))) v)).
(14)
The following theorem provides convergence guarantee for the distillation process under the above assumptions.
The following theorem provides a quantitative bound on how closely the student model's empirical risk can align with that of the teacher ensemble.
Theorem 2. Suppose {0%)-2o are generated by 0
0% - 17 V LLD. Mf 5h,, under the above assumptions.
Then for all r 2 and < n < 1, the following bound holds:
L,"
Theorem 1. Suppose the loss function ((j. y) (e.g., cross- entropy loss) is L-Lipschitz continuous with respect to the logit vector ÿ. Given a C-class classification task, the soft- max ouiputs lie within the interval (a, - -a) for some 0 < < 1 across C' classes, and temperature parameter T >>0 0, let the student model ( be trained to minimize the knowledge distillation loss (kD( as defined in Eq. 12. Then, the empirical risk discrepancy between the student and teacher models is bounded as follows:
11110% 01111 S 7" 100g - 01112 +
Σx (20)
P))
L
7=0
where 7 (1- 2np
B = 12 0² and
P
0% is the stationary point of the distillation objective LKD We defer the proof of Theorems to Appendix B.3.
4. Experiments
Rs( Rs(() - R(A R(AA)0)) S
L.CT a[l-a]
(Axatol
+3) (Aango()+))
(15)
While Theorems 1 and 2 theoretically demonstrate that our proposed FOL framework enhances accuracy for each par- ticipant in the system, it is essential to quantify these accu- racy gains empirically. To this end, we conduct extensive experiments to evaluate the effectiveness of FOL.
The proof of Theorem is provided in Appendix B.2. Note that the assumption that the cross-entropy loss function
6

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
Table Test accuracies (%) ) on Wildfire and Hurricane (e) 0.7), reported as mean std.
Dataset
Wildfire
Hurricane
Satellite #
13
28
48
35
32
44
Methods
V
0.7
Local
94.23 : 1.84
94.12 + 1,80
90.53
1.57
86.93 + 1.56
87.34 + 1.60
89.82 2 1.82
FOL-A (E=1)
97.19 1.53
97.16 1.24
95.97 + 1.55
95.34 1.42
96.18 1.02
97.61 + 1.68
FOL-A (E=2)
97.50 1.12
97.52 1.17
97.33 1.23
96.59 1.76
96.97 1.41
97.87 1.22
FOL-A (F=3)
97.53
97.70 0.98
0.99
96.90
9
98.20 1.03
FOL (E=1)
94.94 1.38
95.21 + 1.32
91.26 1.62
90.09
0.69
91.62 0.58
FOL (F=2)
91.60 0.72
91.23
9
95.21 1.49
FOL (E=3)
96.32
95.75
91.95 1.31
92.26
92.41 1.68
95.81 = 1.88
FOL-AN (E=1)
94.38
9
91.28 1.82
1.82
91.14 1.13
92.81 = 1.10
FOL-AN (E=2)
1.40
93.29 1.43
92.47 9..09
94.01 1.70
FOL-AN (E=3)
0.71
95.97 1.43 0.65
93.19
9.19
96.41 # 1.26
FOL-N (E=1)
93,44
94.68 1.79
2.331
1.85
0.93
91.62 1.19
FOL-N (E=2)
94.69
90.60 0.88
90.21 1.31
92.22 1.65
FOL-N (E=3)
95.31
0.92
0.97
90.51 0.57
94.61 = 0.73
DENSE
88.75
1.63
83.22 1.57
6..49
1.70
73.05 1.62
Co-Boosting
90.31 1.26
1.13
88.02 1.25
1.51
7.745
74.04 1.54
FedAvg (E=1)
73.19 1.73
73.94 1.96
68.18 2.002
60.21 1.73
02.03$1199
66.26 = 1.62
FedAvg (E=2)
73.13 + 1.91
72.29 + 1.74
66.92 1.55
59.44
64.33 1.33
69.88 = 1.57
FedAvg (E=3)
74.61
71.58 + 1.16
68.48 1.23
63.70
65.16 1.14
67.82 + 0.92
Table Test accuracies (%) on Wildfire Hurricane (+) E (0.5, 0.3, 0.1)). reported mean std.
Dataset
Wildfire
Hurricane
Satellite #
32
43
48
8
26
44
Methods
vb = 0.5
</ = 0.3
V/ = 0.1
= 0.5
y = 0.3
0.1
Local
79.07 + 1.71
90.37 + 1.76
85.50 + 2.16
86.77 + 1.90
57.14 + 2.87
77.78 # 1.35
FOL-A (F=1)
95.35 + 1.42
94.07 # 1.89
90.63 + 1.92
95.04 1.70
90.48 1.57
88.89 1.92
FOL-A (E=2)
96.52 1.02
94.92 + 1.25
96.14 1.16
95.34 1.16
91.72 1.26
91.67 1.26
FOL-A (E=3)
97.67
95.76 0.85
96.88 +
95.87 + 1.03
93.65
1.14
94.44 0.87
FOL (E=1)
90.70 1.75
90.68 + 1.01
88.46 1.99
89.26
84.13 1.57
83.33 1.69
FOL (E=2)
91.96 1.09
91.53 1.78
1.77
90.08 + 1.74
85.71 1.38
84.43 # 1.92
FOL (F=3)
03.02
92.37
93.75
90.91
$7.30
86.11 # 1.18
FOL-AN (E=1)
90.77 1.38
91.53
2.32
9..34
2.55
86.73 1.94
FOL-AN (E=2)
93.22 1.85
93.22
90.63
92.56
8
88.67 = 1.75
FOL-AN (E=3)
94.07 #
93.39 1.37
90.48 1.55
91.39 + 1.26
FOL-N (E=1)
86.05
88.14
85.13
85.95
76.19 1.73
80.56 2.11
FOL-N (E=2)
1.41
$
86.38
86.74
80.95 1.94
81.94 1.38
FOL-N (E=3)
1.55
1.50
1.37
87.60
8..54
83.37 # 1.56
DENSE
1.79
78.63 1.96
2.003
58.73 6..10
46.14 1.81
Co-Boosting
86.05
85.59
1..51
7
52.38 1.85
48.78 1.50
FedAvg (E=1)
13.11+1.82
6.825
22.76
66.12
41.27
46.14 1.72
FedAvg (F=2)
56.03 2.53
67.52 1.92
45.16
58.79
45.16 1.26
42.61 # 1.86
FedAvg (E=3)
51.07
66.10
42.86
60.33
44.44 1.76
43.33 + 1.46
Datasets. We evaluate FOL's performance using four diverse datasets: Wildfire (Aaba, 2023), Hurricane (Park, 2021), CIFAR-I0 (Krizhevsky, 2009), CIFAR-100 (Krizhevsky, 2009), and SVHN (Netzer et al., 10. Specif- ically, the Wildfire dataset consists of 42,850 satellite images with resolution of 350x350x3, designed for binary clas- sification task distinguishing wildfire events from non-fire regions. Similarly, the Hurricane dataset includes 14,000 satellite images with a resolution of 128x 128x3, and it also
focuses on a binary classification task that identifies hur- ricane weather patterns. CIFAR-10 and CIFAR-100 each provide 60,000 natural images (32 x x 3) spanning 10 and 100 classes, respectively. Lastly, SVHN (Street View House Numbers) offers 73,257 training and 26,032 testing images (32 32 x 3) for 10-class digit recognition, sourced from real-world house number imagery. Please refer to Appendix C.3 for the summary of these datasets.
Data partition. order to simulate the diverse local
7

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
datasets of LEO satellite constellations, we utilize a Dirich- let distribution-baseld approach to create non-III data par- titions across 70 clients (i.e., satellites). Specifically, we sample the proportion PL. of data from each class i allo- cated to client k using a Dirichlet distribution Pk ~ Dir( where v controls the degree of data heterogeneity among clients. To enhance realism, we apply data augmentation techniques (e.g.. rotation, flipping, cropping, resizing, nor- malization, and the addition of small Gaussian noise) for each image during the partitioning process. Using these augmentations, we mimic the variability introduced by dif- ferent satellite perspectives. This ensures each client has unique data capturing the nuanced differences of real-world satellite imagery. Recognizing that newly launched LEO satellites, such as the Disaster Monitoring Constellation and OroraTech, typically need to collect several hundred to thousand images before conducting model training and classification tasks (Walden et al., 2020; Maskey & Cho, 2020), we modify the partitioning function to guarantee that each client has a minimum dataset size of 100 images. Fol- lowing the partitioning process, each client splits its local dataset into training. validation, and testing subsets in pro- portions of 70%, 15%, and 15%, respectively. This splitting is performed uniformly to maintain the same label distri- bution across all subsets, ensuring a rigorous evaluation of the personalization effectiveness of the FOL framework in handling non-IID data scenarios.
and FOL-N omit the fine-tuning, pruning, and post-pruning fine-tuning stages but otherwise follow the same procedures as FOL-A and OFL, respectively.
Model Configurations. We consider a system with 70 satel- lites participating in E = 3 model collection rounds. Each satellite can store up to Q = 29 neighbor models at any point. The top-K parameter is set to K 10. meaning that after the first round (where 30 models are stored), each satellite retains the 10 highest-perfoneet and then gathers 20 additional models in the subsequent round, maintaining a total of 30. We adopt different hy- perparameter configurations for different datasets. For the Wildfire and Hurricane datasets, we use Stochastic Gradient Descent (SGD) with a momentum of 0.9, a weight decay of 0.001, a learning rate of 0.001, a batch size of 32, a patience of 20, and local training for 200 epochs. For CIFAR-10. CIFAR- 100, and SVHN, we use SGD with a momentum of 0.9, a weight decay of 0.001, a learning rate of 0.01, a batch size of 128, a patience of 20, and local training for 300 epochs. All baselines follow the same hyperparame- ter settings as ours. Results are reported as the average of runs with different random seeds. All models are built in PyTorch and trained/tested on two GeForce RTX 4090 GPUs. Details on the network architectures are provided in Appendix C.5.
models locally
Evaluation on Satellite Datasets. Using traditional OFL evaluation metrics, which focus on average accuracy across all clients, is insufficient for assessing personalized OFL. This is because the primary goal of personalized OFL is to optimize performance for individual clients rather than to achieve a uniform global model. In personalized OFL, each client's data may be significantly different (i.e., het- erogeneous), meaning average accuracy fails to capture the individualized improvements gained through personaliza- tion. Instead, evaluating a random subset of clients provides a more meaningful and realistic assessment, as it highlights how effectively method adapts to diverse data distributions and addresses the distinct needs of each client. Ideally, we would conduct experiments for all 70 clients across each dataset setting. However, due to the constraints of our lab's 's computing resources, we are limited to a subset of experi- ments for each configuration. To be realistic, for so = 0.7 on the Wildfire and Hurricane datasets, we select three satel- lites: one is chosen randomly, and the other two are selected based varying dataset sizes (ranging from a few hundred to a couple thousand images; see Appendix C.6). For other datasets and & values, we randomly select one satellite to evaluate FOL's effectiveness, with the expectation that these results are representative of the general performance across the entire client population.
Counterparts. Given that each satellite has only a single opportunity to exchange models with the same neighbor, we compare the performance of FOL against three of the most relevant existing methods (baselines): FedAvg (McMahan et al., 2017), DENSE (Zhang et al., 2022a), and Co-Boosting (Dai et al., 2024). For FedAvg, we employ its standard configuration, where each satellite directly averages the pa- rameters of all models received from its neighbors. The aggregated model then evaluated on the respective satel- lite's 's local test dataset Diss- Both DENSE and Co-Boosting are OFLebased methods. To ensure a convincing compari- son, DENSE and Co- are evaluated using their final server models under their best performing configurations. Specifically, when the number of clients is set to 5, both the original reports and our experimental results (see Ap- pendix C.4) indicate that these methods achieve their best performance. Consequently, we partition the dataset among 5 clients for these two method and train their final server models accordingly. Their performance is then evaluated on each satellite's 's respective local test dataset Di lest
In addition, we investigate four FOL variants to highlight the impact of individual components. FOL represents the standard proposed framework, which combines fine-tuning, pruning, post-pruning fine-tuning, top-K model selection, ensemble, and final knowledge distillation into a personal- ized model. FOL-A denotes the ensemble model. FOL-AN
We begin by examining the Wildfire and Hurricane datasets
under both moderate (अ) = 0.7) and high (2) E {0.5,
8

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
0.3, 0.1) levels of data heterogeneity. Table 1 presents the performance at db 0.7, where the ensemble model FOL- A consistently achieves the highest accuracy during each model collection round. In addition, the final personal- ized model in FOL demonstrates substantial improvements over the best baseline, surpassing it by up to 6.56% on the Wildfire dataset (satellite #28) and by 21. 21.77% on the Hurri- cane dataset (satellite #44). The larger improvement on the Hurricane dataset is likely due to its higher classification complexity, driven by intricate cloud formations, wind pat- terns, and other atmospheric phenomena, which challenge synthetic image generation methods like DENSE and Co- Boosting. These results highlight the ability of our FOL framework to enhance each satellite's s model performance, which is especially beneficial for challenging classification tasks. Table 2 further explores the impact of increasing data skew by reducing s). While most methods exhibit perfor- mance drops under more pronounced non-IID conditions, FOL-A and FOL remain robust. Notably, FOL exceeds the best baseline by up to 39.24% on the Wildfire dataset (satel- lite #48) and 37.33% on the Wildfire dataset (satellite #44) when & 0.1. These findings validate the effectiveness of our proposed FOL framework in non-IID satellite environ- ments, consistently achieving significant improvements in both ensemble and personalized models.
and data distribution. Nevertheless, FOL-A consistently achieves the highest accuracy, and FOL surpasses the best baseline by up to 3.72%, 5.97%, and 9.27% on CIFAR- 10 (client #9), CIFAR- 100 (client #14), and SVHN (client #21), respectively. These improvements highlight our frame- work's ' capability to adapt effectively across diverse do- mains, suggesting its potential applicability in a wide range of one-shot federated learning scenarios.
Effects of the Proposed Components. We further assess the contributions of our proposed model refinement process, including fine-tuning, structured pruning with alignment regularization, and post-pruning fine-tuning. Tables 1-3 demonstrate that FOL-A consistently outperforms FOL-AN in ensemble accuracy, while FOL. consistently surpasses FOL-N in final personalized model accuracy. These substan- tial gains underscore the effectiveness of the proposed model refinement process in adapting received models to each client's s unique data distribution, thereby significantly en- hancing both model accuracy and robustness. Moreover, the performance of each FOL variant progressively improves as the number of model collection rounds E increases (from to 3). In each round, the updated local model o( from the previous round serves as the initial local model for the next, enabling a gradual refinement process. As result, the ensemble accuracy (FOL-A) steadily increases, while the final personalized model accuracy (FOL. also benefits from more effective knowledge distillation. This iterative en- hancement ensures that each subsequent round begins with a stronger local model, leading to continuous performance gains across rounds. Overall, these results confirm that the integration of fine-tuning, structured pruning with align- ment regularization, and post-pruning fine-tuning, along with ensemble- knowledge distillation, is fundamental to the effectiveness of our FOL framework, particularly in handling non-lID satellite environments.
Table Test accuracies (%) on CIFAR-1O, CIFAR-100, and SVHN, reported mean t std.
Dataset
CIFAR-10
CIFAR-100
SVHN
Satellite#
21
Methods
V 0.7
0.7
à = 0.5
Local
60.06 + 1.97
30.41 = 2.28
78.97 # 1.75
FOL-A (F=1)
70.73 + 2.09
46.32 2.12
85.73 1.63
FOL-A (E=2)
70.93 1.16
47.72
86.26 1.28
FOL-A (E=3)
71.02
49.24
88.37 0.92
FOL (E=1)
2.11
2.43
81.09 1.54
5. Conclusion
FOL (B=2)
66.06 1.05
37.43
81.62 1.36
FOL (E=3)
39.42 0.82
82.85 1.18
FOL-AN (E=1)
2.37
79.62 2133
We introduced FOL, a distributed OPFL framework de- signed to handle communication constraints, high mobility, and diverse tasks in real-world environments. By integrat- ing fine-tuning, pruning, post fine-tuning, ensemble refine- ment, and knowledge distillation, FOL enables clients to extract relevant knowledge from neighbors while aligning with their local models. We provide theoretical guarantees on the bounded risk discrepancy between the student and teacher models during knowledge distillation, as well as the convergence of the distillation process. Experiments on real- world datasets (Wildfire and Hurricane) and benchmarks (CIFAR-1O, 10, CIFAR-100, and SVHN) show that FOL con- sistently outperforms state-of-the- one-shot FL methods, including FedAvg, DENSE, and Co-Boosting. Future work includes developing advanced personalization techniques and integrating stronger privacy guarantees into OPFL.
FOL-AN (E=2)
1.75
80.92#1.41
FOL-AN (E=3)
0.70
32.05
83.15
FOL-N (E=1)
2.44
2.57
80.04
FOL-N (E=2)
60.44
30.99
80.39
FOL-N (E=3)
1.43
31.11
1.22
DENSE
2.00
69.53 29.59
Co-Boosting
1..11
2.122
73.58 1.48
FedAvg (F=1)
2.63
3.916
5..0
FedAvg (F=2)
44.90
12.75
3.23
58.72 1.79
FedAvg (B=3)
45.57 + 1.79
12.40 + 2.89
55.49 1.93
Evaluation CIFAR-10, CIFAR-100, and SVHN. To illustrate the general applicability of our approach beyond satellite imaging classification tasks, we evaluate FOL on the CIFAR-10, 100, and SVHN benchmarks us- ing eb = 0.7 (see Table 3). These datasets differ substan- tially from Wildfire and Hurricane in both task complexity
9

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
Acknowledgments
Dai, R., Zhang, Y., Li, A., Liu, T., Yang, X., and Han, B. Enhaneing one-shot federated learning through data and ensemble co-boosting. arXiv, 1ax2402.150700 2024, https: /ttp//aaivvv//..////i///////..aa
This work is supported in part by the United States National Science Foundation (NSF) under grants CNS-2308761 and CNS-2006998. Any opinions, findings, conclusions, or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the NSF.
Danilenka, A., Bogacka, K., and Wasielewska- Michniewska. K. One-shot federated learning with self-adversarial data. In 2023 18th Conference on Computer Science and Intelligence Systems (FedCSIS), pp. 919-924. IEEE, 2023.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must specifically highlighted here.
Deng, D., Wu, X., Zhang, T. Tang, X., Du, H., Kang, J., Liu, J., and Niyato, D. Fedasa: A personalized federated learning with adaptive model aggregation for heteroge- neous mobile edge computing. IEEE Transactions on Mobile Computing. 2024.
References
Aaba, A. Wildfire prediction dataset. https://www www.
Deng, Y., Kamani, M. M., and Mahdavi, M. Adaptive personalized federated learning. arXiv, abtp20003.1346 2020. https: / / arxiv. org/ abs 22003 13461.
kaggle wi hwtpc///aatttttttaaaaeeeeeeeeaaa Accessed: 2025-01-20.
2023.
Dery, L.. Kolawole, S., Kagy, J.-F. Smith, V., Neubig, G., and Talwalkar, A. Everybody prune now: Struetured pruning of ulms with only forward passes. arXiv preprint arXhv:2402.0544 2024.
Amazon. Project kuiper overview. attps://wwi
amazon. cww/ homm/uupprr/,.. 2022.
An, Y., Zhao, X., Yu, T., Tang, M. and Wang, J. Fluctuation- based adaptive structured pruning for large language mod- els. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38. pp. 10865-10873, 2024.
Devlin, J., Chang, M., Lee, K., and Toutanova, K. Pre- training of deep bidirectional transformers for language understanding in: Proceedings of the 2019 conference of the north american chapter of the association for computa- tional linguistics: Human language technologies. volume (long and short papers). Minneapolis, MN: Association for Computational Linguistics, pp. 4171 2019.
Bendersky, E. The softmax function and its derivative. https: /ttpss//eliss thegreenpheeeenne ace. net/201 6/ che-softmax- www.:fttax--unnciiiooo www.ooonnnnnnnnnninneeeeeeeeeeee 2016. Accessed: January 9, 2025.
Diao, Y., Li, Q., and He, B. Towards addressing label skews in one-shot federated learning. In The Eleventh International Conference on Learning Representations, 2023.
Birdal, T. How to prove the 2-norm of an in- vertible matrix is exactly the reciprocal of its minimum singular value? scicomp wtw.kmppsstcckeccaggeg.... com/ 10
https://
Doe, J. and Smith, A. Deployment strategies for leo satellite constellations. Journal of Satellite Communications, 12: 34 2019.
ann-innertibble--mttii--------
-tee retherreciproccl-of--- 2019.
Accessed:
January 9, 2025.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 Transformers for image recognition at scale. arXiv preprint arXiv:2010. 2990: 1929, 2020.
Brown, and Green, M. Connectivity challenges in leo satellite networks. IEEE Transactions on Aerospace and Electronic Systems, 57(3):1234-1222 2021.
words:
Collins, L., Hassani, H., Mokhtari, A., and Shakkottai, S. Exploiting shared representations for personalized feder- ated learning. In International conference on machine learning, pp. 2089-2099. PMLR, 2021.
Elmahallawy, M. and Luo, T. One-shot federated learning for leo constellations that reduces convergence time from days to 90 minutes. In 2023 24th IEEE International Conference on Mobile Data Management (MDM), pp. 45 54. IEEE, 2023.
Collins, L., Hassani, H., Mokhtari, A., and Shakkottai, S. Fedavg with fine tuning: Local updates lead to representa- tion learning. Advances in Neural Information Processing Systems, 35:72 2022.
Gong, X., Sharma, A., Karanam, S., Wu, Z., Chen, T., Doer- mann, D.,, and Innanje, A. Ensemble attention distillation for privacy-preserving federated learning. In Proceedings
10

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
of the IEEE/CVF International Conference on Computer Vision, pp. 15076-15086, 2021.
Li, Q.. He, B., and Song, D. Practical one-shot federated learning for cross-silo setting. arXiv, abs/2010.01017, 2020a. https: /ttp:///aaivvveee org/ abs/20s0..000 01017.
Guha, N., Talwalkar, A., and Smith, V. One-shot feder- ated learning. arXiv, abs/1902.11175, 2019. https: //arxiv. .ww. aba wat.//992..111 11175.
Li, T., Sahu, A. K., Talwalkar, A., and Smith, V. Feder- ated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3):50-60 2020b.
He, Y. and Xiao, L. Structured pruning for deep convolu- tional neural networks: A survey. IEEE transactions on paltern analysis and machine intelligence, 2023.
Li, T., Hu, S., Beirami, A., and Smith, V. Ditto: Fair and robust federated learning through personalization. In International conference on machine learning, pp. 6357- 6368. PMLR, 2021.
Heinbaugh, C. E.,, Luz-Ricca, E., and Shao, H. Data-free one-shot federated learning under very high statistical heterogeneity. In The Eleventh International Conference on Learning Representations, 2023.
Li, Z., Sun, Y., Shao, J., Mao, Y., Wang, J. H., and Zhang, J. Feature matching data synthesis for non-iid federated learning. IEEE Transactions on Mobile Computing, 2024.
Huang, G. and Shu, A global-local probsparse self- attention transformer for leo satellite orbit prediction. In 2024 International Conference on Machine Learn- ing and Applications (ICMLA), pp. 91-98, 2024. doi: D0.109/CCMLAA1116222240000000
Liang, P. P., Liu, T., Liu, Z., Allen, N. B., Auerbach, R. P., Brent, D., Salakhutdinov, R., and Morency, L.-P. Think locally, act globally: Federated learning with local and global representations. arXiv, abs/2001.01522 2020. https: http//axivvvv.o/. wwtp//ggggg...oo/./////////////t 01523.
Huang, W., Ye, M., Shi, Z., and Du, B. Generalizable heterogeneous federated cross- cross-correlation and instance similarity learning. IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 2023.
Liu, J., Wu, J., Chen, J., Hu, M., Zhou, Y., and Wu, D. Feddwa: Personalized federated learning with dynamic weight adjustment. arXiv preprint urXiv:2305. 061244 2023a.
Huang, W., Ye, M., Shi, Z., Wan, G., H., Du, B., and Yang, Q. Federated learning for generalization, robust- ness, fairness: A survey and benchmark. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2024.
Liu, X., Liu, L., Ye, F., Shen, Y., Li, X., Jiang, L., and Li, J. Fedlpa: Personalized one-shot federated learning with layer-wise posterior aggregation. arXiv, abs/2310.00339, 2023b. https: / arxii. www.vv.org/bbbbaoa 2210. 00339.
Kang, M., Chikontwe, P., Kim, S., Jin, K. H., Adeli, E., Pohl, K. M., and Park, S. H. One-shot federated learning on medical data using knowledge distillation with image synthesis and client model adaptation. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 521-531. Springer, 2023.
Luo, J., Mendieta. M.. Chen, C., and Wu, S. Pgfed: Person alize each client's 's global objective for federated learning. Proceedings of the IEEE/CVF International Confer- ence on Computer Vision pp. 3946-390 -6566, 2023.
Mao, A., Mohri, M., and Zhong, Y. Cross-entropy loss func- tions: Theoretical analysis and applications. In Interna- tional conference on Machine learning, pp. 23803-23828. PMLR, 2023.
Kim, H. and Chen, Y. Internet of things: Addressing spo- radic connectivity. IEEE Internet of Things Journal, 6(2): 345-356, 2019.
Konečný, J., McMahan, H. B., Ramage, D., and Richtárik, P. Federated optimization: Distributed machine learning for on-device intelligence. arXiv, abs/ 6610.2257.... 2016. https: //tp///axvvvv//va//////eee/a/.aa hbtgg//b///ee
Maskey, A. and Cho, M. Cubesatnet: Ultralight convolu- tional neural network designed for on-orbit binary image classification on a lu cubesat. Engineering Applications of Artificial Intelligence, 103952,, 2020.
02527.
Krizhevsky, A. Learning multiple layers of features from tiny images. Technical Report, University of Toronto, 2009. URL https: hwwp//wwwwwwccooo ca. toronto..... edu/ -kriz/ wwriz//earnigg-eeeee Pages 32-33.
McMahan, B., Moore, E.,, Ramage, D., Hampson, S., and y Arcas, B. A. Communication-effciieenn learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282, PMIR, 2017.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A. Y., et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
Lee, K. and Park, S. Connectivity solutions for autonomous vehicles in isolated regions. In Proceedings of the Inter- national Conference on Autonomous Systems, pp. 18889, 2020.

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
Oh, J., Kim, S., and Yun, S.-Y. Fedbabu: Towards enhanced representation for federated image classification. arXiv, abs/2106.06042, 2021. https: eetps://arxivy http//axxvvv.oo////aa wwtpxiiiorr///asee / 2106. 06042.
Tang, Z., Zhang, Y., Dong, P., Cheung, Y.-m,, Zhou, A. C., Han, B., and Chu, X. Fusefl: One-shot fed- erated learning through the lens of causality with pro- gressive model fusion. arXiv, abs/2410.20380,, 2024. https: http://wxxxvv....aaa// 241 410.011011101 20380.
Web. Oneweb satellite network. https: //www. oneweb. com/, 2021.
Taylor, R. and Nguyen, T. Mobile device connectivity in rural areas. Journal of Mobile Computing, 25(4):567-580, 2018
Park, Y. Satellite images of hurricane damage. https::/www. http:://wwww..aggle... coom/ooee /ooodd//yeaaaa////
satellite- images----------- www-ooof-h-rreeeo hurricane- dameed 2021. Accessed: 2025-01-20.
Vogt, J. E. and Roth, V. The group-lasso: 11, regularization versus 11, 2 regularization. In Joint Pattern Recognition Symposium, pp. 252-261. Springer, 2010.
Regatti, J. R., Lu, S., Gupta, A., and Shroff, N. Condi- tional moment alignment for improved generalization in federated learning. In Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurlPS 2022), 2022.
Walden, R., Dosunmu, T., Jefferies, S. M., and Pimentel-
Alarcón, D. Satellite characterization via restoration-free imaging: A novel machine learning paradigm for ssa. In Proceedings of the Advanced Maui Optical and Space Surveillance Technologies (AMOS) Conference, Maui, Hawaii, 2020. Advanced Maui Optical and Space Surveil- lance Technologies Conference, AMOS. URL https: / //mostecchhccccccco httpr//chhh.cceeeeeeeeeaaaaaaers
Safaryan, M., Peste, A., and Alistarh, D. Knowledge dis- tillation performs partial variance reduction. Advances in Neural Information Processing Systems, 36:75229- 75258, 2023.
Ma Wa Wa pdf.
Song, C., Saxena, D., Cao, J., and Zhao, Y. Fed- distill: Global model distillation for local model de- biasing in non-iid federated learning. arXiv preprint arXiv:2404.09210. 2024.
Wallach, N. R. The inverse eigenvalue prob- lem: An introduction the general theory and recent results. https: /tt///aatbbb...... www.wwwww..........o edu/ -nwal -nwallach/ wwwhh//iieeeeeeeeeeeeeeeeeeeaeee [1] pdf, 2005, Accessed: January 11, 2025.
SpaceX. Starlink satellite constellation. https: /www//wwwwww. starlink. com/, 2020.
Su, R., Pang. X., and Wang, H. A novel parameter de- coupling approach of personalised federated learning for image analysis. IET Computer Vision, 17(8):913-92 924, 2023a.
Wang. H., Li. Y., Xu, W., Li. R., Zhan, Y., and Zeng, Z. Dalkd: Domain-aware federated knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2-422,, 2023,
Su, S., Li, B., and Xue, X. One-shot federated learning without server-side training. Neural Networks, 164:203- 215, 2023b.
Wu, X., Liu, X., Niu, J., Wang, H., Tang, S., Zhu, G., and Su, H. Decoupling general and personalized knowledge in federated learning via additive and low-rank decom- position. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 7172-7181, 2024.
Sun, X. and Shi, H. Towards better structured pruning saliency by reorganizing convolution. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 2204-2214, 2024.
Yang, M., Su, S., Li, B., and Xue, X. Exploring one-shot semi-supervised federated learning with pre-trained dif- fusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16325-16333, 2024a.
T Dinh, C., Tran, N., and Nguyen, J. Personalized federated learning with moreau envelopes. Advances in neural information processing systems, 33:21394-21405, 2020.
Tan, A. Z., Yu, H., Cui, L., and Yang, Q. Towards per- sonalized federated learning. IEEE transactions on neu- ral networks and learning systems, 34(12)95577-603, 2022a.
Yang, M., Su, S., Li, B., and Xue, X. Feddeo: Description- enhanced one-shot federated learning with diffusion mod- els. In Proceedings of the 32nd ACM International Con- ference on Multimedia, pp.
2024b.
Tan, Y. Long, G., Liu, I.,, Zhou, T., Lu, Q., Jiang, J., and Zhang, C. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAA/ Confer- ence on Artificial Intelligence, volume 36, pp. -432-8440, 2022b.
Yang, Q., Liu, Y., Chen, T., and Tong, Y. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-99, 2019.
12

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
Yang, X., Huang, W., and Ye, M. Fedas: Bridging inconsis- tency in personalized federated learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1 11986-119999 11995, 2024c.
Yao, D. D., Zhou, X. Y., Zhang, H., Li, D., Qian, F., and Fu, P. Variance minimization in stochastic systems. Stochas- tic Modeling and Optimization: With Applications in Queues, Finance, and Supply Chains, pp. 311-332, 2003.
Ye, M., Fang. X., Du, B., Yuen, C., and Tao, D. Hetero- geneous federated learning: State-of-the-ard and research challenges. ACM Computing Surveys, 56(3):1-44, 2023.
Zeng, H., Xu, M., Zhou, T., Wu, X., Kang. J., Cai, Z., and Niyato, D. tne-shot-but--ot-derradd federated learning. In Proceedings of the 32nd ACM International Confer- on Multimedia, pp. 11070-11079 2024.
Zhang, J., Chen, C., Li, B., Lyu, L., Wu, S., Ding, S., Shen, C., and Wu, C. Dense: Data-free one-shot federated learning. Advances in Neural Information Processing Systems, 35:21414-211 21428. 2022a.
Zhang. L., Shen, L., Ding. L., Tao, D., and Duan, L.-Y. Fine- tuming global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10174-10183, 2022b.
Zhou, Y., Pu, G., Ma, X., Li, X., and Wu, D. Distilled one- shot federated learning. arXiv, ats/2009.07999) 2020. https: / / arxiv. org/ abs 2009 2009.0799.
13

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
A. Related Works
A.1. One-Shot Federated Learning
One-Shot Federated Learning (OFL) has emerged as a promising paradigm that aims to train a global model using only a single communication round between clients and central server (Konečný et al., 2016; McMahan al., 2017; Guha et al., 2019; Li et al., 2020a; Su et al., 2023b; Elmahallawy & Luo, 2023; Kang et al., 2023; Danilenka al., 2023: Dai et al., 2024; Zeng et al., 2024; Yang al., 2024b;a). Existing OFL. methods primarily employ two key strategies: ensemble refinement and synthesized data enhancement. (1) Ensemble refinement aims to enhance the aggregation of local models into more generalized global ensemble by utilizing techniques such as parameter alignment, neuron matching, and weighted model combination, including MA-Fcho-OFL, (Su et al., 2023b), FedL.P (Liu et al., 2023b), FuseFl. ( (Tang et al., 2024), and Co-Boosting (Dai et al., 2024). (2) Synthesized data enhancement strategies, on the other hand, aim to generate surrogate or distilled data on the server side to facilitate global model training without direct access to raw client data. Methods in this category rely on dataset distillation, generative modeling, or data-free synthesis, such as DOSFL (Zhou al., 2020), DENSE (Zhang et al., 2022a), FEDCVAE (Heinbaugh al., 2023), FedISCA (Kang et al., 2023), LEOShot (Elmahallawy & Luo, 2023), IntactOFL (Zeng et al., 2024), FedDISC (Yang et al., 2024a), and FedDEO (Yang al., 2024b).
While one-shot federated learning approaches significamtly reduce communication overhead and are well-suited for resource- constrained environments, their effectiveness diminishes in highly heterogeneous settings where local data distributions vary substantially. This is because these methods prioritize generating a single global model that aims to perform generally across all clients. Consequently, the global-first paradigm restricts their applicability in scenarios requiring personalized federated learning, as unified model cannot sufficiently address the specific needs of individual clients.
A.2. Personalized Federated Learning
Personalized Federated Learning (PFL) has recently gained traction to address the limitations of "onn-sizeefits-all" model in non-IID scenarios (Liang al., 2020; Deng et al., 2020; T Dinh et al., 2020: Li et al., 2021; Oh al., 2021; Collins et al., 2021; Tan et al., 2022b; Luo al., 2023; Liu al., 2023a; Su et al., 2023a; Wu al., 2024; Yang et al., 2024c). PFL aims to customize each local model to perform well on client-specific data while still exploiting the shared knowledge from collaboration. Broadly, PFL methods can be categorized into two main groups (1) Parameter decoupling approaches, which separate the model into shared backbone (feature extractor) and a personalized head (classifier). The shared backbone captures universal features learned across the entire FL process, while the personalized head adapts to the specific data distribution of each client, including LG-FEDAVG (Liang et al., 2020), FedBABU (Oh et al., 2021), Fedproto (Tan et al., 2022b), PCCFED (Su et al., 2023a), and FedDecomp (Wu al., 2024). (2) Optimization- based approaches, which leverage advanced techniques such as regularization methods, dynamic aggregation, and second-order optimization to constrain local models and adjust aggregation weights. This balance facilitates global collaboration while allowing for local personalization. Examples in this category include pFedMe (T Dinh et al., 2020), Ditto (Li et al., 2021), PGFed (Luo et al., 2023), Feddwa (Liu et al., 2023a), and FedAS (Yang et al., 2024c).
These PFL algorithms are particularly effective for clients with heterogeneous data distributions. To achieve strong performance in each client's local model, they rely on iterative updates across multiple communication rounds. However, this reliance poses significant challenges in time-sensitive or resource-constrained environments. For instance, in LEO satellite networks, where satellites have limited communication windows and high mobility, frequent model exchanges are impractical. Therefore, it is imperative to develop practical one-shot PFL method that can effectively deliver high-quality personalized models within single communication round.
14

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
B. Supplementary Material on Methodology
B.1. Definition Shared and Unshared Layers
Definition of Shared Layers. Given a classification task, where each client maintains a neural network, let: Ly be the set of all layers in the local model = Ly be the set of all layers in the received neighbor model of j Specifically, of=)= have layers 1
(e) Layer,, and Pj- have layers {Layer,, Layer,
L.,,
We let each layer Layerg. Lg and ,, -,.. Then each layer L, L, can be characterized by both layer type (e-g., convolution, fully connected) and dimensionality or shape (e.g., number of filters, kernel size, input-output feature map dimensions). Formally, we let type(Layer,,) and dim Layer, denote the layer type and dimensionality of ,, respectively. Likewise for the neighbor model, we have type(Layer,.') and dim(Layer,,v)
Then we define the set shared layers, I shared to index pairs (1,1') l') such that:
The layers share the same type:
type(Layer,.)) type(Layer,,r),
(21)
2. Their dimensionalittie match:
dim(Layerg,i) = dim(Layer,.:).
(22)
Thus, formally define:
Lsharea {(.1)) type(Layer,,)) = type(Layer,.r), dim(Layer,,) dim(Layer,,r)
(23)
where each (1, pair selected based on a first-available match to prevent multiple local layers from matching a single neighbor layer.
Definition of Unshared Layers. Any layer in of that does not have a matching counterpart in of0) is considered unshared. We define:
Lnsshared (j + 1=)= L, \rr e Lk (1,1) Labured
(k: (k:
(24)
These layers have no direct local equivalent, meaning they must be pruned or retained based purely on their local utility, rather than structural alignment.
Example: Identifying Shared & Unshared Layers: Consider two models:
Client k's local CNN (0(e)
Conv(3 x 3,32,, BN(32), Conv(3 3, 64), FC(256, 10)
2. Neighbor j's received CNN (6 )=)=)):
Conv(3 32), Conv(3 x 3, 34), BN(64), FC(256, 10)
Using our shared layer definition, obtain:
Lshared = (Conv(3 x 3, 3, 32)x. Conv(3 x 3, 32))), (Conv(3 x 3, 64),. Conv(3 x 3, 64);), (FC(256, 10),, FC(256, 10),))
(25)
Thus, these layers form:
1 unshured
(BN(32), BN(64),)
(26)
15

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
B.2. Proof of Theorem
Proof. We begin by clarifying how our regularization-busel objective (i.e., KL X-penalty) still leads the same KL bounds as the pure distillation loss. Specifically, recall the regularized objective:
A Ante (ma)
[u( 0/-2
Σ
KL KT
softmax
A 1110 - -0(0)-12
(27)
ID
T
T
where the second term is nonnegative. Hence, for any of
KL....] & (
(28)
D
DiED2
KL-ooly
In other words, minimizing the entire (KL + regularization) objective also forces the pure KL portion to be small. Consequently, all subsequent bounding arguments (Pinsker's 's inequality, logit mismatch, etc.) remain valid under the regularized objective, since they only require that this KL. KL. portion be small.
We now prove Theorem
three main steps. First, we bound the discrepancy in empirical losses between the teacher and the student by showing that bounding logit mismatch (teacher vs. student logits) controls the cross-entropy discrepancy, Second, we link the logit mismatch KL divergence via temperature scaling. Finally, combine these steps to derive the risk discrepancy bound.
denote the teacher ensemble's logits, and let (a;;; o(()) = Logits (xy; of( be the logits of the student model for the same input Tz. Since ((, ys) assumed L-Lipschitz w.r.t. 2, we have, for any &1, 2g E IRya and label ys.
Bounding the Empirical Loss Discrepancy via Logit Mismatch. Let (Tre) = Logits (Aw (1:;; {0| )(.))
[8(21, y:) E(22, v1)1 = L 1| 8211.
(29)
Hence, for each (Ii,8i) E D
[((2s(x;;0)) ((0 v6) ((8T(x)).31)) I L [1ās(2;;) 0(0) - Sr(x))]
(30)
Taking expectation,
[ R(Aw.)l S L. (E..DDDDD [12s (())) (x(x))1
(31)
Thus, bounding logit mismatch indeed bounds the cross-entropy discrepancy.
Linking the KL Divergence Logit Mismatch under Temperature Scaling, Recall that in the pure KL-based distillation (temperature T >>> 0),
27 (1:)) softmax Σ KI (Tssttmaxx
2s (2 o(:))
LKD (0(
(32)
DD train
T
T
RED Bai
and minimizing it makes the student's softened outputs ps close the teacher's softened outputs pT. (Because of the nonnegative regularization in equation 12, minimising the full objective enforces at least the same KL closeness.) We define
2 (r(2)) T
), ps(a;; 0(:)) softmax( 2s(I;;
pr(x.) softmax(
(33)
T
Hence, minimizing (p(r o())) drives the student distribution ps(a;; o()) to approximate the teacher distribution (r(x.). Next, we aim to relate (L(pr(r:)ls((;;; ( to the Euclidean distance between the logits ( (())) - (22) To achieve this, we proceed through the following steps.
First, we work on bounding the distribution (softmax) discrepancy via logit mismatch. Consider the softmax function with temperature scaling:
(z(2)= exp(ze)
&c c= c=1,... C.
where =. T
(34)
ΣE exp(2;)
16

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
Recall that the softmax outputs satisfy: a S Pe(2) a-a, for some 0 < a This is because if any class has probability of exactly 1, all other class probabilities would need to be exactly 0, which is impossible since all exponentials exp(z) 0.
In practice, especially with temperature scaling, the softmax outputs typically lie within the interval [10-3, .0.99, and thus conservative bound on a is in the range 110-3,10-3
Define the vector function p RC > RC by:
p(z) = so)tmax(2).
The Jacobian matrix Jp(2) of the softmax function is:
0p1
82 Opa
Rsc
Ops
Dps
(2)
0p
821
B2g
Bac
1 (diag((()) p(z)p(z)T)
(35)
Oz
:
T
0pc
8pc
U22
8sc
where each element that insides the matrix is given by (according to (Bendersky, 2016)):
Opc p(d())
if d, if d.
(36)
Ozd (-tpo(=)pa(s),
Note that, the Jacobian matrix (z) of the softmax function inherently rank-deficient, having rank of C due the constraint Ec=1 P=(=) = 1. This rank deficiency poses a challenge for applying the Inverse Function Theorem directly, which is essential for linking the distribution discrepancy to the logit mismatch. To circumvent this issue, we will introduce reduced softmax function.
Without loss of generality, we fix the last logit pc(2') (e.g., can fix the last logit 0.) to eliminate the redundancy introduced by the probability simplex constraint. Define the reduced softmax function as:
p(z') = (p1(2'),p2(4 ... PC- 1
(37)
where z' = 22,,,2,... ..-.)) and pa(=') ) for = 1,....CC C 1 are defined as:
......C..
exp(=')
Pe(2')= 1+ 1+Σ j=1 exp(=')
(38)
Hence, the Jacobian matrix J2(3') of the reduced softmax function is:
Op dz
0p: 0zg
Op:
Jzc-1
0pg 02
0pg
Op2
Jo(2) =
0p Oz
82g
(ding(ple'))p(ee((ee))) - -==2))((2)))
(39)
0pc-1 00
Opc-1 02g
Opc-1
8
Note that (z') is full rank due to two reasons. (1), The diagonal elements satisfy Joc (c(2)( > 0, so no row or
column of the Jacobian matrix is entirely zero. (2). The off- off-diagonal elements of (2') satisfy Jed = -P=((pp)p(p)) c d. They not induce linear dependencies between rows or columns because the probabilities pc(z) and (=()) are strictly bounded by a and remain the probability simplex.
Since the reduced softmax function is composed of exponentials and sums, which smooth and differentiable, and its Jacobian matrix is invertible. According to the Inverse Function Cheorem(Wallach) 2005), the inverse Jacobian Jy-1 (p) at (x;)) can be defined as:
(p) = =
(40)
Next, we need to bound the operator norm of the inverse Jacobian Jp (p). The operator norm of a matrix A, denoted [|A]2-=2, is defined as:
|| 1 sup |
(41)
||=||==1
17

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
For the inverse Jacobian ()- according (Birdal, 2019), the operator can be bounded in terms of the smallest singular value mmin ( (2:))) of (x:():
Mg>:(D')lee O (Jz(2"<x:)))
(42)
We then need to give the bound of min (ainnJjp (Jz(z')). Note that the full Jacobian and the reduced Jacobian coincide the (C-ie)-iimnsionnl tangent space of the simplex, their positive singular values are identical. This is because the full Jacobian simply carries one extra singular value 0 corresponding to the all-ones direction. Without loss generality, we simplify our notation in the following: Let p = (Pi,...,Pc) E IR be a probability vector satisfying:
Pj (a, 1 a) for all i,
Σ 1=1
Pj =
(43)
We then define the matrix M M diag(p) pp". Since E Pi = 1, the matrix M has rank at most C 1. Furthermore, we have:
M1 = diag(p) 1 - p 1) p
0.
(44)
Hence the vector 1 = 1,....... 1) spans the nullspace of M with the eigenvalue O. Our goal is to give the bound for the minimum value of the positive nonzero eigenvalues of M. Formally, by the definition of Rayleigh quotient the minimum eigenvalues of can be defined as:
x M α
min (M) = min
(45)
where is nonzero vector, and x 1 can ensure the eigenvalue ,f 0.
x MM does not change if we scale by nonzero constant. Hence, without loss of generality, Tx
Since the Rayleigh quotient
we impose E x = 1, i.e. | 22= 1. Under this normalization,
Ain (M)
min 2= a1=0
M
(46)
Next, we expand x Mæ. Let xc) RC, we have:
x Mx [diag(p) - pp 1xc.
(47)
Split this into two terms:
x diag(p) x
and
x: [ [[ppp" pp xx.
(48)
Term A
Term
Term A: x diag(p) æ. Since diag(p) is diagonal with entries diag(p); = we have
diag(p) (piz), p2d2 potc) T
(49)
Hence, we have:
C
x [diag(p)) (prr)) Σ
Pi x
(50)
1=1
Term B: x. T pp" T]
æ. Since pp is non-diagonal with entries (ppT)s )is = PiPj. we have:
(pp = = (EprRa)
PL.TK)
(51)
Rm
18

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
Hence, can get:
2
x (pp x Σ Pitts
(52)
Add the negative sign:
c
2
σ
pp [p 1-ppp'
=
Σ
pi 24
(53)
Put eveything together, we get:
C
T M x =
Σ
p
-
Ps Iz
(54)
i=1
i=1
To simplify this equation, let 2
Σi piiti and since Eip: Eq. (54) can be rewritten as:
C
C
Σ
Pi a - x² = Epil - 20,z + =2] -
p:
x2
Σ pi(a - 2)²
(55)
Now, the problem becomes to find the the global minimum of the following:
Σ E 2
Σ Li=1 x = 1,
pi (x; 2)²
subject to
(56)
Pt E (a, a);
E,P, E,P.L. =1
Next, to solve this optimization problem, we follow (Yao et al., 2003), let 1 C {1...., C} where x; + a for i T+ let I_ C (I....,C) where x; b for i I_, and let M |44 and N [|_), so that M + N C. Hence, the constrains be rewritten
Σ 2
Σ a+
Σ La + (- = La No =0 0 = La= No.
(57)
EEL+
iel-
L.a
N-(-b)
Similarly, we have:
2 = La + No = 1.
(58)
we define: P+ = Eier, Ps, p- = EiEL_ Pt and p+ +p- 1, where p+, P_ E (a, 1 1-a) a) because each ps E (a, 1 1-a). The weighted mean can be rewritten as:
2
Σ p.z. = (+a) + (--)(-b) a p-
(59)
The objective function can be rewritten as:
(x; - 2)2 = p+ (a =)²2 + p- (-b 2)²
(60)
We then define the Lagrangian:
C(a, b,; )1 =p+ (a (a- 2)² + (1 p4) (-) 2)² + X (I.a Nb) p(..² + N 1),
(61)
where 2 a (1-P+)b P+ + p b p+ (a+b)
19

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
We then can solve this optimal function by taking partial derivatives = N, a =, b, p+ = a, p- = a.
ac da
0,
ac 26
0,
ab
0. The results gives
apt
By substituting everything back, we can get 2 == a a) + -1-a)(-o0) 0) (1- a)) have
(1-
(2a 1) a. Hence, we
(2xx 1) 2(1 a)a, 0,
(2a 1) - -2a
Therefore, we have:
C
Σ 2
Ps (z. - 2)² = a 21² + (1- - a) I-a - 2/?
[2(1 a)a|² (1 aa
(62)
4o² a (1 a)² + (1 - a) a²
4a² a (1 - ox).
Recall that La +N N a² =Ca² = 1, thus a² = I/C, so Eq. (62) can be rewritten as:
c
Σ
pi (z: - =)== = c a
(63)
By using the result of Eq. (63), include the parameter T, we get:
4a a)
mmin (Jp(2')
(64)
CT
We define:
CT
GT 4a(1-a)
(65)
which serves upper bound for the operator norm of the inverse Jacobian.
Using the bounded inverse Jacobian, we then relate the differences in logits to the differences in the softmax outputs. Consider the student and teacher models with reduced softmax function for input æ,;
(a ) = p(: )(
pf((1)) = (x:)((x))).
(66)
Hence, we have:
=p <s( ( (rs((; -=::"))))===""
(x:():)).
(67)
Applying the Mean Value Theorem for vector-valued functions, there exists a point & on the line segment between pls (x,;; (())) and pf such that:
(x:1; - (x;) - (e) (ps(a, - (x:()
(68)
Taking the L2 norm both sides:
(a (()) (x:) 125 (91= o()) (xx)22)
(69)
Substituting the bound on the inverse Jacobian
2 (z ol - (1))lllsSaal (125( 1000--ppas (2;)lle.
(70)
We then extend the bound in Eq. (70) by relax the reduced softmax to the full softmax. First, we extend the LHS. Recall that 25 and are the full C-dimensional logit vectors for the student and teacher models, respectively. The reduced logit vectors z'g and 2/y exclude the last component (fixed to 0), i.e., z's = (25,1, 28,2, - and similarly for 2/.- Since the last logit is fixed (2s,c, 2T,C = 0), the Euclidean distance between the full logits is equal to the Euclidean distance between the reduced logits:
G-1
|| ||
2T 11 Σ [28 2T,,)²=
Σ cml
(2',e - zT.0)² + (2s,c 2T,0)² = 1| - +++++0
(71)
cml
20

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
To extend the RHS of Eq. (70), recall that ps softmax(zs). pr soflmax (2r), P's (Ps,..P........ Ps,C-1), P1 = (pr,1, Pr,cc-.,,, PS.C = Lo- C011 c-l Ps,e, and pr,C = -1- Pr.c- The norm of full softmax distributions
can be express as:
lpss - 1 Σ (ps.c - pr,,)² + (ps.c- -r.c))
(72)
For the RHS of Eq. (72), we have:
C-1
C-1
C-1
P PT,C
1 Σ
PS.c
1
Σ
PT.c
Σ
(r, PS,.)
(73)
Plugging this back into Eq. (72), we have:
C- (ps.c PT, +
C-1
2
llps pr pTl12
Σ
(PT.c PS.0)
(74)
c=
Notice that (E (PT,0- - (PT.e-Ps.c) 2 0. Hence, we have:
Ips PT pTl13>
Σ (ps.c-P7) - PT,c)² (ps,o-PT,,)2- = Ill's prr122
(75)
Therefore, the bound in Eq. (70) can be extended by:
a 21 1 12s (T) II''s PT11225 (T) llps Prl12-
(76)
To connect the KL divergence to the Euclidean distance between the softmax outputs, utilize Pinsker's Inequality along with norm relationships:
(T)pp((x)lllss z (lr((a)) ps p= o6 )|?
(77)
Additionally, in RC, the L and 2 norms satisfy:
( (a ol))) 125 (|r((x;) (()(=:;(
(78)
Derivation of [[p())) - p(z2) 12 (m((1) p(22))11:
We start with the L1 norm and L2 norm definitions: (lp(2)) p(22)111 = EiElle((1) - pr(22)). [lp(z)) - p(22)112 =
Ee ²²² Then, we square of the Linorm:
C [ii( pi(22)
C
[p(=)) - p(22)) = Σ (z1) n((2))]
(79)
1=1
J=1
Eq. (79) can be decomposed as:
C - 12(22)1++
- - -p: - E
(2)()))
+ - -2)))) (3)3)
(80)
imj
igj
This implies:
[[((z)) - [|p()))) -(22) -I - (((e)))))))))) pp(22))
(81)
Therefore, we proofed the LHS of Eq. (78):
llp(z)) - p(22)113 (1p(2)) (
(76)
21

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
Combining Eq. (77) and Eq. (78), obtain:
[pr(x.) - p2(2,; )125 v (L(pr(4;))pss (f))))
(82)
Establishing the Upper Bound on Logit Mismatch
By combining Eq. (76) and Eq. (82), we can get the logit mismatch bound:
|| (1:; o( 2T(x))l2s G. (T) V2 KTopr(as))lssss: (2)
(83)
To establish linear relationship, we employ the inequality v - 2 + 1/2
2 (L(pp((;; ps(2;; 1)
V2 (L(pr(z;) (1 (1s(a,;0)))
(84)
2
2
Substituting back:
2 (2.KT(pr(a)) (1
[2,(2;; 0(00
(85)
2
2
Hence, we have:
[&s(=;;(k)) -r((xi)ll Ga(T) (L(pr(=:) (1:;( )) + Ga(T)
(86)
2
Define:
K =G 50 Ga(T)
(87)
2
we obtain:
|| - k (L(PT(Z:) ps(2 0) + 80-
(88)
Deriving the final bound
By averaging the above inequality over all samples in the training dataset Di we get:
E ED.
||ás(z:; ( ST(x:))l2 k. LKD (0(:)) + on.
(89)
where:
CiD (0(e))
(Lipr(r:)lps(a;
(90)
DR train
Substituting back the regularized objective LKD e and by using Eq. (28) we have:
E [ 060) (rr)))l s K. LKD (0()))= + S x (KD + 80.
(91)
By substituting back the Ga and do, we have
E E(..,)DD [1()(1))(()) 21(2x)lla s
2a(1 a)) CkD(O( + 8a(1 a))
CT
CT
(92)
[s(((()) R(Aws)l S L. EEEEDDDDD [1(5((0)((0) &r(x))]] Let's then combine Eq. (93) and Eq. (31):
(93)
(s(e))) - R(Aw,)) <
LT 2a(1 a)
(k)) + L CT 80 - a)
(94)
Final Bound and Conclusion. Combining the above mismatch and KL relationships, we obtain
Rs (((()) R(Aww) <
CT
(kD(
+
1
(95)
a(1 a)
2
8
This completes the proof of Theorem
22

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
B.3. Proof of Theorem
Proof. In order to prove Theorem 2, we first need expand the SGD update rule:
Recall the update rule:
0,+1 = 0% 7) V &LDD. ( (D)
(96)
Since we want to bound 110+ 0% 11.... 1, let's expand the squared norm:
|1o 0- 110% 0% 1y VCKDDA(0% EDI
(97)
Using (u- - u)² 1luu 1) + ||a/1 we have:
|0 1 0,112= 110 11 2 ( t%. VLKDA((% (T) + 1/² [VLKD, (DI
(98)
We then take conditional expectation given 0f on both side of Eq. (98),
El||0 - 11 0 = 110 - -2 - 2n - 0% VLKDA (%%, EEM + n" LD,, EDI
(99)
Term A
Term B
By the definition of the stochastic gradient (the expectation of the stochastic gradient equals the true gradient), we have:
E[VLKD, EE) = ((.))
(100)
Hence, Term A = (0) 8% (LKD,&(Of))
The derivation for the Term
By the definition of the inner product for vectors, have:
(0% - 0% (OK
Σ 01 (L VLKD (D].,
(101)
1=1
where [1]. [1]. and VLKD (0% [t] are the i-th components of 0%. 0% and the stochastic gradient, respectively.
We then take the expectation on both side of Eq. (101) over En
1 0% LLKD, CDM = Σ [o (1) LLLD, [
(102)
Since the expectation operator E. is linear, rearrange the above equation as:
(00 0% LLKD (%, (D))
(M
0g(1) ElV (O,
(103)
Substituting Eq. (100) into the above equation, we have:
[o (1(1 LLVL MMM Σ Corl) (%(1)
LLDD, (OP))
(104)
Note that the summation Σ [f][)] fgl+) VLKD (0 the definition of the inner product between 0 0 and V LKDD (0T):
(0g Of VLKD = Σ
(o - 0g() V VLDDA (OD)).
(105)
111
Thus, Term A can be defined as:
E( 0% LLDD, (O),
(b1) (0) 0% L(KD,e(f))
(106)
23

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
For Term B, we the assumption 3:
E| L (DI < [V))
(107)
Therefore, we have:
1 0211 [g]
<
10% 001l
2n 10 02 VLKD..(0)) + o² (1D)
(108)
Next, we need to leverage the p-strong convexity property to establish certain key facts. Since L.DD&(.) it is j--ttrnn convexity, by the definition, we have:
p 110 011
(0) 2 CKD,(00) (VLKD,(O), 0' 0)
(109)
Eq. (109) can be rewritten as:
(VL,) - (Ok), 0% - 0z) 2 1% 0211
(110)
The derivation Eq. (110);
Let's define function C(0) is jestrrogly convex if for all 0, 0' e Rº:
L((') 2 [()) + (VL()) 0' 0) + 1110 01 012
(111)
Using the strong convexity inequality at both 0 and 0', we get two inequalities:
At
[(0') 2 (()) (VC(0), 03 + -ーッ 11000 0112
(112)
2. At At 6":
(()) 2 (V 0) 1 01112
(113)
Rearranging these two inequalities, we have:
(E(0") (())) 2 (VL(0), - -0) 0 013
(114)
2
(C(O)- 2(0')) 2 (VC()), 03) 0 0'13
(115)
2
Adding these two inequalities together, we have:
0 0 (VC(0), - + + 0¹) 1|10 0'112
(116)
Simplify:
(VC(0) VC(0),0- 0) 4|10 0112
(117)
Hence, we have:
(VLKD, V.KD, 0% 0 2 1 16 071
(109)
Since optimum Oz. VLKD
0, we have:
(VLKD.(()), 0% 0%) 2 110 1211..
(118)
Therefore, we have:
2n (0% - 0% LLKDD (05) S 2n 110 - 1
(119)
Since function LKD,E is j--troon convexity, it also satisfies the Polyak (PL) condition with parameter >> 0 if for all 0 E IRP Hence, we have:
2 VLKD ((CKD.(()) (()))
(120)
24

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
The derivation of Eq. (120):
Applying the definition of p-strong convexity with 0' = 0%. we have:
(KD, (k) + (VLKD.<00) - 11% 011².
(121)
Rearranging the above equation, we have:
LKD,a(0) LKD),(0f) S (VLKD,((0),0- -20 # 110 -011..
(122)
Using the Cauchy-Schree Schwarz inequality, we have:
(VLKD,(O),,0- 02) & [VC 0211-
(123)
Applying Young's s inequality ab - 2μ + nb² 2
with a =
| |VLKD (0)1 and b ||0 - 0g we have:
1 -0] =110 -
(O)I -
0313
(124)
2μ
Substituting back, we have:
(0) (k) S
[C]k+
0tll M 100 0211.
(125)
2p
2
Simplifying the above equation, we have:
1
(KD. (KDDN(O)) - (K)D)
(126)
2p
Multiplying both sides by 24,, have:
M(CKD, -(CKDA(O)=<KD.HHA (x)) s (V =
(127)
Therefore, we have:
1
[LKD, ((CkD,,(0) (kD))(0))).
(120)
Building on the results above, we proceed to bound V using the smooohhhness and p-strong convexity properties of (-).
From the PL condition, we have:
[VCkD,(0)] p (CKD,)(O) (xD,
(128)
Rearranging the above equation, we have:
[VLkD,< 2p (CKD.a - LKD. (02))
(119)
Since at optimum 07 VLKD, 0, from T.,-smoothness, specifically the gradient norm bound, we have:
[LKD, L, 0%
(129)
Squaring both sides, we have:
[C)D,, 1.2 12
(130)
25

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
From Eq. (120) and Eq. (130), have:
2/ (o,,)))) [VCkn.,<00) 2 0211
(131)
Hence, we have:
2μ (CKD,A(0) & - L 02ll?
(132)
Dividing both sides by 2/s, we have:
L2
(KD - (KD.
- 10 1:112
(133)
2
Hence, we have:
L2
L3
[VLKD,((0)] S 2 La (CKD.(0)) (CKDA(())LLD..)) LKD. S 2 L.
10
0/1
110
-0
(134)
2p
/a
L3
For simplicity, define C
so
[V)k),((0)) C C| 6f1l
(135)
Next, we use the above results to simplify the Eq. (108), we get:
E[ll0t+ 01 [%] I 110% 0/11
2n (0) 6%, LKD, (05) + +1² (IVLKD,
gradient alignment
(136)
110 02112 - - 110 0/12 1/ C1107 - 0i1l + 0²²0²
(1 (1-2 2n Cnn²) 110% - 2 + 0². 0².
The second inequality used the fact that - 2n 10% - 02 VL) < -2 110 - 02l1 (Eq. (119)) and the bound
[| C' - 01 from Eq. (135).
To simplify the above equation, we define:
= 21-2n Cr7), B 12
(137)
Finally, we expand Eq. (136) to derive its global (unconditional) expectation version. Specifically, we define the global (unconditional) expectation of the squared distance at iteration as:
gr == 1 - 12].
(138)
From Eq. (136), we have:
Fl F110r=1 0212 0%
<
110 - 0211 + B.
(139)
Since, we have: [|X ri] [[X], by taking expectation on both sides, we have:
|0 - 0]|[]]
s >E |00 0|||] + B,
(140)
Hence, we can get:
gr+1 Elllet+1
2 s -- 3.
(141)
We now write Eq. (141) for each r = 0, 1....
In particular:
13 < 0000 + B,
32 < 734 +
s 00 +B)++ 3 = -0 + 7 = 200 + (x+1)
(142)
g3 < _8t +
S (2- 8d + ( 1)8) + = "000 + (62 8 + = --0 + +
26

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
Observing the pattern, we find:
33 S 7 + (7-1++1 + + + 1)
(143)
Therefore, have
7-1
m 1 0 - 7" 100% 0:
Σ T=0
r" 3
(144)
This completes the proof of Theorem 2.
C. Additional Experiment Materials
C.1. Additional Experiments on the Hurricane Dataset at d = 0.7
To demonstrate the performance improvement achieved by satellites other than those three evaluated in the main text, we conduct six additional experiments on the Hurricane dataset additional satellites. The results confirm that both FOL-A and FOL maintain consistent performance gains across broader range of clients.
0.7. Table reports the test accuracies (%) for these six
Table 4. Test accuracies (%) for six additional clients the Hurricane dataset with 0.7.
Dataset
Hurricane
Satellite #
41
3
22
51
Methods
/
0.7
Local
90.45
82.35
88.63
90.67
86.01
91.18
FOL-A (E=1)
94.27
91.18
92.73
93.10
93.87
96.57
FOL-A (E=2)
95.54
94.12
93.64
93.68
95.16
97.06
FOL-A (E=3)
96.18
96.06
94.09
95.40
95.74
97.55
FOL (E=1)
93.11
85,29
90.02
91.95
89.81
93.63
FOL (E=2)
93.63
91.33
91.82
92.53
90.07
94.12
FOL (E=3)
94.27
93.04
92.27
94.25
91.92
95.59
DENSE
70.02
67.35
68.13
71.31
69.57
70.16
Co-Boosting
74.61
69.16
72.51
73.63
75.21
74.47
To demonstrate convergence of clients under our proposed FOL method, we provide the loss-vs-epoch curves for the six additional satellites during distillation in Figure 1. All six clients exhibit rapid initial decrease in training loss and steadily converge to a stable low-loss plateau, confirming the robustness of the FOL distillation process across diverse satellites.
lst Satallite
JbI Satelitt #9
[b] Satellite #:41
0.25
0.5
0.4
0.33
4.2
W
3.33
0.2
0.1
mmm
22
30
2
20
60
20
50
Epoch
Eporh
Epoch
[d] Satellite #22
(d1 Satelitte
Satellite #56
2.45
2.40
2.25
0.4
0.3
0.0
Spuch
Figure 1. Training loss convergence 83 epachs for six additional Hurricane clients a 0.7.
27

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
C.2. Additional Hyperparameter Settings
In our experiments on CIFAR-1O, CIFAR-IO0, SVHN, and the satellite datasets (Wildfire and Hurricane), we set A, 0.1, Yshared = 0.05, Tunshared 0.02 in Equation (5), and we set the distillation regularization weight in Equation (12) to 0.01. These values were selected via cross-validation and were found to consistently yield robust performance, effectively balancing alignment, diversity retention, and model personalization.
C.3. Summary of Datasets
Note that, by default, some datasets provide separate training, validation, and testing subsets. To streamline the presentation and ensure consistency, we merged the validation samples into the testing dataset in the table below.
Table 5. Summary of datascts used in the experiments.
Name
#Training Samples
#Testing Samples
#Classes
Image Size
Wildfire
30,250
12,600
2
350x350x3
Hurricane
10,000
4,000
2
128x128x3
CIFAR-I0
50,000
10,000
32x32x3
CIFAR-I00 100
50,000
10,000
100
32x32x3
SVHN
73,257
26,032
10
32x32*3
C.4. Performance of DENSE and Co-Boosting on the Wildfire and Hurricane Datasets
The original authors of DENSE and So-Boosting have extensively evaluated their methods on various image classification benchmark datasets, consistently demonstrating that these methods achieve optimal performance when n = 5. In this study, we adopt their methods. evaluation metries (testing the model on the entire testing dataset), and default configurations to further assess their performance. Specifically, we evaluate whether these methods retain their optimal performance on real-world satellite image datasets when = 5. To simulate non-IlD data distributions, the datasets were divided among 5 and 10 clients using Dirichlet distribution with parameter & 0.7 for the binary classification tasks Wildfire and Hurricane. For each configuration, results are reported as the average of 5 runs with different random seeds.
Table 6. Test Accuracy (%) of the Final Server Model on the Wildfire and Hurricane Datasets.
Dataset
Wildfire
Hurricane
Method
n 5, 1 0.7
10,
n 5,
0.7
10, 1
DENSE
89.87 + 1.32
83.38 1.57
69.35 + 1.69
63.95 + 1.55
Co- Co-Boosting
92.35 + 0.86
85.46
1.12
75.45 + 1.27
68.05 + 1.42
C.5. Neural Network Structures for Each Dataset
Table 7. Cifar-10 Network Structure (Model Architecture for Each Base Model).
Layer (type)
Input Shape
Output Shape
Param #
Conv2d
[1.3, 224, 224]
[1. 128, 222, 222]
3,584
MaxPool2d
[1, 128, 222, 222]
[1, 128, 111, 111]
0
Conv2d
[1, 128, 111, 111]
[1, 128, 109, 1091
147,584
MaxPool2d
128, 109, 109]
[1, 128, 54, 54]
0
Conv2d
[1. 128, 54, 54]
128, 52,
147,584
Linear
[169, 2048]
[169, 10]
20,490
Table 8. Network structure Hurricane and CIFAR-100 Datasets.
Dataset
Structure
Usage
CIFAR-100 ResNet18 (
Model architecture for each base model
Hurricane ResNet18 18 (Non-pretrained)
Model architecture for each base model
28

---

Federated Oriented Learning: A Practical One-Shot Personalized Federated Learning Framework
Table 9. Wildfire network structure (Model Architecture for Each Base Model).
Layer (type)
Input Shape
Output Shape
Param #
Conv2d
[1,3, 224, 224]
[1. 64, 224, 224]
1,792 128
BatchNorm2d
[1, 64, 224, 224]
[1, 64, 224, 224]
ReLU
[1, 64, 224, 224]
[1. 64, 224, 224]
MaxPool2d
[1, [4, 224, 224]
[1, 64, 112, 112]
Conv2d
61, 112, 112]
[1, 128, 112, 112]
73,856
BatchNorm2d
[1, 128, 112, 112]
[1, 128, 112, 112]
256
ReI.U
[1, 128, 112, 112]
128, 112, 112]
MaxPool2d
[1, 128, 112, 112]
[1, 128. 56, 56]
Conv2d
128, 56, 56]
[1. 256, 56, 56]
295,168
BatchNorm2d
[1,256, 56, 56]
[1. 256. 56]
512
ReLU
[1, 56, 561
[1, 56,
0
MaxPool2d
[1, 56,
[1, 256, 281
Conv2d
[1, 28, 28]
[1, 281
1,800,60
BatchNorm2d
[1,512, 28, 28]
[1, 28, 28]
1,024
ReLU
[1,512, 28, 28]
[1, 28, 28]
0
MaxPool2d
[1, 28, 28]
[1, 14, 14]
AdgpoiveAvgPool22
[1, 14, 14]
[1,522 1J
Flatten
[1.
[1,
Linear
[1,512]
[1.2]
1,026
C.6. Selected Satellites and Dataset Sizes
The table below summarizes the selected satellites and their corresponding dataset sizes, including the number of training, validation, and testing samples used for evaluation under 3i 0.7. For the Wildfire dataset, satellites #13, #28, and #48 were chosen, with training dataset sizes ranging from 693 to 2,627 samples. Similarly, for the Hurricane dataset, satellites #35, #32, and #44 were selected, with training dataset sizes ranging from 732 to 1,507 samples.
Table 10. Selected Satellites and Dataset Sizes for Evaluation.
0.7
Dataset Name
Satellite #
# #Training
#Validation
# #Testing
13
1,488
319
320
Wildfire
2,627
563
564
48
693
148
149
1,507
Hurricane
32
732
157
158
44
778
167
167
29


Reviews:
[{"review_id": "p6p4GKEogj", "reviewer": "Reviewer_rxR9", "summary": "This paper proposes Federated Oriented Learning, a novel framework for One-Shot Personalized Federated Learning designed for environments with constrained or infrequent communication or limited contact windows. The authors further provide two theoretical guarantees on empirical risk discrepancy between student and teacher models and the convergence of the distillation process.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Regarding Evaluation on 1–3 Clients per Setting.**\n\n**Response:** The accuracy of 6 additional clients are given in following Table. Similar performance trends as those 3 shown in our original paper can be observed in this table (i.e., those 3 shown in our original table indeed are representative). Together with these 6 additional clients, we have shown the performance of 9 clients. We hope this has addressed the concern of the reviewer.  \n\n\n**Table:** Test accuracies (%) for six additional clients on the *Hurricane* dataset with $\\psi$ = 0.7.\n\n| **Dataset**     |               |          | **Hurricane** |          |               |          |\n|-----------------|---------------|----------|---------------|----------|---------------|----------|\n| **Satellite #** | **41**        | **3**    | **9**         | **22**   | **56**        | **51**   |\n| **Methods**     |               |          | **$\\psi = 0.7$** |          |               |          |\n| Local           | 90.45%        | 82.35%   | 88.63%        | 90.67%   | 86.01%        | 91.18%   |\n| FOL-A (E=1)     | 94.27%        | 91.18%   | 92.73%        | 93.10%   | 93.87%        | 96.57%   |\n| FOL-A (E=2)     | 95.54%        | 94.12%   | 93.64%        | 93.68%   | 95.16%        | 97.06%   |\n| FOL-A (E=3)     | 96.18%        | 96.06%   | 94.09%        | 95.40%   | 95.74%        | 97.55%   |\n| FOL (E=1)       | 93.11%        | 85.29%   | 90.02%        | 91.95%   | 89.81%        | 93.63%   |\n| FOL (E=2)       | 93.63%        | 91.33%   | 91.82%        | 92.53%   | 90.07%        | 94.12%   |\n| FOL (E=3)       | 94.27%        | 93.04%   | 92.27%        | 94.25%   | 91.92%        | 95.59%   |\n| DENSE           | 70.02%        | 67.35%   | 68.13%        | 71.31%   | 69.57%        | 70.16%   |\n| Co-Boosting     | 74.61%        | 69.16%   | 72.51%        | 73.63%   | 75.21%        | 74.47%   |\n\n**2. Suggestion to Include a Convergence Trend Figure.**\n\n**Response:** Because the rebuttal policy of ICML does not allow us to upload figures as part of our rebuttal, we present a representative trace of the loss value in the training of the distilled FOL model in the following table. The convergence trend of the training can be clearly observed in the table. We will present the same trace as a figure in our final paper, as suggested by the reviewer.\n\n\n**Table:** FOL distillation training loss vs. epoch id for one of the satellites over the Hurricane dataset.\n\n| **Epoch**         | 1      | 2      | 3      | 4      | 5      | 6      | 7      | 8      | 9      |\n|-------------------|--------|--------|--------|--------|--------|--------|--------|--------|--------|\n| **Training Loss** | 0.135  | 0.1001 | 0.0818 | 0.1109 | 0.097  | 0.0968 | 0.0839 | 0.0859 | 0.0891 |\n\n\n**3. Concerning the Scalability of FOL.**\n\n**Response:** Our proposed FOL framework is designed to scale efficiently. The storage, computation, and communication cost do not change with the number of clients. These costs increase proportionally with the size of local model.", "claims_and_evidence": "Most claims are supported, with real-world and benchmark datasets and theoretical contribution", "methods_and_evaluation_criteria": "The multi-stage pipeline is methodologically valid and appropriate for OPFL. Non-IID data is considered with different levels of heterogeneity.", "theoretical_claims": "Theoretical contribution includes (1) risk discrepancy between student and teacher model with KL-divergence and (2) convergence of distillation with standard theoretical assumptions.", "experimental_designs_or_analyses": "The experimental settings are valid for OPFL with class imbalance and natural non-IID splits settings.\nAblation studies demonstrates the contribution of each component.\nConsiders heterogeneity and real-world dataset.\n\nI wonder if evaluation is on 1–3 clients per setting, which might seems insufficient for personalization-focused FL research where client-wise variance matters.", "relation_to_broader_scientific_literature": "This paper is closed connected to FL, PFL and OFL, with model distillation and pruning.\n\nIntegrating one-shot FL, pruning with alignment, and distillation for personalization seems novel.", "essential_references_not_discussed": "Some multi-round methods could be mentioned here.", "other_strengths_and_weaknesses": "Strength:\nPaper is well-written and experiments carefully designed with theoretical analysis.\n\nWeaknesses:\nA figure showing the trend of convergence is desired here.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "How does FOL scale with increased client count or larger models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xo4IMGV6dj", "reviewer": "Reviewer_DHoW", "summary": "This paper first introduces an important limitation of existing Personalized Federated Learning methods, which is the need of multiple communication rounds to update models. This will lead to massive communication costs and impracticable for the real-world scenarios. Moreover, the authors argue that personalizing the global model is not feasible in practice, as the global model produced by federated learning (FL) algorithms typically lacks the adaptive modules necessary for effective local adaptation. Based on this, the authors propose a novel algorithm for one-shot personalized federated learning (PFL), called Federated Oriented Learning (FOL), which operates in a decentralized manner and enables clients to iteratively enhance their local models by learning from their neighbors through a single round of local model communication. FOL consists of several key stages: model pretraining, model collection, fine-tuning, pruning, post fine-tuning, ensemble refinement, and knowledge distillation. Additionally, the authors establish two theoretical guarantees: one on the empirical risk discrepancy between the student and teacher models, and another on the convergence of the distillation process.\n\nThe paper demonstrates a well-motivated and innovative approach; however, certain critical steps are described in a vague and potentially misleading manner, which could hinder the clarity of the methodology. It is recommended that the authors enhance the readability of these sections to ensure a more precise and accessible presentation. Additionally, the complexity of the proposed method appears to be notably high, raising concerns about its practical applicability. A more detailed discussion on computational efficiency and scalability would greatly strengthen the paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Optimal Weighted Ensemble Clarification.** *Optimal Weighted Ensemble described in Section 3.3 appears to be misleading...; Additionally, the statement that \"the ensemble outcome (a.k.a. the teacher model) is K times larger...*\n\n**Response:** We respectfully clarify that the \"Optimal Weighted Ensemble\" in Section 3.3 is performed entirely at the logit level. In our approach, we aggregate the outputs (logits) of the K selected base models using a weighted sum (as shown in Equation (10)), so the ensemble’s final output retains the same dimensionality as that of a single model. More formally, let vector $\\mathbf{g}_i = [g_i^{(1)}, \\ldots, g_i^{(N)}]$\n\n denote the $N$-dimensional logit output of base model $i$, then the output of the ensemble model can be written as $\\mathbf{G} = \\sum_{i=0}^K w_i \\mathbf{g}_i$, where $K$ is the number of base models participating in the ensemble. \n\nFurthermore, in our original text, the phrase “K times larger” referred to the fact that the ensemble model is composed of K base models, and hence the number of parameters in the ensemble model is roughly $K$ times of the size of a base model.  It was never intended to imply that the final output dimension (i.e., the dimensionality of $\\mathbf{G}$) increases by a factor of $K$. \n\n**2. Role of Knowledge Distillation.** *Given that the Ensemble Model (FOL-A) consistently outperforms ... it raises the question ... should be removed. Alternatively, is there a more effective personalization strategy that could replace knowledge distillation?*\n\n**Response:** The knowledge distillation component is necessary in order to ensure the final personalized model has the same size as the initial local model. In particular, the ensemble step FOL-A bloats the model size by $K$ time, the distillation step compresses the bloated model by a factor of $K$, leading to a final model roughly at the same size of the initial model.\n\nFurthermore, based on our best knowledge, knowledge distillation achieves the best performance among all other personalization strategies in one‑shot federated learning scenarios.\n\n**3. Computational Complexity and Practical Applicability.** *Each client collects the local models of its neighbors and performs operations...; Moreover, the authors do not clarify whether each local model is trained on the local dataset only once or for multiple epochs...*\n\n**Response:** We want to point out that our proposed FOL method has much lower computation and communication overhead than DENSE and Co-Boosting (i.e., the baselines). Specifically, DENSE and Co-Boosting rely on a modified GAN structure in which the generator is composed of multiple large fully connected layers and runs for 30 epochs per distillation epoch over a total of 200 epochs (i.e, in total 6000 epochs for GAN alone). In FOL, the computation cost of various components are as follows:  fine tuning: 30 epochs, structured pruning: 30 epochs, post fine tuning: 30 epochs, ensemble refinement: 10 epochs, knowledge distillation: at most 500 epochs. In total: at most 600 epochs, which is much smaller than that of DENSE and Co-Boosting.  Moreover, these additional processes incur only local computational cost, with no extra communication overhead. Furthermore, we note that FOL is designed for one-shot communication scenarios, e.g., LEO satellite networks or intermittent IoT systems, for which limited communication bandwidth, instead of the computation power, is the primary constraint. \n\n\n**4. Experiments on Data Diversity and Baseline Selection.** *The authors claim that their method is effective for clients with highly diverse datasets...; The number of baseline methods appears to be somewhat limited.*\n\n**Response:** In our experimental settings, we use a Dirichlet distribution to partition datasets across 70 clients, thereby explicitly simulating highly non-IID conditions with varying degrees of data heterogeneity (with $\\psi$ values ranging from 0.1 to 0.7). \n\nRegarding baseline methods, we compare against widely recognized state‑of‑the‑art one‑shot federated learning approaches, namely DENSE and Co‑Boosting. We believe that our experimental design thoroughly demonstrates the effectiveness of our method.\n\n\n\n\n**5.Suggestion to Include the Convergence Analysis of Equation (5).**\n\n**Response:** We want to point out that structured pruning is just one of the several intermediate steps within FOL. Instead of proving the convergence of this intermediate step, we have proved the convergence of the final step of the proposed FOL method in Theorem 2 of the paper. We believe proving the convergence of the final step is more meaningful and important than just proving the convergence of an intermediate step. In our extensive experiments, we have observed that all intermediate steps consistently converge.\n\n**6. Minor Writing Issues Eq. (28).**\n\n**Response:** We appreciate the reviewer pointing out this typo and will ensure all equations are correctly presented in the final paper.", "claims_and_evidence": "1.\tThe authors provide a thorough and detailed analysis of existing research and the problem at hand in the introduction section, effectively highlighting the significance of the issue and presenting a well-justified motivation.", "methods_and_evaluation_criteria": "1.\tEach client collects the local models of its neighbors and performs operations such as fine-tuning and models alignment by structured pruning on its own dataset. Essentially, every step involves training all the neighbor’s local models on the local dataset, which is undoubtedly unsuitable for resource-constrained devices. Moreover, the authors do not clarify whether each local model is trained on the local dataset only once or for multiple epochs. The authors should reduce unnecessary computational burdens to enhance the practical applicability of the proposed method.\n\n2.\tOptimal Weighted Ensemble described in Section 3.3 appears to be misleading and requires further clarification. In Line 231, it seems to suggest assembling models to form a new model, but later descriptions indicate that this step is more akin to ensembling logits. Additionally, the statement that **\"the ensemble outcome (a.k.a. the teacher model) is K times larger\"** is unclear, as Equation (10) implies that the size of the outcome remains unchanged, with the corresponding elements being weighted sums.", "theoretical_claims": "1.\tThe paper would benefit significantly from a convergence analysis of Equation (5), as it would strengthen the theoretical foundation of the proposed method.", "experimental_designs_or_analyses": "1.\tThe authors claim that their method is effective for clients with highly diverse datasets; however, the experiments do not include any scenarios that validate this claim.\n\n2.\tGiven that the Ensemble Model  (FOL-A) consistently outperforms other approaches in the experimental results, including full FOL, it raises the question of whether the Knowledge Distillation component should be removed. Alternatively, is there a more effective personalization strategy that could replace knowledge distillation?\n\n3.\tThe number of baseline methods appears to be somewhat limited.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WoZu7RzdW3", "reviewer": "Reviewer_hUmY", "summary": "•\tIn order to address the situation of limited client communication in federated learning, this paper introduces a novel federated learning paradigm - OPFL and presents a four-stage one-shot PFL algorithm FOL (Federated Oriented Learning). FOL can learn a personalized model for each client without the need of central server to generate a global model. The convergence analysis is also discussed in the paper. Under the scenario of Low Earth Orbit (LEO), FOL can demonstrate good performance on the Wildfire and Hurricane satellite image datasets, as well as on CIFAR10 and CIFAR100.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Fairness Issue in the One-Shot Communication Setting.**\n\n**Response:** Please note that for personalized learning, fairness does not mean equal accuracy across individual users, but instead it means every user has comparable opportunity to improve its accuracy (i.e., opportunity of learning). Under this definition of fairness, the major issue in the learning example given by the reviewer, i.e., $(t_0: K1 ⟷  K2), (t_1: K2 ⟷  K3)$, where $t_1>t_0$ and $(t: a ⟷ b)$ denotes a learning at time $t$ between users $a$ and $b$ (because they meet at that moment), is that it only considers a short interval $[t_0, t_1]$ in the learning period of these users while has artificially neglected what could happen after that interval, e.g., $K_1$ may later meet and learn from someone who has learned from $K_3$ earlier. Specifically, consider the following example sequence of learning that extends the reviewer's example: $(t_0: K1 ⟷  K2), (t_1: K2 ⟷  K3), (t_2: K3⟷  K4), (t_3: K1⟷  K4)$. So, at moment $t_3$, $K1$ makes up its knowledge on $K3$ by learning from $K4$ who just learned from $K3$ at moment $t_2$. To make our above example more concrete, we have conducted experiment on the aforementioned sequence of learning, and list the accuracy of $K1$ at different moments in Table 1. From this table, it can be observed that the accuracy of $K1$ was improved at $t_0$ and $t_3$, indicating that $K1$ indeed obtained the opportunity of improving its accuracy by learning at these two moments, comparable to the learning opportunities possessed by $K3$. In general, by considering the more realistic scenario where users encounter with each other over a wider time horizon, the fairness issue raised by the reviewer will diminish due to the knowledge propagation among users.\n\n**Table: Accuracy of $K1$ at different moments over Hurricane and CIFAR-10.**\n\n| Methods                                     | Hurricane ($\\psi=0.3$) | CIFAR-10 ($\\psi=0.7$) |\n|---------------------------------------------|-------------------------|------------------------|\n| Initial (i.e., Local)                       | 82.14\\%                 | 60.47\\%                |\n| FOL-A ($t_0: K1 ⟷  K2$)      | 91.07\\%                 | 63.18\\%                |\n| FOL-A ($t_3: K1 ⟷  K4$)      | 92.86\\%                 | 67.15\\%                |\n| FOL ($t_0: K1 ⟷  K2$)        | 85.71\\%                 | 61.91\\%                |\n| FOL ($t_3: K1 ⟷  K4$)        | 89.07\\%                 | 63.13\\%                |\n\n**2. Gating Parameters and Pruning Mechanism.**\n\n**Response:** Compared with the common **hard** pruning method, wherein weights are either strictly set to zero or fully retained based on a fixed/unified threshold,  our **soft** pruning has the unique advantage of adaptive, fine-grained selection of threshold to prune for each individual connections (i.e., threshold $\\alpha_l$ for weight $W_l$). These gating parameters/thresholds are optimized during training, allowing the model to selectively prune less important connections, which ultimately results in a more robust model. This strategy is also commonly used in recent differentiable pruning methods.\n\n**3.Hyperparameter Settings in Equations (5) and (12).**\n\n**Response:** In our experiments, for CIFAR-10, CIFAR-100, and the satellite datasets (Wildfire and Hurricane), we set $\\lambda = 0.1$, $\\gamma_{\\text{shared}} = 0.05$, and $\\gamma_{\\text{unshared}} = 0.02$ in Equation (5), and we set the distillation regularization weight in Equation (12) to $0.01$. These values were selected via cross-validation and were found to consistently yield robust performance, effectively balancing alignment, diversity retention, and model personalization. We will include these information  in the final paper.\n\n**4. Minor Writing Issues.**\n\n**Response:** We acknowledge that Equation (28) was not fully expressed, that the same constant “L” is used for both the L-smoothness and L-Lipschitz conditions, and that the same notation $\\lambda$ appears in Equations (5) and (12). In the revised manuscript, we will update the notation to clearly distinguish between the different constants and ensure that all equations are fully expressed. Importantly, these issues do not affect the correctness of the proofs, as confirmed by our analysis and by the reviewer’s own inspection of Theorem 2.\n\n**5. Suggestion to Include SVHN.**\n\n**Response:** We want to point out that our validation in the paper is based on more comprehensive and domain-relevant datasets than those done for DENSE and Co-Boosting. Specifically, none of those datasets used to validate DENSE and Co-Boosting in their original papers are relevant to satellite applications, which are the typical applications our proposed FOL method targets for. In contrast, the wildfire and hurricane datasets used in our paper are more relevant to the satellite application of the proposed FOL method.", "claims_and_evidence": "In the reality of Low Earth Orbit, each client can often only communicate with neighboring clients, and adding a global server for all Low Earth Orbits can be very expensive. This paper attempts to address this situation.", "methods_and_evaluation_criteria": "For the communication situation faced by Low Earth Orbit (LEO), FOL can learn a good model for each orbit relatively well.", "theoretical_claims": "I mainly checked the correctness of Theorem 2, and the main part of the theoretical proof in the paper is correct, with some minor writing issues.\n(1) Equation (28) has not been fully expressed.\n(2) In equation (17), the parameter L is defined as the parameter of L-smoothness, but in the Proof of Theorem 1 of B.3, the parameter for the L-Lipschitz condition is also L.", "experimental_designs_or_analyses": "The authors mainly focus on the validation of satellite datasets. The SVHN dataset is used in both DENSE and Co-Boosting papers, but this paper does not use this dataset. I think this paper should validation the method on more datasets.", "relation_to_broader_scientific_literature": "This paper may be helpful for federated learning in satellites. The authors mainly conduct experiments on relevant satellite datasets. Prior to this, there have been some works on one-shot federated learning, but they main focus on learning a global model. This paper proposes to learn a personalized model for each client in one-shot federated learning.", "essential_references_not_discussed": "The authors' research on relevant papers is relatively comprehensive, and they give a relatively complete introduction to one-shot Federated Learning and Personalized Federated Learning, which are most relevant to this paper.", "other_strengths_and_weaknesses": "Weakness:\n(1) This paper should consider the fairness issue in federated learning. \nThis paper assumes a setting where each client can only communicate with its neighboring clients. Consider the following scenario: there are three groups of clients, K1, K2, and K3, which cannot communicate internally among themselves. Each client in K1 and K3 can only communicate once, while K1 can communicate with K2, K2 can communicate with K3, but K1 cannot communicate directly with K3.\nIf all clients in K1 first communicate with all clients in K2, and then all clients in K2 communicate with all clients in K3, when the number of clients in K1 and K3 is the same, each client in these groups will have the same communication cost. However, each client in K1 can only collect knowledge from clients in K2, while clients in K3 can collect knowledge from both K1 and K2 (since K2 processes and aggregates K1's knowledge). Therefore, at the same communication cost, K3 obtains more knowledge than K1. This disparity becomes even greater when there are more clients in K1. It is unfair to them.\n(2) In P4, 'Each gating parameter \\alpha_{l,i}\\in [0,1] controls the retention or pruning of the i-th filter or neuron.'Aaccording to this, \\alpha_{l,i}) is the gating parameter, which is a real number in [0,1], such as 0.5. This is equivalent to scaling each weight to a certain extent, and this is not a common pruning process. Common pruning methods either set the weights to 0 or retain the weights.", "other_comments_or_suggestions": "There are some writing errors in the paper, such as using the same hyperparameter λ in equations (5) and (12), but the hyperparameters are different. \nIn P5 ’The proof of Theorem 2 is provided in Appendix B.2.’. Perhaps it should be 'The proof of Theorem 1 is provided in Appendix B.3.'", "questions_for_authors": "In equation (5), there are hyperparameters \\lambda, \\gamma_shared, \\gamma_unshared. How to set the value for the hyperparameters in equation (5) and equation (12) is not mentioned in the paper. I did not see any experimental results on them.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NqZrWgxLTN", "reviewer": "Reviewer_bQPg", "summary": "The paper introduces Federated Oriented Learning (FOL), a novel one-shot personalized federated learning (OPFL) framework designed for communication-constrained environments such as LEO satellite networks. FOL integrates multi-stage processes—fine-tuning, structured pruning with alignment regularization, ensemble refinement, and knowledge distillation—to enable clients to adaptively integrate knowledge from neighboring models under stringent communication constraints. Theoretical guarantees on empirical risk discrepancy and convergence are provided. Extensive experiments on wildfire, hurricane, CIFAR-10, and CIFAR-100 datasets demonstrate FOL’s superiority over baselines like FedAvg, DENSE, and Co-Boosting, achieving accuracy improvements.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Concern About the Local Validation Set Has the Same Distribution as the Final Test Set.**\n\n**Response:** We would like to clarify that, in personalized learning, it is standard practice to assume that the local validation set and the test set follow the same distribution. In real-world scenarios, such as LEO satellite constellations, each client’s data is collected by the client itself, and the training, validation, and test sets are just uniformly sampled from these data. This naturally implies that the distribution of these validation and test sets are aligned.\n\n\n**2. Definition of \"Neighbor Models\" and Communication Cost Comparison.**\n\n**Response:** Regarding the definition of neighbor models, we clarify that in our framework a neighbor is any client reachable via a one-hop connection. For example, in a Starlink LEO satellite network, each satellite continuously broadcasts beacons, and any satellite that receives a beacon is considered a neighbor. \n\nRegarding the communication costs, we respectfully clarify that in our one-shot setting, each client exchanges models only once with its one-hop neighbors, incurring significantly lower communication overhead than the iterative rounds required by FL setup with a central server.\n\n**3. Personalized Model Performance and Ensemble Storage Use.**\n*Furthermore, the performance of naive local training  is actually quite similar to......, If the 30 times the storage space used....*\n\n**Response:** The reviewer seems to have mis-read the performance comparison between the naive local model and our personalized model shown in Tables 1-3 in the paper. We respectfully clarify that our final personalized model consistently outperforms naive local training. As shown in these tables, FOL improves accuracy by up to 13.95\\% on Wildfire ($\\psi$ = 0.5), 30.16\\% on Hurricane ($\\psi$ = 0.3), 6.77\\% on CIFAR-10 ($\\psi$ = 0.7), and 9.01\\% on CIFAR-100 ($\\psi$ = 0.7). These consistent and substantial gains confirm that our method effectively extracts and integrates valuable knowledge from neighbor models.\n\nAs for the suggestion of using the storage budget (30 times of model size) to train a larger local model instead of using them to fuse neighbors' models, we note that increasing model size alone can potentially increase the risk of overfitting and cannot achieve the benefits of aggregating (new) knowledge from non-IID clients. Additionally, we also want to point out that the 30-times storage cost is only needed at the ensemble step. After the ensemble model is distilled, the final personalized model has the same size as a standard local model.\n\n**4.Privacy Concerns.**\n\n **Response:** In this work, our primary focus is on improving model accuracy, and thus the issue of privacy is out of the scope of this paper. In real-world applications such as Starlink LEO satellite networks, those satellites typically belong to the same operator, therefore there is no privacy issue. Moreover, if privacy is required in other applications, homomorphic encryption can be integrated into the parameter sharing process, enabling computations directly on encrypted data without compromising performance.\n\n**5. Training Time or Communication Costs Against Baselines.** *The communication cost of peer-to-peer communication......needs to be experimentally compared with....*\n\n**Response:** We want to point out that our proposed FOL method has lower computation and communication overhead than DENSE and Co-Boosting (i.e., the baselines). Specifically,   DENSE and Co-Boosting rely on a modified GAN structure in which the generator is composed of multiple large fully connected layers and runs for 30 epochs per distillation epoch over a total of 200 epochs (i.e, in total 6000 epochs for GAN alone). In FOL, the computation cost of various components are as follows:  fine tuning: 30 epochs, structured pruning: 30 epochs, post fine tuning: 30 epochs, ensemble refinement: 10 epochs, knowledge distillation: at most 500 epochs. In total: at most 600 epochs, which is much smaller than that of DENSE and Co-Boosting.  Moreover, these additional processes incur only local computational cost, with no extra communication overhead.\n\n\n**6. Related Work In Decentralized Learning.**\n\n**Response:** In the revised version, we will add additional citations and discussion of relevant decentralized learning approaches. It is important to note that to the best of our knowledge, no existing decentralized learning work has provided a one-shot personalized features. Our work is the first to provide such a feature with both theoretical guarantees and experiment validation.", "claims_and_evidence": "The assumption that the local validation set has the same distribution as the final test set, and using it to determine the contribution of each model in the ensemble, can lead to overfitting on the final test set. This is unrealistic in practical scenarios.", "methods_and_evaluation_criteria": "The communication cost of peer-to-peer communication between all nodes and their neighboring nodes needs to be experimentally compared with the communication cost in a federated learning setup with a central server.", "theoretical_claims": "No.I haven't checked the details in the proofs. Broadly speaking, such assumptions and results are reasonable.", "experimental_designs_or_analyses": "1. FOL involves multiple stages (e.g., fine-tuning, pruning), but the paper does not compare training time or communication costs against baselines. For resource-constrained environments, this omission limits practical applicability assessment.\n\n2.  In the experiments of this paper, the authors emphasize the performance of the ensemble model, but the ultimate goal of the algorithm should be a single personalized model. Furthermore, the performance of naive local training is actually quite similar to the personalized model produced by the algorithm, especially considering the significant increase in storage and computational costs introduced by the proposed method. If the 30 times the storage space used for the ensemble model were instead allocated to improving the local model, the performance of the local model could potentially be further enhanced.", "relation_to_broader_scientific_literature": "The authors focus on a setting where there is no central server, and clients communicate directly with their neighbors, which represents a decentralized learning framework. This field has already been extensively studied, and the paper should reference related works in this area.", "essential_references_not_discussed": "I think some related decentralized learning works should be cited,", "other_strengths_and_weaknesses": "The authors focus on a setting where there is no central server, and clients communicate directly with their neighbors, which represents a decentralized learning framework. This field has already been extensively studied, and the paper should reference related works in this area.\n\nIn the proposed setup, where clients directly communicate model parameters with their neighbors, serious privacy concerns arise. Additionally, the assumption that the local validation set has the same distribution as the final test set, and using it to determine the contribution of each model in the ensemble, can lead to overfitting on the final test set. This is unrealistic in practical scenarios.\n\nThe definition of \"neighbor models\" in the paper lacks sufficient detail. Specifically, it remains unclear how the adjacency or relationship between neighbors is determined and whether it changes over time.\n\nThe communication cost of peer-to-peer communication between all nodes and their neighboring nodes needs to be experimentally compared with the communication cost in a federated learning setup with a central server.\n\nIn the experiments of this paper, the authors emphasize the performance of the ensemble model, but the ultimate goal of the algorithm should be a single personalized model. Furthermore, the performance of naive local training is actually quite similar to the personalized model produced by the algorithm, especially considering the significant increase in storage and computational costs introduced by the proposed method. If the 30 times the storage space used for the ensemble model were instead allocated to improving the local model, the performance of the local model could potentially be further enhanced.", "other_comments_or_suggestions": "In the problem statement part, “Given an image classification task” may be not necessary. As far as I know, other PFL articles usually clarify the specific task in the experiment instead of problem setting. Personally, I think deleting this sentence makes the method look more generalized rather than being limited in a specific domain. Of course, this is just a suggestion, not a necessity.", "questions_for_authors": "The details have been discussed in the strengths and weaknesses part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes FOL, to iteratively enhance the local training via utilizing neighboring models. The authors also propose theoretical analysis regarding the discrepancy and the distillation.
Decision: Accept (poster)