Federated Learning for Feature Generalization with Convex Constraints
Authors: Dongwon Kim
Donghee Kim
Sung Kuk Shyn
Kwangsu Kim
Venue: ICML 2025 poster (2025)
Keywords: Federated Learning
Convex Optimization
Primary area: general_machine_learning
TL;DR: FedCONST improves feature generalization in Federated Learning (FL) by adaptively adjusting model updates based on parameter strength using convex constraints, leading to state-of-the-art performance.

Abstract: Federated learning (FL) often struggles with generalization due to heterogeneous client data. Local models are prone to overfitting their local data distributions, and even transferable features can be distorted during aggregation. To address these challenges, we propose FedCONST, an approach that adaptively modulates update magnitudes based on the global model’s parameter strength. This prevents over-emphasizing well-learned parameters while reinforcing underdeveloped ones. Specifically, FedCONST employs linear convex constraints to ensure training stability and preserve locally learned generalization capabilities during aggregation. A Gradient Signal-to-Noise Ratio (GSNR) analysis further validates FedCONST's effectiveness in enhancing feature transferability and robustness. As a result, FedCONST effectively aligns local and global objectives, mitigating overfitting and promoting stronger generalization across diverse FL environments, achieving state-of-the-art performance.

Federated Learning for Feature Generalization with Convex Constraints
Dongwon Kim
Donghee Kim
Sung Kuk Shyn? 2
Kwangsu Kim
Abstract
Federated learning (FL) often struggles with gen- eralization due to heterogeneous client data. Lo- cal models are prone overfitting their local data distributions, and even transferable features can be distorted during aggregation. To address these challenges, we propose FedCONST, ap- proach that adaptively modulates update mag- nitudes based on the global model's parameter strength. This prevents over-emphasizing well- learned parameters while reinforcing underdevel- oped ones. Specifically, FedCONST employs lin- car convex constraints to ensure training stabil- ity and preserve locally learned generalization capabilities during aggregation. A Gradient Sig- nal to Noise Ratio (GSNR) analysis further val- idates FedCONST's effectiveness in enhancing feature transferability and robustness. FedCONST effectively aligns local and global objectives, mitigating overfitting and promoting stronger generalization across diverse FL environ- ments, achieving state-of-ther oortr-rrrt performance.
Without Constraint
Non-Convex Constraint
Proposed Method
aggregation
Generalization
Constraint
Training
Figure I. Illustration of the parameter space in FL. (1) Vanilla FL. drives the optimization process away from the generalization area. (2) Optimization with non-convex constraints stabilizes the loss
training process within the generalization
but may of generalization during aggregation. (3) Our convex constraints stabilize the training process and align the aggregation with the gencralization area, ensuring improved global generalization.
result,
et al., 2022). (2) Even when some clients learn features that could generalize, these can become distorted during aggregation, misaligning them with global objectives and undermining performance(Lee & Yoon, 2024).
1. Introduction
Previous approaches, including regularization(An et al., 2024; Li et al., 2021a; 2020), normalization (Li et al., 2021b; Andreux et al., 2020; Wang et al., 2023) and correction (Acar et al., 2021: :: Karimireddy et al., 2020; Varno et al., 2022) aim mitigate these issues by aligning local model updates with the global model. However, by focusing solely preserving global model information, these methods have neglected to ensure generalization, inevitably resulting in overfitting under limited data conditions.
Federated Learning (FaaMMnaahan et al., 2017) has emerged as a promising paradigm that enables multiple clients to collaboratively learn shared model while keep- ing their data localized. pivotal challenge in FL arises from the sparse and heterogeneous data distribution across clients, which leads to significant problems on the perfor- mance of global model. (1) Local models often overfit their own distributions, limiting their capacity to generalize across the entire data distribution((u al., 2022; Mendieta
Recently, some studies have turned their attention to gener- alization of local learning. FedAlign(Menddita al., 2022) and FedSAM(Qu et al., 2022) focused on the generaliza- tion of local learning employing generalization term loss. However, by failing to preserve coherent optimization objec- tives across clients during aggregation, these methods allow the model's generalization capabilities to become distorted, ultimately degrading its overall performance.
'Department Computer Science and Engineering, Uni- versity of Sungkyunkwan, Suwon, Korea "Kim Jaechul Grad- uate School of AI, Korea Advanced Institute of Science and Technology (KAIST). Daejeon, Korea. Correspondence to: Dongwon Kim 2kdwaha@skku.eduu skku, Kwangsu Kim <kim.kwangsuee skku.edu>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In light of these issues, we ask: What kind of constraint

---

Federated Learning for Feature Generalization with Convex Constraints
enhances feature generalization during local training while
2. Related work
remaining unaffected by the aggregation process?
2.1. Federated Learning on Non-IID Data
To answer this question, We propose FedCONST (Federated Learning with Convex Constraints for Global Model Gen- eralization). FedCONST applies client-consistent convex constraints derived from the global model's weight magni- tudes, which serve as proxies for feature importance across the entire data distribution. Concretely, well-learned (sta- ble) features in the global model are constrained to remain close during local updates, while under-learned (unstable) features are emphasized for further training. Because the constraints are convex and shared among all clients, they preserve the intended generalization effect after aggregation as desribed in Figure
The challenge of non-lID data across clients leads to unsta- ble local learning diverge the global model from consistent optima. To mitigate these issues, regularization methods have been prominently adopted. Techniques like FedProx (Li et al., 2020), FedMRUR(An et al., 2024), and MOON(Li et al., 2021a) incorporate explicit regularization mechanisms to align local updates with global objectives more effectively. These approaches ensure that the local models have common features for global objectives. However, common features are often spurious as well, making global model suffer from the overfitting problem.
To directly address the challenges posed by heteroge- neous gradients, correction methods like FedDyn(Acan et al., 2021), AdaBest( Varno et al., 2022), and SCAF- FOLD(Karimireddy et al., 2020) introduce correction terms that aim to align client updates with the global model. These strategies utilize stateful operations to align client updates FL environment with limited local data. Aside from the risk on alignment of overfitting problem, this stateful operations require extra communications.
Our method is inspired by insights from Domain Gener- alization (DG) method(Michalkiewiic et al., 2023), where strong features (often measured by gradient statistics such as Gradient Signal-to-Noise Ratio, GSNR) are crucial to robust performance across diverse domains. Directly track- ing GSNR is typically infeasible in FL due to communi- cation and computational bottlenecks. Instead, we show that global weight magnitudes are reliable proxy for feature strength, aligning well with the GSNR perspective. This design choice makes FedCONST simple, communication- efficient, and broadly applicable.
On the other hand, methods such as FedSAM((u et al., 2022) and MMendiia et al., 2022) focus on enhane- ing generalization across clients without imposing restric- tions for alignment with global objectives. These techniques prioritize generalization of local training limited data but unaligned approach to handle the heterogeneity inherent in FL.
In this paper, we provide the motivation behind our work and a theoretical foundation for our method. We demonstrate higher stability in local training through reduced gradient variance and improved convexity of the global model loss landscape. Consequently, our experiments show that Fed- CONST significantly outperforms existing FL methods in various models, datasets, and levels of heterogeneity in both cross-device and cross-silo settings, while maintaining high computational and communication efficiency.
2.2. Generalization of Neural Networks
To analyze generalization performance during the training process, study proposed the concept of One Step Gen- eralization Ratio (OSGR)(Liu et al.). OSGR is defined as the ratio between the decrease in loss on test data and on training data:
In summary, our contributions are as follows.
We propose a simple, yet yet effective approach that retains well-learned features while focusing on under-learned ones. This framework introduces new insights into how generalization can be enhanced in FL.
Rz (
(I)
ED-z0 [ACp]
where ALDs and ALD denote the decrease in loss on train- ing data D) and test data D' within single optimization step.
Our theoretical and empirical analyses guarantee that our method boosts generalization by imposing more updates with larger probabilities to unaee-d fea- tures.
To facilitate the prediction of OSGR during training, the authors further propose the following theorem.
We validate the effectiveness of FedCONST with a wide range of dataset and model architectures and show that it significantly outperforms existing FL methods with SOTA performance.
Proposition 2.1 (From Paper (Liu et al.)). The generaliza- tion of gradient updates can be expressed using the follow- ing OSGR value:
Rz,
1-
1 n1
Σ
W
1
(2)
2

---

Federated Learning for Feature Generalization with Convex Constraints
where the number of samples, gÿ the squared gradient magnitude for feature j and 03 is the corresponding noise variance, and The weight 5 (satisfying E, W = =) weighting term.
Client Training Pivoting to the FedAvg blueprint, the local training trajectory is captured
with w. Σ LET
Im. t w + Aww
(6)
Proposition 2.1 indicates that features with higher Gradient Signal-to-Noise Ratios(GSNR) defined by:
Here, w typifies the global model during global round k. gin, e is the gradient corresponding to client m at timestep of the 1thh global round, and r denotes the learning rate.
Age Aggregation Transitioning to the global aggregation phase, the mechanics unfold as:
yield larger values of OSGR, thereby contributing more significantly to generalization performance.
Based on these insights, a Paper (Michalkiewicz al., 2023) proposed GSNR-based dropout method for DG tasks, aim- ing to enhance robustness by preserving parameters with higher GSNR values while promoting updates in those with lower GSNR. However, the distributed nature of FL compli- cates gradient collection and aggregation across clients.
W
Σ MEM
w = =
Σ MEM
Aw m
M
M
W
Au
In this matrix, Au* epitomizes the average model updates accumulated from all clients during the kas global round.
3. Our Approach
3.2. FedCONST:Federated Learning for Feature Generalization with Convex Constraints
To begin with, we define standard learning process is to train a deep neural network f (a;; W), where f X
+ Y is a
For promoting generalization in FL, we begin by proposing a conjecture and two principles to design our constraints. Next, we demonstrate that our method adheres to these prin- ciples. Finally, we present evidence support the validity of the conjecture.
neural network with L neural layers:
W = {w', 2²
(4)
Especially, We are interested in training each weight w' [w". w...)) with respect to the corresponding fea- ture/channel on input layer I.
3.2.1. CONSTRAINED OPTIMIZATION FOR CLIENT TRAINING
3.1. Federated Learning
We propose a Constrained Weight Optimization framework that transfers the key insight of Feature introduced in DG methods-to a FL scenario. Our approach is motivated by Conjecture 1, which states that the magni- tude of weight serve a proxy for how well feature is learned.
We consider the standard FL that trains a model collabora- tively from decentralized client devices. For every client, denoted as in the set M, there are Nm, training samples given by pairs (x4, 8/) for 1 to Nm. Here, x; represents the image from set X and y; is the corresponding label from set Y. These pairs are independently and identically distributed, drawn from a distribution specific to the device, symbolized as Dm (I. y) when D = Um Dm- With this setting, we follow the framework of FedAvg follows:
Conjecture 1 Large weights indicate well-learned (strong) features. Small weights signify weaker features requiring additional training.
By leveraging this conjecture, we can selectively preserve strong features while reinforcing weaker ones, preserving the core principle of DG methods without direct access to each client's detailed gradients. To operationalize Conjec- ture within federated learning, we introduce a constrained optimization framework:
Dmml
C(W)=
Σ
L. (W)
MEM D
(5)
where (W) = E E(rr. in. [C( 34: W)]
The global objective, denoted L, can be broken down into individual empirical loss f. specific to each client data. Because of the separation of clients' data, L can not be optimized directly. FedAvg addresses this challenge by alternating between local training on each client's 's dataset and global aggregation step.
min Lm W
(8)
s.t. G. - Go)
1 Vc,l
Here, wi represents the weight vector for the i-th output feature in the I-th layer of client model, and G., means
3

---

Federated Learning for Feature Generalization with Convex Constraints
Client- Side
Visualization Parameter Space
Server - Side
Uploads
Projection
Σ
cicents
Orthogonal
Centeriigg
Clica
Frojection
Poor Features
Downloads
Update Preserve
Globst Mode
Cliert
Cilent
Features
Client
Clint
W
AW
Generalization Area
Construints
of Clicst on Round Boration
Parunnte Upskicc Clicnt on Round
Medi
Updatcc Global Model on Round
Figure Schematic representation FedCONST: Local learning on clients with weight constrained optimization preserve robust paramter of the global model. Aggregation phase where convex constraints guide diverse client models toward common, optimal representation, facilitating better alignment and performance of the global model.
the corresponding weight vector of global model. 1 is ture/channel i follow spherical Gaussian distribution: all-one vector.
AWeg N (0, o2) where & = 1,2.....
(9)
This design originated form the insight that constraints should adjust feature learning while maintaining general- ization ability after the aggregation phase. This can be summarized the following condition:
0² Σ, (AW! 2.9)
(10)
We project the weight changes onto the hyperplane orthogo- nal to the initial client weight vector W which is initialized to match the global parameter G and define the projec- tion as follows:
Condition 1. (Feature Adjustment) The constraints boost weak features and preserve already strong features for gen- eralization.
Condition 2. (Convex Constraints) The constraints should be conserved after aggregation to preserve generalization ability.
W |W
P
"
(11)
the projected weight update is:
Our design consists of two constraints, the centralization constraint and the orthogonal constraint. The centralization constraint(1 w/ total update impact on a feature is equalized, while the or-
Awl PAw, Cov(Aw = P Cov(Aw) pT = -2p
(12)
on Equation (8)) ensures that the
thogonal constraint(G' (w: GD) = 0 on Equation (8)) mitigates redundant updates already strong signals. To-
The resulting variance aligns with:
Var Var (. = 0² (1
=0² W! 2 c.g I
gether, these constraints satisfy Condition by promoting generalization during local training and Condition 2 by be- ing linear and convex.
(13)
So these discussion can be summarized as the following proposition:
3.2.2. FEATURE ADJUSTMENT
Theorem 3.1
Updates under Centering and Orthogonality). If we impose centering and orthogo- nality constraints, and if W S W then
In this section, we justify our constraints prevent redun- dant training on strong features. We assume that weight changes due to gradient updates on parameters W! (W! wi...., Wl 3 with respect the corresponding fea-
Var(AW' Var(AW: ))

---

Federated Learning for Feature Generalization with Convex Constraints
which means that updates are inversely correlated with the weight size, i.e,,
the strong features. This adjustment translates the constraint such that the local gradient is orthogonal to the initial global model:
Pr(1AW IAW 2 Pr(IAW",, - SAW
(w0²) Im,
(18)
Therefore, constraints ensure that updates have in- verse correlation with weight size, promoting stability and avoiding overfitting to features with larger weights.
then it results in
Σ ( Im,t = =w²)) (-n) Σ smm.)
TET
TET
(19)
3.2.3. CONVEX CONSTRAINTS
= =ww*) Aw 0
Each client optimizes its local model independently on pri- vate data, and these models are then aggregated to form a global model without alignment. To address this misalign- ment, we refine constraints to be convex, ensuring they satisfy the conditions for generalization both during local training and after aggregation. Although these constraints are applied per weight vector for channel/feature e of layer 1, we omit è and in the notation for brevity and to focus on the convex characteristic.
During aggregation, Equation (19) aligns the global update
as:
Σ (w") Aw"" = (w*)) " Aw = (20)
M
MEM
In Algorithm 1, we apply the projection operator Pus to each update direction to ensure that local updates remain orthogonal to the initial global parameter p = w* /||w*|i
Centralization Constraint(1 T wn 0 Equation (8)) The Centralization Constraint is specifically applied to the gradient of each client's local model to stabilize the local training process by maintaining the mean to be zero. If each gradient of every client is centralized as:
Par (w) = (I pp )m²,
(21)
where I is the identity matrix, and pp" is the outer product of p with itself. This projection operator is used in our algo- rithm to enforce the orthogonality constraint by projecting updates onto the tangent space of the global model direction.
1 gm.t = 0
(14)
then it results in
3.3. Weight Size as a Feature Strength
Σ 'ET
T Im,t
fm,t=11 =
(-n Σ f = 1T A =0 (15)
In this section, we discuss about validity conjecture 1. Fortunately, the paper (Liu et al.) also provided a detailed analysis of the relationship between GSNR and weight size, indicating positive correlation. They considered fully connected network with parameters
'ET
Consequently, the total change of a client model during local training becomes zero, ensuring unbiased weight on client models.
By Equation (15), the global update adhering to the Cen-
-WW W'mnx)
(22)
tralization Constraint is represented as:
where W b/ are the weight matrix and bias of the first layer, respectively, and on. The activations of the I-th layer are denoted by
Σ A = 1 Aw =0 MEA
(16)
M
a' a (a'
(23)
In Algorithm to apply the centralization constraint 1T on = in practice, we define the following centering
where is the index for nodes/channels, and
function:
01-1= [W- (===,...,,W''->
C(w) =
w
(17)
(24)
where n denotes the number of parameters in w. This func- tion is directly used in our algorithm enforce the central- ization constraint during local updates.
is the collection of parameters in the layers before 1. The for- ward pass on data sample i, where (a'((0'-1) is multiplied by the weight matrix W defined as:
Orthogonal Constraint(G! T (w!. - G.) = 0 on Equa- tion (8))
oe Σ W! ((.
(25)
The orthogonal constraint align the parameters of local and global models onto the same hyperplane orthogonal to the initial global model at each round, mitigating the grow in
where o' = {of_e } is the output for the i-th data sample on the 1-th layer, and e the index for nodes/channels. We use
5

---

ge to denote the average gradient of weights of the I-th layer W i.e. Federated Learning for Feature Generalization with Convex Constraints
Algorithm 1 Training procedure of FedCONST
Input: Batch size B, communication rounds K, number of clients M, local steps T. dataset D Ume[M/ Dm
ac;
g'
(26)
aw
Output: Global model parameters w
Server executes:
where C is the loss of the i-th sample.
Initialize w with He Initialization
In this setting, the authors showed the following correlation between gradient change and gradient norm size:
for k = O..... , K
do
for m 1, .............................. . M in parallel do
Send w² to client
Ag'.c
A n
Σ
Wi 8,0
1 n2 1=1
ac,
0okic"
w++1 + FedCONST: Client executes(m, w
do
00;-
(27)
8, --
end
+ other terms
w 8+1 E m
[D k+1
To u
where À is the learning rate, assumed to be small enough.
end
This expression implies that if the gradient change Ag'.c and the corresponding weight W s.c. have different signs, they contribute to a more stable state by constructing positive feedback during training, which increases the size of both values. Conversely, they have the same signs, a negative feedback loop during training decreases the size of both values until one of them changes its sign, resulting in stable state.
return w
FedCONST: Client executes (m, w²):: Assign global model to the local model w + w
for each local epoch = 1..... T do
for batch 11 F.,,1.B) Dm do
Per layer and channel/feature c,
Center gradient: gm.t + C(om..) Project gradient: Iin,t + Park (gtn.t) Apply update: w +- w ngm,t
Therefore, considering only stable states, the size of the weight W s,c is directly correlated with the gradient change
end end return w++1
g on Equation (3). Here, we decided to use the size of W s.c of the global model instead of collecting g'cc. statistics to calculate GSNR values for the entire dataset. As a result, rather than utilizing GSNR as an indicator of feature strength, which requires collecting gradients from each client, we adopt the weight magnitude of the global model a proxy for feature strength.
Agh.cs positively affecting the GSNR value of ry P
to server
spanned both cross-silo and cross- settings. For the cross-silo setup, we involved a total of 10 clients, while in the cross-device setting, 10% of clients were randomly selected from pool of 50 or 100 participants. The data dis- tribution among clients was governed by a Dirichlet distribu- tion, with the value determining the degree of heterogene- ity; a lower value corresponds to more heterogeneous distribution. For an extremely heterogeneous environment, we used a Dirichlet a of 0.2 with 10 local training epochs, while a more typical environment utilized an alpha of 0.5 with 5 local training epochs. Our tests were conducted on both the LeNet-5 and ResNet- 18 architectures. More detailed settings of experiments are in the supplementary materials.
3.4. Training Process
As shown in Algorithm 1, FedCONST merely changes the local learning process to constrained optimization convex linear constraints for Global Model Generalization. This approach has two main advantages: (I) Using convex constraints based on common global model, we align local training across client models without additional communica- tion cost in stateless manner. (2) By employing the weight size estimate the generalizability of the features, we en- sure the generalizability of the FL process. Moreover, this insight is well motivated by GSNR based analysis.
4.2. Performance Comparison
As detailed in Table 1, the LeNet-5 model consistently outperformed the existing algorithms across all settings. As shown in Table 1, our approach surpassed the perfor- mance of existing algorithms in all settings on CIFAR-10 and demonstrated commendable performance on CIFAR- 100. Table further reveals that a cross-device setting, our algorithm consistently exceeded the performance of the other algorithms. Moreover, performance enhancement was
4. Experiments
4.1. Experimental Setup
We conducted experiments using the CIFAR-10 and CIFAR-
100 datasets (Krizhevsky et al., 2009). Our experiments
6

---

Federated Learning for Feature Generalization with Convex Constraints
Table Top-1 accuracy (%) comparison of LeNet-5 and ResNet-18 18 models under Pross-Device and Cross-Sllo settings. The numbers inside the parentheses represent the accuracy differences when Constraint was applied to the training of client models.
BROSS-DEVEE
CROSS- SIOS
CIFAR-I0
CIFAR-10
CIFAR-I0
CIFAR-100
MODEL
ALGORITHM
= 0.5
= 0.2
= 0.5
a 0.5
FEDAVG
46.12
46.42
53.12
17.46
FEDAVG + CONST
54.28 (+8.16)
54.79 (+8.37)
$9.66 (+6.54)
26.86 (-9.40)
FEDPROX
45.58
45.27
35.15
18.42
FEDPROX CONST
53.09 (47.51)
56.18 (+10.91)
60.70 (65.55)
26.78 (-8.36)
LENET-5
MOON
43.89
46.66
55.79
18.72
MOON CONST
48.66 (-4.77)
52.88 (+6.22)
59.86 (+.00))
26.76 (-8.04)
SCAFFOLD
45.66
45.67
32.74
17.66
SCAFFOLD CONST
53.82 (+8.16) 44.93
56.62 (+10.95)
63.03 (.10.29)
26.74 (-9.08)
FEDDYN
48.05
51.05
16.79
FEDDYN CONST
54.07 (-9.14)
55.67 (+7.62)
59.76 (+8.71)
27.14 (-10.35)
FEDAVG
54.07
57.04
64.25
33.51
FEDAVG + CONST
66.51 (-12.44)
68.41 (411.37)
72.44 (+8.19)
36.82 (-3.31)
FEDPROX
56.79
33.92
64.51
34.11
FEDPROX + CONST
63,51 (-6.72)
68.07 (+14.15)
71.96 (+7.45)
36.56 (-2.45)
RESNET-18
MOON
57.84
51.51
68.45
35.19
MOON CONST
66.94 (49.10) 56,47
62.52 (+11.0)
71.84 (03.39)
36.80 (-1.61)
SCAFFOLD
$5.30
64.50
37.18
SCAFFOLD CONST
63.49 (-7.02)
68.63
75.09 (+10.59)
38.93 (-1.75)
FEDDYN
52.64
35.09
65.50
35.07
FEDDyN + CONST
64.29 (-11.65)
66.00 (+10.91)
71.76 (+6.26)
37.22 (-2.15)
FEDSAM
62.52
61.35
69.45
38.43
FEDSAM CONST
63.45
68.87
72.64 (+3.19)
39.61
observed when constraints applied to algorithms repre- sentative of either aligmment or local learning generalization methods. This indicates that our method fills the gaps and enhances areas where existing methods fall short, leading to a more robust and efficient FL, process.
4.3.2. MODEL ALIGNMENT ACROSS CLIENT
In analysis of loss landscape, shifted analyti- cal focus from sharpness based metrics convexity- based metrics. Previous research primarily utilized the maximum eigenvalue Amas of the model's 's Hessian matrix to measure sharpness, which correlates with generalization performance (Mendieta et al., 2022). However, our analysis suggests that observing the ratio of the absolute maximum eigenvalue the minimum eigenvalue Amaz//Amic vexity (Rangwani et al., 2022) more effectively captures the essence of model alignment.
4.3. Experiment Analysis
4.3.1. STABILITY ON LOCAL TRAINING
measure of con-
Figure 3(a) demonstrates that our method reduces gradi- ent variance compared to FedAvg, thereby stabilizing local updates environments with sparse data. Moreover, Fig- 3(b) illustrates that our method enhances drift diversity,
defined EmCa [2
Aw
(Li al., 2023) ensuring
Table 2. Loss Landscape Convexness. The metric Cconves rep- resents the /mmez /mmnn values of ResNet-18 model in Cross- Device settings. The term Hirace denotes Hessian trace value.
that each client learns effectively and can fully reflect its own unique data characteristics even with constraints. The increase drift diversity compared to FedAvg indicates that method results in larger client updates after local train- ing, enabling effective model updates despite sparse data or parameter constraints. Overall, these results demonstrate that our approach effectively mitigates instability in local training, ensuring sufficient updates within the imposed constraints.
w/o CONSTRAINTS
CONSTRAINTS
ALGORITHM
Comnvs
Hıracs
Cconver
Hirace
FEDAVG
2.466
-4951
31.7
10102
FEDPROX
2.739
-3873
23.09
12294
MOON
3.015
-3416
16.09
9640
SCAFFOLD
2,914
-3245
21.81
9145
FEDDYN
2.16
-4590
12.82
9121
FEDCONST
31.7
10102

---

Federated Learning for Feature Generalization with Convex Constraints
1071
halay
Teckug (thep quart
holDyn
TecDyn ialap
MOON lstep, goart
guar)
50
300
250
200
250
20
30
Epoch
Round
(a)) Gradient Variance
((b)) Drift Diversity
Figure 3. Gradient variance and Drift diversity. Our method reduces the gradient variance, thereby stabilizing the local training of client models. Simultancously, it enhances drift diversity, ensuring that each client learns effectively even with constraints.
Table 2 shows how the application of constraints secures proper alignment, thereby sculpting more convex and advantageous loss landscape for the global model. Fur- thermore, a negative trace value of the Hessian suggests convergence to a saddle point-a non-ideal scenario. Hence, Table 2 also demonstrates that the implementation of con- straints contributes to more convex loss landscape, steering the model away from saddle points towards optimal conver- gence.
all clients. This increased consistency contributes struc- tured and predictable optimization trajectory, reinforcing the benefits of convexity based alignment.
4.3.3. GENERALIZATION ON FRATURES OF GLOBAL MODEL
Additionally, we introduce client consistency another key metric. which quantifies the consistency of local models across clients. This consistency defined as ΣMEM [Dml A 12. where lower values indicate that client updates remain proportionally aligned, preventing excessive deviations in heterogeneous data environments.
Doech
Figure 5. Weight sizes and GSNR values. Weight magnitudes and GSNR values sampled from the top 10% and bottom 10% of gradient update magnitudes across clients. metrics show correlation with gradient updates, suggesting their relevance to GSNR the federated learning setting.
1001
Our conjecture is that if already generalizable global feature is changed during local training of client, it harms general- ization ability of FL process. To prove our hypothesis, we sampled top-10% and bottom 10% of gradient update size on changes. and we observe the GSNR value and weights size corresponding feature to validate our hypothesis. And finally, compared the GSNR value of vanilla Fe- dAvg and our method indicating improved generalization performance.
Figure Consistency. Our method enhances consistency among clients, by ensuring more aligned learning experience across all clients.
The experimental results in Figure confirm that our method significantly enhances client consistency compared to Fe- dAvg, ensuring a more aligned learning experience across
First, Weight sizes corresponding to top 10% of Gradient update sizes have larger value than bottom 10%. Obser-
8

---

Federated Learning for Feature Generalization with Convex Constraints
datasets. However, real-world industrial data is often dis- tributed, imbalanced, and noisy, making it challenging apply academic research directly to practical settings de- spite the enormous potential of distributed data. Federated Learning (FI.) is a promising technology to address this lim- itation, yet further improvements are needed for real-world deployment. This paper proposes a method to enhance FL performance by introducing a novel generalization perspec- tive that mitigates its inherent limitations.
Epace
References
Figure 6. The sum of GSNR values on a Client. The sum of GSNR values on the initial local epoch of the proposed method is higher, indicating that features aligned with the global model are properly enhanced.
Acar, D. A. E., Zhao, Y., Navarro, R. M., Mattina, M., Whatmough, P. N., and Saligrama, V. Federated learn- ing based on dynamic regularization. arXiv preprint arXiv:211 11fi:21,1..4263,, 2021.
An, X., Shen, L., Hu, H.,, and Luo, Y. Federated learning with manifold regularization and normalized update reag- gregation. Advances in Neural Information Processing Systems, 36, 2024.
vation on Figure 5 indicates gradient update on client is associated with global generalizable feature, which harms generalization ability of global model. On GSNR values, top 10% also has larger values showing more direct impact of client training on generalization of global model.
Andreux, M., du Terrail, J. O., Beguier, C., and Tramel, E. W. Siloed federated learning for multi-centri histopathology datasets. In Domain Adaptation and Rep- resentation Transfer. and Distributed and Collaborative Learning: Second MICCAI Workshop, DART 2020, and First MICCAI Workshop, DCL 2020, Held Conjunc- tion with MICCAI 2020, Lima, Peru, October 4-8, 2020, Proceedings 2, pp. 129 139...9.600 Springer, 2020.
After we impose our constraints that conserve common generalizable features, Figure shows increase of GSNR values overall weight with constraints proving our method work as intention.
5. Conclusion
In this research, we have introduced FedCONST, a novel algorithm that leverages convex constraints during opti- mization of client model for increasing generalization. This algorithm fosters stable local training on convex constraints, leading to more generalizable global model through learn- ing common features based on corresponding weight size of the global model. Our comprehensive experiments have shown that FedCONST not only stabilizes the learning pro- cess at the client level but also ensures consistent alignment to generalizable features. various experimental settings, especially in the presence of highly heterogeneous data, FedCONST consistently outperformed existing algorithms.
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S.,
and Suresh, A. T. Scaffold: Stochastic controlled averag- ing for federated learning. In International conference on machine learning, pp. 5132-5143. PMLR, 2020.
Krizhevsky, A., Hinton, G., al. Learning multiple layers of features from tiny images. 2009.
Lee, and Yoon, S. W. Rethinking the flat minima searching in federated learning. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scar- lett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference Machine Leurn- ing, volume 235 of Proceedings of Machine Learn- ing Research, pp. 27037 PMLR, 21-27 Jul 2024. URL https: hppp///peecigggseee/ee/ee/e mlr press/ v235/1 4 html.
Acknowledgements
This work was supported by Korea Internet & Secu- rity Agency(KISA) grant funded by the Korea govern- ment(PIPC) $6o.RS-2023--022 1200, Development of per- sonal video information privacy protection technology capa- ble of AI learning in an autonomous driving environment)
Li, B., Schmidt, M. N., Alstrøm, T. S., and Stich, S. U. On the effectiveness of partial variance reduction in federated learning with heterogeneous data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3964-3973, 2023.
Impact Statement
Many studies on deep neural networks are conducted under the centralized learning paradigm with well-preprocessed
Li, Q. He, B., and Song, D. Model-contrastive federated learning. In Proceedings of the IEEE/CVE conference

---

Federated Learning for Feature Generalization with Convex Constraints
compuler vision and paltern recognition, pp. 10713- 10722, 2021a.
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 2020.
Li, X., Jiang, M., Zhang, X., Kamp, M., and Dou, Q. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623, 2021b.
Liu, J., Bai, Y., Jiang, G., Chen, T., and Wang, H. Under- standing why neural networks generalize well through gsnr of parameters. In International Conference on Learn- ing Representations.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and
Arcas, B. A. Communication-effiieenn learning of deep networks from decentralized data. Artificial intelli- gence and statisties, pp. 1282-1282. PMLR, 2017.
Mendieta, M., Yang, T., Wang. P., Lee, M., Ding, Z., and Chen, C. Local learning matters: Rethinking data het- crogeneity federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8397-8406, 2022.
Michalkiewicz, M., Faraki, M., Yu, X., Chandraker, M., and Baktashmotlaghh, M. Domain generalization guided by gradient signal noise ratio of parameters. Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6177-6188, 2023.
Qu, Z., Li, Duan, R., Liu, Y., Tang. B., and Lu, Gen- eralized federated learning via sharpness aware minimiza- tion. In International conference on machine learning, pp. 18250-18288 PMLR, 2022.
Rangwani, H., Aithal, S. K., Mishra, M., et al. Escap- ing saddle points for effective generalization on class- imbalanced data. Advances in Neural Information Pro- cessing Systems, 35:22791-22805, 2022.
Varno, F., Saghayi, M., Rafiee Sevyeri, L., Gupta, S., Matwin, S., and Havaei, M. Adabest: Minimizing client drift in federated learning via adaptive bias estimation. In European Conference on Computer Vision, pp. 710-726. Springer, 2022.
Wang, Y., Shi, Q., and Chang, T.-H. Why batch normaliza- tion damage federated learning on non-iid data? IEEE Transactions on Neural Networks and Learning Systems, 2023.
10

---

Federated Learning for Feature Generalization with Convex Constraints
A. Generalization Area and Implementation on Constraints
Figure conceptually illustrates how aligning the aggregation step with the generalization area improves global generalization aggregation. For our method be effective, the constraint region must be contained within the generalization area. We argue that this condition is satisfied in practice, the generalization area is sufficiently large under typical training regimes.
FL. This motivates our use of convex constraints that stabilize both local training and
We provide an intuitive argument to suggest that the generalization area can be sufficiently large, based on the behavior of I-th layer representation of neural network:
e__ - = ((ee(l11)) (e(l)))TT!!)))...
(28)
where é is activation function (e.g., tanh), and _ denotes representation within the generalization regime. We consider whether the perturbed form
(o-- (w: + +AW!)) Z
(29)
still holds under certain conditions.
Case 1: Large [
( (Saturation Regime). When | || is large, the activation function saturates, and the output
becomes relatively insensitive to A !!! Thus, wide range of perturbations can yield generalizable representations.
Case 2: Small
|W!]|| (Linear Regime). When ! is small, the activation function behaves almost linearly:
e s "111) !!!.
(30)
We apply Chebyshev's inequality:
Var(e (1)
P((50 C.goal
P
(31)
E²
Thus, when the variance is small, the representation stays close to the generalization target with high probability.
We note that gradient space dignment--sppecinne orthogonal @!" helpful under our convexity assumptions, and that using OSGR-based preconditioning encourages high-GSNR updates that remain within the generalization zone.
Furthermore, generalization often means consistent loss across training and test-even if predictions are wrong so the region itself is inherently wide.
B. Implementation Details
B.1. Training settings
Hyperparameters. In our experiments, we configured various algorithms with specific hyperparameters:
MOON: 0.01, Temperature =
FedProx: 0.01
FedDyn: 1
FedSAM: p 1.0
Model Configuration. We employed both the ResNet-18 18 and LenNet-5 architectures for our experiments. When applying our constraints, we removed the batch normalization layer to leverage the weight normalization (WN) effect. Additionally, biases were omitted from the models our experiments, as they had only a minor effect on the overall model performance.

---

Federated Learning for Feature Generalization with Convex Constraints
Other Experimental Settings. For the training parameters, we the local learning momentum 0.9, applied weight decay of 1e-5, and used batch size of 50. The learning rate 0.01 for local training and 1.0 for global updates. All experimental evaluations were executed utilizing two Nvidia 3090 GPUs.
B.2. Data Partitioning
Datasets. Our experiments conducted using two well-known datasets: CIFAR-10 and CIFAR-100.
Data Distribution Across Clients. To simulate varying degrees data heterogeneity across clients, we used Dirichlet distributions with different Alpha values: 0.5, 0.2, and 0.05. The distribution of data across clients, under these settings, illustrated in Figure
Local Test Data. For the evaluation of test loss on client data, we partitioned the data such that 10% of each client's data was reserved as local test data. This approach ensures that the test loss reflects the performance of the model under the specific data distribution of each client.
C. Additional Experiments
C.1. Ablation Study
Impact of Constraints Learning. To understand the influence of each constraint on the learning process, we conducted an ablation study, examining accuracy graph and Hessian values.
Accuracy Improvements. indicated in Table (Accuracy), applying centralization and sphere constraints indepen- dently resulted in performance enhancements. The highest improvement was observed when both constraints were applied
together.
Orthogonal
Center
Performance (%)
Cconvez
53.12
34.83
56.77
65.90
0
51.99
32.27
0
59.66
84.58
Table Ablation study on the FedAvg algorithm assessing impact of Sphere Center optimization and their combined application performance. The experiments were conducted using the LeNet-5 model the CIFAR-10 dataset, with Dirichlet distribution of 0.5. Performance is measured terms of Top-1 accuracy (%) and Coonves is defined as Amaz/Aman)
Learning Curves and Overfitting. Observations from Figure (Learning Curves) reveal that the application of sphere constraint helps prevent overfitting, contributing more generalized local learning.
Hessian Value Analysis. Upon examining the Hessian values Table 3), we found that orthogomalization constraints tend to make the model more convex, implying better alignment among client models. centralization constraint, on the other hand, increases the speed of training the model.
Our analysis indicates that both othogonalization and centralization significantly impact performance. Specifically, orthogonalization constraint align client models effectively, while centralization constraint boosts local learning, enhancing local learning capabilities.
C.2. Hessian Values and Loss Landscape
Analysis of Hessian Values. Our observations, as detailed in Table 4, indicate an increase in the maximum eigenvalue despite the application constraints. This challenges the conventional interpretation correlating the decrease in maximum eigenvalue with improved generalization, particularly in FL contexts. However, we noted a consistent increase in convexity with the application of constraints, suggesting that convexity might be a more reliable indicator in FL environments.
12

---

Federated Learning for Feature Generalization with Convex Constraints
Ablation Experiments on Condtraints
0.00
0.57
0.40
Redrig
0.25
0.30
Bath
Figure? 7. This figure represents the Top-1 accuracy per global epoch for the FedAvg algorithm under different constraint applications using the LeNet-5 model on CIFAR-10.
w/o constraints
constraints
Algorithm
\//A5
X
\/A
X1
FedAvg
1.217
10.53
1.751
252.8
FedProx
1.268
10.45
1.698
265.2
MOON
1.252
9.51
1.718
258.5
SCAFFOLD
1.278
10.05
1.594
177.6
FedDyn 1.326
10.71
2.167
280.8
Table 4. In this analysis, we utilized the ratios /As and the maximum eigenvalue a of the Hessian matrix to assess the sharpness of the loss landscape. A lower value these metrics typically indicates a flatter loss landscape, which commonly associated with better generalization performance. Here, represents the largest eigenvalue, while As denotes the fifth largest eigenvalue of the Hessian matrix. The table above demonstrates how the application of constraints leads to a sharper loss landscape, as indicated by these metrics.
Convexity and Model Alignment. The relationship between constraints and a more convex loss landscape is evident in Figure 10 (Loss Landscape). This convexity, indicative of effective model alignment, further supported by Figure 11 (Eigen Spectral Density), which implies that constraints align the model towards more convex points.
These findings demonstrate the importance of considering convexity in the analysis of Hessian matrices in FL setting, Unlike traditional settings where the focus is often on the maximum eigenvalue generalization indicator, our results highlight the significance of convexity in understanding model alignment and performance in FL. Therefore, observing convexity in the loss landscape and Hessian matrices could offer more effective approach for analyzing and enhancing model performance in federated environments.
C.3. More analysis on Learning Dynamics
Cosine Similarity and Local Learning. The Cosine Similarity FFiuure 15) analysis reveals that the application of constraints does not hinder the variability of cosine similarity, In fact, we observe an increased change on cosine similarity, suggesting that local learning is not restricted but appropriately regulated by the constraints. This indicates a balanced approach, where constraints guide the learning process without stifling the model's ability of local learning.
GSNR and Model Generalization The GSNR analysis highlights the impact of constraints on model alignment and generalization. Figure 14(a) presents the sum of GSNR values on a client at the initial local training step of each round. A higher GSNR at this stage suggests that FedCONST effectively leverages alignment with the global model to extract more generalizable features than FedAvg. Conversely, Figure 14(b) shows the sum of GSNR values at the final local training step of each round, where client models tend to drift from the global model. The observed decrease in GSNR values in FedCONST indicates that the constraints mitigate overfitting to client-specific data. This suggests that FedCONST maintains a more stable generalization process by preventing excessive reliance on localized information while preserving the overall adaptability of the model.
13

---

Federated Learning for Feature Generalization with Convex Constraints
Conjecture Weight Magnitude and Generalization In Conjecture we proposed that the magnitude of model parameters may reflect feature generality, and that preserving high magnitude weights could promote better generalization. To support this, we present simple empirical analysis using t-SNE visualizations of global feature representations.
Specifically, we compare two variants of the global model: one where the bottom 20% of weights (by magnitude) are zeroed out, and another where the top 20% are removed.
observe that excluding the bottom 20% of weights results in more clearly clustered and semantically aligned feature representations. In contrast, removing the top 20% of weights yields less structured outputs. This supports our conjecture that small-magnitude weights contribute more noise than signal, and that weight magnitude encodes meaningful signals about feature generality.
((a)) Excluding bottom 20% of weights
((b) Excluding top 20% of weights
Figure 8. L-SNE visualization of feature representations from the global model. Removing small-magnitude weights (left) results in more clearly clustered and semantically aligned features, while removing lagge-magnitude weights (right) does not significantly improve semantic structure. This supports conjecture that weight magnitude correlates with feature generality.
14

---

Federated Learning for Feature Generalization with Convex Constraints
(()) 10 clients, CIFAR-100, a 0.2
((b)) 10 clients, CIFAR-100, 0.05
((c) 10 clients, CIFAR-10, 0.5
(()) 10 clients, CIFAR- 10, 0.2
((e) 50 clients, CIFAR- 10, 0.5
((D) 100 clients, CIFAR 10, 0.5
Figure 9. Example of data distribution according to (Client Number, Dataset, Dirichlet alpha). Each subfigure represents the data distribution under different client settings and Dirichlet values.
15

---

Federated Learning for Feature Generalization with Convex Constraints
(()) FedAvg
(b)) FedAvg with con- straints
(()) FedProx
((d) FedProx with con- straints
(()) SCAFFOLD
(()) SCAFFOLD with constraints
(g)) FedDyn
((h) FedDyn con- straints
(i) MOON
(i)) MOON with con- straints
Figure 10. This figure presents the experimental results of the loss landscape for the ResNet-18 model in cross-device setting of ??. Noise was introduced to the weight [|W]] in the form of random vector e, scaled such that the ratio ||e||/]| ||[ ranged from to 1. The results demonstrate that applying constraints leads more convex loss landscape, indicating enhanced generalization capability under these conditions.
(())) FedAvg
((b) FedAvg with straints
(()) FedProx
((d) FedProx with straints
((e) SCAFFOLD
(()) SCAFFOLD with constraints
((g)) FedDyn
((h)) FedDyn with con- straints
(i) MOON
(i)) MOON with con- straints
Figure This figure illustrates experimental results the Figen Spectral Density Hessian Matrix ResNet- model in drosse setting. Prior the application of constraints, the density of negative eigenvalues is significant, indicating the presence saddle points the loss landscape.
16

---

Federated Learning for Feature Generalization with Convex Constraints
Global
Spech Glabo
652
0.50
016-
0.59
X
0.30
h:3Pay
TuSSAM
NUESAM
038-
0.20
(()) CIFAR- 10, 0.2, local epochs 10, LeNet-5
(b)) CIFAR-I0 10, & 0.5, local epochs 5, LeNet-5
Epuch vs Global Test ACC
Spech vs Globe Test ACC
ACSENC
TodSAV
SCAPFOUS
SCAPFOUD
02.
hullyn
PulDyn
Spec1
Spect
(()) CIFAR-10, 10, 0.2, local epochs 10, ResNet-18
((d)) CIFAR- 10, 0.5, local epochs 5, ResNet-18
Epach vs
spech
0.43
0.53
0.30
200495
0.35
0.25
0.23
0.30
HISPV
IEEEA 20
010
000
ledion IeSCONS
185CONS"
0.33
20
(())) CIFAR- 100, a = 0.5, local epochs = 5, LcNet-5
(f)) CIFAR- 1000, a = 0.5, local epochs = ResNet-18
Epoch vs Global Test ACC
FuSAM
035
SCOPOOLD
032
Ezoch
(g)) CIFAR-I00. 0.05, local epochs = 5, ResNet-18
Figure 12. T accuracy per global epoch for various algorithms conducted under cross-silo settings, with specific conditions (Dataset, Dirichlet alpha, local epoch, Model). This comparison highlights the performance variations across algorithms and the impact of different environments.
17

---

Federated Learning for Feature Generalization with Convex Constraints
Epoch Auree
Epoch
redtng
Inding
FedCOMST
FedCONGT
Eaecs
Epee7
((a)) LeNet-5 with 100 clients
(b)) ResNet-18 with clients
Figure 13. Top-1 accuracy global epoch for various algorithms on CIFAR-10 under a cross-device setting, with Dirichlet alpha of 0.5 and 10% client participation per round. Figure (3(a) shows results for LeNet-5 with 100 clients, Figure 13(b) for ResNet-18 with 50 clients.
Fnck DMST
PoSCONS
Fpoch
Fpoch
(()) GSNR value Initial Local Epoch
(bb) GSNR value Final Local Epoch
Figure 14. Figure 14(a) shows the sum of GSNR values on a client at initial local training step of each round. When client model generalizable features than FedAvg. Figure 14(b) shows the of GSNR values
aligned with global model, FedCONST harvests on client at final local training step of each round. When client model drifted from glohal model, there are less generalizable common features. Decrease GSNR values FedCONST indicates less overfitting client data.
18

---

Federated Learning for Feature Generalization with Convex Constraints
Epoch vs Cosine Similanity
Epoch vs Weight Norm
1422
30
Tedlag
1650
feSrONS
2525
1875
bodbin
Fed:AM
SCOFFOLD
Fed2ym
((a)) CIFAR- 10, & 0.3, local epochs = 5, LcNet-5
((b) CIFACIFRRRUU 10, a 0.5, local epochs = 5, LeNet-5
Epoch vs Cosine Similarity
Ecoch vs Weight Norm
110
Hedrag
NOSCONS"
40
V0000
0.88
Fedtig
870
PotSMM
10-
(()) CIFAR-IO, a 0.2, Jocal epochs = LeNet-S
((d)) CIFAR- 10, 0.2, local epochs 10, LeNet-5
Epoch Similanty
Epach vs Weight Norm
03278
Inding
SCARFOLD
83900
holloy
04650
03175
Epcc7
((e)) CIFAR- R000, 0.05, local epochs R8
(()) CIFAR- 100, 0.05, local epochs 5, ResNet- 18
Figure 15. This figure displays the L2 norm of client model and the cosine similarity between a client and the global model at cach local epoch, for various algorithms implemented under cross-silo settings. The experiments were conducted with specific conditions (Dataset, Dirichlet alpha, local epoch, Model). The application of constraints consistently maintains the weight's L2 norm throughout training. In case of ResNet-13 18, which includes batch normalization layers, the weight norm is naturally consistent. Notably, a large change cosine similarity during training with constraints suggests that local learning is dynamically evolving and not overly restricted.
19


Reviews:
[{"review_id": "x8TTbKj5Be", "reviewer": "Reviewer_wAnr", "summary": "This paper proposes FedCONST, a federated learning (FL) framework, to boost generalization under heterogeneous client data distributions. Specifically, the authors adaptively modulate the magnitudes of updates based on the global model's parameter strength by applying convex constraints during client training. This prevents overemphasis on well-learned features while reinforcing underdeveloped ones. The authors further supported their approach with theoretical analyses and a series of experiments on different datasets, which validated its effectiveness in enhancing generalization.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive feedback and valuable suggestions. Below, we address each point in detail:\n\n---\n\n## Dataset diversity  \nWe appreciate your suggestion. While we agree that evaluating on larger datasets such as Tiny-ImageNet would further strengthen the experimental scope, we note that several recent works in the federated learning literature—including Wang et al. (CVPR 2024), Li et al. (ICML 2023), Qu et al. (ICML 2022), and Lee \\& Yoon (ICML 2024)—also did not include Tiny-ImageNet in their evaluations.\n\nInstead, we extended our experiments to the EMNIST dataset. Preliminary results show the improvement using FedCONST, and we will include these results in the final version of the paper.\n\n### EMNIST Accuracy Comparison\n\n| Method    | Accuracy (%) |\n|-----------|--------------|\n| FedAvg    | 83.54        |\n| FedCONST  | 85.10        |\n---\n\n## Comparison with recent FL methods  \nWe agree that comparing with more recent federated learning methods is important. While our main focus was on validating the fundamental effect of convex constraints through widely used baselines such as **FedAvg** and **FedProx**, we have additionally conducted experiments on recent algorithms, including **FedLAW**, **FedWon**, and **FedFA**, to strengthen the empirical comparison.\n\nThe results below demonstrate that our proposed method performs competitively across both cross-device and cross-silo settings:\n\n| Algorithm       | Cross-Device CIFAR-10 (α=0.5) | Cross-Device CIFAR-10 (α=0.2) | Cross-Silo CIFAR-10 (α=0.5) | Cross-Silo CIFAR-100 (α=0.5) |\n|----------------|-------------------------------|-------------------------------|------------------------------|-------------------------------|\n| FedLAW (Wang et al., 2024) | 47.22                         | 45.15                         | 52.36                        | 17.80                         |\n| FedWon (Zhuang & Lyu, 2023)        | 35.80                         | 31.30                         | 39.31                        | 2.81                          |\n| FedFA (Qu et al., 2022)           | 48.14                         | 45.61                         | 50.95                        | 20.21                         |\n| **FedCONST (ours)**         | **54.28**                         | **54.79**                         | **59.66**                        | **26.86**                         |\n\nWe will include these updated results in the final version.\n\n---\n\n## Statistical significance (p-values / confidence intervals)  \nThank you for pointing this out. We are currently computing **95% confidence intervals** based on repeated runs. Early results indicate that the performance improvements remain statistically significant. We will incorporate these confidence intervals in the final version of the results table.\n\n---\n\n## Gaussian assumption in the proof  \nIn our setting, local updates result from multiple gradient steps. By the Central Limit Theorem (CLT), their distribution tends to be Gaussian as the number of steps increases. Additionally, momentum-based optimizers accumulate gradients over time, reinforcing this tendency. We also assume the model has sufficiently many parameters per channel, allowing grouped parameters to be approximated as Gaussian. Thus, the Gaussian assumption is a practical and reasonable approximation for theoretical analysis in FL. We thank the reviewer for pointing this out and will include a more rigorous discussion on its validity in the revised version.\n\n---\n\n## Limited citations of recent work  \nWe appreciate the feedback and acknowledge that our related work section can be improved. We will revise it to incorporate more recent and relevant literature, particularly those that address generalization in FL and representation robustness.\n\n---\n## Clarification on Symbol Consistency in Equation 13\nIn Equation 13, we used the same symbol for local updates and global model parameters since the local model starts from the global one. The weight update is computed as a delta from the global model, so the notational unification was intended for simplicity. However, we admit this may cause confusion, and we will revise the notation for clarity in the final version.\n\n---\n### References  \n- Wang, Y., Fu, H., Kanagavelu, R., Wei, Q., Liu, Y., & Goh, R. S. M. (2024). *An aggregation-free federated learning for tackling data heterogeneity*. In CVPR.  \n- Li, Z., Lin, T., Shang, X., & Wu, C. (2023). *Revisiting weighted aggregation in federated learning with neural networks*. In ICML.  \n- Qu, Z., Li, X., Duan, R., Liu, Y., Tang, B., & Lu, Z. (2022). *Generalized federated learning via sharpness aware minimization*. In ICML.  \n- Lee, T., & Yoon, S. W. (2024). *Rethinking the flat minima searching in federated learning*. In ICML.  \n- Zhuang, W., & Lyu, L. (2023). *FedWon: Triumphing multi-domain federated learning without normalization*. arXiv:2306.05879.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The paper situates itself well within the federated learning and domain generalization literature. It builds on established methods such as FedAvg, FedProx, and FedSAM while addressing known issues like client drift and feature misalignment. The theoretical inspiration drawn from GSNR-based dropout and domain generalization techniques is articulated, positioning the contribution as a natural yet innovative extension to the FL paradigm.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. The paper proposes an innovative and conceptually straightforward method to tackle overfitting and misalignment in FL.\n2. It provides theoretical justification and extensive experimental validation for the strategy.\n3. The writing is well-organized, with intuitive illustrations.\n\nWeakness\n1. The experimental evaluation relies solely on the CIFAR-10 and CIFAR-100 datasets. It is recommended that the experiments be supplemented with Tiny-ImageNet, as in some related works.\n2. The baseline methods compared in the paper (e.g., FedAvg, FedProx) can validate the fundamental effectiveness of FedCONST. However, the selection of baseline models is somewhat outdated, failing to cover more recent federated learning methods, such as FedALA and FedFA.\n3. Results in Table 1 report accuracy improvements but omit p-values or confidence intervals, leaving the significance of improvements unclear.\n4. The proof assumes Gaussian-distributed weight updates (Equation 9), which may not hold in non-convex neural network optimization. Some discussion may be needed here.\n5. The paper cites relatively few works and fails to cover many of the recent advances in the field. More recent relevant works need to be incorporated.", "other_comments_or_suggestions": "No.", "questions_for_authors": "In Section 3.2.2, while the projection matrix is constructed based on global model parameters, local model parameters are directly used in the derivation of Equation 13. Why does such a symbol substitution occur?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oQMpo5SAWq", "reviewer": "Reviewer_fYb2", "summary": "This paper targets at addressing the generalization challenges in federated learning (FL). The authors propose FedCONST, an approach that adaptively adjusts update magnitudes based on the global model's parameter strength, preventing overemphasis on well-learned parameters and reinforcing underdeveloped ones. FedCONST employs linear convex constraints to maintain training stability and preserve locally learned generalization capabilities during aggregation. FedCONST aligns local and global objectives, mitigating overfitting and enhancing generalization across diverse FL environments, achieving state-of-the-art performance.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback. Your comments have helped us refine our intuition and better communicate the contributions of this work.\n\n## Justification on Conjecture\n\nWe appreciate your comment regarding the lack of direct evidence supporting *Conjecture 1*. In our paper, we proposed that the magnitude of global model parameters can serve as a proxy for feature generality, and that adjusting updates based on weight size could improve generalization. We provide further theoretical discussion in our response to Reviewer ETbn.\n\nTo offer additional empirical support, we present a t-SNE visualization of feature representations from the global model (see [here](https://imgur.com/a/additional-validation-on-conjecture-UQB05mN); fully anonymized). In this analysis, we excluded the top 20% and bottom 20% of weights by magnitude to isolate the role of mid-range weights.\n\nInterestingly, removing the bottom 20% (i.e., the smallest-magnitude weights) resulted in *clearer clustering and more semantically aligned* representations. In contrast, removing the top 20% did not produce the same clarity.\n\nThis observation supports our conjecture that small-magnitude weights may contribute more noise than signal, while mid-to-high magnitude weights are more aligned with generalizable features. This lends empirical credibility to our claim that **weight magnitude encodes meaningful signals about feature generality**, which justifies the use of convex constraints based on parameter norms to improve generalization.\n\n## Contribution of this Work\n\nThank you for the suggestion regarding terminology. We will revise terms like “Theorem 3.1” in the final version to avoid confusion. Our initial choice was to highlight the central theoretical result connecting constrained optimization with generalization.\n\nThe main contribution of our work is to offer a new perspective on Federated Learning:  \nThe magnitude of global model parameters can implicitly encode information about the data distribution—information which can be harnessed even when local data is inaccessible.\n\nBuilding upon Theorem 2.1, we demonstrate that *simple convex constraints*, applied solely to global model weights, can influence generalization across clients. This is especially relevant in federated settings, where privacy constraints prevent access to raw data.\n\nMoreover, our work reframes the problem from *what* the model learns to *how* it learns—highlighting the role of directional constraints in implicit regularization and representation control in decentralized learning.\n\nFinally, we emphasize that our method is *lightweight and architecture-agnostic*:\n- No additional loss terms  \n- No auxiliary models  \n- No heavy computation\n\nThis makes it highly practical for federated learning, where efficiency, privacy, and scalability are key concerns.", "claims_and_evidence": "The claim on line 92 that \"this paper provides theoretical and empirical analyses guarantee that the proposed method boosts generalization by imposing more updates with larger probabilities to under-learned features\" is misleading, as the theoretical results in Theorem 2.1 appear to be directly adopted from existing research.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes, there are no outstanding issues in the designs and analyses of this paper.", "relation_to_broader_scientific_literature": "The essential motivation behind the proposed method is discussed in Conjecture 1, which, however, is not verified by either theoretical or empirical evidence in this paper. As a result, the contributions of this work to the relevant community remain unclear.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. Extensive experiments on multiple benchmark datasets, including CIFAR-100, demonstrate the superior performance of the proposed method in diverse scenarios.\n\n**Weaknesses:**\n\n1. The essential motivation behind the proposed method is discussed in Conjecture 1, which, however, is not verified by either theoretical or empirical evidence in this paper. The soundness of this work would be improved if more evidence were provided to demonstrate the correctness of the claim in Conjecture 1.\n\n2. The contributions of this work to the federated learning community are unclear. It appears that the theoretical results in Theorem 2.1 are directly adopted from an existing research paper. If this is the case, it would be more appropriate to title it as a Proposition rather than a Theorem.\n\n3. The soundness of the evaluation section can be improved. For example, more datasets (e.g., EMNIST, ImageNet) should be considered to enhance the evaluation.", "other_comments_or_suggestions": "Please refer to the weaknesses listed in the previous section.", "questions_for_authors": "Please refer to the weaknesses listed in the previous section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qe462JHQqR", "reviewer": "Reviewer_9QZ9", "summary": "This paper introduces FedCONST, a novel federated learning (FL) algorithm that addresses the challenges of generalization and overfitting in FL environments with heterogeneous data. By employing convex constraints based on the global model's parameter strengths, FedCONST adaptively modulates update magnitudes to prevent overemphasis on well-learned parameters while reinforcing underdeveloped ones. This approach not only stabilizes local training but also enhances feature transferability and robustness across diverse FL settings. The authors validate their method through extensive experiments on various datasets and model architectures, demonstrating state-of-the-art performance compared to existing FL techniques.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewers for their constructive feedback and thoughtful suggestions.\n\n## Theoretical Overhead:  \nThe proposed constraints—centralization and orthogonal projection—are implemented using simple linear operations with negligible computational overhead. Specifically, for a gradient vector of dimension $n$, each constraint introduces only $\\mathcal{O}(n)$ additional computation per update during local training. No additional backpropagation steps or network modules are required. As a result, our method maintains computational efficiency and scalability within the federated learning setting.\n\n## Practical Convergence Speed:  \nAs shown in Supplementary B.3, we provide convergence plots (test accuracy vs. global rounds) under various experimental settings. These results demonstrate that models trained with our constraint-based approach not only achieve better generalization but also converge more rapidly in terms of test accuracy. This indicates that the added constraints enhance both the learning dynamics and the final performance.", "claims_and_evidence": "Yes. The author's claims regarding theoretical and experimental contributions are well-supported by concrete content.", "methods_and_evaluation_criteria": "The proposed method, FedCONST , draws inspiration from Domain Generalization (DG) by leveraging Gradient Signal-to-Noise Ratio (GSNR) insights and applying convex constraints to enhance feature generalization in Federated Learning (FL). This approach is particularly effective in cross-silo settings, where it demonstrates significant improvements in generalization. However, the authors should provide a more detailed analysis of the computational overhead introduced by the method. Specifically:\n\n- Theoretical Overhead : Discuss the additional computational cost of applying centralization and orthogonality constraints during local training.\n- Practical Convergence Speed : Include convergence curves (e.g., test accuracy vs. global rounds) to assess whether the added complexity translates into faster or more robust convergence.", "theoretical_claims": "While the paper introduces several formulaic expressions and provides a theoretical foundation for the proposed method, it lacks rigorous proofs for some of its core claims, particularly regarding convergence and generalization .", "experimental_designs_or_analyses": "The experimental results effectively demonstrate the benefits of FedCONST in enhancing generalization and stability. However, in Figure 14(e), the large fluctuations in the L2 norm and cosine similarity for FedCONST toward the end of training raise concerns about potential instability.", "relation_to_broader_scientific_literature": "FedCONST adapts the concept of GSNR from DG to FL, leveraging global weight magnitudes as a proxy for feature importance. This innovation aligns with recent efforts in model optimization research to enhance generalization under data heterogeneity, offering a computationally efficient and scalable solution tailored to FL's distributed nature. By stabilizing local training and preserving generalizable features during aggregation, FedCONST addresses key limitations of existing methods, such as overfitting and misalignment. These contributions inspire further exploration into constraint-based optimization strategies, particularly in scenarios involving sparse or imbalanced data distributions, and position the work as a meaningful advancement in both FL and general machine learning optimization paradigms.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "It is recommended to carefully review the figures and equations for potential typos. For example, in Figure 2 , the \"Client - Side\" section should likely be labeled as Client 1, Client 2, ..., Client N instead of the current notation, which appears to be inconsistent with standard representation.", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4L5lZ8CoFX", "reviewer": "Reviewer_ETbn", "summary": "For generalization in FL framework, this paper proposes FedCONST which impose more updates with larger probabilities to under-learned features by centralization and orthogonal constraints at clients. Various FL algorithms could get performance improvement with the proposed constraints.\n\n## update after rebuttal\nThank you for your rebuttal. I have reviewed the authors’ rebuttals to all the reviews. I think most of the concerns are resolved in the rebuttal. Therefore, I want to increase my score to 3. Weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Generalization Area\n\nWe appreciate your insightful concern regarding the relationship between the convex constraint region and the generalization area. You raised an important point: if the generalization area is narrower than the constraint region, then the aggregated model could potentially drift away from the generalizable zone. In response, we offer an intuitive argument suggesting that the generalization area can, in fact, be sufficiently large.\n\nConsider the $l$-th layer representation of a neural network:  \n$$\n\\Phi^l_c = \\sigma((\\Phi^{l-1})^\\top W^l_c)\n$$\n\nwhere $\\sigma$ is the activation function (e.g., $\\tanh$), and $\\Phi^l_c$ denotes a representation within the generalization regime.  \nWe ask: under what conditions does  \n$$\n\\sigma((\\Phi^{l-1})^\\top (W^l_c + \\Delta W^l_c)) \\approx \\Phi^l_c\n$$  \nstill hold?\n\n### Case 1: Large $\\|W^l_c\\|$ (saturation regime)\n\nIf the norm of $W^l_c + \\Delta W^l_c$ is large, then the activation function saturates.  \nIn this case, $\\sigma$ becomes relatively insensitive to $\\Delta W^l_c$, so a wide range of perturbations still yield generalizable representations.\n\n### Case 2: Small $\\|W^l_c\\|$ (linear regime)\n\nWhen weights are small, $\\sigma$ behaves almost linearly:  \n$$\n\\Phi^l_c \\approx (\\Phi^{l-1})^\\top W^l_c\n$$\n\nWe can apply Chebyshev’s inequality:  \n$$\nP(|\\Phi^l_c - \\Phi^l_{c,\\text{gen}}| \\geq \\epsilon) \\leq \\frac{\\text{Var}(\\Phi^l_c)}{\\epsilon^2}\n$$\n\nHere, the variance is small, so the output remains close to generalization with high probability.\n\nThese cases show that the perturbation space $\\Delta W^l_c$ that preserves generalization is often broad, especially when orthogonal to $W^l_c$—a behavior encouraged by our convex constraints.\n\nFurthermore, generalization often means consistent loss across training and test—even if predictions are wrong—so the region itself is inherently wide.  \nOur formulation characterizes the generalization region by high GSNR parameters, which span a broad zone, especially early in training (Liu et al., ICLR 2020).\n\nBecause our convex constraints encourage high-GSNR directions, they not only prevent drift but also promote convergence within the generalization zone.\n\n---\n\n## Implementation on Constraints\n\nWe appreciate the reviewer’s observation. Indeed, when both constraints are affine (e.g., linear equalities), it is possible to compute a closed-form projection directly onto their intersection:\n\n$$\n\\mathbf{x}^* = \\mathbf{v} - A^\\top (A A^\\top)^{-1} (A \\mathbf{v} - \\mathbf{b})\n$$\n\nWhere $A$ stacks the constraint vectors, and $\\mathbf{b}$ contains target values.\n\nHowever, our implementation uses two-step sequential projection for simplicity and modularity:\n\n- Centering constraint:  \n  $$\n  C(g_c^l) = g_c^l - \\frac{1}{n_c^l} \\mathbf{1}^\\top g_c^l\n  $$\n\n- Projection constraint:  \n  $$\n  P_{G^l_c}(g_c^l) = (I - G^l_c G^{l\\top}_c) g_c^l\n  $$\n\nThis modular approach keeps each step interpretable and separable.  \nNonetheless, we agree a single-step version might be more efficient and plan to explore this in the future.\n\n---\n\n## Transformer Architecture\n\nWhile we didn’t test on Transformers, our method is architecture-agnostic.\n\nOur constraints apply at the weight update level. In Transformers, this means we can project updates in attention matrices ($W^Q$, $W^K$, $W^V$) and feedforward layers ($W_1$, $W_2$), just as with CNNs.\n\nBecause the projection is direction-aware and lightweight, it can be plugged into Transformer training pipelines without architectural changes.  \nWe plan to validate this in future work.\n\n### Reference\n- Liu et al., \"Understanding Why Neural Networks Generalize Well Through GSNR\", ICLR 2020", "claims_and_evidence": "Positive correlation between weight magnitude and GSNR, gradient variance reduction, drift diversity are supported by experiments.", "methods_and_evaluation_criteria": "Evaluations of using various FL methods with the additional proposed convex constraints for cross-device and cross-silo conditions makes sense.", "theoretical_claims": "Theoretical analysis mainly based on “Understanding Why Neural Networks Generalize Well Through GSNR of Parameters, ICLR’20”", "experimental_designs_or_analyses": "Experiments are somewhat small-scale. Federated domain generalization benchmark datasets such as PACS and OfficeHome would be better to show the effectiveness of the generalization.", "relation_to_broader_scientific_literature": "This paper proposes a federated domain generalization method by introducing convex constraints.", "essential_references_not_discussed": "Most of the related papers are well cited.", "other_strengths_and_weaknesses": "- Strengths \n  - It is an interesting idea to include convex constraint to the local model updates by considering the aggregated global model. \n  - Impressive performance improvement can be observed in Table 1.\n- Weaknesses\n  - Due to the centralization constraint, the aggregated weight exists within the convex constraint area. However, for the aggregated model to exist within the generation area, the convex constraint area should be within the generation area. I guess that the generalization area is generally smaller than the convex constraint area. \n  - Some missing details and typos make it difficult to understand. E.g. formulations of center gradient and project gradient, ‘i' is used for sample or channel. \n  - The experiments were conducted on small-scale datasets with limited model architectures (mainly conv networks). It would be better to evaluate the Conjecture 1 (Global weight magnitude is a reliably proxy for feature strength) with various network architectures.", "other_comments_or_suggestions": "$w_i^l$ might be $w_c^l$ below Eq. (8)", "questions_for_authors": "1) I am not sure why the convex constraints guarantee the aggregated global model belongs to the generalization area as shown in Figure 1. Could you explain it with more intuitive examples?\n2) In Algorithm 1, what are the exact formulations of the Center gradient $C(g^k_{m,t})$ and Project gradient $P(g^k_{m,t})$. And why are center gradient and project gradient calculated serially?\n3) Is the proposed method applicable to transformer architectures?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach using convex constraints to improve generalization in FL.
The approach depends on the conjecture that large weights indicate well-learned (strong) features while small weights signify weaker features requiring additional training.
The approach is orthogonal to many FL algorithms and can be applied to a variety of algorithms and the empirical results are significant.

While there was some concern about missing datasets or baseline methods, the paper seems to make a meaningful contribution that could apply more broadly and added onto prior approaches. Thus, the potential for impact is higher. In addition, the approach is simple and scalable.

The main suggestion is to increase the explanation and evidence for conjecture 1 and how this overall drives the paper. Also, more datasets and baselines would improve the evidence for the paper's contributions.
Decision: Accept (poster)