NormFit: A Lightweight Solution for Few-Shot Federated Learning with Non-IID Data
Authors: Azadeh Motamedi
Jae-Mo Kang
Il-Min Kim
Venue: NeurIPS 2025 spotlight ()
Keywords: Federated Learning
Vision-Language Models
Few-Shot Fine-Tuning
Data Heterogeneity
Communication and Computation Cost
Primary area: deep_learning
TL;DR: NormFit is a novel fine-tuning framework designed specifically for few-shot federated learning scenarios characterized by heterogeneous and imbalanced data distributions

Abstract: Vision–Language Models (VLMs) have recently attracted considerable attention in Federated Learning (FL) due to their strong and robust performance. In particular, few-shot adaptation with pre-trained VLMs like CLIP enhances the performance of downstream tasks. However, existing methods still suffer from substantial communication overhead, high local computational demands, and suboptimal performance under non-IID user data. To simultaneously address all those limitations, we propose NormFit, a lightweight solution that selectively fine-tunes only a very small portion of the model parameters, specifically only the Pre-LayerNorm parameters of the vision encoder within a VLM. Overcoming the existing tradeoff between performance and communication/computation efficiency in few-shot FL, NormFit sets a new benchmark by simultaneously achieving superior accuracy and substantially reduced communication and computational demands. Theoretically, we show that NormFit yields a considerably smaller generalization gap compared to tuning all LayerNorm parameters. 
Importantly, NormFit can function effectively as a standalone solution or integrate seamlessly with existing few-shot fine-tuning methods to further enhance their performance. Notably, NormFit offers implementation simplicity, achieving these improvements without any algorithmic modifications, changes to the underlying model architecture, or the addition of external parameters.

NormFit: A Lightweight Solution for Few-Shot Federated Learning with Non-IID Data
Azadeh Motamedi'
Jae-Mo Kang
II-iin Kim Queen's University Kingston, Canada
Queen's 's University
Kyungpook National University
Kingston, Canada
Daegu, South Korea jmkang@knu.c
A. wwM.Motmd@@uueeeteeee @queesu..cu
aw..auaaccc..... ac. kr
ilmin. kim@queensu.ca
Abstract
Vision-Language Models (VLMs) have recently attracted considerable attention in Federated Learning (FL) due to their strong and robust performance. In particular, few-shot adaptation with pre-trained VLMs like CLIP enhances the performance downstream tasks. However, existing methods still suffer from substantial communication overhead, high local computational demands, and suboptimal per- formance under non-llD user data. To simultaneously address all those limitations, we propose NormFit, lightweight solution that selectively fine-tunes only very small portion of the model parameters, specifically only the Pre-LayerNorm pa- rameters of the vision encoder within VLM. Overcoming the existing tradeoff efficiency few-shot FL,
between performance and communication/compuaaiii NormFit a benchmark by simultaneously achieving superior accuracy and substantially reduced communication and computational demands. Theoret- ically, we show that NormFit yields a considerably smaller generalization gap compared to tuning all LayerNorm parameters. Importantly, NormFit can function effectively standalone solution or integrate seamlessly with existing few-shot fine-tuning methods to further enhance their performance. Notably, NormFit offers implementation simplicity, achieving these improvements without any algorithmic modifications, changes to the underlying model architecture, or the addition of external parameters.
Introduction
Federated Learning (FL) has emerged as promising paradigm for decentralized machine learning. allowing multiple
collaboratively train shared model without sharing data. This approach preserves user data privacy, making it suitable in sensitive domains such as healthcare and finance. However, FL faces several challenges that limit its scalability and effectiveness. As the model size expands, the communication overhead between the central server and users increases significantly. Additionally, the computational demands the user side substantially grow with larger models and more sophisticated training algorithms. Furthermore, the inherent nature of hon-independent and identically distributed (non-III) users data continues adversely impact model performance.
Vision-Language Models (VLMs), such as Contrastive Language-Image Pretraining (CLIP) [Radford al., 2021], pre-trained large-scale image-text pairs, have recently attracted significant attention in the FL community for their strong zero-shot performance. Pre-trained VLMs are advantageous
Corresponning author
Thhe code is available httts:: wttp:tthw..comm/aaataa/////////
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Method
params (K)
Truinable
cost (KB)
Comm
Comp. cost
(GFLOP))
Codp cocoop
e FeeDAT
TEP TEP-Adapter a FeeTPG FLORA
@ FescLiP * Noraiis
74
Full fine-tuning
1.3 10
4.1 x 10
8.4 10
Accuraes
Standard VLM
CoOp
2.5 10
6.8 x 10
2.3 10a
78
few-shot methods
CoCoOp
8.9 x 10
1,4
2.4 x 10
2.9
2.9 x 10
adopted in FL.
TIP-Adapter-F
F.9
2.2 x 10
3.1 x 10
86
FedCLIP
5.3 10
1.5 x 10
1.0 109
60
FL VLM
FedTPG
3.2 10
8.7 x 10"
3.3 10"
64
few-shot
methods
FodDAT
7.5 x 10
2.1 x 10
8.5 x 10
62
18:
10'
10'
10
Communication Burden (KB) per User
FLoRA
2.3 10
6.8 x 10
2.8 10
(b)
NormFit (ours)
1.5
4.1
8.5x
74
Average Accurasy
72
(a)
78
Figure 1: (a) Number of trainable parameters (K) per user, communication burden (KB) between the server and each user, and computation cost (GFLOPs) per user in few-shot FL. Averaged accuracy in few-shot FL (over the 10 datasets in Table 1) versus (b) communication burden per user and (c) computation cost per user.
68
60
157
18
Computation Cost (GFLOPS) per User
(c)
particularly for FL, because the pre-trained knowledge embedded in VLMs can serve strong initialization, reducing the number of communication rounds and computation cost required for convergence in FL. while providing robust performance on different tasks. Yet, in practice, effectively adapting VLMs to specific downstream tasks typically necessitates few-shot fine-tuning, For VLMs, few-shot learning commonly achieved either by introducing additional trainable parameters [Zhou et al., 2022b,a, Gao et al.. 2024, Zhang et al., 2021, Zanella and Ben Ayed, 2024] or by selectively fine-tuning a subset of existing model parameters [Zaken et al., 2021, Li et al., 2024].
Recently, few-shot fine-tuning for VLMs has also been explored in the context of FL. FedCLIP [Lu et al., 2023] applied adapters the deeper layers VLMs, which limits the adaptability of early-layer ceatures-potentiall problematic when local data is significantly non-IID, FedTPG [Qiu et al., 2024] adopted personalized prompt tuning for non-IID data across users, aiming to reduce communication overhead and user drift. FedDAT [Chen et al., 2024] fine-tuned small modules inserted into a frozen VLM, adapting models to local distributions while reducing communication costs. FLORA [Nguyen et al., 2024] introduced small trainable residual adapters in VLMs.
Although existing methods have made meaningful progress in the area, they continue to face two major limitations in non-IID few-shot FL. First, as illustrated in Figure 1(a), these methods still require training a large number of parameters, resulting substantial communication overhead between users and the server. Particularly, these methods incur high local computation costs, typically involving many local training iterations for convergence, which poses critical challenge for users with constrained computational resources. This suggests that the parameters selected for training do not converge quickly in non- few-shot FL. Second, although these methods have improved performance in few-shot FL. their effectiveness under non-III conditions remains suboptimal, as illustrated in Figures 1(b) and (c). This unsatisfactory performance primarily due to those classes with fewer samples, which are inherently more challenging to learn.
To simultaneously address these challenges, we propose a lightweight few-shot FL solution named NormFüt, which fine-tunes only the Pre-LayerNorm parameters of the vision encoder within a VLM (in this study, CLIP adopted as a representative example). NormFit offers two key advantages. First, by training only a minimal number of parameters, it significantly reduces communication overhead between users and the central server, as illustrated in Figure ((a). Particularly, training of the selected parameters (i.e., the Pre-LayerNorm parameters) converges rapidly in non-III few-shot FL, resulting substantially lower local computational costs a critical benefit for resource- coouuce-constrained users. Second, fine-tuning the Pre-LayerNorm of the vision encoder achieves state-of -atte-of-the-ar (SOTA) accuracy in non-lII few-shot FL. By adjusting these early-stage parameters, the model effectively adapts to the skewed class distributions that commonly arise in non-IlI few-shot settings. As shown in Figures (bb) and (c), this targeted approach not only achieves superior accuracy but also substantially reduces communication costs and computational burdens.

---

Two additional major advantages of NormFit are its versatility and implementation simplicity. Norm- Fit can be employed either as a standalone efficient few-shot solution it can be seamlessly integrated as an enhancement to existing few-shot learning methods. Also, NormFit achieves all the improve- ments without requiring any algorithmic modifications, changes to the underlying model architecture, or additional external parameters. In this paper, we also provide theoretical justification supporting the effectiveness of NormFit. Our contributions are summarized as follows:
A Novel, Effective, and Efficient Tinnin Strategy: We propose NirmFit, new fine-tuning method that selectively updates only the Pre-LayerNorm parameters of the vision encoder in a VLM. NormFit overcomes the existing tradeoff between accuracy and efficiency non-IID few-shot FL, establishing new benchmark.
Theoretical Justification: We theoretically demonstrate that NormFit leads to smaller general- ization gap, explaining why NormFit performs robustly in few-shot non-IID FL scenarios.
Comprehensive Validation of Superior Performance: Through extensive experiments on diverse datasets, we empirically demonstrate that NormFit achieves SOTA accuracy while significantly reducing communication and computational costs compared to existing methods.
Flexible Integration and Versatility: We empirically show that NormFit can serve either as an effective standalone method or as a complementary add-on that further improves the performance of existing few-shot fine-tuning techniques without requiring additional modifications.
Implementation Simplicity: All improvements introduced by NormFit require algorithmic modifications, changes to the underlying model architecture, or additional external parameters.
2
Related Works
In this section, we review the existing methods designed to address non-III data in few-shot FL..
2.1 Adapter Tuning in Few-Shot FL.
CLIP- [Gao et al., 2021] and Tip-Adapter [Zhang et al., 2021] are two prominent adapter tuning methods in the centralized setting. CLIp--dapter proposed lightweight bottleneck archi- tecture to leverage the knowledge stored in CLIP and the newly acquired knowledge from few-shot samples. Instead of the adapter. Tip-Adapter constructed a query-key cache model to derive the adapter's weights, achieving better performance and lower computational complexity compared to CLIP-Adapter. In the FL setting, FedCLIP [Lu et al., 2023] introduced an attention-based adapter for CLIP in FL, enabling efficient personalization and generalization across users. FLORA INguyen et al., 2024], inspired by Low-Rank Adaptation (LoRA) [Zanella and Ben Ayed, 2024], focused on updating small subset of the model's parameters in FL, specifically by adding trainable low-rank adaptation modules to certain layers of a pre-trained model. While adapter tuning performs well in centralized learning, often struggles to effectively address the issue of non-III data in FL.
2.2 Prompt Tuning in Few-Shot FL
Prompt tuning typically optimizes task-related textual tokens in order to embed task -ask-fpcc knowl- edge for prediction. The hand-crafted template "a photo of [Class]" in CLIP is used model the class embedding for zero-shot prediction. Context Optimization (CoOp) [Zhou et al., 2022b) and Conditional Context Optimization (CoCoOp) [Zhou et al., 2022a], the well-known prompt tuning methods in centralized learning, replaced the hand-crafted prompts with a set of learnable prompts inferred by the labeled few-shot samples. pFedPrompt [Guo al., 2023] and FedPrompt [Zhao et al., 2023] explored prompt tuning in FL, aiming to create personalized models while addressing the issue of non-IID data. FedTPG [Qiu al., 2024] proposed task-specific prompt generation method to enhance the relevance of the generated prompts to each user's data, improving the performance on personalized tasks. FedAPT [Su et al., 2024] extends the idea of CoCoOp to FL, emphasizing -opaiii-pecific adaptation across users. Despite being promising, prompt tuning is often less effec- tive in adapting to non-IID data as it inherently relies on degree of consistency in the underlying distributions [Zhou et al., 2022b,a]. In federated prompt tuning, there is risk that prompts may overfit to a user's data, reducing their ability to generalize well across users. Also, prompt tuning requires frequent updates to parameters leading increased communication cost.
3

---

Lona
Original
All VisionyText
All Vision LayerNorms
All Text
Text Pre-
Vision Pre- LayerNorm
Image
Layortiorms
LayerNorrms
Layerhoomm
Feed Forward
Feed Forward
Layerform
Lavenform
+
LayerNorm
LayerNorm
+
(b)
LayerNomm
Pre-LaverNomm
Word Embedding B
Patch Embedding S
Figure 3: Grad-CAM visualizations illustrating the effect of different LayerNorm one-ttnnng strategies in CLIP. Each column corresponds to a specific tuning configuration: all LayerNorms of both vision and text encoders, all vision Lay- erNorms, all text I LayerNorms, text Pre-LayerNNrm only, and vision Pre-LayerNorm only (NormFit). Rows (a) and (b) correspond to the majority and minority classes, respec- tively. Fine-tunii tune-tuning only the vision Pre-LayerNorm yields sharper and more semantically aligned attention on both classes, notably better focus on the minority class.
photo a (cless
datay
sunflower
Figure 2: NormFit fine-tunes only the Pre-LayerNorm of CLIP's vision encoder to adapt feature statistics at the input stage. Snowflake and flame icons denote frozen parameters and fine-tuned parameters, respectively.
3 Proposed Method: NormFit
NormFit is lightweight fine-tuning strategy for VLMs (in this study, CLIP is adopted representa- tive example) in few-shot FL. As depicted in Figure 2, its core idea is to adapt only the Pre-LayerNorm parameters of the CLIP vision encoder while freezing all other weights the remainder of the vi- sion encoder and the entire text encoder. Fine-tuning utilizes a few labeled images per class and textual prompts of the form "a photo of [class]". During each FL round, users locally perform back- pack-propagttion on those minimal parameters (i.e., Pre-LayerNorm) and transmit these updates the central server. The server aggregates these parameters and broadcasts the aggregated values back to users. Updating the minimal parameters ensures very low communication, while preserving the rich prior knowledge captured during CLIP's pre-training. Moreover, as the Pre-LayerNorm resides at the earliest stage of the model, adapting its parameters facilitates a rapid and effective aligmment of the model's feature representations with the local data distribution, decreasing local computational costs. Furthermore, fine-tuning the Pre-LayerNorm enhances the representation quality for underrepresenttd classes and, consequently, overall accuracy under non-lID conditions.
In the following, we provide detailed discussions explaining why fine-tuning only the Pre-LayerNorm parameters of the vision encoder is effective, followed by mathematical justification support these discussions. In Section 4, we will confirm the discussions and analysis through extensive experiments.
3.1 Which to Fine-Tune: Vision Encoder or Text Encoder?
Our answer to this question is the vision encoder because it is more sensitive to distribution shift. Visual inputs often exhibit greater variability than their textual counterparts due domain-specinn differences in lighting, background, scale, etc. This makes the visual features more prone to domain misalignment. Previous work shows that even lightweight adaptations meaningfully recalibrate the internal statistics of the visual backbone. For example, the authors in [Fahes et al., 2024] demonstrate that fine-tuning the visual projection head outperforms prompt tuning, suggesting that the vision encoder is the primary driver of adaptation quality in few-shot image classification.
By contrast, the text encoder is inherently more robust. CLIP text encoder captures semantically rich, stable features that generalize across domains [Radford et al., 2021]. As noted in CoOp [Zhou et al., 2022b], modifyyng the text side yields only modest improvements, and performance is relatively insensitive to the capacity of the text encoder. Prompt tuning often leverages this stability by keeping the text encoder frozen and instead learning task-adaptive prompts [Zhou al., 2022a]. While this avoids catastrophic forgetting. it limits flexibility under severe non-IID scenarios.
These findings suggest consistent trend: when constrained to fine-tune a small subset of parameters, allocating those to the vision encoder yields more substantial gains than adapting the text encoder. Related work spans multiple settings, including image classification [Fahes et al., 2024, Zhou et al.,

---

2022b] and federated settings [Lu et al., 2023]. Text encoder tuning can be useful in some settings but often requires vision-side updates to reach peak performance.
3.2 Why Fine-Tune Only the Pre-LayerNorm?
There are two primary reasons. First, the Pre-LayerNorm contains a very small number of parameters; updating only these parameters helps preserve the rich prior knowledge embedded in the pretrained model. addition, because the Pre-LayerNorm is located at the earliest stage of the model, adapting its parameters rapidly aligns the entire model's feature representations with local data distributions. In the context of FL. these minimal and earliest-stage updates significantly reduce communication and computation costs each user, which is particularly beneficial in resource- constrained environments.
Second, fine-tuning the Pre-LayerNorm effective for improving the representational quality, es- pecially for the classes with fewer samples (namely, minority classes), which are inherently more challenging to learn. Grad-CAM visualizations in Figure 3 show the impact of various LayerNorm fine-tuning strategies applied to the CLIP model in a Fl. setting with 10 users. The columns represent different tuning configurations: fine-tuning all LayerNorms of both vision and text encoders, all LayerNorms of vision encoder, all LayerNorms of text encoder, text encoder's ' Pre-LayerNorm only, and finally, vision encoder's Pre-LayerNorm only, which is NormFit. Row (a) displays an example from majority class (having an average of 10 samples per user), and row (b) provides an example of a minority class (having an average of samples per user). Fine-tuning only the vision encoder's Pre- Pre-LayyrNomm (NormFit) generates sharper, more precise attention maps for both majority and minority classes. The benefit of NormFit is particularly notable for the minority class with an improvement in the clarity and relevance of highlighted regions. This suggests that selectively tuning those earliest-stage parameters in the vision encoder can effectively mitigate biases from class imbalance, enhancing model robustness and predictive accuracy in few-shot non-IID scenarios.
3.3 Theoretical Justification
We analytically substantiate why NormFit, which only fine-tunes the Pre-LayerNorm, provides excel- lent generalization performance. Specifically, we analyze the generalization gap, which quantifies the discrepancy between the model's performance on training samples (empirical risk) and its expected performance unseen data (true risk). A smaller generalization gap signifies that the model has captured meaningful patterns rather than simply memorizing training instances, thereby enhancing its robustness in diverse scenarios. We formally define this concept and rigorously analyze it within the FL. framework introduced in [Mohri et al., 2019] in the following.
Definition 1 (Generalization Gap e in FL) [Mohri et al., 2019]:
Let f(((z), y) denote loss function evaluated if each user k has its own distribution Dx, then for any weight vector X (Ai,..., AM) in the probability simplex, the target distribution is defined as Drx = =E N₂DR and the true risk of ((((())
input-output pair (I, In FL with M users,
M
hypothesis h is given by LDD = E(a.)))D [e(h(z), v)l. Similarly, each user k has its own
empirical distribution (based on its local data), denoted by Dr. the empirical risk on the mixture is LD, (h) = E (e [6(h(I), ((E,))) y]], where Dx= EkE Dr. The generalization gap is defined as:
E(h) == sup XEΛ
(t)
In the following theorems, we present a foundational analysis comparing the impact of fine-tuning only the Pre-LayerNorm versus fine-tuning all LayerNorms on the model's generalization performance.
Theorem 1 (Generalization Gap Ratio Eat/Epre in FL: Pre-LayerNorm All LayerNorms Fine- Tine-Tuning within the Vision Encoder):
Let Ere and Eau denote the generalization gaps under the same FL selup as in [Mohri el al., 2019] when fine-tuning (i) only the Pre Layerooo and (ūi) all LayerNorms, respectively, for a Vision Transformer (ViT) with Pre-LayerNorm and N transformer layers each containing two LayerNorms
5

---

with embedding dimension d. Then the ratio of the generalization gaps is bounded as follows:
< (2N (2N++ By (1+ log(2N 1) log(2d)
Call
Epre
Proof. The proof is provided in Appendix A.
If we consider VIT-B/32 (base vision encoder model in the standard CLIP), we have N = 12 and 768. Then the bound becomes:
Fall
29.98.
Epre
This suggests that, for the standard CLIP, the generalization gap when fine-tuning LayerNorms could be up to about 30 times larger than when fine-tuning a single LayerNorm, all else being equal. This large multiplicative factor provides clear theoretical rationale for why NormFit achieves robust performance with minimal parameter update.
Complementing Theorem 1, we present Theorem 2, which analytically demonstrates that the optimal choice is the Pre-LayerNorm, rather than any deeper alternative in the network. This result highlights the unique influence of the Pre-LayerNorm within the network. To formalize this locational advantage, we leverage Lipschitz constants, which are an essential tool in deep neural network analysis.
Definition 2 (Lipschitz Continuous Function) [Boyd and Vandenberghe, 2004]: A function f: IR" + R is defined as Lipschitz continuous if there exists a non-negative constant L such that for any two inputs 1 D2, the following inequality holds:
(1(x1) - [xx2)] ((xz)) < L||a1 x2ll-
(4)
The smallest such L is the Lipschitz constant of f.
In the following theorem, provide the gradient of the global Lipschitz bound with respect the parameters of any arbitrary LayerNorm (including the Pre-LayerNorm as a special case).
Theorem (Influence of Fine-tuning a LayerNorm):
Let fe be ViT of N layers, f fN o...o fI, where & represents the set of trainable parameters of all LayerNorms in the VIT. All other parts, including attention blocks and MLPs are frozen. Let on its global Lipschitz constant, where Le(O) is the Lipschitz constant of layer Let O₂ denote the set of trainable parameters of the k-th LayerNorm (in layer k). The gradient of the global Lipschitz bound with respect to O2 is given by:
Leghtbal = I L((O) be upper bound
Vo,L DIT + CITk:
(5)
where the Direct Influence Term, DIT,, is given by
DIT = (Ve, Ink) AEs
(6)
with amplification factor
A-1
N
As
Li
Lg
(7)
and the Cascading Influence Term, CITk, is given by
CITy = Σ (VenL))
II Li
(8)
1=
Proof.
The proof is provided in Appendix
Theorem 2 reveals the unique locational influence of Pre-LayerNon First, in the Direct Influence Term, we have A As, which means that parameters of the first LayerNorm possess uniquely amplified influence the global Lipschitz bound, positioning them at point of maximum leverage

---

for model regularization. To see why A₁ > As holds, note that the amplification factor for k = is A1 =II Lj, while for > 1 it becomes A& Iī L, - (L1/Ls))a Clearly, A1 > A& holds if and only if Li < La. Pre-LayerNorm at k receives input directly from the patch embeddings, which have a well-controleed, low-variance distribution. layer k > 1 receives hidden states that have been progressively amplified by the preceding k -1 transformer blocks. These hidden states exhibit higher variance due to the accumulation of nonlinear transformations and residual connections. To handle this increased variance, deeper layers must be more expansive, leading to larger effective Lipschitz constants; that is, L1 < L
contrast, a LayerNorm at any deeper
Second, C is also different. For the Pre-LayeNorm (k 1), CIT, is sum of terms describing how changes in O propagate to affect the Lipschitz constants of all (N 1) subsequent layers, 2, ...... N. In contrast, for any deeper layer k > 1, CIT₂ is sum of terms describing how changes in On propagate to affect the Lipschitz constants of only the (N - k) subsequent layers, + I,.... N.
Intuitively, changes to the Pre-LayerNorm parameters (k 1) have direct impact that is amplified by the entire depth of the network, alongside cascading effect that ripples through every subsequent layer. This provides a powerful, global control. In contrast, changes to deeper LayerNorm parameters (k > 1) have direct impact that amplified by a structurally different, and smaller, product of constants. Also, the cascading effect is more localized, influencing only the remaining deeper layers.
4 Experiments
4.1 Simulation Setting
Datasets: We present our results on a variety of datasets, including CIFARIO [Krizhevsky et al., 2009], ImageNet [Deng et al., 2009], Caltech101 [Fei-Fei et al., 2004], OxfordPet [Parkhi et al., 2012], Cars [Krause et al., 2013], Flowers102 [Nilsback and Zisserman, 2008], Food101 [Bossard et al., 2014], and FGVCAircraft [Maji et al., 2013] for classification; SUN397 [Xiao et al., 2010] for scene recognition; and DTD [Cimpoi et al., 2014] for texture classification.
Baselines: As the most representative VLM few-shot methods, we adopted CoOp [Zhou et al., 2022b] and CoCoOp [Zhou al., 2022a] baselines for prompt tuning; and TIP-Adapter [Zhang et al., 2021] ] as a baseline for adapter tuning in FI. (Table 1). Additionally, we compare our approach against SOTA few-shot FL methods for VLMs, including FedTPG [Qiu et al., 2024] as a baseline for prompt tuning; and FedCLIP [Lu et al., 2023], FedDAT [Chen et al., 2024], and FLoRA [Nguyen al., 2024] as baselines for adapter tuning (e.g., in Table 2).
Evaluation metrics: Each user evaluates the model using the global test set, as our primary goal is to develop global model in FL rather than personalized one. The evaluation emphasizes the spsk-specific performance metric, top-1 accuracy. Furthermore, we assess the communication efficiency (between the server and each user) and the computation cost (per user).
We consider the FI. setting of 100 users, each having CLIP ViT-B/32 model. The learning rate, batch size, and weight decay are 5 x 10 4, 64, and 10-4, respectively. All experiments were conducted on a single A 100 GPU with 48 GB of RAM. To create non-lID data, the widely-adopted Dirichlet distribution [Yurochkin et al., 2019] with small value of concentration parameter 8 = 0.1 is used and the maximum number of samples per class limited to 16 (unless otherwise specified). More details for the non-lID setting and the data distribution visualizations provided in Figures A. and A.2 in Appendix D.
Implementation details:
4.2 Simulation Results: NormFit as a Standalone Strategy
NormFit's flexibility allows it to function
standalone fine-tuning strategy an add-on to existing fine-tuning methods. We evaluate NormFit as a standalone solution in this subsection and, in Section 4.3, assess its performance as an add-on integrated with existing fine-tuning techniques.
4.2.1 Comparison to the Most Representative VLM Fine-Tuning Methods Adopted in FL
Table shows the accuracy of NormFit and the most representative VL.M few-shot fine-tuning methods adopted in FL in non-III scenarios with 3 = 0.1. We report the results for more skewed IID setting with 3 0.05 in Table A.2 in Appendix D. All results demonstrate that
7

---

Table I: Accuracy of NormFit and the most representative VLM few-shot methods adopted in FL.
Biseline
CIRARIO
ImageNett
Caltech
Oxtocdiees
Cars
Flowers
Food
Aircratt
SUN
DTD
Average
Capp
28.1
138112014
74,8220117
$1.9020117
82.6020.15 64 0.00 20 15
$84220011
55.30 1s
62:0220116
CoCoOp
81 81.90=0.16
51,724011
59.36+00100
86,59+0011
60.30+0012
75.17=0.12
33.0000.10
2043-0111
00.96:0113
50,53+00113
0..1
TiP-Alepter
801440071
62.140081
50.11
85.27:00000
20.12:0.80
#0.66
3061=0332
63 63,842003
$7.25
NoenFii (ouns
00.560000 62.57:0211 89.92:006 90.22:0018
70.46:00225 01.27:00.08 81.662033
32.922044
7001::002
10.42::100
7372200233
Table 2: Accuracy of NormFit and the most recent FL VLM few-shot methods.
Diseline
CUVAR1O
ImageNet
Cattchh
OxtocdPess
Cars
Flowers
Food
Aircratt
SUN
DTD
Average
FedC' IP
85.9600.00 10 58.4620113
85.7990114
56.9620.14 68 1770 87.12:0.16 7645:20112 21 55:0000 64:21:0008
58.55+0002
00.35
Fed1PG
M:40=00:22 59924-017
80,74+0016 38.23+00.19
68.13+0.18 87 07.46+0.000 78:57=0.11 22 22:46:00119
56.39-0227
59.32+00.04
70337
FodDAT
84.9900334 61,70:0040
83,4340229
87,5720.66
66.9940.19
88.02004
78225=0447 23,6220.30
65.2320211
60
0031
FLoRA
88.99-0.36
61.29:9026
87.93-00.77 90,02+0.115 70.17+0225
90.76+00099 79.35=0.16 32.224-0.25 67,164-0112 59,09-00.14
72.64:02221
NocmHit louns]
89.54=0.07 62.57-0121 59.19=000 90.22++0.18 70.46++0.25 91.27=0005 X1.06=1.31 32.59:00.44 7001=-062 60.42+00100
73.2240.23
NormFit exhibits robust performance, mostly outperforming all baseline methods adopted in FL, while substantially reducing communication burden and computation cost, shown in Figure (a).
Prompt tuning methods like CoOp and CoCoOp fail to effectively steer the model toward the underrepresented classes due to two reasons. First, prompts cannot modify the frozen backbone, which limits their ability to represent rare or minority classes. Second, prompts may overfit to the majority classes in few-shot scenarios. Adapter tuning, although not entirely immune to the class imbalance, provides better robustness than prompt tuning, while still under- peder-performing NormFit.
Figure 4 shows Grad-CAM visualizations comparing NormFit with CoOp and CLIP-Adapter for one majority and one minority class. Both CoOp and CLIP-Adapter experience performance degradation for the minority class, indicating difficulties in capturing meaningful representations of underrepre- sented classes. In contrast, NormFit consistently produces clear and meaningful activation patterns for the minority class, demonstrating its superior capability in managing class- maass-imbllaneed data.
4.2.2 Comparison to Most Recent FL Few-Shot Methods for VLMS
Table 2 compares NormFit to the most recent FL fine-tuning methods designed for VLMs, including FedCLIP [Lu et al., 2023], FedTPG [Qiu et al., 2024], FedDAT [Chen et al., 2024], and FLoRA [Nguyen et al., 2024]. FedCL.IP trains an adapter federally, yet does not modify CLIPP' core parameters, leaving representations biased toward the pretraining data and limiting gemeralization. FedTPG learns unified prompt generation network but lacks the capacity to fully address non-IID data. FedDAT uses data augmentation to handle non-IID data, but it risks overfitting by generating slight variations of minority samples rather than new data. FLoRA integrates LoRA adapters into CLIP's text encoder, reducing communication cost and improving adaptability. NormFit outperforms all those methods, while further reducing communication overhead and computation cost (Figure 1).
4.3 Simulation Results: NormFit as an Add-On
We now evaluate the effectiveness of NormFit an add-on method. Table 3 presents the simulation results combining NormFit with CoOp and TIP-Adapter in non-lID few-shot FL scenarios. The results clearly demonstrate that NormFit seamlessly enhances existing few-shot fine-tuning methods for VLMs, significantly improving their performance in non-IID few-shot FL.
4.4
Communication Burden and Computation Cost
We assess the communication burden of NormFit against other baselines by examining the size of per-user updates (in KB) shown in Figure 1(a). Because NormFit only transmits the Pre-LayerNorm parameters of the vision encoder (approximately 1.5 K parameters per user), the total data exchanged between the server and each user only 4.1 KB. In contrast, other methods send tens to hundreds of times more parameters. This substantial reduction makes NormFit especially attractive for environments where bandwidth is scarce.
In Figure (a)), the computation cost savings of NormFit are even more pronounced, which arise from two key design choices. First, whereas baseline methods add and train new external modules that require random initialization, NormFit updates only existing LayerNorm parameters, leveraging their
8

---

Original Image
CoOp
Aip-Adapter
NormFit
Figure 4: Grad-CAM visualizations for the Flowers dataset, comparing CoOp. Cdip-Adapter and Norm- Fit. Rows (a) and (b) correspond to the majority and minority classes, respectively. NormFit yields the best semantically aligned attention on both classes, notably better focus on the minority class.
(a]
(b)
Table 3: Accuracy of NormFit as add-on to existing fine-tuning methods in few-shot FL.
Baselize
CIFARIO
ImageNer
Cakeehh
OxfondPess
Curs
Flowers
Food
Auscra
SUN
DTD
Averaze
CcOp
76.71=0.19
50.33=0.15
83,81=00.14
74,4 5..90+00117
92.00-0.113 64,90+
20.36-0.15
58,12+0.11
55.3240.166
00.16
CoUp Nombin
007610224
89177
7277510.11 91.4
1011 3305 0.24
(9.54
TIP-Adaper
89. 21
61 14001
85.27500000
20 12.00.00 so 89 20.0 72.6120000
V06i =0 92, 61.8422001
57.2620.50
70.81 20.73
TIP-Adaper NornFil 91226=0922 628840077 91.33:100 1.077 72,94:0944 91.650089 7543=0344 34.000040 66.17:1114 59.2640.68
73
Table 4: NormFit vs. fine-tuning additional LayerNorms, in few-shot FL.
Melad
CIFARIO
EmazeNNes
Callchh
OxfoedPets
Cars
Flowers
Food
Alrena
SUS
DTD
Avenge
All LayerSorms
Sa5SuuT 61 Rus0os M.
MU 05 MELLO. to M0.7120011 7800LE020 20 20 202000 05.8520000
56.88:10114
09.5020010
aynnNemss
M.182009 624522008 88.822012 87.0610
32 1e
69.02201
71.722010
Text encoder LayerNocms
813610119 01:25:0000 07.16:10008
86 1..0010.16
27:01110177
09.06100.99
79.53.10122
3006610015
67.18200111
56,02.10:25
71.29110.14
Test encodirr Pres syerNoem
89 17 A0912000 88,2320007 60.10200 to 84g000
88.28200100
90.06:0011
32.47200100
142002 58-36-00.10
12.820011
Nommiiz (ouns)
MAsELuTT 62.5720.21
70.46:10.25
V1.27120.08 81.6610331
32.89110.44
7000220622 00.42.20.11)
73:7220225
pretrained values. Second, by restricting updates to the Pre-LayerNorm at the network's 's input stage, NormFit rapidly realigns feature statistics using minimal additional FLOPs. Consequently, each user incurs only 8.5 x 10 GFLOPs substantially lower than those competing methods.
4.5 Ablation Study
4.5.1 NormFit vs. Fine- Tine-Tuning Additional LayerNorms
Table 4 presents the experimental results for various LayerNorm fine-tuning strategies in few-shot FL with non-IID data. The results directly confirm the earlier discussions in Sections 3.1 and 3.2 as well as the theoretical analysis in Section 3.3. Specifically, as discussed in Section 3.1, fine-tuning parameters in the vision encoder is more effective than the text encoder for adapting to distribution shifts. Section 3.2 further explained that fine-tuning only the Pre-LayerNorm enables rapid alignment of feature statistics while minimizing parameter updates, and Section 3.3 theoretically justified that fine-tuning fewer parameters leads to significantly smaller generalization gap. Consistent with these insights, Table 4 shows that NormFit, which fine-tunes only the vision encoder's 's Pre-LayerNorm, achieves the best performance among all strategies.
To gain deeper insight into the effect of fine-tuning additional LayerNorms in the model, Figure 5 presents the results of progressively unfreezing the LayerNorms in both the vision and text encoders. Each encoder consists of Pre-LayerNorm and 12 transformer layers, each containing two LayerNorms. The horizontal axis in Figure 5 represents the cumulative number of LayerNorms fine- tuned, starting from the Pre-LayerNorm of the vision encoder (NormFit), then gradually including all LayerNorms of the vision encoder, followed by the Pre-LayerNorm of the text encoder, and finally all I LayerNorms of the text encoder. As more LayerNorms are fine-tuned, the performance tends to degrade, highlighting that excessive fine-tuning disrupts the pretrained alignment between the vision and text modalities in CLIP.
4.5.2 NormFit with Different Maximum Sample Sizes per Class in Few-Shot FL
In our FL setup, we first sample per-user class proportions from a Dirichlet distribution, then cap the resulting per-class sample counts at predefined maximum of 16. To assess how this cap influences learning, we vary the per-class sample limit from 1 to 16 and measure the resulting accuracy. Figure and Table A.3 present these results, demonstrating that the larger the cap (thus providing richer local datasets), the better NormFit performs in few-shot FL.

---

90
82
70
Accuragy
88
SMN
Aircraft
SUM
Feod
Aircraft
Cara
Food
48
Pats
Cars
Flovers
Pats
39
20
18
28
T
40
Number LayerNorms Fine-Tuned Model
Maximum Number Samples per Class (8-0.)) 0.1)
Figure 5: Fine-tuning more LayerNorms both encoders in few-shot FI.. '1' denotes fine- tuning the Pre-LayerNorm of the vision en- coder (NormFit). Tuning more and deeper lay- ers degrades the performance, underscoring the critical role of early normalization layers.
Figure 6: Impact increasing the maximum number of samples per class on accuracy across different datasets in few-shot FI.. Ac- curacy improves as the maximum number of samples per class increases, highlighting the benefit of more data.
Table 5: Zero-shot performance of NormFit.
(a) Trained on Food.
(b) Trained on Pets.
NormFir
LayerNorms
NormFit
LayerNorms
Food
81.0::0.511
78.80-40
76.42
90.22-0.33
84.24::0.09
85.35
CIFAR-10
03.92+0.36
76.53-4-.11
88.73
CIFAR-10
85.11=0.42
74.50:10.27
88.73
Flowers
6.675+00.0
52.49-+0.02
66.10
Flowers
0.24
50.43++0.10
66.10
4.5.3 NormFit in Zero-shot mode
Table 5(a) and (b) present the zero-shot performance of NormFit compared fine-tuning all Lay- erNorms, with Food and Pets used as training datasets, respectively. The results highlight that fine-tuning all LayerNorms can degrade robustness, whereas NormFit offers better trade-off be- tween adaptation and generalization.
4.5.4 NormFit in Centralized Few-Shot Learning
While NormFit is designed for few-shot FL with non-III data, is also effective to the centralized setup when the few-shot training data is class-imbalanced. In Appendix E, we present the results for the centralized learning case with masssimmallaceed few-shot data. Specifically, Table A.4 compares the accuracy of NormFit with that of the other few-shot fine-tuning baselines in centralized few-shot learning. Table A.5 evaluates NormFit as an add-on, and Table A.6 explores the effect of fine-tuning more LayerNorms in centralized few-shot learning.
5 Conclusions
We introduced NormFit, a novel framework designed to address the issue of non-lID data in few-shot FL for VLMs.. By fine-tuning only the Pre-LayerNorm in the vision encoder of VLM, NormFit effectively mitigated the bias toward majority classes while enhancing the representations minority or underrepresented classes. NormFit demonstrated superior performance across wide range of datasets, often achieving notable gains in accuracy compared to SOTA approaches. It substantially reduces both communication overhead and local computational costs, making it an ideal solution for bandwidth-ling limited FL with eesource-constriinng users. When integrated with traditional few-shot fine-tuning methods, NormFit further improved the performance, underscoring its versatility.
While being effective in practical non-IID FL scenarios, NormFit may be less suited for extreme domain shifts. Additionally, NormFit applies uniform strategy users in FL without personal- izing for differences data quality or domain. Also, in high data regime, other traditional methods may outperform Norm in FI..
10

---

Acknowledgments
This work was supported by the Digital Research Alliance of Canada through the Researcher Resource Grant (RRG) 5193 (2025), and by the Natural Sciences and Engineering Research Council of Canada (NSERC) through the Discovery Grant RGPIN-2025-04708.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's s contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in Section Conclusions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The paper provides the full set of assumptions in Definition 1 and Theorem 1. The complete (and correct) proof provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 4.
Guidelines:
All the information needed to reproduce the main experimental results of the paper are included in the paper.
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
12

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code is provided supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
wwwpsssioooooooioo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /twpp/.ccppublll//// cealiic/giieee http/guddes///eee https//ooSSSuuiisseeeee ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
All the training and test details are specified in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
13

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 4.
Guidelines: The computation resource information provided.
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://rrrpps.ccpppull//tttttiea
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiciio))
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: the Introduction, it stated that federated learning is an effective mechanism to protect data privacy.
14

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The owners of assets used in the paper, are properly mentioned and respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
15

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com comddataees has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
16

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy /ttpp::///eurps..... for what should or should not be described.
17

---

References
Peter L BBrrlett, Nick Harvey, Christopher Liaw, and Abbas Mchrabian. Nearly-tight ve-dimension and pscudodimension bounds for piecewise linear neural networks. Journal of Machine Learning Research, 20(63):1-17, 2019.
Lukas Bossard, Matthicu Guillaumin, and Lue Van Gool. Food-101- min-n discriminative compo- nents with random forests. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pages 446-461. Springer, 2014.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Haokun Chen, Yao Zhang, Denis Krompass, Jindong Gu, and Volker Tresp. Feddat: An approach for foundation model finetuning in multi-modal heterogeneous federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1285-112933 11293, 2024.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 245-255. leee, 2009.
Mohammad Fahes, Tuan-Hung Vu, Andrei Bursuc, Patrick Pérez, and Raoul de Charette. Fine-tuning clip's last visual projector: A few-shot coruucopia. arXiv preprint arXiv:2410.05270, 2024.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178-178. IEEE, 2004.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint
2021.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-lange language models with feature adapters. International Journal of Computer Vision, 132(2):511-595, 2024.
Tao Guo. Song Guo, and Junxiao Wang. Pfedprompt: Learning personalized prompt for vision- language models in federated learning. In Proceedings of the ACM Web Conference 2023, pages 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine fine-graine grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Ming Li, Jike Zhong, Chenxin Li, Liuzhuozheng Li, Nie Lin, and Masashi Sugiyama. Vision-language model fine-tuning via simple parameter-efficient modification. arXiv preprint arXiv:2409. /67 18,8. 2024.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1 11976 2022.
Wang Lu, Xixu Hu, Jindong Wang, and Xing Xie. Fedclip: Fast generalization and personalization for clip in federated learning. arXiv preprint 234.883.....388... 2023.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 2013.
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In ICMI., pages 4615-4625, PMLR, 2019.
18

---

Duy Phuong Nguyen, J Pablo Munoz, and Ali Jannesari. Flora: Enhancing vision-languee models with parameter-efficien federated learning. arXiv preprint arXiv:2404.15182, 2024.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference computer vision, graphics & image processing, pages 722-729. IEEE, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference computer vision and pattern recognition, pages -998-3505. IEEE, 2012.
Chen Qiu, Xingyu Li, Chaithanya Kumar Mummadi, Madan Ravi Ganesh, Zhenzhen Li, Lu Peng, and Wan-Y Yi Lin. Federated text-driven prompt generation for vision-language models. In The Twelfth International Conference on Learning Representations, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103. 00000000000...0 2021.
Shangchao Su, Mingzhao Yang. Bin Li, and Xiangyang Xue. Federated adaptive prompt tuning for multi-domain collaborative learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15117-15125, 2024.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages -485-3492. IEFE, 2010.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametrie federated learning of neural networks. In International conference on machine learning, pages 7252-7261. PMLR, 2019.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficien fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106. 2021.
Maxime Zanella and Ismail Ben Ayed. Low-rank few-shot adaptation vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1593-1603 2024.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pages 18123-18133 18133, 2022.
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang L.i, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:21 / 03990, 2021.
Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. Fedprompt: Communication efficient and privacy-puns preaccppee prompt tuning in federated learning. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16816-16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-2348, 2022b.
19

---

Appendix
The Appendix includes the following items:
Content
Page
Proof of Theorem
Proof of Theorem 2
21
NormFit with Other Vision Encoders
Dirichlet Distribution-Bast Non-IID Partitioning
22
NormFit with More Skewed Data Distribution in Few-Shot FT.
22
NormFit with More Skewed Data Distribution in Zero-Shot Mode
NormFit in Centralized Few-Shot Learning with Class-Imbalaneea Data
23
NormFit as a Standalone Strategy in Centralized Few-Shot Learning with
Data
NormFit Add-On Existing Fine-Tuning Methods in Centralized Few-Shot Learning with Class Imbalanced Data
23
Ablation study: Fine-tuning More LayerNorms in the Centralized Setting
24
A Proof of Theorem 1
We start from the generalization gap bound given in Theorem 2 of [Mohri et al., 2019]:
s(A [m) [Ac
4h E H. sup πEA
[Tx (1, s 29 29, (9, + Me My 2m
[.D
log
(A.1)
where R (9, denotes the weighted Rademacher complexity over the loss class G, m the skewness, m = > my. and
(mi, ... mar) the empirical sampling distribution, m L m. Note that the bound can be expressed as
S 2m (g, A) + (terms independent of model complexity).
(A.2)
We now focus bounding the dominant term Rm (G, A) in terms of model capacity. By Lemma in [Mohri et al., 2019], for a hypothesis class H with Vapmik-Chervoneniis (VC)-dimension L, the weighted Rademacher complexity satisfies:
2L
em (T,
Rm (G,
(A](m)) log
(A.3)
The VC-dimension bound L is upper bounded by Eq. (2) in [Bartlett et al., 2019]:
L S epg log(p).
(A.4)
where c is a constant, is the number of trainable parameters, and q is the number of layers.
We now consider two cases: fine-tuning (i) Pre-LayerNorm only and (ii) all LayerNorms including the Try-LayerNomm the model. First, for the case fine-tuning the Pre-LayerNorm only, because the scale and shift of the single layer trained (p 2d and 9 1). have
Lpre 2cd log
(A.5)
20

---

and the generalization gap is upper- soper-bounde as:
Acd Aod
Epre
s(AlIm) log 2cd
(A.6)
m
For the second case of fine-tuning all LayerNorms including the pre- pre-LayerNomm and those Layer- Norms in the transformer layers, we have 9 (2.N 1) and p 2d( + 1) because each transform layer has two LayerNorms. It follows that
Laul S 2cd + 1)a log(2d(2N +
(A.7)
and the generalization gap is upper-hounded as:
4od(2N + 1)² log (2d(2N + +1))
Calt S
log 2ed(2N - log( + 1))
m
(A.8)
Their ratio becomes:
log 2od(2N 1)² (2d(22
Eall
(2N +
+ 1))
(A.9)
Epre
log(2d)
log
Toyr (dd
In many practical regimes (i.e., when the capacity depeedett logarithmic terms vary slowly with the model complexity), the ratio the two logarithmic terms is upper bound by a constant. Dropping the constant, the bound simplified to
Eall Epre
(2N 1)2 (1+
log(2N +1)
2
V
(A.10)
log (2d)
B Proof of Theorem 2
We model each layer as a transformation of the form
hy (1
(A.11)
where ho x is the input to the model and h is the output of layer l. Layer-wise Lipschitz constant, Li, is the supremum of the norm of its Jacobian with respect to its input, h11:
= sup 11 Or)I h
(A.12)
Due to chain of dependencies on all preceding layers, we have
h-1 = fi- θ--1) = fi-1(fi fi (1)...))...) 0;-1).
(A.13)
Thus, the Lipschitz constant L, of layer / is implicit function of all parameters to and including layer
L = 02...
(A.14)
For an arbitrary layer k, we take the gradient of T.global with respect to O₁. Applying the multivariate chain rule, we have
21

---

Table A.1; NormFit with different types of vision encoders in few-shot FL.
Reseline
Imngehii
OsfoniPats
Avarge
NormFat
91.0110:00 6X1210.10 914810007
72021002 631610007
75.1510.12
NoraFit (ComNNX0))
58.4660166 87.19:0229 89 89,440021
717760.16 900840018 79.1940.30 30,5540441 70.6360229 6133900100
72,34:023
(our)))
59.4+0007
62.57:0.21 890960006
W.22HaL1 70.66:0225 912774000 8106400331 32.59:00444 2061 6042260110
13,72-0225
Hggoball VorLi,
Vo, Lglobal
Σ ويا
(A.15)
where
II Li- (F)
(A.16)
ÖT.j
Since the parameters O₁ only influence its Lipschitz constant La and the Lipschitz constants of subsequent layers Lj (j k), the gradient term Ve, Ljj zero for all j <
N Σ Le.e Lj)
Ve Lglobal
II L
(A.17)
Separating the first term of the sum (where j = k) from the rest the terms, we have
Ve,, Ligiobal = (Ve,LN) Ix)
II igck
Li
+ Σ
(Ve.L,) Lz)
I Li
(A.18)
j=b+1
C NormFit with Other Vision Encoders
While our experiments mainly focus on CLIP with ViT. NormFit is broadly applicable to VLMs with other vision encoders. In Table A.I, we further demonstrate its effectiveness with LiT [Zhai al., 2022] and ConvNeXt [Liu et al., 2022], showing that NormFit consistently improves generalization.
D Dirichlet Distribution-Based Non-IID Partitioning
To simulate realistic non-lID data for users in FL, we adopt the widely accepted Dirichlet distribution [Yurochkin et al., 2019] to partition class-labeled data among users. This method provides flexible and tunable way to control the degree of non-IlDD non-IID-ness: across users. A smaller concentration parameter leads higher skewness, where each user receives data from fewer classes, mimicking strong non-lID scenarios. Conversely, larger values of 3 produce more balanced distributions across users. Figures A. and A.2 illustrate this effect by visualizing non-IID distribution across 10 users with 10 classes for two different values of 3 0.1 and 0.05, respectively.
D.I NormFit with More Skewed Data Distribution in Few-Shot FI.
Table A.2 presents the accuracy of NormFit and the most representative VLM few-shot methods adopted in non-IID few-shot FL with more skewed data distribution of 3 0.05.
22

---

User IDs
66
User IDs 3
5
5
4
3
2
012
1
0
1
3
4
5
6
7
8
0
1 2
4
5
6
9
Class IDs
Class IDs
Figure A.1: Class distribution across 10 users for 10 classes with non-IID data 3 0.1.
Figure A.2: Class distribution across 10 users for 10 classes with non-IID data 0.05.
Table A.2:
Accuracy of NormFit and the most representative VLM few-shot methods adopted in FL with more skewed data distribution (3 0.05)
Baseline
CIFARIO
ImaggeNet
Caltech
OxfordPets
Cars
Flowers
Food
Aiscraft
SUN
DTD
Average
CoOp
76.27
47.46
80.39
71.61
50.11
80.56
64.18
19.50
56.40
54.23
60.07
CoCoOp
80.23
50.09
87.42
85.18
58.92
72.19
71.70
19.82
61.03
49.75
63.63
TIP-Adapter
88.37
61.25
87.64
85.00
69.74
88.55
30.23
62.94
56.52
70.24
NormFit (ours)
88.70
60.40
88.26
90.00
69.95
90.83
80.18
30.71
68.47
60.03
72.75
D.2 NormFit and Different Maximum Number of Samples per Class in 3 0.01
Table A.3 shows how increasing the number of samples per class improves accuracy across three datasets (Flowers, Food, Pets) in few-shot FL with 3 = 0.01. The results indicate that adding more samples per class consistently boosts performance, with diminishing gains beyond 16-32 samples.
E NormFit in Centralized Few-Shot Learning with Class-Imbalanced Data
In this appendix, we provide the simulation results of NormFit the centralized few-shot learning setting with class-imbalances data (i.e., the case of a single user having class- dass-imbalanced few-shot training data) in comparison to the existing few-shot fine-tuning methods in the -msalanebd scenario with = 0.1.
E.1 NirmFit Standalone Strategy in Centralized Few-Shot Learning with Tlass-lmbalanced Data
Table A.4 presents the accuracy of NormFit compared with the most representative few-shot fine- tuning methods in a centralized few-shot learning setting under class-imbalanced data conditions (B = 0.1). NormFit continues to achieve improved performance even in centralized scenarios with class imbalance, consistent with its effectiveness in non-III few-shot FL scenarios shown previously in Table I.
E.2
NormFit Add-On to Existing Fine- Fine-Tuning Methods in Centralized Few-Shot Learning with Mlass-mmalaaced Data
Similar to its application in non-IID few-shot FL scenarios, NormFit can serve as an add-on to existing fine-tuning methods, further enhancing their performance in centralized few-shot learning with class-imbalanced data. Simulation results demonstrating this capability are presented in Table A.5. Specifically, fine-tuning the Pre-LayerNorm parameters of the vision encoder (NormFit) consistently improves the performance of CoOp in centralized few-shot scenarios under class imbalance.
23

---

Table A.3: Impact of increasing the maximum number of samples per class on accuracy across datasets in few-shot FL (B 0.01).
Dataset
16
Flowers
66.37710.11 86 39.1910441
87.8310.60
88.11
Food 44.50-20.10
63.36:1:0.25
25.62
76.09:00.07
Pets
65.76.10.60 67.81 67.81 1.0.00 88.19 0.229 88.45.10.09
Table A.4: Accuracy of NormFit and most representative fine-tuning methods adopted in few-shot centralized learning.
Baseline
CIFARIO
ImageNet
Caltech
OxfordPets
Cars
Flowers
Food
Aircraft
SUN
DTD
Average
CoOp
78.16
53.42
78.12
76.23
58.23
21.52
59.70
63.75
CoCoOp
82.36
56.30
85.17
88.26
63.76
74.50
75.90
20.32
64.90
51.59
64.31
TIP-Adapter
90.20
64.71
87.92
87.19
71.80
89.83
75.46
32.39
67.21
60.43
72.71
NormFit (ours)
90.76
65.11
88.50
90.13
71.92
80.93
34.17
70.24
61.09
74.40
Table A.5: Accuracy of NormFit as an add-on to the fine-tuning methods in few-shot centralized learning.
Baseline
CIFARIO
ImageNet
Caltech
OxfordPets
Cars
Flowers
Food
Aircraft
SUN
DTD
Average
CoOp
78.16
53.42
78.12
76.23
58.23
85.11
21.52
63.95
CoOptNormFit (ours)
91.18
67.77
89,60
92.27
73.51
9243
81,66
34,04
69.42
61.29
75.92
Table A.6: NormFit vs. fine-tuning additional LayerNorms in few-shot centralized learning.
Baseline
CIFARIO
ImageNet
Caltech
OxfordPets
Cars
Flowers
Food
Aircraft
SUN
DYTD
Average
LayerNooss
Vision encoder LayerNorms
91.01
6-1.51
88.42
90.03
70.22
89.23
80.33
35.92
68.59
59.32
70.03
LayerNomms
Text encoder Pre LayerNerm
89.72
62.25
88.34
89.44
71.50
89.01
73.16
34.47
67.12
51.78
67.48
NormFüt (ours)
90.76
65.11
88.50
90.13
71.92
91.19
80.93
34.17
70.24
61.09
70.90
E.3
Ablation study: Fine-tuning More LayerNorms in the Centralized Setting
Table A.6 presents the simulation results of fine-tuning various combinations of LayerNorm pa- rameters in centralized few-shot settings with class-imbalanced data. Fine-tuning all LayerNorms yields overall unsatisfactory performance, notably datasets such Aireraft and DTD. Fine-tuning only the vision encoder LayerNorms is beneficial primarily for datasets characterized by strong visual features, such as OxfordPets. Meanwhile, fine-tuning only the text encoder LayerNorms is more effective on datasets where textual context clarifies visual concepts, such as Caltech and Food. Importantly, NormFit consistently achieves the highest accuracy across most datasets. The minor performance differences observed for CIFAR-10 and Aircraft in Table A.6 indicate that certain datasets with unique characteristies might slightly benefit from broader fine-tuning. However, these marginal improvements incur drastically higher computational costs, significantly restricting their practicality in resource-constrained scenarios. NormFit, in contrast, emphasizes minimal yet highly effective parameter updates, defining superior balance between performance and efficiency.
24


Reviews:
[{"review_id": "u8ILgV9jb3", "reviewer": "Reviewer_s3pM", "summary": "This paper proposes NormFit, a lightweight fine-tuning strategy for the CLIP model in few-shot Federated Learning settings with non-IID data. The core idea is to selectively fine-tune only the Pre-LayerNorm parameters of the vision encoder, while keeping other parameters frozen. This approach significantly reduces communication overhead and local computational demands, while simultaneously achieving superior accuracy, particularly for underrepresented classes in non-IID scenarios. A theoretical justification for a smaller generalization gap is provided, and empirical results on various datasets demonstrate NormFit's effectiveness both as a standalone method and as an add-on to existing fine-tuning techniques.", "questions": "As pointed out in the weaknesses, I have the following main concerns:\n\n1. While fine-tuning only the Pre-LayerNorm parameters demonstrates effectiveness, the approach is relatively simple and lacks strong connections to the federated learning scenario. The paper would benefit from a more detailed discussion on the research motivation, as well as the relationship between the proposed method and key aspects of federated learning such as privacy preservation and data security.\n2. Given the stochastic nature of data distributions under non-IID settings, repeating experiments and reporting error bars or confidence intervals would significantly enhance the credibility of the results.\n3.  The comparison with CoOp and CoCoOp is insufficient, as both methods are outdated and focus solely on fine-tuning the text encoder of CLIP. This differs significantly from NormFit. Furthermore, comparisons with non-VLM-based federated learning approaches are also necessary to better contextualize the contribution of the proposed method.\n4. The authors claim that fine-tuning only the Pre-LayerNorm helps preserve prior knowledge; however, this claim is not supported by experimental evidence. Including an evaluation of the model’s zero-shot performance after fine-tuning would help substantiate this argument and improve the quality of the paper.\n\nAdditionally, the reviewer has a question regarding**Figure B.2**. It appears that, for class 3, no samples were assigned to any of the users. Is this a valid outcome under the considered data partitioning strategy? Alternatively, could it be due to low color contrast in the figure? Clarification on this point would be helpful.", "limitations": "The authors discuss the limitations of the paper to a certain extent, such as the inability to adapt to excessive domain shift and the lack of personalization ability, but do not mention the solution ideas and future research content. The reviewers suggest further discussion on how NormFit could be extended or combined with personalization techniques within FL.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "*Strengths*:\n\n1. The inclusion of a theoretical analysis on the generalization gap provides a valuable mathematical foundation for why NormFit performs robustly with minimal parameter updates. This strengthens the paper's claims beyond empirical observations.\n2. NormFit achieves improvements without requiring algorithmic modifications, architectural changes, or additional external parameters, enabling seamless integration with existing FL pipelines.\n3. Extensive experiments across diverse datasets and comparisons with state-of-the-art methods (e.g., FedCLIP, FLoRA) demonstrate consistent improvements in accuracy, communication efficiency, and computational cost.\n\n*Weaknesses*:\n\n1. The novelty of the work is incremental. While updating only the Pre-LayerNorm parameters of the vision encoder demonstrates effectiveness, the approach is relatively straightforward and does not constitute a fundamentally new algorithm. Similar strategies have already been applied in \\[1].\n2. The proposed method has limited relevance to key concerns in federated learning such as privacy and security. The focus is solely on communication efficiency in distributed learning frameworks, which weakens the motivation and scope of the study.\n3. The experimental results lack statistical rigor. Only single-run results or average accuracies are reported, without error bars or confidence intervals. Given the stochasticity of data distributions in non-IID settings, this raises concerns regarding the robustness and generalizability of the observed performance improvements.\n4. The baseline methods used for comparison—CoOp and CoCoOp—are outdated, and both involve fine-tuning only the text encoder of CLIP, which differs significantly from NormFit’s strategy. A more appropriate comparison would be with \\[2]. Moreover, comparisons with non-VLM-based federated learning approaches are also needed to better contextualize the contributions.\n5. The authors claim that fine-tuning only the Pre-LayerNorm helps preserve prior knowledge; however, this assertion lacks further analysis or theoretical justification.\n\n\\[1] Tent: Fully Test-Time Adaptation by Entropy Minimization (ICLR 2021)\n\n\\[2] MaPLe: Multi-modal Prompt Learning (CVPR2023)", "was_revised": true, "final_justification": "The authors have addressed the concerns in my initial review. Therefore, I revised my score upwards accordingly.", "rebuttal": "**Thank you very much for your thoughtful and constructive feedback.**\n\n**[Weakness 1]** \n\nWe respectfully emphasize that the novelty of NormFit lies in its *innovative* identification of Pre-LayerNorm as the optimal fine-tuning target—an approach that is *fundamentally* distinct from prior tuning strategies. Furthermore, we now introduce additional new and significant theoretical result (Theorem 2) to analytically show why the Pre-LayerNorm is the optimal target for fine-tuning. The result complements Theorem 1. Please see our response to Reviewer **Qst8**. \n\nRegarding TENT [1], comparison with TENT actually highlights the novelty of NormFit. While it is true that both methods touch normalization-related parameters, NormFit exactly *rejects* TENT’s strategy of tuning ALL normalization layers. Instead, NormFit shows that only Pre-LayerNorm of ViT should be fine-tuned in few-shot FL. Applying TENT-style full fine-tuning (whether ViT, the text encoder, or both) notably worsens accuracy compared NormFit (Table 4 and Figure 5). Moreover, such broad tuning increases communication and computing burden by orders of magnitude — clearly undesired in FL. Theoretically, Theorem 1 provides a formal justification that TENT’s style incurs a significantly larger generalization gap (up to 30×) compared to NormFit. \n\nThe root cause of this divergence lies in the fundamental differences of the problem formulation, system setting, learning objective, and operating constraints. TENT is designed for *centralized, unsupervised*, and *test-time adaptation* in *inference*. To compensate for the absence of supervision, TENT updates ALL normalization layers to minimize prediction entropy on-the-fly. In contrast, NormFit operates in a *decentralized (federated), supervised, few-shot*, and *training* regime. In this context, updating all normalization layers introduces a risk of overfitting and violates efficiency requirements of FL. NormFit addresses all these issues. \n\n**[Weakness 2, Question 1]** \n\nWe respectfully argue that “being simple” is a critical strength (not weakness), especially for real-world FL deployments. Simple and lightweight methods are far more likely to be adopted—especially at scale—than those requiring complex changes or coordination. In this spirit, NormFit’s design is *intentionally* minimal yet highly effective.\n\nNormFit resolves three *key* concerns of *FL* (not just communication efficiency): (1) achieving SOTA accuracy, which is unquestionably crucial; (2) reducing communication overhead, which is essential for scalability in practical FL deployments; and (3) minimizing local computation, which is critical for resource-constrained users (e.g., edge devices). Importantly, NormFit achieves the highest efficiency while outperforming current SOTA methods in accuracy. \n\nRegarding privacy and security, NormFit is orthogonal to—and fully compatible with—any existing or emerging security and privacy mechanisms such as differential privacy, secure aggregation, and homomorphic encryption (HE). NormFit’s lightweight design makes these mechanisms implementable in real-world FL settings. For example, HE is notoriously resource-intensive and often impractical for complex FL setups. The minimalism of NormFit makes HE far more feasible in practice.\n\nImportantly, NormFit’s extreme efficiency brings significant benefits to privacy and security. Its ultra-low communication footprint reduces the risk of gradient inversion attacks due to its severe information bottleneck. Reconstructing high-dimensional inputs from a very low-dimensional update vector (~ 1.5K parameters) can be a difficult inverse problem, making high-fidelity recovery challenging for an adversary. Crucially, the parameters that NormFit updates—the scale and shifts of the Pre-LayerNorm—encode low-level statistical information rather than high-level semantic content. These parameters capture very coarse-grained statistics at the *earliest* stage, making it more difficult for an adversary to extract semantically meaningful inputs from their gradients. In contrast, methods that tune adapters or prompts modify high-level feature representations that are richer in semantic detail, and thus more susceptible to inversion.\n\nFurthermore, NormFit’s design provides architectural robustness against model poisoning attacks. Such attacks typically rely on manipulating high-capacity components of the model to introduce backdoors or malicious behaviors. NormFit’s extremely constrained update space restricts the expressive capacity available to a malicious client. Attackers are, by design, constrained from modifying core components such as attention blocks or MLP layers that govern the model’s semantic reasoning and classification behavior.\n\n**[Weakness 3, Question 2]**\n\nAll simulation results in the paper were obtained by running each experiment three times and reporting the average. In this rebuttal, we include the standard error of the mean (SEM) to quantify variability. Due to the rebuttal time constraints, we prioritized computing SEMs for the two strongest competing baselines. NormFit consistently outperforms these methods, with margins that mostly exceed their SEMs. For the results where SEM could not be computed within time, we use \"x.xx\" as a placeholder. In the final version, we will report SEMs for all results.\n\n**Table: Accuracy of NormFit and VLM few-shot methods in FL (selected datasets)**\n\n| Baseline       | Caltech        | Flowers         | Food           | Aircraft        | DTD            | SUN             |\n|----------------|----------------|------------------|----------------|------------------|----------------|------------------|\n| TIP-Adapter    | 88.11 ± 0.94   | 89.01 ± 0.66     | 72.61 ± 0.60   | 30.61 ± 0.32     | 57.25 ± 0.50   | 63.84 ± x.xx     |\n| FLoRA          | 87.93 ± 0.27   | 90.76 ± 0.09     | 79.35 ± 0.16   | 32.22 ± 0.25     | 59.09 ± 0.14   | 67.16 ± 0.12     |\n| **NormFit (ours)** | **89.09 ± 0.06** | **91.27 ± 0.08** | **81.06 ± 0.31** | **32.59 ± 0.44** | **60.42 ± 0.10** | **70.01 ± 0.62** |\n\n**[Weakness 4, Question 3]** \n\nWe would like to clarify that our NormFit consistently and substantially outperforms FedTPG (Table 2 of our paper), which was published at ICLR 2024 and cited explicitly as [Qiu et al., 2024]. In Tables 1–3 of [Qiu et al., 2024], MaPLe is adapted to FL (referred to as FedMaple) and directly compared to FedTPG. The results show that FedMaple substantially underperforms FedTPG, and even underperforms FedCoOp. The core issue is that federated averaging breaks the V–L coupling that MaPLe relies on. While V–L coupling is effective in centralized learning, it becomes problematic in FL. For this reason, we did not include FedMaple as a baseline in our study. By contrast, we re-implemented CoOp and CoCoOp in our FL setup, as they demonstrated stronger performance than MaPLe in FL and could be more meaningful baselines. \n\nThe current SOTA method in FL is FLoRA, and it consistently outperforms FedTPG (Table 2 of our paper). Crucially, NormFit, consistently outperforms FLoRA (Table 2 of our paper), establishing a new SOTA. Furthermore, FedTPG itself outperforms FedCoOp, which in turn outperforms FedMaple [Qiu et al., 2024], resulting in:\n\n**NormFit > FLoRA > FedTPG > FedCoOp > FedMaple**\n\nApplying MaPLe in FL also raises critical security and privacy risks. The V–L coupling produces information-rich “super-gradients,” which are more vulnerable to gradient inversion attacks. This coupling also introduces potential vectors for cross-modal leakage, where private data from one modality could be reconstructed using updates from the other. \n\nFurthermore, MaPLe introduces 3.55M additional parameters, significantly increasing communication and computation overheads—detrimental for FL. The added complexity also elevates the risk of membership inference attacks.\n\nRegarding non-VLM FL methods, a direct comparison would be uninformative due to fundamental differences in problem formulation and assumptions. In the traditional Non-VLM FL, the question is: How can we collaboratively train a relatively small, task-specific model (e.g., a ResNet) from scratch on decentralized data? The core challenges revolve around achieving model convergence under statistical heterogeneity. By contrast, in VLM-based FL (NormFit's Domain), the question is: How can we efficiently adapt a pre-trained foundation model to downstream tasks using decentralized, few-shot data? The challenges here are minimizing the crippling communication/computation costs and adapting the model's vast prior knowledge effectively without catastrophic forgetting. These two paradigms are not direct competitors. \n\n**[Weakness 5, Question 4]** \n\nWe conducted the requested zero-shot evaluations after fine-tuning. First, the results demonstrate that NormFit is an effective adaptation method, outperforming baseline approaches on the held-out splits of the training datasets. Importantly, NormFit provides strong zero-shot performance on unseen datasets, supporting that catastrophic forgetting is effectively mitigated.\n\n**Table 2: Zero-Shot Performance**\n\n| Training Dataset | Testing Dataset | NormFit | All LayerNorms | Zero-shot CLIP |\n| :--- | :--- | :---: | :---: | :---: |\n| **Food** | Food | $81.06 \\pm 0.51$ | $78.80 \\pm 0.49$ | $76.42$ |\n| | CIFAR-10 | $83.92 \\pm 0.36$ | $76.53 \\pm 0.19$ | $88.73$ |\n| | Flowers | $63.75 \\pm 0.60$ | $52.49 \\pm 0.02$ | $66.10$ |\n| — | — | — | — | — |\n| **Pets** | Pets | $90.22 \\pm 0.33$ | $84.24 \\pm 0.09$ | $85.35$ |\n| | CIFAR-10 | $85.11 \\pm 0.42$ | $74.50 \\pm 0.27$ | $88.73$ |\n| | Flowers | $64.38 \\pm 0.24$ | $50.43 \\pm 0.10$ | $66.10$ |\n\n**[Question for Figure B.2]**\n\nYes, it is due to lower color contrast in the figure. We will improve this in the final version.\n\n**[Limitations]**\n\nA promising direction is to combine NormFit with client-specific adaptation modules (e.g., lightweight task-specific heads) trained locally to capture user-specific nuances."}, {"review_id": "iEoUk0JJPu", "reviewer": "Reviewer_Qst8", "summary": "This paper introduces NormFit, a novel and lightweight fine-tuning method for adapting Vision-Language Models (VLMs) like CLIP in the context of few-shot Federated Learning (FL) with non-IID data. The core problem addressed is the trade-off between model performance, communication overhead, and local computation costs, which plagues existing methods. NormFit's key idea is its simplicity and efficiency: it selectively fine-tunes only the Pre-LayerNorm parameters of the vision encoder, while keeping the rest of the VLM frozen.", "questions": "1. In Table 4, does the results of \"NormFit as an Add-On\" applying Tip-Adapter require extra hyper-parameter searching?\n2. Does a larger number of samples per class (>16) achieve higher accuracy in Figure 6?\n3. Can NormFit be transferred to CNN-based encoders?", "limitations": "Please refer to the questions section.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The proposed method, NormFit, is elegant in its simplicity. While parameter-efficient fine-tuning (PEFT) is a well-established field, the specific choice to tune only the Pre-LayerNorm of the vision encoder is a novel and non-obvious insight. This simplicity makes it easy to implement without any changes to model architecture or complex algorithms.\n2. The paper presents compelling evidence that NormFit resolves the critical trade-off between accuracy and efficiency in few-shot FL. The results in Figure 1 and Tables 1 & 2 are particularly strong, showing that NormFit simultaneously achieves the highest accuracy and the lowest communication/computation costs among all compared methods.\n\nWeaknesses\n1. The experiments are primarily focused on the CLIP model with a Vision Transformer (ViT) or Locked-image Text tuning (LiT) backbone. While these are representative, the paper's claims of general applicability would be stronger if the principle was tested on other VLM architectures or even different model families (e.g., CNN-based vision encoders) to demonstrate broader robustness.\n2. The generalization gap analysis is a significant strength, but it primarily explains why tuning fewer parameters is better. It does not fully provide a theoretical reason for the specific effectiveness of the Pre-LayerNorm layer over other layers, beyond the intuition that it is at an \"early stage\".", "was_revised": true, "final_justification": "The author's rebuttal partly resolved my doubts, and I decided to maintain the previous rating unchanged.", "rebuttal": "**Thank you very much for your thoughtful and constructive feedback.**\n\n**[Weakness 1, Question 3]**\n\nNormFit is potentially applicable to any models having normalization parameters, because the core idea of NormFit is to fine-tune a minimal set of normalization parameters at an early-stage. \n\nRegarding applicability to CNNs, recent advancements have made NormFit directly applicable to modern, high-performance, SOTA convolutional architectures like ConvNeXt. Designed to compete with Transformers, ConvNeXt has explicitly adopted Layer Normalization. Since NormFit targets fine-tuning early layer normalization parameters, it can be seamlessly applied to a ConvNeXt-based vision encoder without modification. \n\nConceptually, the principle of NormFit also extends to traditional CNNs like ResNet. A direct application would be to fine-tune only the affine parameters (gamma and beta) of early Batch Normalization (BN) layers, offering a principled and lightweight alternative to ad-hoc fine-tuning practices.\n\nDuring the rebuttal period, we have identified suitable open-source, pre-trained VLMs based on ConvNeXt (e.g., OLA-VLM) and ResNet to specifically test and apply NormFit in our forthcoming experiments. Unfortunately, due to the time constraint, we were unable to run and include these experiments in this rebuttal. However, all authors are committed to conducting these experiments on VLMs with ConvNeXt and ResNet encoders, and the results will be surely included in the final version of the paper.\n\n**[Weakness 2]**\n\nWe fully agree that Theorem 1 offers only theoretical support for the benefit of fine-tuning a single LayerNorm over fine-tuning multiple or all LayerNorms. Complementing Theorem 1, we now introduce additional new and significant theoretical result, Theorem 2, to analytically show why the optimal choice must be the first one (i.e., the Pre-LayerNorm), instead of one located deeper in the network. This new result highlights the unique influence of the first LayerNorm within the network. To formalize this locational advantage, we leverage Lipschitz constants, which are an essential tool in deep neural network analysis.\n\nA function $f: \\mathbb{R}^n \\to \\mathbb{R}^m$ is defined as Lipschitz continuous if there exists a non-negative constant $L$ such that for any two inputs $x_1, x_2$, the inequality $\\|\\|f(x_1) - f(x_2)\\|\\| \\le L \\|\\|x_1 - x_2\\|\\|$ holds. The smallest such $L$  is *the* Lipschitz constant of $f$. \nIn the following theorem, we provide the gradient of the global Lipschitz bound with respect to the parameters of any arbitrary LayerNorm (including the Pre-LayerNorm as a special case). \n\n**Theorem 2 (Influence of fine-tuning a LayerNorm)**\nLet $f_{\\Theta}$ be a ViT of $N$ layers, $f = f_N \\circ \\dots \\circ f_1$, where  $\\Theta$ represents the set of trainable parameters of all LayerNorms in the ViT. All other parts including attention blocks and MLPs are frozen.  Let $L_{global}(\\Theta) = \\prod_{l=1}^{N} L_l(\\Theta)$ be an upper bound on its global Lipschitz constant, where $L_k(\\Theta)$ is the Lipschitz constant of layer $k$. Let $\\Theta_k$ denote the set of trainable parameters of the $k$-th LayerNorm (in layer $k$). The gradient of the global Lipschitz bound with respect to $\\Theta_k$ is given by:\n$$\\nabla_{\\Theta_k} L_{global} = DIT_k + CIT_k$$\nwhere the Direct Influence Term, $DIT_k$, is given by \n$$DIT_k= \\left( \\nabla_{\\Theta_k} L_k \\right) A_k$$\nwith  amplification factor \n$$A_k=\\left( \\prod_{i =1}^{k-1} L_i \\right) \\left( \\prod_{i = k+1}^N L_i \\right)$$\nand the Cascading Influence Term, $CIT_k$, is given by \n$$CIT_k=\\sum_{j=k+1}^{N} \\left( \\nabla_{\\Theta_k} L_j \\right) \\left( \\prod_{i \\ne j} L_i \\right)$$  \n\n\n**Proof:** \nEach layer is represented by transformation $h_l = f_l(h_{l-1}, \\Theta_l)$,\nwhere $h_0 = x$ is the input to the model and $h_l$ is the output of layer $l$. Layer-wise Lipschitz constant, $L_l$, is the supremum of the norm of its Jacobian $J$ with respect to its input, $h_{l-1}$:\n$$L_l = \\sup_{h_{l-1}} \\|\\| J_{f_l}(h_{l-1}, \\Theta_l) \\|\\|$$\nDue to a chain of dependencies on all preceding layers, we have\n$$h_{l-1} = f_{l-1}(h_{l-2}, \\Theta_{l-1}) = f_{l-1}(f_{l-2}(\\dots f_1(x, \\Theta_1)\\dots), \\Theta_{l-1})$$\nThus, the Lipschitz constant $L_l$ of layer $l$ is an implicit function of all parameters up to and including layer $l$:\n$$L_l = L_l(\\Theta_1, \\Theta_2, \\dots, \\Theta_l)$$\n \nFor an arbitrary layer $k$, we take the gradient of $L_{global}$ with respect to $\\Theta_k$. Applying the multivariate chain rule, we have \n$$\\nabla_{\\Theta_k} L_{global} = \\sum_{j=1}^{N} \\frac{\\partial L_{global}}{\\partial L_j} \\nabla_{\\Theta_k} L_j$$\nwhere \n$$\\frac{\\partial L_{global}}{\\partial L_j} = \\prod_{i \\ne j} L_i$$ \nSince the parameters $\\Theta_k$ can only influence its own Lipschitz constant $L_k$ and the Lipschitz constants of subsequent layers $L_j$ ($j > k$), the gradient term $\\nabla_{\\Theta_k} L_j$ is zero for all $j < k$: \n    $$\\nabla_{\\Theta_k} L_{global} = \\sum_{j=k}^{N} (\\nabla_{\\Theta_k} L_j) \\left( \\prod_{i \\ne j} L_i \\right)$$\nSeparating the first term of the sum (where $j=k$) from the rest of the terms, we have\n    $$\\nabla_{\\Theta_k} L_{global} = \\left( \\nabla_{\\Theta_k} L_k \\right) \\left( \\prod_{i \\ne k} L_i \\right) + \\sum_{j=k+1}^{N} \\left( \\nabla_{\\Theta_k} L_j \\right) \\left( \\prod_{i \\ne j} L_i \\right)$$\n\n$\\blacksquare$\n\nTheorem 2 clearly reveals the unique locational influence of Pre-LayerNorm. First, in the Direct Influence Term, the amplification factor has the property of $A_1 > A_k$ for $k>1$, which means that parameters of the first LayerNorm possess a uniquely amplified influence on the global Lipschitz bound, positioning them at a point of maximum leverage for model regularization. We now show why $A_1 > A_k$ holds. The amplification factor  is  $A_1 = \\prod_{j=2}^{N} L_j$ for $k=1$, while it is $A_k = \\prod_{j\\neq k} L_j = (L_1/ L_k) A_1$ for $k > 1$. Clearly,  $A_1 > A_k$ holds if and only if $L_1 < L_k$. Pre-LayerNorm at $k = 1$ receives input directly from the patch embeddings, which have a well-controlled, low-variance distribution. In contrast, a LayerNorm at any deeper layer $k > 1$ receives hidden states that have been progressively amplified by the preceding $k -1$ transformer blocks. These hidden states exhibit higher variance due to the accumulation of nonlinear transformations and residual connections. To handle this increased variance, deeper layers must be more expansive, leading to larger effective Lipschitz constants; that is, $L_1 < L_k$. \n\nSecond, the Cascading Influence Term is also different. For the Pre-LayeNorm ($k=1$), it is a sum of terms describing how changes in $\\Theta_1$ propagate to affect the Lipschitz constants of all $(N-1)$ subsequent layers, $2, \\dots, N$. In contrast, for any deeper layer $k > 1$, it is a sum of terms describing how changes in $\\Theta_k$ propagate to affect the Lipschitz constants of only the $(N-k)$ subsequent layers, ${k+1}, \\dots, N$.\n\n\nIntuitively, changes to the Pre-LayerNorm parameters $(k=1)$ have a direct impact that is amplified by the entire depth of the network, alongside a cascading effect that ripples through every subsequent layer. This provides a powerful, global control mechanism. In contrast, changes to deeper LayerNorm parameters $(k>1)$ have a direct impact that is amplified by a structurally different, and smaller, product of constants. Also, the cascading effect is more localized, influencing only the remaining deeper layers.  \n\n**[Question 1]**\n\nFor the question “*In Table 3, does the results of \"NormFit as an Add-On\" applying Tip-Adapter require extra hyper-parameter searching?*”, the answer is no. The results for “NormFit as an Add-On applied to Tip-Adapter” were obtained using Tip-Adapter’s existing training setup, without any extra hyper-parameter search or special adjustments. This is because NormFit does not introduce new hyper-parameters or external modules of its own: NormFit simply fine-tunes the Pre-LayerNorm parameters already present in the model. In other words, NormFit operates within the original configuration of Tip-Adapter, so no separate tuning was necessary to integrate the two. \n\nAs the paper emphasizes, the performance gains obtained by adding NormFit were achieved without requiring any algorithmic changes, architectural modifications, or additional parameters, underscoring that we did not perform any extra hyper-parameter sweeping for the combined method.\n\n**[Question 2]**\n\nWe have conducted additional experiments on three datasets for the case of 32 samples/class. The results indicate that, with NormFit, increasing the number of samples per class from 16 to 32 continues to improve accuracy. However, the gains beyond 16 samples are relatively modest, suggesting diminishing returns. In the final version of the paper, additional results for 64 samples/class will be included. \n\n**Table: NormFit with increasing number of samples per class on accuracy in few-shot FL ($\\beta$ = 0.01)**\n\n| Dataset  |   1 sample/class   |  4 samples/class  | 16 samples/class  | 32 samples/class  |\n|----------|--------------------|-------------------|-------------------|-------------------|\n| Flowers  |   66.37 ± 0.11     |   86.39 ± 0.41    |   87.83 ± 0.60    |   88.11 ± 0.50    |\n| Food     |   44.50 ± 0.10     |   63.36 ± 0.25    |   75.62 ± 0.19    |   76.09 ± 0.07    |\n| Pets     |   65.76 ± 0.60     |   67.81 ± 0.30    |   88.19 ± 0.29    |   88.45 ± 0.09    |"}, {"review_id": "PNy0GSVQzG", "reviewer": "Reviewer_Rw1u", "summary": "The paper show that finetuning just the pre-LayerNorm parameters of VLMs like CLIP would reduce the communication overhead and computation cost in the problem setting of few shot federated learning, along with the improved results compared to various few shot-based methods in the same line of work. Authors show that focusing the finetuning on pre-LayerNorm parameters improve the model’s attention to object of interest compared to finetuning other LayerNorm parameters in the model. Improved results are shown across 10 different benchmarks like CIFAR10, ImageNet, Caltech etc. They also show that their proposed finetuning can also be seen as add-on to existing works.", "questions": "-\tAs the parameters to be finetuned is model dependent, and the results in Figure 1 show the CLIP ViT-B/32, how does the number of trainable parameters, communication cost and computational cost would look like for other CLIP architectures like ViT-B/16, ViT-L/14, ViT-H/14 in comparison to baselines?\n\n-\tI understand that the problem setting is focused on few-shot FL and thus the choice of 16 samples per class. Does increasing more training examples per class to 32 or 64 still keep the improvement gap of the proposed approach against baselines?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n-\tIt is a simple but effective finetuning strategy utilizing existing parameters in few-shot FL that is shown to outperform existing baselines which introduce additional set of parameters. \n\n-\tThis choice of pre-LayerNorm finetuning shown to greatly reduce the communication overhead and computational cost at the user side compared to previous works. To the best of my knowledge, this is the first work to show that adapting a small set of parameters of a layer would result in improvements and reduce computational cost that turn out to be beneficial in the few shot federated learning setting.\n\n-\tAuthors shown the effect of finetuning only pre-LayerNorm result in semantically aligned attention on object of interest compared to finetuning other LayerNorm layers. I appreciate this analysis as it sheds some understanding on the benefit of finetuning this specific layer.\n\n-\tExperiments are conducted on 10 different benchmarks, compared against various baselines like FedCLIP, FedTPG, FedDAT and FLoRA, and demonstrated improvements over them.\n\n-\tPaper is also easy to read and follow through the storyline.\n\nWeakness:\n\n-\tI don’t find major weaknesses in the paper, as the paper try to understand the effect of pre-LayerNorm through Grad-CAM activations and makes a discussion on the benefits of this specific choice, compare against SOTA methods across multiple datasets, and also conduct ablation study on finetuning additional layer norms. However, I have few questions (please refer to Questions below).", "was_revised": true, "final_justification": "I find that the paper proposed a simple and effective finetuning strategy in few-shot FL that outperform existing baselines. From the rebuttal, the evidence that NormFit keeps its efficiency margin while scaling to a larger backbone is reassuring, and the diminishing-returns analysis for >16 samples/class clarifies when the method is most useful. I believe this work would be of interest to the community for its ability to reduce communication overhead and computational cost while still outperforming few shot FL baselines.", "rebuttal": "**Thank you very much for your thoughtful and constructive feedback.**\n\n**[Question 1]**\n\nWe have conducted additional experiments using a larger CLIP architecture, ViT-B/16, selected for rebuttal analysis due to evaluation feasibility within the review window.\n\nThe table below summarizes the number of trainable parameters, communication cost, and computation cost for NormFit compared to the two best-performing baselines: TIP-Adapter and FLoRA. Given the limited time available during the rebuttal period, we focused on these two baselines because they are the most competitive and meaningful baselines. Specifically, FLoRA is currently the best performing (SOTA) method and also the most lightweight among all baseline methods and TIP-Adapter is the second-best performing method and remains relatively lightweight. \n\nAs shown in the table, NormFit consistently uses dramatically fewer trainable parameters and incurs far lower communication and computation costs.  \n\nIn the final version of the paper, complete results for other baselines and additional results for ViT-L/14 and ViT-H/14 will be included in an appendix. \n\n**Table: Comparison of communication, computation, and parameter overhead in few-shot FL settings with CLIP with ViT-B/16.**\n\n| Method               | Trainable Params (K) | Comm. Cost (KB) | Comp. Cost (GFLOPs) |\n|----------------------|----------------------|------------------|----------------------|\n| TIP-Adapter        | 7.9 × 10²            | 2.4 × 10²        | 1.2 × 10⁶            |\n| FLoRA                | 2.5 × 10¹            | 9.6 × 10¹        | 1.1 × 10⁶            |\n| **NormFit (ours)**   | **1.5**              | **4.1**          | **4.9 × 10²**        |\n\n**[Question 2]**\n\nWe have conducted additional experiments on three datasets for the case of 32 samples/class. The results indicate that, with NormFit, increasing the number of samples per class from 16 to 32 continues to improve accuracy. However, the gains beyond 16 samples are relatively modest, suggesting diminishing returns. In the final version of the paper, additional results for 64 samples/class will be included. \n\n**Table: NormFit with increasing number of samples per class on accuracy in few-shot FL ($\\beta$ = 0.01)**\n\n| Dataset  |   1 sample/class   |  4 samples/class  | 16 samples/class  | 32 samples/class  |\n|----------|--------------------|-------------------|-------------------|-------------------|\n| Flowers  |   66.37 ± 0.11     |   86.39 ± 0.41    |   87.83 ± 0.60    |   88.11 ± 0.50    |\n| Food     |   44.50 ± 0.10     |   63.36 ± 0.25    |   75.62 ± 0.19    |   76.09 ± 0.07    |\n| Pets     |   65.76 ± 0.60     |   67.81 ± 0.30    |   88.19 ± 0.29    |   88.45 ± 0.09    |"}]

Author rebuttal:

Meta-review: The paper proposes NormFit, a minimal PEFT strategy that tunes only the Pre-LayerNorm of the vision encoder in VLMs for few-shot FL with non-IID data. Reviewers find the work technically solid with clear empirical gains and strong efficiency: it matches/exceeds SOTA accuracy while dramatically reducing communication and local compute. Post-rebuttal, two reviews remain at Accept (5) and the third explicitly raised its score after concerns were addressed. No significant ethics issues were noted. Thus, the AC agrees with the reviewers to accept the submission.
Decision: Accept (spotlight)