Title: NormFit: A Lightweight Solution for Few-Shot Federated Learning with Non-IID Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: Federated Learning

Vision-Language Models

Few-Shot Fine-Tuning

Data Heterogeneity

Communication and Computation Cost
Abstract: Vision–Language Models (VLMs) have recently attracted considerable attention in Federated Learning (FL) due to their strong and robust performance. In particular, few-shot adaptation with pre-trained VLMs like CLIP enhances the performance of downstream tasks. However, existing methods still suffer from substantial communication overhead, high local computational demands, and suboptimal performance under non-IID user data. To simultaneously address all those limitations, we propose NormFit, a lightweight solution that selectively fine-tunes only a very small portion of the model parameters, specifically only the Pre-LayerNorm parameters of the vision encoder within a VLM. Overcoming the existing tradeoff between performance and communication/computation efficiency in few-shot FL, NormFit sets a new benchmark by simultaneously achieving superior accuracy and substantially reduced communication and computational demands. Theoretically, we show that NormFit yields a considerably smaller generalization gap compared to tuning all LayerNorm parameters. 
Importantly, NormFit can function effectively as a standalone solution or integrate seamlessly with existing few-shot fine-tuning methods to further enhance their performance. Notably, NormFit offers implementation simplicity, achieving these improvements without any algorithmic modifications, changes to the underlying model architecture, or the addition of external parameters.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper show that finetuning just the pre-LayerNorm parameters of VLMs like CLIP would reduce the communication overhead and computation cost in the problem setting of few shot federated learning, along with the improved results compared to various few shot-based methods in the same line of work. Authors show that focusing the finetuning on pre-LayerNorm parameters improve the model’s attention to object of interest compared to finetuning other LayerNorm parameters in the model. Improved results are shown across 10 different benchmarks like CIFAR10, ImageNet, Caltech etc. They also show that their proposed finetuning can also be seen as add-on to existing works.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n-\tIt is a simple but effective finetuning strategy utilizing existing parameters in few-shot FL that is shown to outperform existing baselines which introduce additional set of parameters. \n\n-\tThis choice of pre-LayerNorm finetuning shown to greatly reduce the communication overhead and computational cost at the user side compared to previous works. To the best of my knowledge, this is the first work to show that adapting a small set of parameters of a layer would result in improvements and reduce computational cost that turn out to be beneficial in the few shot federated learning setting.\n\n-\tAuthors shown the effect of finetuning only pre-LayerNorm result in semantically aligned attention on object of interest compared to finetuning other LayerNorm layers. I appreciate this analysis as it sheds some understanding on the benefit of finetuning this specific layer.\n\n-\tExperiments are conducted on 10 different benchmarks, compared against various baselines like FedCLIP, FedTPG, FedDAT and FLoRA, and demonstrated improvements over them.\n\n-\tPaper is also easy to read and follow through the storyline.\n\nWeakness:\n\n-\tI don’t find major weaknesses in the paper, as the paper try to understand the effect of pre-LayerNorm through Grad-CAM activations and makes a discussion on the benefits of this specific choice, compare against SOTA methods across multiple datasets, and also conduct ablation study on finetuning additional layer norms. However, I have few questions (please refer to Questions below).\n\nQUESTIONS:\n-\tAs the parameters to be finetuned is model dependent, and the results in Figure 1 show the CLIP ViT-B/32, how does the number of trainable parameters, communication cost and computational cost would look like for other CLIP architectures like ViT-B/16, ViT-L/14, ViT-H/14 in comparison to baselines?\n\n-\tI understand that the problem setting is focused on few-shot FL and thus the choice of 16 samples per class. Does increasing more training examples per class to 32 or 64 still keep the improvement gap of the proposed approach against baselines?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces NormFit, a novel and lightweight fine-tuning method for adapting Vision-Language Models (VLMs) like CLIP in the context of few-shot Federated Learning (FL) with non-IID data. The core problem addressed is the trade-off between model performance, communication overhead, and local computation costs, which plagues existing methods. NormFit's key idea is its simplicity and efficiency: it selectively fine-tunes only the Pre-LayerNorm parameters of the vision encoder, while keeping the rest of the VLM frozen.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The proposed method, NormFit, is elegant in its simplicity. While parameter-efficient fine-tuning (PEFT) is a well-established field, the specific choice to tune only the Pre-LayerNorm of the vision encoder is a novel and non-obvious insight. This simplicity makes it easy to implement without any changes to model architecture or complex algorithms.\n2. The paper presents compelling evidence that NormFit resolves the critical trade-off between accuracy and efficiency in few-shot FL. The results in Figure 1 and Tables 1 & 2 are particularly strong, showing that NormFit simultaneously achieves the highest accuracy and the lowest communication/computation costs among all compared methods.\n\nWeaknesses\n1. The experiments are primarily focused on the CLIP model with a Vision Transformer (ViT) or Locked-image Text tuning (LiT) backbone. While these are representative, the paper's claims of general applicability would be stronger if the principle was tested on other VLM architectures or even different model families (e.g., CNN-based vision encoders) to demonstrate broader robustness.\n2. The generalization gap analysis is a significant strength, but it primarily explains why tuning fewer parameters is better. It does not fully provide a theoretical reason for the specific effectiveness of the Pre-LayerNorm layer over other layers, beyond the intuition that it is at an \"early stage\".\n\nQUESTIONS:\n1. In Table 4, does the results of \"NormFit as an Add-On\" applying Tip-Adapter require extra hyper-parameter searching?\n2. Does a larger number of samples per class (>16) achieve higher accuracy in Figure 6?\n3. Can NormFit be transferred to CNN-based encoders?\n\nLIMITATIONS:\nPlease refer to the questions section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes NormFit, a lightweight fine-tuning strategy for the CLIP model in few-shot Federated Learning settings with non-IID data. The core idea is to selectively fine-tune only the Pre-LayerNorm parameters of the vision encoder, while keeping other parameters frozen. This approach significantly reduces communication overhead and local computational demands, while simultaneously achieving superior accuracy, particularly for underrepresented classes in non-IID scenarios. A theoretical justification for a smaller generalization gap is provided, and empirical results on various datasets demonstrate NormFit's effectiveness both as a standalone method and as an add-on to existing fine-tuning techniques.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*:\n\n1. The inclusion of a theoretical analysis on the generalization gap provides a valuable mathematical foundation for why NormFit performs robustly with minimal parameter updates. This strengthens the paper's claims beyond empirical observations.\n2. NormFit achieves improvements without requiring algorithmic modifications, architectural changes, or additional external parameters, enabling seamless integration with existing FL pipelines.\n3. Extensive experiments across diverse datasets and comparisons with state-of-the-art methods (e.g., FedCLIP, FLoRA) demonstrate consistent improvements in accuracy, communication efficiency, and computational cost.\n\n*Weaknesses*:\n\n1. The novelty of the work is incremental. While updating only the Pre-LayerNorm parameters of the vision encoder demonstrates effectiveness, the approach is relatively straightforward and does not constitute a fundamentally new algorithm. Similar strategies have already been applied in \\[1].\n2. The proposed method has limited relevance to key concerns in federated learning such as privacy and security. The focus is solely on communication efficiency in distributed learning frameworks, which weakens the motivation and scope of the study.\n3. The experimental results lack statistical rigor. Only single-run results or average accuracies are reported, without error bars or confidence intervals. Given the stochasticity of data distributions in non-IID settings, this raises concerns regarding the robustness and generalizability of the observed performance improvements.\n4. The baseline methods used for comparison—CoOp and CoCoOp—are outdated, and both involve fine-tuning only the text encoder of CLIP, which differs significantly from NormFit’s strategy. A more appropriate comparison would be with \\[2]. Moreover, comparisons with non-VLM-based federated learning approaches are also needed to better contextualize the contributions.\n5. The authors claim that fine-tuning only the Pre-LayerNorm helps preserve prior knowledge; however, this assertion lacks further analysis or theoretical justification.\n\n\\[1] Tent: Fully Test-Time Adaptation by Entropy Minimization (ICLR 2021)\n\n\\[2] MaPLe: Multi-modal Prompt Learning (CVPR2023)\n\nQUESTIONS:\nAs pointed out in the weaknesses, I have the following main concerns:\n\n1. While fine-tuning only the Pre-LayerNorm parameters demonstrates effectiveness, the approach is relatively simple and lacks strong connections to the federated learning scenario. The paper would benefit from a more detailed discussion on the research motivation, as well as the relationship between the proposed method and key aspects of federated learning such as privacy preservation and data security.\n2. Given the stochastic nature of data distributions under non-IID settings, repeating experiments and reporting error bars or confidence intervals would significantly enhance the credibility of the results.\n3.  The comparison with CoOp and CoCoOp is insufficient, as both methods are outdated and focus solely on fine-tuning the text encoder of CLIP. This differs significantly from NormFit. Furthermore, comparisons with non-VLM-based federated learning approaches are also necessary to better contextualize the contribution of the proposed method.\n4. The authors claim that fine-tuning only the Pre-LayerNorm helps preserve prior knowledge; however, this claim is not supported by experimental evidence. Including an evaluation of the model’s zero-shot performance after fine-tuning would help substantiate this argument and improve the quality of the paper.\n\nAdditionally, the reviewer has a question regarding**Figure B.2**. It appears that, for class 3, no samples were assigned to any of the users. Is this a valid outcome under the considered data partitioning strategy? Alternatively, could it be due to low color contrast in the figure? Clarification on this point would be helpful.\n\nLIMITATIONS:\nThe authors discuss the limitations of the paper to a certain extent, such as the inability to adapt to excessive domain shift and the lack of personalization ability, but do not mention the solution ideas and future research content. The reviewers suggest further discussion on how NormFit could be extended or combined with personalization techniques within FL.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}