Improving Visual Prompt Tuning by Gaussian Neighborhood Minimization for Long-Tailed Visual Recognition

Authors: mengke  li
ye  liu
yang  lu
yiqun  zhang
hui  huang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-02 05:32:57

Abstract: Long-tailed visual recognition has received increasing attention recently. Despite fine-tuning techniques represented by visual prompt tuning (VPT) achieving substantial performance improvement by leveraging pre-trained knowledge, models still exhibit unsatisfactory generalization performance on tail classes. To address this issue, we propose a novel optimization strategy called Gaussian neighborhood minimization prompt tuning (GNM-PT), for VPT to address the long-tail learning problem. We introduce a novel Gaussian neighborhood loss, which provides a tight upper bound on the loss function of data distribution, facilitating a flattened loss landscape correlated to improved model generalization. Specifically, GNM-PT seeks the gradient descent direction within a random parameter neighborhood, independent of input samples, during each gradient update. Ultimately, GNM-PT enhances generalization across all classes while simultaneously reducing computational overhead. The proposed GNM-PT achieves state-of-the-art classification accuracies of 90.3%, 76.5%, and 50.1% on benchmark datasets CIFAR100-LT (IR 100), iNaturalist 2018, and Places-LT, respectively. The source code is available at https://github.com/Keke921/GNM-PT.

Main paper: 
 introduction: 
 Long-tailed visual recognition provides solutions to the challenges posed by the prevalent imbalance and multitude of classes in real-world data. Its training data mirror the real-world distribution, wherein a few categories (head classes) boast abundant samples, while a substantial number of categories (tail classes) exhibit very few samples, conforming to a long-tail distribution [58] . Given its ubiquity and practicality, long-tailed visual recognition has attracted considerable attention, and numerous approaches have been proposed in recent years. Based on the data processing workflow, these methods can be broadly categorized into three types [30] : 1) data manipulation [17, 49, 8, 59, 35] , 2) representation improvement [68, 23, 72, 22] , and 3) model output modification [2, 48, 42, 32, 33] . These methods address the challenge of long-tailed learning from diverse perspectives by extending the traditional training-from-scratch approach. The dataset used is CIFAR100-LT with an imbalance ratio of 100.Recently, leveraging the robust discriminative capabilities of pre-trained models through the integration of multi-head self-attention (MHSA) based networks [54, 11] and parameter-efficient fine-tuning (PEFT) techniques [21, 69, 63, 15, 16] has led to substantial enhancements in model performance on long-tailed data. For example, Tian et al. [52] introduced the text modality by CLIP [46] to aid in visual representation. Dong et al. [10] exploited visual prompt tuning (VPT) to learn class-shared and group-specific prompts for long-tailed data. These methods essentially increase model compatibility. However, even with the assistance of large-scale pre-trained knowledge, PEFT represented by VPT still exhibits inferior generalization performance on tail classes compared to head classes.Chen et al. [5] emphasize that converged ViTs exhibit extremely sharp local minima, hindering their generalization [12] , particularly for tail classes with limited samples. The imperative necessity to improve tail-class accuracy resides in advancing the generalization capability of PEFT, a facet extensively elucidated within the optimization framework [18] . Searching for flat minima by sharpness-aware minimization (SAM) [14] represents a promising optimization technique to improve model performance (as shown in Figure 1 ). SAM first captures the sharpness of loss landscape, which correlates with the generalization gap, based on gradient directions, and then searches for flat minima. Nevertheless, SAM encounters two challenges when applied to long-tail data: 1) flat minima primarily target head classes [71, 70] , and 2) it involves two sequential gradients computation. This paper proposes a novel optimization strategy, named Gaussian neighborhood minimization prompt tuning (GNM-PT), inspired by SAM of flattening the loss landscape to enhance model generalization. Since the widespread usage, the flexibility of prompt and the suitability amount of trainable parameters for visualizing the loss landscape, we select VPT as a representative of PEFT technology to study. GNM-PT shows superior performance than SAM-based methods, particularly targeting long-tailed visual recognition tasks. Specifically, we propose to minimize a novel Gaussian neighborhood loss named Gaussian neighborhood minimization (GNM) to obtain flat minima, substantiated by rigorous theoretical proof. GNM minimizes the mean value of the loss function within the parameter neighborhood, in contrast to the approaches of minimizing the maximum value of the parameter neighborhood employed by SAM [14, 26, 36, 43] . The mean value is a tighter upper bound than the maximum. It is evident from Figure 1 (b) that GNM yields a distinctly pronounced convexity, characterized by relatively lower loss values, thereby leading to a more optimal solution. The calculation is achieved by random sampling from a normal distribution as the perturbation for the training parameters. The proposed GNM equally constrains smoothness optimization through a sample-independent perturbation without extra gradient calculations, which eventually improves model performance for long-tailed data. As shown in Figure 1 (a), GNM, resembling SAM, flattens the landscape of cross-entropy (CE) loss. To further enhance the classification capability of the PEFT methods exemplified by VPT, we harness information from high-level prompts by merging the prompt with the class token for the ultimate classification. We theoretically validate the rationale behind the proposed method. Extensive experiments on benchmark datasets demonstrate that GNM-PT shows great generalization ability on long-tail data, surpassing existing methods. Ablation experiments further prove that GNM improves model performance as well as maintains computational efficiency. Our main contributions are summarized as follows: 1) We identify pressing concerns, explicitly focusing on the imperative need for pretrained models to enhance the generalization abilities across all classes while concurrently mitigating computational time. 2) We propose the efficient GNM-PT, tailored for long-tailed visual recognition based on VPT, which can improve model generalization while minimizing computational overhead. 3) We provide theoretical evidence supporting the superiority of the proposed GNM-PT. Comprehensive experiments also demonstrate that GNM-PT outperforms its state-of-the-art counterparts. 
related work 2.1 dnn-based model for long-tailed learning: 
 Deep neural networks (DNNs) have made significant advancements in long-tail visual recognition in the last few decades. Re-balancing the data distribution, including re-sampling the input data [3, 40, 23, 56, 64] and re-weighting the loss function [13, 49, 8, 24, 44] is the most direct and effective way to improve the performance of the tail classes. Re-margining methods [2, 33, 42, 32, 55] leave larger margins for tail classes than for head classes to improve the separability of tail classes, which can alleviate overfitting and improve model generalization. However, these methods, while improving the performance of tail classes, come at the cost of sacrificing the accuracy in head classes. Ensembling learning encompasses redundant ensembling [57, 1, 27, 29, 22] , which aggregates separate classifiers or networks in a multi-expert framework, and complementary ensembling [68, 6, 61] , which involves statistically selecting different data divisions. Studies have demonstrated that ensembling methods, particularly redundant ensembling, can achieve SOTA performance and generate more robust predictions by reducing model variance [27, 57] and/or increasing data diversity [29, 34, 61] . Additionally, various alternative methodologies within the realm of DNN-based longtailed learning. For example, data augmentation [45, 65] , decoupling representation [23, 66] , logit adjustment [2, 42, 32, 31] , to name a few.Recent advancements in the field of computer vision have harnessed the potential of pre-trained MHSA-Based models, as exemplified by CLIP [46] and the Visual Transformers (ViTs) [21] . In contrast to the conventional practice of training DNNs from scratch, recent proposed PEFT techniques [21, 4, 20] , adopted in RAC [38] , VL-LTR [52] , LPT [10] , and PEL [50] , to name a few, showcase that the meticulous fine-tuning of pre-trained models can yield surprising improvements in the performance of long-tailed visual recognition tasks. For example, VL-LTR employs a contrastive language-image pre-training approach, known as CLIP, and integrates supplementary image-text web data for fine-tuning. LPT fine-tunes a vision Transformer using visual prompt tuning [21] , employing a two-stage training strategy. Nevertheless, it is worth noting that their performance in tail classes still exhibits inferior results compared to that in head classes.Hochreiter and Schmidhuber [18] first pointed out that flat minima corresponds to low network complexity and high generalization performance. Li et al. [28] proposed to visualize the loss landscape and used it to find the flat minima. Subsequently, Foret et al. [14] proposed SAM to seek flat minima and minimize the loss function, so as to improve model generalization capability. Chen et al. [5] demonstrated that ViTs converge at extremely sharp local minima, and they can surpass ResNets in both accuracy and robustness when combined with SAM optimizer. As for long-tailed data, models often showcase varying levels of generalization performance across different classes, with the tail class typically exhibiting inferior performance. Based on this, CCSAM [71] scales the intensity of SAM for classifier inversely with the number of samples available for each class. Zhou et al. [70] observed that, despite the utilization of SAM, the tail classes, due to their substantially smaller sample sizes in comparison to head classes, have limited influence on model parameters. As a result, the loss landscape for tail classes lacks the desired flatness. To address this issue, they proposed ImbSAM, which isolates SAM for head classes and concentrates exclusively on tail classes. Both CCSAM and ImbSAM are designed to bolster generalization capabilities, particularly for tail classes, albeit at the cost of a slight reduction in the performance of head classes.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 Visual Prompt Tuning. In VPT [21] , n p prompt tokens P = [p 1 , p 2 , • • • , p np ] ∈ R np×D are trained to facilitate transfer learning on new datasets with a constraint on the number of learnable parameters, where D is the dimension of tokens in the pre-trained ViT [11] . The prompt tokens encode task-specific information through collaboration with patch representations obtained from ViT blocks. The only parameters that need training are prompt and classification header. There are two variations: 1) VPT-Shallow, which inserts prompts only at the first block, and 2) VPT-Deep, which inserts prompts at all blocks. Take VPT-deep as an example, it is expressed as:EQUATIONwhere p l is the learnable prompt, z l cls is the class token, andZ l = z l 1 , z l 2 , • • • , z lNz represents N z patch tokens. Block l denotes the l-th layer of the pre-trained ViT model.VPT demonstrates notable efficacy in low-data scenarios and maintains its advantages across varying data scales [21] . Despite achieving significant performance improvements, it is noteworthy that VPT exhibits substantial differences in accuracy across different categories. For example, on CIFAR100-LT, the original VPT achieves Top-1 accuracies of 92.11%, 82.86%, and 64.83% for the head, median, and tail classes, respectively. Its generalization ability towards tail classes can be further improved.Sharpness-Aware Minimization. SAM [14] can improve the generalization ability for models by finding an optimal with low curvature. That is, SAM minimizes a specific point and its neighborhoods in the loss landscape of criterion L D (θ) w.r.t data distribution D. It is derived from PAC-Bayesian generalization bound [41] and following [12, 71] , which is, for any ρ > 0 and distribution D, with probability 1 -ρ over a training set T i.i.d sampled from D, the criterion L D (θ) satisfies:EQUATIONwhere h is a strictly increasing function. It can be theoretically substituted by an L 2 weight decay regularizer λ 2 ∥θ∥ 2 2 due to its monotonicity. λ denotes the weight decay coefficient. Foret et al. EQUATIONWhen comparing Equation (3) to the standard training loss, it requires that the maximum loss value of the parameter within the neighborhood of radius ρ centered on θ also remains low. The direction of the gradient of L T (θ) indicates the maximum value of the loss within the neighborhood. Subsequently, for step t, the optimal perturbation vector εt obtained based on the gradient of L T (θ) to obtain L SAM T . The parameters are updated w.r.t. the perturbed model parameters θ + ε:EQUATIONEQUATION)where ρ SAM represents the radius of the parameter neighborhood for SAM, and α t denotes the learning rate scheduled in step t.Despite the effectiveness of SAM and its strong theoretical foundation, it exhibits twofold deficiencies:• For long-tailed data, ε for tail classes is often negligible due to the dominance of head classes with a large number of samples in determining the gradient direction. Consequently, this leads to a challenge in achieving effective generalization for tail classes [71, 70] .• At each step, two gradient computations are required, namely ∇ θ L T (θ t ) and ∇ θ L T (θ t + εt ), resulting in a duplication of the computational overhead.To address the aforementioned issues of head-class dominant optimization and double gradient computation, we propose GNM-PT.To mitigate the presence of sharp minima and enhance the performance of VPT on long-tailed data, we can directly minimize the loss within the parameter neighborhood, thereby attaining a flattened loss landscape. We introduce the Gaussian neighborhood loss L GN T on T , which is defined as:EQUATIOÑ EQUATIONEQUATION)ρ GN M in Equation ( 7) represents the radius of the parameter neighborhood for GNM. The detailed derivation of the gradient for GNM can be found in Appendix A. Figure 2 schematically illustrates a single GNM parameter update.Remark 1. Handling Long-Tailed Data. GNM is better suited for long-tailed data.Proof. If T is a long-tailed training set i.i.d. sampled from D, the direction of gradients in existing methods such as SGD and SAM is predominantly influenced by head classes. (The detailed proof can be found in Appendix B.) Consequently, optimization through Equation ( 4), which is sampledependent, will be determined mainly by the head classes. Conversely, Equation ( 7) is in a sampleindependent manner, avoiding classes with large numbers of samples that dominate the direction of the perturbation vector.Remark 2. Computational Time. GNM saves computational overhead compared to SAM.Proof. Even when disregarding second-and higher-order terms (for example, see Foret et al. [14] , Zhou et al. [71] , Mi et al. [43] for more details), it is apparent from Equation (4) that solving for εt necessitates the computation of one gradient involving a forward and backward pass, while calculating θ t+1 requires another forward and backward pass. As a result, in SAM calculation, which retains only first-order terms, the parameter update already requires an additional forward and backward pass, undesirably doubling the computation time. Conversely, Equation ( 7) mitigates the computational burden and improves the precision of perturbations.Remark 3. Loss landscape. GNM can achieve a flat loss landscape for VPT.Figure 1 and Section 4.4 empirically demonstrate the loss landscape obtained by GNM for VPT is flattened than the original VPT and SAM. Appendix I and Appendix L demonstrate that besides VPT, GNM can also improve other PEFT methods such as AdapterFormer [4] and other backbones such as ResNet based models.Although GNM is not affected by class sizes and improves the generalization performance of each category equally, head-class bias caused by the classifier still exists. Two-stage strategy [56, 23, 10] is effective. Classifier re-balance strategy, including deferred re-weighting/sampling (DRW or DRS) [2] , classifier Re-training (cRT) [23] , nearest class mean classifier (NCM) [23] , to name a few, can be employed. The overall training procedure of GNM-PT is summarized in Appendix D.This section explains GNM from the theoretical perspective. We introduce the following theorem to demonstrate the compactness of the upper bound of the loss function across the distribution D.Theorem 1. For any 0 < δ < 1, and number of samples n ∈ N + , with probability 1 -δ over the training set T sampled i.i.d. from a distribution D, the following generalization bound w.r.t. model parameters θ holds:EQUATIONwhere h : R + → R + is a strictly increasing function.Proof. Based on the condition that adding Gaussian perturbation should not decrease the test error, L D satisfy:EQUATION) By Theorem 2 in Foret et al. [14] and Theorem 1 in Zhou et al. [71] , the Gaussian perturbation satisfy:EQUATION)where k is the dimension of θ. Since log(1 + x) < x holds for all x > 0, Equation ( 11) can be simplified to:EQUATION)The term containing square roots in the above expression is a strictly increasing function. Therefore, by combining it with Equation ( 10), Theorem 1 is proved.Similar to [14, 71] , h in Equation ( 9) can be replaced by L 2 weight decay regularizer λ 2 ∥θ∥ 2 2 . Minimizing L D can be achieved by solving the following GNM problem:EQUATIONHence, the parameter updates given by Equation (7) and Equation ( 8) for GNM can minimize the upper bound given of L D (θ) by Theorem 1. Remark 4. Upper Bound for Loss Function. GNM achieves a tighter upper bound for loss function than SAM.Proof. According to Equation (2), L SAM T is obtained by minimizing the maximum of the loss within the parameter neighborhood of radius ρ SAM . By adjusting the variance σ, L GN T is obtained by minimizing the average value of the loss function within the parameter neighborhood r GN M . It is evident that whenρ SAM ≥ ρ GN M , E ρ SAM [L T (θ + ε)] ≤ max ρ GN M [L T (θ + ε)]. Therefore, L GNT is a tighter upper bound on the loss over D than L SAM T .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Answer: [Yes] Justification: The datasets used in this work are publicly available, and the source code is provided in the abstract.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiment: 
 4.1 Datasets CIFAR100-LT. We adopt the same settings utilized in [8, 2] to establish the long-tailed version by downsampling the original CIFAR100 dataset [25] with different imbalance ratios IR = n max /n min , where n max , n min represent the class sizes of the most and the least frequent classes, respectively. Following [32] , we set the imbalance ratios at 200, 100, 50, and 10.It is artificially truncated from its balanced version, Places365 [67] . The long-tailed version was first created by Liu et al. [37] . Places-LT consists of 62.5K training images with an imbalance ratio of 996.iNaturalist2018. iNaturalist is a substantial real-world dataset that inherently exhibits an exceedingly imbalanced distribution. In our experiments, we leverage the widely employed 2018 version [53] (iNat for short), encompassing 437.5K images across 8,142 distinct species. This dataset features an imbalance ratio of 512.Evaluation Protocol. Following the fundamental assumption that every class carries equal importance, all classes with varying frequencies in the training set are granted an equal number of samples during testing. Top-1 classification accuracy is the primary metric for assessing the performance of various methods. Following Liu et al. [37] , we additionally provide accuracy measurements for three class splits based on the number of training data: Head (> 100 images), Medium (Med for short, 20 ∼ 100 images), and Tail ( ≤ 20 images).Model and Parameter Settings. Following Dong et al. [10] , we employ ViT-B/16 pre-trained on ImageNet-21K and VPT-deep for prompt tuning, and GCL [32] as the loss function. The same data augmentation strategies outlined in [48, 29, 22 ] are adopted, consistent with widely adopted practices among mainstream methods. We employ SGD with GNM as an optimizer and set the batch size to 128, a learning rate of 0.01, accompanied by a cosine learning rate scheduler. For parameter settings of the Gaussian distribution parameters (0, σ 2 ) mentioned in Section 3.2, we exploit the same strategy as Li et al. [32] . Specifically, we set σ = 1 3 meanwhile clamping ε within the range [-1, 1] to ensure that its amplitude remains within one and use a hyper-parameter ρ to control the perturbation magnitude. We adopt DRW for classifier re-balance. Notably, LPT [10] also employs a re-balance strategy during the group prompt tuning stage. For CIFAR100-LT and iNat, we fine-tuned models for 70 epochs, with the final 10 epochs for DRW. For Places-LT, the models undergo a fine-tuning process spanning 100 epochs, with the last 40 epochs for DRW.We compare our proposed GNM-PT with several state-of-the-art methods, broadly categorized into two main types. DNN-based model. We compare with (1) two-stage methods, namely BBN [68] , LWS [23] , MiSLAS [66] ; (2) logit adjustment methods, i.e., GCL [32] and LDAM [2] ; (3) ensembling learning methods, including, RIDE [57] , NCL [29] , and SHIKE [22] ; and (4) contrastive learning, represented by GPaCo [7] . Additionally, we compared two recently proposed SAM-based methods, CCSAM [71] and Imb-SAM [70] , explicitly designed to address longtail data.Recently, MHSA-based models represented by ViT have been employed in long-tail visual recognition. We compare with visual-only methods, including LiVT [62] , VPT [21] , BAL-LAD [39] , LPT [10] , and Decoder [60] . All methods were implemented using ViT-B/16 for a fair comparison. In addition, VL-LTR [52] , RAC [38] and GML [51] are also MHSA-based models which use supplementary data (i.e., text information). We also report the results obtained by these methods for reference.Comparison on CIFAR100-LT. We present the comparison results for CIFAR100-LT in Table 1 .Our proposed GNM-PT exhibits superior performance across all commonly used imbalance ratios compared to the competing methods. Notably, as the imbalance ratio increases, the effectiveness of our GNM-PT becomes increasingly apparent on CIFAR100-LT. Specifically, our proposed method achieves improvements of 1.3%, 1.2%, 1.2%, and 0.8% over the second-best method, namely LPT [10] , for imbalance ratios of 200, 100, 50, and 10, respectively.Comparison on iNat. Table 2 provides results on iNat. The proposed GNM-PT achieves a top-1 classification accuracy of 76.5%, surpassing DNN-based methods. Compared with other visual-only MHSA-based methods that exclusively rely on visual data, our improvement may not be substantial (76.5% However, it is noteworthy that GNM-PT is trained with a relatively small number of epochs (70 and 80 epochs without and with DRW, respectively). In contrast, LPT [10] requires 160 epochs (80 for shared prompt tuning and 80 for group prompt tuning), while LiVT [62] requires 100 epochs.Our proposed method can even achieve results comparable to those with supplementary data. For example, VL-LTR [52] , which requires image-text pairs, achieves an accuracy of 76.8%, only 0.3% and 0.5% higher than GNM-PT with and without DRW, respectively. Notably, the adoption of DRW in GNM-PT has the potential to enhance overall performance, albeit with the trade-off of sacrificing head-class accuracy to bolster tail-class accuracy. This observation may be attributed to suboptimal parameter selection in calculating effective numbers in DRW, an aspect that we plan to delve into in future work. Comparison on Places-LT. From Table 3 , we can observe that GNM-PT continues to outperform existing methods. Similarly to iNat, GNM-PT obtains performance equivalent to LPT with fewer training epochs and outperforms LiVT by nearly 10%. Even when compared to VL-LTR and RAC, which leverage additional auxiliary data, GNM-PT still achieves satisfying performance. Additionally, from Table 3 , it can be observed that DRW improves tail classes at the expense of significant degradation in head classes on Places-LT. While it resulted in an overall improvement of 1%, the head classes decreased by 2%. This indicates that the chosen effective number employed by DRW may not be optimal, warranting further investigation. More results on imageNet-LT can be found in Appendix G.To ensure a fair comparison, the experiments in this section are all executed on the following hardware: Core(TM) i9-13900K, operating at 3.00GHz, equipped with 128GB RAM, and a single NVIDIA GeForce RTX 4090 GPU. The dataset is CIFAR100-LT with IR = 100.GNM vs. SAM. To demonstrate the superiority of GNM, we conducted a comparative analysis with the SAM from two perspectives: classification accuracy and computational efficiency. We employ both CE loss and GCL loss utilizing the CIFAR100-LT dataset with an imbalance ratio of 100. Except for incorporating the optimization of SAM or GNM, all other settings remain identical. Table 4 shows the results. We can observe that SAM entails a computation time exceeding 1.8 times that of the original method compared to the baseline methods without additional optimization technologies. In contrast, GNM incurs only a negligible increase in computation time, namely, less than 2 seconds per epoch. In addition to time savings, GNM also manifests a discernible improvement in accuracy. Despite our straightforward adoption of random perturbation vector ε (as detailed in Section 3.2), its performance is superior to that of the theoretical optimal perturbation vector ε for seeking the maximum loss value within the neighborhood. It is worth noting that all experiments in this section employ the same number of training epochs. SAM and GNM do not affect the convergence speed of the network. The effectiveness across various classes is visualized in Figure 3 . While SAM declines the performance of GCL within the tail classes, our proposed GNM consistently improves performance across all categories in every scenario. Additional comparison results for the long-tailed SAM method can be found in Appendix J and Appendix K. Further comparisons for balanced softmax (BASM) [48] and ResNet-152 backbone can be found in Appendix L Visualization of Loss Landscape. We employ the method in [28] to visualize the loss landscape of the model. Figure 4 shows the results of the learnable prompts obtained by different optimizers. The absence of a perceptible change in flatness explains the marginal improvement of GCL+DRW+SAM over GCL+DRW in Table 4 . In comparison, GNM, by inducing a flattened loss landscape, further accentuates the improvement over GCL+DRW. An unforeseen advantage is that GNM results in a smaller loss, indicating that GNM enhances the model fitting to the training data. A flatter landscape with lower minima contributes to discovering a more optimal solution. By referring to Figure 1 in Section 1, it can be observed that GNM is effective across various loss functions.  
concluding remarks: 
 In this paper, we observed that class biases persist even when employing large-scale pre-trained models such as VPT in long-tail learning. While SAM can enhance the generalization performance of the VPT model on long-tailed data, it still exhibits several shortcomings: neglecting higher-order terms leads to a suboptimal perturbation vector, additional forward and backward passes double the computational time, and the generalization is relatively affected by gradients predominantly originating from head classes. Based on this, we have proposed GNM-PT, which involves fine-tuning pre-trained models using the innovative Gaussian Neighborhood Minimization (GNM) optimizer. GNM leverages random noise as a substitute for gradients in the initial step of SAM. The proposed GNM not only balances the generalization capabilities of both head and tail classes but also reduces computational time. To fully leverage model information, enhance classifier robustness, and enable end-to-end training, we additionally employ merging prompt strategy. We have conducted extensive comparative experiments and ablation studies to demonstrate the effectiveness of the proposed method and the individual component.While GNM-PT proves effective, it is not exempt from limitations. Table 2 and Table 3 show that we still need to further re-balance the classifier. However, the re-balanced strategy adopted compromises performance in head classes to enhance overall performance. Our further research will focus on a more effective optimization strategy that simultaneously improves feature representation and classifier performance, while also enhancing the generalization ability across all classes.We replace the expectation computation in L GN T with random sampling during training. For a single gradient computation, we have:EQUATIONFor simplicity, we omit the subscript indicating the training epoch t.If T is a long-tail training set i.i.d. sampled from D, the direction of gradients in existing methods such as SGD and SAM is predominantly influenced by head classes. For the most widely adopted SGD, a lot of previous works, such as Wang et al. [55] , Hsieh et al. [19] , Li et al. [31] have theoretically and empirically demonstrated that the gradient for head classes far exceeds that of tail classes. Therefore, the optimization is dominated by head classes. Here, we provide proof establishing that the perturbations within SAM are dominated by head classes.Proof. For SAM, we analyze the perturbations class by class. Through Eq. ( 4), we have:EQUATIONThe ρ1 ∥∇ θ L T (θ)∥ 2 2can be considered to be the same value V for both the head and tail classes. Wang et al. [55] , Li et al. [31] have theoretically and empirically shown that the gradient for head classes far exceeds that of tail classes, that is, ∇ θ L T Tail ≫ ∇ θ L T Tail . Therefore, the perturbation obtained based on tail classes can be ignored:EQUATIONConsequently, SAM optimization tends to prioritize generalization for head classes.In contrast, the perturbations obtained by GNM are:EQUATIONThis perturbation remains unaffected by the input samples and their quantities.C Robust Training strategy for Prompt Tuning.In Eq. ( 1), only z l cls is fed into the linear classifier for classification. However, all the learnable prompts are trained on the fine-tuning dataset and thus contain newly learned information. As the deep block output incorporates global attention [11] , we propose merging prompt (MP) that merges the last prompt with z L cls (assuming that we have L blocks). Subsequently, we utilize this merged token as the ultimate class token:EQUATIONwhere w p and w z are hyper-parameters used to control the merging ratio. ẑcls is eventually fed into the linear classifier for classification. We use this Merge Prompt technique in our experiments.The training procedure for GNM-PT is outlined in Algorithm 1. To validate the effectiveness of each proposed component, we conducted an ablation study using the CIFAR100-LT dataset with an imbalance ratio of 100, employing GCL loss. For "Merge" in Eq. ( 19), we employ addition and assign equal weights to both w p and w z , setting them to 0.5. The summarized results are presented in Table 5 . Incorporating DRW has been observed to enhance overall performance, resulting in a performance improvement of 1.07%. Additionally, GNM-PT derives benefits from the design choices made in each individual component. The proposed merge prompt and GNM optimization further improve the performance from 89.32% to 89.58% and 90.28%, respectively. Similar to Place-LT, imageNet-LT is also artificially truncated from its balanced version, namely ImageNet [9] and its long-tailed version is created by Liu et al. [37] . ImageNet-LT comprises 115.8K training images spanning across 1,000 categories, with an imbalance ratio of 256.We report the accuracy in Table 6 . Considering that the model pre-trained on ImageNet-21K contains information about ImageNet-1K, namely the balanced of ImageNet-LT, we compare GNM-PT only with the methods using the same pre-trained models. GNM-PT achieves superior performance, attaining an 80.4% top-1 classification accuracy, outperforming GML and VPT with a notable margin of 2.4% and 3.2%, respectively. Furthermore, GNM-PT surpasses competing methods across all scale classes, demonstrating its outstanding performance. In the comparison experiment with SAM, we used the radius (r SAM ) recommended by the paper in SAM, which is 0.05. For GNM, we set the amplitude (a) for Gaussian noise based on the radius in SAM, that is, the actual radius (r GN M ) used in GNM is ρ GN M = a × ρ SAM . We use a = 0.1 in all experiments. We conducted the ablation study towards the hyper-parameter a in GNM using CIFAR100 with an imbalance ratio of 100. The results are listed in Table 7 . When α → 0, the interference becomes negligible, effectively restricting the loss function to attain its minimum value within a small area. The extreme case is α = 0, meaning no additional optimization techniques are employed. Therefore, the smaller α is, the less pronounced its effect. When α increases: the disturbance area expands. A large α introduces significant perturbations, potentially deviating from the basic gradient descent path. Excessively large values of α, for example α = 2, lead to performance degradation. In the extreme case of α = ∞, the model fails to converge.Table 8 presents ablation studies on the different choices of the variance of the Gaussian distribution. Additionally, we include the results of using a uniform distribution within the range of [-1,1]. The results indicate that the variance impacts model performance. This finding demonstrates that, in addition to the amplitude of ε, the distribution also influences model performance and is worthy of further study. AdapterFormer [4] is one of the recently proposed PEFT techniques. We show the efficacy of GNM when applied to AdapterFormer. The results are summarized in Table 9 . In contrast to prompt tuningbased approaches, GNM demonstrates relatively modest improvements on AdapterFormer. The reason is that, compared to prompt tuning methods, AdapterFormer has fewer learnable parameters, comprising 1.01M and 0.18M parameters, respectively. Table 10 and Table 11 compare the SAM-based method for long-tailed data, that is, CC-SAM [71] . GNM consistently enhances generalization performance across each class compared to CC-SAM.Overall, GNM can serve as an optimization method that not only enhances the performance of the VPTbased model but also improves the performance of CNNs trained from scratch or with full fine-tuning. Tables 12 and 13 present the comparison of existing optimization strategies and our GNM under the same training paradigm. We implement the experiments using CIFAR-100-LT with an imbalance ratio of 100 and take fine-tuning pre-trained ViT-B/16 by VPT with GCL loss as a base training paradigm, named Base. The re-balancing strategy employed in the second stage can influence the performance of optimization methods on a per-class level. We conducted a comparative analysis for various optimization methods, both without and with the application of the rebalancing strategy and exhibit the result in Tables 12 and 13 , respectively. DRW is utilized as the re-balancing strategy. LPT also employs two stages that include a re-balance strategy, therefore we present it in Table 13 .As observed from Table 12 , without the re-balancing strategy to adjust the classifier bias, imbSAM achieves better overall accuracy. However, ImbSAM has little impact on the head and middle classes. Additionally, both CCSAM and ImbSAM require two back propagations, thereby doubling the computation time. Compared to VPT with GCL, which does not include additional optimization, GNM incurs only a small computational overhead.Table 13 shows that the re-balancing strategy sacrifices a small amount of head class performance in exchange for significantly improving tail class performance. imbSAM essentially does not employ additional optimization for head classes, whereas CCSAM and GNM use additional optimization for all classes. However, compared to imbSAM, CCSAM and GNM result in a greater reduction in the accuracy of head classes. The optimization strategies may have a limited impact on the rebalancing training process. We will investigate this in detail in future work.Besides, we provide a detailed analysis of why GNM-PT cannot outperform imbSAM in the first stage but outperform in the second stage as shown in Table 12 and Table 13 . Stage 1 of ImbSAM already applies strong regularization to tail classes, the rebalancing strategy in stage 2 essentially duplicates this effect. In contrast, GNM applies the same level of regularization to all classes without specifically intensifying it for tail classes and thus the strong regularization for tail classes can still work in stage 2. Table 14 presents the results of applying GNM with balanced softmax (BASM) [48] . The backbone is ResNet-152 pretrained on imageNet-1k. GNM exhibits improvements across all classes, especially tail classes. Consequently, the overall performance is enhanced in comparison to SAM.Figure 6 presents the visualization results for the places-LT trained with Resnet-152. Although, the performance differences are small as shown in Table 14 , it can still be observed that SGD and SAM exhibit steeper gradients and some irregular protrusions. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Section 4.1 and Appendix H.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Section 4.3.2 and Section 4.4.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The benchmark datasets used in this paper are publicly available. The original paper that produced the code package and the dataset are properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The source code is temporarily available at the anonymized URL provided in the abstract. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.The loss landscape for the background adheres to the same settings in Figure2of SAM[14].The results are obtained with the assistance of textual data.The result is obtained without DRW.The results are obtained by reproducing the original paper with the recommended settings.The results are obtained with the assistance of textual data. 
 
 Appendix:  


Decision: This paper proposes an optimization strategy, namely Gaussian neighborhood minimization prompt tuning, to tackle the long-tail classification problem. The proposed method employs the Gaussian neighborhood loss to provide a tighter upper bound on the loss and facilitate a flattened loss landscape. It seeks the gradient descent direction within a random parameter neighborhood, and thus reduces the computational overhead compared to sharpness-aware minimization.

Initial concerns raised by reviewers include that the effectiveness of some key designs is not well supported, the evaluation of the loss landscape is insufficient, there lacks several aspects in the experimental comparisons, and the influence of hyperparameters is not analyzed, The authors’ rebuttal well addressed most of these concerns and all reviewers were positive in their final recommendations. The AC agrees with the reviewers that the proposed method is simple yet effective, supported by theoretical analysis. Therefore, the AC recommends to accept. Reviewers did raise valuable concerns that should be addressed. The camera-ready version should include the discussions and results provided in the rebuttal.

Reviews: review: paper_summary: paper_summary: This work addresses the long-tailed learning problem by adding a tight upper bound on the loss function of data distribution and improving the generality of the model through flattening the loss landscape.
strength_weakness: strength_weakness: 1. The authors mention that GNM flattened the loss landscape and thus improved model generalization. I doubt this conclusion, since the flatness improvement from the proposed approach is not significant enough. Specifically, the original landscape is already flat enough in Figures 1 and 4 (i.e., compared with the original bumpy landscape in Figure 1 of [ref1] and Figure 1 of [ref2]). The original approach already possessed the ability to achieve generality without needing any more skills, and thus the GNM seems to contribute little. Hence, it comes to the confusion: does GNM really help the model improve generalization through this perspective (flattening the loss)?

2. I hope to have more quantitative metrics about the loss landscape. Since the only value in Figures 1 and 4 is loss, which only means lower loss buttum with higher accuracy (i.e., it is hard to compare the convexity of various approaches since they are both very flat), Could the author measure the level of convexity? For example, you can compute the eigenvalues of the Hessian following [ref2].

3. I noticed that the authors only chose to visualize the loss landscape of CIFAR100-LT. Could you clarify why you only chose this one? And does it have a similar phenomenon in the other two datasets? Could you provide more visualization results in the appendix?

[ref1] Sharpness-aware minimization for efficiently improving generalization. ICLR 2021
[ref2] Visualizing the Loss Landscape of Neural Nets. NeurIPS 2018
questions: questions: See "Weaknesses”
limitations: limitations: The authors included the limitation section.
paper_summary: paper_summary: The paper proposes an optimization approach called Gaussian neighborhood minimization prompt tuning (GNM-PT) for long-tailed visual recognition. Compared to sharpness-aware minimization (SAM), it excels in lower computational overhead, tighter upper bound for loss function and superior performance. GNM-PT utilize Gaussian neighborhood instead of the gradient direction of current parameters employed in SAM.
strength_weakness: strength_weakness: 1, For the comparative method LDAM [1] mentioned in the paper, the authors did not provide experimental results.
2, The experimental results are insufficient. Specifically, this paper investigates the optimization strategies for long-tailed recognition under parameter-efficient fine-tuning. Accordingly, existing optimization strategies should be compared under the same training paradigm (i.e., PEFT) and the same backbone. For example, in Tables 1 to Table 3, the performance of “LPT+SAM”, “LPT+CCSAM”, and “LPT+IMBSAM” should be provided to illustrate the outstanding superiority of the proposed GNM-PT.
3, I wonder whether the variance of the Gaussian distribution has an impact on the results. The authors should provide ablation studies on the different choices of variance.
4, As illustrated in Section 3.2, GNM-PT improves the generalization performance of each category equally. However, the results presented in Table 2 and Table 3 show that GNM-PT achieves inferior performance than the baseline LPT for tail category, please explain the reason.
5, There are some typos: In the explanation of Eq. (2), what do “k” and “n” represent?

[1] Cao K, Wei C, Gaidon A, et al. Learning imbalanced datasets with label-distribution-aware margin loss[J]. Advances in neural information processing systems, 2019, 32.
questions: questions: See weakness.
limitations: limitations: GNM-PT method must collaborate with the rebalancing strategy to ensure overall performance across all classes.
paper_summary: paper_summary: This paper proposes Gaussian neighborhood minimization (GNM) to enhance prompt tuning methods in long-tailed recognition. GNM is inspired by a recent work SAM, which aims to achieve flat minima by capturing the sharpness of loss landscape. Theoretical evidence shows that GNM can achieve a tighter upper bound and optimize the loss to a lower value. Experiments on multiple long-tailed datasets demonstrate that GNM can improve the performance of both head and tail classes and can reduce the computational cost compared to SAM.
strength_weakness: strength_weakness: 1. In Tables 1-3, the SAM-based methods are equipped with DNN-based model, which is unfair when compared to GNM-PT. Have you experimented the SAM-based methods with MHSA-based models such as ViT-B/16?
2. Since you have already used the GCL loss function, why do you use DRW? Will this make the learning process overly focused on tail classes?
3. The performance gain is marginal in some datasets, for example, when compared to LPT on iNaturalist and Places-LT. There may be space for further refinement.
questions: questions: See weaknesses above.
limitations: limitations: The authors have discussed the limitations in their paper.
paper_summary: paper_summary: The author proposes a novel optimization strategy, Gaussian neighborhood minimization prompt tuning (GNM-PT), for VPT under long-tailed distribution. This method is improved based on the SAM optimizer. Specifically, during each gradient update, GNM-PT searches for a gradient descent direction within a random parameter neighborhood, independent of the input sample, which eliminates the influence of imbalance. Further experiments show the effectiveness of the method.
strength_weakness: strength_weakness: 1) It is not clear why the author applied GNM to VPT and not to adapterformer or lora or any other PEFT-based method or even training from scratch under CNN based model. All of these also face the problem of local minimum.
2) How to optimize under long-tailed distributions has been explored [a][b]. The authors should experiment with them directly under VPT and compare with GNM-PT and discuss their performance and training costs.
3) I admit that the flat minimum is mainly for the head class, but can we apply loss reweighting or logit adjustment[c] to alleviate this?
4) The author has done the experiment with the hyper-parameters in Appendix.G, but does not give any analysis of how this parameter influences the performance.

[a] Class-conditional sharpness-aware minimization for deep long-tailed recognition, CVPR

[b] A closer look at sharpness-aware minimization in class-imbalanced recognition, ICCV

[c] Long-tail learning via logit adjustment, ICLR2021
questions: questions: see weakness
limitations: limitations: yes

Comments: title: Response to Authors
Reminder -- please reply to rebuttal
Reminder -- please reply to rebuttal
comment: Dear Reviewer X9cb,

We are glad to hear that most of your concerns have been addressed. Thank you for your thoughtful consideration. Your positive rating is greatly appreciated. We hope that our paper can meet your expectations.

Should you have any further concerns, please let us know and we are more than willing to engage in a detailed discussion before the discussion phase deadline. We are committed to addressing all feedback to the best of our ability to ensure the paper aligns with the high standards of the conference.

Best regards,

Authors
Thanks for the rebuttal, and most of my concerns are addressed. I will keep my positive rating to this paper.
Dear Reviewer ZADq,

Thank you again for reviewing this paper. Since the reviewer-author discussion phase is closing soon, could you please read through all the reviews and see if the authors' responses have addressed your concerns?

Best,

AC
Dear Reviewer X9cb,

Thank you again for reviewing this paper. Since the reviewer-author discussion phase is closing soon, could you please read through all the reviews and see if the authors' responses have addressed your concerns?

Best,

AC
Dear Reviewer veiW,

We sincerely appreciate your thorough review and recognition of our work. Your valuable suggestions will significantly contribute to improving the quality of our paper.

Best regards,

The Authors
**(Continued from previous)**

--- 
> Q3. Two-stage training procedure is tedious.

** A3:** The two-stage method is one of the most widely used training strategies for long-tail learning [R1-R7]. It requires only a few additional epochs for the classifier to be trained and does not introduce significant overhead. DRW [R1] can even be trained end-to-end without requiring additional computational parameters, time, or other resources.

--- 
> Q4. GNM-PT cannot surpass imbSAM in the first stage.

**A4: ** We admit the necessity of a rebalancing strategy for GNM-PT is the limitation and have discussed it in the concluding remarks.

GNM-PT does not apply strong regularization to tail classes, resulting in less improvement for these classes compared to imbSAM for one-stage training. Therefore, we apply an additional re-balance stage for strong regularization to tail classes and for further balancing the classifier.  GNM-PT demonstrates improvements across all class scales in Stage 1 and surpasses imbSAM after Stage 2.

Computational overhead is also an important metric to consider. Although GNM-PT is slightly inferior to imbSAM in stage 1, it is worth noting that imbSAM requires 88.97 seconds per epoch due to the need for two forward and backward propagations. In contrast, GNM only requires 42.77 seconds per epoch, which significantly reduces the computational burden, as analyzed in Remark 2 (lines 183-190). 

Reference:

[R1] K. Cao, et al., Learning imbalanced datasets with label-distribution-aware margin loss, in NeurIPS 2019.

[R2] B. Kang, et al., Decoupling representation and classifier for long-tailed recognition, in ICLR 2020.

[R3] Z. Zhong, et al., Improving calibration for long-tailed recognition, in CVPR 2021.

[R4] M. Li, et al., Long-tailed visual recognition via gaussian clouded logit adjustment, in CVPR 2022.

[R5] B. Dong, et al., LPT: long-tailed prompt tuning for image classification, in ICLR 2023.

[R6] J.X. Shi, et al., How Re-sampling Helps for Long-Tail Learning? in NeurIPS 2023.

[R7] M. Li, et al., Feature Fusion from Head to Tail for Long-Tailed Visual Recognition, in AAAI 2024.
Thank you for your time and continued discussion. Below, we provide our responses to your comments and hope they address your concerns.

---
> Q1. Analysis for GNM-PT cannot outperform imbSAM in the first stage but outperform in the second stage.

**A1:** In brief, stage 1 of imbSAM already applies strong regularization to tail classes, the rebalancing strategy in stage 2 essentially duplicates this effect. In contrast, GNM applies the same level of regularization to all classes without specifically intensifying it for tail classes and thus the strong regularization for tail classes can still work in stage 2.

In detail, the rebalancing strategy can be viewed as a strong regularization for tail classes. ImbSAM filters the gradient restrictions on head and median classes and only exerts perturbation on tail classes. As shown in Table R5, the performance gain of imbSAM in stage 1 primarily comes from tail classes, while the head classes see a slight improvement, and the median classes even exhibit a decline in performance. In stage 2, the rebalancing strategy that focuses on training tail classes duplicates the effect of stage 1, resulting in relatively smaller performance improvements.

Differently, GNM imposes constraints on minimizing the optimal point and its neighborhood of the loss function to all classes equally. It comprehensively improves the performance across all classes. Compared to imbSAM, GNM applies balanced regularization across all classes. Therefore, adding rebalancing can enhance overall performance by significantly improving the performance of the tail class. 

---
> Q2. Why the GNM-PT can outperform previous SAM-based methods.

**A2:** In brief, for long-tailed learning, SAM tends to favor head classes during optimization, CCSAM and imbSAM are biased towards optimizing tail classes, while GNM offers a balanced optimization across all classes.

For GNM: The optimization constraint in GNM is sample-independent, preventing classes with large sample sizes from dominating the direction of the perturbation vector. GNM can effectively improve all classes, avoiding optimization biases towards head classes and ensuring that the optimization focus is not solely on tail classes. Table R5 supports this as well.

For SAM: The head class dominates gradient optimization as the perturbations are dependent on class size. Remark 1 (lines 176-182) and Appendix B (lines 528-542) provide a detailed analysis and proof.

For CCSAM: It scales the perturbations in SAM utilizing a weight inversely proportional to the class sizes.
For imbSAM: It directly filters out the perturbations from the head class and median classes in SAM, focusing solely on adding smooth constraints to the sharpness of tail classes. 
Both of these two methods prioritize the sharpness constraint of tail classes while overlooking the other classes.

In addition, GNM has another significant advantage: it requires negligible computational overhead due to its single forward and backward pass per gradient update. In contrast, other SAM-based methods require an additional forward and backward pass per gradient update, and thus double the computation time. Remark 2 (lines 183-190 in the paper) provides a detailed analysis.
I am glad to learn our discussion is helpful. I have made my final rating. Good luck!
Thanks for the response. Part of my problems have been solved. However, I find that GNM-PT cannot outperform imbSAM in the first stage but outperform in the second stage. The author may give some analysis of this phenomenon. Furthermore, why the GNM-PT can outperform previous SAM-based methods also needs to be analyzed to help us understand your method.

In addition, the two-stage training procedure is tedious, and it is unsatisfactory that GNM-PT cannot surpass imbSAM in the first stage.
Dear Reviewer veiW,

We would like to thank you again for your precious time and the insightful recommendations. In addition to the revisions already made, we will incorporate a detailed analysis and discussion of these two important works relevant to VPT.  

Moreover, please let us know if you have any further concerns. We are more than willing to discuss them with you and will address all concerns to the best of our ability to ensure the paper meets the high standards of the conference.

Best regards, 

Authors
Thank you for the detailed responses. My concerns have been successfully addressed. 

In additional to my existing comments, I think it would be beneficial for the authors to include a discussion of [1][2] in the revision. [1] approaches VPT by considering the Transformer architecture, while [2] provides a thorough analysis of VPT and its usability.

[1] E^ 2VPT: An Effective and Efficient Approach for Visual Prompt Tuning;
[2] Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?