Parameter-Efficient Complementary Expert Learning for Long-Tailed Visual Recognition

Authors: lixiang  ru
xin  guo
lei  yu
yingying  zhang
jiangwei  lao
jian  wang
jingdong  chen
yansheng  li
ming  yang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-01 11:11:30

Abstract: Long-tailed recognition (LTR) aims to learn balanced models from extremely unbalanced training data. Fine-tuning pretrained foundation models has recently emerged as a promising research direction for LTR. However, we observe that the fine-tuning process tends to degrade the intrinsic representation capability of pretrained models and lead to model bias towards certain classes, thereby hindering the overall recognition performance. To unleash the intrinsic representation capability of pretrained foundation models, in this work, we propose a new Parameter-Efficient Complementary Expert Learning (PECEL) for LTR. Specifically, PECEL consists of multiple experts, where individual experts are trained via Parameter-Efficient Fine-Tuning (PEFT) and encouraged to learn different expertise on complementary sub-categories via a new sample-aware logit adjustment loss. By aggregating the predictions of different experts, PECEL effectively achieves a balanced performance on long-tailed classes. Nevertheless, learning multiple experts generally introduces extra trainable parameters. To ensure parameter efficiency, we further propose a parameter sharing strategy which decomposes and shares the parameters in each expert. Extensive experiments on 4 LTR benchmarks show that the proposed PECEL can effectively learn multiple complementary experts without increasing the trainable parameters and achieve new state-of-the-art performance.

Main paper: 
 introduction: 
 Real-world data usually conforms to a long-tailed distribution, where a minority of classes (i.e., head classes) cover the majority of samples, while the other classes (i.e., tail classes) comprise only a few samples. Such data imbalance poses a formidable challenge for learning long-tailed recognition (LTR) models, since the learned models are prone to be biased towards the head classes with most of data but exhibit poor performance on other classes. To address this challenge, researchers have proposed many approaches [17, 23, 36, 41, 51, 54] to learn competent models from unbalanced long-tailed training data.Existing LTR approaches can be broadly categorized into training from scratch methods and fine-tuning methods. Training from scratch models are trained from randomly initialized models, and usually exhibit unsatisfactory performance [6, 55] . In contrast, LTR methods via fine-tuning [10, 16, 34, 43, 46] mitigate the data imbalance issue by adapting pretrained foundation models [24, 38, 40] . Foundation models are typically trained on extensive data, and inherently learn generalizable representations across different semantic categories [24, 40] . Therefore, LTR methods via fine-tuning foundation models often yield higher accuracies [34, 46] . For example, BALLAD [34] , VL-LTR [46] and UDCPG [16] fine-tune the visual and linguistic branches of CLIP [40] for LTR. To accelerate the training process and prevent undermining the representation of pretrained models in full fine-tuning, LPT [10] and LIFT [43] Figure 1 : Accuracy of each class in ImageNet-LT [9] by Zero-Shot recognition, Full Fine-tuning, PEFT (AdaptFormer [5] ) and our proposed method. Here we use the pretrained CLIP-ViT-B [40] . For better visualization, the accuracy values are smoothed. Best viewed in color.introduce Parameter-Efficient Fine-Tuning (PEFT) methods to LTR and achieve notable improvements. However, we empirically observe that full fine-tuning and PEFT on long-tailed data tend to degrade the inherent representation capability of pretrained models. To verify this, in Figure 1 , we present the accuracy values of each class in ImageNet-LT [9] , which are obtained by zero-shot recognition, full fine-tuning, PEFT (Adapt-Former [5] ) and our proposed method, respectively. The class indices are sorted according to the class frequencies. Figure 1 shows that zero-shot recognition can achieve a relatively balanced performance on different class groups, i.e., many-shot (with > 100 samples), medium-shot (with 20-100 samples) and few-shot classes (with < 20 samples), suggesting the pretrained model learns satisfactory representations for different classes. Though full finetuning can achieve favorable accuracy on the many-shot classes, the performance on the medium and few-shot classes is inferior to zero-shot recognition, indicating the representation capability on these classes is degraded due to fine-tuning on long-tailed data. Similarly, PEFT exhibits a lower accuracy than zero-shot recognition on the few-shot classes, which indicates the representations of these classes are also hindered and become less discriminative.To unleash the potential of pretrained foundation models, in this work, we propose the Parameter-Efficient Complementary Expert Learning (PECEL) for LTR. Specifically, PECEL learns multiple expert models from the pretrained models via PEFT, which are encouraged to attain different expertise on complementary classes. To ensure this, we propose a sample-aware logit adjustment (SLA) loss, which adjusts the learning difficulty of individual experts on different classes and samples. Thus, an expert only needs to focus on learning certain classes and pay less attention to other classes. By aggregating the predictions of these experts, PECEL can achieve a balanced performance across different classes. Besides, directly learning multiple experts generally increases the number of trainable parameters by multiple times. To alleviate this, we propose a parameter sharing strategy. For each expert, the parameters in different PEFT blocks are decomposed as shared components and low-dimensional separate components. By distributing the shared components across different blocks, each expert only needs to maintain a small number of trainable parameters. To demonstrate the efficacy of PECEL, we conduct extensive experiments on the CIFAR100-LT [26] , ImageNet-LT [9] , Places-LT [57] and iNaturalist 2018 [47] . As shown in Figure 2 , our proposed PECEL can perform on par with the state-of-the-art methods with notably fewer trainable parameters, and achieve higher accuracy with equivalent numbers of parameters.To summarize, our contributions in this work include the following aspects.• To the best of our knowledge, the proposed PECEL is the first work to effectively address the representation degradation issue when adapting foundation models for long-tailed recognition. [31, 39, 42] or balancing the loss of different classes with re-weighting [6, 8, 13, 41] and re-margining [4] . Logit adjustment methods [28, 36, 56] sought to alleviate the class imbalance by incorporating dedicated biases or weights into the output logits. Representation learning methods [7, 22, 37, 58] aimed at learning balanced and discriminative representations from the long-tailed data via metric learning [35, 50, 53] or contrastive learning [18, 23, 29, 58] . Ensemble learning-based LTR methods [1, 45, 51, 54, 55] usually learned multiple diverse expert models, and aggregated the outputs of different experts. Specifically, RIDE [51] proposed a routing module and selected the most appropriate expert in testing time. SADE [54] trained complementary experts and adaptively aggregated the predictions to handle different data distributions. MDCS [55] and LGLA [45] aggregated skill-diverse experts to achieve more balanced performance. Compared to recent LTR methods that fine-tuned pretrained models [16, 43, 46] , training from scratch methods usually exhibited inferior performance. Fine-tuning Pretrained Models. Enlightened by the unprecedented success of foundation models [24, 38, 40] , recent LTR methods [16, 34, 46 ] also resorted to pretrained foundation models to tackle the data imbalance. For example, BALLAD [34] trained the visual and linguistic encoder of CLIP [40] with text prompts and input images, then fine-tuned a linear adapter on the balanced training samples. Tian et al. proposed VL-LTR [46] , which aligned the images and text embeddings via contrastive learning, and trained the LTR classifier with cross-modality attention. GML [44] tackled LTR as a mutual information maximization problem [3] between the latent embeddings and ground-truth labels. UDCPG [16] addressed the feature collapse in training by aligning the image and text features with prototype-based contrastive learning. To improve the training efficiency and preserve the inherent representation capability [10, 43] , some LTR methods also explored PEFT techniques for LTR. LPT [10] adopted two-stage visual prompt tuning [21] to further empower the pretrained model's capability on fine-grained discrimination. LIFT [43] proposed to integrate PEFT modules (such as AdaptFormer [5] ) and designed a simple yet highly effective framework for LTR. As illustrated in Figure 1 , we empirically observed that PEFT still degraded the representation capability of pretrained models. In this work, we propose to learn multiple complementary experts via PEFT to alleviate this issue and further unleash the potential of pretrained foundation models for LTR. 
parameter-efficient fine-tuning: 
 Parameter-efficient fine-tuning (PEFT) aims to relieve the training and storage cost of large pretrained models [38, 40] by only updating a small number of parameters while keeping other parameters frozen [30] . VPT [21] proposed to freeze the pretrained backbone and prepend some extra trainable prompt tokens to fine-tune visual recognition tasks. LoRA [20] proposed to inject low-rank trainable matrices to each layer in the backbone. To further explore parameter efficiency, VeRA [25] proposed to decompose the projection matrix in PEFT modules into a fixed random part and a learnable part. Different from LoRA-style methods that re-parameterized the projection weights, Adapter [19] inserted a bottleneck module after the feed-forward network (FFN) of Transformer blocks [48] , which only comprised simple linear projections and nonlinear layers. AdaptFormer [5] extended the sequential design of Adapter to parallel and added an extra scaling factor, which demonstrated better performance. However, existing PEFT methods typically maintain separate parameters in each block, which will remarkably increase the number of trainable parameters when learning multiple experts for LTR. To address this issue, in this work, we propose a parameter sharing strategy by decomposing and sharing the parameters in different PEFT blocks. 
methodology: 
 In this section, we first briefly introduce some preliminary knowledge, and then present the overall framework of the proposed Parameter-Efficient Complementary Expert Learning (PECEL). How to learn multiple complementary experts and reduce the number of trainable parameters in each expert model are elaborated in Section 3.3 and 3.4, respectively. Finally, the overall training objective and analysis on the number of trainable parameters of PECEL are illustrated in Section 3.5 and 3.6, respectively.Logit Adjustment. Logit adjustment [36] tackles the class imbalance issue by infusing class-level biases (i.e., relative class frequency) into the predicted logits in the training process. Let z ∈ R 𝐶 denotes the model predicted logits, where 𝐶 denotes the number of classes, the logit adjustment loss is calculated as:EQUATIONwhere y ∈ R 𝐶 denotes the one-hot ground-truth label, b 𝑖 is the relative frequency of class 𝑖. Intuitively, Eq. 1 adds larger biases to the head classes, which can lower the learning difficulty, thereby enforcing the model to focus more on the tail classes and achieving more balanced performance.AdaptFormer. The previous method LIFT [43] empirically demonstrates that conducting PEFT with AdaptFormer [5] can achieve favorable LTR performance. Therefore, in this work, we also adopt AdaptFormer as the basic PEFT block. Given the input feature X ∈ R 𝑁 ×𝐷 , where 𝑁 and 𝐷 denote the batch size and feature dimension, respectively, the output of an AdaptFormer block isEQUATIONwhere 𝐿𝑁 (•) denotes Layer Normalization [2] , W 𝑑𝑜𝑤𝑛 ∈ R 𝐷 ×𝑑 and W 𝑢𝑝 ∈ R 𝑑 ×𝐷 are the down and up projection matrix (𝑑 is the bottleneck dimension and 𝑑 ≪ 𝐷), respectively, and 𝑠 is a learnable scalar. An AdaptFormer block is typically used as a parallel branch of the feed-forward network (FFN) in a Transformer block [11, 48] , updating the outputs to X 𝑜𝑢𝑡 + 𝐹 𝐹 𝑁 (X). The overall framework of the proposed Parameter-Efficient Complementary Expert Learning (PECEL) for LTR is illustrated in Figure 3 . As shown in Figure 3 , PECEL consists of multiple experts, where each expert is composed of a frozen pretrained backbone and trainable PEFT modules. In this work, following the previous works [43, 46] , we also use the pretrained CLIP-ViT-B [40] and Adapt-Former [5] as the pretrained model and basic PEFT block in this work. To learn expertise on complementary classes, the output logits of different experts are supervised with the proposed sampleaware logit adjustment loss conditioned by different weighted factors. Since learning multiple experts introduces extra parameters, to ensure parameter efficiency, we propose a parameter sharing strategy to decompose and share the parameters in PEFT blocks. In the inference stage, the outputs of different experts are aggregated via averaging without adjusting the logits.As illustrated in Figure 1 , the fine-tuning process undermines the inherent representation capability of pretrained models, then results in the model bias towards certain semantic classes. To take full advantage of pretrained models for LTR, we propose to learn multiple complementary experts, where each expert is enforced to learn the expertise on different subcategories via PEFT. The technical details of PEFT blocks in each expert are presented in Figure 3 and depicted in Section 3.4. Unlike the fixed relative class frequency b in Eq. 1, to learn experts specializing in different class groups, we propose to condition b with different weight factors. Specifically, assuming the output logits of the 𝑘 -𝑡ℎ expert E 𝑘 is z 𝑘 , the loss function for E 𝑘 is calculated as:EQUATIONwhere 𝜏 𝑘 is a weight factor to enforce expert E 𝑘 to focus on specific classes. Particularly, when 𝜏 𝑘 = 1, Eq. 3 degrades to Eq. 1, and can facilitate a relatively balanced performance on different classes. When 𝜏 𝑘 > 1, the imbalance ratio between different classes is enlarged, so the penalty to the head classes is increased, encouraging the expert E 𝑘 to focus and achieve better performance on tail classes. On the contrary, when 𝜏 𝑘 < 1, the imbalance ratio between different classes drops, therefore improving the performance on head classes for expert E 𝑘 . By employing different weight factors, different experts can learn knowledge on complementary classes. In practice, we set PECEL with 3 experts to learn complementary knowledge on many-shot, medium-shot and few-shot classes by setting 𝜏 > 1, 𝜏 = 1 and 𝜏 < 1, respectively. Note that Eq. 3 only adjusts the learning difficulty according to the class frequency. However, due to the intra-class diversity, the learning difficulty of samples in the same class can be very different [56] . To address this issue and enforce the learning of each expert, in this section, we further adapt Eq. 3 and propose the Sample-aware Logit Adjustment (SLA). Specifically, SLA further adjusts the logit bias in Eq. 3 for the misclassified hard samples. Based on Eq. 3, the calculation of SLA loss isEQUATIONwhere 𝛽 ∈ R 𝐶 is a bias vector and its values depend on the prediction results. Intuitively, if the current sample is correctly classified, 𝛽 = 0, degrading Eq. 4 to Eq. 3. While the current sample is misclassified, 𝛽 𝑖 𝑝 = 𝜖, 𝛽 𝑖 𝑦 = -𝜖 and 𝛽 𝑖 = 0 for other values, where 𝑖 𝑝 and 𝑖 𝑦 denote the index of the predicted and ground-truth class label, respectively. 𝜖 denotes a small positive value. For the misclassified samples, Eq. 4 can enlarge the logit bias on the ground-truth class, and decrease the bias on the misclassified class, thereby enforcing the model focus more on the misclassified samples.Learning with Eq. 4 can yield multiple complementary experts. However, the number of trainable parameters will also increase linearly by multiple times. To ensure the parameter efficiency of PECEL, different from previous PEFT methods that maintain separate trainable parameters in different blocks [5, 20] , we propose a parameter sharing strategy to decompose and share the parameters across different PEFT modules. Parameter Sharing. As illustrated in Figure 3 , in each expert, the projection matrices in PEFT blocks are decomposed into shared base matrices and standalone low-dimensional matrices. Specifically, for the 𝑖 -𝑡ℎ PEFT block, the down and up projection matrices are decomposed as:EQUATIONwhere P 𝑑𝑜𝑤𝑛 ∈ R 𝐷 ×𝑑 and P 𝑢𝑝 ∈ R 𝑑 ×𝐷 denote the base down and up projection matrices, respectively. P 𝑑𝑜𝑤𝑛 and P 𝑢𝑝 are shared across different blocks. Q 𝑖 𝑑𝑜𝑤𝑛 ∈ R 𝑑 ×𝑑 and Q 𝑖 𝑢𝑝 ∈ R 𝑑 ×𝑑 are the standalone low-dimensional matrices for down projection and up projection in the 𝑖 -𝑡ℎ block, respectively. Therefore, integrating Eq. 2, the outputs of the 𝑖 -𝑡ℎ shared PEFT block is calculated as:EQUATIONAs shown in Figure 3 , X 𝑖 𝑜𝑢𝑡 and the outputs of pretrained backbone block are integrated via addition. Note that Q 𝑑𝑜𝑤𝑛 and Q 𝑢𝑝 are separate, the individual blocks can therefore maintain their own down and up projection matrices. Since P 𝑑𝑜𝑤𝑛 and P 𝑢𝑝 are shared, and given 𝑑 ≪ 𝐷, in each expert, the number of trainable parameters can be reduced by a large margin. Shared Parameter Regularization. The reduction of trainable parameters requires the network to learn discriminative shared parameters. To facilitate this, we design a simple regularization loss for the shared parameters. Specifically, the shared parameters are restricted to orthogonal matrices. For the shared down projection matrix P 𝑑𝑜𝑤𝑛 , the loss isEQUATIONwhere |•| denotes the mean absolute error loss. I denotes the identity matrix. The calculation of L 𝑟𝑒𝑔 (P 𝑢𝑝 ) is in a similar way.As shown in Figure 3 , for each expert, the training loss includes the sample-aware logit adjustment loss in Eq. 4 and regularization loss in Eq. 7. Assuming there are 𝐾 experts, the overall training loss for PECEL isEQUATIONwhere P 𝑖 𝑑𝑜𝑤𝑛 and P 𝑖 𝑢𝑝 denote the projection matrices for expert 𝑖.To demonstrate the parameter efficiency of the proposed method, we calculate the exact number of trainable parameters when applying AdaptFormer with and without parameter sharing. In practice, the down and up projections are usually instantiated by linear layers, which include the weight matrices and bias vectors. Therefore, an AdaptFormer block has (2𝑑𝐷 + 𝐷 + 𝑑) parameters.  
experiments 4.1 experimental settings: 
 Datasets and Evaluation. We conduct experiments on the CIFAR100-LT [26] , ImageNet-LT [9] , Places-LT [57] and iNaturalist 2018 [47] to verify the efficacy of the proposed method. CIFAR100-LT [26] is the long-tailed version of CIFAR100, including 100 classes with 50K images for training and 10K images for validation. We follow the split in [43, 51, 54] and generate CIFAR100-LT with different imbalance ratios (IR) [45, 55] . IR is defined as the ratio of the maximum and minimum number of samples per class.ImageNet-LT [9] and Places-LT [57] are the long-tailed version of ImageNet-1k [9] and Places365 [57] By default, we report the Top-1 overall accuracy as the evaluation criterion. Following the common practice [43, 55] , we also report the accuracy of many-shot (with > 100 samples), medium-shot (with 20 -100 samples) and few-shot classes (with < 20 samples). Due to the page limits, the results on Places-LT are reported in the supplementary material. Implementation Details. For a fair comparison with previous works [16, 34, 43, 46] , we adopt CLIP [40] as the pretrained foundation model. Specifically, the backbone architecture of PECEL is ViT-B [11] , and initialized with the pretrained CLIP visual encoder. For parameters in PEFT modules, the up projection layers and learnable scale factors are initialized with 0 to avoid influencing the features at the initial training stage. The other parameters are initialized via Kaiming Normalization [14] . We use the semanticaware initialization and test-time ensembling proposed in LIFT [43] to initialize the classifiers and further improve the test accuracy. For the experiments on CIFAR100-LT, ImageNet-LT and Place-LT, we use the SGD optimizer to perform optimization. The initial learning rate and total epochs are set as 0.1 and 10. The learning rate decays linearly to 0 by epoch. Since iNaturalist 2018 consists of more training samples and classes, we use the AdamW optimizer [33] for better convergence in experiments on iNaturalist 2018. The initial learning rate and total epochs are set as 5𝑒-4 and 20, respectively.Unless specified otherwise, the number of experts in PECEL is set as 3, with 𝜏 in Eq. 4 set as [0.5, 1.0, 1.5] to enforce different experts to learn expertise on the many-shot, medium-shot and few-shot classes, respectively. The bias factor 𝜖 in 𝛽 (Eq. 4) is set as 0.1. The bottleneck dimension 𝑑 in the PEFT blocks is set as 14/64/24/224 for CIFAR100-LT/ImageNet-LT/Places-LT/iNaturalist 2018 to align the number of trainable parameters with LIFT [43] . More implementation details can be found in the supplementary.CIFAR100-LT. In Table 1 , we report the classification accuracy on CIFAR100-LT with different imbalance ratios (IR). † denotes PECEL with a lower bottleneck dimension, i.e., 𝑑 = 1 for CIFAR100-LT. As shown in Table 1 , owing to the multiple complementary experts, the proposed PECEL can achieve 84.3%/84.6%/86.4% Top-1 accuracy on CIFAR100-LT with IR=100/50/10, which outperforms previous state-of-the-art LIFT [43] by 2.6%/1.2%/1.4%. Besides, compared to LIFT [43] , due to the proposed parameter sharing strategy, the proposed PECEL can achieve comparable performance with ∼70% fewer trainable parameters. ImageNet-LT & iNaturalist 2018. The results on ImageNet-LT and iNaturalist 2018 are reported in Table 2 . † denotes PECEL with a lower bottleneck dimension, i.e., 𝑑 = 32 for ImageNet-LT and 𝑑 = 128 for iNaturalist 2018. In Table 2 , the proposed PE-CEL achieves 79.2% and 82.5% classification accuracy on ImageNet-LT and iNaturalist 2018, respectively, which remarkably surpass the previous state-of-the-art method LIFT [43] . For the parameter efficiency, compared with LIFT, PECEL can achieve comparable performance with 60.4%/62.3% fewer parameters on ImageNet-LT/iNaturalist 2018, which is attributed to the parameter sharing strategy in Section 3.4. In Table 2 , we also report the accuracy of many-shot classes (Many), medium-shot classes (Med) and few-shot classes (Few). As shown in Table 2 , due to the proposed skill-diverse and complementary expert learning in Eq. 4, compared with LIFT [43] , PECEL can achieve a more balanced performance across different class groups and higher performance on each class group.Ablation Study. To demonstrate the effectiveness of the proposed modules, we conduct ablation experiments on CIFAR100-LT (IR=100) and iNaturalist 2018 (iNat 2018). The results are reported in Table 3 . Our baseline is LIFT [43] , which adopts PEFT with AdaptFormer [5] for LTR. In Table 3 , we observe that the parameter sharing (Share) strategy can reduce the number of trainable parameters, yet at the cost of reducing the representation capability of all expert models. The proposed parameter regularization (Reg) strategy can reduce representation redundancy and improve the recognition accuracy. Incorporating complementary expert learning (CEL) can substantially improve the overall accuracy of CIFAR100-LT and iNaturalist 2018 to 83.7% and 81.6%, respectively. Moreover, since CEL via sample-aware logit adjustment (CEL-SLA) can facilitate learning on misclassified samples, the final recognition accuracy on CIFAR100-LT and iNaturalist 2018 is further promoted to 84.3% and 82.5%, respectively. Complementary Expert Learning. Table 3 shows the proposed modules can effectively improve the overall accuracy. We need to further verify the contribution of individual experts. In Table 4 , we present the classification accuracy of each expert in PECEL on each class group. Table 4 shows that different experts (i.e., E 1 , E 2 , E 3 ) conditioned with 𝜏 = [0.5, 1.0, 1.5] can competently learn expertise on many-shot, medium-shot and few-shot classes, respectively, which demonstrates the effectiveness of complementary expert learning in Eq. 4. Besides, Table 4 also shows that aggregating these complementary experts yields higher overall accuracy than each single expert, which also verifies the necessity of learning multiple complementary experts. Expert Numbers. As mentioned in Section 4.1, by default, PECEL consists of 3 experts conditioned by the weight factors of 0.5, 1.0 and 1.5, respectively. In Table 5 , we report the accuracy of PECEL with different expert numbers and weight factors 𝜏. For example, [0.5, 1.5] denotes PECEL consists of two experts which are conditioned by 0.5 and 1.5, respectively. As shown in Table 5 , PECEL with different 𝜏 consistently performs better than LIFT in Table 1 . Since the performance of few-shot and many-shot classes will be hindered more remarkably when 𝜏 = 0.5 and 2.0, PECEL with 𝜏 = [0.0, 1.0, 2.0] exhibits lower performance than 𝜏 = [0.5, 1.0, 1.5]. It's also noted that PECEL achieves 83.9% accuracy when 𝜏 = [1.0, 1.0, 1.0], indicating that ensembling experts with similar expertise can also help. Sample-aware Logit Adjustment. In addition to learning complementary experts, Sample-aware Logit Adjustment (SLA) also aims to shift more attention to the misclassified samples. To verify this, in Figure 4 , we present the accuracy gains of each class in CIFAR100-LT when using the sample-aware bias in Eq. 4. Figure 4 shows that applying sample-aware bias can improve the average accuracy by 2.0% on the few-shot classes, which usually involve more misclassified samples due to the lower class frequency. Given the ablation results in Table 3 , the proposed SLA can finely achieve the goal of facilitating learning on misclassified samples and improving the overall performance. between performance and parameter efficiency. To further explore the parameter efficiency for PECEL, we 1) share the projection matrices P 𝑑𝑜𝑤𝑛 and P 𝑢𝑝 in Eq. 5 across different experts (denoted as Share); 2) set P 𝑑𝑜𝑤𝑛 and P 𝑢𝑝 as fixed random matrices as done in VeRA [25] (denoted as Random). The experimental results on CIFAR100-LT (IR=100) and ImageNet-LT are presented in Table 6 .As shown in Table 6 , though leveraging 1) and 2) can improve parameter efficiency, the recognition accuracy will also be compromised. The reason is that sharing parameters across experts may confound the learning of skill-diverse experts and the fixed random matrices may weaken the representation learning ability and leave learned representation less discriminative. Bottleneck Dimension. In Figure 5 , we present the recognition accuracy and number of trainable parameters of PECEL with different bottleneck dimensions on CIFAR100-LT and ImageNet-LT. As shown in Figure 5 , the performance of PECEL generally increases with the number of trainable parameters, i.e., the bottleneck dimension. Considering the overall parameter efficiency, the bottleneck dimensions of CIFAR100-LT and ImageNet-LT are set as 14 and 64, respectively. It's also noted that the performance gains of aggregating different experts (i.e., PECEL E2 to PECEL) also generally enlarge with the increasing bottleneck dimension, which indicates that a larger bottleneck dimension would help learn experts with diverse expertise. Generalization Analysis. The proposed PECEL is instantiated with AdaptFormer [5] and ViT [11] , but it's also generalizable to other PEFT methods and backbones. In Table 7 , we present the performance of adopting LoRA [20] and ResNet50 [15] , respectively. Adopting LoRA for PECEL is straightforward since LoRA also uses the down and up projection matrices. For experiments with ResNet50, each convolutional bottleneck block is equipped with a parallel PEFT block. Besides, since the latent feature dimension progressively increases in ResNet, in different stages, different projection matrices P 𝑑𝑜𝑤𝑛 and P 𝑢𝑝 (Eq. 5) with different 𝐷 are used for parameter sharing. As shown in Table 7 , compared with LIFT with the same configuration, the proposed PECEL also performs well and achieves about 2% higher accuracy on CIFAR100-LT, suggesting the proposed PECEL generalizes well to other backbones and PEFT methods. 
conclusion: 
 In this paper, we propose the Parameter-Efficient Complementary Expert Learning (PECEL) to unleash the potential of pretrained models for LTR via learning diverse complementary experts. To alleviate the extra parameter cost brought by multiple experts, we design a cross-block parameter sharing strategy. Besides, we propose parameter regularization and sample-aware logit adjustment to reduce the shared parameter redundancy and facilitate learning on misclassified samples. The extensive experiments on CIFAR100-LT, ImageNet-LT, Places-LT and iNaturalist 2018 show that the proposed PECEL can achieve comparable performance with previous state-of-the-art methods with about 60% fewer trainable parameters, and achieve substantial improvements with equivalent number of trainable parameters. Currently, the proposed method mainly focuses on leveraging the visual encoder in pretrained models. In the future, we'll explore simultaneously incorporating the visual and linguistic knowledge of pretrained multi-modal foundation models for long-tailed recognition. 
 
 Appendix:  


Decision: Three of the reviewers are positive on this paper, and one reviewer recommends rejection. The proposed method is reasonable and sound. It is the first work to effectively learn multiple complementary experts from pre-trained foundation models. There are also some issues in this paper, and most of them have been replied in the rebuttal. AC recommends acceptance. Authors need to improve the final version following reviewers' suggestions.

Reviews: review: paper_summary: paper_summary: The paper proposes a novel method called Parameter-Efficient Complementary Expert Learning (PECEL) to improve long-tailed recognition (LTR) tasks. Long-tailed recognition deals with datasets where a few classes (head classes) have many samples while many classes (tail classes) have few samples. PECEL addresses this by using multiple experts, each fine-tuned to different subsets of classes using Parameter-Efficient Fine-Tuning (PEFT). The experts' predictions are aggregated to achieve balanced performance across all classes. PECEL also introduces a parameter-sharing strategy to maintain efficiency by reducing the number of trainable parameters. The method shows state-of-the-art performance on several benchmarks.
strength_weakness: strength_weakness: Technical Approach: The paper's technical approach is sound, with clear explanations of how PECEL works and the innovations introduced, such as the sample-aware logit adjustment loss and parameter-sharing strategy.
Adequate Evaluation: Extensive experiments on four LTR benchmarks (CIFAR100-LT, ImageNet-LT, Places-LT, iNaturalist 2018) demonstrate the effectiveness of PECEL, showing superior performance compared to existing methods.
Clarity: The paper is well-organized and clearly written, making the complex methodology accessible and understandable.
limitations: limitations: Lack of Novelty: The proposed method appears to be a combination of existing approaches rather than a truly novel solution. For example, the idea of using multiple experts has already been applied to long-tailed visual recognition in NCL [27], and logit adjustment is a common method in this field [1,2]. The authors need to more clearly articulate the unique contributions and novelty of their approach.

[1] Menon, Aditya Krishna, et al. "Long-tail learning via logit adjustment." arXiv preprint arXiv:2007.07314 (2020).
[2] Zhao, Yan, et al. "Adaptive logit adjustment loss for long-tailed visual recognition." Proceedings of the AAAI conference on artificial intelligence. Vol. 36. No. 3. 2022.
Unclear Comparisons: The paper compares training from scratch with fine-tuning pretrained models, but this comparison lacks significance due to the different datasets and backbone networks used. The superiority of fine-tuning pretrained models over training from scratch is well-established and thus does not provide new insights.

Inadequate Parameter Calculation: The method for calculating the number of parameters seems flawed. The comparison in the paper only considers the parameters adjusted by the authors, ignoring the parameters of the pretrained network. This approach can mislead the evaluation of the actual parameter efficiency and computational requirements of the proposed method.

In conclusion, while the authors' work is valuable, it does not meet the standards required for this conference.
paper_summary: paper_summary: This work proposes the Parameter-Efficient Complementary Expert Learning method for Long-Tailed Visual Recognition. The PECEL method learns multiple experts from pretrained foundation models via parameter-efficient fine-tuning. The proposed method achieves the best results on all four datasets.
strength_weakness: strength_weakness: 1. This work proves that combining parameter-efficient fine-tuning and multiple experts is a feasible solution for long-tailed recognition.
limitations: limitations: 1. This work proposes to use multiple experts to address the long-tailed recognition problem. In contrast, previous methods utilizing multiple experts for long-tailed recognition such as RIDE [1], SADE [2], are not mentioned. 
2. Following the previous limitation, the proposed model has 85*3+0.1M parameters in total, which is much larger than other SOTAs. In RIDE and SADE, the backbones are shared and the channels are reduced to maintain a comparable total parameter number with previous works. Thus I consider the proposed model has limited concerns about the computation consumption.
3. The second contribution, i.e., sample-aware logit adjustment, has limited contribution from a technical or effect perspective.


[1] Wang, Xudong, et al. "Long-tailed Recognition by Routing Diverse Distribution-Aware Experts." International Conference on Learning Representations. 2020.
[2] Zhang, Yifan, et al. "Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition." Advances in Neural Information Processing Systems 35 (2022): 34077-34090.

minimal:
L199 UDCPC et al -> UDCPC
paper_summary: paper_summary: To unleash the intrinsic representation capability of pretrained foundation models, this paper proposes a new Parameter-Efficient Complementary Expert Learn-Asaling (PECEL) for LTR. Specifically, PECEL consists of multiple experts, where individual experts are trained by Parameter-Efficient Fine-Tuning (PEFT) and encouraged to learn different expertise on complementary sub-categories via the proposed sample-aware Zero-Shot logit adjustment loss.
strength_weakness: strength_weakness: 1.The proposed PECEL is the first work to effectively learn multiple complementary experts from pretrained foundation models via PEFT for long- tailed recognition.

2.This paper proposes sample-aware logit adjustment and a parameter sharing strategy, which can effectively learn complementary expertise while retaining the parameter efficiency.
limitations: limitations: Please unify the reference format.
paper_summary: paper_summary: This paper addresses the long-tailed recognition (LTR) problem using parameter-efficient fine-tuning (PEFT). It mitigates the bias issue found in previous PEFT long-tailed models (e.g., PEL) by integrating PEFT with complementary experts and a sample-aware logit adjustment loss. Additionally, it reduces computation by sharing some PEFT parameters among experts. Extensive experiments on popular LTR benchmarks demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: This paper has several strengths:

* It is well-written and easy to follow.

* The idea is technically sound, despite its complexity due to multiple components. Fine-tuning from pre-trained models is promising, and PEFT is increasingly popular.

* The results are convincing, as combining diverse experts consistently shows superior performance in LTR.
limitations: limitations: Although this paper is quite readable, some concerns remain:

* The proposed method appears to be a straightforward combination of PEL with diverse experts and LA, both of which are commonly used in LTR, reducing its novelty.

* The motivation for this paper is not clearly stated. While the authors mention that PEL sacrifices performance on tail classes, they do not provide detailed analysis. More empirical evidence would strengthen the paper.

* It is unclear if the topic fits well within the MM community, as the experiments focus on general natural image classification.

Overall, I am somewhat positive about this paper but not fully convinced. I am open to further discussion.