Unleashing Foundation Vision Models: Adaptive Transfer for Diverse Data-Limited Scientific Domains
Authors: Qiankun Li
Feng He
Huabao Chen
Xin Ning
Kun Wang
Zengfu Wang
Venue: NeurIPS 2025 poster ()
Keywords: AI4Science
Cluster attention adapter
Adaptive domain transfer
Primary area: machine_learning_for_sciences
TL;DR: We propose CLAdapter to refine and adapt the rich representations learned from large-scale data for transfer across diverse, data-limited scientific downstream domains.

Abstract: In the big data era, the computer vision field benefits from large-scale datasets such as LAION-2B, LAION-400M, and ImageNet-21K, Kinetics, on which popular models like the ViT and ConvNeXt series have been pre-trained, acquiring substantial knowledge. 
However, numerous downstream tasks in specialized and data-limited scientific domains continue to pose significant challenges. 
In this paper, we propose a novel Cluster Attention Adapter (CLAdapter), which refines and adapts the rich representations learned from large-scale data to various data-limited downstream tasks. Specifically, CLAdapter introduces attention mechanisms and cluster centers to personalize the enhancement of transformed features through distribution correlation and transformation matrices. This enables models fine-tuned with CLAdapter to learn distinct representations tailored to different feature sets, facilitating the models' adaptation from rich pre-trained features to various downstream scenarios effectively. In addition, CLAdapter's unified interface design allows for seamless integration with multiple model architectures, including CNNs and Transformers, in both 2D and 3D contexts.
Through extensive experiments on 10 datasets spanning domains such as generic, multimedia, biological, medical, industrial, agricultural, environmental, geographical, materials science, out-of-distribution (OOD), and 3D analysis, CLAdapter achieves state-of-the-art performance across diverse data-limited scientific domains, demonstrating its effectiveness in unleashing the potential of foundation vision models via adaptive transfer.
Code is available at https://github.com/qklee-lz/CLAdapter.

Unleashing Foundation Vision Models: Adaptive Transfer for Diverse Data-Limited Scientific Domains
Qiankun Li Feng He' Huabao Chen', Xin Ning", Kun Wang" Zengfu Wang"
Uniiversity of Science and Technology of China
2 AnnLab, Institute of Semiconductors, Chinese Academy of Sciences Nanyang Technological University
* *Corresponding Author
{qklee, wk520529) Cmail @mail. ustc. edu, en, 2fwanglustc. edu, en
Abstract
In the big data era, the computer vision field benefits from large-scale datasets such as LAION-2B, LAION-400M, and ImageNet-21K, Kinetics, which pop- ular models like the VIT and ConvNeXt series have been pre-trained, acquiring substantial knowledge. However, numerous downstream tasks in specialized and data-limited scientific domains continue to pose significant challenges. In this pa- per, propose novel Cluster Attention Adapter (CLAdapter), which refines and adapts the rich representations learned from large-scale data to various data-limited downstream tasks. Specifically. CLAdapter introduces attention mechanisms and cluster centers to personalize the enhancement of transformed features through distribution correlation and transformation matrices. This enables models fine- tuned with CLAdapter to learn distinct representations tailored to different feature sets, facilitating the models' adaptation from rich pre-trained features to various downstream scenarios effectively. In addition, CLAdapter's unified interface de- sign allows for seamless integration with multiple model architectures, including CNNs and Transformers, in both 2D and 3D contexts. Through extensive experi- ments on 10 datasets spanning domains such as generic, multimedia, biological, medical, industrial, agricultural, environmental, geographical, materials science, out-of-distribution (OOD), and 3D analysis, CLAdapter achieves state-of-the-arr performance across diverse data-limite scientific domains, demonstrating its ef- fectiveness unleashing the potential of foundation vision models via adaptive transfer. Code is available https: h/tts::/eee //ggthhbbb..../ com com/qklee 1ee-l1///hcc 12/CLAdapter.
Introduction
With the rapid advancement in artificial intelligence, deep learning- bbseed computer vision algorithms have emerged as dominant force [42, 85, 47]. These algorithms are inherently data-driven, capital- izing on substantial datasets to refine their task-specific performance. The digital age's ever-growing data trove has ushered in large-scale datasets, such as ImageNet-21KK [58], LAION-400M [62], and LAION-2B [61], which aim to bolster algorithmic generalization and accuracy through data diversity and volume [25, 44, 10J. Despite these advancements, domain-specific challenges and data scarcity remain significant hurdles scientific visual downstream tasks, where specialized data often limited, heterogeneous,
expensive to acquire [11, 72, 79]. Therefore, developing methods that effectively harness the potential of large-scale pre-trained models to enable robust adaptation in data-limited scientific domains constitutes critical and promising research direction.
Transfer learning through methods like linear probing and full fine-tuning an essential approach for enhancing performance on downstream tasks [77]. This works well when transferring under 39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

normal-sizee sized dataset pre-trained models to in-distribution (ID) downstream tasks. However, transfer- ring adapted knowledge from rich but complex large-scale upstream pretraining poses significant challenges in the current era of large datasets [61, 10]. Furthermore, the scientific domains down- stream tasks often involve -uutt--distribution (OOD) scenarios [4], domain specificity [27], and data limitations [68], intensifying the fine-tuning challenge. L.2-SP fine-tuning [26] introduced 1.2 regularization to preserve the insights of pre-trained weights during task adaptation, although they may lack robustness in OOD contexts. Visual Prompt Tuning (VPT) [34] augmented the input space with task-specific learnable prompts. However, VPT is primarily designed for Vision Transformer (ViT) [70] and lacks the ability to provide stable cross-domain feature transferability. OLOR [30] designed the inn-tuning optimizer from the perspective of pre-trained weights to improve stability, but it lacks task-specific self- self-adaptability.. particularly under the diverse conditions of downstream scientific tasks.
(T [Fmmirsmmntt Fangraphi) (Agriadtura))
Fine-tuning
Pre-training Resources
(Maaoorthh Nceeec)
(Gannea (KID)
Large-scale
Braaklie
HELT (Gentt
CLAdapter
Pre-trained
Backbones
Rgo
(Roeneiiter
(3D Moliierdaa
(Indintral))
(3D Maltinedia)
Overview of the proposed CLAdapter. CLAdapter refines and adapts the rich representations learned from large-scale data to diverse data-limited scientific downstream tasks, achieving state-of- the-art performance across diverse fields 10 datasets.
Figure I:
In this paper, we propose novel Cluster Attention Adapter (CLAdapter), which refines and adapts the rich representations learned from large-scale data diverse data-limited scientific downstream tasks (as illustrated Figure 1). Specifically, Cl.Adapter introduces attention mechanisms and cluster centers to enable customized feature enhancement through distribution correlation and transformation matrices. This facilitates the generation of task-adaptive representations, supporting smooth transition from abundant pre-trained features diverse downstream scenarios. Benefiting from our unified interface design, CLAdapter can seamlessly integrate with mainstream architectures, including CNNs, Transformers, and their 3D versions. In addition, Staged Fine-Tuning (SFT) strategy is presented to collaborate with CLAdapter to further enhance the fine-tuning performance. Through extensive experiments conducted on 10 datasets spanning domains such as generic, multi- media, biological, medical, Industrial, agricultural, environmental, geographical, materials science, distr-ddittiitutinn (OOD), and 3D analysis, CL.Adapter demonstrates its universal applicability and state-of-ther thette-arttt performance. These results underscore the importance of effective knowledge transfer the big data era and advance the reliable and efficient deployment of computer vision foundation models across scientific and industrial domains.
The contributions of this paper are summarized follows:
Adaptive Representation Transfer. We propose novel CLAdapter that leverages large-scale pre-trained knowledge to enhance performance on a variety of data-limited downstream tasks.
Flexible Adaptation Framework. We design a unified interface and a staged fine- tie-ttnnng (SFT) strategy, enabling CLAdapter to integrate seamlessly with mainstream pre-trained models and establish an efficient fine-tuning paradigm.
AI4Science Broad Evaluation. We conduct comprehensive experiments on 10 datasets across diverse domains, including multiple scientific fields where data is limited and heterogeneous. CLAdaater consistently achieves state-of-the-ee performance, demonstrating its potential as generalizable solution for Al-driven scientific applications.

---

2 Related Work
2.1
Pre-Training Resources
With the rapid development of computer vision technology, large number of large-scale datasets [58. 62] and pre-trained models [56, 57, 5, 25, 19] have been proposed, providing a rich feature library for downstream tasks, Pre-training on large-scale datasets can encode rich semantic information, which is useful in solving limited data tasks, domain generalization, and zero-shot learning, ImageNet-21k mainly used for visual image classification, contains about 21k categories and 14 million images. providing a rich and diverse training environment for large models. The LAION-400M dataset [62] is large-scale image and text pairing dataset, containing about 300 million image-text pairs, including natural landscapes, people, everyday items, etc., suitable for training cross-modal models [56, 57J. To cope with these growing resources, new generation of pre-trained models (such as CLIP [56], BEiT [5], MAE [25], and EVA [19]) has emerged, mainly utilizing the architectural principles of ViT [70] and ConvNeXt [50]. However, how to efficiently transfer a large number of complex datasets remains an unresolved ved issue. Therefore, this paper proposes CLAdapter, introducing attention mechanisms and clustering centers to leverage rich pre-training resources, thus effectively improving performance on various downstream tasks.
2.2 Various Downstream Tasks and Fine- Tine-Tuning Methods
The realm of practical downstream visual tasks is vast. However, most are cross-domain with limited data, such as medical image processing [11]. industrial fault diagnosis [60]. pest and disease recognition [72], and natural geographic image classification [83], Materials science research [45], 3D multimedia analysis [37]. Fine-tuning pre-trained models has become a critical method for improving performance downstream tasks. The popular fine-tuning methods are Linear Probing (LP) [26], adjusting only the model's head, and Full Fine-tuning (FT), tuning all layers. Recently, L2-SP fine-tuning introduced an L. regularization keep changes to pre-trained weights minimal, thus preserving initial insights while adapting to new tasks. Visual Prompt Tuning (VPT) [34] inserted trainable prompts at the input, akin NLP's prompt learning, prevent altering the original model weights. VPT proved effective for tasks with numerous parameters and scant data, utilizing pre-trained knowledge and averting overfitting from significant weight modifications. However, VPT is mainly designed for Vision Transformer (ViT) [17] and has cross-domain instability issues. Different from existing fine-tuning methods, CLAdapter introduces attention mechanisms and cluster centers for customized feature representation refinement. By providing uniform interface for various upstream tasks, CLAdapter fine-tune tne-tunes any category of pre-trained models, transferring their knowledge to wide array of downstream tasks.
3 Methods
We propose CLAdapter refine and adapt the rich representations learned from large-scale data for transfer applications in various downstream tasks. It injects a small number of learnable parameters into the original model, offering efficient fine-tuning strategy by freezing the backbone, alongside a staged fine-tuning approach for more gradual adaptation. The framework is depicted in Figure 2.
3.1 Unified Model Interface
Given an input image X, E RCXHxW a standard Vision Transformer (ViT) divides X, into N patches. Each patch is then embedded into D-dimensional latent space, resulting in a set of tokens T E NDD Then, T and a extra learnable classification token Celass ([class) with position embeddings are fed into the transformer layers { Y=o1 Thus, the feature representation extracted by VIT is T (N+x)) Discarding Valass due to its linear combination nature, it might interfere with feature transfer from the pre-train model to downstream tasks.
For CNN-based models, the feature map X E x'C''X'xW'' is extracted. Here, C", H', and W represent the number of channels, height, and width of the feature map, respectively. To facilitate the integration of CNN-based models with Cl.Adapter, we flatten the spatial dimensions of the feature map X f to align with the feature dimensionality used by ViT.
3

---

Pre-train Large-seale Datasets
Fine-tuning Downstream Tasks
Prece
Iirs
CLAdapter
MLP
Buse Transformarion arrices
Large-scale Data
Transformer Rased
Layer-Norm
H
N-D
Head
Cluser
Chasa Canter
Veetees
Laye--Norm
H=WxCC
CNN Basod
Stage
Stage
Stago
Stage
Figure 2: Overview of the CLAdapter. It utilizes large-scale pre- wre-training to enhance various data- limited downstream tasks. The unified interface design and SFT fine-tuning strategy allow CLAdapter to integrate with mainstream pre-trained models and form an efficient fine-tuning paradigm.
In the case of 3D video clips or image sequences Xy E TTxCHxx the extracted features also include an additional temporal dimension T. Although time and spatial dimensions together form tubes rather than patches, they still belong to the internal structure of data. Therefore, we similarly flatten these dimensions to achieve unified representation.
In summary, whether for CNN-based models, Transformer-basee models, their 3D variants, we uniformly apply an interface function for the features extracted by the models, denoted as F(X,/X)) H. where H R NxD represents the extracted features after dimension unification.
3.2 Cluster Attention Adapter (CLAdapter)
For downstream tasks, is crucial to focus on specific information pertinent to their domain. However, while large-scale upstream data encompasses wealth of information, it also introduces complexity and redundancy. This challenge (OOD) tasks, making it more difficult distill the necessary information from the rich pre-traine pre-trained feature representations H. In real-world applications, with the diverse nature of downstream tasks presenting both in-distribution (ID) and OOD scenarios, it is crucial design mechanism capable of adaptively refining and transforming the pre-trained features H into suitable features H' based the characteristics of the downstream tasks. This process can be defined as learning mapping function Fe (H) > H', where Fa denotes a model and represents its learnable parameters.
further exacerbated when dealing with Out-Of-Distribution
CLAdapter aims to refine and adapt the feature representations H learned from large-scale datasets for use in various data-limin data-limited downstream tasks. Notably, embeddings of image categories in feature space often close each other, suggesting the presence of feature cluster centers that represent specific latent information. To exploit this, we introduce multiple learnable vectors to denote these feature cluster centers:
A AA
Axk.
(1)
where A DxK, and K the number of cluster centers. The attention scores derived by calculating the cosine similarity between the pre-trained embeddings H and the cluster centers A. To enhance computational efficiency and convert attention scores relative probability distribution B3 we compute the mean of H to obtain H and apply the softmax function, respectively. The above operation denoted mathematically as follows:
1
H Layer H)),
H
Hip
(2)
N
i=1
7
A
A
Ax
(3)
[
||
|
3 softmax
H
A)
where Layer-Norm reduces the difference in input distribution between different layers, and both H² and A are L2 normalized along the feature dimension. Upon obtaining the attention scores & e k

---

we further introduce learnable transformation matrices M corresponding to the cluster centers A:
M (M, .Mk).
(5)
where each transformation matrix M, E RDDD The weighted transformation matrix M for each pre-trained feature embedding is derived by weighting these matrices with the attention scores:
M' - B.M,
(6)
Consequently, each embedding is subjected to custom transformation matrix based on its cosine similarity with each feature cluster center, facilitating the organized transition from the original upstream feature distribution to a new distribution tailored for downstream tasks. This involves a customized transformation of H using M followed by enhancement with Layer-Norm and MLP layer to improve generalization and introduce non-linearity. The above operation is denoted mathematically follows:
H = -LayerNNormPH.. H.M"),
H' GEL + b, W, + b2.
(8)
where 7 is the result of the customized transformation. In the MLP, GELU represents the Gaussian Error Linear Unit activation function, and W, and W2 are the weight matrices for the first and second linear transformations, respectively, with a default ratio of 4. b, and by are the corresponding bias vectors. The final output H' represents the features adaptively refined and transformed from the rich pre-trained feature embeddings H to suit downstream tasks. Additionally, these transformed features be reshaped back the original feature shape of the upstream model through inverse function of the unified interface defined in Section 3.1, facilitating the integration with their respective heads.
3.3 Fine-tuning Strategy
To adapt a pre-trained model for downstream task, practitioners commonly employ either full fine- tuning (FT), where all model parameters are updated, or linear probing (LP), which only updates the parameters of the final linear classification layer (head). By incorporating our proposed CLAdapter, the LP approach updates both the adapter and the classification layer, whereas the FT approach updates the entire model. These two popular fine-tuning strategies can be formalized as:
CPcl(x) = Wha hOL (H) + badi
(9)
FTc Wha (opre (z)) + bhd:
(10)
where x denotes the input, h.)(.) is the function represented by CLAdapter, Øpre (-) represents the pre-trained backbone, and Whd and bad are the learnable parameters of the head. LP is computation- ally efficient as only updates to the adapter and the head of the model. Although FT provides thorough adaptation to the downstream task, starting CLAdapter training from scratch might distort the backbone's refined data representations, potentially exacerbating domain mismatch in OOD scenarios.
To address this, propose staged fine-tuning (SFT) strategy, beginning with only updates to the CLAdapter and heads in the first stage and progressing to full fine-tuning in the second. This method allows CL.Adapter first to obtain better pre-transfer capabilities for the original pre-training domain, then further fine-tune the entire model to complete the downstream task. Since the fine-tuning overhead of LP is almost negligible compared to FT, the incremental cost of SFT is minimal. The SFT strategy can be represented as:
S SFT (x) W.P.. L.P (p)re(x)) + b
(1))
where LLP. D.P and li represent the head and CLAdapter after the first stage of fine-tuning (LP), respectively. Notably, in some cases, this LP stage often yields satisfactory performance for many tasks, thus reducing costs. SFT strategy efficiently leverages the strengths of both I.P and FT, facilitating effective knowledge transfer from the pre- pre-trained model to diverse downstream tasks.
5

---

Experiments
4.1 Experiment Setup
Pre-training Dataset and Backbones. In the era of big data, popular publicly available large- scale 2D datasets include the ImageNet-21K classification dataset [58] the ten-million level, the LAION-400M image-text dataset [62] at the hundred-million level, and the LAION-2B image-text
dataset [61] at the billion level. For the 3D domain, we utilize the Kinetics-400 [36], large-scale dataset commonly used for action recognition, comprising approximately 260K video clips. On these large-scale datasets, we employ popular pre-trained models such Vision Transformers (ViT) [70], ConvNeXt [50], and Video Swin Transformers (Swin) [49]. Details of these datasets and pre- pre-trained backbones are listed Table 1.
Table 1: Details of the pre-training datasets and the corresponding backbones used.
Dataset
Seale
Type
Backbone
ImageNet-22 21K [58]
14 Million
Images
VIT-B/16, ViT-L/16, ConvNeXt-B
LAION-400M [62]
400 Million
Image-Ter Text Pairs
VIT-B/16, ConvNeXt-B
LAION-22 2B [61]
2000 Million
Image-Text Pairs
VIT-B/16, ConvNeXt-B
Kinetics-400 [8]
0.26 Million
Vidco Clips
Swin-B, Swin-I.
Downstream Tasks. We experiment on 10 benchmarks across a broad spectrum of domains, including generic (Tiny-ImageNet [38]), multimedia (UCF101 [66] and HDMB51 [37]), industrial (InsPLAD-fanl [78]), bimdogiaal&mmedcall (BreakHis [6] and HCRF [68]), agricultural (Apple (WHU-RS19 [81]), materials science (KTH- TIPS-2b [) OOD (PACS [40], and 3D analysis (above video) to demonstrate the versatility and effectiveness of CLAddpter. Full benchmark list (Table and dataset descriptions with processing methods, are provided in Appendix A.1.
Foliar Disease [72]), @
Evaluation Metries. Following the previous works [78], we report the ROC on the InsPLAD-faull dataset. In alignment with established precedents [11], we utilize the precision, recall, accuracy, and Fl score metrics BreakHis and HCRF datasets. For all other datasets, assess model efficacy using 1 1 accuracy. Implementation Details are included in Appendix A.2.
Inpowesea -0
itc.
Scoerha
se-
Imperomet
WHU 8519 KTH--1-P2-B
ROME
IsPLAD Irdit
UCP1OT
HMP85:
Figure 3: Performance comparison CLAdapter against SOTA methods across various application domain datasets. The bar graph illustrates the average scores achieved by the CLAdapter and SOTA each dataset, with the red fold line indicating the percentage improvement offered by the CLAdapter.
4.2 Main Results
Overall Intuitive Performance Comparison. Figure provides intuitive comparison between our method and prior SOTA approaches, while additional intuitive comparisons are presented in Appendix B. Table 6. It demonstrates that method consistently surpasses existing SOTA methods, highlighting its strong a in data- limttel scientific domains and robustness against challenges such as limited data availability, domain distribution shifts, visual semantic variations, and fine- eneigrai feature distinctions. More detailed quantitative comparisons are provided in the following sections.
Comparison with SOTA approaches in diverse scientific domains. As listed in Table 2, CLAAdapeer achieves state-of-thed -he-tarrt results in all benchmarks. Notably, on the industrial defect detection dataset InsPLAD-fat faullt and the 3D multimedia action recognition UCF101 dataset, CLAdapter, fine-tuning

---

Table 2: Comparison of CLAdapter with SOTA methods for diverse scientific downstream tasks. Best performances are highlighted in bold, while the second-bes best are underlined.
(a) Agricullurral [72]
Geography: WHU-RSI9 181]
(c) Materials: KTH-TIPS2-B [51]
Method
Method
Method
MoCo v2 [13]
96.04
DXCA-Fusion [9]
93.56
CDL [80]
76.30
MaskCOV [69]
95.82
GM [89]
88.16
Tinoft [73]
66.30
SPARE [87]
96.70
GLMI6 [89]
92.99
DMD-IFV [52]
76.20
VIT [70]
[18]
VGGVV [14]
DeiT [75]
96.26
MS2AP [7]
98.88
LETRIST [65]
65.30
TransFG [88]
97.14
[17]
96.42
CATex [2]]
66.70
Hybrid [IT170] [70]
96.48
Swin [+8]
97.12
RAMBP [2]
68.90
Swin [48]
98.08
EMTCAL- [70]
97.60
TEX LEX-Nets-LF1B3
78.00
CL.
97.58
SF-MSFormer [84]
97.80
BMCAnet
79.18
Cl.Adaptercannecos
CLAddapte
99.80
CIAAdapteTouneaoss
CLAdapteryzza
98,36
CLAdapter
99.20
CLAdapter
91.26
(d) Biomedicine: Break [6]
(c) Biomedicine: HCRF [68]
Method
Pre
Rec
Method
Pre
Rec
Acc
F1
VIT [17]
80.73
TransMed [15]
94.34
BotNet [67]
79.20
80.72
85.32
79.50
HCRT AM [43]
92.90
91.94
94.24
92.06
TranttooeTanneeeeeee [11]
83.92
83.16
88.10
83,48
GasHis- Transformer [1]]
98.55
97.38
97.97
97.97
LW Gasl Transfommr [11]
84.54
82.99
87.93
83.69
LW GasT Transformer
95.99
96.90
96.43
96.43
CLAdappeec.nNoo....
90.75
CLAdaptercunNeett
98.59
CLAdapler
95.01
92.45
95.32
93.71
CLAdapterynze
95.01
95.00
95.00
95.00
(I) Industrial: InsPLAD-fai fault [78]
(g) 3D Multimedia: Video Recognition
Glass Ins.
Light. RS.
Upper Sha.
Vari Grip
Yoke
Avg
Method
UCFI01 [66]
HMDB51 [37]
Method
Sus.
ROC
McmDPC [23]
86.10
54.50
DifferNet
82.81
99.08
92.42
91.20
96.77
92.46
CoCLR [24]
87.90
$4,60
AtenDDifeeret [63]
86.57
97.38
RSPNet [12]
64.70
FastFlow [1]
70.16
82.02
77.43
65.54
71.48
73.33
VideoMoCo [53]
78,70
49,20
RD+ [63]
86.21
97.54
83.67
93.85
92.46
90.75
VPCLR [16]
89.10
55.70
CS-Flow [55]
85.73
96.60
88.40
91.53
90.70
900.59
CVRL 154]
94,40
70.60
CFLOW-AD [22]
82,22
95.52
86.60
90.37
83.87
87.72
CORPI [29]
93.50
68.00
PatchCore [33]
78.44
78.91
pBYOLP
VideoMAE (74]
96.10
73.30
CLAdapter 96.43
99.94
98.63
96.43
100.00
98.29
CLAdapteryTr 94.64
99.87
98.44
96.07
100.00
97.80
CLAdapler gwin
97.60
75.80
uses only the first stage of SFT, achieving an average AUROC of 98.29% and an accuracy of 97.60% respectively. This demonstrates the efficiency of CLAdapter in feature transformation and model fine-tuning across real-works scenarios. Moreover, CLAdaptercuwweexa. and CLAdapter surpass the best-performing methods on the biomedical BreakHis dataset by 7.97% and 10.02% in Fl score, respectively, highlighting significant impact of CLAdapter on cross-domain medical with limited data. In summary, these results demonstrate the effectiveness of CLAdapter leveraging knowledge from large-scale datasets to adapt and excel in various downstream tasks, pushing the boundaries of computer vision applications across different industry and science domains.
Table 3:
Results on Tiny- Imagene and PACS classic visual datasets. Architecture variants: VIT-L ImageNma ViT-B for PACS. Best results are bold, while the second-best are underlined.
Muny-mmaeNNet [38]
PACS (40) (FT)
PACS [40] (PEFT)
Method
Acc
Mothod
Acc
Method
Acc
CaiT-S/36 [76]
86.74
Linear
71.88
VPT-Adapter[14]
76.76
DeiT-B/16-D [75]
87.29
Full
87.79
LoRA[28]
88.53
Swin-L44 [48]
SET
88.91
DeRA [46]
88.43
VIT-L [31]
86.43
L2-SP
87.74
MoRA [35]
89.09
CIAdapler
94.21
CLAdapter
91.41
CLAdaater
91.41
Comparison with Vision Models and Fine-Tuning Methods on ID and OOD Benchmarks. As listed Table we evaluate CLAAdapeer on both general in-distribution (ID) and out-of-distributioo (OOD) benchmarks to assess its effectiveness in adapting pre-trained models under data-limited condi-
7

---

tions. On the Liny-mmaeNNet dataset, which serves as general classification benchmark, CLAdapter substantially improves ViT-L's baseline performance from 86.43% to 94.21% %, outperforming stronger architectures such as Swin-L by margin of 2.86%. For cross-domain generalization, we conduct evaluations on the OOD benchmark PACS. CLAdapter achieves 1.11 1% accuracy, outperforming full fine-tuning (by 3.62%) and L.2SPP (by 3.67% %).. When compared with parameter-effiieet fine-tuning (PEFT) methods, CLAdapter surpasses VPT by a large margin of 1 14.65%, and maintains at least a 2.32% lead over recent approaches such LoRA, DoRA, and MoRA. The results highlighting the robustness and efficiency of CLAdapter for both ID and OOD scenarios.
4.3 Ablation Study
Discussion on Cluster Center Numbers. The number of cluster centers K in Equation (1) within the CLAdatper is an adjustable hyperparameter. An excessive number of cluster centers might cause the model to overfit, particularly in scenarios where downstream tasks offer limited numbers and diversity of samples. On the other hand, too few cluster centers may fail to transfer all data information, leading underfitting To explore on the BreakHis dataset using the ViT-B model pre-trained on LAION-2B. The results Table indicate that setting K to 20 yields the most improvements for downstream tasks, with optimal F1 score of 93.71% % and an accuracy of 95.32%. Therefore, we recommend 20 as the default setting for the hyperparameter K.
optimal value for this parameter,
conduct experiments
Table 4: Comparison of fine-tuning results on the BreakHis dataset under different cluster center numbers K. The best results are in bold.
Scores
Cluster Center Number (K)
10
25
100
200
Ace F1
92.81
92.81
92.45
95.32
94.24
93.88
93.52
91.73
92.81
90.40
90.73
90.41
93.71
92.59
92.14
91.77
89.03
90.42
Valid
Valid
1.0
122
0.5
400
Epecka
Epecka
(8) Linear Probing VIT-B
(b) Full Fine-tuning VIT-B
Suturation Epoch: 50+, Truining Parums: 7%
Saturation Epoch:
90+, Training Parums: 100%
Modes
Epochs
Spochs
Fpecla
Epecls
(c) CLAdapter with SFT Stage
(d) CLAdapter with SFT Stage 2
Saturation Epoch: 40-, Truining Purams: 7%
Saturation Epoch: 30+, Training Params: 100%
Figure 4: Efficiency comparison of fine-tuning methods. CLAdapter achieves significant performance improvement with fewer training epochs and parameters, indicating both high effectiveness and efficiency. Note that standard augmentations only applied to the training set mitigate overfitting but not to the validation set, which results in lower validation loss than training loss.
Analysis of Efficiency, The proposed CLAdapter maintains high parameter efficiency while signifi- cantly improving performance. As detailed in Appendix B.2 (Table 7), it introduces only 7-10.4% additional parameters to backbone models (C-VVVeX with minimal computational over- head, yet achieves F1-score gains of up to 175.599 775.99. Furthermore, CLAdapter also shows advantages in downstream task fine-tuning. As shown in Figure 4, by only fine-tuning 7% of the model param- eters for 40 epochs in the first stage, CLAdapter achieves results comparable to full fine-tuning of
8

---

100% of the parameters for 90 epochs. Although the second stage of SFT requires tuning 100% of the parameters, saturation is reached just 30 epochs, with an accuracy improvement of 12.44%. These experiments demonstrate the effectiveness and efficiency of our Cl.Adptter in fine-tuning and transferring pre-trained features.
Backbone Features
CLAdapter Features
Backbone Features
CLAdapler Features
(a) BreakHis dataset: red for ductal, yellow for lobular, green for mucinous, and blue for papillary carcinoma.
(b) HCRF dataset: red for normal gastric slices and blue for cancerous gastric slices.
Figure 5: The t-SNE visualizations demonstrating class separability and compactness. The compara- tive analysis highlights the enhanced discriminability of features via CLAdapter.
CLAdapter
Full (Ace: 87.79%
L2-SP (Aec: 87.14%
VPT (Ace: 76, 76%)
Liner (Aec: 71.8%)
(Ace: 91.4%%
Dog
Elephant
Gimfffe
Guitar
Horse
House
Penson
Figure 6: The I-SNE Feature visualization on PACS. The -acc-acyyyycy are reported additionally.
CL.Adapter demonstrated satisfactory separability and compactness.
Visual Analysis of Features. To further validate the effectiveness of CLAdapter in feature transfer, Figure visualizes the features using t-distributed stochastic neighbor embedding (t-SNE) on the BreakHis and HCRF datasets. It compares the features extracted by the Backbone and those refined by CLAdapter. In the BreakHis dataset, post-CLAdapter application, the classes exhibit more distinct clustering. The ductal, lobular, mucinous, and papillary carcinomas demonstrate increased intra-class compactness, underlining the robustness of CLAdapter in feature representation. Similarly, on the HCRF dataset, CLAdapter maximizes the inter-class distances of samples, effectively distinguishing between normal and cancerous gastric slices. In addition, to assess the quality of features extracted, we visualize the feature distributions for all fine-tuning methods on PACS test set using t-SNE. The experiments are performed based on the ViT-B model pre-trained on lageNet-22K. As shown in Figure CLAdapter significantly improves the separability of representation vectors of different classes, exhibiting superior representational capacity.
more separable and
Conclusion & Limitation
This work introduces the Cluster Attention Adapter (CLAdapter), a novel method designed to bridge the gap between large-scale pre-training on diverse datasets and fine-tuning for data-limited down- stream tasks, particularly in diverse scientific domains. By leveraging attention mechanisms and clustering techniques, CLAdapter refines and adapts pre-trained models to enhance their performance significantly on wide array of downstream tasks, showcasing superior adaptability and effectiveness. In addition, benefiting from our unified interface design, CLAdapter effortlessly merges with main- stream models. Moreover, an SFT strategy is presented to collaborate with CLAdapter to enhance the finettnnig performance further. Through rigorous testing across ten diverse datasets, encompass- ing generic, multimedia, biological, medical, industrial, agricultural, environmental, geographical, materials science, OOD, and 3D analysis domains, CLAdapter all achieves a new sfate-of-the-ad performance, highlighting its effectiveness in addressing the unique challenges of data scarcity and domain shift in scientific applications. Limitations: Currently, CL.Adapter has not been specifically designed or validated for detection segmentation. We leave these extensions for future work.

---

References
[1] Marco Aldinucci, Marco Danelutto, Peter Kilpatrick, and Massimo Torquati. Fastflow: High- level and efficient streaming on multicore. Programming Multi-core and Many-core Computing Systems, pages 261-280, 2017.
[2] Mohammad Alkhatib and Adel Hafiane. Robust adaptive median binary pattern for noisy texture classification and retrieval. IEEE Transactions on Image Processing, 28(1 1):5407-5418, 2019.
[3] Rao Muhammad Anwer, Fahad Shahbaz Khan, Joost Van De Weijer, Matthieu Molinier, and Jorma Laaksonen. Binary patterns encoded convolutional neural networks for texture recognition and remote sensing scene classification. ISPRS Journal of Photogrammetry and Remote Sensing, 138:74-85, 2018.
[4] Tewodros Weldebirhan Arega, Stéphanie Bricq, and Fabrice Meriaudeau. Post-hoc out-of- distribution detection for cardiac mri segmentation. Computerized Medical Imaging and Graphics, 199:102 2025.
[5] Hangbo Bao. Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers. arXiv preprint 4rXiv:2106.08254, 2021.
[6] Neslihan Bayramoglu, Juho Kannala, and Janne Heikkilä. Deep learning for magnification independent breast cancer histopathology image classification. In 2016 23rd International Conference Pattern Recognition (ICPR), pages 2440-2445. IEEE, 2016.
[7] Qi Bi, Han Zhang, and Kun Qin. Multi-scale stacking attention pooling for remote sensing scene classification. Neurocomputing, 436: 136:147-161, 2021.
[8] Joao Carreira, Eric Noland, Andras short note about kinetics-600. arXiv preprint arXiv:1808. 01340, 2018
Chloe Hillier, and Andrew Zisserman. A
[9] Souleyman Chaib, Huan Liu, Yanfeng Gu, and Hongxun Yao. Deep feature fusion for vhr remote sensing scene classification. IEEE Transactions on Geoseience and Remote Sensing, 55(8))4775 4788, 2017.
[10] Chia-Ling Chang, Yen-Liang Chen, and Dao-Xuan Jiang. Using large multimodal models to predict outfit compatibility. Decision Support Systems, page 114457, 2025.
[11] Haoyuan Chen, Chen Li, Ge Wang, Xiaoyan Li, Md Mamunur Rahaman, Hongzan Sun, Weiming Hu, Yixin Li, Wanli Liu, Changhao Sun, et al. Gashis-transformer:: A multi-scale visual transformer approach for gastric histopathological image detection. Pattern Recognition, 130: 108827, 2022.
[12] Peihao Chen, Deng Huang, Dongliang He, Xiang Long, Runhao Zeng, Shilei Wen, Mingkui Tan, and Chuang Gan. RspNet: Relative speed perception for unsupervised video representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1045-1053, 2021.
[13] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020.
[14] Mircea Cimpoi, Subhransu Maji. and Andrea Vedaldi. Deep filter banks for texture recognition and segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1828-3836, 2015.
[15] Yin Dai, Yifan Gao, and Fayu Liu. Transmed: Transformers advance image classification. Diagnostics, 11(8):1384, 2021.
medical
[16] Ali Diba, Vivek Sharma, Reza Safdari, Dariush Lotfi, Saquib Sarfraz, Rainer Stiefelhagen, and Lue Van Gool. Vi2clr: Video and image for visual contrastive learning of representation. In Proceedings of the IEEE/CV International Conference on Computer Vision, pages 1502-
12, 2021.
10

---

[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 6 words: Transformers for image recognition at scale. arXiv preprint 2020.
arXiv:2010.1199
[18] Jie Fang, Yuan Yuan, Xiaoqiang Lu, and Yachuang Feng. Robust space-frequency joint representation for remote sensing image scene classification. IEEE Transactions on Geoscience and Remote Sensing, 57(10)7492-7502, 2019.
[19] Yuxin Fang. Wen Wang. Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning scale. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1936,,19369, 2023.
[20] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study unsupervised spatiotemporal representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1299-3309, 2021.
[21] Joao B Florindo and Konradin Metze. cellular automata approach local patterns for texture recognition. Expert Systems with Applications, 179: 15227, 2021.
[22] Denis Gudovskiy, Shun Ishizaka, and Kazuki Kozuka. Cflow-ad: Real-time unsupervised anomaly detection with localization via conditional normalizing flows. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 188-177,20 2022.
[23] Tengda Han, Weidi Xie, and Andrew Zisserman. Memory-augmented dense predictive coding for video representation learning. In European Conference Computer Vision, pages 312-329. Springer, 2020.
[24] Tengda Han, Weidi Xie, and Andrew Zisserman. Self-supervised co-training for video rep- resentation learning. Advances in Neural Information Processing Systems, 33:5679-566 2020.
[25] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[26] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1921-1930, 2019.
[27] Md Ismail Hossen, Mohammad Awrangjeb, Shirui Pan, and Abdullah Al Mamun. Transfer learning in agriculture: a review. Artificial Intelligence Review, 58(4):97, 2025.
[28] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR. 1(2):3, 2022.
[29] Kai Hu, Jie Shao, Yuan Liu, Bhiksha Raj, Marios Savvides, and Zhiqiang Shen. Contrast and order representations for video self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7939-7949, 2021.
[30] Xiaolong Huang, Qiankun Li, Xueran Li, and Xuesong Gao. One step learning, one step review. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22644-12652 2024.
[31] Ethan Huynh. Vision transformers in 2022: An update tiny imagenet. arXiv preprint arXiv:2205.1 10660, 2022.
[32] Ethan Huynh. Vision transformers in 2022: An update on tiny imagenet. arXiv preprint arXiv:2205.I 10660, 2022.
[33] Kengo Ishida, Yuki Takena, Yoshiki Nota, Rinpei Mochizuki, Itaru Matsumura, and Gosuke Ohashi. Sa-patchcore: Anomaly detection in dataset with co-occurrence relationships using self-attention. IEEE Access, 11:3232-3240, 2023.

---

[34] Menglin Jia, Luming Tang, Bor- Ch-C Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727. Springer, 2022.
[35] Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang. et al. Mora: High-rank updating for parameter- efficient fine-tuning, arXiv preprint arXiv:2405 2024.
[36] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 11XXv:1705.06950 705.06550. 2017.
[37] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 2011 International Conference on Computer Vision, pages 2556-2563. IEEE, 2011.
[38] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[39] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[40] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE International Conference on Computer Vision, pages 5542-5550, 2017.
[41] Qiankun Li, Xiaolong Huang, Zhifan Wan, Lanqing Hu, Shuzhe Wu, Jie Zhang, Shiguang Shan, and Zengfu Wang. Data-efficient masked video modeling for self-supervised action recognition. In Proceedings of the 31st ACM International Conference on Multimedia, pages 2723-2733, 2023.
[42] Xiangtai Li, Henghui Ding, Haobo Yuan, Wenwei Zhang. Jiangmiao Pang, Guangliang Cheng, Kai Chen, Ziwei Liu, and Chen Change Loy. Transformer- based visual segmentation: A survey. IEEE transactions on pattern analysis and machine intelligence, 2024.
[43] Yixin Li, Xinran Wu, Chen Li, Xiaoyan Li, Haoyuan Chen, Changhao Sun, Md Mamunur Rahaman, Yudong Yao, Yong Zhang, and Tao Jiang. A hierarchical conditional random field- based attention mechanism approach for gastric histopathology image classification. Applied Intelligence, pages 1-22, 2022.
[44] Zongxia Li, Xiyang Wu, Hongyang Du, Huy Nghiem, and Guangyao Shi. Benchmark evalua- tions, applications, and challenges of large vision language models: A survey. arXiv preprint arXiv:2501.00 002189, 1, 2025.
[45] Mengkun Liu, Licheng Jiao, Xu Liu, Lingling Li, Fang Liu, Shuyuan Yang, and Xian- grong Zhang. Bio-inspired multi-scale contourlet attention networks. IEEE Transactions on Multimedia, 2023.
[46] Shih- Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu- Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min- Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference Machine Learning, 2024.
[47] Xiao Liu, Chenxu Zhang, and Lei Zhang. Vision mamba: A comprehensive survey and taxonomy. arXiv preprint arXiv:2405 004404 2024.
[48] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[49] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3202-3211, 2022.
[50] Zhuang Liu, Hanzi Mao, Chao-Yuaa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 11976- 11976-11986, 2022.
12

---

[51] P Mallikarjuma, Alireza Tavakoli Targhi, Mario Fritz. Eric Hayman, Barbara Caputo, and Jan- Olof Eklundh. The kth- -ips2 database. Computational Vision and Active Perception Laboratory. Stockholm, Sweden, 11 11:12, 2006.
[52] Rakesh Mehta and Karen Egiazarian. Texture classification using dense micro-block difference. IEEE Transactions on Image Processing, (5(4):160-11116 2016.
[53] Tian Pan, Yibing Song, Tianyu Yang. Wenhao Jiang, and Wei Liu. Videomoco: Contrastive video representation learning with temporally adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11205-11214, 2021.
[54] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang. Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6964-6974, 2021.
[55] Ji Qiu, Hongmei Shi, Yuhen Hu, and Zujun Yu. Spatial activation suppression for unsupervised anomaly detectors freight train fault detection. IEEE Transactions on Instrumentation and Measurement, 2023.
[56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 8748-8763. PMLR, 2021.
[57] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[58] Tal Ridnik, Emanuel Ben- Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint MiXx:2104.10972 2021.
[59] Marco Rudolph, Bastian Wandt, and Bodo Rosenhahn. Same same but differnet: Semi- supervised defect detection with normalizing flows. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1907-1916, 2021.
[60] Marco Rudolph, Tom Wehrbein, Bodo Rosenhahn, and Bastian Wandt. Fully convolutional cross-scale-flows for image-based defect detection. Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 1088- 1097, 2022.
[61] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25294, 2022.
[62] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filterd 400 million image-text pairs. arXiv preprint arXiv:21 1tt.::211100222111 2021.
[63] Francisco Simöes, Danny Kowerko, Tobias Schlosser, Felipe Battisti, Veronica Teichrieb, et al. Attention modules improve image-level anomaly detection for industrial inspection: differnet case study. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 8246-8255, 2024.
[64] Francisco Simöes, Danny Kowerko, Tobias Schlosser, Felipe Battisti, Veronica Teichrieb, et al. Attention modules improve image-level anomaly detection for industrial inspection: differnet case study. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 8246 0216-825,,111 2024.
[65] Tiecheng Song, Hongliang Li, Fanman Meng, Qingbo Wu, and Jianfei Cai. Letrist: Lo- cally encoded transform feature histogram for rotation-invariant texture classification. IEFE Transactions on Circuits and Systems for Video Technology, 28(7):155--157, 2017.
13

---

[66] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos the wild. arXiv preprint arXiv: 1IXiv:1212.0402. 2012.
[67] Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16519-16529, 2021.
[68] Changhao Sun, Chen Li, Jinghua Zhang, Md Mamunur Rahaman, Shiliang Ai, Hao Chen, Frank Kulwa, Yixin Li, Xiaoyan Li, and Tao Jiang. Gastric histopathology image segmentation using a hierarchical conditional random field. Biocybereetic and Biomedical Engineering. 40(4):1535-1555 2020.
[69] Yajie Sun, Miaohua Zhang, Xiaohan Yu, Yi Liao, and Yongsheng Gao. A compositional feature embedding and similarity metric for ultra- fine- iine-grained visual categorization, In 2021 Digital Image Computing: Techniques and Applications (DICTA), pages 01-08. IEEE, 2021.
[70] Xu Tang, Mingteng Li, Jingjing Ma, Xiangrong Zhang, Fang Liu, and Licheng Jiao. Emtcal: Efficient multiscale transformer and cross-level attention learning for remote sensing scene classification. IEEE Transactions Geoscience and Remote Sensing, 60: 2022.
[71] Ranjita Thapa, Noah Snavely, Serge Belongie, and Awais Khan. The plant pathology 2020 challenge dataset to classify foliar disease of apples. arXiv preprint arXiv:2004. 11958, 2020.
[72] Ranjita Thapa, Kai Zhang, Noah Snavely, Serge Belongie, and Awais Khan. The plant pathology challenge 2020 data set to classify foliar disease of apples. Applications in Plant Sciences, 8())e1 2020.
[73] Radu Timofte and Lue Van Gool. A training-fres free classification framework for textures, writers, and materials. In BMVC, volume 13, page 14, 2012.
[74] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders data-efficient learners for self-supervised video pre-training. Advances in Neural Information Processing Systems, 15:10088-1009, 2022.
[75] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[76] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Go- ing deeper with image transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 32-42, 2021.
[77] Hsi-Ai Tsao, Lei Hsiung, Pin-Yu Chen, and Tsung- Ho. When does visual prompting out- perform linear probing for vision-language models? a likelihood perspective. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1 1-5. IEEE, 2025.
[78] André Luiz Buarque Vicira e Silva, Heitor de Castro Felix, Franscisco Paulo Magalhães Simöes, Veronica Teichrieb, Michel dos Santos, Hemir Santiago, Virginia Sgotti, and Henrique Lott Neto. Insplad: A dataset and benchmark for power line asset inspection in uav images. International Journal of Remote Sensing, 44(23):7294 7320, 2023.
[79] Junqi Wang, Lanfei Jiang, Hanhui Yu, Zhuangbo Feng, Raúl Castaño- and Shi-jie Cao. Computer vision to advance the sensing and control of built environment towards occupant- centric sustainable development: critical review. Renewable and Sustainable Energy Reviews, 192: 14165,, 2024.
[80] Ruiping Wang, Huimin Guo, Larry $ Davis, and Qionghai Dai. Covariance discriminative learning: natural and efficient approach to image set classification. 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2496 2503. IEEE, 2012.
[81] Gui-Song Xia, Jingwen Hu, Fan Hu, Baoguang Shi, Xiang Bai, Yanfei Zhong, Liangpei Zhang, and Xiaoqiang Lu. Aid: A benchmark data set for performance evaluation of aerial scene classification. IEEE Transactions Geoscience and Remote Sensing, 55(7):396--398, 2017.
14

---

[82] Gui-Song Xia, Jingwen Hu, Fan Hu, Baoguang Shi, Xiang Bai, Yanfei Zhong, Liangpei Zhang and Xiaoqiang Lu. Aid: A benchmark data set for performance evaluation of aerial scene classification. IEEE Transactions on Geoscience and Remote Sensing, 55(7):3955398) 2017.
[83] Yuting Yang, Licheng Jiao, Lingling Li, Xu Liu, Fang Liu, Puhua Chen, and Shuyuan Yang. Lglformer: Local-global lifting transformer for remote sensing scene parsing. IEEE Transactions Geoscience and Remote Sensing, 2023.
[84] Yuting Yang. Licheng Jiao, Fang Liu, Xu Liu, LingLing Li, Puhua Chen, and Shuyuan Yang. An explainable -aequel- multi-scale transformer for remote sensing scene classification. IEEE Transactions on Geoscience and Remote Sensing, 2023.
[85] Weihao Yu, Pan Zhou, Shuicheng Yan, and Xinchao Wang. Inceptionnext: When inception meets convnext. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 1172-5683, 2024.
[86] Xiaohan Yu, Jun Wang. and Yongsheng Gao. Cle-vit: Contrastive learning encoded transformer for altrr-fiie-grainee visual categorization. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 4531-4535 2023.
[87] Xiaohan Yu, Yang Zhao, and Yongsheng Gao. Spare: Self-supervised part crasing for ultra-fine grained visual categorization. Pattern Recognition, 128: 108691, 2022.
[88] Xiaohan Yu, Yang Zhao, Yongsheng Gao, Xiaohui Yuan, and Shengwu Xiong. Benchmark platform for stra-fine-grained visual categorization beyond human performance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10285-10295, 2021.
[89] Yuan Yuan, Jie Fang, Xiaoqiang Lu, and Yachuang Feng. Remote sensing image scene classification using rearranged local features. IEEE Transactions on Geoscience and Remote Sensing, 57(3):1779-1992, 2018.
15

---

A Experiment Configuration
A.1 Dataset Descriptions and Processing Methods
We experiment on 10 benchmarks across a broad spectrum of domains, including generic, multimedia, industrial, biological, medical, agricultural, environmental, geographical, materials science, OOD, and 3D analysis. This demonstrates the versatility and effectiveness of our CLAdapter. The datasets for each domain, class counts, and sample sizes detailed in Table 5.
Table 5: Statistics of datasets used for evaluating downstream task performance.
Dataset
Domains
Class
Train
Val
Test
Tiny-ImageNet [38]
General
200
100000
10000
10000
PACS [40]
General OOD
x
1588
6355
2048
BreakHis [6]
Biomedicine
4
834
278
278
HCRF [68]
Biomedicine
2
70
70
140
Apple Foliar Disease [72]
Agricultural
4
1366
455
WHU-RS19 [81]
Environmental Geography
19
402
100
503
KTH-TIPS-28 [51]
Matcrials Science
11
3564
1188
InsPLAD-fault [78)
Industrial
5 5108
6417
UCF101 [66]
3D Multimedia
101
9537
3783
HMDB51 [37]
3D Multimedia
51
3570
1530
ImageemageNet. Tiny-ImageNet [39] is a simplified version of the larger ImageNet dataset. It comprises 200 classes, each with 500 training images, 50 validation images, and 50 test images, resulting in total of 100,000 images. Liny-ImaeeNet serves a benchmark for evaluating algorithms on a wide array of generic visual recognition tasks, testing both the depth and breadth of models' understanding of visual concepts. In the experiment, the data division adheres official standards.
PACS. The PACS dataset [40] stands as critical benchmark for assessing domain generalization capabilities in general computer vision. It contains images from four distinct domains: Photo, Art Painting, Cartoon, and Sketch, addressing a broad spectrum of visual styles and compositions. With seven common object classes across these domains, the dataset poses significant challenge in learning domain- domain-invariant features. is particularly used for evaluating models on their ability to generalize from seen unseen domains, making and generalization. The Art Painting domain of the PACS dataset is exclusively utilized as the test set to assess cross-domaie performance, while the remaining data are divided into training and validation sets in a 5-fold manner, with a ratio of 1:4.
essential tool for research in domain adaptation
Apple Foliar Disease. The Apple Foliar Disease dataset [71] [] is specialized resource aimed at advancing the field of agricultural and plant disease recognition. It consists of high-quality images that capture various foliar discases affecting apple leaves, including but not limited to apple scab, cedar apple rust, and powdery mildew, as well as images of healthy leaves for comparison. Leveraging such datasets not only validates our CLAdapter's cross-domain effectiveness in agriculture but also aids researchers and agronomists in enhancing precision agriculture, enabling timely and effective disease management improve crop health and yield. The data split method follows the previous work as training and validation sets with a 3:1 ratio.
WHU-RS19. The WHU-RS19 dataset [82] is high-resolution remote sensing dataset, primarily used for the evaluation of land cover and land use classification algorithms in the field of geographical and environmental analysis. Originating from the Wuhan University Remote Sensing Group, this dataset encompasses a diverse collection of 19 classes representing various natural and man-made features, including but not limited to agricultural lands, forests, water bodies, residential areas, and industrial sites, The images in WHU-RS19 are collected from different satellite and acrial sensors, challenging and enhancing classification models' robustness in geographical and environmental fields. For data partitioning, our experiments are consistent with previous studies [84].
KTH -IP--B. The KTH-TIPS2-B B dataset [51] is an extension of the KTH-TIPS dataset, both of which are designed for the task of texture classification and material recognition in the field of computer vision, particularly focusing on the challenges associated with variations in scale, pose, and illumination. This dataset is curated by the KTH Royal Institute of Technology Sweden.
KTH-TIPS2-B consists of images representing set of 11 material categories, such as cotton, wool,
16

---

and aluminum, among others. Each material category includes images captured under different conditions and from multiple angles, providing comprehensive data for evaluating the performance of texture analysis algorithms. The data split method remains consistent with previous work [45].
BreakHis. The BreakHis dataset [6] comprises 7,909 breast cancer images across four magnification levels, divided into eight sub-classes. Originating from 82 anonymous patients in Brazil, BreakHis is key dataset in digital breast histopathology research. Malignant tumor images at a 200x magnification, including ductal carcinoma (DC), lobular carcinoma (LC), mucinous carcinoma (MC), and papillary carcinoma (PC), are used for classification. The dataset is split into training, validation, and testing sets in a 3:1:1 ratio, which is the same as the previous study [11].
The HCRF dataset [68], well-known in gastric histopathology, consists of 560 cancerous and 140 normal images. Following the previous works [1 I, the HCRF dataset is divided into training, validation, and test sets as 1:1:2 random stratified ratio. Available Mendeley Data, it serves an important resource for evaluating model performance in the field of computer vision and biomedicine.
HCRE.
The InsPLAD dataset [78], pivotal for advancing power line asset inspection, includes the InsPLAD-fault subset, a specialized collection designed for anomaly detection tasks in power line components. This subset harnesses real-world images captured by unmanned aerial vehi- cles (UAVs) of operational power line transmission towers, offering a unique challenge in the realm of industrial defect detection. It encapsulates five distinct categories of power line objects, facilitating deep learning models in effectively identifying and classifying anomalies. In the experiment, the data split method remains consistent with previous work [64].
InsPLAD-fault.
UCF101. The UCF101 [66] a widely recognized dataset in the field of action recognition, developed by the University of Central Florida. It's one of the most popular benchmarks for evaluating the performance of video-based action recognition algorithms. The dataset features 101 action categories, encompassing broad range of activities such as sports, playing musical instruments, and human- object interactions. Each category in the UCF101 dataset consists of multiple video clips, amounting to over 13,000 clips and totalling more than 27 hours of video data. The videos are collected from YouTube and represent diverse actors, backgrounds, and lighting conditions. This diversity poses a challenge for action recognition systems, requiring the models to generalize across different environments and subject appearances. The data division method follows the official release.
HMDB51. The HMDB51 [37] is a comprehensive video dataset aimed at the task of human action recognition. Compiled by researchers from Brown University, it consists of 51 action categories, each containing at least 101 video clips, resulting in total of over 6,800 clips. These actions span wide array of human activities, including facial actions, general body movements, and interactions with objects. Due to the limited size of its dataset and the lack of diversity among samples, the HMDB51 dataset poses more challenges than UCF101 Additionally, the data splitting method is consistent with the official.
A.2 Implementation Details
We meticulously design our experimental settings to ensure comparability and reproducibility.
For experiments on the Tiny-ImageNet, follow the protocols established in [32], ensuring consistency with prior benchmarks.
For 3D video analysis on UCF101 and HMDB51 datasets, our experimental configurations align with the previous methods [41], which facilitates direct comparison with existing SOTA approaches.
Regarding the remaining datasets, encompassing cross-domain and various real-world domain appli- cations, we establish uniform implementation details to underscore the adaptability and convenience of CLAdapter. Specifically, we adopt an input resolution of 224 x 224 pixels across all experiments. The learning rate is initialized at le - 4, and models are trained for up to 100 epochs with batch size of 16. We employ the Adam optimizer, configuring it with momentum 31 = 0.9 and weight decay of le - 3, to adapt to the unique challenges presented by these varied datasets. The determination of cluster centers for CLAdapter is fixed at 20, balancing granularity and computational efficiency. Our experimental setup is powered by four Nvidia GeForce RTX 3090 GPUs, boasting 24 GB of memory, under the Ubuntu 20.04 environment. Python 3.8.3 is chosen as the programming language, with the Py Torch 1.13.I framework being utilized for model development.
17

---

B Additional Experimental Analyses
B.1 Intuitive Performance Comparison of Our Method with SOTA and Baseline Methods
In Table of the main manuscript, we present comprehensive comparison of various methods across diverse scientific domains for downstream tasks (subtables a-g). Furthermore, Table 3 highlights the significant performance gains achieved by CLAdapter on both the in-distribution (ID) Ting-et and the nut-of-distribution (OOD) PACS benchmarks.
To provide more intuitive understanding of the advantages of our method, we visualize the perfor- mance comparison, as shown in Figure 3. It can be seen that CLAdapter consistently outperforms state-of-the thertrrrrt methods across different datasets. In addition, results in Table 6. By using the best fine-tuning strategy and keeping the same backbone as ours for comprehensive evaluation, our method still maintains substantial improvements over both baselines and prior SOTA methods.
listed all benchmark baseline
Table 6: Per-Domain Improvement over Baseline (VIT) and SOTA using CLAdapter.
Improve% 1
ID
OOD
Agricultural Geography Materials Biomedicine Industrial 3D Multimedia
Ours Baseline 7.8 3.6
2.0
2.8
2.5
12.2/13.6
4.6
2.5/4.1
Ours vs SOTA
0.9
4.3
10.0/0.6
1.5/2.5
B.2 Analysis of Efficiency
As a universal adapter, our method exhibits relative efficiency under popular pre-trained models. Efficiency analysis results listed in Table 7. For ConvNeXt-B and ViT-B models, CLAdapter adds only 10.4% and 7% more parameters, respectively, while slightly increasing computational complexity (Flops) by 0.44G for ConvNeXt-B and IG for VIT-B. Despite this minimal increase in size and computation, CLAdapter achieves remarkable F1 score improvements of up to 175.59% for ConvNeXt-BB and 51.46% for ViT-B.
Table 7: Comparison of Method efficiency.
Method
Params(M)
Flops(G)
F1
Train
ConvNeXt-B
88.85
15.42
33.26
100%
CLAdapter CLAdapterrcmNeexBssr
99.11
15,86
80.89
10.4%
CLAdapter CLLAdaptercu.NNx..SSS SuwNNoX
99.11
15.86
91.66
100%
VYT-B
85.77
16.86
61.87
100%
CLAdapter VIT---SF.. CI
92.22
17.86
84.34
7.0%
CLAdapter,
VIT-B-SFT-2 CLAdapterVTT..FFF.
92.22
17.86
93.71
100%
B.3 Comparison Using Different Scales of Pre-training Data
To delve into how the scale of large visual datasets influences cross-domain fine-tuning, perform comparative experiments on the PACS and BreakHis datasets using ConvNeXt-B and ViT-B models pre-trained on three large-scale datasets, i.e., ImageNet-21K, LAION-400M and LAION-2B.
Table 8: Results of using different pre-training resources on the PACS dataset.
Method
ImageNet-21K
LAION-2B
Linear
71.88
95.61
Full
87.79
47.17
L2-SP
87.74
45.56
VPT
76.76
97.46
CLAdapter
91.41
97.62
18

---

Table lists the line-tuning results on the PACS dataset based on VIT-B. It is observed that our CLAdapter achieves an accuracy of 97.62% under the LAION-2B pre-training dataset, which is enriched with more diverse knowledge, marking 6.21% % improvement over the scaller-scale ImageNet-21K dataset. Notably, both Full fine-tuning and 1.2-SP exhibit poor performance on LAION-2B, likely due to pattern collapse encountered during the transfer of massive pre-trained features, a problem that VPT and our CLAdapter circumvent through additional parameter transfor- mation. Moreover, our CLAdapter's accuracy under ImageNet-21K pre-training surpasses that of VPT by 14.65% indicating that CLAdapter is also capable of extracting and transforming a sufficient amount of downstream-relewe information from relatively smaller pre- vre-training datasets.
Table 9: Results of using different pre-training resources on the BreakHis dataset.
Method
ImageNet-21K
LA1ON-400M
LAION-2B
SFT Stage !
CLAdapter CONNNEER
80.89
71.72 80.63
75.17
84.34
80.67
SFT Stage
CLAdapter.ommee B
88.60
90.55
91.66
CLAdapter ViBB
90.19
91.35
93.71
The Fl score results on the BreakHis dataset are listed in Table 9. Whether combined with ConvNeXt- or ViT-B pre-trained models, CLAdapter achieves the best fine-tuning results on larger pre-training datasets. Another notable observation is that when only the first stage of the SFT fine-tuning strategy employed (i.e., freezing the weights of the pre-trained model), pre-training on the smaller-scale scale and knowledge-limbee ImageNet-2 2ett-21KK dataset yields better results. This suggests that freezing the pre-trained model weights limits CLAdapter's s capacity for transformation, preventing the thorough transfer and refinement of rich knowledge the downstream tasks. Nevertheless, with just the first fine-tuning stage, CLAdapter still surpasses other SOTA methods on the BreakHis His dataset (as listed in Table 2d).
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose CLAdapter, a cluster-attention adapter that refines foundation vision model representations across CNN and Transformer architectures with negligible overhead. We validate its superior performance on ten diverse data-limited scientific datasets and provide detailed experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper explicitly discusses limitations in the final paragraph the conclu- sion, noting that CLAdapter has not yet been specifically designed evaluated for object detection or segmentation tasks. These directions are identified as future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
20

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Yes, we provide a complete derivation for our proposed CLAdapter in Section
3.2.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all experimental settings and codes are provided in the paper and the Appendix ensure full reproducibility of our main results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
21

---

Please refer to some ablation experiments, hyperparameter settings, and 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We use only publicly available datasets and have released our full code
GitHub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
Baallii/guides/ooeSSubbaee wwm.iiiiiiiiccciooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //iiss... wwwpccccppc/..ec..c...ccce.cetle hwmblc/gguides..../mee/ /ttegguuiee//ooeuuuaiiiii SCodeSeumissionnd ssoooloiiiiio for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
All important settings can be found in our paper and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the training resources we use in the experimental settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :tttpps://nuuurtee /nerrps....pp// cc cccppblic//eeeccccccccccccccclct wtw.ieeeiieleeeeeeeeee line?
Answer: [Yes]
Justification:
Our research conforms with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the broader impacts in the conclusion. The research is inherently scientific, and anticipate adverse socictal impact stemming from its findings.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our method doesn't have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
All the creators or original owners of assets are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper introduces code for CLAdapter and the documentation be found the anonymized URL: http:///wwnooooooooooooo////eeea
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our CLAdapter does not involve LLMs as any important, original, non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (uttps://neur)) htt.r/ppsccc/oooffeeeee////2255 for what should or should not be described.
26


Reviews:
[{"review_id": "Wg7eHEUGIL", "reviewer": "Reviewer_UMqW", "summary": "This paper introduces CLA dapter, a cluster-based attention adapter designed for efficient adaptation of foundation vision models to data-limited scientific domains. By leveraging attention mechanisms and distribution-aware cluster centers, CLAdapter refines pretrained features to better match diverse downstream tasks. It supports both CNNs and Transformers in 2D/3D settings through a unified interface. Experimental results show promising performance, demonstrating the effectiveness in enabling adaptive transfer. The method also achieved top rankings in multiple scientific vision challenges, further validating its practical effectiveness in adaptive transfer.", "questions": "1. Please provide more analysis/discussion with Lora based methods.\n2. Please provide more analysis on inference efficiency or adapter compression.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The CLA adapter design is novel. It introduces multiple residual adapters inserted into multiple layers\n2. The method is validated on multiple datasets covering a wide range of topics, demonstrating versatile transferability in both classification and segmentation tasks.\n3. The method ranks top on multiple vision-related competitions, which reinforces its real-world effectiveness\n\nWeaknesses:\n1. Please provide more analysis/discussion with Lora based methods.\n2. Please provide more analysis on inference efficiency or adapter compression\n3. Please add the dataset/paper reference (e.g. PACS) in the paper body text instead of adding them in the table.", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer UMqW on more Lora-based methods analysis, more inference efficiency analysis, and dataset description position.\n\nThank you very much for your recognition of the good performance, novel design, extensive validation, and real-world effectiveness. \n\nYour constructive comments and suggestions are exceedingly helpful to improve our paper. We have carefully incorporated them in the revised paper. In the following, your comments are first stated and then followed by our point-by-point responses. \n\n### Comment 1:\n> Please provide more analysis/discussion with LoRA-based methods.\n\n### **Authors’ Response:**\n\nThank you for your suggestion. Our **CLAdapter differs significantly from LoRA-based methods** in both *design focus* and *target domains*.\n\n- **1. Design & Applicability Analysis/Discussion**\n    - | Aspect | **LoRA-based** | **CLAdapter (Ours)** |\n      |--------|------------------------|---------------------- |\n      | **Applicable Backbone** | Transformer-based | CNN-based & Transformers-based & their 3D versions |\n      | **Design Motivation** | Large scale **LLM** Efficient training | Cross-domain Transfering: ***foundation vision*** models → **data-limited scientific domains**. |\n      | **Application Domain** | General tasks | *Scientific & Data-Limited domains*: medical, biological, materials, environmental, etc. |\n\n\n\n- **2. Performance Analysis/Discussion**\n    - We already included a ViT-L backbone in the manuscript (Table 2) to provide a fair comparison with LoRA. On **PACS** (popular datasets in the AI field), CLAdapter achieves **91.41%**, surpassing LoRA (**88.53%**) by **+2.88%**. We also outperform advanced versions (**DoRA** and **MoRA** by **+2.98%** and **+2.32%**, respectively).\n\n    - If the **comparison** domain is **changed** to the ***data-limited scientific domain*** where **CLAdapter excels**, CLAdapter can be **expected** to be much better than Lora-based methods (**improve 10%+**).\n\n    | **Method** | PACS (Manuscript Table 2, Acc%) | KTH-TIPS2-B (Acc%) | BreakHis (F1%) |\n    |------------|-------------|------------------------------|------------------------------|\n    | LoRA       | 88.53       | 79.18                        | 80.22                        |\n    | DoRA       | 88.43       | 80.28                        | 79.98                        |\n    | MoRA       | 89.09       | 80.11                        | 83.45                        |\n    | **CLAdapter (Ours)** | **91.41** | **91.26** | **93.71** |\n\n\n---\n\n### Comment 2:\n> Please provide more analysis on inference efficiency or adapter compression.\n\n### **Authors’ Response:**\nThank you for your suggestion. While Figure 4 in the paper already highlights the Fine-tuning efficiency of CLAdapter, we now **additionally** analyze **inference efficiency** and **parameter overhead**.\n\nWe evaluate on an NVIDIA 3090 GPU using two typical backbones:\n- ConvNeXt-B + CLAdapter\n  - GFLOPs: 15.42 → 15.86\n  - Params: 88.85M → 99.11M\n  - → Only ~2 FPS drop in inference speed\n- ViT-B + CLAdapter\n  - GFLOPs: 16.86 → 17.86\n  - Params: 85.77M → 92.22M\n  - → Only ~3.5 FPS drop in inference speed\n\nThese results demonstrate that CLAdapter **adds minimal inference overhead** while delivering significant gains in diverse data-limited scientific domains.\n\n---\n### Comment 3:\n> Please add the dataset/paper reference (e.g. PACS) in the paper body text instead of adding them in the table.\n> \n\n### Authors’ response:\nThank you very much for your valued comment, **following your suggestions:**\n- We have revised the manuscript to explicitly mention and cite all benchmark datasets (e.g., PACS, BreakHis, FoliarDisease, KTH, UCF, etc.) **in the main body text** (***Section 4.1**  [Experiment Setup] (**Paragraph** [Downstream Tasks])*, **rather than** referring to them **only in the Table 5/Appendix A.2**. \n- This revision will improve clarity and paper formatting, we appreciate your guidance!"}, {"review_id": "e4nTtYKl9L", "reviewer": "Reviewer_fSgx", "summary": "Data scarcity remains a significant challenge in scientific vision tasks, limiting the effective training of vision models. While existing transfer learning frameworks perform well when downstream tasks are closely aligned with the distribution of the pretraining data, they often fall short in scientific applications where data is both limited and typically out-of-distribution.\nTo address this, the paper proposes CLAdapter combined with a Staged Fine-Tuning strategy, which adapts pretrained representations—whether from ViT-based or CNN-based models—to better suit downstream tasks under low-data, out-of-distribution conditions.", "questions": "It would be valuable to more clearly highlight how CLAdapter improves performance relative to existing methods, particularly in the context of varying downstream dataset sizes. Specifically, explicitly reporting the number of training examples available for each downstream task would help contextualize the challenges of low-data regimes. Emphasizing CLAdapter’s robustness in tasks with extremely limited data would strengthen the case for its effectiveness, especially in out-of-distribution or scientific vision scenarios where data scarcity is common. This comparison would also underscore the practical advantages of the proposed approach in real-world settings where large-scale annotated datasets are unavailable.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "- The idea of using learnable cluster centroids to model the features relevant to the downstream task is novel and interesting.\n- The paper is technically sound. Extensive experiments and results support the claims.\n- The results are impactful for different scientific domains and the generalizability makes it even more valuable.\n- Highlighting how CLAdapter boosts the performance as compared to the existing methods, while taking into consideration the downstream task’s dataset size would be beneficial.", "was_revised": true, "final_justification": "In rebuttal, the addition of a detailed comparison table showing performance gains across a diverse set of scientific and general datasets with corresponding train sizes effectively demonstrates the method’s robustness in low-data and out-of-distribution settings.\nThis added analysis further supports the practical relevance of CLAdapter, especially in real-world scenarios where annotated data is limited.\nGiven the originality of the proposed approach, the strong experimental results, the other reviewers' comments, I maintain my rating.", "rebuttal": "# Response to Reviewer fSgx on more clearly highlighting CLAdapter's value.\n\nThank you very much for recognizing the strengths of our work, including the novel and interesting design, the strong empirical validation, and the impactful results and valuable across general and scientific domains. \n\n\nBelow, we address your constructive comments and suggestions, and describe the corresponding revisions we have made.\n\n\n\n### Comment 1:\n> It would be valuable to more clearly highlight how CLAdapter improves performance relative to existing methods, particularly in the context of varying downstream dataset sizes. Specifically, explicitly reporting the number of training examples available for each downstream task would help contextualize the challenges of low-data regimes. Emphasizing CLAdapter’s robustness in tasks with extremely limited data would strengthen the case for its effectiveness, especially in out-of-distribution or scientific vision scenarios where data scarcity is common. This comparison would also underscore the practical advantages of the proposed approach in real-world settings where large-scale annotated datasets are unavailable.\n\n### **Authors’ Response:**\nThank you for this important suggestion. \n\nThe aspects you highlighted: ***training sample size*** &  ***performance gains*** & ***effectiveness under limited-data*** & ***scientific scenarios*** are indeed central to the **core value of CLAdapter**. \n\nPresenting these **clearly helps underscore** CLAdpater practical advantages in real-world settings.\n\n- **Following your valuable suggestion**, we compared these dimensions in the following unified table:\n\n    | **Dataset (Domain)** | **Train Size** | **CLAdapter vs. Baseline (\\%)** | **CLAdapter vs. SOTA (\\%)** |\n    |----------------------|----------------|----------------------------------|------------------------------|\n    | Tiny-ImageNet (General)         | 100,000     | +7.8                          | +2.9                        |\n    | PACS (General OOD)              | 1,588       | +3.6                          | +3.6                        |\n    | Apple Foliar Disease (Agri)     | 1,366       | +2.0                          | +0.3                        |\n    | WHU-RS19 (Geography)            | 402         | +2.8                          | +0.9                        |\n    | KTH-TIPS2-B (Materials)         | 3,564       | +2.5                          | +4.3                        |\n    | **BreakHis (Biomedicine)**      | **834**     | **+12.2**                     | **+10.0**                   |\n    | **HCRF (Biomedicine)**          | **70**      | **+13.6**                     | **+0.6**                    |\n    | InsPLAD-fault (Industrial)      | 5,108       | +4.6                          | +4.0                        |\n    | UCF101 (3D Multimedia)          | 9,537       | +2.5                          | +1.5                        |\n    | HMDB51 (3D Multimedia)          | 3,570       | +4.1                          | +2.5                        |\n\n\n- **In addition** to **highlight** the above revised content in the **paper body text**, we have **also included a more clearly and comprehensive presentation** in:\n    -  Appendix A.2 (Table 5, dataset details) \n    -  Appendix B.1 \n        -  Figure 5, clear & vivid comparison **bar chart and line plot**\n        -  Table 6, clear & detail comparison"}, {"review_id": "uiHNuRPvyy", "reviewer": "Reviewer_AwdT", "summary": "This paper proposes a novel method called Cluster Attention Adapter (CLAdapter), which addresses the challenges of transferring large-scale pre-trained vision models to data-limited downstream tasks in scientific domains by leveraging feature clustering and a progressive training strategy.", "questions": "1. In typical classification tasks using Vision Transformers (ViTs), only the class token is used for classification. However, this paper discards the class token and instead adopts a more complex approach based on patch token aggregation. Is there any experimental evidence supporting this design choice? Additionally, why does the class token have a negative effect in the experimental scenarios considered in this work? In Table 1, do all experiments based on ViT models use the same type of output token for classification?\n\n  2. In Figure 3, are the training loss and validation loss labels possibly reversed?\n   \n  I would consider increasing my score if the authors address the identified weaknesses and provide satisfactory responses to the questions raised in the review.", "limitations": "The paper mentions that CLAdapter is not applicable to detection and segmentation tasks. However, as a discriminative task, semantic segmentation is not fundamentally different from image classification. It may be worth exploring the application of CLAdapter to semantic segmentation tasks, although the computational cost could be higher.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "- strength\n  1. By incorporating an attention mechanism and learnable clustering centers, CLAdapter adaptively refines the feature representations of pre-trained models, resulting in significantly improved transfer performance in data-limited scientific domains.\n\n  2. The proposed two-stage fine-tuning strategy first updates only the adapter and classification head, and then performs full model fine-tuning. This approach effectively balances computational efficiency with performance optimization.\n\n- weaknesses\n  1. While a standard classification head typically uses a single linear layer (i.e., linear probing), CLAdapter introduces a new form of classification head that includes nonlinear modules and a larger number of parameters. This makes the experimental comparison potentially unfair. It is strongly recommended that the authors demonstrate that CLAdapter still achieves significantly better performance when compared to nonlinear classification heads with a comparable number of parameters.\n\n  2. Similarly, for the SFT setting, it is recommended to replace CLAdapter with a simple nonlinear classification head that has a comparable number of parameters, and then perform a comparison using the same SFT strategy.\n\n  3. The experimental comparisons appear somewhat disorganized. To properly demonstrate the effectiveness of CLAdapter, a consistent set of pre-trained models (e.g., {MAE-B, MAE-L, DINO-B, DINOv2-B, ...}) should be used across different datasets, comparing their performance with and without CLAdapter. However, in the current version of the paper, even the baseline results for ConvNeXt are missing, which makes it difficult to assess the actual improvement brought by CLAdapter.\n\n  4. Recently, self-supervised pre-trained models have become widely adopted. However, the paper lacks experiments using recent self-supervised models such as DINO, DINOv2, and the CLIP family as base models. This omission makes it unclear whether CLAdapter can provide benefits when applied to modern, high-performing pre-trained models.", "was_revised": true, "final_justification": "The author has effectively addressed my concerns and further demonstrated the efficacy of CLAdapter with more comprehensive experiments.", "rebuttal": "# Response to Reviewer AwdT on more comparisons and details questions.\n\nThank you for recognizing CLAdapter and SFT's novel design, efficiency, and value for data-limited scientific domains.\n\nWe sincerely appreciate your openness to reconsider the score upon addressing the concerns. We’ll carefully revise to fully respond to your insightful feedback.\n\n\n### Question 1:\n> In typical classification tasks using Vision Transformers (ViTs), only the class token is used for classification. However, this paper discards the class token and instead adopts a more complex approach based on patch token aggregation. Is there any experimental evidence supporting this design choice? Additionally, why does the class token have a negative effect in the experimental scenarios considered in this work? In Table 1, do all experiments based on ViT models use the same type of output token for classification?\n\n\n### **Authors’ Response:**\n\nThank you for the question. \n\n- **Training from Scratch (without loading pre-trained weights)**  \n    - use class token\n        - [cls_token] -> classification\n    - not use class token\n        - patch token aggregation (features embedding, via Global Average Pooling)\n    - **Both** yield almost **similar performance**.\n        - **This has been proven** by theoretical analysis and empirical results **in [1] (refer to \"ViT paper Figure 9\")** (*ICLR 2021*).\n\n    - Therefore, in this setting, the use of the **class token is indifferent**.\n     \n- **Fine-Tuning (focus in this paper)**  \n    - use **class token often harms performance**\n        - ***especially in cross-domain transfer to data-limited scientific domin***\n        - Because the class token tends to **preserve the pre-training domain-specific** deep priors.\n        - **Theoretical and experimental** **proofs**, please **refer [2]** (*NeurIPS 2024*)\n\n\nMoreover, influential works like **MAE** [3] and the **BEiT series** [4] also **discarded the class token during downstream classification**.\n\n\n\n- Given this, our CLAdapter intentionally discards the class token and instead focuses on **mining transferable patterns from the upstream patch embeddings**, which better supports adaptation to data-limited scientific domains.\n\n- In addition, regarding Table 1, we confirm that all ViT models use the same type of output token for classification (The experiments are based on timm framework).\n\n___\n\n### Question 2:\n> In Figure 3, are the training loss and validation loss labels possibly reversed?\n\n\n### **Authors’ Response:**\nThank you for your careful observation — we appreciate your attention to detail.\n\n- We understand your concern may come from the fact that **the validation loss appears lower than the training loss in Figure 3**,\n    - which **leads you to think** ***\"loss labels possibly reversed?\"***\n\n- However, the labels in Figure 3 are **correct**. \n    - This is because our training pipeline applies **standard data augmentations** (e.g., random rotation, flip, brightness/contrast adjustment) **only to the training set**, but **not to the validation set**. \n    - This results in the training loss being higher than the validation loss.\n\n\n___\n\n### Weaknesses 1:\n> While a standard classification head typically uses a single linear layer (i.e., linear probing), CLAdapter introduces a new form of classification head that includes nonlinear modules and a larger number of parameters. This makes the experimental comparison potentially unfair. It is strongly recommended that the authors demonstrate that CLAdapter still achieves significantly better performance when compared to nonlinear classification heads with a comparable number of parameters.\n\n### **Authors’ Response:**\nThank you for raising this comment. \n- We would like to clarify that:\n    - **CLAdapter is an adapter module inserted between the backbone and the classification head**, \n    - and is ***\"not a new form of classification head\"*** itself.\n    - In addition, in our CLAdapter-related image experiments, the **classification head remains a standard single linear layer**, consistent with conventional practice. \n    - Therefore, comparing CLAdapter to more complex classification heads is not the focus.\n\n- However, **to address your concerns**, we still provide the following comparison:\n    - **Nonlinear head** (params-similar) set:\n        - Image filed is mostly done with linear heads, but some video field research introduce Spatio-Temporal Attention Heads (nonlinear) [5].\n    - Results on HMDB51 dataset with Pre-trained Swin Backbone\n        - | Method | Classification Head Type | Accuracy (%) |\n          |--------|---------------------------|---------------|\n          | Baseline | Linear Head | 71.70 |\n          | Nonlinear Head | Spatio-TemporalAttention3D Head| 72.11 |\n          | CLAdapter | Linear Head | **75.80** |\n    - This comparison highlight the superior performance of CLAdapter.\n    \n\n\n\n___\n\n### Weaknesses 2:\n> Similarly, for the SFT setting, it is recommended to replace CLAdapter with a simple nonlinear classification head that has a comparable number of parameters, and then perform a comparison using the same SFT strategy.\n\n### **Authors’ Response:**\n- Similarly, the clarifications we need to make are consistent with the response to weakness 1.\n\n- Meanwhile, we also added a corresponding SFT strategy for above Table Nonlinear Head (Spatio-TemporalAttention3D):\n    - Nonlinear Head + SFT = 72.98 (**+ 0.87%**)\n\nThis shows that the **SFT** strategy has a slight **gain for the nonlinear head**, but it is **still far behind our CLAdapter**.\n\n\n___\n\n### Weaknesses 3:\n> The experimental comparisons appear somewhat disorganized. To properly demonstrate the effectiveness of CLAdapter, a consistent set of pre-trained models (e.g., {MAE-B, MAE-L, DINO-B, DINOv2-B, ...}) should be used across different datasets, comparing their performance with and without CLAdapter. However, in the current version of the paper, even the baseline results for ConvNeXt are missing, which makes it difficult to assess the actual improvement brought by CLAdapter.\n\n### **Authors’ Response:**\nThank you for your suggestion. \n\n- We would like to clarify two key points:\n    - (1) **Experimental Comparison Motivation**:  \n       - Our goal is to demonstrate that **CLAdapter achieves SOTA performance across various data-limited scientific domains**, using only **mainstream backbones (ViT, ConvNeXt)**. \n       - Thus, we primarily compare with **recent publication SOTA methods on each dataset domin** rather than exhaustively combining all pre-trained models.\n\n    - (2) **Ablation study w/o CLAdapter** \n        - We mainly conducted ablation (***w/o  CLAdapter***) based **ViT** for **all** datasets (***Appendix Table 6***).\n        - In addition, ConvNeXt, CLIP only on **part** dataset (***Appendix Table 9***)\n\n- **Following your valued suggestions**, we now supplement the corresponding ablation results to demonstrate the **further improvement of CLAdapter**.\n\n    - | Method | FoliarDisease | HCRF | HMDB51|\n      |--------|---------------------------|---------------|----|\n      | MAE-B    | 97.35 | 92.11 | 73.30\n      | +CLAdapter | 97.98 | 92.58 | 74.81\n      | MAE-L    | 96.21 | 93.44 | -|\n      | +CLAdapter  | 97.98 | 96.58 | -|\n      | DINO-B  | 96.12 | 95.94 |59.95|\n      | +CLAdpater  | 98.24 | 97.22| 62.16|\n      | DINOv2-B  | 97.41 | 96.98 | 60.21|\n      | +CLAdapter  | 98.68 | 98.59 | 63.34|\n      | ConvNeXt-B | 96.48| 95.43 |  58.6|\n      | +CLAdpter | 98.36| 98.59 | 62.99|\n\n\n\n---\n\n### Weaknesses 4:\n> Recently, self-supervised pre-trained models have become widely adopted. However, the paper lacks experiments using recent self-supervised models such as DINO, DINOv2, and the CLIP family as base models. This omission makes it unclear whether CLAdapter can provide benefits when applied to modern, high-performing pre-trained models.\n\n### **Authors’ Response:**\n\n- We would like to **clarify** that **we have included CLIP series** in manuscript (***Appendix Table 9***).\n    - We consider not only the high performance and impact of CLIP, but also its versions with different pre-training data sizes (LAION-400M and LAION-2B).\n    - But so soory, we **only wrote the CLIP versions *“LAION-400M” and “LAION-2B”***, which ****caused your misunderstanding and neglect****\n\n- We now write its name **more clearer as follows**:\n    - The results on the BreakHis show that **CLAdapter can be applied to** modern, high-performing pre-trained models, and large-scale pre-training data.\n    - | Method | CLIP-LAION-400M | CLIP-LAION-2B |\n      |--------|---------------------------|---------------|\n      | $\\textbf{CLAdapter}_{\\text{ConvNeXt-B}}$    | 90.55 | 91.66 | 73.30\n      | $\\textbf{CLAdapter}_{\\text{ViT-B}}$ | 91.35 | 93.77 |\n\n- For the DINO series, the previous response (Weaknesses 3) also demonstrated.\n\n\n---\n\n### Reference:\n[1] Dosovitskiy, Alexey, et al. \"An image is worth 16x16 words: Transformers for image recognition at scale.\" arXiv preprint arXiv:2010.11929 (2020).\n\n[2] Zou, Yixiong, et al. \"A closer look at the CLS token for cross-domain few-shot learning.\" Advances in Neural Information Processing Systems 37 (2024): 85523-85545.\n\n[3] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[4] Bao, Hangbo, et al. \"BEIT: Bert pre-training of image transformers.\" arXiv preprint arXiv:2106.08254 (2021).\n\n[5] Pandey, et al. \"Resstanet: deep residual spatio-temporal attention network for violent action recognition.\" International Journal of Information Technology 16.5 (2024): 2891-2900."}, {"review_id": "sBF3DTJf9a", "reviewer": "Reviewer_yjJZ", "summary": "This paper introduces Cluster Attention Adapter (CLAdapter), a module designed to adapt large-scale pre-trained vision models like ViT and ConvNeXt to data-limited scientific domains. By leveraging attention mechanisms and cluster centers, CLAdapter personalizes feature transformation to enhance downstream task performance.", "questions": "1. Clarification on the Focus on Scientific Domains\n  - CLAdapter seems like a general-purpose tuning method. Could the authors clarify why they emphasize scientific downstream tasks? Is there a technical reason (e.g., domain-specific distribution shifts) that makes CLAdapter particularly effective for these tasks, or were these domains selected due to better empirical results?\n  - Suggestion: Please include experiments on more standard benchmarks (e.g., ImageNet, ImageNet-tiny is not enough) to demonstrate broader applicability beyond scientific datasets.\n2. Comparison with Standard Tuning Methods\n  - Since CLAdapter aims to compete with tuning methods like fine-tuning, linear probing, LoRA, and prompt tuning, it is unclear why the paper does not provide a controlled comparison under the same backbone architecture.\n  - Suggestion: Please add direct comparisons of CLAdapter against these standard tuning methods using the same pre-trained model on the ten downstream tasks. This would help isolate the benefits of your method and strengthen the empirical evidence.\n3. Sensitivity to the Number of Cluster Centers\n  - The ablation study indicates that CLAdapter is quite sensitive to the number of cluster centers, which could affect its robustness across tasks.\n  - Suggestion: Could you either propose an automatic way to set this hyperparameter or demonstrate that the performance does not degrade drastically over a reasonable range?\n4. Figure 2 Visualization Improvement\n  - The current version of Figure 2 is somewhat hard to interpret: trainable and frozen components are not clearly distinguishable, and the meaning of left-side annotations (e.g., N×D) is unclear.\n  - Suggestion: Please revise the figure to more clearly distinguish between frozen and trainable parameters (e.g., using different shapes, labels, or highlighting) and clarify the meaning of each notation within the figure caption.\n5. Clarification on Baselines in Table 1\n  - Table 1 compares SOTA methods with different backbones, but it is unclear what tuning strategies were used for these baselines (e.g., full fine-tuning, linear probing).\n  - Suggestion: Please explicitly state the tuning method used for each baseline in the table or in the experimental setup section.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n- The visual analysis clearly demonstrates the difference in cluster distributions before and after applying CLAdapter, helping to intuitively validate the method’s effectiveness.\n- The descriptions of the pretraining datasets and downstream tasks are detailed and comprehensive, providing a clear context for the experiments.\n\n## Weaknesses\n- CLAdapter appears to be a general-purpose tuning method rather than one specifically designed for AI4Science applications. It is unclear why the paper emphasizes scientific downstream tasks—does the method inherently favor these tasks, or were datasets selectively chosen to highlight favorable results? Moreover, the lack of experiments on standard datasets like ImageNet raises concerns about the method’s general applicability.\n- Since CLAdapter targets transfer learning scenarios similar to fine-tuning, linear probing, LoRA, and prompt tuning, it is unclear why the main experiments do not include a direct comparison of these tuning methods under the same backbone. Instead, Table 1 compares different backbones, and it is not specified what tuning strategies were used for the SOTA baselines.\n- The method is highly sensitive to the number of cluster centers in the ablation study, which raises concerns about its robustness and practical usability across different tasks and datasets.\n- In Figure 2, some text elements are misaligned, and the visualization appears overly abstract. The color blocks make it hard to distinguish trainable and frozen parameters. Furthermore, while the left part annotates dimensions like N×D, these notations are not reflected or explained in the right part of the figure, leading to confusion about their intended meaning.", "was_revised": true, "final_justification": "My recommendation is to Accept.\n\nI thank the authors for their detailed rebuttal. The responses have successfully addressed the main concerns and questions I raised in my initial review.\n\nResolved issues:  motivation of focus on scientific domains, visualization improvement\n\nUnresolved issues: none", "rebuttal": "# Response to Reviewer yjJZ.\n\nThank you very much for recognizing the strengths of our work, including the effectiveness, clear background and description.\n\n\nBelow, we address your constructive comments and suggestions, and describe the corresponding revisions we have made.\n\n\n\n### Weaknesses 1 and Question 1:\n> 1. Clarification on the Focus on Scientific Domains\n\n>  - CLAdapter seems like a general-purpose tuning method. Could the authors clarify why they emphasize scientific downstream tasks? Is there a technical reason (e.g., domain-specific distribution shifts) that makes CLAdapter particularly effective for these tasks, or were these domains selected due to better empirical results?\n> - Suggestion: Please include experiments on more standard benchmarks (e.g., ImageNet, ImageNet-tiny is not enough) to demonstrate broader applicability beyond scientific datasets.\n\n### **Authors’ Response:**\nThank you for your comment.\nWe would like to **clarify your misunderstanding** that:\n- **CLAdapter Design Motivation and Technical Reasons**\n    - CLAdapter is tailored for **data-limited scientific domin** rather than the **general-purpose tuning method**\n        - where **data-limited**, **domain distribution shifts**, **vision semantic variations**, **fine-grained feature discrimination.** \n        - This create unique challenges that differ from general-purpose tuning method.\n\n- **Scientific domins data selection principles**\n    - Each dataset selected for each scientific domin:\n        - highly cited\n        - popular\n        - representative     \n    - They are **not** ***\"chosen to highlight favorable results\"***. \n        - Otherwise, it would be impossible to cover more than 10 downstream scenarios.\n\n- **Regarding the use of ImageNet** (***This is a pre-train benchmark, not transfer fine-tuning.***)\n    - Standard/large benchmarks ImageNet are completely **outside the scope of our research.**\n    - **In contrast**, our research aims to leverage models **pre-trained on a general large-scale dataset** (e.g., ImageNet-1K/22K, LAION-400M/2B, etc) to **fine-tune cross-domin transfer** for data-limited **scientific** **domains**.\n\n\n### Weaknesses 2 and Question 2:\n> Since CLAdapter targets transfer learning scenarios similar to fine-tuning, linear probing, LoRA, and prompt tuning, it is unclear why the main experiments do not include a direct comparison of these tuning methods under the same backbone. Instead, Table 1 compares different backbones, and it is not specified what tuning strategies were used for the SOTA baselines.\n\n> 2. Comparison with Standard Tuning Methods\n> - Suggestion: Please add direct comparisons of CLAdapter against these standard tuning methods using the same pre-trained model on the ten downstream tasks. This would help isolate the benefits of your method and strengthen the empirical evidence.\n\n### **Authors’ Response:**\nThank you for your comment.\nWe would like to clarify that:\n\n- **Experimental Comparison Motivation**:  \n     - Our goal is to demonstrate that **CLAdapter achieves SOTA performance across various data-limited scientific domains**, using only **mainstream backbones (ViT, etc)**. \n     - Thus, we primarily compare with **recent publication SOTA methods  on each dataset domin** (Specifically designed for this downstream task with **standard tuning**).\n\n\n- **Following your valuable suggestion**, we compared the performance of the **same** pre-trained **model** (ViT, standard tuning) with CLAdapter **on ten downstream tasks**.\n    - | Improve%↑       | ID  | OOD | Agricultural | Geography | Materials | Biomedicine | Industrial | 3D Multimedia |\n      |-----------------|-----|-----|--------------|-----------|-----------|-------------|------------|---------------|\n      | CLAdapter ***VS*** Same ViT Baseline(pre-trained with ImageNet) | +7.8 | +3.6 | +2.0          | +2.8       | +2.5       | +12.2, +13.6   | +4.6        | +2.5, +4.1       |\n\nThis comparison further vividly highlights the strengthened benefits of our method.\n \n \n \n\n### Weaknesses 3 and Question 3:\n> 3. Sensitivity to the Number of Cluster Centers\n> - The ablation study indicates that CLAdapter is quite sensitive to the number of cluster centers, which could affect its robustness across tasks.\n> - Suggestion: Could you either propose an automatic way to set this hyperparameter or demonstrate that the performance does not degrade drastically over a reasonable range?\n\n\n### **Authors’ Response:**\nThank you for your comment.\nWe would like to clarify that:\n\n- CLAdapter is **not sensitive** when the number of cluster centers is set within a **reasonable range (20–50)**. \n    - | Num. of Clusters  | 20   | 25   | 30   | 35   | 40   | 45   | 50   |\n      |-------------------|------|------|------|------|------|------|------|\n      | Accuracy (%)      | 95.32| 94.24| 93.88| 93.98| 94.01| 93.85| 93.82|\n      | F1 Score (%)      | 93.71| 92.59| 92.14| 92.84| 92.11| 92.82| 92.77|\n\n\n- The results shown in the paper include **extreme values (e.g., 100，200， even 300)** to stress-test the model’s efficiency in very high number cluster centers settings, **which are not needed in practice.**\n    - This may have **led to your misunderstanding**, and we will explain it in the paper.\n\n\n- Possibility of *\"**Automating** this parameter setting\"*\n    - using a classic **simple grid search** within our recommend range (20-50).\n    - In addition, in future work, we also plan to explore smarter automation methods.\n\n\n### Weaknesses 4 and Question 4:\n> 4. Figure 2 Visualization Improvement\n> - The current version of Figure 2 is somewhat hard to interpret: trainable and frozen components are not clearly distinguishable, and the meaning of left-side annotations (e.g., N×D) is unclear.\n> - Suggestion: Please revise the figure to more clearly distinguish between frozen and trainable parameters (e.g., using different shapes, labels, or highlighting) and clarify the meaning of each notation within the figure caption.\n\n### **Authors’ Response:**\nThank you for your careful observation.\n\n- We have revised **Figure 2** to **explicitly distinguish** frozen and trainable modules using light blue and orange components, and **explicitly** added in the labeled below the diagram for better readability\n- The notations have also been clarified:\n  - $N$: number of patch tokens  \n  - $D$: latent space dimension (feature embedding dimension)\n\nWe appreciate your feedback on improving the clarity of our visualizations.\n\n\n### Question 5:\n> 5. Clarification on Baselines in Table 1\n> - Table 1 compares SOTA methods with different backbones, but it is unclear what tuning strategies were used for these baselines (e.g., full fine-tuning, linear probing).\n> - Suggestion: Please explicitly state the tuning method used for each baseline in the table or in the experimental setup section.\n\n### **Authors’ Response:**\nThank you for your comment.\n- These SOTA methods all follow the standard full fine-tuning specified in their papers, and we will add more information in Experiment Configuration Section.\n\nWe sincerely hope that these improvements can facilitate higher scores for our work."}]

Author rebuttal:

Meta-review: This work proposes a method to adapt the representations learnt in large-scale pre-trained vision backbones (e.g., ViT, ConvNext, etc.) to downstream scientific tasks, involving fine-grained discrimination under significant domain shift, and limited training data. A "CLAdaptor" is introduced between the pre-trained backbone and final classification layer which learns to align pre-trained representations with downstream tasks. The adaptor parameters are tuned first, followed by full fine-tuning.

The reviewers appreciated the clarity of writing and strong performance of the proposed adaptor on a number (10) of downstream scientific tasks. Further, clear ablation results wrt the adaptor were presented in the reviewing process. 
Some concerns were highlighted: (1) The method is not applied on more 'general' datasets, but limited to the 'scientific' domain; (2) there is no systematic study of performance across varying (or increasing) amounts of training data (on the same dataset / benchmark).

The proposed method, however, achieves strong performance across several benchmarks, and gives a consistent gain over other adaptation strategies (e.g., non-linear adaptation heads, lora, etc.).

In view of the above, the AC is inclined to recommend this work for acceptance. We strongly urge the authors to incorporate the valuable reviewer feedback and the experiments presented in their response in the final version.
Decision: Accept (poster)