ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts

Authors: Samar Khanna, Medhanie Irgau, David B. Lobell, Stefano Ermon
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: lora, PEFT, parameter-efficient finetuning, parameter-efficient pre-training, vision transformer, ViT, domain adaptation, domain generalization, satellite images, foundation models

Abstract:
Parameter-efficient fine-tuning (PEFT) techniques such as low-rank adaptation (LoRA) can effectively adapt large pre-trained foundation models to downstream tasks using only a small fraction (0.1%-10%) of the original trainable weights. 
An under-explored question of PEFT is in extending the pre-training phase without supervised labels; that is, can we adapt a pre-trained foundation model to a new domain via efficient self-supervised pre-training on this new domain? 
In this work, we introduce ExPLoRA, a highly effective technique to improve transfer learning of pre-trained vision transformers (ViTs) under domain shifts.
Initializing a ViT with pre-trained weights on large, natural-image datasets such as from DinoV2 or MAE, ExPLoRA continues the unsupervised pre-training objective on a new domain, unfreezing 1-2 pre-trained ViT blocks and tuning all other layers with LoRA.
We then fine-tune the resulting model only with LoRA on this new domain for supervised learning. 
Our experiments demonstrate state-of-the-art results on satellite imagery, even outperforming fully pre-training and fine-tuning ViTs. 
Using the DinoV2 training objective, we demonstrate up to 7.5% improvement in linear probing top-1 accuracy on downstream tasks while using <10% of the number of parameters that are used in prior fully-tuned state-of-the art approaches. 
Our ablation studies confirm the efficacy of our approach over other baselines, including PEFT and unfreezing more ViT blocks.

Decision: reject
Reviewer scores: 6, 6, 8, 5, 6

Reviews and discussion:
<thread>
creation date:
1730154035033

thread_type:
official_review

<thread_object>
creator: reviewer_zqza

<content>
summary:
This paper presents an approach for parameter-efficient continual pretraining and fine-tuning of visual foundation models addressing domain shift of the underlying data distribution. To accomplish this, the authors propose to use LoRA during continual pretraining and subsequent fine-tuning.

The domain shift covered in this submission put a major focus on remote sensing imagery as covered by a large part of the experimental section. Nevertheless, the submission also covers some minor experiments including domain shifts of datasets about cell tissue images, wheat images, and animal images.

For all datasets, the authors were able to demonstrate the efficiency of their approach in continual pretraining and fine-tuning up to results able to outperform fine-tuned visual foundation models.

I very much appreciate the work in this area - especially because not everybody has the resources to train visual foundation models and therefore approaches like the one presented are much appreciated. However, overall I see this submission providing only marginal contribution as it combines known methods such as continual pretraining ([mendieta2023towards] proposed it for remote sensing) and low-rank adaptation ([scheibenreif2024parameter] proposed it for remote sensing) together.

Also and since the majority of the experimental section focuses on remote sensing, in times of geo-spatial foundation models such as ScaleMAE, SatMAE, or GMF, I have trouble to see that the first step of training such models is to take a visual foundation models being trained on natural images only.

---
```
@inproceedings{mendieta2023towards,
  title={Towards geospatial foundation models via continual pretraining},
  author={Mendieta, Mat{\'\i}as and Han, Boran and Shi, Xingjian and Zhu, Yi and Chen, Chen},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={16806--16816},
  year={2023}
}
@inproceedings{scheibenreif2024parameter,
    title     = {Parameter Efficient Self-Supervised Geospatial Domain Adaptation},
    author    = {Scheibenreif, Linus and Mommert, Michael and Borth, Damian},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {27841-27851}
}
```

strengths:
**(S1)**: this work covers an important aspect of foundation model training.

**(S2)**: this work provides vast amount of experimental results showing the capabilities of the proposed approach.

**(S3)**: the presented ablation study of the paper provide valuable insights into parameter-efficient domain adaptation of remote sensing imagery.

weaknesses:
**(W1)**: this submission rather combines known approaches than presents methodological or algorithmic novelty. This can be of great value for the research community. However, I think the insights provided in this work might be limited to the ICLR research community.

**(W2)**: given previous work in this area, I miss baseline evaluations against non-LoRA continual pretraining (-> mendieta2023towards) and against pure LoRA (-> scheibenreif2024parameter). Such experiments would provide the opportunity to highlight the capabilities of the proposed approach much more and compare it against previous work.

questions:
**(Q1)**: In section 4 Problem Setup, the target domain data comes from p_{D_T}(x) where D_T is a set of domains, being a subset of all domains. What is then the difference between  p_{D_T}(x) and p_{d_T}(x, y) coming from d_T ∈ D_T? Is this just the formulation that *some* of the domains of p_{D_T}(x) might provide labels (i.e., p_{d_T}(x, y)) and some not? And if yes, is there a significant change in distributions between these? Can we assume that these distributions (p_{D_T}(x) and p_{d_T}(x, y)) are similar with respect to the distribution of x?

**(Q2)**: Is there a reason why for some experiments you do compare against ScaleMAE and in some you do not? For Table 1, the ScaleMAE in its LoRA-r8 version is missing (while SatMEA is provided). It would be interesting to see ScaleMAE performance as 0.8M fine-tuned parameter model. In Table 4 and Table 5 ScaleMAE is entirely missing. Table 6 is listing ScaleMAE as baseline. Is there a rationale behind this?

**(Q3)**: In Table 3, the ablation study, there is an experiment showing [All], which adapts not only the attention matrices but also the MLP matrices. This is great, have you also run experiments showing how MLP adaptation without attention adaptation would perform?

**(Q4)**: In Fig. 2, where can I find U in the figure, which is described in the image caption.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730294460116

thread_type:
official_review

<thread_object>
creator: reviewer_47sp

<content>
summary:
In this work, the authors introduce ExPLoRA to improve transfer learning of pre-trained vision transformers (ViTs) under domain shifts. It initializes a ViT with pre-trained weights and continues unsupervised pre-training on a new domain with some blocks unfrozen and LoRA for other layers. Then it fine-tunes with LoRA for supervised learning. Experiments show state-of-the-art results on satellite imagery. It improves linear probing top-1 accuracy and ablation studies confirm its efficacy over other baselines

strengths:
1. The paper is well organized, the figures are readable and understandable.

2. The proposed method looks logical and technically sound.

3. The experimental results are strong. Consistent improvements have been shown over different baselines.

weaknesses:
1. Concerns about the novelty: The proposed ExPLoRA seems does not have essential differences with the conventional LoRA. What are the differences from LoRA? This paper is more like a technical report than a research paper.

2. Many important experimental comparison results are missing. 1) The authors should compare their proposed method with the recent SOTA ViT-based domain adaptation [1-6] in the same setting. 2) Many PEFT methods besides LoRA, [7-8] should be compared. The reviewer wonders whether ExPLoRA outperforms these types of methods? 3) The results on the widely-used UDA benchmarks like Office-Home, Office-31, VisDA are missing.

[1]. TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation, WACV 2023;

[2]. Safe Self-Refinement for Transformer-based Domain Adaptation, CVPR 2022;

[3]. CDTRANS: CROSS-DOMAIN TRANSFORMER FOR UNSUPERVISED DOMAIN ADAPTATION, ICLR 2022;

[4]. Patch-Mix Transformer for Unsupervised Domain Adaptation: A Game Perspective, CVPR 2023;

[5]. Towards Unsupervised Domain Adaptation via Domain-Transformer, IJCV 2024;

[6]. Making The Best of Both Worlds: A Domain-Oriented Transformer for Unsupervised Domain Adaptation, ACM MM 2022;

[7].Low-Rank Few-Shot Adaptation of Vision-Language Models, CVPR 2024;

[8]. Quantized Prompt for Efficient Generalization of Vision-Language Models, ECCV 2024;

3. Many important references of domain adaptation and PEFT [1-8] are missing. These works should be briefly reviewed in the related work section.

--------------------------------------------------------------After Rebuttal--------------------------------------------------------------

Sorry for the late reply.

After carefully reading the rebuttal and other reviews, I'd like to thank the authors' efforts in response to my concerns. Though some of my concerns have been addressed, I still have concerns in terms of novelty and experimental comparisons:

1) About the novelty. The authors acknowledge that ExPLoRA is a combination of existing strategies, e.g., LoRA and unfreezing blocks, and the authors' explanations in the rebuttal do not convince me. Thus the reviewer thinks it have limited technical contributions. I agree with Reviewer ZQZA's opinion that the technical novelty is limited to the ICLR research community, and I also agree with Reviewer 8d6C's opinion that the whole paper seems like a technical report rather than a research paper.

2) About the experimental comparisons. Thanks the authors for providing the results on VisDA-2017 benchmark. Since this paper targets domain adaptation, the authors should conduct experimental comparisons on many widely-used benchmarks of domain adaptation such as Office31, and DomainNet. Only the results on VisDA-2017 benchmark are insufficient to reveal the effectiveness.

To sum up, I will keep my rating unchanged and suggest the authors revise the paper according to my advice carefully.

questions:
1. What are the differences between ExPLoRA and LoRA?
2. Does the presented method outperform the state-of-the-art DA and PEFT methods in the same setting?
3. What are the results on the widely-used UDA benchmarks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730463816543

thread_type:
official_review

<thread_object>
creator: reviewer_8d6c

<content>
summary:
This paper introduces ExPLoRA, a novel parameter-efficient method for adapting pre-trained vision transformers (ViTs) to new domains through extended unsupervised pre-training. ExPLoRA initializes ViTs with weights from natural-image datasets and continues pre-training on new domains with LoRA. The model is then fine-tuned on the new domain for supervised learning, achieving impressive results on satellite imagery and generalizing to various domains like wildlife, medical, and agricultural imagery. ExPLoRA outperforms fully pre-trained and fine-tuned techniques while using significantly fewer parameters.

strengths:
1. The writing of the article is quite good and the article is logical.
2. Supplementary materials are substantial.
3. The use of PEFT for post-pretraining in the visual domain is innovative.
4. The experiments in the domain migration section are ample and sensible.

weaknesses:
1. Figure 1 seems to express the difference between full fine-tuning and PEFT in a more accurate form.
2. The idea of using PEFT for post-pretraining is a good one. loRA was proposed a few years ago, and there are more efficient PEFT methods in the visual field. Maybe the authors can try to compare with LoRA with newer methods [1-3].
[1] 1% vs 100%: Parameter-efficient low rank adapter for dense predictions.
[2] Pro-tuning: Unified prompt tuning for vision tasks.
[3] Adapter is all you need for tuning visual tasks.

3. The method section is more like a solution derived through experience. It is recommended to analyse whether LoRA has limitations in coping with visual post-pretraining tasks, and optimise based on the analysis to propose your own PEFT method.
4. The whole article seems to be like a technical report, and it is recommended to add some in-depth theoretical analyses and technical innovations for visual features.

----------------------------------------After Rebuttal------------------------------------------------


The rebuttal solves part of my confusion and I improve my score. Still, I think the author could have cited related work that I mentioned or didn't mention. Necessary citations both give the reader a broader view of developments in the field, and are a way of recognising and respecting those who work in the field.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679779259

thread_type:
official_review

<thread_object>
creator: reviewer_7p1n

<content>
summary:
The paper presents ExPLoRA, a method for efficiently adapting pre-trained vision transformers (ViTs) to new domains using parameter-efficient fine-tuning (PEFT) techniques, utilizing LoRA (Low-Rank Adaptation). 
By continuing unsupervised pre-training on the target domain and only unfreezing select model layers, ExPLoRA enables adaptation with minimal computational overhead. 
This approach leverages pre-trained models on natural image datasets like DinoV2, achieving notable performance gains, particularly in challenging domains like satellite imagery. 
For instance, ExPLoRA outperforms fully pre-trained models on satellite classification tasks while using fewer parameters, highlighting its efficiency. 
Beyond satellite data, ExPLoRA generalizes well to other domains, including medical and wildlife imagery, as tested on the WILDS benchmark.

strengths:
1: ExPLoRA excels at adapting large vision transformers to new domains without requiring full re-training, instead leveraging low-rank adaptation. This parameter-efficient approach significantly reduces computational costs, making it suitable for resource-constrained environments.

2: The proposed method is simple yet effective. The ExPLoRA can directly use the well-trained vision foundation models. This compatibility also simplifies its integration with established models.

3: Extensive experiments on RGB satellite images show the proposed methods' effectiveness compared with previous state-of-the-art works.

weaknesses:
1: Analysis of the effect of total training cost. From my understanding, this work's main contribution (claim) is to adapt continuing unsupervised pre-training before supervised training on downstream domains (I don't regard using LoRA fine-tuning and unfreezing the last ViT blocks as this work's contribution, and feel free to point my misunderstanding if it is). It is a two-stage. Therefore, it's important to consider the computational cost of these two stages simultaneously. I have seen Figure 7 which analyzes the effect of pre-training iterations. How about simply putting the equivalent computational cost on the supervised fine-tuning stage? Will unsupervised pre-training speed up the convergence of supervised fine-tuning? If the computational budget is fixed, how should we allocate it across the two different stages?

2: The effect of the "extend unsupervised pre-training stage". Current ablation studies mainly focus on the settings of learnable parameters (unfrozen blocks and LoRA). I hold the perspective that the main claim of this paper is the importance of "extend pertaining" for downstream domain transfer. Therefore, the authors are encouraged to verify the necessity of this stage further. For example, supervised fine-tuning can be performed directly with the current optimal setting of LoRA and unfrozen blocks for transferring.

3: Extending the model to multi-modal models like CLIP and zero-shot settings like using natural language for zero-shot understanding in downstream tasks. The authors could follow the setting of the CLIP paper (the downstream validation dataset).


4: (An open question). How about changing unsupervised training objectives in the extended pertaining stage? Lare-scale pre-trained DinoV2 and MAE both have their advantages. The unsupervised training stage is not limited to the corresponding original training objective. Can the model combine different advantages using different training objectives in this stage?

Overall, this paper presents a simple yet effective method for model adaptation. My main concern exists in the importance of the claimed "extend unsupervised pre-training" stage.

-------------------------------------------------  After Rebuttal ------------------------------------------------

Most of my concerns have been addressed. My main concern exists in the importance of the claim that the extended pre-training stage plays a necessary role in overcoming domain shifts.

Although I still doubt whether the extended pre-training stage is really as important as claimed, this does not prevent this paper from being a very good paper with detailed and comprehensive experiments analysis. Therefore, I decide to increase my score to 6 at this time.

questions:
Potential limitations of the proposed extended pre-training. Can this stage only benefit the domain shits or also work in general scenarios (like general classification and general object detection)? I also did some experiments before and I failed. I would appreciate it if the authors' pipeline works in such scenarios, but I also understand this part is out of the scope of this paper's claim.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730825394670

thread_type:
official_review

<thread_object>
creator: reviewer_anfl

<content>
summary:
This work presents ExPLoRA, which initializes a ViT with pre-trained weights, selectively unfreezes 1 - 2 blocks, tunes remaining weights with LoRA, and continues unsupervised pre-training on a new domain. Then fine-tunes the model on the new domain for supervised learning. This work demonstrates state-of-the-art results on satellite imagery and generalizes to different domains like wildlife, medical, and agricultural imagery.

strengths:
-	The introduction of ExPLoRA, a new parameter-efficient method to extend unsupervised pretraining on target domains. 

-	Conducting comprehensive experiments on various datasets, and showcasing improvements in linear probing top-1 accuracy and outperforming existing techniques on datasets like fMoW. 

-	The authors show the effectiveness of ExPLoRA and analyze the differences in local and global information encoded in the patch representations output by each ViT block.

weaknesses:
-	The authors argue that they selectively unfreeze 1 - 2 blocks, tunes remaining weights with LoRA. But specifically, it is very difficult to evaluate the number of layers to be tuned and which layers to be tuned. Although the authors have shown that the method of tuning 1 - 2 layers is feasible, it is not known whether the number of layers and the specifically selected layers are sensitive for different domains. In other words, the authors‘ method might not be optimal for different domains or datasets.
-	The authors may have missed two baselines. For example, in Table 1, the author only gives a result of using their own method for finetuning. However, one baseline is to use the LoRA method in the pre-training phase, and the author should compare the effect of their method with this baseline. The second baseline is to use the pre-trained MAE for full finetuning directly on the downstream task. Intuitively, it is still unknown whether further pre-training on the downstream task is necessary. This baseline will complete the integrity of the experiment. 
-	In addition to LoRA, the authors lack a comparison of some related PEFT methods [1,2,3,4,5]. It is better to be able to discuss the related methods. Furthermore, as mentioned in the paper, it is also better for the author to be able to give some results on detection and segmentation tasks.

[1] LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning, in NeurIPS2022.

[2] Scaling & shifting Your Features: A New Baseline for Efficient Model Tuning, in NeurIPS2022.

[3] Vision Transformer Adapter for Dense Predictions, in ICLR2023.

[4] Adapters Strike Back, in CVPR2024. 

[5] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning, in NeurIPS2024.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732151012749

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Rebuttal

comment:
We thank the reviewers for their constructive feedback. We're pleased our work is recognized for introducing an innovative perspective on parameter-efficient unsupervised pre-training (ANFL, 7P1n, 8d6C), demonstrating strong and comprehensive results across multiple domains while using fewer parameters (ANFL, 7P1n, 8d6C, 47Sp, ZQZA), and conducting insightful ablations (8d6C, ZQZA). We appreciate ANFL's and ZQZA's acknowledgments of our analysis of encoded information and contributions toward efficient foundation model training.

Our main contribution is demonstrating parameter-efficient unsupervised pre-training for domain adaptation, challenging the paradigm of from-scratch pre-training. Key strengths include:
- Outperforming domain-specific full pre-training with <5-10% of ViT parameters (Tables 1, 2, 4, 5, 6, 11, 12)
- Successfully adapting to diverse domains including satellite, medical, and wildlife images (section 6)
- Providing systematic ablations (section 6.1.2) and interpretability analyses (section 6.3)

Our [revised pdf](https://openreview.net/pdf?id=6BoStmXGBf) has important changes marked in red. We summarize and address reviewer concerns in three key areas:  

## Technical Contributions of ExPLoRA 
- Novel parameter-efficient unsupervised pre-training technique that creates specialized foundation models for new domains, supporting both MAE, DinoV2 objectives. Unlike traditional PEFT, these models enable linear probing, feature extraction, and generalization to downstream tasks beyond supervised fine-tuning 
- Demonstration that combining selective block unfreezing with LoRA significantly improves efficiency and performance over either approach alone 
- Extensive validation of SoTA performance on challenging benchmarks (fMoW-{RGB, temporal, Sentinel}, EuroSAT, Resisc-45, SpaceNet), including successful adaptation to multi-spectral and temporal satellite imagery despite significant domain shifts from RGB. Importantly, ExPLoRA outperforms prior SoTA methods that were fully pre-trained from scratch (SatMAE, ScaleMAE etc.)
- Detailed analysis of intermediate representations through spectral analysis and linear probing of patch embeddings across ViT layers, providing clear guidelines for block selection during pre-training

## Comparisons with Recent Methods 
We've expanded comparisons with: 
- Recent continual pre-training methods (GFM [1], GDA [2]) for remote sensing, outperforming them by >6% in Table 1 and 3% in Table 4. We describe key differences between ExPLoRA and these works in section appendix A.1 and in [our reply to reviewer ZQZA](https://openreview.net/forum?id=6BoStmXGBf&noteId=WaioYIqxOi).
- State-of-the-art PEFT techniques including BOFT [3], Gated VPT [4], AdaLoRA [5], SAVP[6], and newly added Adapters Strike Back [7] and Mona [8,9]. These PEFT techniques do not surpass ExPLoRA's extended pre-training.
- Please also see [our reply to reviewer 47Sp](https://openreview.net/forum?id=6BoStmXGBf&noteId=BYDbSHoOGn) for ExPLoRA's compatibility as an initialization for UDA methods.

We have expanded our references to include other cited works suggested by reviewers. We note that many are either already outperformed or superceded by prior works we have included experiments from. e.g.,

* SA²VP [6] outperforms Pro-Tuning [10] on CIFAR-100, Oxford flowers.
* BOFT [3] and AdaLoRA [5] outperform SSF [11] on VTAB-1K.
* “Adapters Strike Back” [7] includes comparisons with the most recent adapter-based methods, outperforming [11, 12, 13] on VTAB.
* Mona [9] outperforms LoRand [8] and other adapter methods [12] on COCO and other benchmarks

**Importantly**, ExPLoRA remains compatible with any PEFT method during fine-tuning as it preserves the ViT architecture. Instead, our method outperforms _pre-training from scratch_ on new domains, which as we mention in Appendix D, is far more expensive and more environmentally unfriendly.

## Value of Extended Pre-training
Our new analysis in Appendix B2 demonstrates that extended pre-training is crucial:
- For fixed parameter budgets, extended fine-tuning converges to lower accuracy (~1%) than ExPLoRA, despite training for longer
- With fixed compute budgets (measured in GPU-hours), increasing fine-tuning parameters doesn't reach ExPLoRA's accuracy ceiling (also lower by 0.8-1%)
- Pre-training time creates natural performance tradeoffs: more GPU hours in ExPLoRA pre-training improves both convergence and final accuracy
- ExPLoRA, due to unsupervised pre-training, provides unique benefits beyond fine-tuning methods:
  * Works with unlabeled domain data (e.g., unlabeled satellite or medical images)
  * Creates strong feature extractors (7%+ improvement in linear probing SoTA, Table 2)
  * Serves as foundation model initialization for downstream tasks (demonstrated in Tables 6, 11)

We thank the reviewers for their time and hope that they take our response into consideration.

In the following comments, we address reviewer-specific concerns in further detail.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732507014761

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Request for Rebuttal Consideration

comment:
Dear reviewers,

Thank you very much for your helpful reviews and suggestions which have improved our work considerably. As the rebuttal period comes to an end, please let us know if there are any remaining questions or concerns that we may address.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733192288297

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final check-in before discussion phase ends

comment:
Dear Reviewers,

Thank you again for your time and consideration. As we are just a few hours away from the end of the discussion phase, we wanted to extend one final opportunity to provide feedback on our rebuttal. While we are grateful to those who have already responded and increased their evaluations, we welcome any remaining thoughts or questions that we could address during tomorrow's author-only response period.

If you haven't had a chance to review our responses yet, we would greatly appreciate your feedback. For those whose concerns have been addressed by our rebuttal, we kindly request that you consider updating your evaluation accordingly.

Kind regards,  
The Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733293635369

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Experiments and Paper Updates during Discussion Period

comment:
We thank all reviewers for their constructive feedback, which has significantly improved our paper. Below we summarize key changes and experimental additions made during the discussion phase that address reviewer concerns:

**Additional Experimental Results**  
1. Comprehensive comparisons with recent methods in Table 1:
   * Outperform recent continual pre-training (GFM) [1] and parameter-efficient adaptation (GDA) [2] by 6% and 7% respectively
   * Surpass modern PEFT methods: Adapter+ [3] by 1.7% and Mona [4,5] by 6.5%
   * Similar improvements on multi-spectral data ([link](https://openreview.net/forum?id=6BoStmXGBf&noteId=YJRZhJ8vh6)), outperforming GFM and GDA by 2-3% on fMoW-Sentinel

2. New VisDA2017 experiments ([link](https://openreview.net/forum?id=6BoStmXGBf&noteId=BYDbSHoOGn)), to be included in final revision) demonstrating ExPLoRA's effectiveness on synthetic domain data:
   * ExPLoRA initialization elevates TVT [6] to SOTA performance (88.5% mean accuracy)
   * Improves TVT's original results by 2-4% and DinoV2 initialization by 6%
   * Makes TVT competitive with SOTA methods like SSRT [7] and CD-Trans [8]

**Analysis of Extended Pre-training**  
In new section B.2, we analyze two key questions about extended pre-training: (1) given a fixed parameter budget, does equivalent supervised fine-tuning match ExPLoRA + fine-tuning? and (2) given a fixed compute budget (measured in GPU-hours), can supervised fine-tuning alone achieve similar performance? Our experiments show that extended pre-training is crucial - ExPLoRA followed by LoRA-r8 fine-tuning outperforms direct fine-tuning with unfrozen block + LoRA by ≥0.9% in top-1 accuracy while using fewer parameters. Increasing the parameter budget by unfreezing more blocks during fine-tuning doesn't close this gap. Moreover, we observe that increasing pre-training iterations improves initial fine-tuning accuracy, though beyond 100k-150k iterations the gains in final accuracy plateau, demonstrating ExPLoRA's computational efficiency.

Beyond supervised performance, extended pre-training provides unique benefits: it enables learning from large unlabeled domain datasets, creates strong feature extractors (demonstrated by 7%+ improvement in linear probing, Table 2), and produces weights that serve as effective initializations for other downstream tasks within the domain (as shown by SOTA results on Resisc-45 and EuroSAT using the same pre-trained weights).

**Clarified Related Work**  
New appendix A.1 contextualizes ExPLoRA against GFM and GDA. Unlike GFM, ExPLoRA is parameter-efficient, using <10% of parameters. Unlike GDA, it preserves the ViT architecture allowing flexible PEFT methods. ExPLoRA also supports non-MAE objectives (e.g., DinoV2) and provides principled analysis for block selection.

In appendix A.2, we clarify how ExPLoRA differs from UDA: while UDA requires labeled source domain data, ExPLoRA enables unsupervised adaptation using only pre-trained weights, without label set restrictions between domains. This positions ExPLoRA as complementary to UDA methods rather than competitive.

We have also updated the related work section 2 with recent continual pre-training methods, added comparisons with modern PEFT techniques, and included relevant UDA literature.

---

We are very grateful for all of your feedback and your time in reviewing our work. The additions we have made to address your concerns demonstrate ExPLoRA's effectiveness across diverse domains while clarifying its positioning relative to existing methods. The new experiments particularly highlight ExPLoRA's strong performance on challenging domain shifts and its complementarity with existing PEFT/adaptation techniques.

---
References:  
[1] Towards geospatial foundation models via continual pretraining. _ICCV 2023_.  
[2] Parameter Efficient Self-Supervised Geospatial Domain Adaptation. _CVPR 2024_.  
[3] Adapters Strike Back. _CVPR 2024_.  
[4] 5%>100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Tasks. _arXiv:2408.08345 (2024)_.  
[5] 1% vs 100%: Parameter-efficient low rank adapter for dense predictions. _CVPR 2023_.  
[6] TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation. _WACV 2023_.  
[7] Safe Self-Refinement for Transformer-based Domain Adaptation. _CVPR 2022_.  
[8] CDTRANS: Cross-Domain Transformer For Unsupervised Domain Adaptation. _ICLR 2022_.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work introduces ExPLoRA, a method that initializes a Vision Transformer (ViT) with pre-trained weights, selectively unfreezes one to two blocks, fine-tunes the remaining weights using LoRA, and continues unsupervised pre-training on a new domain. Subsequently, the model undergoes supervised fine-tuning for the target domain.

Five experienced reviewers provided a mixed assessment of this submission. Before rebuttal, four reviewers gave the negative reviews. After rebuttal, three reviewers raised their score and indicated that their issues have been solved. Most issues are limited novelty, unconvincing experiment results. However, one reviewer still felt the limited novelty of this work and kept the negative scores with the high confidence.

Thus, the Area Chair (AC) has carefully reviewed the process, including the initial reviews, rebuttal, and discussions between reviewers and authors, as well as the revised submission. The AC agrees with concerns raised by Reviewer ZQZA and Reviewer 47Sp regarding the limited novelty and narrow application scope. Despite the proposed method is simple yet effective, the insights are not enough in current draft, making this submission like a technical report.

For the future submission, the authors are encouraged to either narrow the focus of ExPLoRA or explore a broader range of experimental settings. Additionally, the authors need to provide a clearer differentiation between LoRA and ExPLoRA beyond merely incorporating LoRA.