ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts
Authors: Samar Khanna
Medhanie Irgau
David B. Lobell
Stefano Ermon
Venue: ICML 2025 poster (2025)
Keywords: lora
PEFT
parameter-efficient finetuning
parameter-efficient pre-training
vision transformer
ViT
domain adaptation
domain generalization
satellite images
foundation models
Primary area: deep_learning->selfsupervised_learning
TL;DR: Extended parameter efficient pre-training enables ViTs such as DinoV2 and MAE to effectively and efficiently transfer to different visual domains (i.e. satellite, medical imagery)

Abstract: Parameter-efficient fine-tuning (PEFT) techniques such as low-rank adaptation (LoRA) can effectively adapt large pre-trained foundation models to downstream tasks using only a small fraction (0.1%-10%) of the original trainable weights. An under-explored question of PEFT is in extending the pre-training phase without supervised labels; that is, can we adapt a pre-trained foundation model to a new domain via efficient self-supervised pre-training on this domain? In this work, we introduce ExPLoRA, a highly effective technique to improve transfer learning of pre-trained vision transformers (ViTs) under domain shifts. Initializing a ViT with pre-trained weights on large, natural-image datasets such as from DinoV2 or MAE, ExPLoRA continues the unsupervised pre-training objective on a new domain, unfreezing 1-2 pre-trained ViT blocks and tuning all other layers with LoRA. We then fine-tune the resulting model only with LoRA on this new domain for supervised learning. Our experiments demonstrate state-of-the-art results on satellite imagery, even outperforming fully pre-training and fine-tuning ViTs. Using the DinoV2 training objective, we demonstrate up to 8% improvement in linear probing top-1 accuracy on downstream tasks while using <10% of the number of parameters that are used in prior fully-tuned state-of-the art approaches. Our ablation studies confirm the efficacy of our approach over other baselines such as PEFT. Code is available at: https://samar-khanna.github.io/ExPLoRA/

ExPLoRA: Parameter-Efficien Extended Pre-Training to Adapt Vision Transformers under Domain Shifts
Samar Khanna Medhanie Irgau'
David B. Lobell
Stefano Ermon
Abstract
Parameter-efficient fine-tuning (PEFT) techniques such as low-rank adaptation (LoRA) can effec- tively adapt large pre-trained foundation models to downstream tasks using only small fraction (0. 10%) of the original trainable weights. An under-explored question of PEFT is in extend- ing the pre- pre-training tre-training phase without supervised la- bels; that is, can we adapt a pre- are-trained founda- tion model to new domain via efficient self- supervised pre-training on this domain? In this work, introduce ExPLoRA, highly effec- tive technique to improve transfer learning of pre- trained vision transformers ( (ViTs) under domain shifts. Initializing a ViT with pre-trained weights on large, natural-image datasets such as from Di- noV2 or MAE, ExPLoRA continues the unsu- pervised pre-training objective on a new domain, unfreezing 1-2 pre-trained VIT blocks and tuning all other layers with LoRA. We then fine-tune the resulting model only with LoRA on this new do- main for supervised learning. Our experiments demonstrate statc-of-the-arr results on satellite im- agery, even outperforming fully pre-training and fine-tuning ViTs. Using the DinoV2 training ob- jective, demonstrate up 8% improvement in linear probing top-1 accuracy on downstream tasks while using <10% of the number of param- eters that are used in prior fully-tuned state-of- the art approaches. Our ablation studies confirm the efficacy of our approach over other baselines such as PEFT. Code available at: https: samar khanna github. Bee/xxXLoRR
W/
W
full pen-trueeng tum aouzhh
(1)
4
nitltationn
-4
W
EXPLORA
Figure I: Consider two different image domains, S and T. Above: the traditional paradigm of pre-training from scratch on each do- main S, T' to yield Ws. Wr, and then fine-tuning on target datasets to yield ( A Below: our approach, which is to initial- ize with pre-trained weights from S and then learn unsupervised weights Ar for T in parameter-efffiient manner.
2023
Touvron et al.,
2023 and natural images (He et al.
2022 Rombach et al. 2022 Oquab et al. (024) has histori- cally been computationally intensive, often limited to organi- zations with substantial resources. However, recent advance- ments parameter-ee efficiient fine fine-tuning (PEFT) techniques including low-rank adaptation (LoRA) and others (Hu et al.. 2022 Jia al. 2022 Qiu et al. 2023) have sparked signifi- cant interest. These methods aim to adapt foundation mod- els to downstream fraction (0. %0%) of the model's s trainable weights, with many based on the hypothesis that the required weight up- dates the pre-trained model have a "low intrinsic rank" (Li
snservised-learningg tasks using a small
al. 2018 Aghajanyan al. 2020
In this paper, we focus visual foundation models (VFMs) such as MAE or DinoV2 (He et al.) 2022 Oquab et al., 2024) that were trained on large amount of natural images. Despite the big investments in developing VFMs for natural images, they underperform when applied to other domains with visual data (e.g. medical or satellite images). For ex- ample, fine-tuning model pre-trained on natural images on satellite image classification tasks is not as effective as fine- tuning one that was pre-trained on satellite images (Ayush) ct al. 202T Cong et al. 2022). To bridge this gap. prevail- ing approaches invest similarly large levels of compute to pre-train VFMs on new domains, inspired by techniques developed for natural images (Cong et al. 2022 Zhou et al.
1. Introduction
Pre-training foundation models (Bommasani al., 2021 for natural language (Brown et al. 2020 Chowdhery al.
'Stanford University CCZ Biohub. Correspondence to: Samar Khanna <samarkhannal lertonaaa@cssstttiie Saaaauuthaamesssstttooodddddd.d sdun..r..edu.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ExPLoRA
2023
Moutakanni et al. 2024 Khanna et al. 2024).
stream tasks such classification semantic segmenta-
tion (Grill al. 2020 Chen et al., 2020 He et al. 2022
In this work, we challenge this paradigm (fig. asking whether pre-training from scratch on each new domain is strictly necessary, since doing so is expensive (in compute and time) and precludes knowledge transfer from natu- ral images. Instead, we wish to more efficiently leverage the rich semantic information encoded in natural-image VFMs to adapt them to new domains. Our proposed so- lution addresses these concerns using -effcciette self-supervised learning methods for transfer learning.
Oquab et al. 2024). However, there has also been a rise in
nomain-specint VFMs (Cong et al. 2022 Man et al. 2023
Zhang et al. 2023b Ma et al. 2024] Moutakanni et al., 2024). For instance, SatMAE handles temporal or multi-
spectral satellite image inputs. Since these models contain hundreds of millions of parameters, efficient adaptation to downstream tasks has become key research focus.
PEFT PEFT methods have gained widespread adoption for efficiently adapting large models by updating only a fraction of parameters, mitigating the prohibitive costs of full model tuning. LoRA learns low-rank weight updates to frozen weights, while other methods modify the frequency
We introduce ExPLoRA, which generalizes vision founda- tion models to new domains by extending the pre-training phase with parameter-effiiient techniques. We initialize vision transformer (ViT) (Dosovitskiy et al. (02T) with weights pre-trained from natural-image datasets such as MAE or V2. Selectively unfreezing 1-2 transformer blocks, we tune remaining weights with LoRA and continue unsupervised pre-training on the new domain. Subsequently fine-tuning with linear probing or LoRA on this new domain for supervised learning outperforms prior state-of-the-art (SoTA) approaches while training under 5-10% of the orig- inal weights. On satellite imagery, we demonstrate an 8% improvement in linear probing top-1 accuracy, and even improvement over prior SoTA fully pre-trained and fine- tuned techniques that required up to 16x more trainable pa- rameters and 8x more compute (GPU hours). We conduct an extensive study on RGB, temporal, and multi-spectral satel- lite images, either matching or outperforming prior methods that fully pre-train from scratch. ExPLoRA also outper- forms previous work different domains such wildlife, medical, and agricultural imagery on the WILDS (Koh et al. 2021 benchmark. Our contributions include:
or number of trainable parameters per layer (Hu et al. [022]
Zhang et al. 2023c Chavan et al. 2023 Pu et al. 2023
Others use multiplicative orthogonal updates (Qiu et al.
2023
Liu et al. (2024) or inject adapter modules (Chen et al.
2022
Lian et 2022] Yin et al. [2033 2024 Steitz & Roth
2024
effeetively retaining pre-training knowledge in frozen
weights. Visual prompt tuning (VPT) methods concatenate learnable prompt tokens to image patch sequences, trading improved fine-tuning performance with increased inference
costs (Jia et al. 2022 Yoo et al. 2023 Han et al. 2023 Nie et al. 2023 Tsai et al. 2023
Pei et al. 2024). ExPLoRA aims to supplement rather than replace these methods, and thus can be configured with any existing or future PEFT method for ViT fine-tuning.
Domain Adaptation Domain adaptation enables models trained on a source domain to perform well on a different but related target domain. Traditional ransformer-based methods address this via domain alignment, discriminative feature learning, cross-attention with pseudo- -lbels (Sun)
1. Introducing ExPLoRA, a novel parfmeter-efffcient method that extends unsupervised pre-training on tar- get domains, achieving state-of-thh--rt supervised- learning performance using a fraction of the original ViT weights (section 55).
et al. 2022 Chuan-Xian et al. 2022 Zhu et al. 2023 or adversarial learning with self-refinement (Xu et al. 2021 1
Yang et al. 2023), typically requiring labeled target data or source domain labels. Test-time adaptation methods (Wang et al. 202T: Gao al. 2022 Zhang et al.) (2023a) adapt mod- els without target domain labels, but assume shared label space between domains and thus require models trained with supervised learning on the source domain. While Reed et al. (2022) showed that sequential self-supervised pre-training on source and target datasets improves convergence for su- pervised tasks, their study was limited to ResNet-50 with MoCo and required full model training. Recent work adapts ViTs through different means: e.g., continual pre-training via masked image modeling (Mendieta et al. 2023) and scaled LoRA adapters (Scheibenreif et al. 2024 for satel- lite imagery. ExPLoRA builds on this direction, enabling parameter-efficcent SSL directly on the target domain.
2. Conducting a comprehensive case study on satellite imagery, outperforming existing techniques on datasets like IMoW and showcasing improvements in linear probing accuracy. We also demonstrate generalization to multiple other domains within WILDS (section (
3. Demonstrating ExPLoRA's A's efficacy via ablation stud- ies and by analyzing improvements in local (positional) and global (class) information encoded the patch rep- resentations output by each ViT block (section 7)
2. Related Work
VFMs VFMs such as DinoV2 or masked autoencoders (MAE) that pre-train with self- supf-vupe learning (SSL)
have demonstrated remarkable performance across down-
Further comparisons with related work are in appendix
2

---

ExPLoRA
3. Background
Ws. pre-trained weights obtained via unsupervised
pre-training on images from px(
MAE The is a SSL technique that uses an asymmetric encoder-decoder architecture on images x E RC where patches are masked before being processed by the ViT encoder fe. with parameters 0 and layers L. The masked patches are then reconstructed by smaller decoder 9v with weights v and layers Lp. 15, O are jointly trained using mean-squared error on the reconstructed visible pixels (ge o (x). While effective across domains (Cong et al.) [2022] Bachmann
(MAE) (He et al. 2022
(ii Xr = =x,} px(x), unlabeled dataset of N
images from new domain T
(iii) y { ~ PT pl (x,,), labeled datasets
of ((r) images for each task T in domain T
Our objective is to learn optimal weights w/r)) for each supervised-leaming dataset )x67) in a parameter-efficient manner while leveraging the knowledge stored in Ws.
et al., 2022), MAEs typically require full fine-tuning for downstream tasks, making them computationally expensive.
Traditionally, the approach (fig. 1 has been to begin pre- training from scratch on data Xr from the new domain of interest, and then fine-tune for each dataset 2 given as:
V2 D2noV2 (Oquab et al. 2024) is robust SSL method for ViTs. Unlike MAE, D2 features fx(x) have demonstrated strong zero-shot performance, enabling adaptation to downstream tasks even with a frozen ViT back- bone. During pre-training. D2 maintains two copies of ViT encoder: the student fe (trainable) and the teacher fer, where 9' is updated using exponential-movee moving average of the student's parameters 0. The training objective incor- porates global, image-level loss from Dino (Caron et al. 2021). a patch-based loss from iBOT (Zhou et al. 2021)) ) and regularizers including KoLeo (Delattre & Fournier 2017 and Sinkhorn- Knopp centering (Caron et al. 2020).
w (r) A
(2)
where Wr represents the weights learned from unsuper- vised pre-training on Xr, and ( are the weights learned from supervised fine-tuning Dift) Hooweer, this method is computationally expensive: fully pre-training Wr from scratch for every new target domain requires prohibitively large amounts of additional compute.
LoRA, however, addresses this inefficiency as follows: w ~ W's + A (7) = Ws + B(T)
(3)
LoRA Low-rank adaptation (LoRA) (Hu et al. 2022) as-
The LoRA hypothesis is that the update A (r) resides in a low-rank subspace when adapting pre-trained weights W's to fine-tuned weights w This hypothesis holds well when pre-training and fine-tuning distributions are similar. However, when there is significant domain shift, such as between natural images and multi-spectren satellite data, the low-rank assumption often breaks down (see section 6.1.3)
sumes that the weight update to change set of unsupervised pre-trained weights to supervised fine-tuned weights lives in low-rank subspace,
W R Wo + Aw = Wo+ + BA
(L)
where W E R are the final, task-specific fine-tuned weights, Wo RR are the pre-trained weights, Aw E R is the weight update required to translate the pre- trained weights Wo to the fine-tuned weights W. The key is that Aw = BA, where B E R&xx and A E Rxx form a low-rank factorization of Aw. with r & k2).
Our goal is to learn Wi(T) efficiently while using the knowl- edge encoded in Ws to bridge the large domain shift to T. We propose the following partition of W(T))
w Ws + A + (1)
(4)
where AT kg is an additional structured update matrix learned from unsupervised pre-training X7. Cru- cially, Ar requires only a fraction of the k parameters of Ws. making it significantly more efficient than full-rank pre-training. The resulting model, W7 = Ws + Ay Ro Wr, retains the benefits of unsupervised pre-trained VFMs, in- cluding strong feature extraction, effective linear probing, KNN classification, and generalization to downstream tasks.
4. Problem Setup
Consider a set of image domains D = {1, 2, ... }. where each domain de E D is associated with a data distribution Pa(x), and images x E CCxxHaxXi Wa have domain-specifee channel, height, and width. Let S E 'D represent a source domain (c.g., internet-scale natural image data) and T' D represent a target domain (e.g., satellite imagery). The data from the source domain follow a distribution ps (x), and the target domain data come from (x)).. For the target domain, the joint distributions pf (x,y) describe images x with associated supervised labels y used for each downstream task 7. We then assume access to the following:
5. Method
To learn AT, we propose ExPLoRA (i.e. Extended Pre- training with LoRA), a method that efficiently adapts a pre-trained VIT to a new target domain T, in algorithm []]
3

---

ExPLoRA
EXPLORA VIT Block- Unfrozen Ful-Rank Tuning
Embedding Sequernce
LoRA Turinng AN
Unfrozen U
Norm
Attention
Norm
MLP
MAE or Dinok? Decoders
EXPLoRA VTT Block- LoRA Turing
Embedding Sequence
A
a
A
Norm
Attention
Nom
MLP
Embecdigg
1
2
1-1 L
ViT Encoder Blocks Initialized with Pre-Trained Weights
LoRA
Figure 2: An overview of ExPLoRA. The set L. of I. VIT blocks is partitioned into two sets: 24 (red), which denotes blocks whose parameters are completely unfrozen, and L 2 (blue) which denotes blocks that undergo LoRA tuning (only the Q, V attention matrices). Note that the normalization layers always unfrozen across all blocks.
Algorithm ExPLoRA
Thus, ÂT comprises all weights 2, LoRA matrices in C U, and layer normalization parameters of all blocks.
Input: Ws = pre-trained VIT with layers L
[I,...;II: Xr unlabeled dataset
For DinoV2, we initialize ViT-L with Ws from DinoV2's VIT encoder, without registers (Darcet et al., 2024). Since Dino pre-trained checkpoints don't 't include Dino or iBOT linear heads, we initialize shared head go from scratch, which fully trained during extended pre-training.
2: Initialize frozen VIT with Ws from source domain S
(e.g., V2 or MAE weights).
3: Unfreeze all parameters of a subset of blocks U c C.
(e.g.. = } or = {))
4: Apply LoRA (with rank r) on Q and weights in attention layers of frozen blocks in \ and unfreeze normalization layers in these blocks.
For MAE, we initialize a ViT-L with Ws from the MAE ViT-L encoder and use provided weights to initialize de- coder ge (He al., 2022). During extended pre-training, beyond the ExPLoRA recipe in algorithm we apply LoRA with rank r' on the Q, V matrices of each attention layer in Iv- The LoRA rank r' may differ from rank used in the ViT encoder fo (appendix B.4]. All other decoder weights except layer-normalization are frozen, with no block fully unfrozen in Jo to minimize trainable parameters.
5: Train all unfrozen parameters År on the unlabeled dataset Xy using the same unsupervised objective as what was used for Ws (e.g., DinoV2 or MAE).
6: Output: A new pre-trained model W = Ws + Ar for target domain
Coneretely, we learn AT follows:
For the to additionally unfreeze the positional encoding and patch embedding weights for each group of channels, as part of @U,rr) in eq. (5) These cannot be initialized from Ws. as W's trained on RGB inputs, whereas multi-spectral inputs can have more or different channels.
VIT introduced by SatMAE need
AT arg min (E xedr
min
Ecs Cs (ow (ss),
(5)
where Ws) is VIT feature encoder parameterized by with initialization Ws. G6 is learnable decoder pa- rameterized by a (e.g., MAE/Dino decoder), Cs is the unsupervised loss that was used for Ws (e.g., reconstruction loss), and ((4,r) restricts the trainable parameter space to full-rank updates for blocks in // and LoRA rank-r updates in C \ U. Similar to Goyal ing the same unsupervised loss Cs was used for Ws is beneficial. As we show in section 6 ExPLoRA can even outperform full pre-training new domains from scratch.
Fine- Tine-Tuning post-ExPL.oRA After running ExPLoRA, we receive a new unsupervised model W Ws + AT for the target domain T that functions as any other pre-trained ViT WT. Ie (e.g., the Dino/MAE decoder) discarded as it is not part of the ViT encoder f. Only Ar, consisting of 1-2 unfrozen VIT blocks, LoRA matrices, and layer- normalization weights, are stored for each 7" all of which can be merged into the original ViT, thus preserving architec- ture. Like LoRA, ExPLoRA significantly reduces additional storage requirements compared fully pre- pre-training Wr.
(2023), we find that us-
In terms of notation, D-[L]-r64 refers to a ViT initialized with V2 weights, where = (L), and LoRA rank 64 applied to the Q, V matrices of attention layers in U.

---

ExPLoRA
Finally, solving for (T) on each labeled dataset 2/
6.1.1. RGB SATELLITE IMAGES
(T) arg min
Σ
C' (n2 (fo(x: (6)
Dataset We first consider the functional map of the world (fMoW) dataset of high-resolution satellite images, each paired with one of 62 classification labels (Christic et al., 2018). fMoW is used as a benchmark for satellite-image foundation models (Cong et al. 2022 Reed et al. 2023
(r)(r) (x.))E)
where the ViT fo is now initialized with W7. (1) is
task-speeific ViT decoding head with parameters 6, and C' represents a supervised-leamingg loss function.
We compare our results in table D against both prior fully pre-trained SoTA foundation models as well as PEFT tech- niques applied on ViTs pre-trained with MAE and/or Di- no weights. Our results demonstrate that D-FxPLoRA- [L]-r64 is SoTA in terms of IMoW-RGB average accuracy at 79.28%. ExPLoRA outperforms techniques that require fully and/or continually pre-training ViTs on f'MoW while using 6% of the original ViT encoder parameters and 8x less compute. Further experiments with MAE are in B.6
For any PEFT method, we allow 0 E ( where restricts the trainable parameter space θ of L (c.g., LoRA only on attention Q, V matrices). For linear probing, (0)) - 0, so (1.)) (x; W = T [xx W;). For general fine-tuning, we optimize both 9 E θ and ø unrestricted, modifying (T) as per the task (e.g., a decoder head for segmentation). With A (7), eq. (4 gives us final model weights wr(", which can be used for classification, segmentation, detection etc.
with LoRA fine-tuning outper- form other unsupervised initializations paired with PEFT techniques by 1-3%, including SoTA matrix-adapiation methods like AdaLoRA (Zhang et al.) [202cc), BOFT (Liu et al. 2024). VPT approaches such as GVPT (Yoo et al., SA SA2VP VP (Pei et al. 2024). and adapter methods
6. Experiments
Our experimental results consist of case study on satellite imagery (section66.1). with an ablation study in section 6.1.2 We evaluate on multiple downstream tasks in sections 6.1.3
2023
6.1.4 and 6.2 Additional experiments and ablations are provided in appendix B and training hyperparameter and
like Adapter+ (Steitz & Roth 2024). We also outper- form satellite image continual pre- pre-training methods such as
compute configurations are mentioned in appendix C
GFM (Mendieta et al. (023) and GDA (Scheibenreif et al.) 2024) by 6%. Additionally, applying SA" to ExPLoRA. initialized ViTs improves performance over D2no by 1% (rows 16 vs 17), showcasing A'ssoRA's compatibility with other PEFT methods and its versatility as an initialization for new domains.
We report both performance metrics and computational re- quirements for our experiments. As pre-training technique, ExPLoRA offers an efficient alternative to full domain- specific pre-training. Thus, unlike task- task-specific fine-tuning, its compute costs are amortized across all downstream appli- cations of the resulting model including feature extraction, linear probing, and various supervised tasks.
Using our strongest performing variant (i.e. EXPLORA with D2n, we investigate on fMoW-RGB compared with prior SoTA methods in ta- ble 2 Linear-probing represents freezing the backbone and then training a linear head on the features extracted from the frozen backbone, serving as a desirable metric of the quality of extracted embeddings. Our results demonstrate an improvement of over 18.2% in top 1 average accuracy over prior So'A methods, demonstrating that ExPLoRA learns robust unsupervised representations for its target do- main without requiring expensive from-scracch pre- trainini. Importantly, ExPLoRA A outperforms domain-specific prior SoTA solutions (rows 1-4), as well as V2n which sug- gests successful transfer learning on the target domain by leveraging knowledge from pre-training on natural images.
performance
Our results achieve a new SoTA top I accuracy of 79.3% (t1.5% on the competitive fMoW-RGB benchmark. outper- forming fully pre-trained and fine-tuned models while using 6% of the ViT encoder parameters and requiring only 100 GPU hours compared to 960+ hours for full pre-training- We also achieve a 18.2% improvement in linear probing accuracy on the same dataset. Across other satellite datasets, ExPLoRA matches or exceeds helly-pretraieed prior state- of-the-art methods while requiring 8x-10x less compute and 16x fewer trainable parameters and demonstrates competi- tive performance on WiLDS benchmark datasets as well.
6.1. Case Study: Satellite Imagery
1.2. ABLATION STUDY
We examine satellite images given their importance towards societal applications (section 8) and since they represent significant domain shift from natural images. There is large and growing body of research on developing founda- tion models for satellite imagery from scratch (Cong et al.] 2022 Reed et al., 2023 Tang et al., 2024), thus presenting a good benchmark for ExPLoRA.
We perform an ablation study (table on linear-probing performance for fMoW-RGB to determinee whether our pro- posed configuration performs optimally.
A natural question is whether the improvement in perfor- mance stems primarily from unfreezing blocks, or from
5

---

ExPLORA
Pre-Train
Fine- Tune
Pre-Train Train GPU hours
Fine-Tune GPU hours
Model
PEFT
Top 1 Acc.
Params
Params
ScaleMAE 566
Full
303.3M
303. 3M
960
610
77.80
SatMAE 17
Full
303.3M
303.3M
960
610
77.78
SatMAE
17
LoRA-r8
30
303.3M
0.8M
960
220
76.10
ScalcMAE
56
LoRA-r8
30
303.3M
0.8M
960
220
78.01
GFM 45
LoRA-r8
30
303.3M
0.8M
900
220
73.03
6
GDA 58
GDA-r16
8.5M
8.5M
310
310
71.88
7
MAE 28
LoRA-r8
30
0.8M
220
76.21
8
M- [1]
-664
LoRA-r8
30
18.7M
0.8M
80
220
76.55
V2
48
LoRA-r8
30
0.8M
220
78.08
Dino
48
BOFT-62m8
0.9M
230
72.40
Dino
48
Mona 73
7.1M
220
72.80
Dino V2
48
VPT-100
K1
0.4M
500
77.29
Dino
48
GVPT-100
74
0.4M
500
76.22
Dino V2
48
AdaLoRA-rS
1.2M
220
78.87
Dino V2
48
Adapter+
1.4M
220
78.16
D2
48
SA'VP 50
1.IM
410
77.53
D- D-[1]-r0 --64
SA-VP
50
18.7M
1.IM
100
410
78.51
18
D- D-[1]-r66 -r64
LoRA-r8
18.7M
0.8M
100
220
79.28
Table 1: Results the MMOWRGGB validation dataset. "Pre-train / Fine-tune Params" refer trainable parameters of the VIT-L encoder -r64 --64
required on the new domain, i.e. satellite images. M- M D2noV2 weights, respectively (section 5). Our measurements for GPU hours use standardized hardware platforms for fair comparison.
D- [L] -r64 refer ExPLoRA models initialized with MAE
Method
Arch.
Top Acc.
Blocks Unfrazen
LoRA
Norm
LoRA
Num.
GPU Top
Rank Unfrozen
Layers
Params hours
Acc.
GASSL
ResNet
68.32
SatMAF
17
ViT-I.
65.94
69.00
ScaleMAE [56
VIT-B
67.30
2
All
N/A
Yes
11
303.3M
1200
54.29
CrossScaleMAE 63
VIT-B
69.20
Dino V2 48
VIT-I
67.60
3
[G]
[]
12.7M
90
74.83
D2noV21 48
VIT-L
69.00
4
[-- 1,
b]
0
V
[]
25.3M
130
75.97
[]
256
V
[Q, v]
25.9M
180
75.51
D- II -664
VIT-L 76,86
128
V
A11
33.1M
220
55.03
D- [ -664
VIT-L 77.48
64
Mip
16.5M
48.55
Table 2: Linear-probing on IMoW-RGi The first four rows fully train the dataset. i denotes concatenating features from the last VIT blocks. All other rows features of the last ViT block.
8
[1]
64
[0, VI
18.7M
100
75.97
[9]
64
[Q.
18.7M
100
75.45
[L-1]
64
[Q,
18.7M
100
77.40
[L]
VPT-100
12.8M
430
70.14
12
[L]
64
X
[Q.
18.6M
100
76.78
[L]]
8
V
[Q,
13.4M
90
76.31
LoRA-tuning the rest of the VIT. We investigate this by unfreezing blocks [L, L-1} (with no LoRA) in row 4, and comparing that with EXPLORA-L-TS in row 13. As seen, un- freezing an extra block consumes almost double the number of parameters, but fails to yield the same improvement in performance 0.34%. Thus, simply increasing the number of unfrozen blocks will likely improve performance, but will not do so as effectively ExPLoRA, and will also significantly and sharply decrease the carfmeter-efficiinncy
[L]
32
V
[Q.V]
15.7M
100 76.40
64
V
1Q,VI
18.7M
100 77.48
[L-1,L]
64
1Q,V1
31.1M
140
77.76
1111111,1
64
[Q,
43.4M
180
78.04
Table 3: Ablation study using DinoV2-ExPI.oRA, measuring lincar-prohing accuracy on fMoW-RGB. The second row performs full pre- pre-training from scratch. All results are obtained by using concatenated features from the last 4 ViT blocks.
Next, we investigate whether applying high-rank LoRA to all matrices (including MLP) outperforms targeting only attention Q, V matrices. Surprisingly, LoRA-r123 on all matrices (row 6) only MLP matrices (row 7) signifi- cantly harms performance compared to L5RA-r256 on just Q matrices (row 5). However, both rows and 6 are much less parameter-efffcient than ExPLoRA (rows 13-15).
as effective as U/ = (L-1} or L4 = (L), ceteris paribus. To understand this result further, see section 7 We also notice slight drop in accuracy from leaving the normalization layers across the ViT frozen, seen in row 12.
Lastly, we investigate the impact of LoRA rank and in- creasing the set of unfrozen blocks 2 on ExPLoRA, ef- fectively varying the parameter budget r), As seen in table 3, changing the rank from 8 to 32 yields a mod- est improvement (1 0.09%, rows 13 vs 1 14), but increasing
The choice of l4 matters as well. As seen in rows 8-10, and 15, for the Dino V2 objective, {1} {9} are not
6

---

ExPLoRA
from 32 64 brings about a much larger gain (t 1.08%, rows 14 vs 15), with equivalent 3M parameter increase.
PT/ Params
PT/
Top
Method
PEFT
GPU hours
Acc.
Meanwhile, expanding the set of unfrozen blocks from U = [1] to U- = {L-1, L} (rows 15 vs 16) and further to 11 = (1, I-1, I.} (rows 16 vs 17) yield consistent improvements of t 0.28%.
1 MAE 28
Full
-/ /03.33
- 77770
51.61
2 SaMMA 17
Full
303.3M/ / 303.3M
1150/770
61.48
MAE 28
LoRA-r8
- / 0.88
-/ 2/2
46.97
SatMAE 7
LoRA-r8
303.3M / 008MM
1150/
59.48
GFM 45
LoRA-r8
303.3M / 0.8M
960/ 060/290
57.55
These results indicate that strategically constraining the pa- rameter space via @(4,r) r) in ExPLoRA yields better per- formance more efficiently than either uniform high-rank updates across all layers (row 5) or simply unfreezing more blocks without LoRA (row 4). This balance between full- rank updates in key transformer blocks and low-rank up- dates elsewhere more effectively captures domain-specine knowledge during pre-training.
6 GDA 58
GDA-r16
7.3M /7.3M
560/410
55.23
MAE
LoRA-r8
51.5M / 0.88M
380 /80/290
54.12
8 M- [+[1)-33
LoRA-r8
16.2M / 00.8M
290 / 290
51.84
M- [1,3
LoRA-r8
29.7M/ / 0088M
320/ / 290 60.15
Table 4: Results on MMoWSSentnee (validation), with ViT-L. "PT" and "FT"" refer to pre-training and fine-tuning, respectively. "Params" refers to trainable parameters required on the new do- main, ie. multi-specti izing a SatMAE model with MAE weights and then pre-training with blocks 1, 1,2,23,24 unfrozen. ExPLoRA achieves competitive performance while requiring significantly less compute than full domain-specific specific pre-training.
satellite images. "MAEs" refers initial-
Notably, full pre-training from scratch (row 2) achieves only 54.29% accuracy despite using 1200 GPU hours, suggest- ing that significantly more compute might be needed for performant from-scratch D2 checkpoint. In contrast, our standard configuration for ExPLoRA in row 15 achieves 77.48% accuracy with just 100 GPU hours (a 12x reduction in compute) while using only 18.7M parameters (6% of the full model). While rows 16-17 show further performance improvements with additional unfrozen blocks, row 15 rep- resents our preferred trade-off between performance and efficiency for most applications. Further ablations on com- pute efficiency (B.2). data efficiency (B.3), MAE decoder rank B.4. and VIT backbone size B.5 are in appendix B
FT Params
PT/ FT
Top 1 Acc.
Method
PEFT
GPU hours
GASSL 2
Full
23.5M/ / 23.5M
380 220
74.11
2 SatMAE 17
Full
303.3M/ / 303.3M
1120 610
79.69
3 MAE 28
LoRA-r8
0.8M
250
69.30
4 SatMAE 17
LoRA-r8
303.3M / 0.8M
1120 250 75.27
5 M- [-[L] -r32
LoRA-r8
18.7M / 0.8M
100 / 250 75.98
Table 5: Classification results on the validation set of fMoW- Temporal. "PT"" and "FT" refer to pre-training and fine-tuning, respectively. All MAE/SatMAE experiments use ViT-L, while GASSL uses a ResNet. ExPLoRA (M- (M)3)) achieves the best PEFT performance while using only fraction of the compute required for full pre-training.
1.3.3. MULTI- SATELLITE IMAGES
Dataset Next, we consider the IMoW-Sentinel dataset, large dataset of Sentinel-2 2 images introduced by Cong al.] (2022). Each image consists of 13 spectral bands and is paired with one of 62 classes.
1.4.4. ADDITIONAL SATELLITE DATASETS
With Mentinln,,,,, we evaluate transfer from natural im- ages to multi-spectral, low-resolution satellite images- a domain with significant distribution shift from RGB images due the absence of non-RGB bands in S. We use the group-channe VIT-L from Cong et al. (2022), initialized with MAE. During ExPLoRA, we additionally unfreeze only the patch embedding layers due to architectural differences.
We perform extensive experiments on downstream satellite datasets, with further results in B.
-emporrl Each input is a sequence of up to 3 fMo (Christie et al. 2018) images of location, distributed temporally, and paired with one of 62 classes. Since the inputs are now temporal sequences, we initialize the temporal MAE architecture from Cong et al. (2022) with MAE weights, and pre-train on Xr with 21 = [L] and LoRA rank 32. ExPLoRA then outperforms temporal SatMAE for PEFT (table 55) demonstrating successful transfer learning at fraction of the pre-training parameters and compute.
Table 4 higghlggts the challenge: direct fine-tuning of MAE weights on this domain results in a substantial 9% perfor- mance gap compared to SatMAE (rows 1 vs 2). LoRA tun- ing from MAE performs worse (row 3), and unfreezing four transformer blocks (row 7) fails to help. ExPLoRA bridges this gap effectively while requiring just 320 GPU hours and <1 <10% trainable parameters for pre-training compared to 1 150 hours for SatMAE. This demonstrates ExPLoRA's ability to efficiently adapt to domains with substantial distri- bution shifts from natural images. preserving performance while dramatically reducing computational requirements.
SpaceNet-v1 This dataset contains high resolution satel- lite images, each paired with segmentation mask for build- ings (Van Etten et al. 2018). The training and test sets consist of 5000 and 1940 images, respectively. For Ex- PLoRA, we pre-train on the training set. However, many images in the dataset contain extensive blacked-out regions,
7

---

ExPLoRA
SpaceNet
Resise45
Pre-Train/ Cee-Trii/Fiie-tune GPU hours
Method
PEFT
Method
PEFT
Top 1 Acc.
mloU
Acc.
SatMAE 17
Full
78.07
94.80
1
CLater 54
Full
93.90
ScaleMAE 56
Full
78.90
95.70
ICON
Full
90.10
DinoV2 48
LoRA-r8
76.69
97.60
MAE 28
LoRA-r8
-/120
92.13
D-[1]rr64
LoRA-r8
76.69
97.65
M- [L] +332
LoRA-r8
90/ 120
92.24
D2
48
Lin. Probe
1/
93.27
SatMAE 17
Lin. Probe 50.89
88.30
D2
48
LoRA-r8
-/ 120
92.97
ScaleMAE 56
Lin. Probe
47.17
89.60
D2 48
Lin. Probe 76.21
96.34
7
D- [L]-r32
Lin. Probe
90 10
94.41
D- [L]-r32
LoRA-r8
90 120
94.21
D- [L]
Lin. Probe 76.34
97.32
Table 7: Classification results on the validation set of Camelyon
Table 6: Results on the validation sets of SpaceNet-vl, segmen- tation task, and R5sisc-45, a classification task. The D- [L]-r64 ExPLoRA model from the final row of Table achieves state- of-the-art performance on Resisc-45 and competitive results on SpaceNet-vl without requiring pre-training.
ExPLoRA models achieve the best performance in both linear probing and LoRA-based fine-tuning configurations.
Pre-Train/Fine-tur GPU hours
Method
PEFT
Top I Acc.
MAE
28
LoRA-r8
-0
60.07
indicating limits of the visible region. Considering this limitation and the small dataset size, is not clear whether additional pre-training is effective. We find that, despite this, ExPLoRA remains on par with the LoRA-tuned D2noV2 model and remains competitive with the fully pre-trained and fully fine-tuned domain-specine models (table
2
M- [L] -r32
LoRA-r8
70/90 900
61.86
D2
48
Lin. Probe
-/ 10
66.04
D2
48
LoRA-r8
-/90
67.10
D- [LL-r33)
Lin. Probe
70/ 10
62.95
D- [r]-
LoRA-r8
70/ 90
68.07
Table 8: Classification results on the validation set of iWildcam.
Pre-Trin Fre-Train/Fine-tnee
Top Acc.
AR 0.5:0.95
AP@
Method
RESISC-45 The RESISC-45 (Cheng et al. 2017) bench- mark dataset consists of 31, 31,500 satellite images of varying resolution (0. (0.2m-30m GSD), with 45 classes. The data is split into 25,200 training and 6,300 validation images, per Reed et al. (2023). In table 6 our D-ExPLoRA pre- trained on only high-resolution fMoW-RGB images (last row of table [ achieves SoTA results of 97.32% on multi- resolution RESISC-45 images, with just linear- aheearprobbig... We demonstrate successful transfer learning from ExPLoRA pre-training, without requiring any additional modifications for scale-aware representation learning (Reed et al.) 2023).
GPU hours
0,5:0.95
ICON 35
68.9
MAE 28
-/ 190
82.5
53.8
58.7
M- [1]-r64
30/ 190
79.3
51.6
56.6
D2 48
190
82.3
52.1
57.1
D- D-[1]
30/ 190
82.7
54.5
59.2
Table 9: Object detection results the validation set of Global- Wheat, AP and AR stand for average precision and average recall. ExPLoRA outperforms the baseline methods across all metrics.
iWildeam iWildcam classification requires identifying one of 182 animal species given an image. We pre-train on the training set, finding that this outperforms pre-training on the extra-unlabeled set. In table 8 we find an improvement over Dino using LoRA-r8 PEFT. Surprisingly, the linear probing performance of the ExPLoRA suffers in comparison with Dino suggesting possible overfitting due a small domain gap. This is likely because natural image datasets such as ImageNet (Deng et al. (2099) used for pre-training DinoV2 already contain many images of animals.
6.2. WILDS Datasets
We test ExPLoRA on the WILDS (Koh et al. (021) bench- mark, specifically on Camelyon (Bandi et al., 2018), iWildcam (Beery et al., 2020) and GlobalWheat (David et al. 2020.22002 (021) datasets, representing domain transfers to medical, wildlife, and agricultural imagery, respectively.
Camelyon17 The WILDS Camelyon17 dataset consists of images of cancerous and non-cancerous cell tissue. We use the "train-unlabeled" split for pre-training ExPLoRA, and either use LoRA fine-tuning or linear probing on the training set of the labeled split. We report accuracy the binary classification problem and compare with entries on the WILDS leaderboard which use unlabeled data. Our results in table demonstrate improved performance over domain domain-specific methods as well as Din once again successfully bridging the domain gap.
The GlobalWheat dataset consists of an ob- ject detection task, where images of wheat fields are associ- ated with bounding boxes on the visible wheat heads (David) et al. 2020 2021). ExPLoRA extends pre-training on the training set, and then we fine-tune using Detectron2 code for object-detection with ViTs (Wu et al. 2019). ExPLoRA out- performs both fully pre-trained baselines from the WILDS leaderboard and strong VFMs Dino and MAE on top I accuracy, average precision, and average recall.
GlobalWheat
8

---

ExPLoRA
7. Further Analysis
empirically confirmed in table where linear probing accu- racy correlates inversely with the mean eigenvalue of each block (i.e., block 23 block 22 block 0> block 9). The attention maps in fig. 9 further support this, showing that deeper layers focus more clearly on central objects, while earlier layers (e.g.. blocks 9, 10) exhibit diffuse attention patterns spread around the border.
Backrtte Stans hicppel Spanatum
Diccoeas Verters Prie Operatio We
Findings and Unfreezing Strategy for MAE: For MAE, we see similar, but less pronounced trend. However, MAE is only trained for reconstruction, and so retains more local information across the ViT's layers. This is reflected by its lower patch-wise wise eigenvalues, higher localization accuracy, and lower global accuracies than Dino.
Figure 3: The mean of the eigen- Figure 4: The variance of the values of the feature map out- eigenvalues of the feature map putted by each VIT block.
outputted by each VIT block.
Coll
ExPLoRA's Impact: D-ExPLoRA preserves local infor- mation in the middle layers but also improves localization accuracy in the last few layers. Importantly, it also enhances the global information contained in the patches for deeper model layers. This indicates a better understanding of the target domain, as seen in B.8 where ExPLoRA's attention highlights the central object more clearly.
Figure 5: Linear probing patches Figure 6: Linear probing patches for position (local information), for classification (global infor- across all VIT blocks.
mation), across all VIT blocks.
The key design choice of ExPLoRA is to constrain the trainable parameter space during pre-training via 0(4, r) in eq. (5). where 44 C C layers of the ViT undergo full-rank training while the remaining frozen layers C U receive low-rank updates. For parameter-efffcieneyy, we wish to keep [4| < [C] and make an informed choice of which layers to unfreeze based on their potential to improve representation learning during extended pre-training.
Thus, our analysis provides guidelines for practitioners to select which blocks unfreeze based the eigenvalue properties and classification accuracy patterns of different ViT layers, offering a systematic approach to constraining @(4, r) when pre-training with ExPLoRA (eq. (
8. Conclusion and Discussion
In this paper, introduce ExPLoRA, a novel pre-training strategy to adapt pre-trained ViT foundation models for nat- ural images to additional visual domains such as satellite im- agery or medical data. We challenge the common paradigm of expensive pre-training from scratch for each new visual domain by offering a solution to transfer knowledge from foundation models that matches or outperforms domain- specific foundation models. ExPLoRA makes powerful foundation models accessible to researchers with limited computational resources while using 8- 8-10x less compute and 16x fewer parameters. Our hope is that ExPLoRA en- ables further use of VFMs on domains other than natural images without requiring vast computational resources for pre-training.
We conduct an investigation on 5 models using a sample of XD,, These models are DinoV2, D-ExPLoRA- [L]- r64, SatMAE, MAE, and M-EXPLoRA- [L] [L]-664. -644 We do the following analyses: (i) PCA measure the mean and variance of eigenvalues of patch feature vectors for each VIT block, in figs. 3 and 4 (ii) linear probing for local or global information Darcet et al. 2024 by training logistic regression classifiers on each block's 's patch feature vectors, to predict either patch position (fig. 5 or image class (fi..T6).
Findings and Unfreezing Strategy for DinoV2. Our anal- ysis reveals that the spectral properties of block's feature map (fig. and the ability to retrieve local information from its output patch tokens (fig. 5) are correlated. The classification accuracy for position and the mean and vari- ance of the principal eigenvalues peak in the middle-layers of the model, suggesting that the middle blocks capture fine-graine fine-grained local properties of patches (e.g., texture. rela- tive position). Meanwhile, deeper blocks focus on global semantic understanding, shown by increased classifica- tion accuracy for image class prediction in fig. 6 and lower variance in feature map eigenvalues in fig. Combined, these results suggest that unfreezing deeper layers, such as u= = {L}. allows the model to better capture global features without overfitting to local details of images of T. This is
While effective, many aspects of ExPLoRA deserve fur- ther study. The strategy of fully training small budget of weights combines well with PEFT techniques like LoRA- understanding this further would be valuable. Future work might explore whether other parameter-efficient techniques could improve ExPLoRA during pre-training more effec- tively than unfreezing blocks. Lastly, an investigation of ExPLoRA on large language models would be valuable.

---

ExPLoRA
Impact Statement
Multi-modal multi-task masked autoencoders. arXiv preprint arXiv:2204001678 2022.
Multimae:
As the scale of models and datasets grows exponentially, access to the computing power necessary to develop and use foundation models is increasingly restricted to the hands of a few organizations. This leaves many researchers in academia or smaller companies reliant on the resources of such organizations for ML research and applications. Tech- niques such as PEFT can alleviate this dependence and enable those with fewer computational resources to adapt, investigate, and customize models for their own needs. We hope that ExPLoRA furthers this goal, allowing ML practi- tioners to tailor foundation models with minimal compute, thus broadening access to powerful ML tools for critical fields like sustainability and medicine.
[4] Bandi, P., Geessink, O., Manson, Q. Van Dijk, M., Balkenhol, M., Hermsen, M., Bejnordi, B. E.,, Lee, B., Paeng, K. Zhong, A., et al. From detection of individual metastases classification of lymph node status the pa- tient level: the camelyon17 challenge. IEEE transactions on medical imaging, 38(2):550 550, 2018
[5] Beery, S., Cole, E.,, and Gioka, A. The iwildcam 2020 competition dataset. CoRR, abs/2004.44444344 2020. URL
[6] Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.0722 2021.
For example, automated analysis of satellite imagery can inform social, economic, and environmental policies, but manual curation is expensive, and pre-training models on such data has significant costs, both environmental and oth- erwise (appendix (D). ExPLoRA offers a more efficient way to create new foundation models for different visual do- mains via distilling knowledge from existing foundation models trained on natural images. This can sharply reduce costs, aid researchers and policymakers, and enable flcxible downstream applications.
[7] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
[8] Caron, M., Misra. I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912 2299924, 2020.
Acknowledgements
This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), In- telligence Advanced Research Projects Activity (IARPA), via 2021-201 1000004, NSF(#1551565) ARO (W91 INF- 21-1-0125), ONR (N00014-23-1-21599) CZ Biohub, HAI. The views and conclusions contained herein are those of the authors and should not be interpreted necessarily rep- resenting the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Gov- ernment is authorized to reproduce and distribute reprints for governmental purposes not-withstanding any copyright annotation therein.
[9] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
[10] Chavan, A., Liu, Z.,, Gupta, D., Xing, E., and Shen, Z. One-for-all: Generalized lora for parfmeter-efficient fine-tuning, arXiv preprint arXiv:2306.07961 2023.
[11] Chen, X., Fan, H., Girshick, R., and He, K. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020.
References
[12] Chen, Z., Duan, Y., Wang, W., He, J., Lu, T., Dai,
[1] Aghajanyan, A., Zettlemoyer, L.,, and Gupta, S. Intrin- sie dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint 13255 2020.
J., and Qiao, Y. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.085. 2022.
[13] Cheng, G., Han, J., and Lu, X. Remote sensing im- age scene classification: Benchmark and state of the art. Proceedings of the IEFE, 105(10)1185--188) 2017.
[2] Ayush, K., Uzkent, B., Meng, C., Tanmay, K., Burke, M., Lobell, D., and Ermon, S. Geography-awan self- supervised learning. In Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 10181 10190, 2021.
[14] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sut- ton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113 2023.
[3] Bachmann, R., Mizrahi, D., Atanov, A., and Zamir, A.
10

---

ExPLoRA
[15) Christie, G., Fendley, N., Wilson, J., and Mukherjee, R. Functional map of the world. In CVPR, 2018.
[26] Grill, J.-B., Strub, F,, Altché, F,, Tallec, C., Richemond, P. Buchatskaya, E.,, Doersch, C., Avila Pires, B., Guo, Z.., Gheshlaghi Azar, M., et al. Bootstrap your own latent-a new approach to self-suupr learning. Advances in neural information processing systems, 33:21271-21284, 2020.
[16] Chuan-Xian, R., Yi-Ming, Z., You- Wei, L.., and Hong, Y. Towards unsupervised domain adaptation via domain- transformer. arXiv preprint arXiv: 13777 2022.
[17] Cong. Y., Khanna, S., Meng, C., Liu, P., Rozi, E., He, Y., Burke, M., Lobell, D., and Ermon, S. Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery. Advances in Neural Information Pro- cessing Systems, 35:197-211, 2022.
[27] Han, C., Wang, Q., Cui, Y., Cao, Z., Wang, W., Qi, S., and Liu, D. E. 2vpt: An effective and effi- cient approach for visual prompt tuning. arXiv preprint urXiv:2307.33770, 2023.
[28] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Gir- shick, R. Masked autoencoders are scalable vision learn- ers. In Proceedings of the IEEE/CVF conference com- puter vision and pattern recognition, pp. 16000-16009, 2022.
[18] Darcet, T., Oquab, M., Mairal, J., and Bojanowski, P. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024. URL httts:://ppnreeieeeeee.e http://wwwwweeee.reeeeeee
id 2dn0033LLJJ
[29] Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 12(7):2217-22) 2226, 2019.
[19] David, E., Madec, S., Sadeghi-Tehran, P., Aasen, H., Zheng, B., Liu, S., Kirchgessner, N., Ishikawa, G., Na- gasawa, K., Badhon, M. A.. et al. Global wheat head detection (gwhd) dataset: A large and diverse dataset of high-resolution rgb-labelled images to develop and bench- mark wheat head detection methods. Plant Phenomics, 2020.
[30] Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang. L.. and Chen, W. LoRA: Low-rank adap- tation of large language models. In International Confer- ence Learning Representations, 2022. URL https: //openreview www. for Fww.mooiiiiiooo http://www....ooooommmmoooo//e/e
[20] David, E., Scrouart, M., Smith, D., Madec, S., Velumani, K., Liu, S., Wang, X., Espinosa, F. P., Shafiee, S., Tahir, I. S., et al. Global wheat head dataset 2021: An update to improve the benchmarking wheat head localiza- tion with more diversity. CoRR, 2021.
[31] Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727, Springer, 2022.
[21] Delattre, S. and Fournier, N. On the kozachenko- leonenko entropy estimator. Journal of Statistical Plan- ning and Inference, 185:69-93, 2017.
[32] Kang. G., Jiang, L., Yang, Y., and Hauptmann, A. G. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 4893- 4902. 2019.
[22] Deng. J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
[23] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weis- senborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transform- ers for image recognition at scale. ICLR, 2021.
[33] Khanna, S., Vaidyanath, S., and Velu, A. Differen- tiable weight masks for domain transfer. arXiv preprint arXiv:2308. 130.27308...95,, 2023.
[34] Khanna, S., Liu, P., Zhou, L., Meng, C., Rombach, R., Burke, M., Lobell, D. B., and Ermon, S. Diffusionsat: A generative foundation model for satellite imagery. In The Twelfth International Conference on Learning Represen- tations, 2024. URL htttp::/ net/ Forum SerummiddISwwbbDDD..
[24] Gao, Y., Shi, X., Zhu, Y., Wang, H., Tang, Z., Zhou, X., Li, M., and Metaxas, D. N. Visual prompt tun- ing for test-time domain adaptation. arXiv preprint arXiv:2210.048311 2022.
[25) Goyal, S., Kumar, A., Garg, S., Kolter, Z., and Raghu- nathan, A. Finetune like you pretrain: Improved finetun- ing of zero-shot vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19338-19333 19347,1947, 2023.
[35] Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M.. Phillips, R. L., Gao, I., et al. Wilds: A benchmark of in- the-wild distribution shifts. In International conference on machine learning, pp. 5637-5664. PMIR, 2021.

---

ExPLoRA
[36] Lacoste, A., Luccioni, A., Schmidt, V., and Dandres, T. Quantifying the carbon emissions of machine learning. CoRR, abs/1910.09700,, 2019. URL http://arxiv
[47] Nie, X., Ni, B., Chang, J., Meng. G., Huo, C.,, Xiang, S., and Tian, Q- Pro-tuning: Unified prompt tuning for vision tasks. IEEE Transactions Circuits and Systems for Video Technology, 2023.
org/abs/1910.. 0970C
[37] Larsson, G., Maire, M., and Shakhnarovich, G. Fractal- net: Ultra-deep neural networks without residuals. arXiv
[48] Oquab, M., Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. DINOv2: Learn- ing robust visual features without supervision. Transac- tions on Machine Learning Research, 2024. ISSN 2835- 8856. URL https: /openreviewwwe.e net/ forum? Featured Certification.
preprint arXiv: 160500788, 2016.
[38] Li, C., Farkhoor, H., Liu, R., and Yosinski, J. Measur- ing the intrinsic dimension of objective landscapes. arXiv preprint arXiv: /00v:/80..000 08838, 2018.
id=a 61sUt. 62Ft
[39] Lian, D., Zhou, D., Feng, J., and Wang, X. Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Sys- tems, 35:109-123, 2022.
[49] Oren, Y., Sagawa, S., Hashimoto, T. B., and Liang, P. Distributionally robust language modeling, arXiv preprint arXiv: /e0.//99900000 02060, 2019.
[50] Pei. W., Xia, T., Chen, F., Li, Tian, J., and Lu, G. Sa²vp: Spatially higgeddannd-adapee
[40] Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
visual prompt. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pp. 4450 0400-4458,000 2024.
[51] Peng. X., Usman, B., Kaushik, N., Hoffman, J., Wang, D., and Saenko, K. Visda: The visual domain adaptation challenge. arXiv preprint 17XXi:171000690 2017.
[41] Liu, W., Qiu, Z., Feng, Y., Xiu, Y., Xue, Y., Yu, L.., Feng, H., Liu, Z., Heo, J., Peng, S., Wen, Y., Black, M. J., Weller, A., and Schölkopf, B. Parameter-efffiient orthog- onal finetuning via butterfly factorization. In The Twelfth International Conference on Learning Representations, 2024. URL ctttps///peenrviie..nrt
[52] Pu, G., Jain, A., Yin, J., and Kaplan, R. Empirical anal- ysis of the strengths and weaknesses of peft techniques for llms. arXiv preprint arXiv:2304. 14999, 2023.
[53] Qiu, Z., Liu, W., Feng, H., Xue, Y., Feng, Y., Liu, Z., Zhang, D.,. Weller, A., and Schölkopf, B. Controlling text- to- to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36:79320- 79362, 2023.
id-7NzgkEdGyr
[42] Ma, J., He, Y., Li, F., Han, L., You, C., and Wang, B. Segment anything in medical images. Nature Communi- cations, 15(1):654, 2024.
[54] Qu, H. and Xie, S. M. Connect later: Improving fine- tuning for robustness with targeted augmentations. arXiv preprint arXiv:2-40..03325 2024.
[43] Man, X., Zhang, C., Feng, J., Li, C., and Shao, J. W-mac: Pre-trained weather model with masked au- weather forecasting. arXiv preprint arXiv:2304.08755 2023.
toencoder for
[55] Reed, C. J., Yue, X., Nrusimha, A., Ebrahimi, S., Vijaykumar, V., Mao, R., Li, B., Zhang, S., Guillory, D., Metzger, S., et al. Self-supervised pretraining im- proves self-supervised pretraining. In Proceedings of the IEEF/CVF Winter Conference on Applications of Com- puter Vision, pp. 2584-2594 2022.
[44] Mañas, O., Lacoste, A., Giro-i Nieto, X., Vazquez, D., and Rodriguez, P. Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9414-9423, 2021.
[56] Reed, C. J., Gupta, R., Li, S., Brockman, S., Funk, C., Clipp, B., Keutzer, K., Candido, S., Uyttendaele, M., and Darrell, T. Scale-mae: A scale-aware masked autoen- coder for multiscale geospatial representation learning. In Proceedings of the IEEE/CVV International Conference Computer Vision, pp. 4088 2023.
[45] Mendieta, M., Han, B., Shi, X., Zhu, Y., and Chen, C. Towards geospatial foundation models via continual pre- training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 16806-16816 2023.
[46] Moutakanni, T., Bojanowski, P., Chassagnon, G., Hude- lot, C., Joulin, A., LeCun, Y., Muckley, M., Oquab, M., Revel, M.-P., and Vakalopoulou, M. Advancing human- centric ai for robust x-ray analysis through holistic self- supervised learning. arXiv preprint arXiv:2405.01465 2024.
[57] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
12

---

ExPLoRA
[58] Scheibenreif, L., Mommert, M., and Borth, D. Param- eter efficient self-super geospatial domain adapta- tion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp.
[69] Wu, Y., Kirillov, A., Massa, F, Lo, W.-Y, and
Girshick, R. Detectron2. ht.tps://gi htt//githain thub. com/
facebookresearcch/eettttzzz
2019.
[70] Xu, T., Chen, W., Wang. P., Wang, F., Li, H., and Jin, R. Cdtrans: Cross-domain transformer for unsupervised domain adaptation. arXiv preprint arXiv:2109006166 2021.
27841-278511 June 2024.
[59] Shen, S., Seneviratne, S., Wanyan, X., and Kirley, M. Firerisk: A remote sensing dataset for fire risk assessment with benchmarks using supervised and self-supervised learning. In 2023 International Conference on Digital Image Computing: Techniques and Applications (DICTA),
[71] Yang. J., Liu, J., Xu, N., and Huang, Tvt: Transfer- able vision transformer for unsupervised domain adapta- tion. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 520-530, 2023.
pp. 189-196. IEEE, 2023.
[60] Singhal, P., Walambe, R., Ramanna, S., and Kotecha, K. Domain adaptation: challenges, methods, datasets, and applications. IEEE access, 11:6973-7020, 2023.
[72] Yin, D., Hu, L.,, Li, B., and Zhang, Y. Adapter all you need for tuning visual tasks. arXiv preprint 2023.
urXiv:2311.1500
[61] Steitz, J.-M. O and Roth, S. Adapters strike back. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 23449-23459, 2024.
[73] Yin, D., Hu, L., Li, B., Zhang, Y., and Yang, X. 5% 100%: Breaking performance shackles of full fine-tuning on visual recognition tasks. arXiv preprint 2024.
[62] Sun, T,, Lu, C., Zhang, T., and Ling, H. Safe self-refinement for transformer-based domain adapta- tion, 2022.
[74] Yoo, S., Kim, E., Jung, D., Lee, J., and Yoon, S. Im- proving visual prompt tuning for self-supervised vision transformers. In International Conference on Machine Learning, pp. 40075-40092. PMLR, 2023.
07683
[63] Tang, M., Cozma, A., Georgiou, K., and Qi, H. Cross- scale mae: tale of multiscale exploitation in remote sensing. Advances in Neural Information Processing Systems, 36, 2024.
[75] Zhang, J., Qi, L., Shi, Y., and Gao, Y. Domainadaptor: A novel approach to test-time adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 18971-18981, 2023a.
[64] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, et al. Llama: Open and ef- ficient foundation language models. arXiv preprint WxXi::230.11397/ 2023.
[76] Zhang, J., Zhou, Z., Mai, G., Mu, L., Hu, M., and Li, S. Text2seg: Remote sensing image semantic segmentation via text-guided visual foundation models. arXiv preprint arXiv:2304. 105977 2023b.
[77] Zhang, Q., Chen, M., Bukharin, A., He, P., Cheng, Y., Chen, W., and Zhao T. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh Interna- tional Conference on Learning Representations, 2023c.
[65] Tsai, Y-Y., Mao, C., and Yang, J. Convolutional visual prompt for robust visual perception. Advances in Neural Information Processing Systems, 36:27897-27921, 2023.
[66] Van Etten, A., Lindenbaum, D., and Bacastow, T. M. Spacenet: A remote sensing dataset and challenge series. arXiv preprint 1xX.:18877.0132. 2018.
[78] Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., and Kong, T. ibot: Image bert pre- pre-training with online tokenizer. arXiv preprint arXiv:2 (xXXx:2211100822 1.07832, 2021.
[67] Venkateswara, H., Eusebio, J., Chakraborty, S., and Panchanathan, S. Deep hashing network for unsuper- vised domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 10118-502,2000
[79] Zhou, Y., Chia, M. A., Wagner, S. K., Ayhan, M. S., Williamson, D. J., Struyven, R. R., Liu, T., Xu. M., Lozano, M. G., Woodward-Cowe -Cuut, P., et al. A foundation model for generalizable disease detection from retinal images. Nature, 622(798 156- 2023.
[68] Wang, D., Shelhamer, F.,, Liu, S., Olshausen, B., and Darrell, T. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021. URL https:: // //penreview net. Fow.mmid=uul1:31 3bZ1kk33333333
[80] Zhu, J., Bai, H., and Wang, L. Patch-mis mix transformer for unsupervised domain adaptation: game perspective. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3561-3571, 2023.
13

---

ExPLORA
Appendix
include supplementary material the following sections. We provide additional contextualization and comparison with recent related work in appendix A] additional experimental results and ablations in appendix B training and hyperparameter details in appendix and an overview of the environmental impact of our work in appendix D
A. Further Contextualization with Related Work
Here, we expand upon ExPLoRA's differences with recent related work.
A.1. Comparison with Geospatial Domain Adaptation
Recent work has explored both continual pre-training (GFM) (Mendieta et al., 2023) and parameter-efficieet domain
adaptation (GDA) (Scheibenreif et al. 2024) for satellite imagery. We compare these approaches with ExPLoRA in table [0
ExPLoRA differs from these approaches in several key aspects. Unlike GFM which trains the full backhone, ExPLoRA achieves superior performance with only fraction of trainable parameters. While GDA is also parameter-efficient, it requires non-mergeable scaling vee- tors that induce inference latency and modify the ViT, whereas ExPLoRA's LoRA adapters can be merged into the VIT's weights. Additionally, ExPLoRA extends be- yond MAE architectures (supporting D2 and others) and allows flexible configurations between pre-training and fine-tuning, including varying LoRA ranks or using different PEFT methods, which GDA doesn't 't support out-of-the-box.
GFM
GDA
ExPLoRA (Ours)
35
68
Paramcter-efficient Training objective
X MAE
V MAE X
V Any V
Arch, preservation Fine-tuning
Any PEFT
Only LoRA
Any PEFT
Table 10: Differences between prior geospatial domain adaptation methods and ExPLoRA
We also demonstrate ExPLoRA's 's broader applicability through experiments on larger datasets (fMoW-RGB, MMoW-Senting which have 400k-800k images vs 90k images in FireRisk (Shen et al.. 2023), the largest dataset used in GDA) and domains beyond remote sensing (i.c. WiLDS). Our analysis section provides insights into block-wise information encoding, offering practitioners systematic approach for block selection during extended pre-training- a unique feature not present in prior work.
A.2. Comparison with Unsupervised Domain Adaptation
Unsupervised domain adaptation (UDA) enables models to generalize unseen domains (Kang al. 2019 Oren al., 2019 Singhal al.
UDA
ExPLoRA
2023 Khanna al. 2023) Traditional UDA assumes:
Source data
Labeled Data Unlabeled
None
Source knowledge
Weights
i Vs y ps(x, y). labeled source domain dataset
Target data
Unlabeled
(ii) X7 {x ~ (x)), an unlabeled target domain dataset
Label constraints
Yr Ys
None
(ii) Yr C Ys, constraining the label-set of I with respect to S
Table 11: Differences between UDA and ExPLoRA
Common UDA benchmarks like Office-Home (Venkateswara et al.
2017) and VisDA-2017 (Peng et al. 2017) follow this setup (Xu et al.
2021
Sun al. 2022 Yang et al.
2023 Zhu et al. 2023).
ExPLoRA's setting in section
different: we only require weights Ws from unsupervised pre-training on (x(x). without source data access or label set restrictions. This enables adaptation across wider domain shifts (e.g., ImageNet to multi-spectral satellite imagery, section 6.33). Thus, rather than competing with UDA methods, ExPLoRA can complement them by providing better initialization than standard natural-image pre-training (as seen empirically in appendix B.7
B. Additional Experimental Results
Below, we include further experimental results as continuation of section 6
14

---

ExPLoRA
B.1. Results on Additional Downstream Datasets
NAIP We consider a land-cover classification dataset used in (Ayush et al. 202T, where each of 244,471 training and 55,529 validation images are paired with one of 66 land cover classes obtained by the USDA's National Agricultural Imagery Program. In table 12 we first demonstrate similar performance between both natural-image backbones (rows 4 and 5), which surprisingly outperform SatMAE, which is pre-trained on IMoW-RGB. We use ExPLoRA to pre-train from D2 to the training set of this dataset (without labels). Our results (row 6) demonstrate comparable performance, suggesting that for this dataset, domain-speedt specific knowledge may not be highly relevant to successfully solve the task.
Method
PEFT
Top Acc.
GASSL 12]
Full
57.63
2 SaLMAE 17
Full
71.77
3 SatMAE 17
LoRA-18
69,45
4 MAE 28
LoRA-18
70.36
5 D2 48
LoRA-r8
70,40
D- [-[]]-rk L-33-33
LoRA-r8
70.40
Table 12: NAIP validation set results
EuroSAT The dataset contains 27,000 13-band satellite images of 10 classes (Helber et al. 2019), sourced from Sentinel-2. For ExPLoRA, A, we don't pre-train on this dataset's training set, and instead use LoRA fine-tuning starting with the pre-trained weights learned in the last row of table 4 DinoV2, and match the performance achieved by the domain-specific SatMAE which was fully pre-trained on fMoW-Sentinel, and fully fine-tuned on EuroSAT (table This demonstrates the successful use of our extended pre-trained model on further downstream datasets.
Method
PEFT
Top Acc.
SeCo [44]
Full
93.14
SatMAE 17
Full
98.98
We demonstrate improved performance over
SatMAE 17
LoRA-r8
98.73
Dino V2 48
BOFT-b8m2
96.60
M- L] -64
LoRA-r8
98.54
Table 13: EuroSAT validation set results
B.2. The Importance of Extended Pre-training
To evaluate ExPLoRA's effectiveness, analyze how its performance scales with computational resources. Specifically, investigate two key questions: first, given fixed compute budget, what is the optimal allocation between extended pre-training and fine-tuning? Second, for a fixed parameter budget, does investing compute in extended pre- training provide advantages over standard fine-tune tuning approaches?
Fine tuninunnng Accuracy (96)
77
ExPLORA-USIU -64--120000 (18.7M. D.M
ExPLORA [L] -64-tt155kk [18.7M. 08M)
EXPLORA 754 1410k100k (18.7M. 0.8M
We address these questions in fig. 7 focusing on D2noV2 models running NVIDIA-A4000 GPUs We evaluate D-EXPLORAA-DE [6]- for different lengths of pre-training (50k, 100k, 150k, and 200k iter- ations), corresponding to 24, 48, 72, and 96 GPU-hours of extended pre-training respectively. Each checkpoint undergoes LoRA-r8 fine- tuning. We compare against three baselines: (i) Direct LoRA-r8 fine-tuning on D2 weights (ii) Fine-tuning DinoV2 with block 24 unfrozen and LoRA-r64 (matching ExPLoRA's parameter bud- get) (iii) Fine-tuning Dino V2 with blocks 0, 1, 23, 24 unfrozen and LoRA-r64 (55.8M parameters vs ExPLoRA's 18.7M).
EXPLORA r64 118.7M,
DinoV2-nt I-, 0.8M)
DinoV2-y- [1)0))-([])__,___)0 Iy 18.7K)
DinoV2-2-
L-,,, (-, 55
50
100
150
200
250
300
550
460
Total Compute (GPU Hours)
Figure 7: Fine-tuning accuracy versus total compute (measured in GPU-hours). Total compute includes both pre-training (if applicable) and fine-tuning phases. Along with the label for each method in the legend, we include #fpre-trriniig params, #finc-tuning params).
Results in fig. 7 demonstrate that ExPLoRA's 's extended pre-training achieves a 1 1.0% improvement in maximum fine-tuning accuracy within the same total compute budget (320 GPU hours). Notably, even increasing the parameter budget during fine-tuning fails to match this performance. While additional pre-training iterations beyond 100k improve initial fine-tuning accuracy, they have minimal impact on the final accuracy ceiling, highlighting ExPLoRA's computational efficiency. Importantly, the total amount of compute invested in pre-training and fine-tuning ExPLoRA is still dwarfed by the compute spent for full pre-training from scratch, e.g., 960 GPU hours for SatMAE (table I).
Lastly, we re-iterate that the benefits of extended pre-training become more pronounced as the domain gap increases. While fig. dedmnnstsssssss a 1 % improvement on RGB satellite images, table Thighlights 8% improvement when using ExPLoRA + PEFT (row 8) over full fine-tuning MAE (row 1) on multi-spectral satellite images- a domain with substantial shift from natural images. ExPLoRA still requires less total compute (610 vs 770 GPU hours), demonstrating that its efficiency advantages scale with domain difficulty.
15

---

ExPLORA
B.3. Convergence and Data Efficiency
Another important question is on EXPLoRA's 's data efficiency- i.e. can ExPLoRA achieve good representations on the target domain without requiring many training iterations?
Accuracy vs. Pre-Training Iterations
In fig. [] we plot the linear-probing accuracy against the number of extended pre- training iterations for ExPLoRA (in blue). ExPLoRA improves quickly, requiring between 100-150k extended pre-training iterations to reach optimal performance. As discussed in section 6.1.2 unfreezing additional transformer blocks (in red)
un. prob Accuracy
-6
24
fails to achieve the same level of performance while requiring more parameters.
70-
EXPLORA
One hypothesis for the effecttivnness of pairing unfreezing blocks with LoRA is that low-rank updates to the ViT backbone "nudge" the sequence of embedded visual tokens from S to those representing T,, which then enables the unfrozen ViT block to efficiently compress global information from the new domain.
0.00 0.25 0.30 0.75
1.00 1.25 1.50
1.75 2.00
Number of pro training iteratioon (1e5)
Figure 8: Lin. prohe accuracy vs, number of training iterations.
B.4. Impact of MAE Decoder Rank
As outlined in section 5 we initialize the MAE decoder Iv (with its LD transformer layers) with pre-trained weights Ws from He et al. (2022), keeping all decoder weights (except layer norm) frozen during extended pre- vre-training on Ny. We apply LoRA with rank r' the Q, V weights of the attention layers in the decoder LD. while unfreezing 1-2 blocks L in the ViT encoder L and applying LoRA with rank the remaining layers L U (algorithm D
Decoder Rank
Top I Acc.
8 59.75
16
59.77
32
60.15
64
59.21
We evaluate ExPLoRA with L-(1, on MeW-Senninnl, using a fixed encoder LoRA rank r = 64, unfreezing blocks 4/ = {1, L). and varying the decoder rank r'. We then fine-tune the resulting model with LoRA r = 8 and measure the highest top accuracy on the validation set of MMoW-Sentinel. Table shows that increasing r' up to 32 improves fine-tuning performance, which then declines by 1 0.94% for r' = 64. This suggests that balancing the unfrozen parameters between the ViT encoder fo (used for fine-tuning) and the MAE decoder JS (discarded post pre- pre-training) is crucial. Larger r' may improve the decoder's ability without benefiting the learned representations of f. This issue doesn't arise in V2, as the Dino-iBOT shared head is fully trained since it isn't provided by Oquab et al. (024)).
Table 14: Ablation on M-[1, L]- r64 on the validation set of MMoW-Sentinel. Here, the LoRA rank used for the ViT-L encoder is fixed at r = 64, while the rank r' for MAE decoder is varied.
B.5. Impact of ViT backbone size
We also test the impact of the ViT backbone for ExPLoRA, varying the
architecture for Di from ViT-B (86M, L = 12 layers, embedding
Top I Acc.
dimension 768), VIT-L (303M parameters, = 24 layers, embedding
Method
Arch.
Last 1/Last 4
dimension 1024), and ViT-G (1 100M parameters, L = 40 layers, embedding dimension 1280) for extended pre-training on fMoW-RGB. The ExPLoRA models we compare against are D- [22]]r6444 for ViT-B, D-[24] --66 for ViT-L, and D- [32] -r32 for ViT-G. We unfreeze the 12th, 24th, and 32nd layers for each of VTT-B, VIT-L, and ViT-G, picking these layers by extending the analysis from section 7 to VIT-B and ViT-G. We find that the 12th (last layer) for ViT-B and the 32nd (out of 40) layer for ViT-G output representations with low mean eigenvalues compared to other layers, thus presenting good candidates for unfreezing.
D2 [48
ViT-B
63.62/65.99
DinoV2 48
VIT-L
67.60/69.00
DinoV2 48
VIT-G
70.07/70.36
D-[12]-r64
VIT-B
74.72/75.11
D-[24]-r64
VIT-L
76.86/77.48
D-132)-r32
ViT-G
77.29/77.79
Table 15: Linear probing results on fMoW-RGB (validation), where we vary the size of the ViT encoder & from ViT-B, ViT-L,, and ViT-G. "Last 1/Last 4" refers to using the output representa- tion from just the last I the last 4 ViT layers.
In table 15 we see that as expected, ViT-G performs the best, but is only 0.31% better in top 1 accuracy compared to ViT-L. while using many more parameters. On the other hand, we see the highest impact for ExPLoRA on VIT-B, where the top 1 accuracy improves by t 9.21% over the original D2 ViT-B. These results further demonstrate the effectiveness and efficiency of ExPLoRA as a powerful technique to create unsupervised foundation models for new visual domains.
16

---

ExPLORA
B.6. Additional PEFT baselines for MAE
As continuation of table we include PEFT methods used on MAE weights, which generally underperform compared with Din For completeness, these results are in table 16
Pre- Train
Fine-Tune
Pre-Train GPU hours
Fine- Tune GPU hours
Model
Arch.
PEFT
Top Acc.
Params
Params
SatMAE
17
VIT-L
LoRA-r8
30
303.3M
0.8M
960
220
76.10
MAE
28
VIT-L
LoRA-r8
30
0.8M
220
76.21
MAE
28
VIT-L
DVPT-10
33
0.4M
500
72.35
MAE
28
VIT-L
GVPT-100
74
0.4M
500
70.86
5
MAE
28
VIT-L
SAAVP (50
1.IM
410
73.55
MAE
28
VIT-L
AdaLoRA-r8 777
1.2M
220
75.25
MAE
28
VIT-L
Adapter+ [61]
1.4M
220
74.10
8
MAE
28
VIT-L
Mona 73
7.1M
220
74.76
9
M- [1] -64
VIT-L
LoRA-r8 30
18.7M
0.8M
220
76.55
Table 16: MAE+PEFT results on IMoW-RGB validation split (table contd.). "Pre-train Params" and "Fine-tune Params" refer to trainable parameters of the ViT encoder required on the new domain (satellite images).
Pre-training ExPLoRA with MAE slightly more efficient than with D2 (i.e. 80 vs 100 GPU hours) due to two factors: the MAE encoder only operates visible image patches, and unlike Dino does not require maintaining a separate "teacher" model copy.
B.7. Results on UDA Benchmarks
As discussed in appendix A.2 while ExPLoRA a traditional unsupervised domain adaptation (UDA) method, it can serve an initialization for ViT-based UDA approaches. We demonstrate this compatibility below.
Method
Arch.
Init.
plane
beyel
bus
horse
knife
meyel
person
plant
sktbrd
train
truck
Mean
CDTrans
70
DEiT IN
97.1
90.5
82.4
77.5
96.6
96.1
93.6
88.6
97.9
86.9
90.3
62.8
88.4
PMTrans
80
ViT-B
IN-21k
98.9
93.7
84.5
73.3
99.0
98.0
96.2
67.8
94.2
98.4
96.6
49.0
87.5
TVT 71
VIT-B
IN-21k
97.1
92.9
85.3
66.4
97.1
97.1
89.3
75.5
95.0
94.7
94.5
55.1
86.7
TVT
VIT-B
IN-21k
95.8
85.8
81.9
68.4
95.9
96.2
91.9
70.3
93.8
93.7
92.9
48.5
84.6
TVT 71
VIT-B
D2
98.9
88.7
90.3
64.2
99.3
74.5
95.3
66.0
85.3
94.6
97.9
54.6
84.1
TVT 71
VIT-B
Ours
97.0
89.9
89.4
73.8
98.0
88.9
94.4
85.9
93.8
94.5
97.7
54.3
88.2
SSRT
VIT-B
IN-2Tk
98.9
87.6
89.1
84.8
98.3
98.7
96.3
81.1
94.9
97.9
94.5
43.1
88.8
SSRT
62
VIT-B
D2
99.2
88.1
89.9
85.4
98.4
98.9
97.6
84.8
96.2
97.1
95.4
48.3
89.9
SSRT
62
VIT-B
Ours
99.4
88.6
91.4
87.9
98.3
99.1
97.1
88.0
95.9
98.1
96.0
51.2
90.9
Table 17: Classification accuracy (%) on VisDA-2017 (validation). Results marked with * our reproduced results. Using ExPLoRA initialization improves UDA performance compared to standard ImageNet initialization.
VisDA2017 The VisDA2017 dataset (Peng al. 2017) contains 152,297 training and 55,388 validation images across 12 object classes. The dataset represents synthetic-to-Pe to-real domain shift: training images are synthetically rendered 3D models under various lighting conditions, while validation images are sourced from MS-COCO (Li et al. 2014).
Table 17 shows ExPLoRA's 's effectiveness when combined with TVT (Yang et al. 2023) and SSRT (Sun et al. 2022). two state-of-the-art UDA methods. Using ExPLoRA D- [-111-r00 -r64 (Dino V--iiiitiialleed ViT-B with last layer un unfrozen and LoRA-164 elsewhere) pre-trained on both synthetic and real domains, we outperform traditional ImageNet-211 (Deng et al., 2009) initialization by 1-3% while achieving more balanced per-class accuracy. Most notably, when using SSRT, we achieve new SoTA accuracy of 90.9%, surpassing both ImageNet and Di initialization by substantial margins (12.1 and 1.0% respectively). These results are particularly significant they show ExPLoRA's unsupervised initialization can outperform methods that rely on supervised ImageNet-21k pre-training. The benefits of ExPLoRA initialization are also clear with TVT. where performance rises to 88.2% to match recent SoTA methods (Xu et al. 2021 Zhu et al. [2033] a marked improvement over both its original results (12.1% 1%) and Di initialization (14.1%) These results demonstrate
that ExPLoRA serves as powerful initialization strategy for UDA, effectively bridging domain gaps while remaining computationally efficient.
17

---

ExPLORA
B.8. Attention Map Visualizations
To aid our analysis in section 7 we visualize attention scores for different ViT blocks across multiple models, including DinoV2, D- [L] -r64 (i.e. the last row of table the second and third rows of table MAE, SatMAE, and M- [L] -r64. These visualizations are shown in fig. f 3 different images from the validation set of IMoW-RGB. Since models are trained without registers, we truncate attention scores more than 5 standard deviations away from the mean, thus removing artifact attention scores with unusually high values on background patches (Darcet et al. 2024).
DINiO DESOLOAA
WIC
blook haok
Book
blok
E20 blok
Figure 9: Attention maps visualized from the validation set of fMoW-RGB. The models considered, from left to right, are: DinoV2, D-ExPLoRA- [L)]--(]]--6] Dino with blocks 22,23 unfrozen during extended pretraining, Dino with LoRA-r256 during extended pre-training, MAE, SaIMAE, and M-ExPLoRA- [1]-r64. We visualize the attention maps the beginning. middle, and end blocks of the ViT-I..
The visualizations in fig. further support the analysis in section For the Dino models, the attention scores of block 9-10 diffuse and spread around the central object of the image, with quite few border pixels highlighted. Conversely, the attention scores of the final layers are concentrated more towards the central object. These visualizations further suggest that the middle layers focus capturing local properties of the images such as texture, while the final layers capture global semantic information such as object-ness. Interestingly, the initial blocks for the Dino models display sparse attention patterns with spikes on seemingly random patches. This might suggest a form of caching to aid the computation of deeper layers that will extract local or global information.
For the MAE models, we see that the original MAE (pre-trained on natural images) seem to highlight more border pixels in the final layers of the VIT. Post extended pre-training with ExPLoRA, the final layers concentrate attention scores on the central object, more closely resembling the patterns of SatMAE (which was fully pre-trained on satellite images). ExPLoRA is thus able to successfully transfer knowledge from its initialized source-domain weights Ws to serve as foundation model Wi on the new target domain T.
C. Training Details
In this section, we describe hyperparameters and hardware configurations used for our models. We also include details datasets used for our experiments.
C.1. Pre-Training
We use the ViT-Large architecture for all experiments. Since image sizes vary, the shorter image size is resized to 224 while preserving aspect ratio, and then center crop is taken to yield images of size 3 224 x 224, representing the channels, height, and width. All pre-training is done on single NVIDA-RTX 6000 Ada GPU, or 4 NVIDIA-RTX A4000 GPUs on an academic GPU cluster.
ExPLoRA with MAE Most of the hyperparameters we use for M-ExPLoRA pre-training follow those in He et al. 2022)
Cong et al. 2022). We use an effective batch size of 1024 (through gradient accumulation), a base learning rate of 4.5 x 10 no weight decay, and a warmup and decaying cosine scheduler, with a warmup of epoch, and a total training time of 200 epochs. We use a masking ratio of 0.75 and we use the norm Lepppiilll loss flag for the MSE loss.
18

---

ExPLoRA
ExPLoRA with V2 Most of the hyperparameters for D-EXPLORA follow the defaults set by Oquab al. 2024
That is, local (small) crops are between 5%-32% of the original image and are resized to 98x98 pixels, and global (large) crops are greater than 32% of the image and resized to 224x224 pixels. We share the parameters of the Dino-iBOT linear head (3 layers), with a bottleneck dimension of 256, hidden dimension of 2048, and an output dimension of 65536, initialized from scratch. For Dino, we use Sinkhorn-Knopp (Caron et al. 2020) centering and Koleo (Delattre & Fournier 2017) regularization with a weight of 0.1. For iBOT, we use masking ratios between 0.1 and 0.5 to mask half of the samples in the batch. The teacher model uses an initial EMA rate of 0.994, with a cosine warmup to 1.000 by the end of training. The teacher warmup and final temperatures are 0.04 and 0.07. The linear Dino-iBOT head is frozen for the first 3k training iterations. We train with the AdamW optimizer (no weight decay), with base learning rate of 2 10-3 that is varied with linear warmup and cosine decay schedule. Training is completed within 200,000 iterations, with a batch size of 32 and with 32 gradient accumulation steps (equalling an effective batch size of 1024), and with an epoch length set to 1000.
C.2. PEFT Fine-Tuming
We fine-tune using 4 NVIDIA-RTX A4000 GPUs. We use a base learning rate of 10 a cosine scheduler with warmup for 1 epoch, and train for 120 epochs. We use an effective batch size of 256, making use of gradient accumulation if the GPU cannot fit the full batch size in memory.
For data augmentations, we only use the drop-path augmentation (Larsson et al., 2016) at a rate of 0.2, with no dropout, mixup, or cutmix. We note that the original LoRA configuration outperforms other PEFT techniques when paired with the drop-path regularization technique. For example, we find that BOFT does not pair well with drop-path, instead performing most effectively with custom multiplicative dropout technique (Liu et al. 2024).
We include the result with the best
hyperparameter configuration for each row in table 1
C.3. Linear Probing
We use a single NVIDIA-RTX A4000 GPU for linear probing. We adapt the code provided by Oquab (024g) for linear probing, with batch size of 256 and collection of different learning rates: 1 x 10 4,1x 10-3 5 x 10-3, x 10 2 x 10-2,5 5 x 10 x 10 10_']. We evaluate both probing on average pooled features as well as on the (CLS) token, and also use output features from just the last block, or the last 4 blocks. All numbers reported represent the best validation set accuracy from the best performing configuration.
et
al.
C.4. Multi-Spectral Images
We use the group-channel ViT-L architecture introduced in (Cong et al. 2022). We don't 't use DinoV2 since there is no such architecture for Dino pre-training. Input images are 13 x 98 x 98, representing 13 multi-spectral bands. We follow the configuration in Cong et al. (2022) of dropping bands B1, B9, B10, and use the same grouping strategy. When loading MAE weights to the ViT-L encoder, the patch embeddings do not match and so the patch embedding and group channel encodings trained from scratch. All other configuration details are the same for M-ExPLoRA A in appendix C.T, except that we use base learning rate of 4.5 x 10-4 for pre-training and train for 50 epochs (given the larger dataset size) on NVIDIA RTX A4000 GPUs for 80 hours.
Fine-tuning details are the same as in
C.5. Dataset Information
We include detailed information about the datasets used in our experiments. Table 18 shows the number of samples in the train and validation splits for all datasets used in this work.
Hyperparameter and training configuration details are the same as in appendix C.I if the images are RGB, and the same as in appendix if the images have more channels or are temporal.
Since each dataset contains varying number of training images, the number of ExPLoRA pre-training iterations should be adjusted accordingly. We include the recommended number of ExPLoRA pre-training iterations in table [8 Note, however, that this number can be varied depending on other hyperparameters such as batch size, learning rate, LoRA rank, and number of unfrozen blocks.
19

---

ExPLORA
Dataset
#Train
#Validation
ExPL.oRA Iters
Domain
MMoWRGBB
363.6k
53.0k
200k
Remote Sensing
MMo--Sentiel
712.9k
84.9k
80k
Remote Sensing (Multi-spectral)
(MoW-Tempors
83,4k
14.2k
80k
Remote Sensing (Temporal)
SpaceNet VI
6.0k
1.5k
10k
Remote Sensing
Resisc-45
18.9k
6.3k
Remote Sensing
NAIP
244.4k
55.5k
200k
Remote Sensing
EuroSAT
16.2k
5.4k
Remote Sensing (Multi-speettrl)
Camelyon17
302.4k
33.6k
200k
Medical Imaging
iWildcam
129.8k
7.3k
150k
Wildlife
GheaalWheat
2.9k
0.4k
80k
Agricultural
VisDA2017
152.3k
55.4k
200k
Synthetic-to-PP
Table 18: Dataset splits and sizes used in our experiments, well suggested number of ExPLoRA pre-training iterations.
In general, we find that ExPLoRA demonstrates the largest performance gains on larger datasets where sufficient diversity prevents overfitting during extended pre-training, For datasets with fewer than 50k training samples (c.g.. SpaceNet VI, GlobalWheat), the improvements from ExPLoRA more modest, the limited data diversity can lead overfitting with extensive pre-training. Conversely, ExPLoRA works exceptionally well on larger datasets such fMoW-(RGB, Sentinel, Temporal), Camelyon17, and VisDA2001, where 100k-200k pre-training iterations provide substantial performance improvements while maintaining computational efficiency compared to full domain-specific pre-training.
The licenses for all datasets are included in the footnotes: fMoW[]] Sentinel-2
EuroS
SpaceNe
Camelyon 1
iWildCan O GlobalWheat
D. Environmental Impact
Following (Cong et al. 2022), we compare the carbon footprint of pre-training using ExPLoRA with domain-specing
solutions such as SatMAE. We use the carbon footprint calculator proposed by Lacoste et al. (2019). Our results are in
table 14
fMoW-RGB
fMo Mon-nentt
fMo (MoW-Temporal
Method
GPU hours
kg CO2 eq.
GPU hours
kg CO2 eq
GPU hours
kg CO2 eq.
SatMAE
768
109.44
576
82.08
768
109.44
ExPLoRA
96
12.44
320
19.35
100
12.96
Table 19: The estimated carbon footprint of pre-training these datasets
Since we initialize with pre-trained weights on natural image domains, ExPLoRA is much less environmentally impactful while achieving similar or higher levels of performance. We achieve 4x-8x reduction total carbon emitted for each of the large pre-training satellite image datasets considered in table 19
'Moo license:
"Sentinel-2 license:
Sentinel_nn Sontinel__aat_Teee Terrtt_T_rm___e
ht http:///h//////eaaaaa.o.o..oo///
'SpaceNet vl license: httpa///ea////eeee////o///oeee/a
license htaa://www.oogoao..../e/../.totn
WWiilCam http://ccnnn..ooooo////////a////
"GlohalWhen hicmne http://wwww...r..eeeeeeeeee/e//e censes/Mi
'Note: while the SatMAE paper reported 768 GPU hours for pre-training MoW-RGB and 576 GPU hours MMoW-Sentinel,
measurements Tables 1-5 main text have a revised numher to ensure consistency of hardware platforms across comparisons.
20


Reviews:
[{"review_id": "0saxuEf2Q6", "reviewer": "Reviewer_W3i1", "summary": "The authors aim to transfer knowledge from large pre-trained vision models to new domains efficiently, and address different downstream tasks. So, given a set of downstream tasks on a new domain, the straightforward approach is to either pre-train from scratch a large model on this new domain and then fine-tune it on the downstream tasks, or to directly fine-tune an existing large model from a different domain on the downstream tasks at hand.\n\nInstead of these approaches, the authors propose Extended Pretraining with LoRA (ExPLoRA), which consists of the following 2 steps. First, a large vision model pre-trained on natural images, like DinoV2 or MAE, is further pre-trained on a new domain, e.g., satellite images. In this pre-training stage there is no use of labels, and the optimization objective is the same as the initial pre-training on the natural images. In this stage, the model in not fully fine-tuned, instead, 1 or 2 layers are fully unfrozen, and LoRA is used for the rest of the layers. In the second step, after the extended pre-training, the model undergoes supervised fine-tuning with a PEFT method on downstream tasks, e.g., classification of satellite images. The core idea is that extended pre-training can be considerably more efficient compared to training from scratch on a new domain, while performance is not compromised, and can even improve, especially compared to directly fine-tuning on downstream tasks.\n\nThe authors conduct multiple experiments to validate their method and explore its behavior. They use DinoV2 and MAE as pre-trained models on natural images, and they show that in most cases, ExPLoRA outperforms both models trained from scratch on new domains, as well as models directly fine-tuned on downstream tasks. Their primary experiments are on satellite images, both RGB and spectral, but they show similar results on different domains as well, e.g., WiLDS datasets. Also, they show promising results across different downstream tasks, e.g., image classification, segmentation, and object detection. In addition, they conduct ablation studies to justify their design choices and explore the features learned by ExPLoRA. Finally, additional experiments and analysis of the models behavior is provided in the supplementary material, e.g., the effect of the model size, or that of the duration of extended pre-training.\n\n## update after rebuttal\nThe authors addressed the main points of my review, so I increased my score from 3 to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and insightful feedback. We appreciate your recognition of ExPLoRA’s novelty in combining fundamental ideas and its strong empirical performance across datasets.\n\n**Q: Discussion of required compute of ExPLoRA vs fine-tuning baselines**  \nWe agree that more details on compute requirements is needed, which we have added to the main text. Upon reviewing the feedback, we'd like to clarify:\n\n* ExPLoRA is a pre-training method and is meant to provide an efficient alternative to domain-specific _full or continual pre-training_ (eg: SatMAE, ScaleMAE, GFM, etc.). These methods require at least 8x more compute than ExPLoRA pre-training and achieve lower or similar performance to ExPLoRA across different domain gaps (eg: RGB, multi-spectral, temporal data).\n* Directly comparing compute requirements for fine-tuning vs extended pre-training + fine-tuning paints an incomplete picture. The ExPLoRA pre-training phase is functionally different from fine-tuning a natural-image pre-trained model. A pre-trained checkpoint serves multiple purposes- feature extraction, linear-probing, fine-tuning on various downstream tasks etc. In contrast, compute used for supervised fine-tuning yields a model usually suited for just one task (eqs 2, 3). Pre-training compute is amortized across multiple downstream tasks, and would be double counted if included in the fine-tuning budgets for each specific task.\n* We realize now that Figure 7 in section B.2 is incomplete. It only compares the compute for fine-tuning a natural image model (eg: DinoV2) vs. pre-training + fine-tuning for ExPLoRA, for the fMoW-RGB task. Crucially, the pre-training + fine-tuning compute required for full domain-specific pre-training (eg: SatMAE, ScaleMAE) is missing. Further, for different domains (eg: multi-spectral satellite images), the gap between fine-tuning a natural-image model (eg: MAE) and domain-specific pre-training is much larger than for fMoW-RGB (i.e. 6%, from Table 4). Lastly, to reiterate, a directly fine-tuned DinoV2 on fMoW RGB is less flexible than an ExPLoRA pre-trained model, as the latter can be used as any _pre-trained model_- for feature extraction, probing, or further task-specific fine-tuning.\n\nWe've expanded Tables 1 and 4 with compute details [linked here](https://imgur.com/a/iMjgOHv). Note that:\n* We only need to allocate 100 GPU hours to ExPLoRA pre-training to achieve the reported gains. For fairness, we use ~220 GPU hours for fine-tuning across models.\n* Providing extra 100 GPU hours to fine-tuning non-ExPLoRA baselines (for an equitable total compute comparison) doesn't improve top-1 accuracy.\n* VPT techniques significantly increase fine-tuning compute by extending token sequence length.\n\n**Q: The DinoV2 baselines reach close to their peak performance in 30 GPU hours while ExPLoRA requires 420 GPU hours**  \nThis is not a direct comparison, since Figure 7 provides different fine-tuning variants. The red curve is DinoV2 fine-tuned with LoRA-r8, which we also use for ExPLoRA models. The orange and purple curves have blocks unfrozen with LoRA-r64, representing a similar parameter budget to ExPLoRA pre-training. The orange curve reaches peak performance at ~260 GPU hours, by which point all ExPLoRA models have surpassed it.\n\n**Q: Experimental results on larger backbone sizes beyond ViT-L**  \nThank you for the suggestion. As you mention, we compare with ViT-B and ViT-G in Table 15 (appendix B.5). ViT-G experiments are more expensive (>3x parameters vs ViT-L), especially on academic compute budgets. Most prior work uses ViT-L as their backbone, so we maintain this architecture for fair comparison in our main experiments.\n\n**Q: More detailed descriptions of datasets**  \nThank you- we'll include further dataset details in the appendix. For fMoW:\n* fMoW-RGB: 363k training, 53k test data points\n* fMoW-Sentinel: 713k training, 85k test data points\n\n**Q: lines 760-761, parameter budget clarification**  \nYou're right - it should be 18.7M.\n\n**Q: ln 765: Is it 320 GPU hours or 420?**  \nWithin 320 total hours (pre-training + fine-tuning), ExPLoRA achieves >1% improvement over the DinoV2 baseline. The 420 hour limit was added as a buffer to demonstrate earlier convergence.\n\n**Q: Eq.6 $\\Theta(r)$ notation**  \nThank you- the notation should be $\\Theta$ to refer to an unconstrained parameter space (regular full-rank weights).\n\n**Q: iWildcam linear probing result**  \nThe unfrozen ViT block likely overfit to the training data due to the small domain gap. This effect disappears with PEFT (showing improvement over DinoV2) since all ViT attention layers' Q,V matrices are tuned with LoRA.\n\n**Q: Row number clarifications and \"blk\" notation in figures 3-6**  \nThank you! We've corrected these mistakes in our manuscript.\n\n---\nThank you for your thorough feedback which has improved our paper. Please let us know if you have follow up questions. If your main concerns are resolved, we kindly request that you reconsider your score.", "claims_and_evidence": "- Performance:\n    - The authors compare against the 2 main settings they aim to improve on, which is training from scratch on a new domain, and fine-tuning a pre-trained model from a different domain. I think the authors demonstrate clearly that ExPLoRA can lead to performance benefits, since it outperforms or achieves comparable performance across multiple datasets of different size and content, downstream tasks, heads (linear probe and original heads), and backbones (DinoV2 and MAE).\n- Efficiency:\n    - It is clear that ExPLoRA uses much fewer parameters compared to training a model from scratch, since it unfreezes 1 or 2 layers, and commonly uses LoRA with up to $r=64$ for the rest of the layers. However, the required compute is not included in any of the Tables or Figures. For example, in Table 1, D-[L]-r64 with LoRA-r8 PEFT outperforms DinoV2 with LoRA-r8 by 1.2%, but at what cost in terms of compute? If I am not mistaken, in this case the ExPLoRA model and the baseline undergo the same fine-tuning on the downstream task, but the ExPLoRA model requires an additional pre-training phase, which according to Section C.1. in the Appendix, it corresponds to 200,000 additional training iterations. I understand that it may worth conducting the pre-training if performance is a priority, or if the ExPLoRA model will be used for multiple downstream tasks, so, I want to clarify that I don’t think ExPLoRA should have comparable training GPU hours with all baselines, but I think this should be clearly mentioned in the experiments, since efficiency is a primary motivation for this work.\n    - In addition, in Section B.2. in the Appendix, we can see that the DinoV2 baselines reach close to their peak performance after as few as approximately 30 GPU hours, while the best ExPLoRA model needs 420 GPU hours to reach its peak performance. So, I think the authors should have more experiments like this included in the main text (not just in the Appendix), and explicitly discuss compute requirements.", "methods_and_evaluation_criteria": "I think the authors use appropriate baselines, datasets and ablation studies. As I mentioned in the previous section, what I think is missing, is an elaborate discussion about compute.", "theoretical_claims": "There aren’t any theoretical claims or proofs.", "experimental_designs_or_analyses": "I think the experiments are well designed, and as I mentioned before, the authors do extensive evaluations, covering diverse scenarios. One remark I have is that in most experiments, the backbone is ViT-L, which raises questions about whether the presented results generalize to bigger scales. To their credit, the authors conduct experiments with more diverse backbones in Section B.5. in the Appendix, where they experiment with ViT-G, so, given the importance of model scale for modern applications, I would suggest to include and/or discuss such experiments in the main text.", "relation_to_broader_scientific_literature": "The proposed method directly combines in a new way 2 fundamental ideas in the literature, unsupervised pre-training (e.g., DinoV2), and PEFT (e.g., LoRA). The main contribution is that PEFT should not be seen solely as a method to address downstream tasks when a large pre-trained model is available, but it is beneficial to first apply PEFT to extend pre-training, and then re-apply PEFT on downstream tasks.", "essential_references_not_discussed": "Nothing to add.", "other_strengths_and_weaknesses": "I would like to mention that the manuscript is very well written and easy to follow. Also, the authors motivate well the importance of satellite images, and why they focus on this image domain.", "other_comments_or_suggestions": "- Eq. 6: I think notation $\\Theta (r)$ is a bit confusing based on the explanation provided “where $r$ controls the fraction of trainable weights”, and the fact that $r \\in [0, \\infty]$. $r$ is usually used to represent the LoRA rank, which is the way it is used in Eq. 5 as well.\n- ln 321, col 1: I think ExPLoRA-L-r8 is in row 11 instead of 10.\n- Table 3 is discussed in Section 6.1.2., and there are multiple references to different rows, e.g., rows 11-13. It would be easier to follow such references if the rows of Table 3 were indexed.\n- ln 314, col 2: I think it should be “row 3” for LoRA-tuned MAE instead of “row 4”. Similarly, in ln 315, should be “row 7” instead of “row 6”.\n- In the iWildcam experiment (paragraph that starts in ln 375, col 2), the authors discuss why the linear probing performance of ExPLoRA suffers in comparison to DinoV2, emphasizing on the small domain gap. However, it is not clear to me why this is a valid hypothesis. In particular, this could explain ExPLoRA and DinoV2 having a small difference in performance, and not DinoV2 clearly outperforming ExPLoRA. I think it would be useful to clarify this argument further.\n- In Fig. 3-6, the legend has names “D-ExPLoRA-blk23r64” and “M-ExPLoRA-blk23r64”, I guess “blk23” corresponds to the unfrozen block, but this notation is not used anywhere else in the paper; the same naming is used in Fig. 9 as well.", "questions_for_authors": "I don’t have additional questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yQc1AuLdV2", "reviewer": "Reviewer_Fy1r", "summary": "The paper proposes ExPLoRA, a parameter-efficient way to extend pre-training of a large vision transformer from its original domain (e.g. natural images) to a new domain (e.g satellite imagery). ExPLoRA accomplishes this by unfreezing 1-2 transformer blocks for full training and applying low-rank (LoRA) updates on the rest. After this unsupervised adaptation, the model can then be fine-tuned using LoRA on labeled tasks and achieves results that match or surpass fully trained domain-specific models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and for recognizing our thorough experimental validation across baselines and datasets and for providing useful insights into in-domain parameter-efficient pre-training. You may find responses to your questions below:\n\n**Q: Further justification for which blocks to unfreeze for ExPLoRA**  \nWe agree that block selection is an important component of our method. In section 7, we systematically analyze the properties of patch embeddings from different transformer blocks:\n\n* Blocks that output patch embeddings with low mean/variance eigenvalues inversely correlate with higher linear probing accuracy on predicting patch position\n* Linear probing accuracy for image class peaks in final layers (which also have low mean/variance eigenvalues), while patch position accuracy peaks in middle layers\n* The eigenvalue patterns and classification accuracy suggest unfreezing block L or L-1 would have highest impact for extended pre-training, which is confirmed in section 6.1.2\n* ExPLoRA results in lower mean/variance eigenvalues in the unfrozen block's output, and higher classification and localization accuracies across all ViT layers\n* Linear probing performance in section 6.1.2 follows: block 23 > block 22 > block 9 > block 1, matching exactly the trend in the mean eigenvalue plot (figure 3)\n\nUpon your suggestion, we've included additional baselines with blocks [L-1,L] and [1,L-1,L] unfrozen in the expanded version of table 3 [linked here](https://imgur.com/a/tWdN4xU). We find that increasing unfrozen blocks with ExPLoRA indeed improves linear probing accuracy, where our highest performing D-ExPLoRA model with blocks [1, L-1, L] unfrozen and LoRA-r64 on the rest achieves 78.04% linear probing accuracy, a 9% improvement over prior SoTA.\n\nNote that while more unfrozen blocks improve performance, each adds to the parameter count. We aim to demonstrate strong results with a limited parameter budget. Our analysis in section 7 provides guidelines for selecting additional blocks to unfreeze when higher parameter budgets are available.\n\nSimilar trends as in figures 3-7 held across different datasets, guiding our design choices in other experiments. \n\n**Q: Provide a DinoV2 full pre-training baseline for linear probing for fMoW-RGB**  \nWe agree that this comparison would be useful, however, we want to note that no such baseline in prior literature exists. The SatMAE/ScaleMAE/CrossScaleMAE models were pre-trained on fMoW-RGB, with which we provide direct comparisons to demonstrate that our efficient in-domain pre-training vastly outperforms existing SoTA pre-training methods in the literature by 8% while using 8x-10x less compute.\n\nUpon your suggestion, we have trained a full DinoV2 baseline on fMoW-RGB in the [revised Table 3 linked above](https://imgur.com/a/tWdN4xU). This required ~1200 GPU hours, more than 12x the compute required for ExPLoRA and failed to match its linear probing performance. This likely suggests that much more compute and time is required to pre-train a DinoV2 on fMoW-RGB.\n\n**Q: The paper expands on existing methods**  \nWith ExPLoRA, we demonstrate a highly effective alternative to expensive full pre training for new image domains. While it combines existing methods, we would like to cite peer conference [NeurIPS guidelines](https://neurips.cc/Conferences/2024/ReviewerGuidelines), which mentions that demonstrating the effectiveness of combining existing techniques can provide substantial research value.\n\n---\nThank you again for your time in reviewing our work. We look forward to addressing any follow-up questions you may have. If your concerns are addressed, we kindly ask that you reconsider your score.", "claims_and_evidence": "In general the paper is sound and the claims are supported by enough evidence. However there are two claims to which I would like to draw attention to:\n“As seen, un- freezing an extra block consumes almost double the number of parameters, but fails to yield the same improvement in performance ↓ 0.34%. Thus, simply increasing the number of unfrozen blocks will likely improve performance, but will not do so as effectively as ExPLoRA, and will also significantly and sharply decrease the parameter-efficiency”\nIn section 6.1.2, the authors provide an ablation study where they attempt to unfreeze blocks at different positions (primarily L and/or L-1). While the table supports this claim, it does not provide sufficient evidence to justify choosing only one block to unfreeze. This table is crucial, as their method depends on having at least one block unfrozen to maintain performance. Therefore, more extensive experimentation with various block positions and different numbers of unfrozen blocks would be valuable, as well as replicating these findings across additional datasets.\n“Our results demonstrate an improvement of over ↑8.2% in top 1 average accuracy over prior SoTA methods”\nIn section 6.1.1, the authors make this claim regarding results from satellite images. This is among the paper's strongest claims (also highlighted in the abstract) and is presented in Table 2. However, the comparison appears unfair because it contrasts Dinov2 pretrained weights without domain adaptation against ExPLoRA, which includes domain adaptation. While insightful, this comparison does not clearly illustrate the main use of their method, which aims at efficient in-domain pretraining.\nAdditionally, I notice the absence of a Dinov2 model fully pretrained in-domain in Table 1. Including such a model would provide a more equitable comparison, as MAE baselines cannot directly be compared with Dinov2.", "methods_and_evaluation_criteria": "They perform extensive analysis in a specific use case which is satellite images. Besides, they report results on 3 of WiLDS datasets with additional 2 datasets in the appendix.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "Experimental designs and analyses are overall adequate. They compare with many different PEFT methods, showing the superiority of their approach in this regard.\nSee section “Claims And Evidence” for missing experiments/baselines.", "relation_to_broader_scientific_literature": "This paper builds upon MAE, DinoV2 and LoRA. They cite appropriately all the methods that they use and state clearly their contributions.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\nThe paper is clear, well-organized, and thorough. It effectively examines the use of LoRA for in-domain pretraining to improve results, offering a straightforward and useful method. The study includes detailed comparisons with different PEFT techniques and covers multiple varied datasets, greatly enhancing the overall analysis.\n\nWeaknesses:\nWhile providing useful insights, the paper's novelty is somewhat limited as it mainly expands on existing methods. Additionally, it is well-known that in-domain pretraining boosts fine-tuning performance. Hence, the paper could focus more on showing efficient methods for achieving this rather than simply restating the performance benefits (see the \"Claims and Evidence\" section for more details).", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jCRGfyQ7nQ", "reviewer": "Reviewer_83s5", "summary": "This paper proposes a continual pre-training method with a parameter-efficient fine-tuning (PEFT) module such as LoRA (Hu et al. 2021) to improve the adaptability of visual foundation models on specific domains. By inserting and training the PEFT module inside the general-domain pre-trained backbone model with the same learning objective that the backbone model was trained, the proposed ExPLoRA induces meaningful initialization for adaptation with labeled data, resulting in somewhat better classification accuracy on downstream tasks compared with the full fine-tuning approach.\n\n\n\n\n\n\n\nHu et al. 2021, LoRA: Low-Rank Adaptation of Large Language Models", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We appreciate your recognition of our extensive, well-designed experimental validation and for ExPLoRA’s improved performance over baselines. Here are our responses:\n\n**Q: Can you provide qualitative analysis that illustrates the difference between pre-trained vs ExPLoRA pre-trained embedding spaces?**  \nSection 7 contains our analysis of patch embeddings output by ViT blocks across different models (ExPLoRA, DinoV2, MAE, SatMAE), revealing important quantitative differences. Please also see our response to reviewer Fy1r [linked here](https://openreview.net/forum?id=OtxLhobhwb&noteId=JMCCQ90U3c).\n\nFigure 9 (appendix B.8) also qualitatively analyzes attention maps across different ViT blocks of these models. ExPLoRA concentrates attention more tightly on central objects, especially in final layers. This correlates with lower mean eigenvalues and higher positional/classification accuracies from linear probing.\n\n**Q: Include discussion of test-time adaptation (TTA) methods.**  \nThank you for mentioning these methods. We have updated our related work section to include these references which will be present in our camera-ready revision. Our work has key differences with TTA methods:\n\n* TTA methods make a critical assumption: the label space Y is shared between the source and target domains. This makes [1,2,3] incompatible with unsupervised pre-trained backbones when domains have different label sets (common in many settings, including ours). Unsupervised pre-training techniques like DinoV2 or ExPLoRA don't assume specific downstream label sets, requiring some supervised adaptation (linear-probing/PEFT) to parameterize $p^{(\\tau)}_T(y|x)$ for downstream task $\\tau$ in target domain $T$. TTA methods could then be applied on top of ExPLoRA backbones just as with any ViT. This means that ExPLoRA is complementary with TTA methods rather than a competing method.\n* [3,4] are tailored for CNNs, not applicable to our work with ViTs, which consistently outperform CNNs on our datasets (eg: fMoW-RGB, fMoW-Sentinel, temporal images etc. as verified in prior work).\n* [2] uses visual-prompt tuning (VPT) for ViTs, but lacks a public codebase. We include multiple recent VPT baselines in Table 1 (VPT [5], GVPT [6], SA2VP [7]). ExPLoRA outperforms all by ~2%, including a pre-training baseline with VPT in Table 3.\n\n**Q: ExPLoRA requires a non-trivial amount of additional training compared to fine-tuning a natural-image pre-trained model for an improvement in performance (Figure 7)**  \nWe would like to refer you to our response to reviewer W3i1, [linked here](https://openreview.net/forum?id=OtxLhobhwb&noteId=AapZp7BqgM). \n\nTo summarize, we realize that our plot in figure 7 paints an incomplete picture. ExPLoRA is a pre-training method, providing an alternative to domain-specific _full pre-training_. While extended pre-training + fine-tuning requires more GPU hours than directly fine-tuning, the extended pre-training phase is amortized across multiple downstream tasks, since the same ExPLoRA model can be re-used for initialization. Further, it can be used for methods such as feature extraction, linear probing etc. that a task-specific fine-tuned model cannot do. For linear probing, ExPLoRA outperforms DinoV2/prior SoTA models by >8%. This difference is not captured via figure 7, since that only compares with fine-tuning.\n\nThus, the fairer comparison is with domain-specific full-pretraining. Here, ExPLoRA requires **8x-10x less compute, 16x fewer parameters** and achieves similar or higher performance. Our expanded [Table 1](https://imgur.com/a/iMjgOHv) demonstrates that ExPLoRA is vastly more efficient than other pre-training methods.\n\nFigure 7 shows only fMoW-RGB performance. On fMoW-Sentinel (larger domain shift to multi-spectral data), full-finetuning a natural-image baseline shows a 9% performance gap with ExPLoRA (row 1, Table 4). Combined, these are significant results since ExPLoRA achieve/surpass fully pre-trained baselines using ~8-10x less compute and 10-16x fewer parameters.\n\n---\n\nThank you for your feedback which has strengthened our work. If your concerns are addressed, we kindly ask that you reconsider your score.\n\n---\nReferences:  \n[1] Tent: Fully Test-time Adaptation by Entropy Minimization, _ICLR 2021_.  \n[2] Visual Prompt Tuning for Test-Time Domain Adaptation, _arxiv 2210.04831_.  \n[3] DomainAdaptor: A Novel Approach to Test-time Adaptation, _ICCV 2023_.  \n[4] Convolutional Visual Prompt for Robust Visual Perception, _NeurIPS 2023_.  \n[5] Visual prompt tuning. _ECCV 2022_.  \n[6] Improving visual prompt tuning for self-supervised vision transformers. _ICML 2023_.  \n[7] SA²VP: Spatially Aligned-and-Adapted Visual Prompt. _AAAI 2024_.", "claims_and_evidence": "The authors claim extended pre-training with LoRA (and a few unfreeze model blocks) via self-supervised learning (SSL) objectives can induce better initialization for specific domain datasets. This claim was well-validated through the conventional evaluation protocol of SSL, i.e., evaluation with linear probing and fine-tuning performance on detection and calibration tasks across diverse domains. \n\n**However**, there is a point that can be further improved. \n* The author explains the rationale of continual pertaining with LoRA based on weight decomposition, i.e., the desired target weight vector $W_{T}^{(\\tau)}$ is the summation of pre-trained weight vector $W_{S}$ on source domain (general knowledge), a vector contains general knowledge on the target domain, $\\Delta_{T}$, and the vector represent the task-specific knowledge on target domain $\\Delta^{(\\tau)}$.\n* If the author can provide some qualitative analysis that illustrates the difference (maybe in the embedding space) between pre-trained weight and ExPLoRA-extended pre-trained weight, this design motivation of ExPLoRA will become more convincing.", "methods_and_evaluation_criteria": "The proposed method is reasonable to address the stated problem -- developing domain adaptable foundation models, and the adopted metric follows the standard of literature.", "theoretical_claims": "There are no formal theorems.", "experimental_designs_or_analyses": "Overall the amount and range of experiments are extensive and well-design, but it would be better if the authors could (1) add some qualitative analyses such as embedding visualization, and baseline for the test-time adaptation method\n\n- Wang et al. 2020, Tent: Fully Test-time Adaptation by Entropy Minimization\n- Zhang et al. 2023, DomainAdaptor: A Novel Approach to Test-time Adaptation", "relation_to_broader_scientific_literature": "The authors claim that the proposed paradigm can indeed contribute to developing the domain-adaptable foundation models by reducing the development cost compared with the from-scratch training approach. However, as shown in Figure 7 of the appendix, the proposed method requires a non-trivial amount of additional training compared with the vanilla pre-trained model for the marginal improvement of downstream performance. Thus the contribution seems not significant.", "essential_references_not_discussed": "I would recommend the author mention test-time adaptation (TTA) approaches (especially the PEFT-based TTA method) somewhere in the main body of the paper. In terms of the goal -- enabling efficient adaptation of backbone model to target domain -- and design of the learning algorithm, TTA approaches are well-aligned with the position of this work.\n\n- Wang et al. 2020, Tent: Fully Test-time Adaptation by Entropy Minimization\n- Gao et al. 2022, VISUAL PROMPT TUNING FOR TEST-TIME DOMAIN ADAPTATION\n- Zhang et al. 2023, DomainAdaptor: A Novel Approach to Test-time Adaptation\n- Tsai et al. 2023, Convolutional Visual Prompt for Robust Visual Perception", "other_strengths_and_weaknesses": "Although the proposed method consistently shows strong results across diverse downstream tasks and domains (`strength`), it requires a non-trivial amount of additional training hours, e.g., 200 hours to achieve marginally improved downstream performance (`weaknesses`).", "other_comments_or_suggestions": "See the above reviews.\n\n# Post-rebuttal\n> I appreciate the authors' professional rebuttal. For now, I will adhere to my rating because I do not agree with authors' statements on the benefits of the proposed method from the perspective of computational efficiency <-> accuracy improvement trade-off. I will go through other rebuttals as well to reconsider my recommendation (if necessary) through the remaining discussion period.", "questions_for_authors": "See the above reviews.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JwpsbMgf3L", "reviewer": "Reviewer_XYuv", "summary": "The authors show that continued pre-training (with PEFT) on the target domains before supervised fine tuning is an effective methodolofy for adapting natural-image foundation models to non-natural image target tasks. They show particular gains on RGB tasks, but also show some benefit on multi-spectral imagery.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and recognition of the empirical performance and novelty of using parameter-efficency for pre-training in ExPLoRA. Here are our responses:\n\n**Q: Is PEFT needed in either pre-training or fine-tuning beyond memory savings?**  \nOur primary motivation for parameter-efficient techniques is _efficiency_:\n* Reduced memory footprint provides significant compute savings with larger batch sizes and/or fewer GPUs, and since gradients for <10% of parameters need to be stored and propagated. This makes ExPLoRA much faster than full pre-training (see [revised Table 1](https://imgur.com/a/iMjgOHv) and [response to reviewer W3i1](https://openreview.net/forum?id=OtxLhobhwb&noteId=AapZp7BqgM))\n* Memory savings enable using models such as ViT-L or ViT-G, otherwise inaccessible with constrained GPU budgets\n* While not our main motivation, constraining the parameter space mitigates catastrophic forgetting from the natural image domain. ExPLoRA outperforms fine-tuning from MAE/DinoV2 or domain-specific SatMAE models by >1% in Table 1, and full continual pre-training methods (eg: GFM) by 5-7%. For linear probing, this gap widens to 8%, suggesting benefits from both original and extended pre-training.\n\nWe've included a baseline for fully pre-training a ViT-L from scratch on fMoW-RGB with DinoV2 objective in our [revised Table 3](https://imgur.com/a/tWdN4xU). This shows full pre-training cannot match ExPLoRA while requiring >10x compute and >16x parameters.\n\nLastly, our parameter-efficient pre-training is orthogonal to fine-tuning method choice. ExPLoRA preserves the ViT architecture, enabling any fine-tuning approach. While we primarily use PEFT for efficiency, ExPLoRA also outperforms fully fine-tuned SatMAE and ScaleMAE models by 1.4% (Table 1).\n\n**Q: Are there baselines that fully fine-tune natural-image pre-trained models on the domain of interest?**  \nWe use PEFT primarily for efficiency. At your request, we've included fully fine-tuning DinoV2 and ExPLoRA models on fMoW-RGB alongside the SatMAE paper's fully fine-tuned MAE result in our [expanded Table 1](https://imgur.com/a/iMjgOHv).\n\nFull fine-tuning is viable with sufficient resources. ExPLoRA models benefit from it just as well as other baselines and outperform them. However, LoRA-r8 PEFT is much cheaper and equally effective. Table 4 includes a comparison where MAE is directly fully fine-tuned on fMoW-Sentinel, showing poor performance (>9% gap) due to the large domain shift.\n\nWe reiterate that we don't aim to replace full or PEFT fine-tuning, both complementary to ExPLoRA. Instead, we provide an efficient alternative to _full, domain-specific pre-training_.\n\n**Q: Relation to “self-supervised pre-training improves...” (HPT)**  \nWe will update our paper to include this reference. Key differences between ExPLoRA and HPT:\n\n* HPT uses two full backbone pre-training phases on different datasets. ExPLoRA uses a single efficient extended pre-training phase that generalizes to multiple downstream tasks.\n* While HPT offers tuning only batch norm layers in the second phase, it centers around full-rank pre-training, like GFM. ExPLoRA uses parameter-efficient techniques that **reduce compute by >10x and parameters by >16x** while achieving similar or higher performance.\n* HPT studies MoCo with ResNet-50. ExPLoRA addresses ViTs with different SSL objectives (DinoV2, MAE), demonstrating broader applicability.\n\nExPLoRA goes beyond \"modernizing HPT\" by demonstrating parameter-efficient extended pre-training's value via strong performance at reduced costs across diverse datasets, SSL objectives, and tasks.\n\n**Q: Relation to “Finetune like you pretrain...” (FLYP)**  \nWe agree this reference is also valuable. As above, there are important differences with ExPLoRA:\n* FLYP is a fine-tuning technique while ExPLoRA is a pre-training technique\n* FLYP focuses on contrastive SSL methods with ResNet. ExPLoRA is compatible with any ViT SSL objective (as we show with DinoV2, MAE).\n\nFLYP does further justify continuing pre-training with the same loss function (eq. 5). We'll update our final manuscript to include this reference.\n\n**Q: Show additional evidence in Table 4 ,5 for D -[L}-r32 and M-[L]-r32 in Tables 7-9.**  \nWe have now included MAE results in tables 7-9, [linked here](https://imgur.com/a/lx4TaLZ), which underperforms our SoTA Dino-ExPLoRA.\n\nFor tables 4-5, we use MAE SSL from SatMAE. There's no DinoV2 modification for multi-spectral/temporal data in prior work. Creating one would require modifying the ViT and Dino SSL mechanism for non-RGB temporal/multi-spectral sequences, which is out of scope. Our goal is to use the SatMAE SSL architecture with MAE weights, showing ExPLoRA matches/outperforms full pre-training (0.67%) and vastly outperforms full MAE fine-tuning on multi-spectral data by 8.54% (Table 2).\n\n---\nWe hope these answers address your questions. If your main concerns are resolved, we kindly request you reconsider your score.", "claims_and_evidence": "* Claim: Continued pre-training on target domain, with peft, unlocks peft finetuning for specialized tasks. \nThis is supported in Table 1 and other results tables. However, the question of whether PEFT is needed in either stage is useful (beyond memory savings) is not answered.", "methods_and_evaluation_criteria": "Broadly, the method and evaluations both make sense.", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "Reading the paper, the experimental design mostly makes sense, with a few caveats:\n\nTable 2 shows results for both Dinov2 and MAE, but Table 4- 9 don't always show both. Does it still work there? \n\nThere is no comparison that I can find for a model which has continued pre-training and finetuning on the same dataset with full finetuning (not ScaleMAE or SatMae), so the performance charachteristics of using PEFT are not well developed.", "relation_to_broader_scientific_literature": "This paper builds on the idea of \"finetuning like you pretrain\"[1] and \"Self-supervised pretraining improves self-supervised pretraining.\"[2], and adds a PEFT component. They verify this works for adapting natural image models to OOD data domains like sattelite imagery. \n\n\n[1] Goyal, Sachin, et al. \"Finetune like you pretrain: Improved finetuning of zero-shot vision models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n[2].    Reed, Colorado J., et al. \"Self-supervised pretraining improves self-supervised pretraining.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.", "essential_references_not_discussed": "[1] and [2] are relevant and not discussed. \n\n[1] Goyal, Sachin, et al. \"Finetune like you pretrain: Improved finetuning of zero-shot vision models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n[2].    Reed, Colorado J., et al. \"Self-supervised pretraining improves self-supervised pretraining.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.", "other_strengths_and_weaknesses": "Strenghts:\n\nEmpirically, it works! \nRelatively easy to read. \n\nWeaknesses:\n\nIn principle, this modernizes \"Self-supervised pretraining improves self-supervised pretraining.\"  with a stronger backbone and peft finetuning, which is not very novel.", "other_comments_or_suggestions": "My main suggestions for improving are discussing the missed references, demonstrating that PEFT is truly needed (not just full finetune), and  sure to show both Dino and MAE results in Table 4-9 for consistency.", "questions_for_authors": "Here are the points which would lead me to the a higher score.\n\n* Demonstrating that one can't do the proposed two stage finetuning with no PEFT, since PEFT mechnaism is one of the key innovation . I believe this to be true, since it is difficult to not catastrophically forget, but it is the key innovation in the paper, so important to verify \n\n* Show additional evidence in Table 4 ,5 for D -[L}-r32 and M-[L]-r32 in Tables 7-9.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: ExpLoRA investigates a new mode of transfer learning: parameter efficient fine-tuning to first extend self-supervised pre-training of some parameters + LoRAs then supervised learning on LoRAs alone for visual data in the presence of shift. The experiments show the approach is effective on remote sensing and standard benchmarks of shift like WILDS. Ablations and analyses provide additional information for practitioners in computer vision, and the results are relevant by their scale with ViTs and by their improvement over full pre-training and fine-tuning.

Reviewers differ in their scores at 3, 3, 2, 2. The authors provide a rebuttal, 4/4 reviewers acknowledge it, and there is extensive discussion. XYuv raises their score from 2 to 3 and W3i1 raises their score from 3 to 4. 83s5 maintains their score for 2 and argues for rejection due to lack of novelty as an application of LoRA to ViT and insufficient efficiency. In the final AC-reviewer discussion 83s5 moderates their view to borderline and XYuv argues in favor of acceptance because the work provides insight for the transfer of foundational models to new visual domains.

The area chair sides with acceptance: the design and thorough experiments are informative and novel in the context of unsupervised then supervised learning. It is not trivial to coordinate the updates of different parameterizations (frozen/unfrozen layers, LoRAs, etc.) and this work offers practical guidance.
Decision: Accept (poster)