ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts

## Abstract

Abstract Parameter-efficient fine-tuning (PEFT) techniques such as low-rank adaptation (LoRA) can effectively adapt large pre-trained foundation models to downstream tasks using only a small fraction (0.1%-10%) of the original trainable weights.
An under-explored question of PEFT is in extending the pre-training phase without supervised labels; that is, can we adapt a pre-trained foundation model to a new domain via efficient self-supervised pre-training on this new domain?
In this work, we introduce ExPLoRA, a
highly effective technique to improve transfer learning
of pre-trained vision transformers (ViTs) under domain shifts.
Initializing a ViT with pre-trained weights on large, natural-image datasets such as from DinoV2 or MAE, ExPLoRA continues the unsupervised pre-training objective on a new domain, unfreezing 1-2 pre-trained ViT blocks and tuning all other layers with LoRA.
We then fine-tune the resulting model only with LoRA on this new domain for supervised learning.
Our experiments demonstrate state-of-the-art results on satellite imagery, even outperforming fully pre-training and fine-tuning ViTs.
Using the DinoV2 training objective, we demonstrate up to 7.5% improvement in linear probing top-1 accuracy on downstream tasks while using <10% of the number of parameters that are used in prior fully-tuned state-of-the art approaches.
Our ablation studies confirm the efficacy of our approach over other baselines, including PEFT and unfreezing more ViT blocks.
Code is available on the project website: https://samar-khanna.github.io/ExPLoRA/

## 1 Introduction

Pre-training foundation models (Bommasani et al., ) for natural language (Brown et al., ; Chowdhery et al., ; Touvron et al., ; Jiang et al., ) and natural images (Oquab et al., ; He et al., ; Zhou et al., ; Rombach et al., ) has historically been computationally intensive, often limited to organizations with substantial resources.
However, recent advancements in parameter-efficient fine-tuning (PEFT) techniques including low-rank adaptation (LoRA) and others (Hu et al., ; Zhang et al., ; Chavan et al., ; Qiu et al., ; Liu et al., ; Jia et al., ) have sparked significant interest.
These methods aim to adapt foundation models to downstream supervised-learning tasks using a small fraction (0.1%-10%) of the model’s trainable weights, with many based on the hypothesis that the required weight updates to the pre-trained model have a “low intrinsic rank" (Hu et al., ; Li et al., ; Aghajanyan et al., ).

In this paper, we focus on visual foundation models (VFMs) such as DinoV2 or MAE (Oquab et al., ; He et al., ), which were trained on large-scale natural-image datasets.
Despite the large investments in developing such models for natural images, they underperform when applied to other domains with visual data (e.g. medical or satellite images). For example, fine-tuning a model pre-trained on natural images on satellite image classification tasks
is not as effective as
fine-tuning one that was pre-trained on satellite images (Cong et al., ; Ayush et al., ).
To bridge this gap, prevailing approaches invest similarly large levels of compute to pre-train VFMs on new domains, inspired by techniques developed for natural images (Cong et al., ; Reed et al., ; Tang et al., ; Khanna et al., ; Zhou et al., ; Moutakanni et al., ; Man et al., ).

In this work, we challenge this paradigm (fig. 1), asking whether pre-training from scratch on each new domain is strictly necessary, since doing so is expensive (in compute and time) and precludes knowledge transfer from natural images.
Instead, we wish to more efficiently leverage the rich semantic information encoded in natural-image vision foundation models to adapt them to new domains.
Our proposed solution addresses these concerns using PEFT techniques for domain adaptation via self-supervised learning.

Figure: Figure 1: Consider two different image domains, $D_{S}$ and $D_{T}$. Left: the traditional paradigm of pre-training from scratch on each domain to yield $W_{D_{S}}$ and $W_{D_{T}}$, and then fine-tuning on target datasets $i$ to yield $\Delta_{s_{i}},\Delta_{t_{i}}$, for domains $D_{S}$ and $D_{T}$, respectively. Right: our approach, which is to initialize with pre-trained weights from domain $D_{S}$ and then learn unsupervised weights $\Delta_{D_{T}}$ for domain $D_{T}$ in a parameter-efficient manner.
Refer to caption: x1.png

We introduce ExPLoRA, which generalizes vision foundation models to new domains by extending the pre-training phase with parameter-efficient techniques.
We initialize a vision transformer (ViT) (Dosovitskiy et al., ) with pre-trained weights from natural-image datasets such as MAE or DinoV2.
Selectively unfreezing 1-2 transformer blocks, we tune remaining weights with LoRA and continue unsupervised pre-training on the new domain.
Subsequently fine-tuning with linear probing or LoRA on this new domain for supervised learning outperforms prior state-of-the-art (SoTA) approaches while training under 6-10% of the original weights.
On satellite imagery, for example, we demonstrate an 8% improvement in linear probing top-1 accuracy, and even an improvement over prior SoTA fully pre-trained and fine-tuned techniques.
We conduct an extensive study on RGB, temporal, and multi-spectral satellite images, either matching or outperforming prior methods that fully pre-train from scratch.
ExPLoRA also generalizes to different domains such as wildlife, medical, and agricultural imagery on the WILDS (Koh et al., ) benchmark.
Our contributions include:

- 1.
Introducing ExPLoRA, a novel parameter-efficient method that extends unsupervised pre-training on target domains, achieving state-of-the-art supervised-learning performance using a fraction of the original ViT weights (section 5).
- 2.
Conducting a comprehensive case study on satellite imagery, showcasing improvements in linear probing top-1 accuracy and outperforming existing techniques on datasets like fMoW. We also demonstrate generalization to multiple other domains within WILDS (section 6).
- 3.
Demonstrating ExPLoRA’s efficacy via ablation studies and by analyzing the differences in local (eg: positional) and global (eg: class) information encoded in the patch representations output by each ViT block (section 6.3).

## 2 Related Work

##### VFMs

VFMs such as DinoV2 or masked autoencoders (MAE) that pre-train with self-supervised learning (SSL) have demonstrated remarkable performance across downstream tasks such as classification or semantic segmentation Oquab et al. (); He et al. (); Grill et al. (); Chen et al. ().
However, there has also been a rise in domain-specific VFMs (Cong et al., ; Reed et al., ; Moutakanni et al., ; Ma et al., ; Zhang et al., ).
For instance, SatMAE handles temporal or multi-spectral satellite image inputs.
Since these models contain hundreds of millions of parameters, efficient adaptation to downstream tasks has become a key research focus.

##### PEFT

PEFT methods have gained widespread adoption for efficiently adapting large models to various downstream tasks, mitigating the prohibitive costs of full model tuning by updating only a fraction of the parameters.
For example, LoRA learns low-rank weight updates to frozen weights, while other methods modify the frequency or number of trainable parameters per layer (Hu et al., ; Zhang et al., ; Chavan et al., ; Pu et al., ).
Lialin et al. () introduce LoRA for pre-training via adding multiple low-rank matrices, but require full parameter tuning as a “warm start”.
Others use multiplicative orthogonal weight updates, effectively retaining pre-training knowledge in frozen weights (Qiu et al., ; Liu et al., ).
Another family of PEFT techniques include visual prompt tuning (VPT), which concatenate learnable prompt tokens to the sequence of image patch tokens, trading improved fine-tuning performance with increased inference-time latency and memory usage (Jia et al., ; Yoo et al., ; Pei et al., ; Han et al., ).
Our ExPLoRA method aims to supplement existing PEFT methods rather than replace them, and thus can be configured with any existing or future PEFT method intended for fine-tuning ViT backbones.

##### Domain Adaptation

Domain adaptation approaches have attempted to bridge the train-test data distribution shift from several perspectives (Singhal et al., ).
Discrepancy-based methods minimize the difference between feature distributions of the source and target domains using discrepancy metrics (Gretton et al., ; Kang et al., ; Sun et al., ) for domain loss.
Adversarial methods are trained to amplify domain confusion while simultaneously distinguishing between different domains (Ganin et al., ; Li et al., ; Tzeng et al., ).
Group DRO minimizes the loss in the worst-case domain, specifically addressing subpopulation shift, where data distributions differ but may have some overlap (Hu et al., ; Oren et al., ).

## 3 Background

##### MAE

The masked-autoencoder (MAE) (He et al., ) is an effective SSL technique for ViTs that uses an asymmetric encoder-decoder architecture on images ${\mathbf{x}}\in{\mathbb{R}}^{C\times H\times W}$, where patches are masked before being processed by the ViT encoder ${\mathcal{L}}$.
The masked patches are then reconstructed by a smaller decoder ${\mathcal{L}}_{D}$, with both trained jointly using mean-squared error on the reconstructed visible pixels.
While effective across domains (Cong et al., ; Bachmann et al., ), MAEs typically require full fine-tuning for downstream tasks, which makes them computationally expensive.

##### DinoV2

DinoV2 (Oquab et al., ) is a robust SSL method for ViTs. Unlike MAE, DinoV2 features have demonstrated strong zero-shot performance, enabling adaptation to downstream tasks even with a frozen ViT backbone. During pre-training, DinoV2 maintains two copies of a ViT encoder: the student (trainable) and the teacher, which is updated using an exponential-moving average of the student’s parameters. The training objective incorporates a global, image-level loss from Dino (Caron et al., ), a patch-based loss from iBOT (Zhou et al., ), and regularizers including KoLeo (Delattre & Fournier, ) and Sinkhorn-Knopp centering (Caron et al., ).

##### LoRA

Low-rank adaptation (LoRA) (Hu et al., ) assumes that the weight update to change a set of unsupervised pre-trained weights to supervised fine-tuned weights lives in a low-rank subspace,

$$ $W\approx W_{0}+\Delta_{W}=W_{0}+BA$ (1) $$

where $W\in\mathbb{R}^{k_{2}\times k_{1}}$ are the final, task-specific fine-tuned weights, $W_{0}\in\mathbb{R}^{k_{2}\times k_{1}}$ are the pre-trained weights, $\Delta_{W}\in\mathbb{R}^{k_{2}\times k_{1}}$ is the weight update required to translate the pre-trained weights $W_{0}$ to the fine-tuned weights $W$. The key is that $\Delta_{W}=BA$ where $B\in\mathbb{R}^{k_{2}\times r}$ and $A\in\mathbb{R}^{r\times k_{1}}$. That is, $A$ and $B$ form a low-rank factorization of $\Delta_{W}$, where the rank $r\ll\min(k_{1},k_{2})$.

## 4 Problem Setup

Consider a set of image domains ${\mathcal{D}}=\{1,2,\dots\}$, where each domain $d\in{\mathcal{D}}$ is associated with a data distribution $p_{d}({\mathbf{x}})$, and images ${\mathbf{x}}\in{\mathbb{R}}^{C_{d}\times H_{d}\times W_{d}}$ have domain-specific channel, height, and width.
Let $D_{S}\subset{\mathcal{D}}$ represent a set of source domains (e.g., internet-scale natural image data) and $D_{T}\subset{\mathcal{D}}$ represent target domains (e.g., satellite imagery).
The data from the source domains follow a distribution $p_{D_{S}}({\mathbf{x}})$, and the target domain data come from $p_{D_{T}}({\mathbf{x}})$.
For some target domains $d_{T}\in D_{T}$, the joint distributions $p_{d_{T}}({\mathbf{x}},{\mathbf{y}})$ describe images ${\mathbf{x}}$ with associated supervised labels ${\mathbf{y}}$ used for downstream tasks.
We then assume access to the following:

- (i)
$W_{D_{S}}$, pre-trained weights obtained via unsupervised pre-training on images from $p_{D_{S}}({\mathbf{x}})$
- (ii)
${\mathcal{X}}_{D_{T}}=\{{\mathbf{x}}_{i}\}_{i=1}^{N}\sim p_{D_{T}}({\mathbf{x}})$, an unlabeled dataset of $N$ images from new domains $D_{T}$
- (iii)
${\mathcal{Y}}_{d_{T}}=\{{\mathbf{x}}_{j},{\mathbf{y}}_{j}\}_{j=1}^{M_{d_{T}}}
\sim p_{d_{T}}({\mathbf{x}},{\mathbf{y}})$ a labeled dataset of $M_{d_{T}}$ images from domain $d_{T}\in D_{T}$

Our objective is to learn optimal weights $W_{d_{T}}$ for each supervised-learning dataset ${\mathcal{Y}}_{d_{T}}$ in a parameter-efficient manner while leveraging the knowledge stored in $W_{D_{S}}$.

Traditionally, the approach (fig. 1) has been to begin pre-training from scratch on the new domains of interest in ${\mathcal{X}}_{D_{T}}$, and then fine-tune for each dataset ${\mathcal{Y}}_{d_{T}}$, representing the following:

$$ $W_{d_{T}}\approx W_{D_{T}}+\Delta_{d_{T}}$ (2) $$

where $W_{D_{T}}$ represents the weights learned from unsupervised pre-training on ${\mathcal{X}}_{D_{T}}$, and $\Delta_{d_{T}}$ are the weights learned from supervised fine-tuning on ${\mathcal{Y}}_{d_{T}}$.
However, this method is computationally expensive: fully pre-training $W_{D_{T}}$ from scratch for every new domain requires prohibitively large amounts of additional compute.

On the other hand, LoRA addresses this inefficiency in the following way:

$$ $W_{d_{T}}\approx W_{D_{S}}+\Delta_{d_{T}}=W_{D_{S}}+B_{d_{T}}A_{d_{T}}$ (3) $$

The LoRA hypothesis is that the update $\Delta_{d_{T}}$ resides in a low-rank subspace when adapting pre-trained weights $W_{D_{S}}$ to fine-tuned weights $W_{d_{T}}$.
This hypothesis holds well when pre-training and fine-tuning distributions are similar, or where $d_{T}\in D_{S}$.
However, when there is significant domain shift, such as between natural images and multi-spectral satellite data, the low-rank assumption often breaks down (see section 6.1.3).

Our goal is to learn $W_{D_{T}}$ in a parameter-efficient manner to bridge the large domain shift to $D_{T}$ while leveraging the knowledge encoded in $W_{D_{S}}$.
We propose the following factorization of $W_{d_{T}}$:

$$ $W_{d_{T}}\approx W_{D_{S}}+\Delta_{D_{T}}+\Delta_{d_{T}}$ (4) $$

where $\Delta_{D_{T}}\in\mathbb{R}^{k_{2}\times k_{1}}$ is an additional update matrix learned from unsupervised pre-training on ${\mathcal{X}}_{D_{T}}$.
Crucially, $\Delta_{D_{T}}$ requires only a fraction of the $k_{1}k_{2}$ parameters of $W_{D_{S}}$, making it significantly more efficient than full-rank pre-training. The resulting model, $W_{D_{T}}^{\ast}=W_{D_{S}}+\Delta_{D_{T}}\approx W_{D_{T}}$, retains the benefits of unsupervised pre-trained VFMs, including strong feature extraction, effective linear probing, KNN classification, and generalization to downstream tasks.

Figure: Figure 2: An overview of ExPLoRA. The set ${\mathcal{L}}$ of L ViT blocks is partitioned into two sets: ${\mathcal{U}}$, which denotes blocks whose parameters are completely unfrozen, and ${\mathcal{L}}\setminus{\mathcal{U}}$ which denotes blocks that undergo LoRA tuning (only on the $Q,V$ attention matrices). Note that the normalization layers are always unfrozen across all blocks.
Refer to caption: x2.png

Figure: Algorithm 1 ExPLoRA

## 5 Method

To learn $\Delta_{D_{T}}$, we propose ExPLoRA (i.e. Extended Pre-training with LoRA), a method that efficiently adapts pre-trained ViTs for new target domains $D_{T}$, described in algorithm 1.

In terms of notation, D-[L]-$r64$ refers to a ViT initialized with DinoV2 weights (denoted by D), where ${\mathcal{U}}=\{\texttt{L}\}$, and LoRA rank 64 is applied to the $Q,V$ matrices of attention layers in ${\mathcal{L}}\setminus{\mathcal{U}}$.
Thus, $\Delta_{D_{T}}$ comprises of all weights in ${\mathcal{U}}$, LoRA matrices in ${\mathcal{L}}\setminus{\mathcal{U}}$, and normalization layers.
For ${\mathcal{U}}=\{\texttt{L}\}$, $\Delta_{D_{T}}$ consists of only 5% of the original ViT parameters.
As we show in section 6, our extended pre-training approach can match or even outperform full pre-training on new domains from scratch.

##### ExPLoRA for DinoV2

We initialize a ViT-L with $W_{D_{S}}$ from the DinoV2 ViT-L encoder, without registers (Darcet et al., ).
Since the DinoV2 pre-trained checkpoints don’t contain the Dino or iBOT linear heads, we initialize a shared Dino-iBOT linear head from scratch.
This shared head is fully trained during extended pre-training, adding only a minimal number of trainable parameters.

##### ExPLoRA for MAE

We initialize a ViT-L with $W_{D_{S}}$ from the MAE ViT-L encoder.
Since MAE provides the pre-trained decoder, we use these weights to initialize our MAE decoder He et al. ().
During extended pre-training, in addition to the ExPLoRA recipe in algorithm 1, we apply LoRA with rank $r^{\prime}$ on the $Q,V$ matrices of each attention layer in the frozen decoder.
Note that the LoRA rank $r^{\prime}$ may differ from the LoRA rank $r$ used in the ViT encoder (section A.4).
All other decoder weights, apart from the layer-normalization layers, are kept frozen.
No block is fully unfrozen in the MAE decoder, as it will be discarded after extended pre-training.
This helps to minimize the number of additional parameters trained in the decoder.

##### ExPLoRA for Multi-Spectral Inputs

For the multi-spectral ViT introduced by SatMAE we need to additionally unfreeze the positional encoding and the patch embedding weights for each group of channels.
These cannot be initialized from $W_{D_{S}}$, as $W_{D_{S}}$ is trained on RGB inputs, whereas multi-spectral inputs can have more or different channels.
As part of $\Delta_{D_{T}}$ in algorithm 1, the positional encodings and patch embeddings for multi-spectral data are adapted during extended pre-training.
Aside from this, the approach remains unchanged from that of DinoV2 or MAE described earlier.

##### Storage Considerations

After running ExPLoRA, we receive a new unsupervised model $W_{D_{T}}^{\ast}=W_{D_{S}}+\Delta_{D_{T}}$ for the target domains $D_{T}$.
Any components that are not part of the ViT encoder (eg: the Dino linear head or the MAE decoder) are discarded.
Post-ExPLoRA, only $\Delta_{T}$, consisting of 1-2 unfrozen ViT blocks, LoRA matrices, and layer-normalization weights, are stored for each $D_{T}$.
Like LoRA and other PEFT methods, ExPLoRA significantly reduces additional storage requirements compared to fully training $W_{D_{T}}$ from scratch.

##### Fine-Tuning post-ExPLoRA

After extended pre-training with ExPLoRA, the output weights $W_{D_{T}}^{\ast}$ behave as any fully pre-trained model $W_{D_{T}}$.
We can now use the ViT for feature extraction, PEFT, or fine-tuning as desired.
For instance, we could initialize a linear head for classification or a decoder for segmentation, either of which is fully trainable.
We can then freeze all ViT weights and apply LoRA-$r8$ on the $Q,V$ matrices of the attention layers (or use another PEFT method).
Lastly, we use supervised fine-tuning on each labeled dataset ${\mathcal{Y}}_{d_{T}}$ to train the unfrozen parameters $\Delta_{d_{T}}$.
This yields our final model $W_{d_{T}}$ (eq. 4), which can be used for classification, segmentation, detection etc.

## 6 Experiments

Our experimental results consist of a case study on satellite imagery (section 6.1), with an ablation study in section 6.1.2 and analysis in section 6.3. We evaluate on multiple downstream tasks in sections 6.1.3, 6.1.4 and 6.2.
Additional experiments and ablations are provided in appendix A and training hyperparameter and compute configurations are mentioned in appendix B.
Our results achieve a new SoTA top 1 accuracy of 79.2% ($\uparrow$1.4%) on the competitive fMoW-RGB benchmark, outperforming fully pre-trained and fine-tuned models while using 6% of the ViT encoder parameters. We also achieve a $\uparrow$8.2% improvement in linear probing accuracy on the same dataset. Across other satellite datasets, we match fully-pretrained prior state-of-the-art methods, and demonstrate competitive performance on WiLDS benchmark datasets as well.

### 6.1 Case Study: Satellite Imagery

We examine satellite images given their importance towards multiple societal applications (Broader Impact) and since they represent a significant domain shift from natural images.
There is a large and growing body of research on developing foundational models for satellite imagery from scratch Ayush et al. (); Cong et al. (); Reed et al. (); Tang et al. (), thus presenting a good benchmark for ExPLoRA.

#### 6.1.1 RGB Satellite Images

##### Dataset

We first consider the functional map of the world (fMoW) dataset of high-resolution satellite images, each paired with one of 62 classification labels (Christie et al., ). fMoW is used as a benchmark for many satellite-image foundation models (Cong et al., ; Reed et al., ).

**Table 1: Results on the fMoW-RGB validation dataset. The “Pre-train #Params" and “Fine-tune #Params" refer to the trainable parameters of the ViT encoder required on the new domain, i.e. satellite images. M-[L]-$r64$ and D-[L]-$r64$ refer to ExPLoRA models initialized with MAE and DinoV2 weights, respectively (section 5).**
| Model | Arch. | PEFT | Pre-train<br>#Params | Fine-tune<br>#Params | Top 1 Acc. |
| --- | --- | --- | --- | --- | --- |
| ScaleMAE [] | ViT-L | Full | 303.3M | 303.3M | 77.80 |
| SatMAE [] | ViT-L | Full | 303.3M | 303.3M | 77.78 |
| SatMAE [] | ViT-L | LoRA-$r8$ [] | 303.3M | 0.8M | 76.10 |
| MAE [] | ViT-L | LoRA-$r8$ [] | - | 0.8M | 76.21 |
| MAE [] | ViT-L | DVPT-10 [] | - | 0.4M | 72.35 |
| MAE [] | ViT-L | GVPT-100 [] | - | 0.4M | 70.86 |
| MAE [] | ViT-L | SA^2VP [] | - | 1.1M | 73.55 |
| MAE [] | ViT-L | AdaLoRA-$r8$ [] | - | 1.2M | 75.25 |
| M-[L]-$r64$ | ViT-L | LoRA-$r8$ [] | 18.7M | 0.8M | 76.55 |
| DinoV2 [] | ViT-L | LoRA-$r8$ [] | - | 0.8M | 78.08 |
| DinoV2 [] | ViT-L | BOFT-$b2m8$ [] | - | 0.9M | 72.40 |
| DinoV2 [] | ViT-L | VPT-100 [] | - | 0.4M | 77.29 |
| DinoV2 [] | ViT-L | GVPT-100 [] | - | 0.4M | 76.22 |
| DinoV2 [] | ViT-L | AdaLoRA-$r8$ [] | - | 1.2M | 78.87 |
| DinoV2 [] | ViT-L | SA^2VP [] | - | 1.1M | 77.53 |
| D-[L]-$r64$ | ViT-L | SA^2VP [] | 18.7M | 1.1M | 78.51 |
| D-[L]-$r64$ | ViT-L | LoRA-$r8$ [] | 18.7M | 0.8M | 79.15 |

We compare our results in table 1 against both prior fully pre-trained SoTA foundation models as well as PEFT techniques applied on ViTs pre-trained with MAE and/or DinoV2 weights.
Our results demonstrate that D-ExPLoRA-$\tt{[L]}$-$r64$ is SoTA in terms of fMoW-RGB average accuracy at 79.15%.
ExPLoRA outperforms techniques that require fully pre-training ViTs on fMoW while using 6% of the original ViT encoder parameters.

ExPLoRA-initializations with LoRA fine-tuning outperform other unsupervised initializations paired with PEFT techniques by 1-3%, including SoTA matrix-adaptation methods like AdaLoRA (Zhang et al., ), BOFT (Liu et al., ), and VPT-based approaches such as GVPT (Yoo et al., ) and SA^2VP (Pei et al., ).
Additionally, applying SA^2VP to ExPLoRA-initialized ViTs improves performance over a DinoV2 initialization by  1%, showcasing ExPLoRA’s compatibility with other PEFT methods and its versatility as an initialization for new domains.

Using our strongest performing variant (i.e. ExPLoRA with DinoV2), we investigate linear-probing performance on fMoW-RGB compared with prior SoTA methods in table 3. Linear-probing represents freezing the backbone and then training a linear head on the features extracted from the frozen backbone, serving as a desirable metric of the quality of extracted embeddings. Our results demonstrate an improvement of over $\uparrow$8.2% in top 1 average accuracy over prior SoTA methods, demonstrating that ExPLoRA learns robust unsupervised representations for its target domain without requiring expensive from-scratch pre-training. Importantly, ExPLoRA outperforms domain-specific prior SoTA solutions (rows 1-4), as well as DinoV2, which suggests successful transfer learning on the target domain by leveraging knowledge from pre-training on natural images.

#### 6.1.2 Ablation study

**Table 2: Linear-probing on fMoW-RGB. The first four rows fully pre-train on the dataset. $\dagger$ denotes concatenating features from the last 4 ViT blocks. All other rows use the features of the last ViT block.**
| Method | Arch. | Top 1<br>Acc. |
| --- | --- | --- |
| GASSL [] | ResNet | 68.32 |
| SatMAE [] | ViT-L | 65.94 |
| ScaleMAE [] | ViT-B | 67.30 |
| CScaleMAE [] | ViT-B | 69.20 |
| DinoV2 [] | ViT-L | 67.60 |
| DinoV2$\dagger$ [] | ViT-L | 69.00 |
| D-[L]-$r64$ | ViT-L | 76.86 |
| D-[L]-$r64$$\dagger$ | ViT-L | 77.48 |

We perform an ablation study (table 3) on linear-probing performance for fMoW-RGB to determine whether our proposed configuration performs optimally.
A natural question is whether the improvement in performance stems primarily from unfreezing blocks, or from LoRA-tuning the rest of the ViT. We investigate this by unfreezing blocks $\{\texttt{L,L-1}\}$ in row 2 (with no LoRA), and comparing that with ExPLoRA-L-$r8$ in row 10. As seen, unfreezing an extra block consumes almost double the number of parameters, but fails to yield the same improvement in performance $\downarrow 0.34\%$. Thus, simply increasing the number of unfrozen blocks will likely improve performance, but will not do so as effectively as ExPLoRA, and will also significantly and sharply decrease the parameter-efficiency.

Next, we investigate whether high LoRA ranks used on all ViT layers (i.e. all attention and MLP matrices, not just $Q,V$) is beneficial. Surprisingly, this significantly harms learning (row 4). In fact, it is much less effective than using just LoRA-$r256$ on the $Q,V$ matrices of all ${\mathcal{L}}$ blocks (row 3). However, both rows 3 and 4 are much less parameter-efficient than ExPLoRA (rows 5-7, 10-12).

The choice of ${\mathcal{U}}$ matters as well. As seen in rows 4, 5, 6, and 12, for the DinoV2 objective, ${\mathcal{U}}=\{\texttt{1}\}$ or ${\mathcal{U}}=\{\texttt{9}\}$ are not as effective as ${\mathcal{U}}=\{\texttt{L-1}\}$ or ${\mathcal{U}}=\{\texttt{L}\}$, ceteris paribus. To understand this result further, see section 6.3. We also notice a slight drop in accuracy from leaving the normalization layers across the ViT frozen, seen in row 5.

Lastly, we investigate the impact of LoRA rank on ExPLoRA. Changing the rank from 8 to 32 has a small improvement ($\uparrow 0.09\%$), but changing from 32 to 64 brings about a much larger improvement ($\uparrow 1.08\%$), with only a relatively small increase in trainable parameters.
This demonstrates that higher ranks are necessary during pre-training for effective learning on the new domain.
Further ablations on data efficiency (A.3), MAE decoder rank (A.4), and ViT backbone size (A.5) are in appendix A.

#### 6.1.3 Multi-Spectral Satellite Images

##### Dataset

Next, we consider the fMoW-Sentinel dataset, a large dataset of Sentinel-2 images used in Cong et al. (). Each image consists of 13 spectral bands and is paired with one of 62 classes.

With fMoW-Sentinel, we evaluate the challenge of transferring from natural images to multi-spectral, low-resolution satellite images– a much harder task than fMoW-RGB due to the absence of sensor data beyond RGB bands in $D_{S}$. We use the SatMAE group-channel ViT-L model from Cong et al. (), initialized with MAE weights from natural images. Since the patch embedding layers differ, we unfreeze and train them from scratch during algorithm 1, adding minimal parameter overhead.

**Table 4: Results on the fMoW-Sentinel validation set. The “Pre-train #Params" and “Fine-tune #Params" refer to the trainable parameters required on the new domain, i.e. multi-spectral satellite images. “MAE-[1,2,L-1,L]" refers to initializing the group-channel SatMAE model with MAE weights, unfreezing blocks 1,2,23,24 for ViT-L, and then continuing pre-training on fMoW-Sentinel.**
| Model | Backbone | PEFT | Pre-train<br>#Params | Fine-tune<br>#Params | Top 1 Acc. |
| --- | --- | --- | --- | --- | --- |
| ImgNet-Supervised | ResNet152 | Full | 60.3M | 60.3M | 54.46 |
| MAE [] | ViT-L | Full | - | 303.3M | 51.61 |
| SatMAE [] | ViT-L | Full | 303.3M | 303.3M | 61.48 |
| MAE [] | ViT-L | LoRA-r8 | - | 0.8M | 46.97 |
| SatMAE [] | ViT-L | LoRA-r8 | 303.3M | 0.8M | 59.48 |
| MAE-[1,2,L-1,L] | ViT-L | LoRA-r8 | 51.5M | 0.8M | 54.12 |
| M-ExPLoRA-[L]-$r32$ | ViT-L | LoRA-r8 | 16.2M | 0.8M | 51.84 |
| M-ExPLoRA-[1,L]-$r32$ | ViT-L | LoRA-r8 | 29.7M | 0.8M | 60.15 |

Table 4 highlights the difficulty of this domain transfer. Fully fine-tuning from MAE weights leads to a nearly 10% drop in accuracy (row 2), and LoRA tuning from MAE alone performs even worse (row 4). Simply unfreezing four transformer blocks during pre-training (row 6) fails to close the gap.
However, ExPLoRA, with ${\mathcal{U}}=\{\tt{1,L}\}$, yields excellent results, even outperforming full pre-training from scratch for LoRA fine-tuning (row 5 vs. last row).
This demonstrates ExPLoRA’s effectiveness in bridging large domain gaps while using just a fraction (here, <10%) of the parameters.

#### 6.1.4 Additional Satellite Datasets

We perform extensive experiments on downstream satellite datasets, with further results in A.1.

##### fMoW-Temporal

Each input is a sequence of up to 3 fMoW-RGB (Christie et al., ) images of the same location, distributed temporally, and paired with one of 62 classes.
Since the inputs are now temporal sequences,
we initialize the temporal MAE architecture from Cong et al. () with MAE weights, and pre-train on ${\mathcal{X}}_{D_{T}}$ with ${\mathcal{U}}=\tt[L]$ and LoRA rank 32. ExPLoRA then outperforms temporal SatMAE for PEFT (table 6), demonstrating successful transfer learning at a fraction of the pre-training parameters.

##### SpaceNet-v1

This dataset contains high resolution satellite images, each paired with a segmentation mask for buildings (Van Etten et al., ).
The training and test sets consist of 5000 and 1940 images, respectively.
For ExPLoRA, we pre-train on the training set.
However, many images in the dataset contain extensive blacked-out regions, indicating limits of the visible region.
Considering this limitation and the small dataset size, it is not clear whether additional pre-training is effective.
We find that, despite this, ExPLoRA remains on par with the LoRA-tuned DinoV2 model and remains competitive with the fully pre-trained and fully fine-tuned domain-specific models (table 6).

##### RESISC-45

The RESISC-45 (Cheng et al., ) benchmark dataset consists of 31,500 satellite images of varying resolution (0.2m-30m GSD), with 45 classes. The data is split into 25,200 training and 6,300 validation images, as per Reed et al. (). In table 6, our D-ExPLoRA pre-trained on only high-resolution fMoW-RGB images achieves SoTA results of 97.32% on multi-resolution RESISC-45 images, with just linear-probing. Since we use the same pre-trained model as in the last row of table 1, we demonstrate successful transfer learning from ExPLoRA pre-training, without requiring any additional modifications for scale-aware representation learning (Reed et al., ).

**Table 5: fMoW-Temporal validation set results**
| Method | PEFT | Top 1 Acc. |
| --- | --- | --- |
| GASSL [] | Full | 74.11 |
| SatMAE [] | Full | 79.69 |
| MAE [] | LoRA-r8 | 69.30 |
| SatMAE [] | LoRA-r8 | 75.27 |
| M-[L]-$r32$ | LoRA-r8 | 75.98 |

### 6.2 WiLDS Datasets

We test ExPLoRA on the WILDS (Koh et al., ) benchmark, specifically on Camelyon17 (Bandi et al., ), iWildcam (Beery et al., ) and GlobalWheat David et al. (; ) datasets, representing domain transfers to medical, wildlife, and agricultural imagery, respectively.

**Table 7: Classification results on the validation set of Camelyon17.**
| Method | PEFT | Top 1<br>Acc. |
| --- | --- | --- |
| CLater [] | Full | 93.90 |
| ICON | Full | 90.10 |
| DinoV2 [] | Lin. Probe | 93.27 |
| DinoV2 [] | LoRA-r8 | 92.97 |
| D-[L]-$r32$ | Lin. Probe | 94.41 |
| D-[L]-$r32$ | LoRA-r8 | 94.21 |

##### Camelyon17

The WILDS Camelyon17 dataset consists of images of cancerous and non-cancerous cell tissue organized in labeled and unlabeled splits. We use the “train-unlabeled" split for pre-training ExPLoRA, and either use LoRA fine-tuning or linear probing on the training set of the labeled split.
We report accuracy on the binary classification problem and compare with entries on the WILDS leaderboard which use unlabeled data. Our results in table 9 demonstrate improved performance over domain-specific methods as well as DinoV2, once again successfully bridging the domain gap.

##### iWildcam

iWildcam classification requires identifying one of 182 animal species given an image.
We pre-train on the training set, finding that this outperforms pre-training on the extra-unlabeled set. In table 9, we find an improvement over DinoV2 using LoRA-$r8$ PEFT.
Surprisingly, the linear probing performance of the ExPLoRA suffers in comparison with DinoV2, suggesting possible loss in knowledge-transfer due to a small domain gap. Likely because natural image datasets $W_{D_{S}}$ such as ImageNet (Deng et al., ) used for DinoV2 already contain many images of animals.

##### GlobalWheat

The GlobalWheat dataset consists of a wheat head object detection task, where each image of a wheat field is associated with bounding boxes on the visible wheat heads David et al. (; ).
ExPLoRA extends pre-training on the training set, and then we run fine-tuning using Detectron2 code for object-detection with ViTs (Wu et al., ).
ExPLoRA outperforms both fully pre-trained baselines from the WILDS leaderboard and strong VFMs DinoV2 and MAE on top 1 accuracy, average precision, and average recall.

### 6.3 Analyzing ExPLoRA

Table: Figure 3: The mean of the principal components of the feature map outputted by each ViT block.

The key design choice of ExPLoRA is to fully train a small subset ${\mathcal{U}}\subset{\mathcal{L}}$ of the ViT, while applying low-rank updates to the remaining frozen layers ${\mathcal{L}}\setminus{\mathcal{U}}$.
For parameter-efficiency, we aim to keep $|{\mathcal{U}}|\ll|{\mathcal{L}}|$ and make and informed choice of which layers to unfreeze based on their potential to improve learning during extended pre-training.

We conduct an investigation on 5 models using a sample of ${\mathcal{X}}_{D_{T}}$.
These models are DinoV2, D-ExPLoRA-[L]-$r64$, SatMAE, MAE, and M-ExPLoRA-[L]-$r64$.
We do the following analyses:
(i) PCA to measure the mean and variance of eigenvalues of patch feature vectors for each ViT block, in fig. 3
(ii) linear probing for local or global information (Darcet et al., ) by training logistic regression classifiers on each block’s patch feature vectors, to predict either patch position (fig. 4) or image class (fig. 5).

##### Findings and Unfreezing Strategy for DinoV2

Our analysis reveals that the spectral properties of a block’s feature map (fig. 3) and the ability to retrieve local information from its output patch tokens (fig. 4) are correlated.
The classification accuracy for position and the mean of the principal eigenvalues peak in the middle-layers of the model, suggesting that the middle blocks capture local properties of patches (e.g., texture, relative position).
Meanwhile, deeper blocks focus on global semantic understanding, as shown by increased classification accuracy for image class prediction in fig. 5.
Combined, these results suggest that unfreezing deeper layers, such as ${\mathcal{U}}=\{\texttt{L}\}$, allows the model to better capture global features without overfitting to local details of images of $D_{T}$.
This is empirically confirmed in table 3, where linear probing accuracy correlates inversely with the mean eigenvalue of each block (i.e., block 23 > block 22 > block 0 > block 9).
The attention maps in fig. 6 further support this, showing that the deeper layers focus more clearly on central objects, while earlier layers (e.g., blocks 9, 10) exhibit more diffuse attention patterns spread around the border.

##### Findings and Unfreezing Strategy for MAE

For MAE, we see a similar, but less pronounced trend.
However, MAE is only trained for reconstruction, and so retains more local information across the ViT’s layers. This is reflected by its lower patch-wise eigenvalues, higher localization accuracy, and lower global accuracies than Dino.

##### ExPLoRA’s Impact

D-ExPLoRA preserves local information in the middle layers but also improves localization accuracy in the last few layers.
Importantly, it also enhances the global information contained in the patches for deeper model layers.
This indicates a better understanding of the target domain, as seen in A.2, where ExPLoRA’s attention highlights the central object more clearly.

## 7 Conclusion and Discussion

In this paper, we introduce ExPLoRA, a novel pre-training strategy to adapt pre-trained ViT foundation models for natural images to additional visual domains such as satellite imagery or medical data.
We challenge the common paradigm of expensive pre-training from scratch for each new visual domain by offering a solution to transfer knowledge from foundation models that is both parameter-efficient and effective (even outperforming domain-specific foundation models).
Our hope is that ExPLoRA enables further use of foundation models on domains other than natural images without requiring vast computational resources for pre-training.

While effective, there are many aspects of ExPLoRA that deserve further study.
The strategy of fully training a small amount (or budget) of weights combines extremely well with PEFT techniques such as LoRA– we hope that future work investigates the reason behind this in further detail.
Unresolved questions also include whether other parameter-efficient techniques might work better with ExPLoRA during pre-training.
Further work to evaluate ExPLoRA for natural language would be valuable, as would an investigation into whether we can do away entirely with unfreezing a transformer block.

## Broader Impact

As the scale of models and datasets grows exponentially, access to the computing power necessary to develop and make use of foundation models is increasingly restricted to the hands of a few organizations.
This leaves many researchers in academia or smaller companies reliant on the resources of such organizations for ML research and applications.
Techniques such as PEFT can alleviate this dependence and enable those with fewer computational resources to adapt, investigate, and customize models for their own needs.
We hope that ExPLoRA furthers this goal, allowing ML practitioners to tailor foundation models with minimal compute, thus broadening access to powerful ML tools for critical fields like sustainability and medicine.

For example, automated analysis of satellite imagery can inform social, economic, and environmental policies, but manual curation is expensive, and pre-training models on such data has significant costs, both environmental and otherwise (see appendix C).
ExPLoRA offers a more efficient way to distill knowledge from existing foundation models trained on natural images, sharply reducing costs while aiding researchers and policymakers and enabling flexible applications in downstream tasks.

## Acknowledgements

This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via 2021-2011000004, NSF(#1651565), ARO (W911NF-21-1-0125), ONR (N00014-23-1-2159), CZ Biohub, HAI. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes not-withstanding any copyright annotation therein.

## Appendix

We include supplementary material in the following sections.

## Appendix A Additional Experimental Results

We include further experimental results as a continuation of section 6.

### A.1 Results on Additional Downstream Datasets

##### NAIP

**Table 10: NAIP validation set results**
| Method | PEFT | Top 1 Acc. |
| --- | --- | --- |
| GASSL [] | Full | 57.63 |
| SatMAE [] | Full | 71.77 |
| SatMAE [] | LoRA-r8 | 69.45 |
| MAE [] | LoRA-r8 | 70.36 |
| DinoV2 [] | LoRA-r8 | 70.40 |
| D-[L]-$r32$ | LoRA-r8 | 70.40 |

We consider a land-cover classification dataset used in Ayush et al. (), where each of 244,471 training and 55,529 validation images are paired with one of 66 land cover classes obtained by the USDA’s National Agricultural Imagery Program. In table 10, we first demonstrate similar performance between both natural-image backbones (rows 4 and 5), which surprisingly outperform SatMAE, which is pre-trained on fMoW-RGB.
We use ExPLoRA to pre-train from DinoV2 to the training set of this dataset (without labels). Our results (row 6) demonstrate comparable performance, suggesting that for this dataset, domain-specific knowledge may not be highly relevant to successfully solve the task.

##### EuroSAT

**Table 11: EuroSAT validation set results**
| Method | PEFT | Top 1 Acc. |
| --- | --- | --- |
| SeCo [] | Full | 93.14 |
| SatMAE [] | Full | 98.98 |
| SatMAE [] | LoRA-r8 | 98.73 |
| DinoV2 [] | BOFT-b8m2 | 96.60 |
| M-[1,L]-$r64$ | LoRA-r8 | 98.54 |

The dataset contains 27,000 13-band satellite images of 10 classes Helber et al. (), sourced from Sentinel-2. For ExPLoRA, we don’t pre-train on the training set of this dataset, and instead use LoRA fine-tuning starting with the pre-trained weights learned in row 8 of table 4. We demonstrate improved performance over DinoV2, and match the performance achieved by the domain-specific SatMAE which was fully pre-trained on fMoW-Sentinel, and fully fine-tuned on EuroSAT (table 11). This demonstrates the successful use of our extended pre-trained model on further downstream datasets.

### A.2 Attention Map Visualizations

To aid our analysis in section 6.3, we visualize attention scores for different ViT blocks across multiple models, including DinoV2, D-[L]-$r64$ (i.e. the last row of table 3), the second and third rows of table 3, MAE, SatMAE, and M-[L]-$r64$.
These visualizations are shown in fig. 6 for 3 different images from the validation set of fMoW-RGB.
Since our models are trained without registers, we truncate attention scores more than 5 standard deviations away from the mean, thus removing artifact attention scores with unusually high values on background patches (Darcet et al., ).

Figure: Figure 6: Attention maps visualized from the validation set of fMoW-RGB. The models considered, from left to right, are: DinoV2, D-ExPLoRA-[L]-$r64$, Dino with blocks 22,23 unfrozen during extended pretraining, Dino with LoRA-r256 during extended pre-training, MAE, SatMAE, and M-ExPLoRA-[L]-$r64$. We visualize the attention maps at the beginning, middle, and end blocks of the ViT-L.
Refer to caption: x6.png

The visualizations in fig. 6 further support the analysis in section 6.3.
For the Dino models, the attention scores of block 9-10 are diffuse and spread around the central object of the image, with quite a few border pixels highlighted.
Conversely, the attention scores of the final layers are concentrated more towards the central object.
These visualizations further suggest that the middle layers focus on capturing local properties of the images such as texture, while the final layers capture global semantic information such as object-ness.
Interestingly, the initial blocks for the Dino models display sparse attention patterns with spikes on seemingly random patches.
This might suggest a form of caching to aid the computation of deeper layers that will extract local or global information.

For the MAE models, we see that the original MAE (pre-trained on natural images) seem to highlight more border pixels in the final layers of the ViT. Post extended pre-training with ExPLoRA, the final layers concentrate attention scores on the central object, more closely resembling the patterns of SatMAE (which was fully pre-trained on satellite images).
ExPLoRA is thus able to successfully transfer knowledge from its initialized source-domain weights $W_{D_{S}}$ to serve as a foundation model $W_{D_{T}}^{\ast}$ on the new target domain $D_{T}$.

### A.3 Convergence and Data Efficiency

Figure: Figure 7: Lin. probe accuracy vs. number of training iterations.
Refer to caption: x7.png

Another important question is on ExPLoRA’s data efficiency- i.e. can ExPLoRA achieve good representations on the target domain without requiring many training iterations?

In fig. 7, we plot the linear-probing accuracy against the number of extended pre-training iterations for ExPLoRA (in blue). ExPLoRA improves quickly, requiring between 100-150k extended pre-training iterations to reach optimal performance.
As discussed in section 6.1.2, unfreezing additional transformer blocks (in red) fails to achieve the same level of performance while requiring more parameters.

One hypothesis for the effectiveness of pairing unfreezing blocks with LoRA tuning is that the low-rank updates to the ViT backbone “nudge" the sequence of embedded visual tokens from $D_{S}$ to those representing $D_{T}$, which then enables the unfrozen ViT block to efficiently compress global information from the new domain.

### A.4 Impact of MAE Decoder Rank

**Table 12: Ablation on M-$\tt{[1,L]}$-$r64$ on the validation set of fMoW-Sentinel. Here, the LoRA rank used for the ViT-L encoder is fixed at $r=64$, while the rank $r^{\prime}$ for MAE decoder is varied.**
| Decoder<br>Rank $r^{\prime}$ | Top 1 Acc. |
| --- | --- |
| 8 | 59.75 |
| 16 | 59.77 |
| 32 | 60.15 |
| 64 | 59.21 |

As outlined in section 5, we initialize the MAE decoder ${\mathcal{L}}_{D}$ with pre-trained weights $W_{D_{S}}$ from He et al. (), keeping all decoder weights (except layer norm) frozen during extended pre-training on ${\mathcal{X}}_{D_{T}}$. We apply LoRA with rank $r^{\prime}$ to the $Q,V$ weights of the attention layers in the decoder ${\mathcal{L}}_{D}$, while unfreezing 1-2 blocks ${\mathcal{U}}$ in the ViT encoder ${\mathcal{L}}$ and applying LoRA with rank $r$ to the remaining layers ${\mathcal{L}}\setminus{\mathcal{U}}$ (algorithm 1).

We evaluate ExPLoRA with M-$\tt{[1,L]}$-$r64$ on fMoW-Sentinel, using a fixed encoder LoRA rank $r=64$, unfreezing blocks ${\mathcal{U}}=\{\tt{1,L}\}$, and varying the decoder rank $r^{\prime}$
We then fine-tune the resulting model with LoRA $r=8$ and measure the highest top 1 accuracy on the validation set of fMoW-Sentinel.
Results in table 12 show that increasing $r^{\prime}$ up to 32 improves fine-tuning performance, but performance declines when $r^{\prime}=64$, dropping $\downarrow 0.94\%$.
This suggests that balancing the unfrozen parameters between the ViT encoder ${\mathcal{L}}$ (used for fine-tuning) and the MAE decoder ${\mathcal{L}}_{D}$ (discarded post pre-training) is crucial.
Larger $r^{\prime}$ may improve the decoder’s ability without benefiting the learned representations of ${\mathcal{L}}$.
This issue doesn’t arise in DinoV2, as the Dino-iBOT shared head is fully trained since it isn’t provided by Oquab et al. ().

### A.5 Impact of ViT backbone size

**Table 13: Linear probing results on the validation set fMoW-RGB, where we vary the size of the ViT encoder ${\mathcal{L}}$ from ViT-B, ViT-L, and ViT-G. “Last 1/Last 4” refers to using the output representation from just the last 1 or the last 4 ViT layers, respectively.**
| Method | Arch. | Top 1 Acc.<br>Last 1/Last 4 |
| --- | --- | --- |
| DinoV2 [] | ViT-B | 63.62/65.90 |
| DinoV2 [] | ViT-L | 67.60/69.00 |
| DinoV2 [] | ViT-G | 70.07/70.36 |
| D-[12]-$r64$ | ViT-B | 74.72/75.11 |
| D-[24]-$r64$ | ViT-L | 76.86/77.48 |
| D-[32]-$r32$ | ViT-G | 77.29/77.79 |

We also test the impact of the ViT backbone for ExPLoRA, varying the architecture for DinoV2 from ViT-B (86M, $\tt{L}=12$ layers, embedding dimension 768), ViT-L (303M parameters, $\tt{L}=24$ layers, embedding dimension 1024), and ViT-G (1100M parameters, $\tt{L}=40$ layers, embedding dimension 1280) for extended pre-training on fMoW-RGB.
The ExPLoRA models we compare against are D-[12]-$r64$ for ViT-B, D-[24]-$r64$ for ViT-L, and D-[32]-$r32$ for ViT-G.
We unfreeze the 12th, 24th, and 32nd layers for each of ViT-B, ViT-L, and ViT-G, picking these layers by extending the analysis from section 6.3 to ViT-B and ViT-G.
We find that the 12th (last layer) for ViT-B and the 32nd (out of 40) layer for ViT-G output representations with low mean eigenvalues compared to other layers, thus presenting good candidates for unfreezing.

In table 13, we see that as expected, ViT-G performs the best, but is only $\uparrow 0.31\%$ better in top 1 accuracy compared to ViT-L, while using many more parameters.
On the other hand, we see the highest impact for ExPLoRA on ViT-B, where the top 1 accuracy improves by $\uparrow 9.21\%$ over the original DinoV2 ViT-B.
These results further demonstrate the effectiveness and efficiency of ExPLoRA as a powerful technique to create unsupervised foundation models for new visual domains.

## Appendix B Training Details

In this section, we describe hyperparameters and hardware configurations used for our models.

### B.1 Pre-Training

We use the ViT-Large architecture for all experiments.
Since raw image sizes vary, the shorter image size is resized to $224$ while preserving aspect ratio, and then a center crop is taken to yield images of size $3\times 224\times 224$, representing the channels, height, and width.
All pre-training is done on a single NVIDA-RTX 6000 Ada GPU, or 4 NVIDIA-RTX A4000 GPUs on an academic GPU cluster.

##### ExPLoRA with DinoV2

Most of the hyperparameters for D-ExPLoRA follow the defaults set by Oquab et al. (). That is, local (small) crops are between 5%-32% of the original image and are resized to 98x98 pixels, and global (large) crops are greater than 32% of the image and resized to 224x224 pixels.
We share the parameters of the Dino-iBOT linear head (3 layers), with a bottleneck dimension of 256, a hidden dimension of 2048, and an output dimension of 65536, initialized from scratch.
For Dino, we use Sinkhorn-Knopp (Caron et al., ) centering and Koleo (Delattre & Fournier, ) regularization with a weight of 0.1.
For iBOT, we use masking ratios between 0.1 and 0.5 to mask half of the samples in the batch.
The teacher model uses an initial EMA rate of 0.994, with a cosine warmup to 1.000 by the end of training. The teacher warmup and final temperatures are 0.04 and 0.07.
The linear Dino-iBOT head is frozen for the first 3k training iterations.
We train with the AdamW optimizer (no weight decay), with a base learning rate of $2\times 10^{-3}$ that is varied with a linear warmup and cosine decay schedule.
Training is completed within 200,000 iterations, with a batch size of 32 and with 32 gradient accumulation steps (equalling an effective batch size of 1024), and with an epoch length set to 1000.

##### ExPLoRA with MAE

Most of the hyperparameters we use for M-ExPLoRA pre-training follow those in He et al. (); Cong et al. ().
We use an effective batch size of 1024 (through gradient accumulation), a base learning rate of $4.5\times 10^{-4}$, no weight decay, and a warmup and decaying cosine scheduler, with a warmup of 1 epoch, and a total training time of 200 epochs.
We use a masking ratio of $0.75$ and we use the norm_pix_loss flag for the MSE loss.

### B.2 PEFT Fine-Tuning

We fine-tune using 4 NVIDIA-RTX A4000 GPUs.
We use a base learning rate of $10^{-3}$, a cosine scheduler with warmup for 1 epoch, and train for 120 epochs.
We use an effective batch size of 256, making use of gradient accumulation if the GPU cannot fit the full batch size in memory.

For data augmentations, we only use the drop-path augmentation (Larsson et al., ) at a rate of 0.2, with no dropout, mixup, or cutmix.
We note that the original LoRA configuration outperforms other PEFT techniques when paired with the drop-path regularization technique.
For example, we find that BOFT does not pair well with drop-path, instead performing most effectively with a custom multiplicative dropout technique (Liu et al., ).
We include the result with the best hyperparameter configuration for each row in table 1.

### B.3 Linear Probing

We use a single NVIDIA-RTX A4000 GPU for linear probing.
We adapt the code provided by Oquab et al. () for linear probing, with a batch size of 256 and a collection of different learning rates: $\left[1\times 10^{-4},1\times 10^{-3},5\times 10^{-3},1\times 10^{-2},2\times 1
0^{-2},5\times 10^{-2},1\times 10^{-1}\right]$.
We evaluate both probing on average pooled features as well as on the $\tt{[CLS]}$ token, and also use output features from just the last block, or the last 4 blocks.
All numbers reported represent the best validation set accuracy from the best performing configuration.

### B.4 Multi-Spectral Images

We use the group-channel ViT-L architecture introduced in Cong et al. ().
We don’t use DinoV2 since there is no such architecture for DinoV2 pre-training.
Input images are $13\times 98\times 98$, representing 13 multi-spectral bands.
We follow the configuration in Cong et al. () of dropping bands B1, B9, B10, and use the same grouping strategy.
When loading MAE weights to the ViT-L encoder, the patch embeddings do not match and so the patch embedding and group channel encodings are trained from scratch. All other configuration details are the same as for M-ExPLoRA in section B.1, except that we use a base learning rate of $4.5\times 10^{-4}$ for pre-training and train for 50 epochs (given the larger dataset size) on 4 NVIDIA RTX A4000 GPUs for 80 hours.

Fine-tuning details are the same as in B.2.

### B.5 Downstream datasets

Hyperparameter and training configuration details are the same as in section B.1 if the images are RGB, and the same as in section B.4 if the images have more channels or are temporal.

### B.6 Dataset Licenses

The licenses for all datasets are included in the footnotes:
fMoW(^1^11fMoW license: [https://github.com/fMoW/dataset/raw/master/LICENSE](https://github.com/fMoW/dataset/raw/master/LICENSE)), Sentinel-2(^2^22Sentinel-2 license: [https://scihub.copernicus.eu/twiki/pub/SciHubWebPortal/TermsConditions/Sentinel_Data_Terms_and_Conditions.pdf](https://scihub.copernicus.eu/twiki/pub/SciHubWebPortal/TermsConditions/Sentinel_Data_Terms_and_Conditions.pdf)), EuroSAT(^3^33EuroSAT license: [https://creativecommons.org/licenses/by/4.0/](https://creativecommons.org/licenses/by/4.0/)), SpaceNet(^4^44SpaceNet v1 license: [http://creativecommons.org/licenses/by-sa/4.0/](http://creativecommons.org/licenses/by-sa/4.0/)),
Camelyon17(^5^55Camelyon17 license:[https://creativecommons.org/publicdomain/zero/1.0/](https://creativecommons.org/publicdomain/zero/1.0/)), iWildCam(^6^66iWildCam license:[https://cdla.dev/permissive-1-0/](https://cdla.dev/permissive-1-0/)),
GlobalWheat(^7^77GlobalWheat license:[https://opensource.org/licenses/MIT](https://opensource.org/licenses/MIT)).

## Appendix C Environmental Impact

Following Cong et al. (), we compare the carbon footprint of pre-training using ExPLoRA with domain-specific solutions such as SatMAE. We use the carbon footprint calculator proposed by Lacoste et al. (). Our results are in table 14.

**Table 14: The estimated carbon footprint of pre-training on these datasets**
| Method | fMoW-RGB | fMoW-Sentinel | fMoW-Temporal |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| GPU hours | kg $CO_{2}$ eq. | GPU hours | kg $CO_{2}$ eq | GPU hours | kg $CO_{2}$ eq. |  |
| SatMAE | 768 | 109.44 | 576 | 82.08 | 768 | 109.44 |
| ExPLoRA | 96 | 12.44 | 320 | 19.35 | 100 | 12.96 |

Since we initialize with pre-trained weights on natural image domains, ExPLoRA is much less environmentally impactful while achieving similar or higher levels of performance. We achieve a 4x-8x reduction in total carbon emitted for each of the large pre-training satellite image datasets considered in table 14.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: See weaknesses.
soundness: 2
strengths: -	The introduction of ExPLoRA, a new parameter-efficient method to extend unsupervised pretraining on target domains. 

-	Conducting comprehensive experiments on various datasets, and showcasing improvements in linear probing top-1 accuracy and outperforming existing techniques on datasets like fMoW. 

-	The authors show the effectiveness of ExPLoRA and analyze the differences in local and global information encoded in the patch representations output by each ViT block.
summary: This work presents ExPLoRA, which initializes a ViT with pre-trained weights, selectively unfreezes 1 - 2 blocks, tunes remaining weights with LoRA, and continues unsupervised pre-training on a new domain. Then fine-tunes the model on the new domain for supervised learning. This work demonstrates state-of-the-art results on satellite imagery and generalizes to different domains like wildlife, medical, and agricultural imagery.
weaknesses: -	The authors argue that they selectively unfreeze 1 - 2 blocks, tunes remaining weights with LoRA. But specifically, it is very difficult to evaluate the number of layers to be tuned and which layers to be tuned. Although the authors have shown that the method of tuning 1 - 2 layers is feasible, it is not known whether the number of layers and the specifically selected layers are sensitive for different domains. In other words, the authors‘ method might not be optimal for different domains or datasets.
-	The authors may have missed two baselines. For example, in Table 1, the author only gives a result of using their own method for finetuning. However, one baseline is to use the LoRA method in the pre-training phase, and the author should compare the effect of their method with this baseline. The second baseline is to use the pre-trained MAE for full finetuning directly on the downstream task. Intuitively, it is still unknown whether further pre-training on the downstream task is necessary. This baseline will complete the integrity of the experiment. 
-	In addition to LoRA, the authors lack a comparison of some related PEFT methods [1,2,3,4,5]. It is better to be able to discuss the related methods. Furthermore, as mentioned in the paper, it is also better for the author to be able to give some results on detection and segmentation tasks.

[1] LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning, in NeurIPS2022.

[2] Scaling & shifting Your Features: A New Baseline for Efficient Model Tuning, in NeurIPS2022.

[3] Vision Transformer Adapter for Dense Predictions, in ICLR2023.

[4] Adapters Strike Back, in CVPR2024. 

[5] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning, in NeurIPS2024.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Potential limitations of the proposed extended pre-training. Can this stage only benefit the domain shits or also work in general scenarios (like general classification and general object detection)? I also did some experiments before and I failed. I would appreciate it if the authors' pipeline works in such scenarios, but I also understand this part is out of the scope of this paper's claim.
soundness: 3
strengths: 1: ExPLoRA excels at adapting large vision transformers to new domains without requiring full re-training, instead leveraging low-rank adaptation. This parameter-efficient approach significantly reduces computational costs, making it suitable for resource-constrained environments.

2: The proposed method is simple yet effective. The ExPLoRA can directly use the well-trained vision foundation models. This compatibility also simplifies its integration with established models.

3: Extensive experiments on RGB satellite images show the proposed methods' effectiveness compared with previous state-of-the-art works.
summary: The paper presents ExPLoRA, a method for efficiently adapting pre-trained vision transformers (ViTs) to new domains using parameter-efficient fine-tuning (PEFT) techniques, utilizing LoRA (Low-Rank Adaptation). 
By continuing unsupervised pre-training on the target domain and only unfreezing select model layers, ExPLoRA enables adaptation with minimal computational overhead. 
This approach leverages pre-trained models on natural image datasets like DinoV2, achieving notable performance gains, particularly in challenging domains like satellite imagery. 
For instance, ExPLoRA outperforms fully pre-trained models on satellite classification tasks while using fewer parameters, highlighting its efficiency. 
Beyond satellite data, ExPLoRA generalizes well to other domains, including medical and wildlife imagery, as tested on the WILDS benchmark.
weaknesses: 1: Analysis of the effect of total training cost. From my understanding, this work's main contribution (claim) is to adapt continuing unsupervised pre-training before supervised training on downstream domains (I don't regard using LoRA fine-tuning and unfreezing the last ViT blocks as this work's contribution, and feel free to point my misunderstanding if it is). It is a two-stage. Therefore, it's important to consider the computational cost of these two stages simultaneously. I have seen Figure 7 which analyzes the effect of pre-training iterations. How about simply putting the equivalent computational cost on the supervised fine-tuning stage? Will unsupervised pre-training speed up the convergence of supervised fine-tuning? If the computational budget is fixed, how should we allocate it across the two different stages?

2: The effect of the "extend unsupervised pre-training stage". Current ablation studies mainly focus on the settings of learnable parameters (unfrozen blocks and LoRA). I hold the perspective that the main claim of this paper is the importance of "extend pertaining" for downstream domain transfer. Therefore, the authors are encouraged to verify the necessity of this stage further. For example, supervised fine-tuning can be performed directly with the current optimal setting of LoRA and unfrozen blocks for transferring.

3: Extending the model to multi-modal models like CLIP and zero-shot settings like using natural language for zero-shot understanding in downstream tasks. The authors could follow the setting of the CLIP paper (the downstream validation dataset).


4: (An open question). How about changing unsupervised training objectives in the extended pertaining stage? Lare-scale pre-trained DinoV2 and MAE both have their advantages. The unsupervised training stage is not limited to the corresponding original training objective. Can the model combine different advantages using different training objectives in this stage?

Overall, this paper presents a simple yet effective method for model adaptation. My main concern exists in the importance of the claimed "extend unsupervised pre-training" stage.

-------------------------------------------------  After Rebuttal ------------------------------------------------

Most of my concerns have been addressed. My main concern exists in the importance of the claim that the extended pre-training stage plays a necessary role in overcoming domain shifts.

Although I still doubt whether the extended pre-training stage is really as important as claimed, this does not prevent this paper from being a very good paper with detailed and comprehensive experiments analysis. Therefore, I decide to increase my score to 6 at this time.

---

confidence: 5
contribution: 4
initial_rating: 6
presentation: 2
questions: See above.
soundness: 4
strengths: 1. The writing of the article is quite good and the article is logical.
2. Supplementary materials are substantial.
3. The use of PEFT for post-pretraining in the visual domain is innovative.
4. The experiments in the domain migration section are ample and sensible.
summary: This paper introduces ExPLoRA, a novel parameter-efficient method for adapting pre-trained vision transformers (ViTs) to new domains through extended unsupervised pre-training. ExPLoRA initializes ViTs with weights from natural-image datasets and continues pre-training on new domains with LoRA. The model is then fine-tuned on the new domain for supervised learning, achieving impressive results on satellite imagery and generalizing to various domains like wildlife, medical, and agricultural imagery. ExPLoRA outperforms fully pre-trained and fine-tuned techniques while using significantly fewer parameters.
weaknesses: 1. Figure 1 seems to express the difference between full fine-tuning and PEFT in a more accurate form.
2. The idea of using PEFT for post-pretraining is a good one. loRA was proposed a few years ago, and there are more efficient PEFT methods in the visual field. Maybe the authors can try to compare with LoRA with newer methods [1-3].
[1] 1% vs 100%: Parameter-efficient low rank adapter for dense predictions.
[2] Pro-tuning: Unified prompt tuning for vision tasks.
[3] Adapter is all you need for tuning visual tasks.

3. The method section is more like a solution derived through experience. It is recommended to analyse whether LoRA has limitations in coping with visual post-pretraining tasks, and optimise based on the analysis to propose your own PEFT method.
4. The whole article seems to be like a technical report, and it is recommended to add some in-depth theoretical analyses and technical innovations for visual features.

----------------------------------------After Rebuttal------------------------------------------------


The rebuttal solves part of my confusion and I improve my score. Still, I think the author could have cited related work that I mentioned or didn't mention. Necessary citations both give the reader a broader view of developments in the field, and are a way of recognising and respecting those who work in the field.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. What are the differences between ExPLoRA and LoRA?
2. Does the presented method outperform the state-of-the-art DA and PEFT methods in the same setting?
3. What are the results on the widely-used UDA benchmarks?
soundness: 3
strengths: 1. The paper is well organized, the figures are readable and understandable.

2. The proposed method looks logical and technically sound.

3. The experimental results are strong. Consistent improvements have been shown over different baselines.
summary: In this work, the authors introduce ExPLoRA to improve transfer learning of pre-trained vision transformers (ViTs) under domain shifts. It initializes a ViT with pre-trained weights and continues unsupervised pre-training on a new domain with some blocks unfrozen and LoRA for other layers. Then it fine-tunes with LoRA for supervised learning. Experiments show state-of-the-art results on satellite imagery. It improves linear probing top-1 accuracy and ablation studies confirm its efficacy over other baselines
weaknesses: 1. Concerns about the novelty: The proposed ExPLoRA seems does not have essential differences with the conventional LoRA. What are the differences from LoRA? This paper is more like a technical report than a research paper.

2. Many important experimental comparison results are missing. 1) The authors should compare their proposed method with the recent SOTA ViT-based domain adaptation [1-6] in the same setting. 2) Many PEFT methods besides LoRA, [7-8] should be compared. The reviewer wonders whether ExPLoRA outperforms these types of methods? 3) The results on the widely-used UDA benchmarks like Office-Home, Office-31, VisDA are missing.

[1]. TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation, WACV 2023;

[2]. Safe Self-Refinement for Transformer-based Domain Adaptation, CVPR 2022;

[3]. CDTRANS: CROSS-DOMAIN TRANSFORMER FOR UNSUPERVISED DOMAIN ADAPTATION, ICLR 2022;

[4]. Patch-Mix Transformer for Unsupervised Domain Adaptation: A Game Perspective, CVPR 2023;

[5]. Towards Unsupervised Domain Adaptation via Domain-Transformer, IJCV 2024;

[6]. Making The Best of Both Worlds: A Domain-Oriented Transformer for Unsupervised Domain Adaptation, ACM MM 2022;

[7].Low-Rank Few-Shot Adaptation of Vision-Language Models, CVPR 2024;

[8]. Quantized Prompt for Efficient Generalization of Vision-Language Models, ECCV 2024;

3. Many important references of domain adaptation and PEFT [1-8] are missing. These works should be briefly reviewed in the related work section.

--------------------------------------------------------------After Rebuttal--------------------------------------------------------------

Sorry for the late reply.

After carefully reading the rebuttal and other reviews, I'd like to thank the authors' efforts in response to my concerns. Though some of my concerns have been addressed, I still have concerns in terms of novelty and experimental comparisons:

1) About the novelty. The authors acknowledge that ExPLoRA is a combination of existing strategies, e.g., LoRA and unfreezing blocks, and the authors' explanations in the rebuttal do not convince me. Thus the reviewer thinks it have limited technical contributions. I agree with Reviewer ZQZA's opinion that the technical novelty is limited to the ICLR research community, and I also agree with Reviewer 8d6C's opinion that the whole paper seems like a technical report rather than a research paper.

2) About the experimental comparisons. Thanks the authors for providing the results on VisDA-2017 benchmark. Since this paper targets domain adaptation, the authors should conduct experimental comparisons on many widely-used benchmarks of domain adaptation such as Office31, and DomainNet. Only the results on VisDA-2017 benchmark are insufficient to reveal the effectiveness.

To sum up, I will keep my rating unchanged and suggest the authors revise the paper according to my advice carefully.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: **(Q1)**: In section 4 Problem Setup, the target domain data comes from p_{D_T}(x) where D_T is a set of domains, being a subset of all domains. What is then the difference between  p_{D_T}(x) and p_{d_T}(x, y) coming from d_T ∈ D_T? Is this just the formulation that *some* of the domains of p_{D_T}(x) might provide labels (i.e., p_{d_T}(x, y)) and some not? And if yes, is there a significant change in distributions between these? Can we assume that these distributions (p_{D_T}(x) and p_{d_T}(x, y)) are similar with respect to the distribution of x?

**(Q2)**: Is there a reason why for some experiments you do compare against ScaleMAE and in some you do not? For Table 1, the ScaleMAE in its LoRA-r8 version is missing (while SatMEA is provided). It would be interesting to see ScaleMAE performance as 0.8M fine-tuned parameter model. In Table 4 and Table 5 ScaleMAE is entirely missing. Table 6 is listing ScaleMAE as baseline. Is there a rationale behind this?

**(Q3)**: In Table 3, the ablation study, there is an experiment showing [All], which adapts not only the attention matrices but also the MLP matrices. This is great, have you also run experiments showing how MLP adaptation without attention adaptation would perform?

**(Q4)**: In Fig. 2, where can I find U in the figure, which is described in the image caption.
soundness: 3
strengths: **(S1)**: this work covers an important aspect of foundation model training.

**(S2)**: this work provides vast amount of experimental results showing the capabilities of the proposed approach.

**(S3)**: the presented ablation study of the paper provide valuable insights into parameter-efficient domain adaptation of remote sensing imagery.
summary: This paper presents an approach for parameter-efficient continual pretraining and fine-tuning of visual foundation models addressing domain shift of the underlying data distribution. To accomplish this, the authors propose to use LoRA during continual pretraining and subsequent fine-tuning.

The domain shift covered in this submission put a major focus on remote sensing imagery as covered by a large part of the experimental section. Nevertheless, the submission also covers some minor experiments including domain shifts of datasets about cell tissue images, wheat images, and animal images.

For all datasets, the authors were able to demonstrate the efficiency of their approach in continual pretraining and fine-tuning up to results able to outperform fine-tuned visual foundation models.

I very much appreciate the work in this area - especially because not everybody has the resources to train visual foundation models and therefore approaches like the one presented are much appreciated. However, overall I see this submission providing only marginal contribution as it combines known methods such as continual pretraining ([mendieta2023towards] proposed it for remote sensing) and low-rank adaptation ([scheibenreif2024parameter] proposed it for remote sensing) together.

Also and since the majority of the experimental section focuses on remote sensing, in times of geo-spatial foundation models such as ScaleMAE, SatMAE, or GMF, I have trouble to see that the first step of training such models is to take a visual foundation models being trained on natural images only.

---
```
@inproceedings{mendieta2023towards,
  title={Towards geospatial foundation models via continual pretraining},
  author={Mendieta, Mat{\'\i}as and Han, Boran and Shi, Xingjian and Zhu, Yi and Chen, Chen},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={16806--16816},
  year={2023}
}
@inproceedings{scheibenreif2024parameter,
    title     = {Parameter Efficient Self-Supervised Geospatial Domain Adaptation},
    author    = {Scheibenreif, Linus and Mommert, Michael and Borth, Damian},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {27841-27851}
}
```
weaknesses: **(W1)**: this submission rather combines known approaches than presents methodological or algorithmic novelty. This can be of great value for the research community. However, I think the insights provided in this work might be limited to the ICLR research community.

**(W2)**: given previous work in this area, I miss baseline evaluations against non-LoRA continual pretraining (-> mendieta2023towards) and against pure LoRA (-> scheibenreif2024parameter). Such experiments would provide the opportunity to highlight the capabilities of the proposed approach much more and compare it against previous work.

Meta-review: This work introduces ExPLoRA, a method that initializes a Vision Transformer (ViT) with pre-trained weights, selectively unfreezes one to two blocks, fine-tunes the remaining weights using LoRA, and continues unsupervised pre-training on a new domain. Subsequently, the model undergoes supervised fine-tuning for the target domain.

Five experienced reviewers provided a mixed assessment of this submission. Before rebuttal, four reviewers gave the negative reviews. After rebuttal, three reviewers raised their score and indicated that their issues have been solved. Most issues are limited novelty, unconvincing experiment results. However, one reviewer still felt the limited novelty of this work and kept the negative scores with the high confidence.

Thus, the Area Chair (AC) has carefully reviewed the process, including the initial reviews, rebuttal, and discussions between reviewers and authors, as well as the revised submission. The AC agrees with concerns raised by Reviewer ZQZA and Reviewer 47Sp regarding the limited novelty and narrow application scope. Despite the proposed method is simple yet effective, the insights are not enough in current draft, making this submission like a technical report.

For the future submission, the authors are encouraged to either narrow the focus of ExPLoRA or explore a broader range of experimental settings. Additionally, the authors need to provide a clearer differentiation between LoRA and ExPLoRA beyond merely incorporating LoRA.

Decision: Reject