Learning from Offline Foundation Features with Tensor Augmentations
Authors: Emir Konuk
Christos Matsoukas
Moein Sorkhei
Phitchapha Lertsiravarameth
Kevin Smith
Venue: NeurIPS 2024 poster (2024)
Keywords: Adaptation
Foundation Models
Transfer Learning
Primary area: machine_vision

Abstract: We introduce Learning from Offline Foundation Features with Tensor Augmentations (LOFF-TA), an efficient training scheme designed to harness the capabilities of foundation models in limited resource settings where their direct development is not feasible. LOFF-TA involves training a compact classifier on cached feature embeddings  from a frozen foundation model, resulting in up to  $37\times$ faster training and up to $26\times$ reduced GPU memory usage. Because the embeddings of augmented images would be too numerous to store, yet the augmentation process is essential for training, we propose to apply tensor augmentations to the cached embeddings of the original non-augmented images. LOFF-TA makes it possible to leverage the power of foundation models, regardless of their size, in settings with limited computational capacity. Moreover, LOFF-TA can be used to apply foundation models to high-resolution images without increasing compute.    In certain scenarios, we find that training with LOFF-TA yields better results than directly fine-tuning the foundation model.

Learning from Offline Foundation Features with Tensor Augmentations
Emir Konuk! Christos Matsoukas 1,2, Moein Sorkhei 1,2 Phitchapha Lertsiravaramet 1,2
Kevin Smith 1,2
KTH Royal Institute of Technology, Stockholm, Sweden Science for Life Laboratory, Stockholm, Sweden {ekonuk, ksmi th)
Abstract
We introduce Learning from Offline Foundation Features with Tensor Augmentations (LOFF-TA), efficient training scheme designed to harness the capabilities of foundation models in limited resource settings where their direct development is not feasible. LOFF-TA involves training compact classifier on cached feature embeddings from a frozen foundation model, resulting in up to 37x faster training and up to 26x reduced GPU memory usage. Because the embeddings of augmented images would be too numerous store, yet the augmentation process is essential for training, we propose to apply tensor augmentations to the cached embeddings of the original non-augmented images. LOFF-TA makes it possible to leverage the power of foundation models, regardless of their size, in settings with limited computational capacity. Moreover, LOFF-TA can be used apply foundation models to high-resolution images without increasing compute. In certain scenarios, we find that training with LOFF-TA yields better results than directly fine-tuning the foundation model.
Introduction
Large and expensive foundation models, designed to capture general-purposs knowledge, have become significant focus in machine learning and computer vision research [3]. These models excel in zero- and few-shot learning [4, 23] and adapt to various domains, especially in data-scarce scenarios, through transfer learning. But adapting these models for a specific task is a resource-intensive process [35]. The cost of fine-tuning large foundation models today is already prohibitive to most individuals and organizations. As they continue to grow in size, the rising costs of foundation models risk excluding all but the wealthiest organizations. To mitigate this, efffciet fine-tuning methods have been proposed. These methods incorporate rank-deficient [18] and simple affine modules [31] or learnable prompt parameters injected at the input stage [20]. Their core principle is to limit the number of parameters that need to trained. We extend this principle to its logical conclusion by introducing no intermediate parameters or prompts, and investigate whether it is possible to completely separate the foundation model from the training process.
In this study, we explore this complete separation of the resource-intensin foundation model from the training process. As seen in Figure 1, training data passed through the foundation model at a one-time cost, and then cached. The cached feature embeddings are later loaded and used train lightweight classifier. By adopting this caching strategy we can train at a significantly faster rate using less memory resources and achieve similar,
38th Conference Neural Information Processing Systems (NeurIPS 2024).

---

Method
Size
Performance
Mem,
augmentations
Classifier
Prozens Inner
LOFF-TA
256
256
0%
132
72
Foeling
512
16.1%
132
minevntiie
Butuar
Unfeozen linaa
236
1375
5452
Figure 1: Learning from Offline Foundation Features with Tensor Augmentations (LOFF-TA). Training data is passed through foundation model and cached. The cached embeddings are loaded and spatial tensor augmentations are applied in lieu of standard image augmentations. lightweight classifier is trained on the cached, augmented features. This enables the use of arbitrarily large foundation models and high-resolution images at no additional cost.
or even in some cases, better performance. This framework enables us to leverage the power foundation models of any size limited resource settings. Moreover, tasks requiring high-resolution imaging, such as medical image diagnosis can benefit from the power of foundation models without increasing computational costs. However, these benefits come at the cost of inference speed, which can be slower as a consequence of our paradigm.
We call this approach "Learning from Offline Foundation Features with Tensor Augmen- tations", or LOFF-TA. It a simple approach, but has not yet been explored as it is complementary existing adaptation methods. LOFF-TA can be trivially combined with existing adaptation methods [18, 31, 20] by caching features from the adapted foundation models themselves achieve even better performance. We conduct series of comprehen- sive experiments using LOFF-TA on eleven well-known image classification benchmark datasets to demonstrate its benefits and limitations. Our key findings and contributions are outlined as follows:
We propose, LOFF-TA which decouples the training process from the resource- intensive foundation model - classifier is trained on cached features from founda- tion models instead of images.
Since integrating image augmentations within LOFF-TA presents challenge due to the tremendous storage cost of caching the embeddings of augmented images, we propose apply spatial tensor augmentations the cached embeddings of original images when training the compact classifier. We show that they perform nearly as well as standard image augmentations.
We show that, using LOFF-TA, it is possible to achieve similar performance to fine-tuned foundation model at a fraction of the computational cost training speed is accelerated up to 37x, and GPU memory usage is reduced up to 26x. Additionally, LOFF-TA allows flexibility in choosing any input image size or foundation model according to need and available resources.
Surprisingly, in some cases, LOFF-TA outperforms fine-tuned foundation model.
Despite the simplicity of our approach, our findings indicate that there are many potential benefits, in terms of both performance and economics, to training from cached foundation features. The source code used in this work can be found at https:// cttthuub.com/miikonnuk..
2 Related work
Foundation models [3] such as LLaMA [38] and GPT [4] have revolutionized natural language processing through large-scale training, requiring huge datasets and substantial computational resources. In a rapidly advancing competitive landscape, these models are
It was not possible to train VIT-G on single GPU with batch size of 64. Instead we report the memory footprint across NVIDIA Quadro RIX 8000 using distributed training.

---

Step Feature extraction
Foundation modell Fi
Pataap
Pocking
Tensor sugmentitionn modide
Step Classifter training
CLd segeneend
Tenoet augmemaatoro
clauifrr
Figure/ 2: Overview of LOFF-TA. Step 1: We leverage a foundation model process the training data and store the extracted features offline. Step 2: The cached tensors are loaded, tensor augmentations are applied, then the augmented tensors are passed through projection and normalization layers and used to train a lightweight classifier. The tensor augmentations include spatial-based transforms, such as flips and crops, along with additive Gaussian noise. An optional pooling step (dashed operation) reduces the spatial dimension of the stored features, allowing for training with high-resolution images at no additional cost.
scaling up dramatically to achieve unprecedented capabilities. In computer vision, this trend is mirrored with models like DINOv2 [35], CLIP [37], OpenCIIP [19], BLIP [30], SAM [23] and SEEM [47].
The standard approach for adapting a foundation model for a specific task involves fine- tuning its parameters integrating additional trainable layers or classifiers [3, 13]. However, as the sizes of foundation models expand, these computational demands will escalate, posing significant challenges. To overcome these challenges, researchers have developed several strategies. These include fine-tuning only a subset of a model's parameters [46] and using techniques like gradient check-pointing for resource optimization [7]. Adaptation methods like [31, 17, 6, 18] introduce a limited number of tunable modules within the foundation model and incorporate learnable 'prompts' in the stem [20]. The result from these approaches is an efficiently fine-tuned, slightly bigger foundation model. Prior to deep learning, machine learning typically involved two-stage process: first extract relevant features from data, then train a model on these features [2]. LOFF-TA, with its caching of features, echoes this latter approach [15]. As such, it complements the adaptation methods [31, 17, 6, 18, 20] and can be used in conjunction with them to improve overall performance, as show in Section 5.5.
Developing novel image augmentation strategies mostly rely on visual insights to design augmentations [45]. The underlying principle is the manifold hypothesis [5] and effective augmentations should not move samples too far from the image manifold. While previous work feature augmentations relied interpolating features of different images [42] and adding noise to these interpolations [11] to ensure the augmented features stay close to the manifold, work, we show that it is surprisingly beneficial to apply spatial augmentations to features, in manner similar to image augmentations.
Augmentations.
find that feature augmentations need not be limited. In this
Methods
In this work, we propose to pass training data through the foundation model, cache its features, and use them to train a lightweight classifier. We term this Learning from Offline Foundation Features (LOFF). A key challenge LOFF's inability incorporate image augmentations. To overcome this, apply spatial and noise tensor augmentations directly
3

---

to the pre-stored foundation features, resulting in LOFF-TA. Finally, to allow the use of high-resolution images, we pool the foundation features.
3.1 LOFF
Our approach embraces a straightforward yet powerful idea: to separate feature extraction using a powerful foundation model from the training process using a lightweight classifier, as depicted in 2. We use foundation model to process training data upfront, then extract and store the output features (Step 1). These serve as rich representations of the data which can be retrieved at a later time for training one (or more) classifiers, as depicted in Step 2.
In detail, given a dataset D, we generate and store a new dataset T' by applying foundation model Fo to all images from D. Each sample f in I' is a k x d dimensional tensor where k is the number of tokens for each input image x and d is the embedding dimensionality.
= E
Rkddd RR
t= Fx(x))
(1)
Importantly, this phase only needs to be performed once. The extracted features, T, along with the corresponding labels for each image, are cached and used as replacement for D. The offline foundation features T are retrieved to train a lightweight classification model Fg. Finally, we employ standard cross-entropy loss, denoted as L:
arg min L(y, Fo(PPro;(t)))
(2)
0
where, is the true label, f E T' the stored features and Pproj is a projection module. The role of the projection module Pproj is to match the cached features to the dimensions expected by the classification model. Pproj consists of a learnable linear projection layer followed by Layer Normalization (LN) operation [1], which we found improve performance.
High-resolution images Many foundation models are trained handle images larger than 224 x 224, such as [,3, but GPU memory limitations on conventional hardware make impossible for them to be fine-tuned on high-resolution images. LOFF can mitigate this issue by pooling the features before storing them as shown in Fig. 2 (Step 1). The reduced size of the pooled features allow them to be stored and used to train a classifier efficiently.
We investigate both average pooling and max pooling operations [27] and assess their impact on classification performance and computational cost.
3.2 LOFF-TA
Given that image augmentations crucial for effective model training, the inability to apply them in the LOFF framework poses a significant challenge. The obstacle lies in the impracticality of storing tensors from augmented images during Step 1, which would result in prohibítive storage demands. To address this drawback, we introduce tensor augmentations (t) the features t E T. A tensor augmentation module is applied dynamically online, before a batch of features is fed into the classifier, as depicted in 2. This method, named Learning from Offlie Foundation Features with Tensor Augmentationns (LOFF-TA), allows augmentations to be used for overcoming the aforementioned obstacle with image augmentations.
In a standard setting, the objective for image classification is given by
arg min L(y, Fo Fo(Aimg(x))
where Fg is the model to trained on each sample (x, y) from the dataset D and Aing indicates stochastic image augmentations. LOFF-TA changes the optimization task to become
arg min Lly, Pe(Oproj(Aeerseee (!))))
(4)
where Atensor denotes our tensor augmentation operator. The stored features I T and projection module Pproj remain the same as in LOFF.

---

Table 1: Main results. We train models on features extracted by DINOv2 [35] ViT-B and VIT-G models. We report the results using LOFF (no augmentations) and LOFF-TA (with tensor augmentations). We consider features extracted from 256 256 and 512 512 images (using pooling as described in 3. Frozen + linear and Unfrozen + linear are points of comparison consisting of frozen/ unfrozen foundation model with linear layer trained on images directly, with image augmentations.
Method
Size
APTOS,
Aoc. 10,000
DDSM, AUCT 10,239
ISIC, Rec. 1
NABirds, Acc. 45,562
TP, Im/ec Train (Infer.)
Mem.Ch Training
Frozen linear
236
88.6 = 0.3.
+.1
9
51.7 =
860 = 0.1
(313)
1.8
9
VIL-B
LOFF-IA
236
90.4 0.6
92.3
9
728
83.
227 (236)
13.2
LOFP Pool
LOFF-TA
512
89.4
+
95.3
95.5
74.3
862
(61)
(61)
13.2
Unfrowen linear
236
4 0.9
93.7 0.8
93.3 a
76.8 0.7
85.8 a
77
28.2
Frosen linear
256
88.2 0.3
92.8 02
9.
66. 1.1
89.8 0.2
(4))
7.2
VIF-G
LOFF-TA
LOFF
236
88.6 1.5
89.9
4..
94.0 0.2
9.3
76.0 0.7
73.1
88.5 0.2
874
218 (27)
222 (27)
13.2
13.2
LOFF-TA Pool
LOHE +
512
90.3
0.3
+4.6+002
9.6
79.9 0.2
74.0
90.1
218
222
13.2
13.2
Unfrozen linear
256
0.
96.2
9..
6.
902
6(28)
345.
3.3 Tensor augmentations
Spatial relationships in image data are crucial for understanding the content of the image. Directly applying augmentations haphazardly the unstructured output tokens from the foundation model may lead to undesirable results. A key aspect of our approach involves the utilization of spatial tensor augmentations during the training phase, as depicted in 2. These augmentations are chosen to consider the spatial relationships in the data, similar how image augmentations operate. We apply spatial augmentations, denoted as Atensor, to foundation features i E after reshaping operation. These augmentations are conceptually analogous to image augmentations, treating the foundation features as they were oowrresolution, hyper-spectral images. We select set of spatial augmentations suited for feature-level transformations, chosen to enhance the training process while maintaining the integrity of spatial relationships. We flip by mirroring the tensor on its height or width axis, resize by upsampling or downsampling its spatial dimensions using linear interpolation. We shear the tensor using nearest neighbor interpolation and transiate it by shifting along its spatial dimensions. We rotate the tensor in its spatial dimensions around its spatial center using nearest neighbor interpolation. In addition to spatial augmentations, we apply additive Gaussian noise with zero mean to the feature tensor, similar to [11]. Although we considered channel augmentations, analogous contrast or color augmentations in images, we did not find them to be beneficial. We also note that, like image augmentations, not all tensor augmentations types are appropriate in every setting?.
4
Experimental setup
To evaluate the effectiveness of LOFF-TA we benchmark over eleven datasets from various domains using different foundation models, model capacities and image resolutions.
4.1 Models and implementation details
Foundation models. We employ two foundation model families: DINOv2 [35] and CLIP [37] (implemented by OPENCLIP [19]) as the basis for our investigations. For the majority of our experiments, we utilize the ViT-B and ViT-G architectures.
Classifiers. LOFFand LOFF-TA train a lightweight classifier on the features from foundation model. While, in principle, any classifier can be used in this role, our experiments use
DRrT-S [39]. The classifier is initialized using IMACENET [10] pre-trained weights, as we found empirically this gave significant improvement over random initialization [14]. In some
We omit vertical flips for the SUN397 dataset [44].

---

Table 2: Expanded results on seven standard datasets. We compare LOFF (no augmentations) and LOFF-TA (with tensor augmentations) against baselines Frozen + linear, Unfrozen + linear and Frozen + DEIT-S consisting of frozen/ sozzen/unfroen foundation model with linear layer/ DEIT-S classifier trained on images directly (with image augmentations). Results are reported for features extracted from OPENCLIP [19] and DINOv2 [35] using 256 x 256 images.
Method
Oxford-1ll Pet a = 7,549
Flowers102 R 8,189
Caltech-101 - 8,77
StanfordCars 1 - 16,185
Dogn " - 20,580
SUN397 # 39,700
NABirds
TP/Mem. Im/s T/ Gb
a 48,562
Frezen linear
95.7 + 0.1
a1
967 04
879 0.1
8.1
12
860 01
153 / 1.8
DINDv2
VIT-8
LOFF-TA
LOFF
94.5 s 04
95.2 0.1
0.5
877 a 0.6
889 0.4
0,3
02
7.5
2.
83
227 13.2
228 / 13.2
Unfrozen linear
91.8 a 0.2
99.0 0.1
97.1
93.7 0.1
0.4
72
85.8
77 26.2
Dell-S
9
8.4
2
94/ 13,8
Frozen linear
962 0.1
907=00 00
9
8
782 0.1
8988
14 7.2
DINOx2
VITGG
LOFF-TA
LOFF
95.5
95.8 2 0.
0.1
96.8 a 0.3
96.
918
9.1
892
89.0
79.2 0.3
782
87.4
58.5 = 02
218/ 13.2
Unfrozen linear
96.0 0.2
0.1
97.5 0.1
915 0.5
90.1
79.6 0.6
90.2 0.1
6/ 31522
Prozen DefT-S
:0
9
a.3
8.1
787
88.9 02
13/ 18.2
Prozen
91.7=
9044:66
9
761
=
61.0
OrrNCLIP
LOFF
91.4 0.3
89.2
946+
93. 0.2
72.7 1.2
77.5
2077 0.1
VIT-8
LOTE-TA
91
91.1 07
9.1
4.1
76.
77.7 022
02
227 13.2
Unfrozen linear
92.5.002
96644 02
0.
942
80.1 6.7
7.7
79.1 0.1
101
Prosen DelT-S
92.9..02
03
0.1
914
7.5
7.
72
121 14.5
experiments, we compare against a frozen/unfroze foundation model as a benchmark, with either linear layer or DEIT-S on top as classifier.
Implementation details. In our experiments, we utilize the AdamW optimizer [32], and a batch size of 64. We incorporate a learning rate warm-up strategy and manually decrease the learning rate by a factor of 0.1 when the validation performance plateaus. For lightweight classifier in LOFF and LOFF-TA, we implement modifications to the DETT-S architecture [39]. We remove the patchifier from the model's stem and introduce a linear projection layer followed by normalization layer detailed 3.
4.2 Datasets
Our evaluation spans eleven image classification datasets, covering a diverse spectrum of object categories. We begin with datasets with high-resolution images, as this setting highlights new capabilities made possible by LOFF-TA. We include APTOS2019 [21] for diabetic retinopathy detection, DDSM [29] for identifying masses in mammography, ISIC [40, 8, 9] for skin lesion classification, AID [43] for aerial image classification, and NABirds [41] for fine-grained bird species classification. The resolution of these datasets varies, but we resize them to 512 x 512. We extend our evaluation to a number of standard 256 x 256 resolution benchmark datasets: Flowers102 [34], NABirds [41], StanfordCars [26], StanfordDogs [22], Oxford-Ill Pet [36], Caltech-101 [12], and SUN397 [44]. For each dataset, we report metrics appropriate to its specific evaluation criteria. We adhere to official train/ validationn / test splits when available, or follow [24] in their absence. Standard practice of image normalization is maintained, using the mean and variance from the training sets.
to
Results
In this section we show that LOFF-TA achieves competitive, sometimes superior, results compared to the baselines while significantly reducing memory usage and training time.
5.1 Effectiveness of LOFF-TA
We begin our analysis with Table where we focus on the cost-benefit trade-off of using LOFF-TA, considering DINOv2 as the foundation model. Our evaluation spans five datasets where higher resolution images are known to improve performance. We measure each approach in terms of performance, throughput (TP), and memory (Mem.) footprint. The study includes comparison with two key configurations: a baseline approach, where línear classifier is appended to a frozen foundation model (frozen + linear) and an upper- bound where the entire foundation model is fine-tuned in typical fashion for transfer

---

Table 3: Ablations. We systematically remove components of LOFF-TA to investigate the impact of each contribution. Results are reported with DINOv2 [35] as the foundation model, adding trivial augment [33] as an augmentation strategy.
Foundation CLS
Layer norm
Gaussian noise
Spatial
Trivial augment [33]
Oxford-Il1 Pet
Caltech-101
X
94,
95.6
X
94.9
94.8
K
X
94.9
95.4
95.1
96.2
X
95.2
96.4
96.8
learning (unfrrozen linear). Both employ standard image augmentations. In contrast, LOFF operates without augmentations, while LOFF-TA introduces tensor augmentations. We also explore the effects of pooling on foundational features, increasing image resolution from 256 to 512, to understand how these adjustments influence the performance and efficiency.
Our observations reveal several trends. Firstly, all LOFF and LOFF-TA variants (with exception of NABirds) surpass the baseline in performance, with only slight increase in memory usage but significantly faster training. Secondly, LOFF-TA consistently outperforms LOFF, confirming the importance tensor augmentations. Moreover, upgrading to a larger foundation model (from ViT-B ViT-G) doesn't alter memory or training speed. Pooling and working with higher resolution further improves performance, again without affecting memory or throughput. Remarkably, in many cases (6 out of 10), LOFF-TA with pooling exceeds the performance of the intended upper-bound model (unfrozen + linear). Finally, it's worth noting the training speed and memory efficiency of LOFF and LOFF-TA compared to fine-ture tuning the foundation model (unfrozen + linear); our approach offers a 37x acceleration during training, and a remarkable savings in memory as well in fact, ViT-G too large to fine-tune on a conventional GPU with a batch size of 64.
5.2 Further evidence
In Table 2, we continue our analysis in a similar fashion to Table but in this case focusing on seven standard visual object recognition datasets. Our focus remains on assessing performance, throughput (TP), and memory usage (Mem.), but with image resolution of 256. The analysis compares LOFF, LOFF-TA, frozen + linear, and unfrozen + linear, and introduces frozen DEST-S benchmark for image vs. tensor augmentations (see Section 5.3).
frozenn
The findings in Table reinforce the patterns observed in Table 1. LOFF-TA maintains its superiority over LOFF, highlighting the efficacy of tensor augmentations. It's important to note that in this set of experiments, the best performing LOFF-TA + Pooling configuration using high-resolution images is not considered due to the datasets' inherent resolution limit of 256. Despite this, LOFF-TA generally matches or surpasses the baseline frozen + linear in performance, while delivering a notable increase in throughput. Although frozen + linear claims the smallest memory footprint, LOFF and LOFF-TA again show significant savings compared fine-tuning a foundation model (unfrozen linear). Comparing foundation models of similar capacity, we notice that DINOv2 outperforms OPENCLIP in five of the seven datasets using LOFF-TA. Finally, we once again see that swapping the foundation model (between DINOv2 VIT-B, OPENCLIP ViT-B, and DINOv2 ViT-G) results in no appreciable change in throughput or memory consumption for LOFF-TA, only differences in performance.
5.3 Image vs. tensor augmentations
Looking at Table 2 again, we investigate the effect of image augmentations versus tensor augmentations. LOFF-TA and frozen DETT-S share the same architecture (a cascaded model that consists of a frozen foundation model with an appended DrTT-S), and in both cases the foundation frozen - the only difference is LOFF-TA uses tensor augmentations while frozen + DEiT-S uses image augmentations. We also report results (Table 5 in the Appendix) when we unfreeze the foundation model in the cascaded setting for both OPENCLIP and DINOv2 for completeness. In Table 2, we observe that image augmentations outperform
7

---

tensor augmentations, As one might expect, tensor augmentations such as tensor rotation or cropping can not replicate the exact effects of image rotation or cropping since foundation models are not linear operators. Yet, the performance impact is surprisingly less than anticipated, which, considering LOFF-TA's significant computational savings, noteworthy.
5.4 Ablation study
In this section, we try to understand the impact of each contribution to LOFF-TA. Utilizing DINOv2 [35], we conduct a study where components are systematically removed and then tested on the Oxford-III Pet and Caltech-101 datasets. Results are presented in Table 3.
CLS token. A perhaps obvious, but critical, insight from our study is the important role played by the foundation model's CLS token in contributing to classifier performance. When this token, representing global information, is integrated into the classifier training, it proves to be a key contributor to performance gains. However, the choice of how to integrate it was not trivial. We opted to integrate the offline CLS token from the foundation model with the learned CLS token of the classifier model by summation. Other possible ways to incorporate could be to initialize the classifier's CLS to the offline foundation CLS, or concatenate them although our non-exhaustive experiments testing these approaches were inferior.
Layer norm. We applied Layer Norm [1] operation in the projection layer at the stem of the classifier. Similar to the CLS token ablation, we find that adding a layer norm operation before passing the foundation features to the classifier boosts performance. By centering input features, we believe the normalization plays a pivotal role similar to that in Dual Patchnorm [28], aligning and standardizing the input.
Pooling. Table 1 shows that pooling larger image features enhances performance without extra computational costs, enabling the use of large models for high-resolution tasks. In Appendix Table 6, we compare max and average pooling, finding both perform similarly, with max pooling slightly outperforming in larger models, possibly due to better noise reduction. Thus, max pooling recommended for larger models. While this work focuses on pooling, alternative dimensionality reduction methods like strided convolutions or bi-linear interpolation could also be effective.
Augmentations. Since the cached features contain long-range spatial information that may potentially be harmed by our tensor augmentations, is important to assess if these augmentations have a meaningful contribution to the performance of LOFF-TA. Comparing Row 3 and Row 4 of Table 3, we see that adding spatial augmentations results in significant boost in performance.
smaller boost is observed when Gaussian noise is added in Row 5. Replacing our augmentations with Trivial augment [33] provides further boost to performance, although this setup was not used in our other experiments.
5.5 LOFF-TA vs. foundation adaptation methods
LOFF-TA enables practitioners to use large foundation models without any modifications. A class of methods exists that modify foundation models for new tasks, so-called adaptation methods. Adaptation methods can be used in conjunction with LOFF-TA, but also provide a comparison between them in Table 4. The results show that standalone LOFF-TA achieves competitive performance with VPT [20], Adaptformer [6] and SSF [31]. When LOFF-TA is combined with these methods, we observe noticeable improvements across the board, demonstrating that computationally efficient SOTA performance be achieved by adapting foundation models and then applying LOFF-TA.
6 Discussion
Performance-efficiincy trade-off. The aim of this paper is to consider the use of foundation models in resource-limited setting. To do so, we propose to work with cached foundational features. The benefits of doing so are that training speed is significantly accelerated (up to 37x), and GPU memory usage is significantly reduced (up to 26x). Of course, these benefits
8

---

Table 4: LOFF-TA can be used alongside foundation adaptation methods. Below we report results using ViT-B. Standalone LOFF-TA performs comparable with other foundation adaptation methods and can easily be combined with them further enhance performance.
Method
AITOS,,
AID, Acc. = 10,000
DDSM, AUC T " 10,239
ISIC, Rec. T " = 25,333
NABirds, Acc. T : 48,562
n = 3,662
LOFF-TA
90.4 + 0.6
92.3 + 0.7
94.4 0.1
72.8 1.7
83.5 : 0.3
VPT [20]
89.6 + 0.1
93.0 0.1
91.4 + 0.3 92.4 0.3
75.2 1.1
85.8 0.2
VPT LOFF-TA
90.8 + 0.4
93.1 # 0.3
79.7 + 0.9
83.7 0.1
SSF [31]
90.2 0.1 91.1 + 0.7
92.1 + 0.2
96.7 0.6
76.4 0.9
88.2 + 0.0
SSF + LOFF-TA
93.1 + 0.0
97.2 0.3
81.6 + 1.5
85.6 0.1
AdaptFormer [6]
89.6 # 0.6
94.3 0.1
91.8 0.8
82.6 1.0
87.1 # 0.3
AdaptRormer + LOFF-TA 90.0 + U.3
94.3 : U.2
93.2 # 0.5
83.5 + 0.3
85,3 # 0.2
come with trade-offs. In most cases, LOFF-IA will perform slightly worse than directly fine-tuning the foundation model, although we did observe some cases where LOFF-IA was superior. Also, the computational cost savings at training time do not translate to inference.
An important benefit of LOFF-TA that it affords flexibility, allowing practitioners to choose the most suitable foundation model for their task from range of sizes and pretraining methods (refer to Table 2), including the capacity to handle high-resolution images (see Table 1). The caching and pooling of features makes these different choices come with equivalent computational costs during training, determined by the compact classifier.
Societal impact: As foundation models grow, their computational demands are expected to increase, potentially creating digital divide where advanced models are accessible only to well-resourced organizations, excluding smaller entities and individual researchers. This trend underscores the importance of developing efficient methods to utilize these models, ensuring broad access and fostering progress in various application areas. There is significant potential to impact fields like medical imaging and remote sensing, which can greatly benefit developing regions and marginalized groups. But these applications often require high-resolution image processing, which is resource-intessive. LOFF-TA facilitates wider, fairer access to sophisticated foundation models in resource-limited settings.
Limitations: LOFF-TA is a training strategy meant for low resource settings. Given enough computational resources, full fine-tuning will generally outperform LOFF-TA or adaptation methods like SSF [31]. Another limitation LOFF-TA is that at inference time it is less efficient than standalone foundation models. However, we note that due to their latency, foundation models are costly to be deployed scale without distillation [16] (or some other measure) to limit cost which would also alleviate the limitation for LOFF-TA.
7 Conclusion
Foundation models have significantly impacted the community and will likely become more crucial they grow in size and capability. However, adapting these models for specific tasks is increasingly challenging due to their high resource demands. Revisiting the classical machine learning paradigm of perceive, then reason is pragmatic this context. Using foundation models as powerful feature extractors allows us to benefit from their rich representations, while at the same time mitigating computational costs by training lightweight classifier on their cached features. LOFF-TA achieves a low memory footprint and high throughput during training, regardless of the chosen foundation model or input image size. It is particularly beneficial for high-resolution image processing when pooling is applied, allowing LOFF-TÁ to outperform linear classifiers using frozen foundation models and compete with surpasses fine-tuned foundation models.
Acknowledgements. This work was supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) and the Development and Promotion of Science and Technology Talents Project. We acknowledge the Berzelius computational resources provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.

---

References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. "Layer normalization". In: arXiv preprint ar xix:::6607 Ntt:1607.004535 .06450 (2016).
[2] Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning.
Vol. 4. Springer, 2006.
[3] Rishi Bommasani et al. "On the opportunities and risks of foundation models". In: arXiv preprint Sttt:2108.072555 (2021).
[4] Tom Brown et al. "Language models are few-shot learners". In: Advances neural information processing systems 33 (2020), pp. 1877-1901.
[5] Lawrence Cayton et al. Algorithms for manifold learning. eScholarship, University of California, 2008.
[6] Shoufa Chen et al. "Adaptformer: Adapting vision transformers for scalable visual recognition". In: Advances in Neural Information Processing Systems 35 (2022), pp. 16664- 16678.
[7] Tianqi Chen et al. "Training deep nets with sublinear memory cost". In: Xiv preprint arXiv:1604.06111 (2016).
[8] Noel CF Codella et al. "Skin lesion analysis toward melanoma detection: A challenge the 2017 international symposium on biomedical imaging (isbi), hosted by the inter- national skin imaging collaboration (isic)" In: 2018 IEEE 15th international symposium biomedical imaging (ISBI 2018). IEEE. 2018, pp. 168-172.
[9] Marc Combalia et al. "BCN20000: Dermoscopic lesions in the wild". In: Xiv preprint ar Arx::81908.0 002288 (2019).
[10] Jia Deng al. "Imagenet: A large-scale hierarchical image database". In: 2009 IEEE conference on computer vision and pattern recognition. leee. 2009, pp. 248-255.
[11] Terrance DeVries and Graham W Taylor. "Dataset augmentation in feature space". In: Xrviv preprint ar Xie:1702.00538 (2017).
[12] Fei-Fei, Rob Fergus, and Pietro Perona. "Learning generative visual models from few
training examples: An incremental bayesian approach tested 101 object categories". In: 2004 conference on computer vision and pattern recognition workshop. IEEE. 2004,
PP. 178-178.
[13]
Xu Han al. "Pre-trained models: Past, present and future". In: AI Open (2021),
PP. 225-250.
[14] Kaiming He et al. "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification" In: Proceedings of the IEEE international conference on computer vision. 2015, Pp. 1026-1034.
[15] John Hennessy and David A Patterson. Computer architecture: quantitative approach. Elsevier, 2011.
[16] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in neural network". In: arviv preprini arXiv:1503. 02553 (2015).
[17] Neil Houlsby et al. "Parameter-efficient transfer learning for NLP". In: International Conference on Machine Learning. PMLR. 2019, pp. 2790-2799.
[18] Edward Hu et al. "Lora: Low-rank adaptation of large language models". In: arv preprint ar 21ix::2000006883 (2021).
[19] Gabriel Ilharco al. OpenCLIP. If you use this software, please cite as below. July 2021. 5143773. URL: https: http:://ddo..ooog/// wrrg/10.......oo... 5281/220odo 5143773.
DOL: 10.
[20] Menglin Jia et al. "Visual prompt tuning". In: European Conference on Computer Vision.
Springer. 2022, pp. 709-727.
[21]
Sohier Dane Karthik Maggie. APTOS 2019 Blindness Delection. 2019. URL: https: / //kaggle. com/ compettiionss/pptss/it httns//ppttt22ll--iinnneeee-e--e
[22] Aditya Khosla et al. "Novel dataset for fine-grained image categorization: Stanford dogs". In: Proc. CVPR workshop on fine-grained visual categorization (FGVC). Vol. 2. Citeseer. 2011.
[23] Alexander Kirillov et al. "Segment anything". : In: ar Xiv preprint arXio:2304.0262 (2023).
10

---

[24] Simon Kornblith, Jonathon Shlens, and Quoe V Le. "Do better imagenet models transfer better?" In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019, pp. 2661-2671.
[25] Simon Kornblith et al. "Similarity of neural network representations revisited". In: International Conference on Machine Learning. PMLR. 2019, pp. 3519-3529.
[26] Jonathan Krause et al. "3d object representations for fine-grained categorization', In: Proceedings of the IEEE international conference on computer vision workshops. 2013, Pp. 554-561.
[27]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. "Imagenet classification
with deep convolutional neural networks". In: Communications of the ACM 60.6 (2017),
Pp. 84-90.
[28] Manoj Kumar, Mostafa Dehghani, and Neil Houlsby. "Dual PatchNorm". In: arv preprint arXio:2302.0132 (2023).
[29] Rebecca Sawyer Lee et al. "A curated mammography data set for use in computer-aided detection and diagnosis research". In: Scientific data 4.1 (2017), pp. 1-9.
[30] Junnan Li et al. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision- Language Understanding and Generation. 2022. arXiv: 2201. 12086 [cs.CV]
[31] Dongze Lian et al. "Scaling & shifting your features: A new baseline for efficient model tuning". In: Advances in Neural Information Processing Systems 35 (2022), pp. 109-123, [32] Ilya Loshchilov and Frank Hutter. "Decoupled weight decay regularization". In: arviv preprint 1xXi:1111.051000 (2017).
[33] Samuel G Müller and Frank Hutter. "Trivialaugment: Tuning-free yet state-of-the-a data augmentation". In: Proceedings of the IEEE/CVE international conference on computer vision. 2021, pp. 774-782.
[34] Maria-Elena Nilsback and Andrew Zisserman. "Automated flower classification over a large number of classes". In: 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. IEEE. 2008, pp. 722-729.
[35]
Maxime Oquab et al. "Dinov2: Learning robust visual features without supervision". In: ar Xiv preprint ar Xio:2304.007199 (2023).
[36]
Omkar M Parkhi et al. "Cats and dogs". In: 2012 IEEE conference on computer vision and puttern recognition. IEEE. 2012, pp. 3498-3505.
[37] Alec Radford et al. "Learning transferable visual models from natural language supervision". In: Infernational conference on machine learning. PMLR. 2021, pp. 8748- 8763.
[38] Hugo Touvron et al. "LLaMA: Open and Efficient Foundation Language Models". In: arXiv preprint ar xit::2302..1397 (2023). URL:
Languau
[39]
Hugo Touvron al. "Training data-efficient image transformers distillation through attention. arXiv 2020". In: ar Xiv preprint arXie:2012.12877 (2020).
[40] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. "The HAM10000 dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions" In: Scientific data 5.1 (2018), pp. 1-9.
[41] Grant Van Horn al. "Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection", " In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, PP. 595-604
[42] Vikas Verma et al. "Manifold mixup: Better representations by interpolating hidden states". In: International conference machine learning. PMLR. 2019, pp. 6438-6447,
[43] Gui-Song Xia et al. AID: A benchmark data set for performance evaluation of aerial scene classification". ". In: IEEE Transactions on Geoscience and Remote Sensing 55.7 (2017),
pp. 3965-3981.
[44] Jianxiong Xiao al. "Sun database: Large-scale scene recognition from abbey to zoo". In: 2010 fEFE computer society conference on computer vision and pattern recognition. IEEE. 2010, pp. 3485-3492.
[45] Suorong Yang et al. "Image data augmentation for deep learning: A survey". In: Xiv preprint 22:4: (2022).
11

---

[46] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. "Bitfit: Simple parameter- efficient tinettuing for transformer-based masked masked language-modell" II: ar Xio preprint arXim:2106.1011 (2021).
[47] Xueyan Zou et al. "Segment everything everywhere all at once". In: ar Xio preprint arXio:2304006718 (2023).
12

---

Appendix
A Representation similarities
To examine the similarities between classifiers trained on images and trained on tensors using LOFF, we employed Centered Kernel Alignment (CKA) [25], technique commonly used analyze representational similarity in neural networks. Figure presents the CKA analysis results for classifiers trained on Oxford-II Pet. In the top panels, we observe that the LOFF classifier's low-to-mid layers exhibit similarity to the higher layers of the image- trained classifier, indicating that LOFF learns features similar to the image-trainets classifier's high-level features. This suggests that LOFF's offline features are sufficiently informative for learning high-level features earlier in the network, with subsequent layers adapting to novel features. The middle panels demonstrate that the cascaded frozen foundation model classifier displays similar behavior to the LOFF classifier. The right panel reveals high similarities between corresponding layers of the frozen foundation and LOFF classifiers, particularly in the earlier layers, indicating shared learned features. The bottom panel illustrates the internal representational similarity of each classifier with itself before and after fine-tuning. The left panel shows strong similarity between the image-trained classifier's layers, indicating the retention of pretrained features during fine-tuning, Conversely, the middle and right panels indicate that the frozen foundation and LOFF classifiers retain similarity in the low-to-mid layers while learning new features in the higher layers.
Figure CKA similarities betwven different models. Left: Representation similarity of different classifiers after fine-tuning on Oxford-IIJ -le. Right: Representation similarity of the internal layers of each classifier with itself before and after fine-tuning.
B Unfreezing the Cascaded Models
We evaluate the model's performance in the cascaded setting (foundation model + DEIT-S classifier) when the whole model is fine-tuned. We present our findings in Table 5, comparing the use of DINOv2 and OPENCLIP foundation models. Once again, we find that DINOv2 outperforms OPENCLIP, in most cases. When DINOv2 is unfrozen, either surpasses or matches the performance of its frozen alternative (see Table 2), though the margin of improvement is rather small. OPENCLIP shows a similar trend, but its performance seems more dependent the dataset it is evaluated appear to outperform their frozen alternatives. However, their computational cost makes them prohibitive for larger models image sizes.
Overall, unfrozen foundation models
Table 5: Fine-tuning a foundation model + DEIT-S classifier. We append a DRIT-S classifier after a DINOv2 [35] or OPENCLIP [19] foundation model and the whole cascaded model.
Foundation
Oxford-111 Pet 7,349
Flowers102 8,189
Caltech-101
StanfordCars 16,185
Stanford 20,580
SUN397
NABirds
8,677
39,700
48,562
DINOv2 VIT-B
95.2 0.2
99.3 0.1
97.4 0.2
93.9 0.4
87.0 0.3
76.2 0.7
86.3 0.2
OLINCLIP ViT-B
91.8 a 0.5
95.4 = 1.2
96.8 a 0.2
94.5 a 0.2
81.3 a 0.3
7488 a 0.2
77.8 0.1
13

---

C Tensor augmentations
In our experiments, we consistently observe performance improvement when applying tensor augmentations. However, the need for task-specific and domain-appropriate aug- mentation strategies should be emphasized. For example, in scene classification using the SUN397 dataset, applying vertical flips to images when training DEIT-S (pretrained on (MageNet) led to decreased accuracy (-2%), likely due the unrealistic expectation of upside-down building facades during testing. One might expect similar effect with tensor augmentations, since the feature space preserves the spatial orientations of objects (see Figure 4). Interestingly we vertical flips as tensor sugmentations on this dataset (-0.2%). This suggests LOFF-TA's 's tensor augmentations exhibit greater resilience to improper augmentations compared to the image domain. This observation merits further exploration to understand its underlying mechanisms and consequences.
much smaller performance drop when we apply
Intriguingly, LOFF-TA demonstrates benefits from spatial tensor augmentations despite their potential conflict with the spatial information present in feature tokens. Feature tokens in foundation models carry positional data, informed by positional embeddings and attention mechanisms. Spatial augmentations, such horizontal flips, disrupt this positional context, yet our experiments, especially with Trivial Augment, show a notable performance enhancement (see Table 3). This paradox suggests that these disruptions may actually bolster the classifier's 's ability to learn robust features, thereby improving classification outcomes. The precise dynamics of this phenomenon remain unclear, presenting exciting avenue for future research, particularly in the realm auto augment strategies for foundation features.
Figure 4: Robustness and spatial consistency of features. Images along with a random channel of the corresponding foundation features reveal the spatial consistency between objects in the image and feature spaces. This consistency allows insights from the image space to guide tensor augmentation choice, egg. if vertical flips are harmful for a building facade dataset in image space, they are likely to be harmful in feature space. However, we observe that training with LOFF-TA is more robust against 'incorrect' augmentation choices compared to standard classifier training on images.
D Pooling ablation
Table 6: Pooling enables larger resolution. We compare different approaches to pooling used for LOFF and LOFF-TA: pooling, max pooling, and average pooling. Results are reported for foundation features extracted by DINOv2 [35] ViT-B and ViT-G models from standard 256 256 without pooling versus larger 512 images with pooling (note that the compute costs are equivalent).
Method
Pooling
Size
APTOS,
AID, 10,000
DDSM, AUC 10,239
ISIC, Rec. 25,333
NABirds, Acc. 48,562
и 3,662
LOFF
89.6 0.2
9..
94.2 1.2
70.8 2.1
83.0 2 0.1
LOFF-TA
256
904
92.3 0.7
94.4
72.8 1.7
83.5 + 0.3
ViT-B
LOFF
90.3 0.2
0.7
95.7 0.3
7.1
86.3 0.3
LOFF-TA
Average
512
90.7 0.8
93.7 + 0.5
96.1 +.1
77.9
86.7 + 0.3
LOFF
89.4 1.5.
93.2 0.6
95.3
1.5
86.2 0.3
Max
512
LOFF-TA
90.5 = 1.0
93.7 + 0.3
95.5 0.1
774 0.0
86.8 + 0.4
LOFF-TA
LOFF
x
256
88.6 1.5
89.9
93.3 + 0.5 9..0
9.1
94.8 +
7.70
73.1 0.5
87.4 0.2
88.5
Average
512
89.0
94.0
96.0 1.010.
77.5
89.0 0.2
LOFF-TA
90.1
9.23
9.6
7.84
90.0 0.2
0.6
94.1
95.4
74.0
0.1
LOFF-TA
Max
512
91.8 0.3
94.6 0.2
96.3
79.9 0.2
90.1 0.2
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provided a list of contributions in Section Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and / /or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification: We have listed the limitations of our proposed method and under which assumptions it works as described.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech- to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
15

---

Question:
For each theoretical result, does the paper provide the full set of assump- tions and a complete (and correct) proof?
Answer: [NA]
Justification: This is an empirical study and we do not report any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems,
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be com- plemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main ( claims and/ / or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We provided the architecture details, hyperparameter selection details ie. default hyperparameters except the learning rate and schedule for which we provided the selection strategy. We also listed the set of augmentations we utilized. We will provide the full training logs along with the configuration files (stored online at wandb. com) after the review process.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/ /or/ model, the authors should describe the steps taken make their results reproducible verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., large language model), then there should either be a way to access this model for reproducing the results or way to reproduce the model (e-g., with an open-source dataset or instructions for how to construct the dataset).
16

---

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model limited some way (e.g,, registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The datasets are all public and sources are referenced in the manuscript. We will provide the code after publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlP'S code and data submission guidelines (https://nips. cc/ cc/public/ httc//ggiie//eeeeeeeeeeeeeeeeeee
www.aimiiiinnna.. ssionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central the contribution (e-g. for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submis- sion guidelines for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for
the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicablle.
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We have used the default settings defined in the original papers. We referenced which dataset splitting guidelines we used in the absence of official splits. We listed the optimizer we used and described our learning rate selection strategy.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
17

---

Answer: [Yes]
Justification: Wepeptee our experiments with different random seeds and reported the standard deviation of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/ test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer:
[Yes]
Justification: We provided the GPU memory and throughput for both training and inference for all experiments, along with the hardware specifications.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e-g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ //tt:///neu....eee cttsssccpppuiii htc/public/Ethicsiiii deisesssdei lines?
Answer:
[Yes]
Justification: No human subjects or crowdsourcing were used during the study. We adhered to the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
18

---

The authors should make sure to preserve anonymity (e.g,, if there is a special consideration due laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We provided societal impact statement addressing potential positive and negative impacts of the study.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g,, disinformation, generating fake profiles, surveillance), fairness considerations (e.g,, deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
the quality
The authors should consider possible harms that could arise when the technol- ogy being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our study is about efficient training strategies and it does not pose high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
19

---

Question: Are the creators or original owners of assets (e.g,, code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited all datasets and pretrained weights used in this study, We referenced both the original versions these assets and modifications to them.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g, CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the
license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We will release the code after publication. No other assets will be released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/ code// model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer:
[NA]
Justification:
No human subjects crowdsourcing were used during the study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
20

---

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, partlppaaaiiiii whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/reviee based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects crowdsourcing were used during the study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "UKfuq0fx4S", "reviewer": "Reviewer_PGYR", "summary": "This paper proposed a framework to efficiently use foundation features for online serving. The idea (offline feature extraction from foundation model and online inference with lightweight model) is simple but interesting/effective.", "questions": "1. I agree the tensor augmentation is reasonable for image embeddings (such as flip, rotate, translate and resizing). Is there any way to expand the the tensor augmentation w.r.t spatial operators to the video embeddings?", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and easy to follow.\n2. The paper proposed a simple yet effective framework on how to efficiently use visual foundation model features for online serving.\n3. The experimental results demonstrate the proposed method is promising.", "weaknesses": "1. Can you please list the classification accuracy of the raw visual foundation models (VFM)? If the performance of the raw VFM is high, we can directly output the probability score and save them in a database instead of extracting features offline and then do online inference with a followup lightweight model.\n2. I'm concerned the potential application scenarios of the proposed scheme is limited, since it may only be able to be leveraged on image classification scenarios.\n3. When the authors claim efficiency improvement, the baseline is somewhat unfair, because the proposed methods actually need an offline feature extraction step which would also consume computation resources.\n4. The tensor augmentation is actually limited to several spatial augmentations, while there are many for raw image classification, for example, color-jittering and mixup..", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments.\n\n**Weaknesses**\n\n1 - Reporting such an accuracy is not possible. Generally, the classes included during pre-training and fine tuning are different, e.g. self-supervised pre-training of Dino vs. supervised fine-tuning on retinopathy images. Furthermore, the output dimensionality of pre-training and fine-tuning tasks would rarely match, thus a direct mapping for accuracy measurement is difficult, e.g., APTOS2019 having 5 classes vs. the very high dimensionality of Dino outputs during pre-training.\n\n2- “The method is for image classification” cannot be a limitation. We ask the reviewer to consider many valuable studies in the literature only addressing image classification tasks.\n\n3 - Offline feature extraction is run only once, without computing or storing gradients, and can be done with any batch size. This cost does not affect computational requirements in any meaningful way. Due to the reviewer’s request, we ran a small experiment and found embedding generation to take approximately 1/10th of one fine-tuning run, most of this time being spent on read/write operations on a HDD. Recognizing that in most experiments the bulk of computation is spent on hyperparameter search, the time spent on a *single* no-gradient forward pass through the dataset is not impactful.\n\n4 - As we discussed in the paper, our results are surprising exactly because tensor augmentations are more limited and less well-researched than image augmentations yet their impact is comparable. We believe our work can pave the way for more research regarding tensor augmentations.\n\n**Questions**\n\n1 - All of our proposed tensor augmentations can be directly applied to video embeddings along with uniform frame sampling or tubelet embedding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ekWiW2eAbM", "reviewer": "Reviewer_TzTJ", "summary": "The authors propose to store low dimensional representations of images that are obtained after passing them through pretrained (foundation) models. Only afterwards augmentation strategies (\"tensor transformations\") for training the downstream classifier are employed. This has the benefit that the foundation model is not part of the training processm, which is computationally cheaper. More importantly, some dataset specific optimization is obtained, without the need of finetuning the complex / expensive foundation model itself.  The authors present a set of relevant tensor transformation and compare against the baseline.", "questions": "* See above", "limitations": "All very well done.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The authors follow an interesting idea - using (and augmenting) offline features rather than full images for training.\n* Using the foundation model as a \"feature extractor\" is interesting. Even more interesting is identifying a set of operations that mimick standard data augmentation in the resulting feature / tensor domain\n* Results are promising. To the best of my knowledge the concept of the tensor transformation is novel.\n* The paper is well written", "weaknesses": "* It would have been interesting to study tensor augmentations in a more systematic / more formal way\n* I am not sure why this approach would be particular suited for high-resolution images. The authors describe how foundation model optimized for large images can be used to deal with small images via the offline eature / tensor augmentation approach. This implies that the presented method is particulary well suited deal with_low-resolution_ images, but not the opposite, isn't it?\n* Using a pretrained (foundation) model as a feature extractor is not necessarily an overly innovative idea.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments.\n\n1- We have made efforts to choose our wording carefully and clearly outline the limitations of our study. We welcome any suggestions for improvement and would be grateful if the reviewer could identify specific sections of the manuscript that might need further refinement.\n\n2 - LOFFTA caches foundation model embeddings using a single forward pass through the dataset. We apply pooling before storing the embeddings, then load them and apply tensor augmentations on reduced dimensional representations to train a small classifier. When the input image resolution increases, the GPU memory requirements normally increase quadratically, mostly due to gradient computation. Since we store embeddings from a single forward pass without gradient computation, it's feasible to use higher resolution images for this purpose. However, as noted in Table 1, using these same high-resolution images for regular fine-tuning is not practical in settings with limited computational resources.\n \nWe know that increasing the resolution improves performance in some tasks, e.g. detecting small tumors becomes easier in higher resolutions. The computational efficiency and reduced GPU memory requirements of our method enables practitioners to freely use images with higher resolutions, making it particularly suited for high resolution images. We will clarify the manuscript regarding this topic.\n\n3 - To the best of our knowledge, we are the first to employ caching in this context. We believe the simplicity of our idea is a benefit (in addition to its efficiency). However, we are happy to cite previous work that we have missed if the reviewer can point those out.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "erLcg1IYof", "reviewer": "Reviewer_1T2n", "summary": "This paper introduced a training scheme considering offline foundation features with tensor augmentations LOFF-TA, focusing on limited resource settings. They basically trained a classifier on cached features from frozen foundation models with an augmentation process to these cached embedded features. Their proposed training mechanism is claimed to speed up the training process by $37\\times$ with appx. $26\\times$ reduction of GPU memory usage. The authors validated their approach on eleven image classification benchmarks and showed similar performances as standard image augmentations and fine-tuned foundation models.", "questions": "Please see the weaknesses section. I tried to discuss all the findings and questions there. Following, please find some of my concerns and suggestions the author might want to rebut in the rebuttal\n\n1. The validation of their model on high-dimensional medical images is missing. The author might want to show very brief experiments on how their proposed approach performs on 3D medical classification tasks. \n\n2. How the offline tensor augmentations preserve the anatomical features compared to the online tensor/image augmentations? Consider the small anatomical features such as ventricles that are very important when we experiment on finding the presence of neurodegenerative disease from 3D brain MRIs.", "limitations": "The authors carefully addressed the limitations and societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**S1. Simple yet rich architecture.** I appreciate the authors' approach to augment the frozen foundation model features that are being used to train a compact classifier resulting in solid training time and memory space improvement.\n\n**S2. Strong experiments and ablation studies.** The authors validated their model on eleven image classification benchmarks under ViT-B and ViT-G backbones. The ablations studies showed the effectiveness of tensor augmentations and their variances, pooling operation under different network sizes, and presence of [foundation CLS, layer norm]. The results show promising improvements over the unfrozen-linear method under all the network backbones.\n\n**S3. Paper writing and presentations.** The paper is very well written. I like their overall presentation, justifications, and motivations.", "weaknesses": "**W1. Adaptation of spatial augmentations.** For high-resolution images, performing augmentations on cached embedded features highly depends on the size of the latent space from where the offline foundation features are being cached. Typically in the medical image domain where one needs to deal with very high-resolution images/volumes (e.g., $256\\times256\\times256$), it is necessary to consider the topological features when we generate/augment images from raw data that should not change the image class. In that case, where the authors are performing spatial augmentations on the latent features, there might be scenarios where the topological features get distorted from the flip or the classes change from other spatial augmentation operations. Though I understand that spatial augmentations might be beneficial in terms of natural images, these kinds of augmentations might suffer from experimenting on high-dimensional medical images. \n\n\n**W2. LOFF-TA generalizability on high dimensional 3D volumes.** I appreciate the authors for experimenting with diverse 2D datasets ranging from natural to medical images. However, in the medical domain, we typically follow 3D volumes for complex disease diagnoses such as classifying tumors, neurodegenerative disease, myocardial infarction, etc., which are considered to be high dimensional, e.g., $128^3$ or $256^3$, where it is pretty complex to preserve anatomical information by augmenting on low-dimensional latent features. Experimenting with this type of application on datasets like OASIS, ADNI, ACDC, Synapse multi-organ, etc. would further strengthen the argument about offline tensor augmentations on high-dimensional medical images.\n\n\n**W3. Pictorial latent space depictions.** To validate the proposed model's efficiency compared to image augmentations, the authors might want to visualize the latent feature spaces on both image/tensor augmentations. This would further ensure that offline latent tensors preserve the important spatial features similar to the latent image features. After that, augmenting the offline tensors would make more sense in the carried-out image analysis tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments.\n\n**Weaknesses**\n\nW1 - We do not think spatial tensor augmentations will harm anatomical features more than voxel space augmentations would. However, some care must be taken in the selection of the tensor augmentations. In Appendix C, we pointed out that common sense insights from the 2D pixel space carries over to the tensor space, e.g. applying vertical flips to a building facade classification task is harmful, both in image and tensor spaces. Surprisingly, we noted that the harmful impact of unsuitable augmentations are somewhat mitigated in the tensor space and we believe a similar phenomenon might occur when it comes to voxel space augmentations.\n\nW2 - We agree with the reviewer that demonstrating how our method performs on tasks with 3D volumes would be valuable. We will add these experiments to our study.\n\nW3 - We will include this to our study. We note that we made a related point in our discussion. Due to the non-linearity of the foundation models, applying a vertical flip in pixel space and then generating the embedding will be different from generating an embedding and then applying a tensor space vertical flip. However, we believe the resulting tensors in both scenarios will be visually alike (after dimensionality reduction), as those we provided in Appendix C. Thus, we believe the structures will be preserved, but some small numeric variations will be observed.\n\n**Questions**\n\n1 - We will add these experiments to our study. \n\n2 - We believe our approach will not harm the anatomical features more than feature extraction by using a foundation model does. We will empirically validate this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rLWX0m1A5F", "reviewer": "Reviewer_AUq9", "summary": "The paper discusses leveraging the large foundational models in resource-scare settings to fine-tune an image classification task. The authors propose saving the features from the frozen foundational model and then using them to train/fine-tune a smaller classifier model. They introduce tensor augmentations to improve the method's robustness and demonstrate the advantages of reduction in GPU memory usage and improved training speed and performance on different open-source dataset tasks. The performance of the proposed method is comparable to the full fine-tuning of the foundations model along with a linear layer.", "questions": "See above", "limitations": "There is a lack of clarity in some of the choices in the design. I have a few questions on some of the results and the missing ablation experiments.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This work is essential in low-resource settings, utilizing a much better backbone foundational model to improve classification accuracy. \n\n- Results show improvement over the baseline methods and simple fine-tuning as well.", "weaknesses": "- Table 1 results. Question about the inference throughput. The table shows a considerable reduction compared to the original model with linear. I assume that inference is done end-to-end, i.e., the image passed through the Foundational Model + the extra bits of the proposed LOFF-TA/LOFF pipeline. While the improved training speed is appreciable, the drop in inference is much less desirable, especially when deployed in scenarios that require faster throughput.  So although the resource saving is highly beneficial for the training scenarios, unfortunately for inference the whole system still needs to be loaded onto the GPU because caching is not possible. I would appreciate the author's insight on this. \n\n- The effectiveness of these augmentations depends heavily on the type of features extracted from the foundation model. Specifically, different layers within a neural network capture varying level of information: earlier layers focus on the spatial structure of the image, while deeper layers learn more abstract and semantic features. The paper needs to clarify what layer of the ViT features were extracted and cached. Further, there is a strong dependency on this choice of layers, and I don't see any ablations on this choice. \n\n- It's interesting that pooling actually improves the results. It's surprising because I would assume pooling reduces some of the features when reducing the dimension, but it somehow actually helps. Can the authors comment on this? It's a little counterintuitive, in my opinion. \n\n- Authors propose a $\\phi_{proj}$ adapter to match the cached features to the dimensions expected in the classifier model. Is the layer trained end-to-end along with the classifier?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments.\n\n1 - This is the main limitation of our study that we highlighted in the Limitations subsection - much like any caching strategy, there is a trade off. However, we note that foundation models are inherently less suitable for low latency inference tasks due to their size. Therefore, for practitioners planning to use foundation models, such as in medical imaging, latency should not be a primary concern from the start. It's important to recognize that in such applications, the challenge isn't inference throughput but rather the capacity to learn a model that can handle high-resolution images effectively.\n\nMoreover, if we solely focus on real-time applications, foundation models typically require distillation or similar techniques to enhance inference latency for feasible deployment. Should distillation be employed, our approach would match the latency of a standalone foundation model.\n\n2 - We used the features from the last layer, i.e., before the linear classifier. We will clarify this in the manuscript. Early on in development, we experimented with trimming parts of the foundation models to generate the offline features. Using the DDSM dataset, we found that keeping only the first 5 blocks of the Dinov2-base model to generate the embeddings yielded a slight performance boost. We did not further investigate in this direction since choosing the layer to utilize would be another hyperparameter to select with arguably limited returns. Regardless, we agree with the reviewer that the impact of hierarchically less abstract features on LOFFTA is interesting and we will add an ablation on trimming foundation layers to our work.\n\n3 - The performance boost due to pooling may seem surprising but we believe pooling helps the model better focus on features pertinent to the classification task by suppressing the noise inherent in the high dimensional feature space. The benefits of pooling is also another indicator for the robustness of the foundation model features.\n\n4 - The projection layer is trained end-to-end. We will make the necessary corrections in Eqs. 2 & 4 such that the argmin operation is over both $\\theta$ and the parameters of $\\Phi_{proj}$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the manuscript "Learning from Offline Foundation Features with Tensor Augmentations" (LOFF-TA) presents a valuable and innovative contribution to the field. The proposed approach, which enables efficient training of classifiers using cached embeddings from a foundation model, shows significant potential, particularly in resource-constrained settings. LOFF-TA not only substantially reduces computational and memory requirements but also demonstrates competitive performance across a variety of benchmarks.

The reviewers commend the authors for clarifying the suitability of LOFF-TA for high-resolution images, highlighting that the method’s computational efficiency and reduced memory usage make it especially advantageous in these scenarios. Furthermore, the reviewers appreciate the authors' commitment to extending their work to include experiments on 3D volumes and to visualizing the latent feature spaces. These additions will undoubtedly strengthen the paper’s contributions.

The reviewers encourage the authors to incorporate the proposed revisions and additional experiments into the final manuscript. Doing so will enhance the clarity of the work and further validate the claims made.

In summary, the paper presents a promising approach with the potential to significantly impact the field, especially in environments with limited computational resources. The reviewers recommend accepting this paper, with the expectation that the authors will implement the proposed improvements.
Decision: Accept (poster)