Specialized Foundation Models Struggle to Beat Supervised Baselines
Authors: Zongzhe Xu
Ritvik Gupta
Wenduo Cheng
Alexander Shen
Junhong Shen
Ameet Talwalkar
Mikhail Khodak
Venue: ICLR 2025 Poster (2025)
Keywords: foundation models
supervised learning
neural architecture search
hyperparameter optimization
convolutional networks
autoregressive models
genomics
satellite imaging
time series
Primary area: foundation or frontier models, including LLMs
TL;DR: Foundation models in specialized domains have in many cases not yet surpassed much smaller supervised models that are trained on much less data.

Abstract: Following its success for vision and text, the "foundation model" (FM) paradigm&#151;pretraining large models on massive data, then fine-tuning on target tasks&#151;has rapidly expanded to domains in the sciences, engineering, healthcare, and beyond. Has this achieved what the original FMs accomplished, i.e. the supplanting of traditional supervised learning in their domains? To answer we look at three modalities&#151;genomics, satellite imaging, and time series&#151;with multiple recent FMs and compare them to a standard supervised learning workflow: model development, hyperparameter tuning, and training, all using only data from the target task. Across these three specialized domains, we find that it is consistently possible to train simple supervised models&#151;no more complicated than a lightly modified wide ResNet or UNet&#151;that match or even outperform the latest foundation models. Our work demonstrates that the benefits of large-scale pretraining have yet to be realized in many specialized areas, reinforces the need to compare new FMs to strong, well-tuned baselines, and introduces two new, easy-to-use, open-source, and automated workflows for doing so.

Published as a conference paper at ICLR 2025
SPECIALIZED FOUNDATION MODELS STRUGGLE TO BEAT SUPERVISED BASELINES
Zongzhe Xu," Ritvik Gupta," Wenduo Cheng, Alexander Shen, Junhong Shen Carnegie Mellon University
(zongzhex, ri ritvi kgu, wenduoc, ajshen, Junhongs Gandreww cmu edu * denotes equal contribution; order decided by coin flip
Ameet Talwalkar
Mikhail Khodak
Carnegie Mellon University & Datadog, Inc. talwalkar(faa kaa.aarrraaaaaa. edu eduu
Princeton University
mkhodakēcs princeton, edu
ABSTRACT
Following its success for vision and text, the "foundation model" (FM) pretraining large models on massive data, then fine-tuning on target tasks-has rapidly expanded to domains the sciences, engineering, healthcare, and beyond. Has this achieved what the original FMs accomplished, i.e. the supplanting traditional supervised learning in their domains? To answer we look at three modalities- modallttes-gemommics, satellite imaging, and time series-with multiple recent FMs and compare them to a standard supervised learning work- flow: model development, hyperparameter tuning, and training, all using only data from the target task. Across these three specialized domains, we find that it is consistently possible to train simple supervised models-no more complicated than a lightly modified wide ResNet or UNet-that match or even outperform the latest foundation models. Our work demonstrates that the benefits of large-scale pretraining have yet to be realized in many specialized areas, reinforces the need compare new FMs to strong, well-tuned baselines, and introduces two new, easy-to-use, open-source, and automated workflows for doing so.
INTRODUCTION
Recent years have witnessed a shift towards large-scale pretraining across domains like computer vision and natural language processing. This workflow generally consists of two stages: pretraining on vast amounts of domain-specific data to capture general knowledge followed by fine-tuning on target tasks (Radford Narasimhan, 2018). This cestrain-then-ffeeee dously successful, enabling foundation models (Bommasani et al., 2021) to consistently outcompete traditional supervised learning methods on a wide variety of downstream tasks in the vision and language domains (Dosovitskiy et al., 2021; Liu al., 2021; Devlin et al., 2019).
paradigm has been tremen-
Driven by this success, the foundation model approach has been adapted to various specialized domains, which we define to be ML application areas e.g. genomics, satellite imaging, and time series-whose data modalities lie outside those of classical Al tasks, i.e. natural images and text. Such domains have seen the introduction of many new FMs claiming to leverage large pretraining datasets achieve breakthrough performance on downstream tasks (Dalla Torle- et al., 2023; Nguyen et al., 2024; Zhou et al., 2023b; Avsec et al., 2021; Ji et al., 2021; Fuller et al., 2023; Cong al., 2022; Mendieta al., 2023). These claims underlie our study's 's motivating question:
these new specialized FMs outperform traditional supervised learning applied the same tasks?
Answering this question is critical because supervised workflows are usually much less expensive to implement and deploy, but FMs that allow for effective transfer learning have the potential to funda- mentally transform these domains, as we have seen with language and vision processing in the past decade. However, despite ongoing efforts to promote their fair and comprehensive evaluation (Liang et al., 2022; Bommasani & Liang, 2021), many new FMs have not been adequately compared to simpler, often more efficient basclines. Indeed, found that many works only benchmark their proposed models against other FMs, essentially creating a comparison echo chamber (Fuller et al., 2023; Mendieta al., 2023; Nguyen al., 2024; Zhou al., 2023b).

---

Published as a conference paper at ICLR 2025
supervised method
best foundation model
other foundation models
NLP (2018 2019)
mean score Improvement
Genomics
0.3
0.3
BERT
BILSTM
tunee CNN
Caducous
-0.1
-0.1
HyenaDNA
NT-ullipeece
10
10
total pretraining + fine-tuning tokens (log-Scale)
total pretraining fine-turing base-pairs (log-scale)
imgrovement:
Satellite Imaging
mian RASE Improvament
Time Series
0.3
0.3
buned
SWIMAE
DOFA
tuned
STP LLM
MOMENT
-0.1
0.1
10a
total pretraining fine-tuning images (lag-scale)
total protraining flne-tuuiing series (lag-scale)
Figure 1: Across genomics, satellite imaging, and time series, specialized FMs fail to significantly improve upon supervised learning despite using two-to-five orders of magnitude more data. In con- trast, breakthrough FMs such as BERT dramatically outperformed supervised NLP baselines (top left), causing the field to switch to fine-tuning as the default approach. For each domain we plot to- tal pretraining and fine-tuning data vs. mean improvement over the supervised state-of-the-arr, The NLP results are from the GLUE benchmark ( (Wang et al., 2019) while evaluations of the last three do- mains are in Section Note that for the NLP x-axis we ignore word embedding pretraining tokens.
We answer our motivating question by considering a representative set of three specialized Nomains-chosee according to the presence of multiple FMs and standard set of evaluation tasks- and comparing their performance on those tasks with that of traditional supervised learning work- flow. The latter is three step process (i.e., model development, hyperparameter tuning, training) in which all steps use only data from the target task, in contrast to the FM workflow, which uses vast amounts of pretraining data (c.f. Figure 2). By leveraging model selection tools ranging from clas- sical information criteria to cutting-ede edge architecture search, we build automated pipelines that effi- ciently develop and train strong supervised models on over fifty tasks across three distinct domains.
Our main result is negative: despite pretraining on massive data, specialized FMs struggle and often fail to outperform models trained solely on downstream task data with traditional supervised learn- ing (c.f. Figure 1). Specifically, lightly adapted convolutional neural network (CNN) architectures such as wide ResNet and UNet attain state-of-the-ee on the Nucleotide Transformer benchmark in genomics and match the latest satellite FMs on downstream classification. Furthermore, tuned linear auto-regression (AR) matches or outperforms every open-souse seen-source time series FM on a standard suite of forecasting tasks, despite using four or more orders of magnitude fewer parameters and data.
These findings demonstrate that genomies, satellite imaging, and time series have not yet had their "BERT moment" (Devlin et al., 2019), i.e. these domains have not yet pretrained FMs that dominate traditional supervised approaches. This is despite the fact that all them have BERT-scale FMs and many already witnessing shift towards not comparing with supervised approaches, in natural language processing (NLP) post- BERT. More broadly, since these domains are among the most high-profil areas with specialized FMs, results suggest that practitioners should still consider supervised models in addition to FMs. They also reinforce the need for robust and well- tuned baselines, with surprising findings such (a) simply tuning kernel sizes and dilation rates in standard CNNs dominates genomics benchmark and (b) rescuing the century-old AR forecaster from obsolescence as easy as considering lookback parameters larger than five and training on GPU. To facilitate ongoing research in these and other domains, we make code associated with both our CNN- -tuning pipeline (DASHA) and our AR-on- on-GPU workflow (Auto-AR) publicly available.
Models with 100M+ parameters trained on 100x or more data than supervised tasks in the domain are given. Available https gi thub. com/ i a 99/nASнa and pss / //g thub, nom/ Rongxha- Xo/ Aut oAR.
2

---

Published as a conference paper at ICLR 2025
Foundation model (FM) workflow
pretrain FM on massive dataset
hyper- parameter optimization
fine-tune FM on target task
traditional
model development
hyper- parameter
train model on target task
seed
optimization
neural
architecture search
simulated
Supervised learning workflow
Our goal is to compare the pretrain-thert fien---fittuteee paradigm (top) with a standard super- vised workflow (bottom) on the tasks on which specialized FMs are evaluated. While for time series we go through a traditional process of developing and tuning a supervised model, this manual ap- proach does not scale to many domains; as a result, in Section 3.1 we develop a way to simulate it us- ing architecture search. Note that FM fine-tuning hyperaarameters are not always tuned in practice, but we assume their creators make a best-effort attempt to present their own method in the best light.
Figure 2:
2 RELATED WORK
Foundation models have been trained in numerous specialized domains beyond vision and text, in- cluding genomics (Ji et al., 2021), satellite imaging (Cong et al., 2022), time series (Goswami et al., 2024), weather (Bodnar et al., 2024), pathology (Zimmermann solving (Sun et al., 2024), web traffic (Zhao et al., 2023), and beyond. To get representative sense of their success, focus domains that combine the following properties: (a) multiple BERT-scale FMs, (b) standard suite of evaluation tasks, and (c) significant applied interest. These restrictions suggest looking three domains, all of which have least five FMs evaluated on at least nine tasks: genomics (which has some of the arrgst--vvailabl non-text FMs (Dalla-Tori) et al., 2023)), satellite imaging (which has large ongoing benchmarking effort (Lacoste et al., 2024)), and time series (which has already seen significant industry interest (Cohen et al., 2024)). The remainder of this section examines how different learning workflows approach problems these domains.
al., 2024), differential equation
Specialized foundation models: Collectively our three target domains have more than thirty FMs, many developed via the "lift-and-shift approach- bothh-bborownngg terminology from Rolf et al. i2024)-in which techniques from core areas such as vision and language processing are applied with modest tailoring to specialized domains. In particular, many methods are built on out-of-domain models such as BERT, Swin, and Hyena (Ji et al., 2021; Mendieta et al., 2023; Nguyen et al., 2024; Shen et al., 2024b), with adaptations including specialized tokenizations, embeddings, and model modifications for handling domain-specific considerations like long-range dependencies (Dalla-Tore et al., 2023; Zhou al., 2023b; Shen & Yang, 2021; Das et al., 2023; Shen et al., 2024a;c; 2025; Cohen et al., 2024; Liang et al., 2025) or multispectral data (Cong al., 2022). While the "lift-and-shift" approach point, its widespread use underlines the need for strong in-domain baselines to make sure that the combination of out-of-domain tooling and massive pretraining data is actually helpful. Such com- parisons are not always conducted, c.g. the satellite FM SatMAE (Cong et al., 2022) is compared to ImageNet-initiaaized and randomly initialized ResNet-50 (He al., 2015), while most of the time series FMS we consider only do a full comparison to one linear baseline, DLinear (Zeng et al., 2023). This can sometimes be justified-e.2 in the case of NLP p but our results suggest that for now, specialized FMs should still compare to in-domain supervised model development.
often be useful at least good starting
We note that are not the first to take critical look at specialized FMs. For example, Yang et al. (2024) questioned the dominance of Transformers for protein sequence FMs by showing that con- volutions could do just well, which is related to discovery that (supervised) CNNs were com- petitive with (largely seaeformer-basedd genomics FMs. Another study by Kedzierska et al. (2023) found that training in-domain generative model could outperform pretraining in single-cell biol-
3

---

Published as a conference paper at ICLR 2025
ogy applications. In satellite imaging, Jakubik et al. (2023) showed that supervised UNet performs competitively with an FM on a single non-standard task, while Xiong et al. (2024) found that super- vised models sometimes outperform linearly probed FMs. Corley et al. (2024) further demonstrated can be effective in this domain when properly tailored. Finally, in the time series domain, Tan et al. (2024) show that the popular approach of fine-tuning text-pretrained LLMs on time series tasks often underperforms supervised models (in their case randomly initialized attention); our results generalize this other time series FMs and use an even simpler supervised model (AR) to do so. Overall, these studies focus on specific cases, whereas our results identify a broader trend of FM underperformance and/or weak baselines across multiple fields.
that ImageNet-pretrainin
Specialized baselines: Both automated supervised pipelines that we develop are heavily influ- enced by successful in-domain model development. In particular, the NAS-based pipeline we use to achieve our results in genomics and satellite imaging is inspired by the success of the human-driven specification of kernel sizes and dilation rates in successful architectures like TCN (Lea et al., 2016) and ConvNeXt (Liu et al., 2022). At the same time, for time series our approach is based upon well-tuned GPU implementation of perhaps the most basic forecasting model, AR.
AutoML for specialized domains: While often evaluated on domains such as vision, auto- mated techniques are also important in specialized domains. An important example is Auto- ARIMA (Hyndman & Khandakar, 2008) for time series, although it underperforms on the tasks we consider (Challu et al., 2022). However, to avoid requiring significant expertise in any one domain, we also make use of AutoML methods developed specifically for diverse tasks (Roberts et al., 202 Shen et al., 2023), in particular the NAS method DASH (Shen et al., 2022) that can discover good kernel sizes and dilation rates for a CNN backbone faster than it can be trained from scratch.
3 METHODOLOGY
Recall that our goal is to conduct a robust comparison between traditional supervised learning and specialized FMs; the natural way do this is take existing benchmarks used evaluate FMs in our three target domains and run a typical supervised workflow on the same tasks, As depicted in Figure 2, this pipeline involves three steps: (1) model development, (2) hyperparameter tuning, and (3) training. The first stage involves using both reasoning and trial-and-ere to find good architec- ture to tune and train on the data; for example, Lea et al. (2016) developed the temporal convolutional network (TCN) architecture with multi-layer dilation rate pattern specifically suited to sequential data, while Liu et al. (2022) designed the breakthrough ConvNeXt architecture by methodically exploring ways to make CNNs more like Transformers without introducing attention. The second stage (hyperparameter tuning) can also be done via human-driven iteration, but there exist effective automated procedures for it well (Li et al., 2020). Lastly, the third step of the pipeline involves simply training the selected model with the selected configuration on the data of the target task.
While it is standard to automate the last two steps of the procedure, model development is typically done by hand and is difficult to do for fifty tasks across three domains. As a result, we settle for approximating the traditional supervised workflow by simulating the model development com- ponent via neural architecture search. To ensure fair comparison and reduce computation, we use low-fidelity NAS methods that return an architecture in less time than it takes to train it. Our results can thus be viewed as lower bounds on the performance of supervised learning, as model develop- ment might be significantly improved using hess-heu and/or human-driven architecture design.
In the remainder of this section we detail how we handle the different steps of the supervised pipeline. Note that our NAS NAS-dependent workflow (DASHA) we cover in the first part of this section-yields our main results for genomics and satellite imaging but not for time series. There we find its performance to be less competitive and instead focus on an even simpler approach based on linear regeesegr,,,,,,, whose model development and tuning describe in the second subsection.
3.1 DASHA: SIMULATING THE SUPERVISED WORKFLOW USING NAS
To simulate model development we need a search space over architectures that is (a) efficient, (b) flexible, and (c) applicable to the types of high-dimensional unstructured data that arise in domains targeted by specialized FMs; these requirements make CNN-based search spaces nat- ural choice. In particular, inspired by the success of hand-tuned kernel sizes and dilation rates in traditional model development (Lea et al., 2016; Bai et al., 2018; Liu et al., 2022), we apply DASH (Shen et al., 2022), a NAS method that starts with an existing CNN backbone-c. ResNet (Zagoruyko & Komodakis, 2017)-and and uses the weight-sharing heuristic (Liu et al., 2018)
a wide
4

---

Published as a conference paper at ICLR 2025
Algorithm 1: Pseudocode for the DASHA workflow. Starting with a set of backbone CNNs, we use DASH (Shen et al., 2022) to set the right kernel size and dilation rate for each of its convolutional layers and then use ASHA (Li et al., 2020) to configure training routine for the resulting architecture. Lastly, we pick the best backbone using validation data and train it.
Input: target task dataset D, candidate CNN backbone architectures A for CNN backbone E do
// set kernel size and dilation rate for each layer of archa + DASH(D,a)
// tune hyperparameters for the discovered architecture archa configa, scoor, + ASHA(D,
// train the architecture with the highest validation score a f arg A val.scre,
Output: train(D, arch, configa)
to determine the right kernel size and dilation rate to use at each convolutional layer. DASH has been successfully used in AutoML. competitions (Roberts et al., 2021a) and advance the state-of-the-re on NAS benchmarks (Tu et al., 2022), making it likely to be useful beyond the domains we consider.
As described in Algorithm 1, we augment DASH in two ways: (1) trying more than one CNN backbone (e.g. both wide ResNet and UNet (Ronneberger et al., 2015)) and (2) using the popular hyperparameter optimization (HPO) method ASHA (Li et al., 2020) to configure training settings for the discovered model. Following the NAS and tuning stages, we train the discovered architecture with the selected configuration on the target data. Further details, including the resources given to the three steps of the pipeline and the exact search spaces used by DASH and ASHA, are provided in Appendix B. Note that, while focus is data-efficient baselines, we do ensure that the entire workflow is never substantially more computationally expensive than fine-tuning an FM.
3.2 AUTO-AR: MAKING A BASELINE STRONGER BY MAKING IT SIMPLER
While DASHA can be applied to forecasting tasks, is not competitive with state-of-the-art time series FMs. At the same time, the field of time series forecasting has long employed automated workflows, notably the Auto-ARIMA approach of Hyndman & Khandakar (2008) that uses statis- tical tests and information criteria to tune ARIMA's lookback and differencing parameters. Auto- ARIMA was evaluated on the time series tasks we consider by Challu et al. (2022), who found that performed poorly compared to deep learning approaches. However, their implementation does not make use of multi-channel data and tunes up to lookback window of at most five, which is much less data than used by time series FMs. While tuning ARIMA with larger lookback parameters is computationally costly, we find the following simplified tuning pipeline to be effective:
1. use the KPSS test (Kwiatkowski et al., 1992) to decide whether to take first differences
2. use the Bayesian Information Criterion to select the maximum lookback parameter of the auto- regressive (AR) component of ARIMA, ignoring the moving average (MA) part
3. maximize the multi-channel likelihood of AR with the chosen differencing and lookback
By dropping the MA component of the model and running the procedure on GPU, we are able to tune the lookback windows up to the maximum allowable length (usually 512); we find that longer lookbacks are critical for performance. Note that this is just a tuned version of the classic AR model.
4 EMPIRICAL RESULTS
We now present the results of applying the automated pipelines described in the previous section to our three target domains. For each domain, we provide a brief justification the specific FMs and evaluation tasks that we consider, followed by details on how we apply our workflows; further information can be found in Appendices and B. As there are too many separate results to present outside the appendix, in this section mainly present aggregate statistics that summarize our findings for each domain, with detailed results relegated to Appendix C. The domains have different performance metrics, but they can all be aggregated via the following quantities: average score, average rank, and mean / median percentage improvement over baseline. For each domain, define domain-specific baseline and measure the improvement of FMs and our approach relative to it. This standardizes comparisons across tasks of varying scales.
5

---

Published as a conference paper at ICLR 2025
Model Size
Pretraining Base-Pairs
Average Score t
Average Rank 4
Mean %Imp.f
Median
Model
$Imp.
Enformer
252M
4B
0.569
11.86
27.73
27.91
NT-1000G (500M)
500M
20.5T
0.625
10.52
33,48
36.74
NT-1000G (2.58)
2.5B
20.5T
0.656
7.0
36.58
40.86
NT-Multispecies (500M)
500M
174B
0.700
3.81
40.76
45,07
Foundation
NT-Multispccies (2.5B)
2.5B
174B
0.697
4.08
40.51
45.52
Models
DNABERT-2
117M
32.5B
0.680
6.88
38.65
43.59
HyenaDNA-IK
1.6M
3.2B
0,708
6.92
41.2
43.36
HyenaDNA-32K
1.6M
3.2B
0.630
10.22
33,96
36,93
Caduceus-PPS
1.9M
35B
0.689
6.69
39.08
41.38
Caduceus PH
1.9M
35B
0.725
4.69
42.63
45.01
Wide ResNet
2.0M
0.694
6.83
37.16
43.08
Supervised Models
UNet
4.5M
0.68
7.78
38.67
42.69
DASIIA (our workflow)
10.5M
0.761
3.69
46.33
49.08
Table I: Aggregate genomics performance, showing that our supervised workflow (DASHA) attains state-of-the-art on the NT benchmark, outperforming all FMs according all measures while using no pretraining data and oftentimes many fewer parameters. For Mean Median %Imp.. we report % improvement over the Raw Probe baseline from Dalla-Torre al. (2023), and for DASHA the model size refers to the largest configuration across tasks..
4.1 GENOMICS
We begin our investigation in the genomics domain, which has witnessed the development of numerous FMs, including the early Enformer (Avsec al., 2021), the DNABERT series (Ji et al., 2021; Zhou et al., 2023b), the HyenaDNA family (Nguyen et al., 2024), GENA-LM* (Fishman al., 2024), the recent Caduceus family (Schiff et al., 2024), and the NT family (Dalla-Torre al., 2023); The latter includes models with up to 2.5B parameters. To evaluate them, we consider the Nucleotide Transformer (NT) benchmark of Dalla-Torre et al. (2023), which contains eighteen tasks in three main categories: regulatory elements, RNA production, and histone modification. We use this benchmark because of its diversity and because has been evaluated on by all of the aforementioned FMs, allowing to include eight of them in the comparison. Our numbers for these models are taken from Dalla-Tore et al. (2023, Supplementary Table 6); Following Dalla-Torm et al. (2023, Supplementary Table 5), We use FI score and aceuracy to evaluate a subset of regulatory elements and RÑA production tasks, and we use Matthew's Correlation Coefficient (MCC) as the main metric for evaluation on the remaining datasets.
Baselines: CNNs have long been used for genomics tasks (Avsec et al., 2020; Cheng et al., 2024) and so are natural supervised baselines; in particular include ID variants of Wide ResNet (WRN) and UNet, which we find perform better than some domain-specific CNNs. We use these same two backbones as the candidate CNNs tuned and selected from by our DASHA workflow.
Results: Our genomics results are displayed in Table 1, which shows that our supervised workflow (DASHA) consistently outperforms all FMs across all aggregate metrics. As discussed in Appendix C, our strong performance is driven in large part by outstanding performance on the histone modification tasks fc.f. Table 9). The more detailed results also highlight the importance of considering diverse baselines, with Wide ResNet usually being the selected architecture but UNet performing significantly better for promoter and splice site classification tasks. Overall, DASHA sets new sfate-of-the-ure the NT benchmark and certainly demonstrates that supervised methods remain quite competitive in genomics, despite the availability of massive pretraining datasets.
4.2 SATELLITE IMAGING
While they do get as large those in genomies, numerous BERT-scale FMs have also been introduced for satellite imaging, including SeCo (Manas et al., 2021), SatMAE (Cong et al., 2022), CROMA (Fuller et al., 2023), GFM (Mendieta et al., 2023), Scale-MAE (Reed al., 2023), Sat- las (Bastani et al., 2023), Prithvi (Jakubik et al., 2023), DOFA (Xiong et al., 2024), Clay (Clay Foun- dation, 2023). and SkySense (Guo et al.. 2024). Because our evaluation includes GeoBench (Lacoste et al., 2024), recently introduced satellite benchmark that has not been considered by many of these FMs, we obtain all results using our own fine-tuning: therefore we only consider subset of top- performing open-source models. In all cases use the fine-tuning workflow suggested by the
We compare GENA-L. in Appendix reports metrics that differ from those of the NT henchmark.

---

Published as a conference paper at ICLR 2025
Model Size
Pretraining Images
Average Score
Average Rank +
Mean %lmp.t
Median
Model
%Imp.t
SatMAE-Bate
85.6M
700K
76.99
9.89
5.27
3.59
SatMAF-Larrge
303M
700K
77.75
7.56
6.52
4.62
GFM
86.8M
1.3M
77.18
9.00
5.77
4.08
Swin'-Bass
86.8M
14M
76.69
8.72
4.86
1.43
CROMA-
90.6M
77.39
7.22
5.85
4.22
Foundation Models
CROMA- Large
312M
2M
78.03
5,44
6.90
6.09
DOFA-Base
111M
SM
78.44
4.50
7.71
6,23
DOFA-Large
337M
8M
78.80
3,50
8,40
6.88
Satlas
90M
45M
77.47
6.33
5.87
3.86
ScaleMAE
323M
363.6K
77.10
8.21
5,42
4.40
Clay
92M
70M
77.33
7.50
5.99
3.51
ResNe150
23.5M
0
73.76
13.22
0.30
00.07
Supervised Models
Wide ResNet
17.2M
73.97
12.89
0.00
0.00
UNet
17.3M
75.73
9,94
3.01
1.07
DASHA (our workflow)
32.4M
77.85
6.06
6.67
5.16
Table Performance satellite imaging, demonstrating that supervised workflow (DASHA) can get within 1 1% average accuracy of state-of-the art specialized FMs while using pretraining data and having 2-10x fewer parameters. For Mean Median %Imp. we report % improvement over vanilla Wide ResNet, and for DASHA the model size refers to the largest configuration across tasks.
authors of each FM plus some automated hyperparameter tuning: note that even with the extra tun- ing our reproductions on previous benchmarks systematically underperformed results reported in the original works. We select tasks mainly from GeoBench's 's five classification tasks and then add four additional tasks BigEarth (Sumbul et al., 2019), EuroSAT (Helber et al., 2019), Canadian Cropland (Jacques et al., 2023), and Mon--entieee (Cong al., 2022) that are commonly used to evaluate other FMs. As we focus on classification, report top-1 accuracy or mAP as appropriate.
Since satellite imaging resembles RGB imaging, is common to "lift-and-shift" vision models to this domain (Rolf al., 2024). As result we use several CNN backbones baselines and wide ResNet as the candidate architecture for our DASHA workflow. Lastly, we also consider the performance of fine-tuning the mmageNet-pretrainet vision FM SwinT-base (Liu al., 2021).
Baselines:
Results: Table 2 shows that supervised workflow attains competitive performance across all aggregate metrics and is only slightly outperformed by CROMA-large and the DOFA family. Unlike in genomics, the FMs here consistently outperform CNN backhones, likely because the associated papers compare to them as baselines. However, the frequently superior performance of DASHA sug- gests that domain-aware model development would yield good supervised models in this field. An- other contrast with genomics is that the larger FM versions consistently attain superior performance here, suggesting they are making at least somewhat effective use of the pretraining data. Neverthe- less, that this improvement can also be attained by DASHA, which uses no pretraining and produces a model that is 3- 10x smaller, suggests that there remains significant room for improvement.
4.3 TIME SERIES
Our last domain time series, which has many FMs that use the peetrain-then-fiee fine-tune workflow, such GPT4TS (OFA) (Zhou et al., 2023a), LLM4TS (Chang et al., 2023), MOMENT (Goswami et al., 2024), TEST (Sun et al., 2023), S-IP-LLM (Pan al., 2024), CALF (Liu et al., 2024), TTM (Ekambaram al., 2024). and Time-LLM (Jin et al., 2024), and others that evaluate in zero- shot (ZS) regime, such as TEMPO (Cao et al., 2024), TimesFM (Das et al., 2024), Moirai (Woo et al., 2024), and Toto (Cohen et al., 2024). We do not include four of these FMs in main analysis (Ta- ble 16): for Time-LLM, this is because their results have been hard-to-reproduce for both us and past efforts (c.f. Appendix I.3. while the three others-Moirai. LLM4TS, and Toto-either evaluate on only subset of our chosen tasks or are closed-source (or both); we list their reported num- bers in Table 14. As discussed in Appendix C.3.2, Moirai underperforms Auto-AR while LLM4TS performs roughly on par with TTM (A). The most recent FM, Toto, has strong aggregate metrics, mainly due dominant performance on single task, ETTh2 (c.f. Figure 7). As result we do not view these excluded FMs as significantly affecting our conclusions. We also did not include another FM-like approach, TOTEM (Talukder et al., 2024), because of focus on multi-task training rather
II Appendix C report results when excluding tasks where missing channels may affect performance.

---

Published as a conference paper at ICLR 2025
Model Size
Pretraining Series
Average RMSE
Average Rank
Mean IImp.t
Median
Model
%1mp-7
GPT4TS (OFA)
87M
8M
0.555
6.70
31.29
22.71
THST (Hew Shot)
345M
8M
0.603
10.70
25.23
14.59
MOMENT
385M
13M
0,550
5.14
31.95
23.14
TTM (B)
IM
IM
0.543
2.89
32.92
25.36
Foundation Models
TTM (A)
5M
IM
0,538
2.21
33.38
24.96
I'IP-LLM
345M
8M
0.545
3.54
32.59
24.63
CALF
86M
8M
0.568
8,95
29.89
21.60
TEMPO Shot)
345M
8M
0.598
10.54
26.69
22.30
TimesHM (Mero Shot)
200M
5M
0.574
7.38
28.50
19.56
DLinear
700K
O
0.567
8.13
29.68
23.18
Supervised Models
Auto-ARIMA
10
0
0.896
12.82
0.00
0.00
AR
513
0
0.556
6.57
31.17
24.31
Auto-AR (our workflow
513
0
0.551
5.45
31.91
25.36
Table 3: Aggregate forecasting performance across seven tasks, demonstrating that simply tuning a classical AR model is competitive with state-of-the-arr FMs while using no pretraining data and tens of thousands of times fewer parameters. For Mean / Median %Imp. we report % improvement over Auto-ARIMA, and for Auto-AR, the model size refers to the largest configuration across tasks.
than pretraining and because its multi-task results underperform its single-task method, which if anything supports our claim about the limited (realized) benefits of transfer learning time series.
We focus on long-horizon forecasting, which has standard set of datasets (Goswami et al., 2024, Table 11), of which we consider seven. Each consists of four settings corresponding to different time horizons, in total this yields twenty-eight tasks. We compute aggregate metrics using RMSE, not MSE, so that performance scales linearly with prediction error; this choice has no effect on average rank. Lastly, note that since zero-shot models (Table 16) is in a less challenging setting than the one they report numbers for. However, AR can also be used in the ZS setting by training on just the history provided for the test series; reported in Section 5.5, this is competitive with open-source ZS FMs at short time horizons.
focus is supervised baselines, our main comparison with
Baselines: As baselines we mainly use linear forecasters, including the classical (untuned) linear auto-regression (AR), the automated workhorse Auto-ARIMA (Hyndman & Khandakar, 2008), the more recent DLinear (Zeng et al., 2023), and our own workflow Auto-AR described in Section 3.2. Lastly, also evaluate our other approach, DASHA, six of the tasks (c.f. Table 14).
Results: Table 16 shows that on the full seven-dataset evaluation our Auto-AR workflow always attains competitive performances across all aggregate metries considered, and in particular attains the best median improvement over Auto-ARIMA. Specifically, our Auto-AR workflow achieves competitive performances with two other recent time series FMs, namely MOMENT and S- LLM, and outperforms the rest of the FMs. Although TTM surpasses all other methods across three aggregated metrics, the improvements remain relatively marginal. Thus, in this the domain well we find that increases in pretraining dataset size and model scale have not yet resulted in significant performance improvements. Notably, even untuned AR, which uses no differencing and a large lookback window, is quite efffective, doing better than zero-and-fer wnw-and--f-w-oott FMs across all metries.
5 DISCUSSION
At high level, results show that the foundation models in these three domains have not yet surpassed supervised learning, and thus more broadly that the latter remains a strong baseline for specialized FMs. This is surprising and consequential finding due the paradigm's 's popularity and the data and compute costs associated with large-scale pretraining. In this section we discuss lessons and implications for the development of machine learning in these and other application areas.
5.1 THE IMPORTANCE DIVERSE, WELL-TUNED, A AND DOMAIN-SPECIFIC BASELINES
The main lesson of work is select diverse array of baselines, drawing from both "lift-and- shift" and domain-specific approaches, and then to carefully tune them. For example, in genomics the vanilla wide ResNet baseline does remarkably well, with the majority of FMs doing worse than even this "lift-and-shift" baseline on the typical task in the NT benchmark. While satellite FMs
'The two do consider, Exchange and IIII, evaluated by most time series FMs.
8

---

Published as a conference paper at ICLR 2025
traniigg on ali dota
traiiing on 3 22% sutcomple
turtng (NPD NAA) + traihing (inetuninn / neeinannng
trairing only
genomics
sateellite Imaging g
timsc series
genomics
Satellite Imaging
time
sastdr tare
HOwinY
THM
Doowe
<
DasHe
T
Figure 3: Performance of the best FMs when given only 20% much fine-tuning data (c.f.
Figure 4: Tuning costs (excluding FM pre- training but including HPO for all methods and NAS for DASHA) and downstream fine-tuning / re-training costs (c.f. Table 18). Apart from Caduceus, supervised models are much cheaper, and for time series the entire Auto-AR pipeline is 3,5x faster than fine-tuning any FM once (n. tuning costs for time series FMs are unknown).
Table 19). Supervised baselines are competitive even in this data-scarce regime: in satellite imag- ing and time series they are outperformed by just one FM, while in genomics they are beaten by two. Notably the worst of the three genomics FMs in the full setting does best with less data.
do outperform such baselines, lightly modifying these CNNs via different kernel sizes and dilation rates was enough to match state-of-the-art models there well. Lastly, our time series results demonstrate in dramatic fashion the need to carefully tune domain-specifie approaches, as we show that simply allowing the classical AR forecaster to make use of long lookback windows and GPU- based optimization leads better forecasting than all open-source FMs.
5.2 THE IMPORTANCE OF BENCHMARKS THAT ADDRESS ACTUAL FM USE-CASES
A common argument in favor of foundation models is utility in low-data regimes, where transfer might be necessary to cope with limited supervision. However, demonstrate usefulness via this argument, FMs must be evaluated in data-scarce regimes rather than the usual full-data settings they (and thus also we) generally consider. Furthermore, as demonstrated in Figure 3, our supervised baselines remain competitive even when only 20% of the data is provided to each task. Together with the zero-shot Auto-AR results (c.f. Section 5.5), this indicates that significant effort needs to be put in to designing benchmarks that elucidate the claimed potential of existing FMs.
5.3 COMPUTATIONAL EFFICIENCY CONSIDERATIONS
While computational efficiency is not our main focus, we nevertheless note that any performance gains from FMs must also be balanced against any extra cost. In addition the usually massive costs of pretraining, the resulting models are often very big and costly to use. Indeed, apart from the special case of HyenaDNA and Caduceus, the CNN architectures discovered and trained using DASHA are typically over ten times smaller than FMs in the case of genomics and three to ten times smaller in the case of satellite imaging. Moreover, for time series our Auto-AR approach is quick-to-train and yields simple models with less than 1K parameters over x smaller than any Rh-wh attaining performance that is often competitive even with closed source models. We visualize the computational advantages of smaller supervised models in Figure 4, where we compare them against the top FMs in each domain. For genomics, fine-tuning most FMs takes roughly the same time as training our baselines, while for satellite imaging, FMs take roughly twice long- Lastly, in time series our Auto-AR pipeline is over 30x faster than large FMs and over three times faster than smaller models like TTM. This demonstrates efficiency of the supervised approaches sets a high performance bar that FMs need to clear before they can be deemed useful.
5.4 THE POWER OF TUNING KERNEL SIZES AND DILATION RATES
Our results for genomics and satellite imaging are driven by the DASHA workflow, whose crucial component the tuning of kernel sizes and dilation rate in CNN backbones such as wide ResNet. Its success demonstrates that the procedure is an effective surrogate for human-driven model develop- ment, enabling the automated discovery of the types of diverse, domain-specific baselines stressed in Section 5. To understand this further, we study whether the architecture search component selects different kernel sizes and dilation rates for different tasks, and whether does so in a consistent manner. Specifically, we run DASHA on three of the smaller datasets in the NT benchmark with fifteen different random seeds, construct eighteen- dimenssioni vectors of the discovered kernel sizes and dilation rates assigned to each of the nine layers, and project these to two dimensions using principal component analysis (PCA). The result in Figure 5 reveals that the architectures clustered by task, demonstrating that the procedure selects different but consistent wewessttet-wttt--ttttt within-task

---

Published as a conference paper at ICLR 2025
H3K79 99me3
Auto-AR
H4ac
TEMPO
promoter
RMSE (averaged across
TimasFM
PCA component
0.63
Toto
Moiral
0.60
0.55
0.50
192
336
720
component
Horizon
Figure 5; PCA visualization architectures discovered for three different tasks when DASHA is run multiples times. Clustering across tasks reveals the within-task consis- tency of the architecture search component and the utility of diverse models as baselines.
Figure Zero-shot performance on the subset of time series tasks with Moirai and Toto eval- uations. At shorter horizons, Auto-AR trained only the history given by the test series (and thus also given to ZS FMs) is competitive with all FMs except the closed-source Toto.
kernel parameters. This visualization suggests that architecture search is useful surrogate for model development, and consequently that the DASHA workflow may also be useful for automating similar studies and baselining FMs in other domains with high eigh-dimessional, unstructured data.
5.5 THE SURPRISING EFFECTIVENESS OF LINEAR AUTO- REGREESSEO
Perhaps our most surprising finding is the competitiveness of linear auto ruto-reeression (AR), a very method, on long-horizon forecasting. It is likely that the lack comparison with this baseline was driven by existing evaluations (e.g- by Challu et al. (2022)) of Auto-ARIMA (Hyndman & Khandakar, 2008), which is perceived to be a stronger baseline because both combines AR with another model (MA) and tunes the lookback and differencing parameters. However, in most Auto- ARIMA packages the default maximum lookback is around five, whereas we often found much (hundred-fold) larger settings to work best. Since these implementations also generally to support such long lookbacks, the possibility of expanding the hyperparameter space was more likely to be ignored. By implementing an efficient tuning procedure over larger space lookback parameters, our Auto-AR workflow comprises significant contribution to forecasting baselines.
slow
As described in Appendix F, Auto-AR also turns out to be easily extended to and competitive in the zero-shot setting, enabling direct comparison with ZS FMs (c.f. Figure 6). The method involves training a short-lookback AR model on subsequences of the history of the length 512 provided in the 7S regime. This establishes what we believe should be default supervised baseline for time series forecasting in the 7S setting. We provide full evaluations in Tables 20 & 21.
5.6 LIMITATIONS
While our findings are significant according to measures set by past work, they should not be misinterpreted to address all possible scenarios where FMs may be useful; for example, genomics FMs are often used for exploratory science and not prediction. We are also of course computation- ally limited and there many other domains where FMs have been pretrained, and even in three there are other tasks beyond classification and forecasting. Nevertheless, our evaluation is extensive-ove fifty tasks and thirty FMs-and the results provide at least strongly indicator w.r.t. the state of field that uses benchmark performance to motivate and justify pretraining.
CONCLUSION
We conduct a thorough investigation evaluate whether the cost of training specialized foundation models across three major domains justified by their superior performance relative traditional supervised learning. Our results demonstrate that FMs in these domains have not yet surpassed su- pervised workflows and often outperformed by fairly simple methods, including lightly modified CNN backbones (in genomics and satellite imaging) and classical linear forecasters (for time series). As part of our study, we introduce two automated workflows DASHA AAAAHA for simulating in-domain model development of CNNs and Auto-AR for tuning linear ruto-segrossion on GPUs- - we believe will be useful tools for evaluating future work these and other areas. The code for these pipelines and to reproduce our results is publicly available.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Mononito Goswami, Esther Rolf, and Stephan Xie for useful feedback. This work was supported in part by NSF grants IIS1705121, IIS1838017, IIS2046613, IIS2112471, TCS Presidential Fellowship, and funding from Meta, Morgan Stanley, Amazon, Google, and Scribe. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of any of these funding agencies.
REFERENCES
Ziga Avsec, Melanie Weilert, Avanti Shrikumar, Sabrina Krueger, Amr Alexandari, Khyati Dalal, Robin Fropf, Charles McAnany, Julien Gagneur, Anshul Kundaje, and Julia Zeitlinger. Base- resolution models of transcription factor binding reveal soft motif syntax. bioRxiv. 2020. doi: 10.1101/737981. URL https::/ /www www.wiiiiiii.... org.or/g//ooooooooooooe..oo..o.e http://www.....ooooott/////aa/// /ttp///eeeee/aa///////////////// 737981. 6
Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R. Ledsam, Agmieszka Trabska-Barwinsker Kyle R. Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R. Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature Methods, 18 (10::119-1103,, Oct 2021.
doi: 10.1038/441992-0211002255
1, 6, 19
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv, 2018. 4
Favyen Bastani, Piper Wolters, Ritwik Gupta, Joe Ferdinando, and Aniruddha Kembhavi. Satlaspre- train: A large-scale dataset for remote sensing image understanding. In /EEE/CVF International Conference Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 16726-16736. IEEE, 2023, doi: 10.1 100//CCVV1000.2222..0....
URL https: /
ICCV51070 2 2023. 01538. 6
Cristian Bodnar, Wessel Bruinsma, Ana Lucic, Megan Stanley, Johannes Brandstetter, Patrick Garvan, Maik Riechert, Jonathan Weyn, Haiyu Dong, Anna Vaughan, et al. Aurora: A foundation model of the atmosphere. arXiv preprint arXiv:2405. 13063, 2024. 3
Rishi Bommasani and Percy Liang. Reflections on foundation models, 2021. URL https:// hai. st anffrdd. edu /oluu/eewr www.//ww/ee/e//eeeeeee////e http///es///eeeeeeeee wwtp://rfeeeeiioooo coress-oonndiiiio models. 1
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Hen- derson, John Hewitt, Daniel E. Ho. Jenny Hong. Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning. Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nic, Juan Carlos Niebles, Hamed Nilforoshan, itriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong. Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srini- vasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matci A. Zaharia, Michael Zhang. Tianyi Zhang. Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kait- lyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv, 2021. URL https: / frm..m stanford. edu s / rff.rt.pdf..
Nyarko, Giray Ogut, Laurel Orr, Isabel Papadim-
George.E.P. Box and Gwilym M. Jenkins. Time Series Analysis: Forecasting and Control. Holden- Day, 1976. 22
11

---

Published as a conference paper at ICLR 2025
Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. Tempo: Prompt-based generative pre-trained transformer for time series forecasting, 2024. URL https:
//arxiv. org. abs /tabb//2330 /2100. 231.1.09409000 04948. 7
Cristian Challu, Kin G. Olivares, Boris N. Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-HiTS: Neural hierarchical interpolation for time series forecasting, In Proceedings of the 37th AAAI Conference on Artificial Intelligence, 2022. 4, 5, 10
Ching Chang, Wen-Chih Peng, and Tien-Fu Chen. Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained llms. arXiv preprint arXiv:2308. 2023. 7
Wenduo Cheng, Junhong Shen, Mikhail Khodak, Jian Ma, and Ameet Talwalkar. L2g: Re- purposing language models for genomies tasks. bioRxiv, 2024. doi: 10.1101/2024.12.09 627422. URL https: wwt.:w.bww.iv..oo..oooo..oww.... .rgg www.ent/eeallyy/22/////a////
Clay Foundation. Clay (revision 5a2ble7), 2023. URL https://nugginggaeee co/
made-wa with clay/Clay.
Ben Cohen, Emaad Khwaja, Kan Wang, Charles Masson, Elise Ramé, Youssef Doubli, and Oth- mane Abou-Amal. Toto: Time series optimized transformer for observability. arXiv preprint arXiv:2407.07874, 2024. 3.7
Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David B. Lobell, and Stefano Ermon. SatMAE: Pre- trainir transformers for tempo- ral and multi-spectral satellite imagery. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: eeweneeveiveeeeeeeeeeeeeeeeeeeee ew net/ / forum? foormmiddwBBeeeee d=WBh 6KYH. 1,3, 6, 7,
Isaac Corley, Caleb Robinson, Rahul Dodhia, Juan M Lavista Ferres, and Peyman Najafirad. Revis- iting pre-trained remote sensing model benchmarks: resizing and normalization matters. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3162- 3172, 2024. 4
Hugo Dorla-Torre, Liam Gonzalez, Javier Mendoza Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Hassan Sirelkhatim, Guillaume Richard, Marcin Skwark, Karim Beguir, Marie Lopez, and Thomas Pierrot. The nucleotide trans- former: Building and evaluating robust foundation models for human genomics. bioRxiv, 2023. wwtpssiw.orggcooneet//eeerreee coontet//eeerlly.///
doi: 10. 10112222233 11t.:001..11........ 1236779. URL ttspp:ttttt : / www. biorxiv.org///oo.oo............r 2023/01 00033/0/15//02333 01 11 523679 1, 6, 17, 19, 25
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. arXiv preprint arXiv:2310. 10688, 2023.
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting, 2024. URL https: htttt:///axxi......t http//arxivvirr//// at////bs/2213 /2000.///1.... 10688.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. In North American Chapter of the Associ- ation for Computational Linguistics, 2019. URL https: /apppp..../ http:/appp..aat..ee aeeemnntiiccho... .org/
Corpus ID: 52967399, 1,2
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale, 2021. URL https: aaaaiiiv... org/ /abs///// 20000 0.100.1 11929.
Vijay Ekambaram, Arindam Jati, Nam Nguyen, Pankaj Dayama, Chandra Reddy, Wesley M Gifford, and Jayant Kalagnanam. Tims: Fast multi-level tiny time mixers for improved zero-shot and few-shot forecasting of multivariate time series. arXiv preprint arXiv:2401.039. 2024. 7,
22
12

---

Published as a conference paper at ICLR 2025
Veniamin Fishman, Yuri Kuratov, Aleksei Shmelev, Maxim Petrov, Dmitry Penzar, Denis Shepelin, Nikolay Chekanov, Olga Kardymon, and Mikhail Burtsev. Gena-lm: A family of open-source foundational dna language models for long sequences. bioRxiv, 2024. doi: 10. 10.1101/2023.0.112 544594. URL https: : /www. bi orxiv. wwtp///ote///eer///ooo/o wtteeearly////////// 06. 19.12..54455 544594. 6, 19
Anthony Fuller, Koreen Millard, and James R Green. Croma: Remote sensing representations with contrastive radar-optical masked autoencoders. Thirty-eeentth Conference on Neural Information Processing Systems, 2023. 1, 6, 18, 21
Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai. Shuo Li, and Artur Dubrawski. Moment: family of open time-series foundation models, 2024. URL https: htg/abs/2400 03885. 3, 7, 8,
Xin Guo, Jiangwei Lao, Bo Dang, Yingying Zhang, Lei Yu, Lixiang Ru, Liheng Zhong, Ziyuan Huang, Kang Wu, Dingxiang Hu, et Skysense: A multi-modal remote sensing foundation model towards universal interpretation for earth observation imagery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27672-27683, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015. URL https: /ttp:s///axvvvv/////o org/ abs 1111/1111111 03385. 3
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification, 2019. URL https: / //arx farriiv.oo org/ abs/ 1 100. 00029.00000... 7,
R.J. Hyndman and Y. Khandakar. Automatic time series forecasting: The forecast package for R. Journal of Statistical Software, 27(3), 2008. 4, 5, 8, 10, 22
Amanda A. Boatswain Jacques, Abdoulaye Baniré Diallo, and Etienne Lord. The canadian cropland dataset: A new land cover dataset for multitemporal deep learning classification in agriculture, 2023. URL https:////// / arxiv. org abs 220//330.00000 00114. 7,
Johannes Jakubik, Sujit Roy, C. E. Phillips, Paolo Fraccaro, Denys Godwin, Bianca Zadrozny, Daniela Szwareman, Carlos Gomes, Gabby Nyirjesy, Blair Edwards, Daiki Kimura, Naomi Si- mumba, Linsong Chu, S. Karthik Mukkavilli, Devyani Lambhate, Kamal Das, Ranjini Bangalore, Dário A. B. Oliveira, Michal Muszynski, Kumar Ankur, Muthukumaran Ramasubramanian, Ik- sha Gurung, Sam Khallaghi, Hanxi Li, Michael Cecil, Maryam Ahmadi, Fatemeh Kordi, Hamed Alemohammad, Manil Maskey, Raghu K. Ganti, Kommy Weldemariam, and Rahul Ramachan- dran. Foundation models for generalist geospatial artificial intelligence. CoRR, abs/2310.186600, 2023. 4, 6
Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome. Bioinformat- ics, 37(15):112-22120 02 2021. ISSN 1367-4803. doi: (011033bioonfommatiic/taaabb URL https: /ooi./iii.////////// org/10 1093 3 www.biiionnaaiiioioo....o......a / 1, 3, 6, 19
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-llm: Time series forecasting by reprogramming large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL net
7, 22
Kasia Zofia Kedzierska, Lorin Crawford, Ava Pardis Amini, and Alex X. Lu. Assessing the limits of zero-shot foundation models in single-cell biology. bioRxiv, 2023. 3
Denis Kwiatkowski, Peter C.B. Phillips, Peter Schmidt, and Yongcheol Shin. Testing the null hy- pothesis of stationarity against the alternative of a unit root: How sure are we that economic time series have unit root? Journal of Econometrics, 54(1):159-178, 1992. 5
13

---

Published as a conference paper at ICLR 2025
Alexandre Lacoste, Nils Lehmann, Pau Rodriguez, Evan Sherwin, Hannah Kerner, Bjöm Lütjens, Jeremy Irvin, David Dao, Hamed Alemohammad, Alexandre Drouin, et al. Geo-bench, Toward foundation models for earth monitoring, Advances in Neural Information Processing Systems, 36, 2024. 3, 6, 18
Colin Lea, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolutional networks: A unified approach to action segmentation. In Computer -ision-ECCV 2016 Workshops: Am- sterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pp. 47-54. Springer, 2016.
Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-Tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. A system for massively parallel hyperparameter tuning. Proceedings of Machine Learning and Systems, 2:230- 246, 2020. 5,5
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09/ 10, 2022.
Weixin Liang, Junhong Shen, Genghan Zhang, Ning Dong, Luke Zettlemoyer, and Lili Yu. Mixture- of-mamba: Enhancing multi-modal state-space models with modality-awand sparsity, 2025. URL https: /aaxivvv.vvv..//////////////atv/ org/ /ttp/////.11.... 1a111.62999 16295.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv
preprint arXiv:1806.0900 2018. 4
Peiyuan Liu, Hang Guo, Tao Dai, Naiqi Li, Jigang Bao, Xudong Ren, Yong Jiang, and Shu-Tao Xia. Taming pre-trained ilms for generalised time series forecasting via cross-modal knowledge distillation. arXiv preprint arXiv:2403.0755 403.03300, 2024.7 7
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022 2021. 1,7
Zhuang Liu, Hanzi Mao, Chao-Yuse Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1996-11986, 2022.
Oscar Manas, Alexandre Lacoste, Xavier Giró-i Nieto, David Vazquez, and Pau Rodriguez. Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9414-9423, 2021. 6
Matías Mendieta, Boran Han, Xingjian Shi, Yi Zhu, and Chen Chen. Towards geospatial foundation models via continual pretraining, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16806-16816, 2023. 3, 6,
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Brrch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36, 2024. 1, 3, 19
Zijie Pan, Yushan Jiang, Sahil Garg, Anderson Schneider, Yuriy Nevmyvaka, and Dongjin Song- 2 ip-llm: Semantic space informed prompt learning with I1m for time series forecasting. arXiv preprint arXiv:2403.05790 2024. 22
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre- training. 2018. URL atttps://age semanticscholat www.:a/......ooorr..:.:00003333.
Colorado J Reed, Ritwik Gupta, Shufan Li, Sarah Brockman, Christopher Funk, Brian Clipp. Kurt Keutzer, Salvatore Candido, Matt Uyttendaele, and Trevor Darrell. Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 4088-4099, 2023. 6
14

---

Published as a conference paper at ICLR 2025
Nicholas Roberts, Samuel Guo, Cong Xu, Ameet Talwalkar, David Lander, Lvfang Tao, Linhang Cai, Shuaicheng Niu, Jianyu Heng, Hongyang Qin, Minwen Deng, Johannes Hog. Alexander Pf- efferle. Sushil Ammanaghatta Shivakumar, Arjun Krishnakumar, Yubo Wang, Rhea Sukthanker, Frank Hutter, Fuxhen Hasanaj, Tien-Dung Le, Mikhail Khodak, Yuriy Nevmyvaka, Kashif Rasul, Frederic Sala, Anderson Schneider, Junhong Shen, and Evan Randall Sparks. AutoMI. Decathlon: diverse tasks, modern methods, and efficiency at scale. In Advances in Neural Information Pro- cessing Systems: Competition Track, 2021a.
Nicholas Roberts, Mikhail Khodak, Tri Dao, Liam Li, Christopher Ré, and Ameet Talwalkar. Re- thinking neural operations for diverse tasks. Advances in Neural Information Processing Systems, 34:15855-15869, 2021b. 4
Esther Rolf, Konstantin Klemmer, Caleb Robinson, and Hannah Kemer. Mission
data is distinct modality in machine learning. arXiv preprint Stti:2402.00144:: 2024. 3,
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation, 2015. URL atttpa://arke org/ ozg//abs/1555.05557.
Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arXiv:2403.03234, 2024. 6, 19
Junhong Shen and Lin F. Yang. Theoretically principled deep acceleration via nearest neighbor function approximation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(11); 9558-9566, May 2021. doi: wtt.:///.a...5511........ indexx php/ AAAI www.A///atiieeeeeee 1
URL attps://ojsp aaai.araaaa........
Junhong Shen, Mikhail Khodak, and Ameet Talwalkar. Efficient architecture search for diverse tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 4,5
Junhong Shen, Liam Li, Lucio M Dery, Corey Staten, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar. Cross-modal finettning: Align then refine. In International Conference on Machine Learning. pp. 31030-31055 PMLR, 2023. 4
Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Amlekar, Mouad Hadji, Aaron Podolny, and Ameet Talwalkar. Scribeagent: Towards specialized web agents using production-scale workflow data, 2024a. URL hattps://arxet org/ abs//asss 15004.
Junhong Shen, Tanya Marwah, and Ameet Talwalkar. Ups: Towards foundation models for pde solving via cross-modal adaptation. arXiv preprint arXiv:2403.07/88, 2024b.
Junhong Shen, Neil Tenenholtz, James Brian Hall, David Alvarez-Melis, and Nicolo Fusi. Tag-Ilm: Repurposing general-pune purpose Ilms for specialized domains. arXiv preprint arXiv:2402.05140, 2024c.
Junhong Shen, Kushal Tirumala, Michihiro Yasunaga, Ishan Misra, Luke Zettlemoyer, Lili Yu, and Chunting Zhou. Cat: Content-adaptive image tokenization, 2025. URL https://arxiv. borg/bss/ 2501. 03120. 3
Gencer Sumbul, Marcela Charfuelan, Begum Demir, and Volker Markl. Bigearthnet: A large- scale benchmark archive for remote sensing image understanding. IGARSS 20/9 2019 IEEE International Geoscience and Remote Sensing Symposium. IEEE, July 2019, doi: 10. 10.///9/ss 2019 8900522, URL http: /xx doi .rrg//org//a 10. 1109/ IGARSS 2019. 8900532. 18
1109/IGARSS
Chenxi Sun, Hongyan Li, Yaliang Li, and Shenda Hong. Test: Text prototype aligned embedding to activate Ilm's ability for time series. arXiv preprint 20ti::2300.082444 2023.
Jingmin Sun, Yuxuan Liu, Zecheng Zhang, and Hayden Schaeffer. Towards foundation model for partial differential equation: Multi-operator learning and extrapolation. arXiv preprint arXiv:2404. /2355, 2024. 3
Sabera Talukder, Yisong Yue, and Georgia Gkioxari. Totem: Tokenized time series embeddings for general time series analysis. arXiv preprint arXiv: 16412, 2024.
15

---

Published as a conference paper at ICLR 2025
Mingtian Tan, Mike A. Merrill, Tim Althoff Vinayak Gupta, and Thomas Hartvigsen. Are language models actually useful for time series forecasting? In Advances in Neural Information Processing Systems, 2024. 4. 22
Renbo Tu, Nicholas Roberts, Mikhail Khodak, Junhong Shen, Frederic Sala, and Ameet Talwalkar. NAS-Bench-360: Benchmarking diverse tasks for neural architecture search. In Advances in Neural Information Processing Systems: Datasets and Benchmarks Track, 2022. 5
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 7th International Conference on Learning Representations, 2019.
Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Yong Liu, Mingsheng Long, and Jianmin Wang. Deep time series models: A comprehensive survey and benchmark, 2024. URL https://arxiv:
org/abs/2407 /t0/////2/200.1777... 18
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sa- hoo. Unified training of universal time series forecasting transformers, 2024. URL https:
//arxiv. wwwpivvvorg/aoo abs 2402 02592.7
Zhitong Xiong. Yi Wang. Fahong Zhang, Adam J Stewart, Joëlle Hanna, Damian Borth, Ioannis Pa- poutsis, Bertrand Le Saux, Gustau Samps-Valls, and Xiao Xiang Zhu. Neural plasticityiinsprred foundation model for observing the earth crossing modalities. arXiv e-prints, pp. -Xxv-2403, 2024. 4, 6
Kevin K. Yang, Nicolo Fusi, and Alex X. Lu. Convolutions are competitive with transformers for protein sequence pretraining. Cell Systems, 15()):28--29,, 2024. 3
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks, 2017. URL WRRhhttss///axxi org/abs/ / 605. 07146. 4
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11121-11128, 2023. 3,8
Ruijie Zhao, Mingwei Zhan, Xianwen Deng, Yanhao Wang, Yijun Wang, Guan Gui, and Zhi Xue. Yet another traffic classifier: A masked autoencoder based traffic transformer with multi-level flow representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 5420-5427, 2023. 3
Tian Zhou, PeiSong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits plo:r general time series analysis by pretrained Im, 2023a. URL https: http///axvvvv/////
Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. Dnabert-2: Efficient foundation model and benchmark for multi-species genome, 2023b. 3,
Eric Zimmermann, Eugene Vorontsov, Julian Viret, Adam Casson, Michal Zelechowski, George Shaikovski, Neil Tenenholtz, James Hall, David Klimstra, Razik Yousfi, Thomas Fuchs, Nicolo Fusi, Siqi Liu, and Kristen Severson. Virchow2: Scaling self-supervised mixed magnification models in pathology. arXiv, 2024.
16

---

Published as a conference paper at ICLR 2025
A
TASKS
A.1 GENOMICS
For the Genomics domain, we use the eighteen classification tasks from the Nucleotide Transformer benchmark (Dalla-Toree et al., 2023) that has widely been used for other genomics FMs. The bench- mark datasets consist of nucleotide base sequences ranging from 200 600 bases in length. It pro- vides a realistic and biological meaningful benchmark across four main categories: promoter (hu- man/mouse), enhancer (human), splice site (SS; human/mulltsppccees) and histone modifica- tion (yeast). Within the benchmark, the enhancers.tyye and splice.ss www.essstesseee. tasks classification tasks with three classes each, while the remaining tasks are binary classification tasks.
# of samples train
Maximum sequence length
Dataset
# of classes
Metric
test
enhancers
2
14968
400
200
MCC
enhancers.types
3
14968
400
200
MCC
promoter_all
2
53276
5920
300
FI
promoter tata
2
47767
5299
300
F1
promoter
2
5509
621
300
F1
splice sites acceptors
2
19961
2218
600
F1
splice_siiee sites all
3
27000
3000
400
Accuracy
pelice_sites_donor
2
19775
2198
600
F1
H3
2
13468
1497
500
MCC
H3K14ac
2 29743
3305
500
MCC
H3K36me3
2 31392
3488
500
MCC
H3K4me1
2 28509
3168
500
MCC
H3K4me2
2
27614
3069
500
MCC
H3K4me3
2 25953
2884
500
MCC
H3K79me3
2
25953
2884
500
MCC
НЗК9ac
2
25003
2779
500
MCC
H4
2
13140
1461
500
MCC
H4ac
2
30685
3410
500
MCC
Table 4: Statistics for Genomics datasets
17

---

Published as a conference paper at ICLR 2025
A.2 SATELLITE IMAGING
In the satellite imaging domain, we aim to conduct evaluations with real-world relevance to Earth science. To achieve this, we include a variety data from different sources to cover diverse range of tasks, such as brick kiln identification, deforestation prediction, and photovoltaic monitoring, We utilize five classification tasks provided by the GeoBench dataset (Lacoste et al., 2024), a recently developed benchmark that offers clean and carefully curated collection of tasks specifically de- signed for satellite imaging. In addition to GeoBench, we evaluate our model on three additional datasets (Helber et al., 2019; Jacques al., 2023; Sumbul al., 2019) commonly used in the liter- ature as benchmarks for this domain This brings the total to eight datasets, encompassing wide range of features. These tasks vary in complexity, with single-class classification ranging from bi- nary to 62-class problems, as well as two multilabel classification tasks. The datasets are further characterized by diverse input channels, ranging from 3 RGB channels 18 channels that integrate data from both Sentinel-1 and Sentinel-2 formats.
For Beeo-Bench datasets, do not any mi xup and cutmix x augmentations. For other datasets, we universally use mixup 0.8, cutmix 1. O, and switch probability of 0.5, Following Fuller al. (2023), we use only 10% of training from BigEarthNet and [MoW-Sentine while using the full evaluation set for validation.
Image Size
# of samples
# of channels
Dataset
# of classes
train
val
test
m-bigearthnet
120 120
43
20000
1000
1000
12
m-briekkiln
64 64
15063
999
999
m-so2sat
32 32
17
19992
986
986
18
m-forestnet
332 332
12
6464
989
993
m-pv4ger
320 320
2
11814
999
999
3
BigEarthNet
120 120
19
31166
103944
103728
12
EuroSAT
64 64
13
16200
10800
5400
13
Canadian Cropland
120 120
10
53884
11414
11674
12
MMoW-Sentinel
96 96
62
71287
84939
84966
13
Table 5: Statistics for Satellite datasets
A.3
TIME SERIES
In the time series domain. we focus on the long horizon forecasting task. We use a subset of the common benchmark datasets for evaluating models across different domains (ETT, Electricity, Weather, Illness, Traffic, Exchange Rate) (Wang et al., 2024), specifically, the ETT, Weather, Electricity, Illness (ILI), and Traffic datasets. Note that the ETT dataset is actually collection of four series: ETTh1, ETTh2, ETTm1, and ETTm2; we follow the rest of the literature in treating each series as separate dataset. Each dataset contains measurements of one more channels at evenly spaced time steps.
of channels
# of samples val
Dataset
train
test
ETTh1
7
8033
2785
2785
ETTh2
7
8033
2785
2785
ETTml
7
33953
11425
11425
ETTm2
33953
11425
11425
Weather
21
36280
5175
10444
Electricity
321
17805
2537
5165
ILI
7
69
2
98
Traffic
862
11673
1661
3413
Table 6: Statistics for Time Series datasets
18

---

Published as a conference paper at ICLR 2025
B IMPLEMENTATION DETAILS
B.1 DASHA
Following the architecture search, we perform hyperparameter tuning using ASHA. The hyper- parameter search space includes learning rate, weight decay, momentum, drop rate, and random seed for model initialization. We define a continuous search space, with further specific details provided in Table 7. Using ASHA, we evaluate 200 sample configurations over maximum of 20 epochs, using a reduction factor of 2. The low-perforning configurations are pruned based on their validation scores.
Before retraining the final model, we load the model checkpoint corresponding to the optimal hy- perparameter configuration. The model is then trained for 200 epochs on the training data, with the best-performing checkpoint selected based on validation performance. This process is repeated for each backbone architecture, and the best- pesttperforming backbone is selected using the validation score. Finally, the checkpoint for the selected backbone is evaluated on the test set to obtain the final score.
Hyperparameter
Search Space
Type of Search Space
random_seed
0,500] [10-5 5 x 10-1) [0, 10,0.05,0.1 0.13
Integer
1r
Log Uniform
drop_rate
Discrete
weight decay
5 x 10 10-1,5 10-3)
Log Uniform Uniform
momentum
00.9, 1]
Table 7: Hyperparameter Search Space
B.2 AUTO-AR
A fairly complete description is provided in Section 3.2. Here note only that, because mini- mize the total maximum likelihood across (independent) channels, to determine the amount of differ- encing used for each task needed by the majority of the channels. Notably, this results in a differencing of one for each task.
the KPSS test separately each channel and use the differencing
C DETAILED RESULTS
C.I GENOMICS
We include all FMs listed in Dalla-Torre al. (2023, Supplementary Table with addition of the two recently released models, Caduceus (Schiff et al., 2024) and Gena-LM (Fishman et al., 2024); note that the last row of tasks the NT paper(promoter and splice sites) are mislabeled, but we infer an order in combination with previous information obtained from (since-deleted) Huggingface leaderboard." 6 In alignment with the leaderboard, we apply a 0.1 validation split for DASHA during our evaluation. Additionally, we use architecture set that includes both Wide ResNet and UNet for the search with DASHA on these datasets. We use batch size= 128 for all datasets, and cross entropy loss for all the training and finetuning. Individual scores for each task in the benchmark are provided in Tables 9 and 8.
The performance of the models on individual tasks is detailed in Tables 8 and 9. In the regula- tory elements domain (c.f. Table 8), DASHA performs slightly behind the largest models like NT- Multispecies (Dalla-Torre et al., 2023) the enhancers tasks but consistently outperforms models such as DNABERT-1 (Ji et al., 2021), Enformer (Avsec et al., 2021), and HyenaDNA (Nguyen al., 2024); in the promoters task in generally performs worse than all reported FMs. In the RNA production domain, DASHA performs near the middle of the FMs. However, where DASHA truly excels in the histone modification tasks (in Table 9), where
only competes with, but often outperforms, the other FMs, consistently achieving top scores in nearly all tasks.
"See hctps://hhapee iggaaee http://p..ppppcpec.eaeeeaooaeeet http://ccaaeeenttttttpttaate /nual eoo dana tranaforme formor ttw.trranfoomrrbeeethheeeee bhnchhar...
19

---

Published as a conference paper at ICLR 2025
Regulatory Elements promoter all
RNA Production splice sites all
enhancers
enhancers
promoter no.tata
promoter tata
splice sites acceptors
splice
donors
Enformer
0.454
0.312
0.955
0.955
0.959
0.915
0.847
0,906
NT-1000G (500M)
0,509
0.951
0.951
0,936
0.968
0,971
NT-1000C (2.5B)
0.546
0.432
0.965
0.967
0.957
0.98
0.976
0,979
NT-Mullispociis (500M)
0,339
0.438
0.976
0.976
0.965
0.981
0.984
0,987
NT-Multispecies (2.58)
DNABERT-1
0.495
0.367
0.961
0.962
0.956
0.975
DNABERT-2
0.525
0.423
0.972
0.972
0.955
0.975
0,939
0.963
HyenaDNA-IK
0.52
0.403
0.959
0.959
0.944
0.959
0.956
0.947
HAe 32K
0.489
0.352
0,956
0.954
0.939
0.96
0,962
0,957
Caduceus-PS
0,491
0,416
0.967
0.968
0.957
0.936
0,927
0.874
Caduceus- PH
0.546
0.439
0.97
0.969
0,953
0.937
0,94
0,948
Wide ResNet
0,525
0.416
0.952
0.946
0.93
0.821
0.457
0.815
UNet
0.49
0.366
0.956
0.954
0.95
0.956
0.955
0.968
0.527
0.962
0.979
Table 8: Regulatory Elements and RNA Production tasks. 4_" indicates unknown quantities.
Model
113
H.K 14ac
H3K36mm.
H1344mel
H3K4me2
H334mme.
H3K77mm3
H3399c
H4
H4ac
0.724
0.284
0.735
NT: 1000 (500M)
0.736
0.381
0.468
0:38
0.26
0.235
0.562
0.479
0.755
0.342
NT. 1000G (2:5B)
0.754
0.453
0.53
0.418
0.278
0.311
0.574
0.491
0.787
0.408
NT-Mullispociess (500M)
0,786
0.549
0.624
0.55
0.406
0.567
0.799
0.496
NT. Multispecies (2.58)
0.793
0.538
0,618
0.541
0.324
0.408
0622
0.547
0,808
0.492
DNABERT-1
0.396
0,258
DNABERT-2
0.785
0.515
0,591
0.512
0.333
0.353
0.615
0.545
0.797
0.465
HycnaDNA-IK
0.781
0.608
0.614
0.512
0,455
0,55
0.669
0.586
0.763
0.564
HyyenDNA-322K
0.291
Cadaccas-PS
0.799
0.541
0.609
0.488
0.388
0.44
0.679
0.604
0.789
0.525
Cadaceus- PII
0.815
0.631
0.601
0.523
0.487
0.544
0.697
0.622
0.811
0.621
Wide ResNet
0.798
0.667
0.670
0.554
0.541
0.660)
0.706
0.620
0.754
0.657
UNet
0.797
0.647
0.482
0.541
0.553
0.292
0.562
0.624
0.760
0.389
0.790
Table 9: Histone Modification tasks
Note that because GENA-LM reports using MCC on all datasets, which is different than the metrics used by NT and Caduceus paper, we compare its results with DASHA in separate table, where all datasets are evaluated through MCC. As shown in Table 10 and Table 11, DASHA also outperform GENA-LM by large margin terms of average MCC.
Model
H3
H3K14ac
33K36me3
H3K4mel
H3K4me2
H3K4me3
H3K79me3
H3K9ac
H4
H4ac
GENA-LM
0.79
0.6
0.61
0.53
0.46
0.55
0.67
0.61
0.78
0.59
DASHA
0.79
0.53
0.64
0.71
0.72
0.71
0.78
0.74
Table Histone Modification for GENA-LM (all scores are obtained using MCC)
Model
enhancers
enhancers types
promoter all
promoter nojaata
promoter tata
splice_sites acceptors
splice.sites all
splice_sites donors
Average
GENA-LM
0.55
0.45
0.94
0.94
0.91
0.92
0.91
0.91
0.707
DASHA
0.43
0.92
0.92
0.90
0.97
0.96
0.96
0.755
Table Regulatory Elements and RNA Production for GENA-LM. (all scores are obtained using MCC). The last column shows the average MCC across all 18 datasets in Tables 10 and 11.
20

---

Published as a conference paper at ICLR 2025
C.2 SATELLITE IMAGING
Training on satellite datasets requires relatively large computational resources due to the high num- ber of channels and the size of the datasets. To ensure fair comparison, we fine-tuned all the foundation models ourselves by sweeping across a fixed set of base learning rates [5e 3, 2e 2e 4, 4e 5]. We then calculate the actual learning rate from base learning rate following pre- vious work by Ir base__rr batch_size
256 This approach ensures that approximately the same amount of resources were used as during the DASHA tuning process, allowing for balanced evaluation of model performance. For training and finetuning, we universally use batch size 16 and loss function as cross entropy with O. label smoothing for single label classification, and multi-label soft margin loss for multilabel classification. Individual scores for each task are provided Table
We closely followed the reported evaluation processes from previous studies on FMs (Cong et al., 2022; Fuller et al., 2023; Mendicta et al., 2023). These models do not employ validation set for hy- perparameter tuning model selection, and we adhered this same approach when fine-tuning the FMs. However, for DASHA, since we performed extensive hyperparameter optimization over a large search space, used validation set ensure fair and accurate comparisons between DASHA and the FMs. This is a less favorable setting for DASHA, as relies on extensive hyperparameter tuning, but demonstrate that, under these conditions, DASHA matches the performance of the FMs.
Model
Average
m- m-bigearthnet
m-brickkiln
m-soZsat
m-foresinet
m-pv-gger
BigEarth Net
EaroSAT
Canadian Cropland
Sentinel
IMoW
SaLMAE-Bas
72.3
54.56
51,89
97.0
LareAF-Large
77.75
73.82
98.6
$5.79
$3.7
96.92
86.75
98. 86
75.38
59.89
GFM
77.18
71.97
98.35
57.52
59.38
96.54
85.93
99.02
72:03
53.84
SwinT-Base
56.49
97.54
CROMA-Bure
77.39
72.07
98.99
60.04
$4.07
96.6
86.94
98.81
75.87
53.14
CarMA-Lare
78.03
73.36
99.01
59.22
51,96
96.9
87.98
98.98
76.56
$8.32
ResNor50
73.76
98
51.83
54.29
96.79
79.16
2.3
$2.84
Wide ResNet
73.97
69.15
98.95
49.04
52.14
96.34
80.48
98.6
72.05
49.00
UNet
75.73
69.89
98.6
56.87
57.18
97.39
83,9
98.99
72.68
46.11
DASHAA
77.85
72.72
98.92
56.28
57.24
97.4
86.09
99.07
75.69
57.20
Table 12: Satellite imaging tasks
It is also important to note that SatMAE only accepts 3-channel and 12-channel inputs, while CROMA is limited to 12-channel inputs. GeoBench, however, includes a wide range of tasks with varying numbers of input channels, ranging from 3 18. Despite these differences, we include all datasets our evaluation because they are valuable benchmarks in the satellite image domain, and crucial for FMs in this field to generalize across diverse datasets. For datasets where the input size does not match the model requirements, we pad missing channels with zeros and prune any extra channels. However, to ensure fair comparison, in addition to reporting the average scores across all datasets, we also provide average scores excluding m-pv4ger and m-forestnet, where missing channels may affect the performance the FMs. The aggregate scores excluding m-pv4ger and m m-forestnet are presented in Table 13.
Average Score f
Average Rank
Mean %Imp.
Median %Imp.f
Model
SatMAB-Base
77.71
6.00
6.74
4.56
SatMAE-Lard
78.44
4.07
7.87
6.75
Foundation Models
GFM
76.95
5.57
5.40
4.08
SwinT-Base
76.12
6.50
3.98
1.43
CROMA-Base
77.98
3.57
6.95
5.30
CROMA-Large
79.06
2.14
8.84
6.26
ResNet50
73.25
9.00
-0.27
-0.38
Supervised
Wide ResNet
73.90
8.00
0.00
0.00
Models
UNet
75.29
6.57
2.33
0.87
DASHA (our workflow)
78.00
3.57
7.02
5.16
Table 13: Aggregated metrics satellite imaging tasks excluding the m-pv4ger and m-forestnet.
21

---

Published as a conference paper at ICLR 2025
C.3
TIME SERIES
Long horizon forecasting for time series is the following task: at every timestep t, take the L his- torical observations at times (t (t- L 1..... t) for each channel and predict the next H observations (t+1, t- H) for every channel. Following the literature, ILI and (96, 192, 336, 720} otherwise. For most methods, we report results for L = 512. All results reported on a 70/10/20 train/validationneest split for each datasets, except for the ETT datasets which have predefined splits. MSE is reported after all datasets have been scaled by the mean and variance of the training data.
evaluate on H E {24, 36, 48, 60) on
In addition to the results for DASHA, Auto-AR, DLinear, and the FMs, we evaluate the performance of two simple baselines:
1. AR (d=0): The vanilla autoregressive model (Box & Jenkins, 1976) predicts the (scalar) value of time series at t + 1 as a linear combination of the last L timesteps and a constant, i.e. it+1= ag + 01 + 02- ...+. - for learnable parameters au....., We fit these parameters using standard maximum likelihood techniques. Here "d=0" denotes no differencing.
2. Auto-ARIMA: ARIMA is a statistical method used for time series forecasting that combines three components: AutoRegressive (AR), Integrated (I), and Moving Average (MA). The AR component models the relationship between an observation and its lagged (past) values, assum- ing that past values have a linear influence on future ones. The Integrated component applies differencing to the data to remove trends seasonality, making the time series stationary by stabilizing its mean over time. The MA component models the relationship between an obser- vation and the residual errors from moving average model applied to previous observations. ARIMA is characterized by three parameters: (the number of lag observations), d (the number of differencing steps achieve stationarity), ) and q (the number lagged forecast errors). This model is particularly effective for univariate time series forecasting where patterns like trends or seasonality are present. To tune these parameters we use the popular Auto-ARIMA approach of Hyndman & Khandakar (2008) discussed in Section 3.2.
As described, all of our baselines can handle only univariate time series, while all of the benchmark datasets are multivariate (multiple channels). These baselines are trained under channel indepen- dence: each channel of a time series is treated independently. While channel independence fails to take into account cross-channel dependencies, we note that developing methods that leverage cross-channel dependencies for variable number of channels remains open problem.
C.3.1 TIME-LLM
Due to diffcrences in the results reported for Time-LLM between the original paper and reproduc- tions in several works (Goswami et al., 2024; Ekambaram et al., 2024; Pan et al., 2024: Tan et al., 2024), we to attempt our own partial reproduction in order to determine whether to include their numbers. To do so, used code provided by the authors and looked at ETTh1 with input lengths from 96, 192 and ILI with input lengths from 24, 36, 48, 90 (going beyond this was infeasible due to the resources required to finetune LLaMA-7B). Our reproduced MSE for ETThl with an input length of 96 was 0.405 as compared to that of 0.362 reported by Jin et al. (2024). while for input length of 192 it was 0.431 vs. 0.398. On ILI the average increase in MSE across time horizons from their reported results to our reproduction was more than 0.4. Due to these large discrepancies we chose to exclude their model from our analysis.
C.3.2 CLOSED- SOURCE MODELS
In Table 14 we report numbers for three additional FMs that either do not report a complete set of results (Moirai) on all seven datasets or are closed-source (LLM4TS) or both (Toto). We report ag- gregate metrics for a six dataset setting in Table 17, where we see that Toto dominates (despite being zero-shot) while the performance of Auto-AR is comparable to that of LLM4TS. A look at Figure reveals that Toto is not superior across all six tasks, with the aggregated metrics being strongly boosted by its dramatically better performance on one of them (ETTh2). Thus, while its ZS perfor- mance is quite good, it is unclear whether this result would be maintained with additional tasks.
22

---

Published as a conference paper at ICLR 2025
Model
ETTh1
ETTh22
ETTml
ETTma2
Hoeizon
96
192
720
96
192.
720
96
192.
720
96
192.
336
720
GPT4TS (OFA)
0.316
0,416
0.442
0.473
0.285
0.354
0.573
0.405
0.292
0.332
0.306
0.417
0.175
0.229
0.236
0.378
TEST (Few shot)
0.455
0.572
0.611
0.723
0.332
0.401
014088
0.459
0. 1.399
0.425
0.471
0.552
0.288
0.300
0.359
0-452
LLMITS
0.371
0.403
0.42
0.422
0.269
0.328
0.353
0.383
0.285
0.324
0.353
0.408
0.165
0.22
0.268
0.15
MOMENT
0.387
0.4
0.422
0.454
0.288
0.349
0.560
0.403
0.293
0.536
0.552
0.405
0.17
0.227
0.275
0.363
TIM (B)
0.36
0.392
0.401
0.436
0.269
0.336
0.359
0.39
0.291
0.525
0.363
0.419
0.164
0.219
0.277
0.35
TIM (A)
0.363
0.442
0.262
0.524
0.351
0.392
0.283
0.353
0.393
0.158
0.213
0.269
0.369
SPIP-LLM
0,366
0
0,412
0.44
0.278
0.346
0,367
0.4
0.288
0.523
0,399
0.403
0.165
0.223
0.277
0.363
CALE
0,369
0.427
0.456
0.479
0.279
0.253
0.302
0.404
0.325
0.374
0.409
0.477
0.178
0.242
0.300
0.391
TEMPO Zero Shot)
0.4
0.426
0.64]
0.143
0.301
0.355
0.579
0.409
0:438
0.461
0.515
0.59]
0.185
0.243
0.309
0.386
TimesFM (Zero Shat)
0.421
0,472
0.51
0.514
0.326
0.399
11434
0.451
0.357
0.411
0.441
0.507
0.205
0.254
0.367
0473
Moirai (Zero Shot)
0.575
0.399
0.413
0.281
0.54
0.38
0.404
0.435
0.49
0.205
0.261
0.319
0.415
Shoo))
0.307
0,419
0.093
0.204
0.306
0.463
0.764
0.354
DLinear
0.375
0.439
0.472
0.289
0.383
0.605
0.299
0.425
0.167
0.224
0,281
0.397
ARIO-ARIMA
0.646
0.704
0.733
0.738
0.524
0,411
0.456
0.462
1.151
1.173
1.231
0.225
0.298
0.37
0478
AR (d-O)
0.35%
0.39
0.41
0.424
0.271
0.334
01.611
0.395
0.299
0.336
013008
0.426
0.165
0.218
0.271
0.366
AutoAAR
0.357
0.39
0.41
0.422
0.269
0.332
0.359
0.391
0.299
0.336
0.368
0.426
0.163
0.218
0.271
0.367
DASHA
0.369
0.401
0,430
0.478
0.284
0.377
0.396
0.745
0.305
0.335
0.507
0.418
0.169
0.224
0.250
0.378
Model
Weather
Electrictty
11.1
Tratto
Horizon
96
192
336
720
96
192
336
720
96
192
336
720
96
192
336
720
GPT,TS (OFA)
0.162
0.204
0.254
0.526
0.139
0.153
0.169
0.206
2,063
1. 1888
1.79
1.979
0.388
0.407
0.412
0.45
TEST (Few shot)
0.163
0.23
0.278
0.501
0.145
0.158
0.17%
0.23
0.415
0.425
0.436
0.489
LLMITS
0.147
0.191
0.241
0.313
0.128
0.146
0.163
0.2
0.372
0.391
0.405
0.437
MOMENT
0.154
0.246
0.315
0.136
0.152
0.167
0,205
2.728
2.728
0.391
0,414
0.45
TIM (B)
B:146
0.242
0.323
0:129
0,149
0,163
0.2
0.368
0,403
0,395
0.21
TIM (A)
0.149
0.192
0.24
0.318
0.128
0.144
0.162
0.191
0.352
0.359
0.375
0.419
5 LLM
0.145
0.19
0.243
0.312
0.135
0.149
0.167
0.2
0.379
0.397
0.407
0.44
CALF
0.164
0.214
(.269)
0.355
0.145
0.161
0.175
(1.22
0.207
0.43
0.44
0.477
TEMPO (oro Shot)
0.211
0.254
0.292
0.37
0.178
0.198
0.209
0.279
0.476
0.696
0.50
0.528
TimesFM Shot)
0.122
0.169
0.391
0.119
0.158
01.00
0.327
042
Moirai (Zoro Shoo)
0.216
0.26
0.32
0:205
0:22
0.236
0.27
Toto (Zero Shoo)
0.18
0.235
0.252
0.356
0.124
0.138
0.155
0.211
DLineer
0.176
0.22
0.265
0.323
0.14
0.153
0.160
0.203
2.215
1,963
2.13
2.368
0.41
0,423
0.466
0.466
Auto ARIMA
0,217
0.263
0.33
0.425
1:22
1.264
1.311
1.364
5.854
6.94
7.192
6.648
1.997
2.044
2,096
2.138
AR (f-O)
0.171
0,215
0,263
0.332
0.138
0.153
0.17
0.212
2,084
2.04
2,004
2.011
0.398
0.413
0.436
0.644
Auto-AR
0.172
0.215
0.263
0.332
0.138
0.153
0.17
0.212
2,084
2.04
2,004
2.011
0.398
0.413
0.426
0.464
DASHA
0.168
0.205
0.251
0.314
0.186
0.200
Table 14: Time series forecasting tasks (MSE). 22 indicates unknown quantities.
Model
ETTh]
ETTh2
ETTml
ETTm2
Horizon
96
336
720
96
336
720
96
3356
720
96
336
720
GPT4TS (OFA)
0.397
0.418
0.433
01.4%
0.342
0,407
0.441
0.346
0.372
0.394
0.421
0.262
0.341
0.401
TEST (Fow shot)
0:457
0.519
0.594
0.574
0.44
0.4%
0.401
0.426
0.501
0.262
0.
0.302
0.341
0.419
LLM-TS
0.394
0,412
0.422
0.444
0.332
0.377
0.396
0.425
0.343
0.366
0.385
0.49
0.254
0.207
0.306
0.38
MOMENT
0.41
0.426
0.437
0.472
0.345
0.386
0.408
0.430
0.340
0.368
0.584
0.46
0.26
0.297
0.328
0.387
TTM (B)
TTM (A)
SPIP-IIM
0.396
0:42
0,431
0.458
0.34
0.406
0.4%6
0.346
0.418
0:257
0.33
0.30
0.389
0.467
0.331
0.126
0.349
0.438
0.256
0.339
0.393
TEMPO iZero Shot)
0:406
0.421
0.43
0.451
0.353
0.389
0.408
0.44
0:424
0,432
0.467
0.509
0.267
0.304
0.345
0.395
TimestM (Zoro Shot)
0.421
0.472
0.51
0.514
0.525
0.399
0.434
0.451
0.357
0.411
0.441
0.507
0.205
0.294
0.367
0.433
Moirai (Zero Shot)
0.402
0.419
0.429
0.344
0,334
0.373
0.393
0.416
0,383
0.502
0.416
0.437
0,282
0.318
0.355
0.41
Toto (Zero Shoot
0,366
0.368
0.399
0.424
0.197
0.251
0.260
0.355
0.328
0.353
0.389
0.429
0.27
0.315
0.319
0.374
DLineer
0.399
0.416
0.643
0.49
0.353
0.418
0.465
0.551
0.343
0.365
0.386
0.421
0.26
0.303
0.342
0.421
AURO-ARIMA
0.537
0.608
0.368
0.454
0.467
0.659
0.701
0.725
0.296
0.381
0.438
Auto-AR
0,388
0.423
0.848
0.334
0.376
0/402
0.437
0.343
0.118
0.251
0.325
0.381
Model
Weather
Electricity
11.1
Traflic
Hlrizcon
96
192
336
720
96
192
336
720
96
192
336
720
96
192
336
720
GPT4TS (OFA)
0.212
0.248
01286
0.337
0.238
0.251
(12666
0.297
0.881
0.
0.892
0.884
0.957
0.282
0.294
0.312
LLM4TS
0.196
0.238
0.277
0.329
0.225
0.24
0.292
0.259
0.275
0.292
TEST (Fow shot)
0.213
0.263
0.282
0.328
0.235
0.255
0.275
0.311
0.317
0.31
0.338
MOMENT
0,209
0.248
0.285
0.336
0.255
0.247
0.2664
0.205
1.114
1.092
1.098
1.126
0.282
0.257
0.292
0.31
TTM (B)
TTM (A)
1PMP-ILM
0.195
0.235
0.28
0.336
0.23
0.266
0.287
0.274
0.262
0.280
0.301
CALF
0.204
0.291
0.352
0.238
0.252
0.300
0.268
0.281
0.3
TEMPO ieero Shot)
0.254
0.298
0.332
0.379
0.276
0.203
0.309
0.355
0.343
0.355
0.356
0.376
TimestM (Zero Shot)
0.122
0.189
0.242
0.391
0.119
0.137
0.158
0.200
0.327
0.353
0.378
0.42
Moirai (Zero Shoo)
0.212
0.25
0.282
0.522
0.299
0.31
0.323
0.347
Tato (Zero Shoo)
0.225
0.267
0.291
0.356
0.212
0.232
0.249
0.291
DLinear
0.237
0,282
0.319
0.362
0.237
0.249
0.267
0.30]
1,081
0.963
1.024
1.096
0.282
0.287
0.296
0.315
Auto ARIMA
0.267
0.341
0.53
0.441
0.468
0.435
0.54
0.755
0.932
1.243
Aulo-AR
0.225
0.294
0.34]
0.233
0.301
0.964
0.947
0.958
0.28
0.286
0.319
Table 15: Time series forecasting tasks (MAE). "_" indicates unknown quantities.
23

---

Published as a conference paper at ICLR 2025
Average MAE
Average
Mean
Median
Model
Rank |
%Imp.t
%1mp.1
GPT4TS
0.337
4.54
33.01
30.07
TEST (Few Shot)
0.370
6.36
27.35
26.89
Foundation Models
MOMENT
0.336
3.68
33.32
30.19
S'IP-LLM
0.331
2.54
34.22
32.68
CALF
0.335
3.68
33.50
29.63
DLinear
0.350
5.61
29.87
26.10
Supervised
Auto-ARIMA
0.553
7.82
0.00
0.00
Models
Auto-AR (our workflow)
0.333
2.71
33.86
29.46
Table 16: Aggregate performance time series tasks seven tasks based MAE.
Model Size
Pretraining Series
Average RMSE 4
Mean %Imp.t
Median
%Imp.t
Moirai (Zero Shot)
14M
6M
0.566
24.53
18.52
LLM4TS
60M
8M
0.526
29.25
20.96
TTM (A)
5M
1M
0.525
29.38
19.87
Toto (Zero Shot)
103M
0.505
32.40
31.35
Auto-AR
513
O
0.534
28.12
19.63
DASHA
480K
0
0.549
25.94
16.78
Table 17: Aggregate results for the excluded time series FMs in the six-dataset setting discussed in Appendix C.3.2. "_" indicates unknown quantities.
taske
Auto-AR TEMPO
60
0.70
TImesFM
50
LLM4TS
0.85
Tato Auto-AR
RMSE (averaged
improvement
20
ETTh1
ETTh2
ETTm1
ETTm2
Weather
ECL
0.50
96
192
336
720
Figure 7: Scatterplot of improvement in RMSE (averaged across time horizons) two closed-source FMs, LLM4TS and Toto, on a subset of datasets in Table 14. It shows that both FMs usually perform similarly the Auto-AR baseline, with Toto attaining strong aggregate metrics due to dominant performance on ETTh2.
Horizon
Figure 8: Zero-shot performance comparisons on all time series datasets, as measured by the aver- age RMSE across tasks. As in Figure 6, even in the ZS regime the Auto-AR baseline compet- itive the shorter time horizons with all open- source FMs.
D COMPUTATIONAL COSTS
Here we study the costs our automated baselines against the top three FMs. The aggregated results for each domain are shown in Table 18, along with explanation for how they were obtained. For genomics, fine-tuning FMs takes approximately the same time as training our baseline (.36 GPU- hours). Developing and tuning our models takes 1.82 GPU-hours, which is better than all FMs except Caduceus-PPi For satellite data, both tnnettnnin FMs and optimizing the hyperparameters of the fine-tuning procedure takes approximately 2x longer than training baselines. Lastly, for time series data our full pipeline is more than 30 times faster than larger language model (LLM)- based FMs such 12IP-LLLM and more than times faster than TTM.
24

---

Published as a conference paper at ICLR 2025
GPU Hours
model
model development tuning
fine-tuning
NT-Multiipeccies (500M)
2.8
0.28
HyenaDNA-1K
6.0
0.60
Genomics
Caduceus-PH
0.40
0.10
DASHA
1.82
0.36
SatMAE-Lagee
57.60
14,40
Satellite
CROMA-Large
81.88
20.47
Imaging
DOFA-Large
75.04
18.76
DASHA
29.24
8.97
MOMENT
1.22
Time
TTM (A)
0.71
Series
I'IP-LLM
24
Auto-AR
0.20
0.01
Table 18: Computational costs of perpefforring FMs and our methods (DASHA and AutoAR), All experiments were conducted on L40 GPUs (L40S for Genomics). For FMs, model development time only considers the time taken for hyperparameter tuning, and does not count the time taken to pre- train the models. "_" means that the model tuning time is unknown. Similarly, for NT-Multispecies (500M) and HyenaDNA-IK, the model tuning time derived by 10 times the fine-tuning time, based on Section 1.1.4 in Dalla-Torre et al. (2023). For satellite models, the model tuning time is similarly derived by reporting 4 times the fine-tuning time, which is the size of the hyperparameter sweep that we used. While we do not have good tuning estimates for time series, based on the above it is reasonable assume would take least as long as fine-tuning the model once, which informs the pale blue bars in the right plot of Figure 4.
E SUBSAMPLING EXPERIMENTS
common argument in favor of FMs that they perform better in data-limited regime. Here, we examine the sensitivity of our results downscaling the amount of available data. Specifically, for each domain-specific dataset, we use only 20% of the original data to train the supervised learning baseline and fine-tune the two best-performing foundation models in each domain. For satellite and genomics, the subset of the data is randomly selected from the original training dataset. For time series, the data are selected from the last 20% of the training series. We display the mean performance on all datasets in each domain in Table 19. The result demonstrate that modern FMs struggle to outperform simple supervised baselines even in data- limited regimes.
model
100%
20%
metric
NT-Multispecies (500M)
0,700
0.656
Genomics
HyenaDNA-IK
0.708
0.516
average
Caduceus-PPe
0.725
0.610
score (1)
DASHA
0.761
0.595
SatMAE-Large
77.75
70.96
Satellite
CROMA-Large
78.03
73.13
average accuracy (t)
Imaging
DOFA-Large
78.80
74.26
DASHA
77.85
71.49
MOMENT
0,550
0.594
Time
TTM (A)
0,538
0.549
average
Series
I-IP-LLM
0,545
0.564
RMSE (L)
Auto-AR
0.551
0.559
Table Subsampling experiments on 20% subset of finetuning data: the aggregated performances are obtained by averaging individual performances of models on different datasets in each domains. Overall, our supervised methods remain competitive even the limited training data setting.
25

---

Published as a conference paper at ICLR 2025
F SERO AUTO-AR
In this section, we describe how to run Auto-AR in a zero-shot manner, enabling a direct and fair comparison with ZS time series FMs. This approach is check on the performance of such models, serving as a minimal baseline that they should be expected to overcome.
Given an input time series of length L and prediction horizon H, the zero-shot Auto-AR methods works as follows:
1. segment the input into smaller fragments using a rolling window of size W such that W < L, resulting in L W + 1 samples of length W
2. run regular Auto-AR (the three steps described in Section 3.2) on these samples (for efficiency we restrict the search space over the lookback parameter to (64, 96, 128, 192))
3. use the resulting model for prediction
Since this approach uses only the test example of length L (e.g. I. = 512), this approach uses the same data as given to 7S time series FM and so can used as a fair baseline for the latter.
We evaluate ZS Auto-AR on all the tasks we consider and report the results in Tables 20 and 21. Aggregate performance is visualized in Figures 6 and 8, where we compare with the zero-shot forecasting performance of Auto-AR FMs such as TEMPO, TimesFM, Toto, and Moirai across different forecasting horizons. A key expectation that ZS Auto-AR, trained on only a few chunks of temporal data, should struggle significantly compared to FMs, which are pretrained millions of time series. However, at shorter horizons (96 and 192), Auto-AR performs competitively with foundation models, with only a marginal gap in average RMSE. Only at longer forecasting horizons does Auto-AR's performance deteriorates relative to that of FMs, with a widening performance gap. This suggests that ZS FMs do leverage pretraining to maintain robustness over longer horizons but are not significantly better than a simple baseline on short-horizon predictions (apart from Toto).
Model
6TThl
ETTh2
ETTml
ETTm2
Hoeioon
96
192
336
720
96
192
335
720
96
192
336
720
96
192
336
720
TEMPO iZero Shot)
0.4
0:426
0.641
0.443
0.301
0.355
0.379
0.409
0.438
0.161
0.515
0.591
0.185
0.243
0.309
0.386
TisesFM Shot)
0.421
0.472
0.514
0.526
0.399
0.451
0.357
0.441
0.507
0.205
0.294
0.367
0.473
Shot)
0.307
0.329
0.093
0.135
0.294
0.306
0.2
0.354
Moina (Tem Shor))
0.375
0.399
0.412
0.413
0.281
0.34
0.38
0.404
0.435
0.462
0.49
0.205
0.261
0.319
0.15
Auto-AR iZero Shott
0.416
0.464
0.511
0.53
0.311
0.416
0.479
0.555
0.348
0.426
0.489
0.854
0.196
0.272
0.357
0.549
Model
Weather
Flectrictty
11.1
Traffic
Horizon
192
96
192
720
96
720
96
720
TEMPO iZero Shor)
0.211
0.254
0.292
0.37
0.178
0.198
0.309
0.279
0.476
0.496
0.503
0.538
TimesEM iZero Shot)
0,122
0.169
0.242
0.391
0.119
0.137
0.158
0.206
0.327
0.353
0.378
0.42
Toto (Zem Shof)
0.18
0.235
0.252
0.556
0.124
0.158
0.155
0.211
Moirai (Zero Stoot)
0.173
0.216
0.26
0.32
0.205
0.22
0.236
0.27
Auto-AR (Zero Shot)
0.184
0.235
0.3
0.861
0.158
0.177
0.205
0,278
2,731
2.357
2,198
2,074
0.461
0,503
0.557
0.616
Table 20: Zero-shot results (MSE). "L" indicates unknown quantities.
Model
BTThl
E1Th2
ET'ml
ETTTm2
Horison
96
192
336
720
96
192
336
730
96
192
336
720
96
192
336
720
TEMPO (Zero Shot)
0.406
0:421
0.43
0.451
0.355
0.389
0.44
0.424
0.432
0.467
0.509
0.267
0.314
0.345
0.395
TimesFM Shor)
0.421
0.472
0.514
0.336
0.509
0.451
0.357
0.507
0.205
0.473
Moirai (Zero Sbot)
0.402
0.419
0.644
0.334
0.573
0.416
0.383
0.137
0.282
0.318
0.41
Toto (Zero Shot)
0.366
0.368
0.399
0.424
0.197
0.231
0.260
0.355
0.328
0.353
0.389
0.429
0.27
0.315
0.319
0.374
Auto-AR AR iZero Shott
0.416
0.464
0.511
0.53
0.311
0.416
0.479
0.555
0.348
0.426
0.489
0.854
0.196
0.272
0.357
0.549
Model
Wenther
Electrictty
11.1
Traffic
Horizon
96
192
720
96
192
720
96
192
720
96
192
720
TEMPO ieero Shot)
0.254
0.298
0.332
0.379
0.276
0.293
0.309
0.355
0.343
0.355
0.356
0.376
TisestM izero Shot)
0.122
0.169
0.242
0.391
0.119
0.157
0.158
0.206
0.927
0.353
0.578
0.42
Maitai (Zemo Shot)
0.212
0.25
0.262
0.322
0.290
0.31
0.323
0.347
Toto (Zem Shot)
0.223
0.267
0.291
0.356
0.212
0.232
0.349
0.201
Auto-AR (Zero Shot)
0.184
0.235
0.664
0.158
0.177
0.205
0.278
0.987
0.949
0.936
0.960
0.461
0,503
0.557
0.546
Table 21: Zero-shot results (MAE). 5." indicates unknown quantitics.
26


Reviews:
[{"review_id": "OJX6LvImgW", "reviewer": "Reviewer_E1xK", "summary": "This paper aims to challenge the assumption that foundation models (FMs) consistently outperform traditional supervised learning methods. To support this, the authors design an automated machine learning framework, utilizing neural architecture search for genomics and satellite imaging tasks, and an Auto-ARIMA approach for time series analysis. These automated workflows produce competitive supervised models, providing strong baselines against which FMs are compared.\n\nThe study highlights that, unlike in general-purpose AI tasks (e.g., NLP or computer vision), the advantages of FMs in specialized fields remain unproven (FMs frequently fail to surpass traditional supervised learning models that are optimized for each task), emphasizing the need for diverse, well-tuned baselines in FM evaluation.", "questions": "My primary concerns regarding the acceptance of this work are based on points 1 and 2 described in the “weakness” section. These concerns stem from my own experience, particularly regarding point 1, where foundation models (FMs) generally demonstrate greater robustness in small-scale data adaptation through fine-tuning in vision and language tasks. However, this behavior may differ in the other modalities used in this paper.\n\nThe claim that “FMs in these domains have not yet surpassed supervised workflows and are often outperformed by fairly simple methods, including lightly modified CNN backbones (in genomics and satellite imaging) and classical linear forecasters (for time series)” may be somewhat overstated. This is especially relevant when considering factors such as data bias, the challenges of hyperparameter optimization for FMs, and potential out-of-distribution issues that could lead to negative transfer due to pretraining data limitations.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-organized and well-written. The research questions are both interesting and practical, particularly in highlighting that some foundation models (FMs) are not fully comparable to in-domain, supervised baseline models. The experimental setup and results are comprehensive, effectively supporting the study’s objectives and providing valuable insights into the performance of FMs relative to tailored supervised models.\n\n2. The two proposed automated supervised pipelines, DASHA and Auto-AR, are designed with detailed insights into model architecture, resulting in lightweight, simple yet powerful baselines. In particular, the development of Auto-AR highlights previously overlooked limitations and gaps in earlier works, encouraging the research community to rethink classical methods.\n\n3. The paper demonstrates high quality through its methodological rigor and detailed experiments. (1) Thorough Benchmarking: The authors evaluate multiple foundation models (FMs) across three distinct scientific fields, using datasets and tasks that reflect real-world applications. This comprehensive experimental setup ensures that the results are robust and meaningful. (2) Well-Constructed Baselines: The paper establishes carefully optimized baselines using existing supervised models. This level of detail ensures fair and well-founded comparisons, providing a reliable foundation for drawing conclusions.", "weaknesses": "1. The strong performance of supervised methods through the automated pipeline may be largely attributed to the large-scale training data. As noted in the data statistics in the appendix, nearly all datasets contain over 10,000 samples for training, which is sufficient for effective supervised training from scratch, even without pretrained checkpoints. For a comprehensive evaluation and to support the claim that pure supervised learning can outperform pretraining, additional experiments varying data scale (e.g., using different ratios of training data) are needed. This would likely highlight the advantages of large-scale pretraining.\n\n2. In addition to performance metrics like accuracy and model parameters, other essential factors are not addressed in the paper. Specifically, the authors do not mention the time and computing resources required to obtain the optimal architecture and hyperparameters. Compared to directly fine-tuning foundation models (FMs), it is unclear whether the automated pipeline is more time- and compute-intensive. Including these comparisons would provide a fuller understanding of the trade-offs involved in using the automated pipeline.\n\n3. The assumption that “their creators make a best-effort attempt to present their own method in the best light” may not be fully reasonable. Due to the high computational costs and complexity of foundation models, obtaining optimal or even sub-optimal hyperparameters is generally challenging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xK5KwPNFZe", "reviewer": "Reviewer_TNS8", "summary": "The paper examines the effectiveness of foundation models (FMs) in specialized domains like genomics, satellite imaging, and time series data, relative to traditional supervised learning models. The authors aim to challenge the assumption that foundation models necessarily outperform supervised models in these domains. Using a rigorous experimental framework, they contrast FMs with supervised baselines, emphasizing cases where the benefits of large-scale pretraining do not translate into superior downstream performance. Through their novel automated pipelines—DASHA for genomics and satellite imaging, and Auto-AR for time series—the authors provide a streamlined, data-efficient, and computationally lighter alternative to FMs, that demonstrates competitive performance across several tasks. The study raises important considerations about the necessity and cost-efficiency of pretraining-heavy foundation models in specialized fields.", "questions": "Q1. Could the DASHA and Auto-AR workflows be extended or adapted for multi-modal datasets or tasks involving mixed data types? If so, what modifications might be necessary?\nQ2. How do the authors envision the role of foundation models evolving in these domains if computational resources continue to scale? Would certain configurations of FMs potentially start to outperform supervised baselines, or do they anticipate the observed trends to persist?\nQ3. Given the success of NAS in selecting optimal CNN architectures for genomics and satellite imaging, do the authors plan to explore whether reinforcement learning or meta-learning approaches might yield further performance gains?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The benchmarks and datasets chosen for each domain appear to comprehensively cover relevant applications in genomics, satellite imaging, and time series.\n\n2. The paper highlights computational efficiency as a priority by contrasting the costs associated with FMs versus the supervised baselines. Holds high relevance given the fast adoption of large, resource-intensive models in research and industry.\n\n3. Contextualized the results within each domain, providing a nuanced understanding of the model performance across tasks.", "weaknesses": "1. The paper lacks an exploration of their applicability to other types of data or more complex, multi-modal tasks. A broader discussion of potential limitations could help future researchers understand the contexts in which DASHA and Auto-AR might or might not succeed.\n\n2. The choice of baselines (e.g., Wide ResNet and UNet for genomics and satellite imaging) is reasonable but could be expanded. Although the selected baselines demonstrate competitive performance, further comparisons with more recent architectures could reinforce the study’s claims regarding the relative inefficacy of FMs in these domains.\n\n3. Although the paper effectively highlights the efficiency of supervised workflows, the focus on computational cost may not fully account for the trade-offs in scalability and generalization that FMs can offer.\n\n4. For time series, the paper does not sufficiently address the unique challenges of zero-shot forecasting. Including more details on why certain zero-shot FMs performed poorly could provide actionable insights for model improvement in zero-shot contexts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yOxuefMoym", "reviewer": "Reviewer_hQrf", "summary": "This work evaluates the effectiveness of foundation models (FMs) in genomics, satellite data, and time series, finding that well-tuned supervised models can match or outperform FMs in these specialized domains. The results suggest that large-scale pretraining benefits are not yet fully realized in these areas, highlighting the need for solid baseline comparisons.", "questions": "See the weakness section for possible improvements. I am willing to listen to the author's feedback and raise my score during the rebuttal period if the weaknesses are addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1) The paper addresses a highly relevant topic that warrants investigation, given the growing popularity of FMs and the ongoing efforts within the community to adapt these models to specialized domains beyond natural images and text.\n\n2) The results are surprising and critical to keep a fair benchmarking of future FMs against other pipelines. The actual gains in data efficiency scored by the proposed baselines (nice summary in Figure 1) range from three to five orders of magnitude in the three tested domains (satellite imaging, genomics, and time series). The authors also plan to release the code for their baselines, providing the community with a valuable benchmark resource.\n\n3) The experimental setup is comprehensive, with models evaluated on up to 50 tasks. I found the design of the supervised baseline to be rigorous and fair, especially the decision to approximate standard practitioner model development through architecture search. \n\n4) The paper is clear, well-written, and easy to follow, making its main message easy for the reader to understand.", "weaknesses": "1) The paper presents two \"solutions\" from an architectural standpoint, i.e., CNNs for genomics/satellite data and auto-AR for time series. This leads to two questions: \n- What model should a practitioner use when facing a task in a different specialized domain? \n- How come 1D CNNs (or, e.g., RNNs) fail to match the performance of FMs in time series forecasting?\n\nIs it just a matter of domain knowledge and data type? In practice, the paper would benefit from some more convincing justification and evidence for the statement, \"While DASHA can be applied to forecasting tasks, it is not competitive with state-of-the-art time series FMs.\" (Line 240).\nAlso, why in Table 3 the evaluation of DASHA for the \"Full setting\" panel is reported as \"unknown\" (\"-\")? \n\n2) The authors should add a brief discussion either in the Introduction or Related Work with references regarding \"simple tuned baselines\", which were shown to outperform/match more complex approaches, as this fact has been seen multiple times in the machine/deep learning literature and is clearly relevant to this work. Some related literature:\n\n- Evaluating Weakly Supervised Object Localization Methods Right, CVPR 2020\n- In Search of Lost Domain Generalization, ICLR 2021\n- Image Classification with Small Datasets: Overview and Benchmark, IEEE Access 2022\n- OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization, CVPR 2022\n- When Do Neural Nets Outperform Boosted Trees on Tabular Data?, NeurIPS 2023 Datasets & Benchmarks\n- Systematic Comparison of Semi-supervised and Self-supervised Learning for Medical Image Classification, CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AntkMa3Ikv", "reviewer": "Reviewer_9w92", "summary": "The paper conducts a comparitive study of foundation models across three domains of time-series data, genomics and satellite data, whereby a selection of foundation models in each domain are compared to a supervised baseline. More specifically, the authors propose an Auto-ML pipeline based on DASH (Shen et al. 2022)and ASHA (Li et al. 2020). The results show that foundation models are not\nable to consistently beat supervised tuned baselines across these domains.", "questions": "General Questions:\n- I don't really understand the importance of including a ResNet and a UNet, as models for classifcation. From my understanding a UNet is an architecture choice where you need the model output to also be a HxW similar to the input. For classification this would imply that you have to include additional layers to get from that output to a vector-valued classification prediction. Additionally, I think a UNet can have all sorts of different convolutional or transformer encoders, including a ResNet, for example see [here](https://github.com/qubvel-org/segmentation_models.pytorch). Having access to the code would probably answer this question.\n- In line 255, you state that beyond a subset of the GeoBench classification tasks you also include BigEarthNet, and EuroSAT, as well as two additional ones. However, BigEarthNet and EuroSAT are already included in GeoBench\n- CROMA takes as input an optical and/or sar image as the code suggests [here](https://github.com/antofuller/CROMA/blob/59505a6bcadbf36ba20767270154bf9f3067c5e7/use_croma.py#L90). I wonder how that is handled across the selected different classification datasets, as it was not entirely clear to me from the appendix around line 953\n- Is it possible to look at the code already which could help answer some of these questions?\n\nOther Comments:\n- in line 261 you state that \"satellite imaging closely resembles RGB imaging\" but that is only true for some optical satellites. Radar Satellites as well as other images are very different than RGB\n\nMinor Comments:\n- line 525 there is a typo in the word \"work\"\n- I don't find the acronym \"NAS\" to be introduced. Also in line 191 you say \"neural architecture search\" where you use NAS beforehand already", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Foundation models are a hot topic in various machine learning research domains, and it is important to have appropriate evaluation regimes that can demonstrate the efficacy or show weaknesses in developed models and test whether they are able to deliver on their promises. Therefore, the research question posed by the authors is highly relevant and of importance to the community. Similarly, it is good that the authors focus on a rather straightforward simple baseline tuning technique that can be employed across domains.", "weaknesses": "I do understand the authors' idea and motivation to show subpar performance of foundation models against a well tuned supervised baseline across different domains. However, by attempting to do this across quiet different domains, I believe that the paper and methodology is falling short of the specific nuances of each of the included subdomains of genomics, time-series and satellite data. \n\nI will mainly attempt to make this point for the satellite data domain as this is the most closely related one to my background among the three:\n- Earth Observation (EO) data encompasses a wide array of sensors and modalities. Additionally, there are a diverse set of tasks, classification, segmentation and pixel-wise regression just to name a few for optical data. Foundation models aim to perform well across these tasks based on a common encoder that yields relevant representations that can then be \"decoded\" for the task at hand. However, while you do select several classification datasets, the evaluation is limited to classification and does not include segmentation which is arguably more relevant in EO. Therefore, I would argue it is difficult to make your overall claim, if you do not compare across EO tasks, for example including segmentation, as this would align more with the premise of FMs.\n- In line 352 you say \"we only consider a restricted subset of top-performing, open-source, and compatibly-formatted models\". However, I would believe that there are other relevant and easily accessible recent models available like [Clay](https://github.com/Clay-foundation/model), [Scale-MAE](https://github.com/microsoft/torchgeo/blob/main/torchgeo/models/scale_mae.py), the [Satlas-model](https://github.com/microsoft/torchgeo/blob/main/torchgeo/models/swin.py), or [DOFA](https://github.com/microsoft/torchgeo/blob/main/torchgeo/models/dofa.py) that include evaluations against your chosen models of CROMA and Sat-MAE.\n- Regarding the experimental findings, I would argue that this is not necessarily new, because there are works that already show strong performances of supervised baselines. For example in the [Prithvi by Jakubik et al.](https://arxiv.org/abs/2310.18660) Table 4 you observe the competitive UNet performanceand that for subclasses it is often better than the FM. Similarly, in [DOFA by Xiong et al.](https://arxiv.org/abs/2403.15356), Table 1 and 2 show that fully trained supervised baselines can sometimes beat the finetuning schemes. More general a work by [Corley et al.](https://openaccess.thecvf.com/content/CVPR2024W/PBVS/html/Corley_Revisiting_Pre-trained_Remote_Sensing_Model_Benchmarks_Resizing_and_Normalization_Matters_CVPRW_2024_paper.html) conducted a focused study around pretrained weights and demonstrated that carefully finetuned baselines are very competitive.\n- From Table 6 it seems that you have conducted the experiments on quiet large datasets in the domain. However, the more interesting application of FMs is for fine tuning on datasets with scarce/few labels where training a model from scratch would likely just lead to over fitting. This domain relevant scenario is not considered here.\n\nGiven that for the satellite subdomain only a subset of existing FMs were used for classification (which is understandable since this requires significant experimental effort), I think some of the author's general statements do not follow the methodology. For example, the authors lead their discussion by stating in line 470 that \"At a high level, our results show that the foundation models in these three domains have not yet surpassed supervised learning\" but the selected methods are not representative of \"the foundation models in the domain as illustrated for satellite data. As I said in the beginning, I believe that there are too many nuances within the separate domains that can be all included in one paper and allow for the conclusions that the authors draw.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)