Specialized Foundation Models struggle to beat Supervised Baselines

## Abstract

Abstract Following its success for vision and text, the “foundation model” (FM) paradigm—pretraining large models on massive data, then fine-tuning on target tasks—has rapidly expanded to domains in the sciences, engineering, healthcare, and beyond.
Has this achieved what the original FMs accomplished, i.e. the supplanting of traditional supervised learning in their domains?
To answer we look at three modalities—genomics, satellite imaging, and time series—with multiple recent FMs and compare them to a standard supervised learning workflow: model development, hyperparameter tuning, and training, all using only data from the target task.
Across these three specialized domains, we find that it is consistently possible to train simple supervised models—no more complicated than a lightly modified wide ResNet or UNet—that match or even outperform the latest foundation models.
Our work demonstrates that the benefits of large-scale pretraining have yet to be realized in many specialized areas, reinforces the need to compare new FMs to strong, well-tuned baselines, and introduces two new, easy-to-use, open-source, and automated workflows for doing so.

## 1 Introduction

Recent years have witnessed a shift towards large-scale pretraining across domains like computer vision and natural language processing. This workflow generally consists of two stages: pretraining on vast amounts of domain-specific data to capture general knowledge followed by fine-tuning on target tasks (Radford & Narasimhan, ). This pretrain-then-finetune paradigm has been tremendously successful, enabling foundation models (Bommasani et al., ) to consistently outcompete traditional supervised learning methods on a wide variety of downstream tasks in the vision and language domains (Dosovitskiy et al., ; Liu et al., ; Devlin et al., ).

Driven by this success, the foundation model approach has been adapted to various specialized domains, which we define to be ML application areas—e.g. genomics, satellite imaging, and time series—whose data modalities lie outside those of classical AI tasks, i.e. natural images and text.
These domains have seen the introduction of many new FMs claiming to leverage large, domain-specific pretraining datasets to achieve breakthrough performance on downstream tasks (Dalla-Torre et al., ; Nguyen et al., ; Zhou et al., ; Avsec et al., ; Ji et al., ; Fuller et al., ; Cong et al., ; Mendieta et al., ).
These claims underlie our study’s motivating question:

Do these new specialized FMs outperform traditional supervised learning applied to the same tasks?

Figure: Figure 1: Across three domains—genomics, satellite imaging, and time series—specialized FMs fail to significantly improve upon tuned supervised learning despite using two-to-five orders of magnitude more data. In contrast, breakthrough FMs such as BERT dramatically outperformed supervised baselines in NLP (top left), causing the field to switch to fine-tuning as the default approach. For each domain we plot total pretraining and fine-tuning data used vs. the mean improvement across tasks over the supervised state-of-the-art. Specifics of our evaluations on the last three domains are in Section 4, while the NLP results are derived from the GLUE benchmark (Wang et al., ). Note that in the x-axis of the top left figure we ignore tokens used to pretrain word embeddings.
Refer to caption: x1.png

Answering this question is critical because supervised workflows are usually much less expensive to implement and deploy, but FMs that allow for effective transfer learning have the potential to fundamentally transform these domains, as we have seen with language and vision processing in the past decade.
However, despite ongoing efforts to promote their fair and comprehensive evaluation (Liang et al., ; Bommasani & Liang, ), many new FMs have not been adequately compared to simpler, often more efficient baselines.
Indeed, we found that many works only benchmark their proposed models against other FMs, essentially creating a comparison echo chamber (Fuller et al., ; Mendieta et al., ; Nguyen et al., ; Zhou et al., ).

We answer our motivating question by considering a reasonably representative set of three specialized domains—chosen according to the presence of multiple FMs and a standard set of evaluation tasks—and comparing their performance on those tasks with that of a traditional supervised learning workflow.
As depicted in Figure 2, the latter is a model development, hyperparameter tuning, and training process in which all steps use only data from the target task, in contrast to the FM workflow, which uses vast amounts of pretraining data.
By leveraging model selection tools ranging from classical information criteria to cutting-edge architecture search, we build automated pipelines that efficiently develop and train strong supervised models on over fifty tasks across three distinct domains.

Our main result is negative: we find that, despite being pretrained on massive datasets, specialized FMs struggle and very often fail to outperform models trained exclusively on downstream task data with traditional supervised learning (c.f. Figure 1).
Specifically, we show that lightly adapted convolutional neural network (CNN) architectures such as wide ResNet and UNet attain state-of-the-art on the Nucleotide Transformer benchmark in genomics and match the latest pretrained satellite FMs on downstream classification.
Furthermore, we show that tuned linear auto-regression (AR) matches or outperforms every open-source time series FM on a standard suite of seven forecasting tasks, despite using four or more orders of magnitude fewer parameters and data.

These results demonstrate that genomics, satellite imaging, and time series have not yet had their “BERT moment” (Devlin et al., ), i.e. these domains have not yet pretrained FMs that dominate traditional supervised approaches.
This is despite the fact that all them have BERT-scale(^1^11Models with 100M+ parameters trained on 100x or more data than supervised tasks in the domain are given.) FMs and the fact that many of them are already witnessing a shift towards not comparing with supervised approaches, as was seen in natural language processing (NLP) post-BERT.
More broadly, since these domains are among the most high-profile areas with specialized FMs, our results challenge the prevailing assumption that pretrained models yield superior performance.
They also reinforce the need for robust and well-tuned baselines, with surprising findings such as (a) simply tuning kernel sizes and dilation rates in standard CNN backbones dominates a genomics classification benchmark and (b) rescuing the century-old AR forecaster from obsolescence is as easy as considering lookback parameters larger than five and training on a GPU.
To facilitate ongoing research in these and other domains, we make code associated with both our CNN-tuning pipeline (DASHA(^2^22[https://github.com/ritvikgupta199/DASHA](https://github.com/ritvikgupta199/DASHA))) and our AR-on-GPU workflow (Auto-AR(^3^33[https://github.com/Zongzhe-Xu/AutoAR](https://github.com/Zongzhe-Xu/AutoAR))) publicly available.

## 2 Related work

Foundation models have been trained in numerous specialized domains beyond vision and text, including genomics (Ji et al., ), satellite imaging (Cong et al., ), time series (Goswami et al., ), weather (Bodnar et al., ), pathology (Zimmermann et al., ), differential equation solving (Sun et al., ), web traffic (Zhao et al., ), and beyond.
To get a representative sense of their success, we focus on domains that combine the following properties:
(a) multiple BERT-scale FMs, (b) a standard suite of evaluation tasks, and (c) significant applied interest.
These restrictions suggest looking at three domains, all of which have at least five FMs evaluated on at least nine tasks:
genomics (which has some of the largest-available non-text FMs (Dalla-Torre et al., )), satellite imaging (which has a large ongoing benchmarking effort (Lacoste et al., )), and time series (which has already seen significant industry interest (Cohen et al., )).
The remainder of this section examines how different learning workflows approach problems in these domains.

Figure: Figure 2: Our goal is to compare the pretrain-then-fine-tune paradigm (top) with a standard supervised workflow (bottom) on the tasks on which specialized FMs are evaluated. While for time series we go through a traditional process of developing and tuning a supervised model, this manual approach does not scale to many domains; as a result, in Section 3.1 we develop a way to simulate it using architecture search. Note that FM fine-tuning hyperparameters are not always tuned in practice, but we assume their creators make a best-effort attempt to present their own method in the best light.
Refer to caption: x2.png

### 2.1 Specialized foundation models

Collectively our three target domains have more than twenty-five FMs, many developed via the “lift-and-shift” approach—borrowing terminology from Rolf et al. ()—in which techniques from core AI areas such as vision and language processing are applied with modest tailoring to specialized domains.
In particular, many methods are built on out-of-domain models such as BERT, Swin, and Hyena (Ji et al., ; Mendieta et al., ; Nguyen et al., ; Shen et al., ), with adaptations such as specialized tokenizations, embeddings, and model modifications for handling domain-specific considerations such long-range dependencies (Dalla-Torre et al., ; Zhou et al., ; Das et al., ; Cohen et al., ; Shen et al., ) or multispectral data (Cong et al., ).

While the “lift-and-shift” approach can often be useful or at least a good starting point, its widespread use underlines the need for strong in-domain baselines to make sure that the combination of out-of-domain tooling and massive pretraining data is actually helpful.
Such comparisons are not always conducted, e.g. the satellite FM SatMAE (Cong et al., ) is compared to ImageNet-initialized and randomly initialized ResNet-50 (He et al., ), while most of the time series FMs we consider only do a full comparison to one linear baseline, DLinear (Zeng et al., ).
While this can sometimes be justified—e.g. in the case of NLP post-BERT—our results suggest that for now, specialized FMs should still compare to in-domain supervised model development.

Lastly, we note that we are not the first to take a somewhat critical look at specialized FMs.
For example, Yang et al. () questioned the dominance of Transformers for protein sequence FMs by showing that convolutions could do just as well, which is related to our discovery that (supervised) CNNs were competitive with (largely Transformer-based) genomics FMs.
Another study by Kedzierska et al. () found that training an in-domain generative model could outperform pretraining one in single-cell biology applications, a finding generalized by our own results, which demonstrate that the underperformance of specialized FMs relative to in-domain training may be a broader trend across multiple domains.
In the time series domain, Tan et al. () show that the popular approach of fine-tuning text-pretrained LLMs on time series tasks often underperforms supervised models (in their case randomly initialized attention);
our results generalize this to other time series FMs and use an even simpler supervised model (AR) to do so.

### 2.2 Specialized baselines

Both of the automated supervised learning pipelines we develop are heavily influenced by successful in-domain model development.
In particular, the NAS-based pipeline we use to achieve our results in genomics and satellite imaging is inspired by the success of the human-driven specification of kernel sizes and dilation rates in successful architectures like TCN (Lea et al., ) and ConvNeXt (Liu et al., ).
At the same time, for time series our approach is based upon a well-tuned GPU implementation of perhaps the most basic forecasting model, AR.

### 2.3 AutoML for specialized domains

While often evaluated on domains such as vision, automated techniques have long been used in specialized domains as well.
An important example is Auto-ARIMA (Hyndman & Khandakar, ) for time series, although it has been found to underperform on the specific suite of tasks we consider (Challu et al., ).
However, to avoid requiring significant expertise in any one domain, we also make use of AutoML methods developed specifically for diverse tasks (Roberts et al., ; Shen et al., ), in particular the NAS method DASH (Shen et al., ) that can discover good kernel sizes and dilation rates for a CNN backbone faster than it can be trained from scratch.

## 3 Methodology

Recall that our goal is to conduct a robust comparison between traditional supervised learning and specialized FMs;
the natural way to do this is to take existing benchmarks used to evaluate FMs in our three target domains and run a typical supervised workflow on the same tasks.
As depicted in Figure 2, this pipeline involves three steps: (1) model development, (2) hyperparameter tuning, and (3) training.
The first stage involves using both reasoning and trial-and-error to find a good architecture to tune and train on the data;
for example, Lea et al. () developed the temporal convolutional network (TCN) architecture with a multi-layer dilation rate pattern specifically suited to sequential data, while Liu et al. () designed the breakthrough ConvNeXt architecture by methodically exploring ways to make CNNs more like Transformers without introducing attention.
The second stage (hyperparameter tuning) can also be done via human-driven iteration, but there exist effective automated procedures for it as well (Li et al., ).
Lastly, the third step of the pipeline involves simply training the selected model with the selected configuration on the data of the target task.

While it is standard to automate the last two steps of the procedure, model development is typically done by hand and so is difficult to do for fifty tasks across three domains.
As a result, we settle for approximating the traditional supervised learning workflow by simulating the model development component using neural architecture search.
To ensure fair comparison and reduce computational costs, we restrict ourselves to low-fidelity NAS methods that return an architecture in less time than it takes to train it.
The results we obtain using NAS can therefore be viewed as lower bounds on the performance of supervised learning, as the model development might be significantly improved using less-heuristic or human-driven architecture design.

In the remainder of this section we detail how we handle the different steps of the supervised learning pipeline.
Note that our NAS-dependent supervised workflow (DASHA)—which we cover in the first part of this section—yields our main results for genomics and satellite imaging but not for time series;
in that domain we find its performance to be less competitive.
There we instead focus on an even simpler approach based on linear auto-regression, whose model development and tuning we describe in the second subsection.

Figure: Algorithm 1 Pseudocode for the DASHA workflow. Starting with a set of backbone CNNs, we use DASH (Shen et al., ) to set the right kernel size and dilation rate for each of its convolutional layers and then use ASHA (Li et al., ) to configure a training routine for the resulting architecture. Lastly, we pick the best backbone using validation data and train it.

### 3.1 DASHA: Simulating the supervised workflow using NAS

To simulate model development we need a search space over architectures that is (a) efficient, (b) flexible, and (c) applicable to the types of high-dimensional unstructured data that arise in domains targeted by specialized FMs;
these requirements make CNN-based search spaces a natural choice.
In particular, inspired by the success of hand-tuned kernel sizes and dilation rates in traditional model development (Lea et al., ; Bai et al., ; Liu et al., ), we apply DASH (Shen et al., ), a NAS method that starts with an existing CNN backbone—e.g. a wide ResNet (Zagoruyko & Komodakis, )—and uses the weight-sharing heuristic (Liu et al., ) to determine the right kernel size and dilation rate to use at each convolutional layer.
DASH has been successfully used in AutoML competitions (Roberts et al., ) and to advance the state-of-the-art on NAS benchmarks (Tu et al., ), making it likely to be useful beyond the domains we consider.

As described in Algorithm 1, we augment the existing DASH approach in two ways:
(1) trying more than one CNN backbone (e.g. both wide ResNet and UNet (Ronneberger et al., )) and (2) using the well-known hyperparameter tuner ASHA (Li et al., ) to configure architecture-specific training settings.
This combination gives our workflow its name.
Following the NAS and hyperparameter tuning stages, we train the discovered architecture with the selected configuration on the target data.
Further details, including the resources given to the three steps of the pipeline and the exact search spaces used by DASH and ASHA, are provided in Appendix B.1.
Note that, while our focus is on data-efficient baselines, we do ensure that the entire workflow is never substantially more computationally expensive than fine-tuning an FM.

### 3.2 Auto-AR: Making a baseline stronger by making it simpler

While DASHA can be applied to forecasting tasks, it is not competitive with state-of-the-art time series FMs.
At the same time, the field of time series forecasting has long employed automated workflows, notably the Auto-ARIMA approach of Hyndman & Khandakar () that uses statistical tests and information criteria to tune ARIMA’s lookback and differencing parameters.
Auto-ARIMA was evaluated on the time series tasks we consider by Challu et al. (), who found that it performed poorly compared to deep learning approaches.
However, their implementation does not make use of multi-channel data and tunes up to a lookback window of at most five, which is much less data than used by time series FMs.
While tuning ARIMA with larger lookback parameters is computationally costly, we find the following simplified tuning pipeline to be effective:

- 1.
use the KPSS test (Kwiatkowski et al., ) to decide whether to take first differences
- 2.
use the Bayesian Information Criterion to select the maximum lookback parameter of the auto-regressive (AR) component of ARIMA, ignoring the moving average (MA) part
- 3.
maximize the multi-channel likelihood of AR with the chosen differencing and lookback

By dropping the MA component of the model and running the procedure on GPU, we are able to tune the lookback windows up to the maximum allowable length (usually 512);
we find that longer lookbacks are critical for performance.
Note that this is just a tuned version of the classic AR model.

## 4 Empirical results

We now present the results of applying the automated pipelines described in the previous section to our three target domains.
For each domain, we provide a brief justification of the specific FMs and evaluation tasks that we consider, followed by details on how we apply our workflows;
further information can be found in Appendices A and B.
As there are too many separate results to present outside the appendix, in this section we mainly present aggregate statistics that summarize our findings for each domain, with detailed results relegated to Appendix C.
The domains have different performance metrics, but they can all be aggregated via the following quantities:
average score, average rank, and mean / median percentage improvement over a baseline. For each domain, we define a domain-specific baseline and measure the improvement of FMs and our approach relative to it. This standardizes comparisons across tasks of varying scales.

### 4.1 Genomics

We begin our investigation in the genomics domain, which has witnessed the development of numerous FMs, including the early Enformer Avsec et al. (), the DNABERT series (Ji et al., ; Zhou et al., ), the HyenaDNA family (Nguyen et al., ), GENA-LM(^4^44We compare to GENA-LM in Appendix C, as its reported metrics differ from the NT benchmark.) (Fishman et al., ), the recent Caduceus family (Schiff et al., ), and the NT family (Dalla-Torre et al., );
The latter includes models with up to 2.5B parameters.
To evaluate them, we consider the Nucleotide Transformer (NT) benchmark of Dalla-Torre et al. (), which contains eighteen tasks in three main categories: regulatory elements, RNA production, and histone modification.
We use this benchmark because of its diversity and because it has been evaluated on by all of the aforementioned FMs, allowing us to include eight of them in the comparison.

Our numbers for these models are taken from Dalla-Torre et al. (, Supplementary Table 6);
Following Dalla-Torre et al. (, Supplementary Table 5), We use F1 score and accuracy to evaluate a subset of regulatory elements and RNA production tasks, and we use Matthew’s Correlation Coefficient (MCC) as the main metric for evaluation on the remaining datasets.

#### 4.1.1 Baselines

CNNs have long been used for genomics tasks (Avsec et al., ; Zhou & Troyanskaya, ) and so constitute natural supervised baselines;
in particular we include 1D variants of Wide ResNet (WRN) and UNet, which we find perform better than some domain-specific CNNs.
We use these same two backbones as the candidate CNNs tuned and selected from by our DASHA workflow.

**Table 1: Aggregate performance on genomics tasks, showing that our supervised workflow (DASHA) attains state-of-the-art on the NT benchmark, outperforming all FMs according to most measures while using no pretraining data and oftentimes many fewer parameters. For Mean/Median %Imp., we report percentage improvement over the Raw Probe baseline from Dalla-Torre et al. (), and for DASHA the model size refers to the largest configuration across tasks. “-” indicates unknown quantities.**
| Model | Model | Pretraining | Avg. | Avg. | Mean | Median |
| --- | --- | --- | --- | --- | --- | --- |
| Size | Base-Pairs | Score $\uparrow$ | Rank $\downarrow$ | %Imp.$\uparrow$ | %Imp.$\uparrow$ |  |
| Foundation Models |  |  |  |  |  |  |
| Enformer | 252M | 4B | 0.569 | 11.86 | 27.73 | 27.91 |
| NT-1000G (500M) | 500M | 20.5T | 0.625 | 10.52 | 33.48 | 36.74 |
| NT-1000G (2.5B) | 2.5B | 20.5T | 0.656 | 7.0 | 36.58 | 40.86 |
| NT-Multispecies (500M) | 500M | 174B | 0.700 | 3.81 | 40.76 | 45.07 |
| NT-Multispecies (2.5B) | 2.5B | 174B | 0.697 | 4.08 | 40.51 | 45.52 |
| DNABERT-2 | 117M | 32.5B | 0.680 | 6.88 | 38.65 | 43.59 |
| HyenaDNA-1K | 1.6M | 3.2B | 0.708 | 6.92 | 41.2 | 43.36 |
| HyenaDNA-32K | 1.6M | 3.2B | 0.630 | 10.22 | 33.96 | 36.93 |
| Caduceus-PS | 1.9M | 35B | 0.689 | 6.69 | 39.08 | 41.38 |
| Caduceus-PH | 1.9M | 35B | 0.725 | 4.69 | 42.63 | 45.01 |
| Supervised Methods |  |  |  |  |  |  |
| Wide ResNet | 2.0M | 0 | 0.694 | 6.83 | 37.16 | 43.08 |
| UNet | 4.5M | 0 | 0.68 | 7.78 | 38.67 | 42.69 |
| DASHA (our workflow) | 10.5M | 0 | 0.761 | 3.69 | 46.33 | 49.08 |

#### 4.1.2 Results

Our genomics results are displayed in Table 1, which shows that our supervised workflow (DASHA) consistently outperforms all FMs across all aggregate metrics.
As discussed in Appendix C, our strong performance is driven in large part by outstanding performance on the histone modification tasks (c.f. Table 9).
The more detailed results also highlight the importance of considering diverse baselines, with Wide ResNet usually being the selected architecture but UNet performing significantly better for promoter and splice site classification tasks.
Overall, DASHA arguably sets a new state-of-the-art on the NT benchmark and certainly demonstrates that supervised methods remain quite competitive in genomics, despite the availability of massive pretraining datasets.

### 4.2 Satellite imaging

While they do not get as large as those in genomics, numerous BERT-scale FMs have also been introduced for satellite imaging, including SeCo (Manas et al., ), the SatMAE family (Cong et al., ), the CROMA family (Fuller et al., ), GFM (Mendieta et al., ), Scale-MAE (Reed et al., ), Satlas (Bastani et al., ), Prithvi (Jakubik et al., ), and SkySense (Guo et al., ).
Because our evaluation includes GeoBench (Lacoste et al., ), a recently introduced satellite benchmark that has not been considered by many of these FMs, we obtain all results using our own fine-tuning;
therefore we only consider a restricted subset of top-performing, open-source, and compatibly-formatted models.
In all cases we use the fine-tuning workflow suggested by the authors of each FM plus some automated hyperparameter tuning;
note that even with the original code and extra tuning our reproductions on previous benchmarks systematically underperformed results reported in the original works.
We take our tasks mainly from GeoBench’s five classification tasks and then add four additional tasks—BigEarthNet (Sumbul et al., ), EuroSAT (Helber et al., ), Canadian Cropland (Jacques et al., ), and fMoW-Sentinel (Cong et al., )—that are commonly used to evaluate other FMs.(^5^55In Appendix C we report results when excluding tasks where missing channels may affect performance.)
As we focus on classification—sometimes with multiple labels—we report top-1 accuracy or mAP as appropriate.

**Table 2: Aggregate performance on satellite imaging tasks, demonstrating that a supervised learning workflow (DASHA) can match the performance of state-of-the-art specialized FMs, all while using no pretraining data and having two-to-ten times fewer parameters. For Mean/Median %Imp. we report percentage improvement over a vanilla Wide ResNet, and for DASHA the model size refers to the largest configuration across tasks.**
| Model | Model | Pretraining | Average | Average | Mean | Median |
| --- | --- | --- | --- | --- | --- | --- |
| Size | Images | Score $\uparrow$ | Rank $\downarrow$ | %Imp.$\uparrow$ | %Imp.$\uparrow$ |  |
| Foundation Models |  |  |  |  |  |  |
| SatMAE-Base | 85.6M | 700K | 76.99 | 6.22 | 5.27 | 3.59 |
| SatMAE-Large | 303M | 700K | 77.75 | 4.5 | 6.52 | 4.62 |
| GFM | 86.8M | 1.3M | 77.18 | 5.56 | 5.77 | 4.08 |
| SwinT-Base | 86.8M | 14M | 76.69 | 5.28 | 4.86 | 1.43 |
| CROMA-Base | 90.6M | 2M | 77.39 | 4.33 | 5.85 | 4.22 |
| CROMA-Large | 312M | 2M | 78.03 | 3.33 | 6.90 | 6.09 |
| Supervised Methods |  |  |  |  |  |  |
| ResNet50 | 23.5M | 0 | 73.76 | 8.34 | 0.30 | 00.07 |
| Wide ResNet | 17.2M | 0 | 73.97 | 8.22 | 0.00 | 0.00 |
| UNet | 17.3M | 0 | 75.73 | 5.89 | 3.01 | 1.07 |
| DASHA (our workflow) | 32.4M | 0 | 77.85 | 3.33 | 6.67 | 5.16 |

#### 4.2.1 Baselines

Since satellite imaging resembles RGB imaging, it is common to “lift-and-shift” vision models to this domain (Rolf et al., ).
As a result we use several CNN backbones as baselines and wide ResNet as the candidate architecture for our DASHA workflow.
Lastly, we also consider the performance of fine-tuning the ImageNet-pretrained vision FM SwinT-base (Liu et al., ).

#### 4.2.2 Results

Table 2 shows that our supervised workflow attains the best or second-best performance across all aggregate metrics and is only ever slightly outperformed by CROMA-large.
Notably, unlike in genomics, the FMs here consistently outperform CNN backbones, likely because the associated papers compare to them as baselines.
However, the frequently superior performance of DASHA suggests that domain-aware model development would yield good supervised models in this field.
Another contrast with genomics is that the larger versions of the FMs consistently attain superior performance here, suggesting they are making at least somewhat effective use of the pretraining data.
Nevertheless, that this improvement can also be attained by DASHA, which uses no pretraining and produces a model that is ten times smaller, suggests that there remains significant room for improvement.

### 4.3 Time series

**Table 3: Aggregate performance on time series tasks across seven tasks. The latter evaluation demonstrates that simply tuning a classical AR model is competitive with state-of-the-art FMs while using no pretraining data and tens of thousands of times fewer parameters. For Mean/Median %Imp. we report percentage improvement over Auto-ARIMA, and for Auto-AR, the model size refers to the largest configuration across tasks. “-” indicates unknown quantities.**
| Model | Model | Pretraining | Average | Average | Mean | Median |
| --- | --- | --- | --- | --- | --- | --- |
| Size | Series | RMSE $\downarrow$ | Rank $\downarrow$ | %Imp.$\uparrow$ | %Imp.$\uparrow$ |  |
| Foundation Models |  |  |  |  |  |  |
| GPT4TS (OFA) | 87M | 8M | 0.555 | 6.70 | 31.29 | 22.71 |
| TEST (Few Shot) | 345M | 8M | 0.603 | 10.70 | 25.23 | 14.59 |
| MOMENT | 385M | 13M | 0.550 | 5.14 | 31.95 | 23.14 |
| TTM (B) | 1M | 1M | 0.543 | 2.89 | 32.92 | 25.36 |
| TTM (A) | 5M | 1M | 0.538 | 2.21 | 33.38 | 24.96 |
| S^2IP-LLM | 345M | 8M | 0.545 | 3.54 | 32.59 | 24.63 |
| CALF | 86M | 8M | 0.568 | 8.95 | 29.89 | 21.60 |
| TEMPO (Zero Shot) | 345M | 8M | 0.598 | 10.54 | 26.69 | 22.30 |
| TimesFM (Zero Shot) | 200M | 5M | 0.574 | 7.38 | 28.50 | 19.56 |
| Supervised Methods |  |  |  |  |  |  |
| DLinear | 700K | 0 | 0.567 | 8.13 | 29.68 | 23.18 |
| Auto-ARIMA | 10 | 0 | 0.896 | 12.82 | 0.00 | 0.00 |
| AR | 513 | 0 | 0.556 | 6.57 | 31.17 | 24.31 |
| Auto-AR (our workflow) | 513 | 0 | 0.551 | 5.45 | 31.91 | 25.36 |

Our last domain is time series, which has many FMs, including those that use the standard pretrain-then-fine-tune workflow: GPT4TS (OFA) (Zhou et al., ), LLM4TS (Chang et al., ), MOMENT (Goswami et al., ), TEST (Sun et al., ), S^2IP-LLM (Pan et al., ), CALF (Liu et al., ), TTM (Ekambaram et al., ), and Time-LLM (Jin et al., ); and others that evaluate in a zero-shot (ZS) regime: TEMPO (Cao et al., ), TimesFM (Das et al., ), Moirai (Woo et al., ), and Toto (Cohen et al., ).
As we are comparing to supervised baselines, our evaluation of ZS models will be in a less challenging setting than the one they report numbers for.
We study the performance of these FMs and our baselines on the problem of long-horizon forecasting, which has a standard set of tasks (Goswami et al., , Table 11), of which we consider seven.(^6^66The two we do not consider, Exchange and ILI, are not evaluated on by most time series FMs.)
Note that each task consists of four settings corresponding to different time horizons, so in total this yields twenty-eight tasks.
Lastly, we compute aggregate metrics using RMSE, not MSE, so that performance scales linearly with prediction error;
this choice has no effect on average rank.

Note that we do not include four of the above time series FMs in our main analysis.
One of them, Time-LLM, reports strong results but has had difficulty being reproduced by both us and past efforts, as we detail in Appendix C.3.1.
The three others—Moirai, LLM4TS, and Toto—either evaluate on only a subset of the seven tasks or are closed-source (or both);
we list their reported numbers in Table 14.
As discussed in Appendix C.3.2,
Moirai underperforms Auto-AR and LLM4TS performs roughly on par with TTM (A).
The most recently released FM, Toto, does have strong aggregate metrics, but this is mainly due to a dominant performance on a sinle task, ETTh2 (c.f. Figure 4).
As a result we do not view these excluded FMs as significantly affecting our conclusions.

#### 4.3.1 Baselines

To baseline these FMs we use mainly linear forecasting methods, including the classical (untuned) linear auto-regression (AR), the automated workhorse Auto-ARIMA (Hyndman & Khandakar, ), the more recent DLinear (Zeng et al., ), and our own workflow Auto-AR described in Section 3.2.
Lastly, we also evaluate our other approach, DASHA, on six of the tasks (c.f. Table 14).

#### 4.3.2 Results

Table 3 shows that on the full seven-dataset evaluation our Auto-AR workflow always attains competitive performances across all aggregate metrics considered, and in particular attains the best median improvement over Auto-ARIMA.
Specifically, our Auto-AR workflow achieves competitive performances with two other recent time series FMs, namely MOMENT and S^2IP-LLM, and outperforms the rest of the FMs.
Although TTM surpasses all other methods across three aggregated metrics, the improvements remain relatively marginal. This observation aligns with our assertion that the substantial increase in pretraining dataset size and model scale has not yet resulted in significant advancements in model performance.
Notably, the three best performing methods are not zero-shot, which is perhaps not surprising given the extra data.
However, it does reinforce the intuition that settings with high data availability should prefer supervised methods, including simple ones like AR.
Notably, even our untuned implementation of AR that uses no differencing and a large lookback window is quite effective, doing better than ZS FMs across all aggregate metrics and even MOMENT on some of them.

## 5 Discussion

At a high level, our results show that the foundation models in these three domains have not yet surpassed supervised learning, and thus more broadly that the latter remains a strong baseline for specialized FMs.
This is a surprising and consequential finding due the paradigm’s popularity and the data and compute costs associated with large-scale pretraining.
In this section we discuss lessons and implications for the development of machine learning in these and other application areas.

### 5.1 The importance of diverse, well-tuned, and domain-specific baselines

The main lesson of our work is to select a diverse array of baselines, drawing from both “lift-and-shift” and domain-specific approaches, and then to carefully tune them.
For example, in genomics the vanilla wide ResNet baseline does remarkably well, with the majority of FMs doing worse than even this “lift-and-shift” baseline on the typical task in the NT benchmark.
While satellite FMs do outperform such baselines, lightly modifying these CNNs via different kernel sizes and dilation rates was enough to match state-of-the-art models there as well.
Lastly, our time series results demonstrate in dramatic fashion the need to carefully tune domain-specific approaches, as we show that simply allowing the classical AR forecaster to make use of long lookback windows and GPU-based optimization leads better forecasting than all open-source FMs.

### 5.2 Computational efficiency considerations

While not our main focus, we nevertheless highlight that any performance gains from FMs must be balanced against their additional cost.
In addition to the extensive GPU-hours used for pretraining, the resulting models are often much bigger and so lead to much more costly inference.
Indeed, apart from the special case of HyenaDNA, the CNN architectures discovered and trained using our DASHA workflow are typically over ten times smaller than FMs in the case of genomics and three to ten times smaller in the case of satellite imaging.
Moreover, for time series our Auto-AR approach is quick-to-train and yields simple models with less than 1K parameters—over two-thousand times smaller than any FM—while attaining performance that is often competitive even with closed-source models.
In aggregate, these examples further demonstrate the efficiency of supervised approaches and the resulting high performance bar that FMs need to clear before they can be deemed useful.

### 5.3 The power of tuning kernel sizes and dilation rates

Figure: Figure 3: PCA visualization of the architectures discovered for three different tasks when the DASHA workflow is run multiples times. The spatial clustering across tasks demonstrates the within-task consistency of the workflow’s architecture search component and the utility of using diverse models as baselines.
Refer to caption: x3.png

Our results for genomics and satellite imaging are driven by the DASHA workflow, whose crucial component is the tuning of kernel sizes and dilation rate in CNN backbones such as wide ResNet.
Its success demonstrates that the procedure is an effective surrogate for human-driven model development, enabling the automated discovery of the types of diverse, domain-specific baselines stressed in Section 5.1.
To understand this further, we study whether the architecture search component selects different kernel sizes and dilation rates for different tasks, and whether it does so in a consistent manner.
Specifically, we run DASHA on three of the smaller datasets in the NT benchmark with fifteen different random seeds, construct eighteen-dimensional vectors of the discovered kernel sizes and dilation rates assigned to each of the nine layers, and project these to two dimensions using principal component analysis (PCA).
The result in Figure 3 reveals that the architectures are clustered by task, demonstrating that the procedure selects different but consistent-within-task kernel parameters.
This visualization suggests that architecture search is a useful surrogate for model development, and consequently that the DASHA workflow may also be useful for automating similar studies and baselining FMs in other domains with high-dimensional, unstructured data.

### 5.4 The surprising effectiveness of linear auto-regression

Perhaps our most surprising finding is the competitiveness of linear auto-regression (AR), a very old method, on long-horizon forecasting.
It is likely that the lack of comparison with this baseline was driven by existing evaluations (e.g. by Challu et al. ()) of Auto-ARIMA (Hyndman & Khandakar, ), which is perceived to be a stronger baseline because it both combines AR with another model (MA) and tunes the lookback and differencing parameters.
However, in most Auto-ARIMA packages the default maximum lookback is around five, whereas we often found much (hundred-fold) larger settings to work best.
Since these implementations are also generally too slow to support such long lookbacks, the possibility of expanding the hyperparameter space was more likely to be ignored.
By implementing an efficient tuning procedure over a larger space of lookback parameters, our Auto-AR workflow comprises a significant contribution to forecasting baselines.

### 5.5 Limitations

While our findings are significant according to measures set by past work, they should not be misinterpreted to address all possible scenarios where FMs may be useful.
This is most salient for time series FMs motivated by zero-shot concerns, a setting we do not study, and to some extent for genomics FMs, which are often used for exploratory science and not supervised learning.
We are also of course computationally limited and there are many other domains where FMs have been pretrained, and even in our three there are other tasks beyond classification and forecasting.
Nevertheless, our evaluation is extensive—over twenty-five FMs and over fifty tasks—and so are at least strongly suggestive of the state of a field that uses benchmark performance to motivate and justify pretraining.

## 6 Conclusion

We conduct a thorough investigation to evaluate whether the cost of training specialized foundation models across three major domains are justified by their superior performance relative to traditional supervised learning.
Our results demonstrate that FMs in these domains have not yet surpassed supervised workflows and are often outperformed by fairly simple methods, including lightly modified CNN backbones (in genomics and satellite imaging) and classical linear forecasters (for time series).
As part of our study, we introduce two automated workflows—DASHA for simulating in-domain model development of CNNs and Auto-AR for tuning linear auto-regression on GPUs—that we believe will be useful tools for evaluating future work in these and other areas.
The code for these pipelines and to reproduce our results is publicly available.

## Acknowledgments

We thank Mononito Goswami, Esther Rolf, and Stephan Xie for useful feedback. This work was supported in part by the National Science Foundation grants IIS1705121, IIS1838017, IIS2046613, IIS2112471, a TCS Presidential Fellowship, and funding from Meta, Morgan Stanley, Amazon, Google, and Scribe. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of any of these funding agencies.

## Appendix A Tasks

### A.1 Genomics

For the Genomics domain, we use the eighteen classification tasks from the Nucleotide Transformer benchmark (Dalla-Torre et al., ) that has widely been used for other genomics FMs. The benchmark datasets consist of nucleotide base sequences ranging from 200 to 600 bases in length. It provides a realistic and biological meaningful benchmark across four main categories: promoter (human/mouse), enhancer (human), splice site (SS; human/multispecies) and histone modification (yeast). Within the benchmark, the enhancers_types and splice_sites_all tasks are classification tasks with three classes each, while the remaining tasks are binary classification tasks.

**Table 4: Statistics for Genomics datasets**
| Dataset | # of classes | # of samples | Maximum | Metric |  |
| --- | --- | --- | --- | --- | --- |
| train | test | sequence length |  |  |  |
| enhancers | 2 | 14968 | 400 | 200 | MCC |
| enhancers_types | 3 | 14968 | 400 | 200 | MCC |
| promoter_all | 2 | 53276 | 5920 | 300 | F1 |
| promoter_no_tata | 2 | 47767 | 5299 | 300 | F1 |
| promoter_tata | 2 | 5509 | 621 | 300 | F1 |
| splice_sites_acceptors | 2 | 19961 | 2218 | 600 | F1 |
| splice_sites_all | 3 | 27000 | 3000 | 400 | Accuracy |
| splice_sites_donors | 2 | 19775 | 2198 | 600 | F1 |
| H3 | 2 | 13468 | 1497 | 500 | MCC |
| H3K14ac | 2 | 29743 | 3305 | 500 | MCC |
| H3K36me3 | 2 | 31392 | 3488 | 500 | MCC |
| H3K4me1 | 2 | 28509 | 3168 | 500 | MCC |
| H3K4me2 | 2 | 27614 | 3069 | 500 | MCC |
| H3K4me3 | 2 | 25953 | 2884 | 500 | MCC |
| H3K79me3 | 2 | 25953 | 2884 | 500 | MCC |
| H3K9ac | 2 | 25003 | 2779 | 500 | MCC |
| H4 | 2 | 13140 | 1461 | 500 | MCC |
| H4ac | 2 | 30685 | 3410 | 500 | MCC |

### A.2 Satellite imaging

In the satellite imaging domain, we aim to conduct evaluations with real-world relevance to Earth science. To achieve this, we include a variety of data from different sources to cover a diverse range of tasks, such as brick kiln identification, deforestation prediction, and photovoltaic monitoring. We utilize five classification tasks provided by the GeoBench dataset (Lacoste et al., ), a recently developed benchmark that offers a clean and carefully curated collection of tasks specifically designed for satellite imaging. In addition to GeoBench, we evaluate our model on three additional datasets (Helber et al., ; Jacques et al., ; Sumbul et al., ) commonly used in the literature as benchmarks for this domain . This brings the total to eight datasets, encompassing a wide range of features. These tasks vary in complexity, with single-class classification ranging from binary to 62-class problems, as well as two multilabel classification tasks. The datasets are further characterized by diverse input channels, ranging from 3 RGB channels to 18 channels that integrate data from both Sentinel-1 and Sentinel-2 formats.

For Geo-Bench datasets, we do not use any mixup and cutmix augmentations. For other datasets, we universally use mixup = 0.8, cutmix = 1.0, and a switch probability of 0.5. Following Fuller et al. (), we use only $10\%$ of training set from BigEarthNet and fMoW-Sentinel while using the full evaluation set for validation.

**Table 5: Statistics for Satellite datasets**
| Dataset | Image | # of classes | # of samples | # of |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Size | train | val | test | channels |  |  |
| m-bigearthnet | 120 $\times$ 120 | 43 | 20000 | 1000 | 1000 | 12 |
| m-brickkiln | 64 $\times$ 64 | 2 | 15063 | 999 | 999 | 13 |
| m-so2sat | 32 $\times$ 32 | 17 | 19992 | 986 | 986 | 18 |
| m-forestnet | 332 $\times$ 332 | 12 | 6464 | 989 | 993 | 6 |
| m-pv4ger | 320 $\times$ 320 | 2 | 11814 | 999 | 999 | 3 |
| BigEarthNet | 120 $\times$ 120 | 19 | 31166 | 103944 | 103728 | 12 |
| EuroSAT | 64 $\times$ 64 | 13 | 16200 | 10800 | 5400 | 13 |
| Canadian Cropland | 120 $\times$ 120 | 10 | 53884 | 11414 | 11674 | 12 |
| fMoW-Sentinel | 96 $\times$ 96 | 62 | 71287 | 84939 | 84966 | 13 |

### A.3 Time series

In the time series domain, we focus on the long horizon forecasting task. We use a subset of the common benchmark datasets for evaluating models across different domains (ETT, Electricity, Weather, Illness, Traffic, Exchange Rate) (Wang et al., ), specifically, the ETT, Weather, Electricity, Illness (ILI), and Traffic datasets. Note that the ETT dataset is actually a collection of four series: ETTh1, ETTh2, ETTm1, and ETTm2; we follow the rest of the literature in treating each series as a separate dataset. Each dataset contains measurements of one or more channels at evenly spaced time steps.

**Table 6: Statistics for Time Series datasets**
| Dataset | # of | # of samples |  |  |
| --- | --- | --- | --- | --- |
| channels | train | val | test |  |
| ETTh1 | 7 | 8033 | 2785 | 2785 |
| ETTh2 | 7 | 8033 | 2785 | 2785 |
| ETTm1 | 7 | 33953 | 11425 | 11425 |
| ETTm2 | 7 | 33953 | 11425 | 11425 |
| Weather | 21 | 36280 | 5175 | 10444 |
| Electricity | 321 | 17805 | 2537 | 5165 |
| ILI | 7 | 69 | 2 | 98 |
| Traffic | 862 | 11673 | 1661 | 3413 |

## Appendix B Implementation details

### B.1 DASHA

Following the architecture search, we perform hyperparameter tuning using ASHA. The hyperparameter search space includes learning rate, weight decay, momentum, drop rate, and random seed for model initialization. We define a continuous search space, with further specific details provided in Table 7. Using ASHA, we evaluate 200 sample configurations over a maximum of 20 epochs, using a reduction factor of 2. The low-performing configurations are pruned based on their validation scores.

Before retraining the final model, we load the model checkpoint corresponding to the optimal hyperparameter configuration. The model is then trained for 200 epochs on the training data, with the best-performing checkpoint selected based on validation performance. This process is repeated for each backbone architecture, and the best-performing backbone is selected using the validation score. Finally, the checkpoint for the selected backbone is evaluated on the test set to obtain the final score.

**Table 7: Hyperparameter Search Space**
| Hyperparameter | Search Space | Type of Search Space |
| --- | --- | --- |
| random_seed | $[0,500]$ | Integer |
| lr | $[10^{-5},5\times 10^{-1}]$ | Log Uniform |
| drop_rate | $\{0,0.05,0.1\}$ | Discrete |
| weight_decay | $[5\times 10^{-7},5\times 10^{-3}]$ | Log Uniform |
| momentum | $[0.9,1]$ | Uniform |

### B.2 Auto-AR

A fairly complete description is provided in Section 3.2.
Here we note only that, because we minimize the total maximum likelihood across (independent) channels, to determine the amount of differencing used for each task we run the KPSS test separately on each channel and use the differencing needed by the majority of the channels.
Notably, this results in a differencing of one for each task.

## Appendix C Detailed results

### C.1 Genomics

We include all FMs listed in Dalla-Torre et al. (, Supplementary Table 6) with addition of the two recently released models, Caduceus (Schiff et al., ) and Gena-LM (Fishman et al., );
note that the last row of tasks in the NT paper(promoter and splice sites) are mislabeled, but we infer an order in combination with previous information obtained from a (now-deleted) Huggingface leaderboard.(^7^77[https://huggingface.co/spaces/InstaDeepAI/nucleotide_transformer_benchmark](https://huggingface.co/spaces/InstaDeepAI/nucleotide_transformer_benchmark))
In alignment with the leaderboard, we apply a 0.1 validation split for DASHA during our evaluation. Additionally, we use an architecture set that includes both Wide ResNet and UNet for the search with DASHA on these datasets. We use batch size$=$ 128 for all datasets, and cross entropy loss for all the training and finetuning. Individual scores for each task in the benchmark are provided in Tables 9 and 8.

**Table 8: Regulatory Elements and RNA Production Downstream Tasks**
| Model | Regulatory Elements | RNA Production |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| enhancers | enhancers | promoter | promoter | promoter | splice_sites | splice_sites | splice_sites |  |
| types | all | no_tata | tata | acceptors | all | donors |  |  |
| Enformer | 0.454 | 0.312 | 0.955 | 0.955 | 0.959 | 0.915 | 0.847 | 0.906 |
| NT-1000G (500M) | 0.509 | 0.395 | 0.951 | 0.951 | 0.936 | 0.965 | 0.968 | 0.971 |
| NT-1000G (2.5B) | 0.546 | 0.432 | 0.965 | 0.967 | 0.957 | 0.98 | 0.976 | 0.979 |
| NT-Multispecies (500M) | 0.559 | 0.438 | 0.976 | 0.976 | 0.965 | 0.981 | 0.984 | 0.987 |
| NT-Multispecies (2.5B) | 0.545 | 0.444 | 0.975 | 0.977 | 0.959 | 0.986 | 0.982 | 0.987 |
| DNABERT-1 | 0.495 | 0.367 | 0.961 | 0.962 | 0.956 | – | 0.975 | – |
| DNABERT-2 | 0.525 | 0.423 | 0.972 | 0.972 | 0.955 | 0.975 | 0.939 | 0.963 |
| HyenaDNA-1K | 0.52 | 0.403 | 0.959 | 0.959 | 0.944 | 0.959 | 0.956 | 0.947 |
| HyenaDNA-32K | 0.489 | 0.352 | 0.956 | 0.954 | 0.939 | 0.96 | 0.962 | 0.957 |
| Caduceus-PS | 0.491 | 0.416 | 0.967 | 0.968 | 0.957 | 0.936 | 0.927 | 0.874 |
| Caduceus-PH | 0.546 | 0.439 | 0.97 | 0.969 | 0.953 | 0.937 | 0.94 | 0.948 |
| Wide ResNet | 0.525 | 0.416 | 0.952 | 0.946 | 0.93 | 0.821 | 0.457 | 0.815 |
| UNet | 0.49 | 0.366 | 0.956 | 0.954 | 0.95 | 0.956 | 0.955 | 0.968 |
| DASHA | 0.527 | 0.432 | 0.958 | 0.962 | 0.957 | 0.978 | 0.979 | 0.978 |

**Table 9: Histone Modification Downstream Tasks**
| Model | H3 | H3K14ac | H3K36me3 | H3K4me1 | H3K4me2 | H3K4me3 | H3K79me3 | H3K9ac | H4 | H4ac |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Enformer | 0.724 | 0.284 | 0.345 | 0.291 | 0.207 | 0.156 | 0.498 | 0.415 | 0.735 | 0.275 |
| NT-1000G (500M) | 0.736 | 0.381 | 0.468 | 0.38 | 0.26 | 0.235 | 0.562 | 0.479 | 0.755 | 0.342 |
| NT-1000G (2.5B) | 0.754 | 0.453 | 0.53 | 0.418 | 0.278 | 0.311 | 0.574 | 0.491 | 0.787 | 0.408 |
| NT-Multispecies (500M) | 0.786 | 0.549 | 0.624 | 0.55 | 0.32 | 0.406 | 0.63 | 0.567 | 0.799 | 0.496 |
| NT-Multispecies (2.5B) | 0.793 | 0.538 | 0.618 | 0.541 | 0.324 | 0.408 | 0.623 | 0.547 | 0.808 | 0.492 |
| DNABERT-1 | 0.763 | 0.403 | 0.474 | 0.396 | 0.282 | 0.258 | 0.578 | 0.505 | 0.784 | 0.359 |
| DNABERT-2 | 0.785 | 0.515 | 0.591 | 0.512 | 0.333 | 0.353 | 0.615 | 0.545 | 0.797 | 0.465 |
| HyenaDNA-1K | 0.781 | 0.608 | 0.614 | 0.512 | 0.455 | 0.55 | 0.669 | 0.586 | 0.763 | 0.564 |
| HyenaDNA-32K | 0.747 | 0.405 | 0.479 | 0.387 | 0.276 | 0.291 | 0.567 | 0.472 | 0.761 | 0.379 |
| Caduceus-PS | 0.799 | 0.541 | 0.609 | 0.488 | 0.388 | 0.44 | 0.679 | 0.604 | 0.789 | 0.525 |
| Caduceus-PH | 0.815 | 0.631 | 0.601 | 0.523 | 0.487 | 0.544 | 0.697 | 0.622 | 0.811 | 0.621 |
| Wide ResNet | 0.798 | 0.667 | 0.670 | 0.554 | 0.541 | 0.660 | 0.706 | 0.620 | 0.754 | 0.657 |
| UNet | 0.797 | 0.647 | 0.482 | 0.541 | 0.553 | 0.292 | 0.562 | 0.624 | 0.760 | 0.389 |
| DASHA | 0.790 | 0.683 | 0.630 | 0.528 | 0.640 | 0.714 | 0.721 | 0.709 | 0.776 | 0.744 |

The performance of the models on individual tasks is detailed in Tables 8 and 9. In the regulatory elements domain (c.f. Table 8), DASHA performs slightly behind the largest models like NT-Multispecies (Dalla-Torre et al., ) on the enhancers tasks but consistently outperforms models such as DNABERT-1 (Ji et al., ), Enformer (Avsec et al., ), and HyenaDNA (Nguyen et al., );
in the promoters task in generally performs worse than all reported FMs.
In the RNA production domain, DASHA performs near the middle of the FMs.
However, where DASHA truly excels is in the histone modification tasks (in Table 9), where it not only competes with, but often outperforms, the other FMs, consistently achieving top scores in nearly all tasks.

Note that because GENA-LM reports using MCC on all datasets, which is different than the metrics used by NT and Caduceus paper, we compare its results with DASHA in a separate table, where all datasets are evaluated through MCC. As shown in Table 10 and Table 11, DASHA also outperform GENA-LM by a large margin in terms of average MCC.

**Table 10: Histone Modification for GENA-LM (all scores are obtained using MCC)**
| Model | H3 | H3K14ac | H3K36me3 | H3K4me1 | H3K4me2 | H3K4me3 | H3K79me3 | H3K9ac | H4 | H4ac |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GENA-LM | 0.79 | 0.6 | 0.61 | 0.53 | 0.46 | 0.55 | 0.67 | 0.61 | 0.78 | 0.59 |
| DASHA | 0.79 | 0.68 | 0.63 | 0.53 | 0.64 | 0.71 | 0.72 | 0.71 | 0.78 | 0.74 |

**Table 11: Regulatory Elements and RNA Production for GENA-LM. (all scores are obtained using MCC) The last column shows the average MCC across all 18 datasets.**
| Model | enhancers | enhancers | promoter | promoter | promoter | splice_sites | splice_sites | splice_sites | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| types | all | no_tata | tata | acceptors | all | donors |  |  |  |
| GENA-LM | 0.55 | 0.45 | 0.94 | 0.94 | 0.91 | 0.92 | 0.91 | 0.91 | 0.707 |
| DASHA | 0.53 | 0.43 | 0.92 | 0.92 | 0.90 | 0.97 | 0.96 | 0.96 | 0.755 |

### C.2 Satellite imaging

Training on satellite datasets requires relatively large computational resources due to the high number of channels and the size of the datasets. To ensure a fair comparison, we fine-tuned all the foundation models ourselves by sweeping across a fixed set of base learning rates $[5e-3,2e-3,2e-4,4e-5]$. We then calculate the actual learning rate from base learning rate following previous work by $lr=base_lr\cdot\frac{batch_size}{256}$. This approach ensures that approximately the same amount of resources were used as during the DASHA tuning process, allowing for a balanced evaluation of model performance.

We closely followed the reported evaluation processes from previous studies on FMs (Cong et al., ; Fuller et al., ; Mendieta et al., ). These models do not employ a validation set for hyperparameter tuning or model selection, and we adhered to this same approach when fine-tuning the FMs. However, for DASHA, since we performed extensive hyperparameter optimization over a large search space, we used a validation set to ensure fair and accurate comparisons between DASHA and the FMs. This is a less favorable setting for DASHA, as it relies on extensive hyperparameter tuning, but we demonstrate that, even under these conditions, DASHA matches the performance of the FMs.

It is also important to note that SatMAE only accepts 3-channel and 12-channel inputs, while CROMA is limited to 12-channel inputs. GeoBench, however, includes a wide range of tasks with varying numbers of input channels, ranging from 3 to 18. Despite these differences, we include all datasets in our evaluation because they are valuable benchmarks in the satellite image domain, and it is crucial for FMs in this field to generalize across diverse datasets. For datasets where the input size does not match the model requirements, we pad missing channels with zeros and prune any extra channels. However, to ensure a fair comparison, in addition to reporting the average scores across all datasets, we also provide average scores excluding m-pv4ger and m-forestnet, where missing channels may affect the performance of the FMs. The aggregate scores excluding m-pv4ger and m-forestnet are presented in Figure 12.

**Table 12: Aggregated metrics on Satellite imaging tasks excluding the m-pv4ger and m-forestnet.**
| Model | Average | Average | Mean | Median |
| --- | --- | --- | --- | --- |
| Score $\uparrow$ | Rank $\downarrow$ | %Imp.$\uparrow$ | %Imp.$\uparrow$ |  |
| Foundation Model |  |  |  |  |
| SatMAE-Base | 77.71 | 6.00 | 6.74 | 4.56 |
| SatMAE-Large | 78.44 | 4.07 | 7.87 | 6.75 |
| GFM | 76.95 | 5.57 | 5.40 | 4.08 |
| SwinT-Base | 76.12 | 6.50 | 3.98 | 1.43 |
| CROMA-Base | 77.98 | 3.57 | 6.95 | 5.30 |
| CROMA-Large | 79.06 | 2.14 | 8.84 | 6.26 |
| Supervised Methods |  |  |  |  |
| ResNet50 | 73.25 | 9.00 | -0.27 | -0.38 |
| Wide ResNet | 73.90 | 8.00 | 0.00 | 0.00 |
| UNet | 75.29 | 6.57 | 2.33 | 0.87 |
| DASHA | 78.00 | 3.57 | 7.02 | 5.16 |

For training and finetuning, we universally use batch_size = 16 and loss function as cross entropy with 0.1 label smoothing for single label classification, and multi-label soft margin loss for multilabel classification. Individual scores for each task are provided in Table 13.

**Table 13: Satellite Imaging Tasks**
| Model | Average | m-bigearthnet | m-brickkiln | m-so2sat | m-forestnet | m-pv4ger | BigEarth | EuroSAT | Canadian | fMoW |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Net | Cropland | Sentinel |  |  |  |  |  |  |  |  |
| SatMAE-Base | 76.99 | 72.3 | 98.22 | 54.56 | 51.89 | 97.0 | 86.04 | 98.69 | 74.64 | 59.55 |
| SatMAE-Large | 77.75 | 73.82 | 98.6 | 55.79 | 53.7 | 96.92 | 86.75 | 98.86 | 75.38 | 59.89 |
| GFM | 77.18 | 71.97 | 98.35 | 57.52 | 59.38 | 96.54 | 85.93 | 99.02 | 72.03 | 53.84 |
| SwinT-Base | 76.69 | 70.14 | 98.81 | 56.49 | 59.78 | 97.54 | 85.91 | 98.99 | 70.15 | 52.37 |
| CROMA-Base | 77.39 | 72.07 | 98.99 | 60.04 | 54.07 | 96.6 | 86.94 | 98.81 | 75.87 | 53.14 |
| CROMA-Large | 78.03 | 73.36 | 99.01 | 59.22 | 51.96 | 96.9 | 87.98 | 98.98 | 76.56 | 58.32 |
| ResNet50 | 73.76 | 60.31 | 98.4 | 51.83 | 54.29 | 96.79 | 79.16 | 98.23 | 72 | 52.84 |
| Wide ResNet | 73.97 | 69.15 | 98.95 | 49.04 | 52.14 | 96.34 | 80.48 | 98.6 | 72.05 | 49.00 |
| UNet | 75.73 | 69.89 | 98.6 | 56.87 | 57.18 | 97.39 | 83.9 | 98.99 | 72.68 | 46.11 |
| DASHA | 77.85 | 72.72 | 98.92 | 56.28 | 57.24 | 97.4 | 86.09 | 99.07 | 75.69 | 57.20 |

### C.3 Time series

The long horizon forecasting task for a time series can be summarized as follows: at every timestep $t$, take the $L$ historical observations at times $(t-L+1,...,t)$ for each channel and predict the next $H$ observations $(t+1,...,t+H)$ for every channel. Following the literature, we evaluate models on $H\in\{24,36,48,60\}$ for the Illness dataset and $H\in\{96,192,336,720\}$. For most methods, we report results when $L=512$.

**Table 14: Time Series Forecasting Tasks**
| Model | ETTh1 | ETTh2 | ETTm1 | ETTm2 |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Horizon | 96 | 192 | 336 | 720 | 96 | 192 | 336 | 720 | 96 | 192 | 336 | 720 | 96 | 192 | 336 | 720 |
| GPT4TS (OFA) | 0.376 | 0.416 | 0.442 | 0.477 | 0.285 | 0.354 | 0.373 | 0.406 | 0.292 | 0.332 | 0.366 | 0.417 | 0.173 | 0.229 | 0.286 | 0.378 |
| TEST (Few shot) | 0.455 | 0.572 | 0.611 | 0.723 | 0.332 | 0.401 | 0.408 | 0.459 | 0.392 | 0.423 | 0.471 | 0.552 | 0.233 | 0.303 | 0.359 | 0.452 |
| LLM4TS | 0.371 | 0.403 | 0.42 | 0.422 | 0.269 | 0.328 | 0.353 | 0.383 | 0.285 | 0.324 | 0.353 | 0.408 | 0.165 | 0.22 | 0.268 | 0.35 |
| MOMENT | 0.387 | 0.41 | 0.422 | 0.454 | 0.288 | 0.349 | 0.369 | 0.403 | 0.293 | 0.326 | 0.352 | 0.405 | 0.17 | 0.227 | 0.275 | 0.363 |
| TTM (B) | 0.36 | 0.392 | 0.401 | 0.436 | 0.269 | 0.336 | 0.359 | 0.39 | 0.291 | 0.325 | 0.363 | 0.419 | 0.164 | 0.219 | 0.277 | 0.35 |
| TTM (A) | 0.363 | 0.392 | 0.413 | 0.442 | 0.262 | 0.324 | 0.351 | 0.392 | 0.283 | 0.332 | 0.353 | 0.393 | 0.158 | 0.213 | 0.269 | 0.369 |
| $S^{2}$IP-LLM | 0.366 | 0.401 | 0.412 | 0.44 | 0.278 | 0.346 | 0.367 | 0.4 | 0.288 | 0.323 | 0.359 | 0.403 | 0.165 | 0.222 | 0.277 | 0.363 |
| CALF | 0.369 | 0.427 | 0.456 | 0.479 | 0.279 | 0.353 | 0.362 | 0.404 | 0.323 | 0.374 | 0.409 | 0.477 | 0.178 | 0.242 | 0.307 | 0.397 |
| TEMPO (Zero Shot) | 0.4 | 0.426 | 0.441 | 0.443 | 0.301 | 0.355 | 0.379 | 0.409 | 0.438 | 0.461 | 0.515 | 0.591 | 0.185 | 0.243 | 0.309 | 0.386 |
| TimesFM (Zero Shot) | 0.421 | 0.472 | 0.51 | 0.514 | 0.326 | 0.399 | 0.434 | 0.451 | 0.357 | 0.411 | 0.441 | 0.507 | 0.205 | 0.294 | 0.367 | 0.473 |
| Moirai (Zero Shot) | 0.375 | 0.399 | 0.412 | 0.413 | 0.281 | 0.34 | 0.362 | 0.38 | 0.404 | 0.435 | 0.462 | 0.49 | 0.205 | 0.261 | 0.319 | 0.415 |
| Toto (Zero Shot) | 0.307 | 0.329 | 0.396 | 0.419 | 0.093 | 0.135 | 0.16 | 0.294 | 0.306 | 0.328 | 0.39 | 0.463 | 0.2 | 0.269 | 0.264 | 0.354 |
| ARIMA | 0.646 | 0.704 | 0.732 | 0.738 | 0.324 | 0.411 | 0.456 | 0.462 | 1.131 | 1.172 | 1.197 | 1.231 | 0.225 | 0.298 | 0.37 | 0.478 |
| AR (d=0) | 0.358 | 0.39 | 0.41 | 0.424 | 0.271 | 0.334 | 0.361 | 0.395 | 0.299 | 0.336 | 0.368 | 0.426 | 0.163 | 0.218 | 0.271 | 0.366 |
| DLinear | 0.375 | 0.405 | 0.439 | 0.472 | 0.289 | 0.383 | 0.448 | 0.605 | 0.299 | 0.335 | 0.369 | 0.425 | 0.167 | 0.224 | 0.281 | 0.397 |
| Auto-AR | 0.357 | 0.39 | 0.41 | 0.422 | 0.269 | 0.332 | 0.359 | 0.394 | 0.299 | 0.336 | 0.368 | 0.426 | 0.163 | 0.218 | 0.271 | 0.367 |
| DASHA | 0.369 | 0.401 | 0.430 | 0.478 | 0.284 | 0.377 | 0.396 | 0.745 | 0.305 | 0.335 | 0.367 | 0.418 | 0.169 | 0.224 | 0.290 | 0.378 |

In addition to the results for DASHA, Auto-AR, DLinear, and FMs, we evaluate the performance of two simple baselines

- 1.
Vanilla Autoregressive Model (Box & Jenkins, ): This model predicts the (scalar) value of a time series at $t+1$ as a linear combination of the last $L$ timesteps and a constant, i.e. $\hat{x}_{t+1}=\alpha_{0}+\alpha_{1}x_{t}+\alpha_{2}x_{t-1}+...+\alpha_{L}x_{t-
L+1}$ for learnable parameters $\alpha_{0},...,\alpha_{L}$. We fit these parameters using standard maximum likelihood techniques.
- 2.
ARIMA is a statistical method used for time series forecasting that combines three components: AutoRegressive (AR), Integrated (I), and Moving Average (MA). The AR component models the relationship between an observation and its lagged (past) values, assuming that past values have a linear influence on future ones. The Integrated component applies differencing to the data to remove trends or seasonality, making the time series stationary by stabilizing its mean over time. The MA component models the relationship between an observation and the residual errors from a moving average model applied to previous observations. ARIMA is characterized by three parameters: p (the number of lag observations), d (the number of differencing steps to achieve stationarity), and q (the number of lagged forecast errors). This model is particularly effective for univariate time series forecasting where patterns like trends or seasonality are present.

All results are reported on a 70/10/20 train/validation/test split for each datasets, except for the ETT datasets which have predefined splits. MSE is reported after all datasets have been scaled by the mean and variance of the training data. Both autoregressive models have only one tunable hyperparameter (number of lags). Similarly, the linear model has only one tunable hyperparameter (number of training epochs).

The baselines as described can handle only univariate time series, while all of the benchmark datasets are multivariate (multiple channels). These baselines are trained under channel independence: each channel of a time series is treated independently. While channel independence fails to take into account cross-channel dependencies, we note that developing methods that leverage cross-channel dependencies for a variable number of channels remains an open problem.

#### C.3.1 Time-LLM

Due to differences in the results reported for Time-LLM between the original paper and reproductions in several works (Goswami et al., ; Ekambaram et al., ; Pan et al., ; Tan et al., ), we to attempt our own partial reproduction in order to determine whether to include their numbers.
To do so, we used code provided by the authors and looked at ETTh1 with input lengths from ${96,192}$ and ILI with input lengths from ${24,36,48,90}$ (going beyond this was infeasible due to the resources required to finetune LLaMA-7B).

Our reproduced MSE for ETTh1 with an input length of 96 was 0.405 as compared to that of 0.362 reported by Jin et al. (), while for input length of 192 it was 0.431 vs. 0.398.
On ILI the average increase in MSE across time horizons from their reported results to our reproduction was more than 0.4.
Due to these large discrepancies we chose to exclude their model from our analysis.

#### C.3.2 Closed-source models

Figure: Figure 4: Scatterplot of improvement in RMSE (averaged across time horizons) of two closed-source FMs, LLM4TS and Toto, on a subset of datasets in Table 14. It shows that both FMs usually perform similarly to the Auto-AR baseline, with Toto attaining strong aggregate metrics due to a dominant performance on ETTh2.
Refer to caption: x4.png

In Table 14 we consider three additional FMs that either do not report complete set of results (Moirai) on all 7 datasets or are closed-source (Toto and LLM4TS).
While the performance of Auto-AR is comparably to that of LLM4TS, Toto dominates the three aggregated metrics we considered in Table 15;
impressively, it does this despite being zero-shot.
However, a look at Figure 4 reveals that Toto is not superior across all six tasks, with the aggregated metrics being strongly boosted by its dramatically better performance on one of them (ETTh2).
Thus, while its ZS performance is quite good, it is unclear whether this result would be maintained with additional tasks.

**Table 15: Additional time series results on the 6 datasets setting**
| Model |  |  | Partial setting (6 datasets / 24 tasks) |  |  |
| --- | --- | --- | --- | --- | --- |
| Model | Pretraining | Avg. | Avg. | Median |  |
| Size | Series | RMSE $\downarrow$ | %Imp.$\uparrow$ | %Imp.$\uparrow$ |  |
| Moirai (Zero Shot) | 14M | 6M | 0.566 | 24.53 | 18.52 |
| LLM4TS | 60M | 8M | 0.526 | 29.25 | 20.96 |
| TTM (A) | 5M | 1M | 0.525 | 29.38 | 19.87 |
| Toto (Zero Shot) | 103M | - | 0.505 | 32.40 | 31.35 |
| Auto-AR | 513 | 0 | 0.534 | 28.12 | 19.63 |
| DASHA | 480K | 0 | 0.549 | 25.94 | 16.78 |

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: My primary concerns regarding the acceptance of this work are based on points 1 and 2 described in the “weakness” section. These concerns stem from my own experience, particularly regarding point 1, where foundation models (FMs) generally demonstrate greater robustness in small-scale data adaptation through fine-tuning in vision and language tasks. However, this behavior may differ in the other modalities used in this paper.

The claim that “FMs in these domains have not yet surpassed supervised workflows and are often outperformed by fairly simple methods, including lightly modified CNN backbones (in genomics and satellite imaging) and classical linear forecasters (for time series)” may be somewhat overstated. This is especially relevant when considering factors such as data bias, the challenges of hyperparameter optimization for FMs, and potential out-of-distribution issues that could lead to negative transfer due to pretraining data limitations.
soundness: 3
strengths: 1. This paper is well-organized and well-written. The research questions are both interesting and practical, particularly in highlighting that some foundation models (FMs) are not fully comparable to in-domain, supervised baseline models. The experimental setup and results are comprehensive, effectively supporting the study’s objectives and providing valuable insights into the performance of FMs relative to tailored supervised models.

2. The two proposed automated supervised pipelines, DASHA and Auto-AR, are designed with detailed insights into model architecture, resulting in lightweight, simple yet powerful baselines. In particular, the development of Auto-AR highlights previously overlooked limitations and gaps in earlier works, encouraging the research community to rethink classical methods.

3. The paper demonstrates high quality through its methodological rigor and detailed experiments. (1) Thorough Benchmarking: The authors evaluate multiple foundation models (FMs) across three distinct scientific fields, using datasets and tasks that reflect real-world applications. This comprehensive experimental setup ensures that the results are robust and meaningful. (2) Well-Constructed Baselines: The paper establishes carefully optimized baselines using existing supervised models. This level of detail ensures fair and well-founded comparisons, providing a reliable foundation for drawing conclusions.
summary: This paper aims to challenge the assumption that foundation models (FMs) consistently outperform traditional supervised learning methods. To support this, the authors design an automated machine learning framework, utilizing neural architecture search for genomics and satellite imaging tasks, and an Auto-ARIMA approach for time series analysis. These automated workflows produce competitive supervised models, providing strong baselines against which FMs are compared.

The study highlights that, unlike in general-purpose AI tasks (e.g., NLP or computer vision), the advantages of FMs in specialized fields remain unproven (FMs frequently fail to surpass traditional supervised learning models that are optimized for each task), emphasizing the need for diverse, well-tuned baselines in FM evaluation.
weaknesses: 1. The strong performance of supervised methods through the automated pipeline may be largely attributed to the large-scale training data. As noted in the data statistics in the appendix, nearly all datasets contain over 10,000 samples for training, which is sufficient for effective supervised training from scratch, even without pretrained checkpoints. For a comprehensive evaluation and to support the claim that pure supervised learning can outperform pretraining, additional experiments varying data scale (e.g., using different ratios of training data) are needed. This would likely highlight the advantages of large-scale pretraining.

2. In addition to performance metrics like accuracy and model parameters, other essential factors are not addressed in the paper. Specifically, the authors do not mention the time and computing resources required to obtain the optimal architecture and hyperparameters. Compared to directly fine-tuning foundation models (FMs), it is unclear whether the automated pipeline is more time- and compute-intensive. Including these comparisons would provide a fuller understanding of the trade-offs involved in using the automated pipeline.

3. The assumption that “their creators make a best-effort attempt to present their own method in the best light” may not be fully reasonable. Due to the high computational costs and complexity of foundation models, obtaining optimal or even sub-optimal hyperparameters is generally challenging.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Q1. Could the DASHA and Auto-AR workflows be extended or adapted for multi-modal datasets or tasks involving mixed data types? If so, what modifications might be necessary?
Q2. How do the authors envision the role of foundation models evolving in these domains if computational resources continue to scale? Would certain configurations of FMs potentially start to outperform supervised baselines, or do they anticipate the observed trends to persist?
Q3. Given the success of NAS in selecting optimal CNN architectures for genomics and satellite imaging, do the authors plan to explore whether reinforcement learning or meta-learning approaches might yield further performance gains?
soundness: 3
strengths: 1. The benchmarks and datasets chosen for each domain appear to comprehensively cover relevant applications in genomics, satellite imaging, and time series.

2. The paper highlights computational efficiency as a priority by contrasting the costs associated with FMs versus the supervised baselines. Holds high relevance given the fast adoption of large, resource-intensive models in research and industry.

3. Contextualized the results within each domain, providing a nuanced understanding of the model performance across tasks.
summary: The paper examines the effectiveness of foundation models (FMs) in specialized domains like genomics, satellite imaging, and time series data, relative to traditional supervised learning models. The authors aim to challenge the assumption that foundation models necessarily outperform supervised models in these domains. Using a rigorous experimental framework, they contrast FMs with supervised baselines, emphasizing cases where the benefits of large-scale pretraining do not translate into superior downstream performance. Through their novel automated pipelines—DASHA for genomics and satellite imaging, and Auto-AR for time series—the authors provide a streamlined, data-efficient, and computationally lighter alternative to FMs, that demonstrates competitive performance across several tasks. The study raises important considerations about the necessity and cost-efficiency of pretraining-heavy foundation models in specialized fields.
weaknesses: 1. The paper lacks an exploration of their applicability to other types of data or more complex, multi-modal tasks. A broader discussion of potential limitations could help future researchers understand the contexts in which DASHA and Auto-AR might or might not succeed.

2. The choice of baselines (e.g., Wide ResNet and UNet for genomics and satellite imaging) is reasonable but could be expanded. Although the selected baselines demonstrate competitive performance, further comparisons with more recent architectures could reinforce the study’s claims regarding the relative inefficacy of FMs in these domains.

3. Although the paper effectively highlights the efficiency of supervised workflows, the focus on computational cost may not fully account for the trade-offs in scalability and generalization that FMs can offer.

4. For time series, the paper does not sufficiently address the unique challenges of zero-shot forecasting. Including more details on why certain zero-shot FMs performed poorly could provide actionable insights for model improvement in zero-shot contexts.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See the weakness section for possible improvements. I am willing to listen to the author's feedback and raise my score during the rebuttal period if the weaknesses are addressed.
soundness: 4
strengths: 1) The paper addresses a highly relevant topic that warrants investigation, given the growing popularity of FMs and the ongoing efforts within the community to adapt these models to specialized domains beyond natural images and text.

2) The results are surprising and critical to keep a fair benchmarking of future FMs against other pipelines. The actual gains in data efficiency scored by the proposed baselines (nice summary in Figure 1) range from three to five orders of magnitude in the three tested domains (satellite imaging, genomics, and time series). The authors also plan to release the code for their baselines, providing the community with a valuable benchmark resource.

3) The experimental setup is comprehensive, with models evaluated on up to 50 tasks. I found the design of the supervised baseline to be rigorous and fair, especially the decision to approximate standard practitioner model development through architecture search. 

4) The paper is clear, well-written, and easy to follow, making its main message easy for the reader to understand.
summary: This work evaluates the effectiveness of foundation models (FMs) in genomics, satellite data, and time series, finding that well-tuned supervised models can match or outperform FMs in these specialized domains. The results suggest that large-scale pretraining benefits are not yet fully realized in these areas, highlighting the need for solid baseline comparisons.
weaknesses: 1) The paper presents two "solutions" from an architectural standpoint, i.e., CNNs for genomics/satellite data and auto-AR for time series. This leads to two questions: 
- What model should a practitioner use when facing a task in a different specialized domain? 
- How come 1D CNNs (or, e.g., RNNs) fail to match the performance of FMs in time series forecasting?

Is it just a matter of domain knowledge and data type? In practice, the paper would benefit from some more convincing justification and evidence for the statement, "While DASHA can be applied to forecasting tasks, it is not competitive with state-of-the-art time series FMs." (Line 240).
Also, why in Table 3 the evaluation of DASHA for the "Full setting" panel is reported as "unknown" ("-")? 

2) The authors should add a brief discussion either in the Introduction or Related Work with references regarding "simple tuned baselines", which were shown to outperform/match more complex approaches, as this fact has been seen multiple times in the machine/deep learning literature and is clearly relevant to this work. Some related literature:

- Evaluating Weakly Supervised Object Localization Methods Right, CVPR 2020
- In Search of Lost Domain Generalization, ICLR 2021
- Image Classification with Small Datasets: Overview and Benchmark, IEEE Access 2022
- OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization, CVPR 2022
- When Do Neural Nets Outperform Boosted Trees on Tabular Data?, NeurIPS 2023 Datasets & Benchmarks
- Systematic Comparison of Semi-supervised and Self-supervised Learning for Medical Image Classification, CVPR 2024

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: General Questions:
- I don't really understand the importance of including a ResNet and a UNet, as models for classifcation. From my understanding a UNet is an architecture choice where you need the model output to also be a HxW similar to the input. For classification this would imply that you have to include additional layers to get from that output to a vector-valued classification prediction. Additionally, I think a UNet can have all sorts of different convolutional or transformer encoders, including a ResNet, for example see [here](https://github.com/qubvel-org/segmentation_models.pytorch). Having access to the code would probably answer this question.
- In line 255, you state that beyond a subset of the GeoBench classification tasks you also include BigEarthNet, and EuroSAT, as well as two additional ones. However, BigEarthNet and EuroSAT are already included in GeoBench
- CROMA takes as input an optical and/or sar image as the code suggests [here](https://github.com/antofuller/CROMA/blob/59505a6bcadbf36ba20767270154bf9f3067c5e7/use_croma.py#L90). I wonder how that is handled across the selected different classification datasets, as it was not entirely clear to me from the appendix around line 953
- Is it possible to look at the code already which could help answer some of these questions?

Other Comments:
- in line 261 you state that "satellite imaging closely resembles RGB imaging" but that is only true for some optical satellites. Radar Satellites as well as other images are very different than RGB

Minor Comments:
- line 525 there is a typo in the word "work"
- I don't find the acronym "NAS" to be introduced. Also in line 191 you say "neural architecture search" where you use NAS beforehand already
soundness: 2
strengths: Foundation models are a hot topic in various machine learning research domains, and it is important to have appropriate evaluation regimes that can demonstrate the efficacy or show weaknesses in developed models and test whether they are able to deliver on their promises. Therefore, the research question posed by the authors is highly relevant and of importance to the community. Similarly, it is good that the authors focus on a rather straightforward simple baseline tuning technique that can be employed across domains.
summary: The paper conducts a comparitive study of foundation models across three domains of time-series data, genomics and satellite data, whereby a selection of foundation models in each domain are compared to a supervised baseline. More specifically, the authors propose an Auto-ML pipeline based on DASH (Shen et al. 2022)and ASHA (Li et al. 2020). The results show that foundation models are not
able to consistently beat supervised tuned baselines across these domains.
weaknesses: I do understand the authors' idea and motivation to show subpar performance of foundation models against a well tuned supervised baseline across different domains. However, by attempting to do this across quiet different domains, I believe that the paper and methodology is falling short of the specific nuances of each of the included subdomains of genomics, time-series and satellite data. 

I will mainly attempt to make this point for the satellite data domain as this is the most closely related one to my background among the three:
- Earth Observation (EO) data encompasses a wide array of sensors and modalities. Additionally, there are a diverse set of tasks, classification, segmentation and pixel-wise regression just to name a few for optical data. Foundation models aim to perform well across these tasks based on a common encoder that yields relevant representations that can then be "decoded" for the task at hand. However, while you do select several classification datasets, the evaluation is limited to classification and does not include segmentation which is arguably more relevant in EO. Therefore, I would argue it is difficult to make your overall claim, if you do not compare across EO tasks, for example including segmentation, as this would align more with the premise of FMs.
- In line 352 you say "we only consider a restricted subset of top-performing, open-source, and compatibly-formatted models". However, I would believe that there are other relevant and easily accessible recent models available like [Clay](https://github.com/Clay-foundation/model), [Scale-MAE](https://github.com/microsoft/torchgeo/blob/main/torchgeo/models/scale_mae.py), the [Satlas-model](https://github.com/microsoft/torchgeo/blob/main/torchgeo/models/swin.py), or [DOFA](https://github.com/microsoft/torchgeo/blob/main/torchgeo/models/dofa.py) that include evaluations against your chosen models of CROMA and Sat-MAE.
- Regarding the experimental findings, I would argue that this is not necessarily new, because there are works that already show strong performances of supervised baselines. For example in the [Prithvi by Jakubik et al.](https://arxiv.org/abs/2310.18660) Table 4 you observe the competitive UNet performanceand that for subclasses it is often better than the FM. Similarly, in [DOFA by Xiong et al.](https://arxiv.org/abs/2403.15356), Table 1 and 2 show that fully trained supervised baselines can sometimes beat the finetuning schemes. More general a work by [Corley et al.](https://openaccess.thecvf.com/content/CVPR2024W/PBVS/html/Corley_Revisiting_Pre-trained_Remote_Sensing_Model_Benchmarks_Resizing_and_Normalization_Matters_CVPRW_2024_paper.html) conducted a focused study around pretrained weights and demonstrated that carefully finetuned baselines are very competitive.
- From Table 6 it seems that you have conducted the experiments on quiet large datasets in the domain. However, the more interesting application of FMs is for fine tuning on datasets with scarce/few labels where training a model from scratch would likely just lead to over fitting. This domain relevant scenario is not considered here.

Given that for the satellite subdomain only a subset of existing FMs were used for classification (which is understandable since this requires significant experimental effort), I think some of the author's general statements do not follow the methodology. For example, the authors lead their discussion by stating in line 470 that "At a high level, our results show that the foundation models in these three domains have not yet surpassed supervised learning" but the selected methods are not representative of "the foundation models in the domain as illustrated for satellite data. As I said in the beginning, I believe that there are too many nuances within the separate domains that can be all included in one paper and allow for the conclusions that the authors draw.

Meta-review: In this paper, the authors consider three modalities and compare "foundational model" performance to a standard supervised learning workflow.  They show that the latter can beat the former in all modalities considered. This is an interesting piece of work that sanity checks the "foundational model" paradigm. Reviewers appreciated the topic, and the rigorous experimental setup. There were criticisms, but the authors responded well to these which saw three reviewer scores raised. As this paper has entirely positive reviews, and no major flaws I believe this should be accepted.

Decision: Accept (Poster)