Toto: Time Series Optimized Transformer for Observability

## Abstract

Abstract This technical report describes the Time Series Optimized Transformer for Observability (Toto), a new state-of-the-art foundation model for time series forecasting developed by Datadog. In addition to advancing the state of the art on generalized time series benchmarks in domains such as electricity and weather, this model is the first general-purpose time series forecasting foundation model to be specifically tuned for observability metrics. Toto was trained on a dataset of one trillion time series data points – the largest among all currently published time series foundation models. Alongside publicly available time series datasets, 75% of the data used to train Toto consists of fully anonymous numerical metric data points from the Datadog platform. In our experiments, Toto outperforms existing time series foundation models on observability data. It does this while also excelling at general-purpose forecasting tasks, achieving state-of-the-art zero-shot performance on multiple open benchmark datasets. In this report, we detail the following key contributions: • Proportional factorized space-time attention: We introduce an advanced attention mechanism that allows for efficient grouping of multivariate time series features, reducing computational overhead while maintaining high accuracy. • Student-T mixture model head: This novel use of a probabilistic model that robustly generalizes Gaussian mixture models enables Toto to more accurately capture the complex dynamics of time series data and provides superior performance over traditional approaches. • Domain-specific training data: In addition to general multi-domain time series data, Toto is specifically pre-trained on a large-scale dataset of Datadog observability metrics, encompassing unique characteristics not present in open-source datasets. This targeted training ensures enhanced performance in observability metric forecasting. Figure 1 : Toto architecture diagram. Input time series of T 𝑇 T italic_T steps (univariate example used for simplicity here) are first embedded using the patch embedding layer. They then pass through the transformer stack, which contains L 𝐿 L italic_L identical segments. Each segment of the transformer consists of one space-wise transformer block followed by N 𝑁 N italic_N time-wise blocks. The flattened transformer outputs are projected to form the parameters of the Student-T mixture model (SMM) head. The final outputs are the forecasts for the input series, shifted P 𝑃 P italic_P steps (the patch width) into the future.

## 1   Background

Figure: Figure 2: Example of Toto's 96-step zero-shot forecasts on the ETTh1 dataset, showing multivariate probabilistic predictions. Solid lines represent ground truth, dashed lines represent median point forecasts, and shaded regions represent 95% prediction intervals.
Refer to caption: x2.png

We present Toto, a groundbreaking time series forecasting foundation model developed by Datadog. Toto is specifically designed to handle the complexities of observability data, leveraging a state-of-the-art transformer architecture to deliver unparalleled accuracy and performance. Toto is trained on a massive dataset of diverse time series data, enabling it to excel in zero-shot predictions. This model is tailored to meet the demanding requirements of real-time analysis as well as compute and memory-efficient scalability to very large data volumes, providing robust solutions for high-frequency and high-dimensional data commonly encountered in observability metrics.

### 1.1   Observability data

The Datadog observability platform collects a vast array of metrics across multiple subdomains, crucial for monitoring and optimizing modern infrastructure and applications. These metrics include infrastructure data such as memory usage, CPU load, disk I/O, and network throughput, as well as application performance indicators like hit counts, error rates, and latency []. Additionally, Datadog integrates specific metrics from numerous SaaS products, cloud services, open-source frameworks, and other third-party tools. The platform allows users to apply various time series models to proactively alert on anomalous behavior, leading to a reduction in time to detection (TTD) and time to resolution (TTR) of production incidents [].

The complexity and diversity of these metrics present significant challenges for time series forecasting. Observability data often requires high time resolution, down to seconds or minutes, and is typically sparse with many zero-inflated metrics. Moreover, these metrics can display extreme dynamic ranges and right-skewed distributions. The dynamic and nonstationary nature of the systems being monitored further complicates the forecasting task, necessitating advanced models that can adapt and perform under these conditions.

### 1.2   Traditional models

Historically, time series forecasting has relied on classical models such as ARIMA, exponential smoothing, and basic machine learning techniques []. While foundational, these models necessitate individual training for each metric, presenting several limitations []. The need to develop and maintain separate models for each metric impedes scalability, especially given the extensive range of metrics in observability data. Moreover, these models often fail to generalize across different types of metrics, leading to suboptimal performance on diverse datasets [, ]. Continuous retraining and tuning to adapt to evolving data patterns further increase the operational burden. This scaling limitation has hindered the adoption of deep learning–based methods for time series analysis, even as they show promise in terms of accuracy [].

### 1.3   Foundation models

Large neural network-based generative models, often referred to as “foundation models,” have revolutionized time series forecasting by enabling accurate predictions on new data not seen during training, known as zero-shot prediction []. This capability significantly reduces the need for constant retraining on each specific metric, thus saving considerable time and computational resources. Their architecture supports the parallel processing of vast data volumes, facilitating timely insights essential for maintaining system performance and reliability [, ].

Through pretraining on diverse datasets, generative models exhibit strong generalization across various types of time series data. This enhances their robustness and versatility, making them suitable for a wide range of applications. Zero-shot predictions are particularly attractive in the observability domain, where the limitations of traditional methods are felt very acutely. The most common use cases for time series models within an observability platform like Datadog include automated anomaly detection and predictive alerting. It is challenging to scale classical forecasting methods to handle cloud-based applications that can be composed of many ephemeral, dynamically scaling components such as containers, VMs, serverless functions, etc. These entities tend to be both high in cardinality and short-lived in time. This limits the practicality of traditional time series models in two ways:

- •
First, the high cardinality and volume of data can make fitting individual models to each time series computationally expensive or even intractable. The ability to train a single model and perform inference across a wide range of domains has the potential to dramatically improve the efficiency, and thus the coverage, of an autonomous monitoring system.
- •
Second, ephemeral infrastructure elements often lack enough historical data to confidently fit a model. In practice, algorithmic alerting systems often require an adaptation period of days or weeks before they can usefully monitor a new metric. However, if the object being monitored is a container with a lifespan measured in minutes or hours, these classical models are unable to adapt quickly enough to be useful. Real-world systems thus often fall back to crude heuristics, such as threshold-based alerts, which rely on the domain knowledge of users. Zero-shot foundation models can enable accurate predictions with much less historical context, by aggregating and interpolating prior information learned from a massive and diverse dataset.

The integration of transformer-based models [] like Toto into observability data analysis thus promises significant improvements in forecasting accuracy and efficiency. These models offer a robust solution for managing diverse, high-frequency data and delivering zero-shot predictions. With their advanced capabilities, transformer-based models represent a significant leap forward in the field of observability and time series analysis [, , ].

### 1.4   Recent work

The past several years have seen the rise of transformer-based models as powerful tools for time series forecasting. These models leverage multi-head self-attention mechanisms to capture long-range dependencies and intricate patterns in data.

To address the unique challenges of time series data, recent advancements have introduced various modifications to the attention mechanism. For example, Moirai [] uses “any-variate” attention to model dependencies across different series simultaneously. Factorized attention mechanisms [] have been developed to separately capture temporal and spatial (cross-series) interactions, enhancing the ability to understand complex interdependencies. Other models [, ] have used cross-channel attention in conjunction with feed-forward networks for mixing in the time dimension. Additionally, causal masking [] and hierarchical encoding [] can improve the efficiency and accuracy of predictions in time series contexts.

These innovative transformer-based models have demonstrated state-of-the-art performance on benchmark datasets [], frequently surpassing traditional models in both accuracy and robustness. Their capacity to process high-dimensional data efficiently [] makes them ideal for applications involving numerous time series metrics with varying characteristics, such as observability.

Even more recently, a number of time series “foundation models” have been released [, , , , , ]. By pre-training on extensive, multi-domain datasets, these large models achieve impressive zero-shot prediction capabilities, significantly reducing the need for constant retraining. This paradigm is appealing for the observability context, where we constantly have new time series to process and frequent retraining is impractical.

## 2   Problem statement

At Datadog, our time series data encompasses a variety of observability metrics from numerous subdomains. These metrics present several challenges for existing forecasting models:

- •
High time resolution: Users often require data in increments of seconds or minutes, unlike many publicly-available time series datasets that are at hourly frequency or above.
- •
Sparsity: Metrics such as error counts often track rare events, resulting in sparse and zero-inflated time series.
- •
Extreme right skew: Latency measurements in distributed systems exhibit positive, heavy tailed distributions with extreme values at high percentiles.
- •
Dynamic, nonstationary systems: The behavior of monitored systems changes frequently due to code deployments, infrastructure scaling, feature flag management, and other configuration changes, as well as external factors like seasonality and user-behavior-driven trends. Some time series, such as those monitoring fleet deployments, can also have a very low variance, exhibiting a piecewise-constant shape.
- •
High-cardinality multivariate data: Monitoring large fleets of ephemeral cloud infrastructure such as virtual machines (VMs), containers, serverless functions, etc. leads to high cardinality data, with hundreds or thousands of individual time series variates, often with limited historical data for each group.
- •
Historical anomalies: Historical data often contains outliers and anomalies caused by performance regressions or production incidents.

Foundation models pre-trained on other domains struggle to generalize effectively to observability data due to these characteristics. To overcome this, we developed Toto, a state-of-the-art foundation model that excels at observability forecasting while also achieving top performance on standard open benchmarks.

## 3   Model architecture

Toto is a decoder-only forecasting model. This model employs many of the latest techniques from the literature, and introduces a novel method for adapting multi-head attention to multivariate time series data (Fig. 1).

### 3.1   Transformer design

Transformer models for time series forecasting have variously used encoder-decoder [, , ], encoder-only [, , ], and decoder-only architectures [, ]. For Toto, we employ a decoder-only architecture. Decoder architectures have been shown to scale well [, ], and allow for arbitrary prediction horizons. The causal next-patch prediction task also simplifies the pre-training process.

We use techniques from some of the latest large language model (LLM) architectures, including pre-normalization [], RMSNorm [], and SwiGLU feed-forward layers [].

### 3.2   Input embedding

Time series transformers in the literature have used various approaches for creating input embeddings. We use non-overlapping patch projections (Fig. 3), first introduced for Vision Transformers [, ] and popularized in the time series context by PatchTST []. Toto was trained using a fixed patch size of 32.

Figure: Figure 3: The patch embedding takes as input a multivariate time series of $M$ variates by $N$ time steps. It divides each variate along the time dimension into patches of size $P$ and projects these linearly into an embedding space of latent dimension $D$. This results in an output of size $M\times\frac{N}{P}\times D$ which is fed to the transformer decoder.

### 3.3   Attention mechanism

Observability metrics are often high-cardinality, multivariate time series. Therefore, an ideal model will natively handle multivariate forecasting. It should be able to analyze relationships both in the time dimension (what we refer to as “time-wise” interactions) and in the channel dimension (what we refer to as “space-wise” interactions, following the convention in the Datadog platform of describing different groups or tag sets of a metric as the “space” dimension).

In order to model both space and time-wise interactions, we need to adapt the traditional multi-head attention architecture [] from one to two dimensions. Several approaches have been proposed in the literature to do this, including:

- •
Assuming channel independence, and computing attention only in the time dimension []. This is efficient, but throws away all information about space-wise interactions.
- •
Computing attention only in the space dimension, and using a feed-forward network in the time dimension [, ].
- •
Concatenating variates along the time dimension and computing full cross-attention between every space/time location []. This can capture every possible space and time interaction, but it is computationally costly.
- •
Computing “factorized attention,” where each transformer block contains a separate space and time attention computation [, , ]. This allows both space and time mixing, and is more efficient than full cross-attention. However, it doubles the effective depth of the network.

In order to design our attention mechanism, we follow the intuition that for many time series, the time relationships are more important or predictive than the space relationships. As evidence, we observe that even models that completely ignore space-wise relationships (such as PatchTST [] and TimesFM []) can still achieve competitive performance on multivariate datasets. However, other studies (e.g. Moirai []) have shown through ablations that there is some clear benefit to including space-wise relationships.

We therefore propose a novel variant of factorized attention, which we call “Proportional Factorized Space-Time Attention.” We use a mixture of alternating space-wise and time-wise attention blocks. As a configurable hyperparameter, we can change the ratio of time-wise to space-wise blocks, thus allowing us to devote more or less compute budget to each type of attention. For our base model, we selected a configuration with one space-wise attention block for every two time-wise blocks.

In the time-wise attention blocks, we use causal masking and rotary positional embeddings [] with XPOS [] in order to autoregressively model time-dependent features. In the space-wise blocks, by contrast, we use full bidirectional attention in order to preserve permutation invariance of the covariates, with a block-diagonal ID mask to ensure that only related variates attend to each other. This masking allows us to pack multiple independent multivariate time series into the same batch, in order to improve training efficiency and reduce the amount of padding.

Figure: Figure 4: Example metric query in the Datadog platform. The metric name (1) determines which metric is being queried. The filter clause (2) limits which contexts are queried, in this case restricting the query to the prod environment. The space aggregation (3) indicates that the average metric value should be returned for each unique combination of the group-by keys. The time aggregation (4) indicates that metric values should be aggregated to the average for each 60-second interval. The query results will be a multivariate time series with 1-minute time steps, and with separate individual variates for each unique service, datacenter tuple.
Refer to caption: x4.png

### 3.4   Probabilistic prediction head

In order to be useful for forecasting applications, a model should produce probabilistic predictions. A common practice in time series models is to use an output layer where the model regresses the parameters of a probability distribution. This allows for prediction intervals to be computed using Monte Carlo sampling [].

Common choices for an output layer are Normal [] and Student-T [, ], which can improve robustness to outliers. Moirai [] allows for more flexible residual distributions by proposing a novel mixture model incorporating a weighted combination of Gaussian, Student-T, Log-Normal, and Negative-Binomial outputs.

However, real-world time series can often have complex distributions that are challenging to fit, with outliers, heavy tails, extreme skew, and multimodality. In order to accommodate these scenarios, we introduce an even more flexible output likelihood. To do this we employ a method based on Gaussian mixture models (GMMs), which can approximate any density function ([]). To avoid training instability in the presence of outliers, we use a Student-T mixture model (SMM), a robust generalization of GMMs [] that has previously shown promise for modeling heavy-tailed financial time series [, ]. The model predicts $k$ Student-T distributions (where $k$ is a hyperparameter) for each time step, as well as a learned weighting.

When we perform inference, we draw samples from the mixture distribution at each timestamp, then feed each sample back into the decoder for the next prediction. This allows us to produce prediction intervals at any quantile, limited only by the number of samples; for more precise tails, we can choose to spend more computation on sampling (Fig. 2).

### 3.5   Input/output scaling

As in other time series models, we perform instance normalization on input data before passing it through the patch embedding, in order to make the model generalize better to inputs of different scales []. We scale the inputs to have zero mean and unit standard deviation. The output predictions are then rescaled back to the original units.

### 3.6   Training objective

As a decoder-only model, Toto is pre-trained on the next-patch prediction task. We minimize the negative log-likelihood of the next predicted patch with respect to the distribution output of the model. We train the model using the AdamW optimizer [].

### 3.7   Hyperparameters

The hyperparameters used for Toto are detailed in Table A.1, with 103 million total parameters.

## 4   Training data

We pretrained Toto with a dataset of approximately one trillion time series points. Of these, roughly three-quarters are anonymous observability metrics from the Datadog platform. The remaining points come from the LOTSA dataset [], a compilation of publicly-available time series datasets across many different domains.

### 4.1   Datadog dataset

The Datadog platform ingests more than a hundred trillion events per day. However, much of this data is sparse, noisy, or too granular or high in cardinality to be useful in its raw form. To curate a high-quality dataset for efficient model training, we sample queries based on quality and relevance signals from dashboards, monitor alerts, and notebooks. This provides a strong signal that the data resulting from these queries is of critical importance and sufficient quality for observability of real-world applications.

**Table 1: Comparison of different models with Toto on the LSF benchmark datasets. Results are averaged across prediction lengths of 96, 192, 336, and 720 steps. For Toto, we use a stride of 512 steps and a historical context window of 512 steps. For other models, we use the results reported in [] and []. Metrics for each prediction length are available in Table A.2. ^*TimesFM only reports values for MAE on ETTh1, ETTh2, ETTm1, and ETTm2. Key: Best results, Second-best results.**
|  |  | Zero Shot | Full Shot |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | Metric | Toto | MoiraiSmall | MoiraiBase | MoiraiLarge | TimesFM^* | iTransformer | TimesNet | PatchTST | Crossformer | TiDE | DLinear | SCINet | FEDformer |
| ETTh1 | MAE | 0.389 | 0.424 | 0.438 | 0.469 | 0.426 | 0.448 | 0.450 | 0.455 | 0.522 | 0.507 | 0.452 | 0.647 | 0.460 |
|  | MSE | 0.363 | 0.400 | 0.434 | 0.510 | - | 0.454 | 0.458 | 0.469 | 0.529 | 0.541 | 0.456 | 0.747 | 0.440 |
| ETTh2 | MAE | 0.261 | 0.379 | 0.382 | 0.376 | 0.410 | 0.407 | 0.497 | 0.407 | 0.684 | 0.550 | 0.515 | 0.723 | 0.449 |
|  | MSE | 0.170 | 0.341 | 0.345 | 0.354 | - | 0.383 | 0.414 | 0.387 | 0.942 | 0.611 | 0.559 | 0.954 | 0.437 |
| ETTm1 | MAE | 0.375 | 0.409 | 0.388 | 0.389 | 0.388 | 0.410 | 0.406 | 0.400 | 0.495 | 0.419 | 0.407 | 0.481 | 0.452 |
|  | MSE | 0.372 | 0.448 | 0.381 | 0.390 | - | 0.407 | 0.400 | 0.387 | 0.513 | 0.419 | 0.403 | 0.486 | 0.448 |
| ETTm2 | MAE | 0.319 | 0.341 | 0.321 | 0.320 | 0.334 | 0.332 | 0.333 | 0.326 | 0.611 | 0.404 | 0.401 | 0.537 | 0.349 |
|  | MSE | 0.272 | 0.300 | 0.272 | 0.276 | - | 0.288 | 0.291 | 0.281 | 0.757 | 0.358 | 0.350 | 0.571 | 0.305 |
| Electricity | MAE | 0.246 | 0.320 | 0.274 | 0.273 | - | 0.270 | 0.295 | 0.304 | 0.334 | 0.344 | 0.300 | 0.365 | 0.327 |
|  | MSE | 0.157 | 0.233 | 0.188 | 0.188 | - | 0.178 | 0.193 | 0.216 | 0.244 | 0.252 | 0.212 | 0.268 | 0.214 |
| Weather | MAE | 0.284 | 0.267 | 0.261 | 0.275 | - | 0.278 | 0.287 | 0.281 | 0.315 | 0.320 | 0.317 | 0.363 | 0.360 |
|  | MSE | 0.256 | 0.242 | 0.238 | 0.259 | - | 0.258 | 0.259 | 0.259 | 0.259 | 0.271 | 0.265 | 0.292 | 0.309 |
| Mean | MAE | 0.312 | 0.357 | 0.341 | 0.350 | - | 0.357 | 0.378 | 0.362 | 0.493 | 0.424 | 0.399 | 0.519 | 0.400 |
|  | MSE | 0.265 | 0.328 | 0.315 | 0.330 | - | 0.328 | 0.336 | 0.333 | 0.541 | 0.409 | 0.374 | 0.533 | 0.359 |

Datadog metrics are accessed using a specialized query language supporting filters, group-bys, time aggregation, and various transformations and postprocessing functions []. We consider groups returned from the same query to be related variates in a multivariate time series (Fig. 4). After we retrieve the query results, we discard the query strings and group identifiers, keeping only the raw numeric data.

Handling this vast amount of data requires several preprocessing steps to ensure consistency and quality. Initially, we apply padding and masking techniques to align the series lengths, making them divisible by the patch stride. This involves adding necessary left-padding to both the time series data and the ID mask, ensuring compatibility with the model's requirements.

Various data augmentations are employed to enhance the dataset's robustness. We introduce random time offsets to prevent memorization caused by having series always align the same way with the patch grid. After concatenating the Datadog and LOTSA datasets for training, we also implement a variate shuffling strategy to maintain diversity and representation. Specifically, 10% of the time, we combine variates that are not necessarily related, thus creating new, diverse combinations of data points. To sample the indices, we employ a normal distribution with a standard deviation of 1000, favoring data points that were closer together in the original datasets. This Gaussian sampling ensures that, while there is a preference for adjacent data points, significant randomness is introduced to enhance the diversity of the training data. This approach improves the model's ability to generalize across different types of data effectively.

By implementing these rigorous preprocessing steps and sophisticated data handling mechanisms, we ensure that the training data for Toto is of the highest quality, ultimately contributing to the model's superior performance and robustness.

### 4.2   Synthetic data

We use a synthetic data generation process similar to TimesFM [] to supplement our training datasets, improving the diversity of the data and helping to teach the model basic structure. We simulate time series data through the composition of components such as piecewise linear trends, ARMA processes, sinusoidal seasonal patterns, and various residual distributions. We randomly combine five of these processes per variate, introducing patterns not always present in our real-world datasets. The generation process involves creating base series with random transformations, clipping extreme values, and rescaling to a specified range. By making synthetic data approximately 5% of our training dataset, we ensure a wide range of time-series behaviors are captured. This diversity exposes our models to various scenarios during training, improving their ability to generalize and effectively handle real-world data.

## 5   Results

We report experimental results for a pre-trained Toto model in Section 5.1 and Section 5.2.

To evaluate predictions, we sequentially divide a time series into context and forecast segments. We input the context segment into Toto and autoregressively generate output patches by sampling from the Student-T mixture model distribution. We forecast a number of steps equal to the nearest multiple of the patch size, then truncate the predictions to the desired length. In order to keep inference time consistent, we vary the number of samples generated based on the cardinality and length of the dataset, with a minimum of 100 samples. We take the median sample at each time step as the final point prediction. This prediction is then compared against the ground-truth forecast segment for evaluation.

### 5.1   LSF benchmarks

To assess general-purpose time series forecasting performance, we use the Long Sequence Forecasting (LSF) benchmark datasets (ETTh1, ETTh2, ETTm1, ETTm2, Electricity, and Weather) []. We evaluate with forecast lengths of 96, 192, 336, and 720 time steps, in sliding windows with stride 512, and average the results. For Toto, we used a historical context window of 512 steps and took the median of 200 samples. Following standard practice, we report normalized Mean Absolute Error (MAE) and Mean Squared Error (MSE), fitted on a training split, in order to be able to compare performance across different datasets. We compared Toto's performance with the reported results of other recent zero-shot foundation models [, ], as well as full-shot time series forecasting models [, , , , , , , ]. We display these results in Table 1.

Toto demonstrates exceptional performance across a variety of benchmark datasets, excelling in zero-shot scenarios. In the LSF datasets, Toto consistently outperforms other models in terms of MAE and MSE. For example, on the ETTh1 dataset, Toto achieves an MAE of 0.389 and an MSE of 0.363, outperforming all zero-shot models, including the previously reported Moirai series and TimesFM. Macro-averaging across the six LSF datasets, Toto achieves an MAE of 0.312 and MSE of 0.265, again exceeding Moirai's reported zero-shot performance as well as the reported performance of the full-shot models.

Several architectural choices and data features likely contribute to Toto's superior performance. The novel Proportional Factorized Space-Time Attention mechanism allows Toto to efficiently capture both temporal and spatial dependencies within multivariate time series data. Additionally, the extensive training on a diverse dataset of one trillion time series points, including a mix of real-world observability metrics and multi-domain time series data, enhances Toto's ability to handle varied characteristics of different benchmark datasets.

While Toto generally excels, there are areas where its performance is closely matched by other models. In full-shot scenarios, models like PatchTST, Crossformer, and FEDformer show competitive results. For example, on the Electricity dataset, while Toto achieves a leading zero-shot MAE of 0.246 and MSE of 0.157, iTransformer and TimesNet also show strong performance, indicating that these models can catch up when additional training data is available.

Overall, Toto's architectural innovations and extensive training data enable it to achieve state-of-the-art performance across diverse benchmarks, excelling in zero-shot scenarios while remaining highly competitive in full-shot contexts.

**Table 2: Performance of Toto and other zero-shot models on the Datadog benchmark dataset. Key: Best results, Second-best results.**
| Metric | Toto | Chronos-T5Tiny | Chronos-T5Mini | Chronos-T5Small | Chronos-T5Base | Chronos-T5Large | MoiraiSmall | MoiraiBase | MoiraiLarge | TimesFM |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| sMAPE | 0.672 | 0.809 | 0.788 | 0.800 | 0.796 | 0.805 | 0.808 | 0.742 | 0.736 | 1.246 |
| sMdAPE | 0.318 | 0.406 | 0.391 | 0.401 | 0.393 | 0.396 | 0.418 | 0.370 | 0.365 | 0.639 |

### 5.2   Datadog benchmark

We created a benchmark using anonymous Datadog data to assess performance across various observability metrics. To ensure a representative and realistic sample, we sampled data based on quality and relevance signals from dashboards, monitor alerts, and notebooks. This benchmark comprises 983,994 data points from 82 distinct multivariate time series, encompassing 1,122 variates.

We analyzed summary statistics of the series in our benchmark to identify characteristics that make observability time series challenging to forecast. The categories and their definitions are as follows:

- •
Sparse: Series with a low density of observations, indicating infrequent recording of data or rare events.
- •
Extreme right skew: Series with a distribution heavily skewed to the right, characterized by a few very high values and many lower values.
- •
Seasonal: Series exhibiting regular and recurring patterns, often linked to daily, weekly, or yearly cycles.
- •
Flat: Series with minimal variability, showing little to no change over time.

The relative proportion of these cases are displayed in Table 3.

To assess the prediction of other zero-shot models on the DD Benchmark, we follow sampling procedures delineated in their respective manuscripts. In short, for Chronos models, we generate 20 samples and take the median prediction. For Moirai models, we take the median of 100 samples and set the patch size to “auto”. TimesFM only produces point predictions of the mean, so we use those directly. Since TimesFM and Chronos only support univariate forecasting, we process each variate independently. Moirai, on the other hand, like Toto, makes joint predictions for each group of related variates. For Toto, we utilize the same evaluation procedure we used on the LSF benchmarks.

The evaluation results (Table 2) demonstrate that Toto outperforms the other models. We evaluate using a prediction length of 365, the maximum forecast window available for previous time series models within the Datadog platform. We use a historical context window of 512 steps. Because observability data can have extreme variation in both magnitude and dispersion, we select symmetric mean absolute percentage error (sMAPE) as a scale-invariant performance metric []. We also report symmetric median absolute percentage error (sMdAPE), a robust version of sMAPE [] that minimizes the influence of the extreme outliers present in observability data. With the lowest sMAPE of 0.672 and sMdAPE of 0.318, Toto proves to be the most accurate for forecasting observability time series data.

These results suggest that current open datasets may not provide sufficient information to extrapolate to the specific nuances of observability data, highlighting the importance of training on more relevant data as demonstrated by Toto's superior performance.

Case
%
Sparse
12.20
Extreme Right Skew
17.07
Seasonal
80.49
Flat
1.22

Table: Table 3: Breakdown of Datadog dataset based on case, computed based on the average characteristics of variates in each multivariate series. Note that these do not add to 100% because time series may fall into multiple categories.

## 6   Conclusions

Toto, through a novel architecture and pre-training corpus, demonstrates state-of-the-art performance both on public benchmarks and on the Datadog observability benchmark. We look forward to sharing many more technical details, experiments, and benchmark results in a forthcoming paper.

## 7   Impact statement

In developing Toto, Datadog follows a structured approach to ensure responsible development, focusing on identifying, assessing, and mitigating potential risks associated with the use of our model. Given that Toto is not intended for mass distribution and specifically generates time series forecasts for observability data, the potential harms are considerably lower compared to more general-purpose models. At Datadog, our primary focus is on ensuring the accuracy, reliability, and security of the forecasts generated by Toto, which are crucial for maintaining and optimizing infrastructure and application performance.

We carefully analyze the potential for Toto to produce incorrect or misleading forecasts that could impact decision-making processes in critical systems. Additionally, we consider the implications of Toto's performance across diverse datasets, ensuring it can generalize well without introducing significant errors.

## 8   Future directions

Many exciting areas of exploration remain for further study. If you are interested in working with us, please reach out to the authors by email.

Some future research questions that particularly intrigue us include:

- •
Multi-modal inputs: Incorporate additional input modalities such as query metadata and captions to enhance forecast performance.
- •
Autonomous troubleshooting agents: Augment Datadog's AI agents [] for troubleshooting and incident response by integrating modality-specific foundation models like Toto to improve their reasoning and planning capabilities with telemetry data.
- •
Conversational interfaces: Align time series forecasting models with LLMs to develop conversational agents capable of interpreting and reasoning about time series data.
- •
Model enhancements and scaling: Explore ways to improve and scale model performance through optimizations such as new types of input embeddings, attention mechanisms, and examining alternative variate groupings to capture richer interactions.

## 9   Contributions

Contributors are listed in alphabetical order.

Othmane Abou-Amal, Joseph Banks, Mayeul Blanzat, Ben Cohen, Youssef Doubli, Ben Hinthorne, Emaad Khwaja, Jared Ledvina, Charles Masson, Sajid Mehmood, Elise Ramé, Maxime Visonneau, Kan Wang.

## 10   Acknowledgements

Our work is made possible by the efforts of numerous teams at Datadog. Special thanks and acknowledgement to:

Johan Andersen, Roashan Ayene, Romoli Bakshi, Kevin Beach, Bill Birkholz, Rob Boll, Maxim Brown, Benedetto Buratti, Marion Chan-Renous, Jessica Cordonnier, Ben Donohue, Zakaria Fikrat, Quentin François, Erica Hale, Michael Hoang, Joe Jones, Max Livingston, Jesse Mack, Amine Naouas, Sean O'Connor, Brendan Rhoads, Phil Sarin, Vyom Shah, Aaron Taa, Bharath Vontimitta, Dominique West, Steven Zhou.

## Appendix

## A.1   Model architecture

**Table A.1: Hyperparameters for Toto**
| Hyperparameter | Value |
| --- | --- |
| Embedding Dimension | 512 |
| MLP Dimension | 2048 |
| # Layers | 24 |
| # Heads | 8 |
| # Variates | 32 |
| ($\beta_{1}$, $\beta_{2}$) | (0.9, 0.95) |
| Weight Decay | 0.01 |
| Spacewise Layer Cadence | 3 |
| Patch Size | 32 |
| # Student-T Mixture Model Components | 16 |
| Initial Learning Rate | 0.001 |
| Annealing Schedule | Cosine |
| Batch Size | 192 |
| Warmup Steps | 5000 |
| Total Train Steps | 193000 |

## A.2   Results

**Table A.2: Performance metrics for various models. Key: Best results, Second-best results.**
|  | Zero Shot | Full Shot |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | Prediction Length | Metric | Toto | MoiraiSmall | MoiraiBase | MoiraiLarge | TimesFM | iTransformer | TimesNet | PatchTST | Crossformer | TiDE | DLinear | SCINet | FEDformer |
|  | 96 | MAE | 0.366 | 0.402 | 0.402 | 0.398 | 0.398 | 0.405 | 0.402 | 0.419 | 0.448 | 0.464 | 0.400 | 0.599 | 0.419 |
|  |  | MSE | 0.307 | 0.375 | 0.384 | 0.380 | - | 0.386 | 0.384 | 0.414 | 0.423 | 0.479 | 0.386 | 0.654 | 0.376 |
|  | 192 | MAE | 0.368 | 0.419 | 0.429 | 0.434 | 0.424 | 0.436 | 0.429 | 0.445 | 0.474 | 0.492 | 0.432 | 0.631 | 0.448 |
| ETTh1 |  | MSE | 0.329 | 0.399 | 0.425 | 0.440 | - | 0.441 | 0.436 | 0.460 | 0.471 | 0.525 | 0.437 | 0.719 | 0.420 |
|  | 336 | MAE | 0.399 | 0.429 | 0.450 | 0.474 | 0.436 | 0.458 | 0.469 | 0.466 | 0.546 | 0.515 | 0.459 | 0.659 | 0.465 |
|  |  | MSE | 0.396 | 0.412 | 0.456 | 0.514 | - | 0.487 | 0.491 | 0.501 | 0.570 | 0.565 | 0.481 | 0.778 | 0.459 |
|  | 720 | MAE | 0.424 | 0.444 | 0.473 | 0.568 | 0.445 | 0.491 | 0.500 | 0.488 | 0.621 | 0.558 | 0.516 | 0.699 | 0.507 |
|  |  | MSE | 0.419 | 0.413 | 0.470 | 0.705 | - | 0.503 | 0.521 | 0.500 | 0.653 | 0.594 | 0.519 | 0.836 | 0.506 |
|  | 96 | MAE | 0.197 | 0.334 | 0.327 | 0.325 | 0.356 | 0.349 | 0.374 | 0.348 | 0.584 | 0.440 | 0.387 | 0.621 | 0.397 |
|  |  | MSE | 0.093 | 0.281 | 0.277 | 0.287 | - | 0.297 | 0.340 | 0.302 | 0.745 | 0.400 | 0.333 | 0.707 | 0.358 |
|  | 192 | MAE | 0.231 | 0.373 | 0.374 | 0.367 | 0.400 | 0.400 | 0.414 | 0.400 | 0.656 | 0.509 | 0.476 | 0.689 | 0.439 |
| ETTh2 |  | MSE | 0.135 | 0.340 | 0.340 | 0.347 | - | 0.380 | 0.402 | 0.388 | 0.877 | 0.528 | 0.477 | 0.860 | 0.429 |
|  | 336 | MAE | 0.260 | 0.393 | 0.401 | 0.393 | 0.428 | 0.432 | 0.541 | 0.433 | 0.731 | 0.571 | 0.541 | 0.744 | 0.487 |
|  |  | MSE | 0.160 | 0.362 | 0.371 | 0.377 | - | 0.428 | 0.452 | 0.426 | 1.043 | 0.643 | 0.594 | 1.000 | 0.496 |
|  | 720 | MAE | 0.355 | 0.416 | 0.426 | 0.421 | 0.457 | 0.445 | 0.657 | 0.446 | 0.763 | 0.679 | 0.657 | 0.838 | 0.474 |
|  |  | MSE | 0.294 | 0.380 | 0.394 | 0.404 | - | 0.427 | 0.462 | 0.431 | 1.104 | 0.874 | 0.831 | 1.249 | 0.463 |
|  | 96 | MAE | 0.328 | 0.383 | 0.360 | 0.363 | 0.345 | 0.368 | 0.375 | 0.367 | 0.426 | 0.387 | 0.372 | 0.438 | 0.419 |
|  |  | MSE | 0.306 | 0.404 | 0.335 | 0.353 | - | 0.334 | 0.338 | 0.329 | 0.404 | 0.364 | 0.345 | 0.418 | 0.379 |
|  | 192 | MAE | 0.353 | 0.402 | 0.379 | 0.380 | 0.374 | 0.391 | 0.387 | 0.385 | 0.451 | 0.404 | 0.389 | 0.450 | 0.441 |
| ETTm1 |  | MSE | 0.328 | 0.435 | 0.366 | 0.376 | - | 0.377 | 0.374 | 0.367 | 0.450 | 0.398 | 0.380 | 0.439 | 0.426 |
|  | 336 | MAE | 0.389 | 0.416 | 0.394 | 0.395 | 0.397 | 0.420 | 0.411 | 0.410 | 0.515 | 0.425 | 0.413 | 0.485 | 0.459 |
|  |  | MSE | 0.390 | 0.462 | 0.391 | 0.399 | - | 0.426 | 0.410 | 0.399 | 0.532 | 0.428 | 0.413 | 0.490 | 0.445 |
|  | 720 | MAE | 0.429 | 0.437 | 0.419 | 0.417 | 0.436 | 0.459 | 0.450 | 0.439 | 0.589 | 0.461 | 0.453 | 0.550 | 0.490 |
|  |  | MSE | 0.463 | 0.490 | 0.434 | 0.432 | - | 0.491 | 0.478 | 0.454 | 0.666 | 0.487 | 0.474 | 0.595 | 0.543 |
|  | 96 | MAE | 0.270 | 0.282 | 0.269 | 0.260 | 0.263 | 0.264 | 0.267 | 0.259 | 0.366 | 0.305 | 0.292 | 0.377 | 0.287 |
|  |  | MSE | 0.200 | 0.205 | 0.195 | 0.189 | - | 0.180 | 0.187 | 0.175 | 0.287 | 0.207 | 0.193 | 0.286 | 0.203 |
|  | 192 | MAE | 0.315 | 0.318 | 0.303 | 0.300 | 0.309 | 0.309 | 0.309 | 0.302 | 0.492 | 0.364 | 0.362 | 0.445 | 0.328 |
| ETTm2 |  | MSE | 0.269 | 0.261 | 0.247 | 0.247 | - | 0.250 | 0.249 | 0.241 | 0.414 | 0.290 | 0.284 | 0.399 | 0.269 |
|  | 336 | MAE | 0.319 | 0.355 | 0.333 | 0.334 | 0.349 | 0.348 | 0.351 | 0.343 | 0.542 | 0.422 | 0.427 | 0.591 | 0.366 |
|  |  | MSE | 0.264 | 0.319 | 0.291 | 0.295 | - | 0.311 | 0.321 | 0.305 | 0.597 | 0.377 | 0.369 | 0.637 | 0.325 |
|  | 720 | MAE | 0.374 | 0.410 | 0.377 | 0.386 | 0.415 | 0.407 | 0.403 | 0.400 | 1.042 | 0.524 | 0.522 | 0.735 | 0.415 |
|  |  | MSE | 0.354 | 0.415 | 0.355 | 0.372 | - | 0.412 | 0.408 | 0.402 | 1.730 | 0.558 | 0.554 | 0.960 | 0.421 |
|  | 96 | MAE | 0.212 | 0.299 | 0.248 | 0.242 | - | 0.240 | 0.272 | 0.285 | 0.314 | 0.329 | 0.282 | 0.345 | 0.308 |
|  |  | MSE | 0.124 | 0.205 | 0.158 | 0.152 | - | 0.148 | 0.168 | 0.195 | 0.219 | 0.237 | 0.197 | 0.247 | 0.193 |
|  | 192 | MAE | 0.232 | 0.310 | 0.263 | 0.259 | - | 0.253 | 0.289 | 0.289 | 0.322 | 0.330 | 0.285 | 0.355 | 0.315 |
| Electricity |  | MSE | 0.138 | 0.220 | 0.174 | 0.171 | - | 0.162 | 0.184 | 0.199 | 0.231 | 0.236 | 0.196 | 0.257 | 0.201 |
|  | 336 | MAE | 0.249 | 0.323 | 0.278 | 0.278 | - | 0.269 | 0.300 | 0.305 | 0.337 | 0.344 | 0.301 | 0.369 | 0.329 |
|  |  | MSE | 0.155 | 0.236 | 0.191 | 0.192 | - | 0.178 | 0.198 | 0.215 | 0.246 | 0.249 | 0.209 | 0.269 | 0.214 |
|  | 720 | MAE | 0.291 | 0.347 | 0.307 | 0.313 | - | 0.317 | 0.320 | 0.337 | 0.363 | 0.373 | 0.333 | 0.390 | 0.355 |
|  |  | MSE | 0.211 | 0.270 | 0.229 | 0.236 | - | 0.225 | 0.220 | 0.256 | 0.280 | 0.284 | 0.245 | 0.299 | 0.246 |
|  | 96 | MAE | 0.223 | 0.212 | 0.203 | 0.208 | - | 0.214 | 0.220 | 0.218 | 0.230 | 0.261 | 0.255 | 0.306 | 0.296 |
|  |  | MSE | 0.180 | 0.173 | 0.167 | 0.177 | - | 0.174 | 0.172 | 0.177 | 0.158 | 0.202 | 0.196 | 0.221 | 0.217 |
|  | 192 | MAE | 0.267 | 0.250 | 0.241 | 0.249 | - | 0.254 | 0.261 | 0.259 | 0.277 | 0.298 | 0.296 | 0.340 | 0.336 |
| Weather |  | MSE | 0.235 | 0.216 | 0.209 | 0.219 | - | 0.221 | 0.219 | 0.225 | 0.206 | 0.242 | 0.237 | 0.261 | 0.276 |
|  | 336 | MAE | 0.291 | 0.282 | 0.276 | 0.292 | - | 0.296 | 0.306 | 0.297 | 0.335 | 0.335 | 0.335 | 0.378 | 0.380 |
|  |  | MSE | 0.252 | 0.260 | 0.256 | 0.277 | - | 0.278 | 0.280 | 0.278 | 0.272 | 0.287 | 0.283 | 0.309 | 0.339 |
|  | 720 | MAE | 0.356 | 0.322 | 0.323 | 0.350 | - | 0.349 | 0.359 | 0.348 | 0.418 | 0.386 | 0.381 | 0.427 | 0.428 |
|  |  | MSE | 0.356 | 0.320 | 0.321 | 0.365 | - | 0.358 | 0.365 | 0.354 | 0.398 | 0.351 | 0.345 | 0.377 | 0.403 |

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: See above.
soundness: 1
strengths: * The application to time series forecasting in observability is intriguing and valuable. A step toward unified benchmarking and a useful dataset is commendable.
* The design of proportional factorized space-time attention potentially offers an expressive and efficient backbone for multivariate time series modeling.
summary: This paper proposes a novel Transformer-based forecasting model tailored to observability, focusing on metric time series from system monitoring. Key contributions include 1/ a novel transformer block integrating one channel-mixing attention layer followed by multiple blocks of time-wise attention; and 2/ an output head employing a mixture of Student-t distributions. Using a decoder-only architecture, the authors train the model on proprietary observability data, supplemented by open-source and synthetic datasets. The model’s effectiveness is evaluated on both public datasets and a private observability benchmark.
weaknesses: * The work lacks clear motivation; the authors’ objective is difficult to discern.
  * If the aim is a superior observability model, why opt for a foundation model? Any deep forecasting backbone could be trained on similar data for comparison. Instead, only zero-shot models are evaluated on the observability benchmark, where presumably the proposed model was pre-trained.
  * If the goal is to demonstrate generalization of the model trained on observability data to other domains, more comprehensive benchmarks are necessary (e.g., Chronos benchmark; LSF is overly simplistic).
  * If the focus is on the efficiency of the factorized space-time attention mechanism, standard task-specific training regimes would suffice instead of pre-training and zero-shot evaluation.
 * If domain-specific training data is a contribution, details are needed on the pre-training corpus construction, ablation studies, and novelty of data selection.
* The paper’s presentation could be significantly improved in terms of completeness, clarity, and structure.
* The technical contributions of space-time attention and Student-t mixture heads are of limited novelty and provide marginal improvement.
  * Experimental results are insufficient to substantiate the claimed contributions:
  * There is no empirical evidence of the computational efficiency of the proposed attention module.
  * For the observability benchmark, only zero-shot model results are presented. Given the extensive pre-training on observability data, fine-tuned versions of the baseline models would provide a fairer comparison. Additional baselines (e.g., classical models, task-specific deep learning models such as TFT, TiDE) should be included.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - In Experiment 4.1, several baselines are missing, such as Chronos [2] and Timer [3] in the zero-shot setting, and TimeMixer [4] in the full-shot setting. Similarly, Experiment 4.2 lacks Timer as a baseline.

- What are the historical context window lengths for the baseline models in Experiment 4.1? In Experiment 4.2, what are the historical context windows and prediction lengths for Toto and the other baselines?

- The sampling strategy for Chronos and Moirai models in Experiment 4.2 is unclear. Why is the original sampling strategy from the referenced papers followed? Additionally, is a sampling quantity of 20 fair for Chronos?

- What is the sampling quantity for Toto in Experiment 4.2? Is it also set to 200? How would increasing the sampling quantity affect the baselines' performance?

- In Line 303, it is stated that "for more precise tails, we can choose to spend more computation on sampling"; however, this aspect is not demonstrated in the experiments. 


[1] Woo, Gerald, et al. "Unified training of universal time series forecasting transformers." arXiv preprint arXiv:2402.02592 (2024).

[2] Ansari, Abdul Fatir, et al. "Chronos: Learning the language of time series." arXiv preprint arXiv:2403.07815 (2024).

[3] Liu, Yong, et al. "Timer: Transformers for time series analysis at scale." arXiv preprint arXiv:2402.02368 (2024).

[4] Wang, Shiyu, et al. "Timemixer: Decomposable multiscale mixing for time series forecasting." arXiv preprint arXiv:2405.14616 (2024).
soundness: 3
strengths: **S1:** The writing style is concise, and the methodology is well-organized and clearly described.

**S2:** The paper introduces a new dataset, a large dataset of proprietary observability metrics, which contains statistical characteristics absent from existing datasets.

**S3:** The study employs a dual perspective in the attention mechanism (space-wise and time-wise), and uses this model for pre-training a foundation model to validate its performance.
summary: This paper introduces Toto, a foundation model designed for time series forecasting, with a specific emphasis on observability metrics. Toto features two core innovations: a proportional factorized attention mechanism, which groups multivariate time series features to reduce computational overhead, and a Student-T mixture model head, which improves the model’s ability to capture complex time series dynamics beyond what traditional Gaussian models offer. Trained on one trillion data points, including a large dataset of proprietary observability metrics, Toto achieves state-of-the-art zero-shot performance in both general benchmarks and observability-specific tasks.
weaknesses: **W1:** The paper’s two primary contributions are the novel observability data and the newly designed foundation model. However, neither the data nor the model are publicly available, raising concerns about reproducibility and broader applicability.

**W2:** The technical novelty is limited; for instance, the probabilistic forecasting using the Student-T mixture model (SMM) is an extension, as the Student-T distribution as a prediction head has already been proposed [1].

**W3:** In Table 1, the baseline results are cited directly from Woo et al. (2024) and Das et al. (2024). However, the current information provided does not allow for adequate evaluation.

**W4:** The experiments lack essential details, such as the historical context length in Section 4.1, the sampling strategy and quantity, and the configurations for historical context length and prediction length in Section 4.2.

**W5:** The experimental evaluation is insufficient, omitting several important baselines. Additionally, the proposed method lacks a parameter sensitivity analysis.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: I have no further questions.
soundness: 2
strengths: The main strength of the paper of the paper is that it introduces a foundation model specifically for the observability domain, which is an important practical problem. Using a foundation model that can be rolled out broadly on an entire system without local model fitting or retraining is a sound approach to address this problem [**Originality and Significance**]. The paper is clearly written and easy to follow [**Clarity**].
summary: The this paper introduces a time series foundation model for observability data (called Toto). The model utilizes a factorized-space time transformer variant to handle multivariate time series and a mixture student-t head for probabilistic forecasting. The model is trained on a large dataset of proprietary observability data in addition to open source, and synthetic data. The authors evaluate their method on a long sequence forecasting benchmark and on the observability dataset.
weaknesses: The two main weaknesses of this manuscript are that the architectural contributions are incremental and the validation on the long sequence forecasting benchmark requires either revision or clarification. 

**Incremental contribution**: I would argue that the main contribution of this paper is the observability dataset. A high quality dataset for this domain would be helpful to further develop foundation models for observability specifically and for time series foundation models more generally. However, since the data in this work is proprietary this will not benefit the ICLR or larger science community. The other contributions (proportional factorized space-time attention) and the mixture student-t probabilistic output distribution are incremental. The proposed transformer architecture is a minor modification from Arnab et al. ICCV 2021 (https://arxiv.org/abs/2103.15691) with a configurable parameter to adjust the ratio of space and time attention layers. The mixture student-t distribution is a minor modification of the mixture distribution output in Woo et al. ICML 2024 (https://arxiv.org/abs/2402.02592). GluonTS (Alexandrov et al., JMLR 2020) also provides an implementation of a generic mixture distribution that can be instantiated as a student-t mixture. 

Thus, I would regard the contributions as too incremental to warrant publication in ICLR. 

**Evaluation on Long Sequence Forecasting (LSF) Dataset**: Another concern is on the evaluation on the LSF dataset. The setup in this manuscript uses a rolling evaluation with a stride 512, which is inconsistent with the results taken from earlier work that is included in this evaluation (Woo et al. ICML 2024). Specifically, the original setup of this dataset uses a stride of 1 (Zhou et al., AAAI 2021: https://arxiv.org/abs/2012.07436). Hence, the results of Toto and the other results that are taken from earlier manuscripts are not comparable. 

I would kindly ask the reviewer to perform the experiments under the same condition as the manuscripts where the experimental results are reported from. I will consider raising my score if the experiments are updated or it is clarified that the experiments are indeed performed under the same conditions.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 3
questions: - What is meant by "... in sliding windows with stride 512 ..." in line 406? Can you share how many windows you end up with for different prediction lengths? (Let's take ETTh1 as an example)
- I am willing to increase my score if the authors expand their evaluation benchmark. A reasonably simple way to do this would be to report results on the benchmark presented in the Chronos paper. As per my knowledge, the code for this benchmark is available on Github.
soundness: 2
strengths: The paper is well written and provides an insightful discussion on the specific case of observability time series data. Some design choices such as the factorized attention mechanism and mixture of student's-t head have been reasonably justified.
summary: The paper proposes a pretrained time series forecasting model with a focus on observability time series data. Toto is a decoder-only patch-based model trained on large amounts of time series data to predict the next patch of time series. Due to its focus on observability, the training corpus includes a large proportion of proprietary observability time series. Toto uses a mixture of Student's-t distribution head to enable forecasting of heavy-tailed and non-stationary data. It includes a factorized "space-time" attention mechanism allowing it to consume multivariate time series and generate forecasts jointly. Experimental results have been reported on some datasets from the long-term forecasting benchmark and a proprietary observability benchmark.
weaknesses: - The paper proposes another pretrained model for time series such as Chronos, TimesFM and Moirai. When compared with these existing models, the technical contribution of Toto is marginal. The mixture of distributions idea is not new and has been studied in forecasting literature before, most recently even in the context of pretrained models (Moirai). Furthermore, the alternating spatial and temporal attention blocks idea is incremental. 

- In the absence of strong methodological contributions, the empirical investigation needs to be strong to meet the acceptance bar. Unfortunately, the experiments reported in this paper are sub par. Concretely, they have the following limitations.
    - The breadth of experiments is severely limited for a purported "foundation model". In the benchmark on publicly available datasets (Table 1), only 6 datasets have been studied, 5 of which belong to the same domain with 4 being essentially the same dataset (ETTh1, ETTh2, ETTm1, ETTm2). This benchmark is limited even for a regular time series paper. For a pretrained model, this is not sufficient to draw any conclusions. Please check the Chronos and/or TimesFM papers for a better evaluation benchmark.
    - Even on these 6 datasets, it appears that the numbers reported for Toto are not from the same setting as other methods. The numbers for baselines such as Moirai and TimesFM have been reproduced from their papers but the rolling evaluation for Toto appears to have been conducted with a stride of 512 which is not the same setting as in the baselines. Due to the fewer slices being evaluated, the numbers could be inflated. Please clarify if I have misunderstood the setting. See questions for details.
    - The second benchmark for observability time series appears to have been conducted on a proprietary dataset which makes it difficult to verify for the community. Will the dataset or the pretrained model weights be released? In either case, code/model could have been made available to the reviewers. Regardless of reproducibility concerns, the percentage metrics reported are often discouraged in the literature (see [1] for details). Furthermore, it is unclear what the key claim of this experiment is. Since Toto has been trained on large amounts of data from the same observability domain (75% of 1T time series), it's not surprising that it would perform better than other "zero-shot" models. 
    - Beyond reporting the numbers, the paper does not conduct any serious investigation of the models or the results.

- Minor: It's not clear to me what a "foundation model" for time series forecasting in observability really means. With such domain-specific qualifier, the word "foundation" loses its meaning and it would be better replaced with something like "pretrained". 

[1] Hyndman and Koehler (2006). Another look at measures of forecast accuracy.