Title: Synthetic Series-Symbol Data Generation for Time Series Foundation Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Time Series Analysis

Foundation Model

Data Scarcity
Abstract: Foundation models for time series analysis (TSA) have attracted significant attention. However, challenges such as training data scarcity and imbalance continue to hinder their development. Inspired by complex dynamic system theories, we design a series-symbol data generation mechanism, enabling the unrestricted creation of high-quality time series data paired with corresponding symbolic expressions. To leverage series-symbol data pairs with strong correlations, we develop SymTime, a pre-trained foundation model for enhancing time series representation using symbolic information. SymTime demonstrates competitive performance across five major TSA tasks when fine-tunes with downstream tasks, rivaling foundation models pre-trained on real-world datasets. This approach underscores the potential of series-symbol data generation and pretraining mechanisms in overcoming data scarcity and enhancing task performance. The code is available at https://github.com/wwhenxuan/SymTime.


Peer reviews:
comments: SUMMARY:
This paper introduces a novel mechanism for generating unlimited synthetic time series data paired with their underlying symbolic (mathematical) expressions—based on complex dynamical systems theory—to overcome real-world data scarcity and imbalance. Leveraging this dual-modality dataset, the authors pretrain SymTime, a foundation model that fuses masked time-series and symbolic-language modeling with contrastive learning, achieving state-of-the-art performance across five major time-series analysis tasks.

STRENGTHS AND WEAKNESSES:
**Strengths**
- I very much like the careful design of the S² generator and the theoretical background provided around Takens’ theorem.
- The idea of synthetically generating time-series seems with their proposed approach seems to be highly effective. 
- I like the broad experimentation carried out by the authors, and the consistent gains that their method seems to achieve. 
- The ablations performed by the authors support the strength of each of the changes introduced. 

**Weaknesses**

- If I understood correctly, there is no comparison against real-data pre-training with the same architecture. This would have been a valuable addition to the paper. 
- There are no standard deviation results in the tables presented.

QUESTIONS:
- Would the synthetic data generation process be able to reproduce phenomena observed in real-world domain specific time-series datasets, e.g. volatility clustering in finance?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article focuses on improving model performance through data augmentation and the use of foundation models. Specifically, they propose a series-symbol data generation mechanism that alleviates issues of scarcity and imbalance in data. Additionally, they design a time series foundation model consisting of two encoders, where one encoder processes time series data and the other encoder processes symbol inputs. These two encoders are trained using contrastive learning to enhance the representation ability of time series. The approach has achieved good results in multiple time series tasks.\nExperiments verify the ability of data augmentation in terms of stability, predictability, frequency domain analysis, and trendiness. The comparative experiments validate the performance of the foundation model across multiple downstream tasks.\n\nSTRENGTHS AND WEAKNESSES:\nS1: The author proposes a data generation mechanism that addresses issues of data scarcity and imbalance, and experimental results show that it can alleviate these problems.\nS2: The pre-trained model demonstrates strong capabilities in multiple time-series tasks.\n\nW1: Lack of implementation details for the symbol data generation mechanism.\nW2: No explanation was given regarding the complexity issue of the symbol data generation mechanism when dealing with long series, as well as whether the symbol selection strategy needs to be adapted based on specific tasks.\n\nQUESTIONS:\n1. Does the symbol data generation mechanism have a complexity issue when dealing with long or non-stationary time series data?\n2. Is the symbol generation mechanism required to be adjusted according to downstream tasks? Please provide detailed information on how symbols are selected.\n3. How were the 6-layer Transformer encoder for handling time series data and the 6-layer DistilBERT for handling symbol data chosen? Does layer depth have any impact on performance?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach to address data scarcity and imbalance in time series analysis. The authors propose a series-symbol data generation pipeline. Inspired by complex dynamic system theories, the authors generate high-quality univariate time series data paired with corresponding symbolic expressions, enabling the creation of diverse time series datasets. The authors present SymTime, a pre-trained encoder-only time series foundation model, leveraging contrastive learning and masked modeling. It shows notable scalability and generalization performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Quality**: The submission is technically sound, with a strong theoretical foundation and extensive experimental validation. SymTime’s architecture is well-motivated, leveraging contrastive learning and masked modeling on numerical and symbolic tokens. About this architecture option, I wonder if the author has ever tried generative modeling on a decoder-only Transformer like TimesFM and Timer. The benchmarks include five major TSA tasks (forecasting, classification, imputation, anomaly detection). Although it is comprehensive, I'd like to raise my score if the model can be evaluated more public leaderboard, such as GIFT-Eval and FEV.\n\n**Clarity**: The mathematical formulation (e.g., tree-based symbolic sampling, ARMA processes) is well-explained. The paper is generally well-written and organized, but there are some typos that could be improved (e.g., \"prmutation\" -> \"permutation\"; \"pretrain\" -> \"pre-train\").\n\n**Significance**: The paper provides new insights for (1) pre-training generalizable time series foundation models, (2) inspiring new benchmarks for time-series pre-training. As symbolically generated time-series data can effectively replace or augment real-world data for pre-training, it can be beneficial for the community if the author releases the data generation framework. \n\n**Originality**: Different from previous works, which are pre-trained solely on synthetic data or partially synthetic data. The proposed symbolic framework can provide aligned synthetic data and symbolic formulation. In addition to enabling contrastive learning, it is a promising direction is to take symbolic formulation as the aligned textual description for multimodal time series models, which can help current models better understand the underlying dynamics of time series. In terms of the model design, the contributions of SymTime mainly focus on pre-training tasks with contrastive learning. The structure is similar to previous works (such as moment, moirai).\n\nQUESTIONS:\n1. Under the proposed symbolic system, what is the maximum scale of synthetic data that can be constructed?\n2. Are there any suggestions to overcome the diminishing marginal returns when pre-training on an ever-larger scale of synthetic data?\n3. What is the range of different statistical metrics in Figure 4? The generated data has relatively low diversity in some metrics (PE, FFT Mean, Seasonality). How does pre-training based on metrics of different degrees of diversity affect the performance of downstream tasks?\n4. Can the proposed method generate multivariate time series? It may also provide insights about how to define how \"multivariate\" a set of time series is.\n\nLIMITATIONS:\nPlease discuss whether omitted operators (e.g., exponent and higher power) limit the generation of certain dynamics.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a synthetic data generation method for training time series foundation models. At the core of the synthetic data generation method is a composition of symbolic expressions that gets randomly generated through sampling (sampling input/output dimensions, operators, affine transformations, etc.). The input to these symbolic expressions are time series sampled from mixed distributions and ARMA processes. The transformation of the symbolic expression then forms the synthetic time series. The paper also introduces SymTime, a time series foundation model which uses patched encodings, cross-entropy loss, and additional contrastive learning and momentum distillation terms.  \n\nThe manuscript compares the generated time series dataset features with existing datasets from Monash to evaluate whether the generated time series are realistic. The authors then evaluate their models on forecasting, imputation, classification, and anomaly detection tasks. Finally, the paper conducts ablations on the pre-training objectives and analyses the obtained time series and symbol representations.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper introduces an interesting idea for synthetic time series generation based on sampling of symbolic expressions. This offers a flexible way of generating synthetic time series. As the authors state in the paper, pretraining time series models requires synthetic data due to a relative scarcity (compared the text data) and therefore novel ideas for synthetic data generation are needed. Therefore, the explored topic in the paper has significance and is original. Another strength of the paper is that the thorough evaluation of the model on several downstream tasks (forecasting, imputation, classification, anomaly detection) along with ablations and analysis of the generated synthetic time series and SymTime representations. \n\nOne weakness of the paper is the clarity of writing. I found several parts of the paper a bit confusing and even after reading several times I am still not sure how they work. While I understood the data generation process after some re-reads, I still do not understand the contrastive learning part of SymTime. For example, how are positive and negative pairs sampled? I understand that the authors covered a lot of ground and had to accommodate this in the space constraints, but I believe a clearer explanation of the core concepts would strengthen the paper. \n\nAnother major weakness is the specific setup of the forecasting results and the claims that are made on SymTime’s performance in forecasting. The paper for the long-range forecasting dataset, a fixed lookback window of 96 is used for most comparing models. This makes the conclusion that SymTime outperforms the other baseline models on this dataset invalid because it only does so under this (artificial) conditions that make the baseline methods perform worse. For example, the MSE/MAE windows for PatchTST are much lower in its original paper on these datasets, if the proposed lookback window of 336 or 512 time steps is used. Thus, SymTime does not reduce the absolute error on these datasets compared to existing methods. I have a similar criticism on the short-term datasets. Here again, while the results for SymTime are good, the OWA for the state-of-the-art method is also lower (see Gasthaus et al. (2019), \"Spline Quantile Function RNNs\") than the error of the methods presented here. \n\nAdditionally, while the long-term forecasting benchmark has been used extensively in time series forecasting, the value of this benchmark to still measure progress in this field is questionable. This has been explored in recent position papers (https://arxiv.org/abs/2502.14045) and in the NeurIPS 2024 time series workshop (https://neurips.cc/virtual/2024/workshop/84712#wse-detail-108471, see Christoph Bergmeir's talk). I think it is important for the time series forecasting filed to move towards more extensive evaluation which is proposed in several recent paper (Gifteval: https://arxiv.org/abs/2410.10393; FEV (from the Chronos paper): https://openreview.net/forum?id=gerNCVqqtR).\n\nI understand and appreciate that this work also considers other tasks (like classification and anomaly detection, where I have less insight on the SOTA). Therefore, the paper provides additional experimental evidence. However, I would ask the authors to at least run some of the baseline methods with their recommended settings (specifically PatchTST with 336/512 context length) make sure their comparison captures the error that SOTA models are capable of and therefore an accurate reflection of the SOTA.\n\nQUESTIONS:\nHow are positive/negative pairs selected in the contrastive learning step?\n\nLIMITATIONS:\nThe authors appropriately address limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}