Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of
  Experts
Authors: name: Xiaoming Shi; name: Shiyu Wang; name: Yuqi Nie; name: Dianqi Li; name: Zhou Ye; name: Qingsong Wen; name: Ming Jin
Published: 2024-09-24 12:42:18

Abstract: Deep learning for time series forecasting has seen significant advancements
over the past decades. However, despite the success of large-scale pre-training
in language and vision domains, pre-trained time series models remain limited
in scale and operate at a high cost, hindering the development of larger
capable forecasting models in real-world applications. In response, we
introduce Time-MoE, a scalable and unified architecture designed to pre-train
larger, more capable forecasting foundation models while reducing inference
costs. By leveraging a sparse mixture-of-experts (MoE) design, Time-MoE
enhances computational efficiency by activating only a subset of networks for
each prediction, reducing computational load while maintaining high model
capacity. This allows Time-MoE to scale effectively without a corresponding
increase in inference costs. Time-MoE comprises a family of decoder-only
transformer models that operate in an auto-regressive manner and support
flexible forecasting horizons with varying input context lengths. We
pre-trained these models on our newly introduced large-scale data Time-300B,
which spans over 9 domains and encompassing over 300 billion time points. For
the first time, we scaled a time series foundation model up to 2.4 billion
parameters, achieving significantly improved forecasting precision. Our results
validate the applicability of scaling laws for training tokens and model size
in the context of time series forecasting. Compared to dense models with the
same number of activated parameters or equivalent computation budgets, our
models consistently outperform them by large margin. These advancements
position Time-MoE as a state-of-the-art solution for tackling real-world time
series forecasting challenges with superior capability, efficiency, and
flexibility.