Approximately Equivariant Recurrent Generative Models for Quasi-Periodic Time Series with a Progressive Training Scheme
Authors: Ruwen Fulek
Markus Lange-Hegermann
Venue: Accepted by TMLR ()

Abstract: We present a simple yet effective generative model for time series, based on a Recurrent Variational Autoencoder that we refer to as RVAE-ST. Recurrent layers often struggle with unstable optimization and poor convergence when modeling long sequences. To address
these limitations, we introduce a progressive training scheme that gradually increases the sequence length, stabilizing optimization and enabling consistent learning over extended horizons. By composing known components into a recurrent, approximately time-shift-equivariant topology, our model introduces an inductive bias that aligns with the structure of quasi-periodic and nearly stationary time series. Across several benchmark datasets, RVAE-ST matches or surpasses state-of-the-art generative models, particularly on quasi-periodic data, while remaining competitive on more irregular signals. Performance is evaluated through ELBO, Fréchet Distance, discriminative metrics, and visualizations of the learned latent embeddings.

Published in Transactions on Machine Learning Research (03/2026)
Approximately Equivariant Recurrent Generative Models for Quasi-Periodic Time Series with a Progressive Training Scheme
Ruwen Fulek
ruwen.fulekBUt-owwl.cc
Institut für industrielle Informattiontechhikk Technische Hochschule Ostwestfalen- Lippe 32657 Lemgo
Markus Lange- Hegermann Institul für industrielle Informaationstechrik Technische Hochschule Ostwestfulen-Lippe 32657 Lemgo
markus..ange--heeermmmnt
Reviewed on OpenReview: https: openreview. net/ forum? id= KHRSEBBG32
Abstract
We present simple yet effective generative model for time series, based Recurrent Variational Autoencoder that refer to EEQ-RVAE-ST. Recurrent layers often struggle with unstable omiimization and poor convergence when modeling long sequences. To address these limitations, we introduce a training scheme that subsequently increases the sequence length, stabilizing optimization and enabling consistent lcarning over extended horizons. By composing known components inlo a recurrent, approximately time-shiil--qquivariadd topology, our model introduces an inductive bias that aligns with the structure of quasi- periodic and nearly stationary time series. Across several benchmark datasets, AEQ-RVAE- ST matches surpasses state-of-the-art generative models, particularly quasi-periodic data, while remaining competitive more irregular signals. Performance is evaluated through ELBO, Fréchet Distance, discriminative metries, and visualizations of the learned latent embeckings.
1 Introduction
Time series data, particularly sensor data, plays crucial role in science, industry, energy, and health. With the increasing digitization of companies and other institutions, the demand for advanced methods to handle and analyze time series sensor data continues to grow. Sensor data often exhibits distinct characteristics: it is frequently multivariate, capturing several measurements simultaneously, and may involve high temporal resolutions, where certain anomalies or patterns of interest only become detectable in sufficiently long se- quences. Furthermore, such data commonly displays quasi-periodic behavior, reflecting repetitive patterns influenced by the underlying processes. For generative models, this raises the challenge of how to embed inductive biases that emphasize relative temporal dynamics over absolute time, encouraging the model to treat repeating structures consistently regardless of their position in the sequence. These unique properties present both opportunities and challenges in the development of methods for efficient data synthesis and analysis, which are essential for a wide range of applications.
Throughout this paper, we use the term quasi-periodic the applied anomaly-detection sense, referring to time series that exhibit repetitive motifs recurring with imperfect regularity due to variable cycle lengths, phase jitter, drift, noise, missing cycles, occasional anomalies. This usage is consistent with the anomaly-
detection and time-series segmentation literature Yang et al. 2025 Liu et al. 2022 Zangrando et al. 2022

---

Published in Transactions on Machine Learning Research (03/2026)
Tang et al. 2023 and differs from the classical mathematical notion of quasi- which typically refers to structured signals generated by finite number of incommensurate frequencies.
Time series data analysis spans tasks such as forecasting (Siami-Namini et al. 2019), imputation (Tashiro
ct al. 2021 Luo et al.) 2018), anomaly detection (Hammerbacher et al.) 2021). and data generation. Of
these, data generation stands out as the most general task, as advances in generative methods often yield improvements across the entire spectrum of time series applications (Murphy 2022).
Recurrent neural networks, particularly Long Short-Term Memory (LSTM) networks (Hochreiter & Schmid- huber 1997). are well-known for their ability to model temporal dynamics and capture dependencies in sequential data. However, their effectiveness tends to diminish with increasing sequence length, as main- laining long-term dependencies can become challenging (Zhu et. al. 2023) where in contrast, convolutional neural networks (CNNs) {LeCun et al. 1998) demonstrate superior scalability for longer sequences (Bai ct al.] 2018). For instance, TimeGAN (Yoon et al.) 2019) represents a stato-of-the-art approach for gen- erating synthetic time series data, partieularly eflective for short sequence lengths. In its original paper, samples with sequence lengths of = 24, showcasing limitations of LSTM-based architectures. By contrast, model like WaveGAN (Donahue et al. 2019), which is built on convolutional architecture, is trained on significantly longer sequence lengths, with / = 16384 at minimum. This contrast highlights the fundamental differences and capabilities between recurrent and convolut ional networks.
TimeGAN demonstrates its capabilities
The limitations of LSTMs in modeling long-term dependencies are not restricted to time series data but also impact their performance in other domains, such natural language processing (NLP). Early applications of attention mechanisms integrated with recurrent neural networks like LSTMs (Bahdanau 2014) have largely been replaced by Transformer architectures (Vaswani et al. 2017), which excel in data-rich tasks due to their parallel processing capabilities and expressive attention mechanisms. While Transformer architectures have shown exceptional results in NLP (Radford et al. 2019), their application to time series data remains chal- lenging. This is due in part to the self-attention mechanism's 's quadratic scaling in memory and computation with sequence length (Katharopoulos et al. 2020), which makes them less practical for very long sequences. Additionally, the inductive bias of Transformers differs from that of recurrent models: Transformers rely on positional encodings to model temporal structure, whereas recurrent architectures such as LSTMs process data sequentially by design, which inherently embeds a sense of temporal order into the model dynamics. This sequential processing makes recurrent models particularly well-suited for long, approximately stationary time series, where preserving temporal continuity over extended horizons can be highly beneficial.
Among the primary approaches for generative modeling of time series, three dominant frameworks have emerged: Generative Adversarial Networks (GANs) (Goodfellow et al., 2020), Variational Autoencoders (VAEs) (Kingnna & Welling) 2014; Fabius & Van Amersfoort, 2014), and, more recently, Diffusion Models (Ho et al. 2020). Diffusion Models have demonstrated impressive capabilities in modeling complex data distributions, but their significant computational demands, high latency, and complexity make them less practical for many applications (Yang et al. 2024). Morcover, in terms of practical applications, there are often constraints in both time and computational resources, which limit the feasibility of performing extensive fine-tuning for each individual dataset. A general, well-performing approach that is both simple and efficient is therefore more desirable. In this context, VAEs still stand out for their simplicity and direct approach to probabilistic modeling. In our work, we focus on VAEs and propose a method for training VAEs with recurrent layers to handle longer sequence lengths. We argue that VAFs are particularly suited for generation of time series data, as they explicitly learn the underlying data distribution, making them robust, interpretable, and straightforward to implement.
Our major contributions are:
We introduce a novel combination of inductive biases, network topology, and training scheme in a recurrent variational autoencoder architecture. Our model integrates approximate time-shift equiv- ariance into recurrent structure, encouraging invariance to absolute time and thereby providing an inductive bias toward quasi-periodic time series. Unlike existing recurrent or convolutional genera- tive models, our architecture maintains fixed number of parameters, independent of the sequence
2

---

Published in Transactions on Machine Learning Research (03/2026)
length. We further analyze this behavior through the Echo State Property (ESP), which serves diagnostic lens to quantify how strongly the model forgets arbitrary initializations and aligns its dynamics with the input structure.
We propose a simple yet effective training scheme that subsequently increases the sequence length during training, leveraging the eequence-length--ivarieee parameterization of our model. We refer to the combination our architecture with this training scheme as AEQ-RVAE-ST. This scheme miti- gates the typical limitations of recurrent layers in capturing long-range dependencies, particularly suited for quasi-periodic datasets, and contributes significantly our model's performance.
We conduct extensive experiments on five benchmark datasets and compare our method against a broad range of strong baselines, including models based on GANs, VAEs, diffusion processes, convolutions, and Transformers. This diverse set covers the most prominent architectural families in time-series generation and ensures fair and comprehensive evaluation.
To evaluate generative quality of the long generated sequences, we employ comprehensive set of evaluation metrics, including Contextualized Fréchet Inception Distance (Context FID), Discrimi- native Score, and visualizations via PCA and t-SNE.
Our implementation, including preprocessing and model training scripts, is available in branches: main inductive bias.
sine
556
532
338
532
W M M M yrrk
(a) Original data
(b) Diffusion-TS
(c) AEQ (ours)
4.1 with sequence length | 1000: (a) original data, (c) generated by our model. Each row corresponds to a different channel. Our model (c) faithfully reproduces key signal characteristios across all channels, including the pronounced voltage peak volatility (row 1), the sawtooth-like DC-bus pattern (row 2), and both the high- and low-frequency components the stator current (row 3). Diffusion-TS (b) captures the general frequency structure but loses fino-grained details such the sawtooth shape and high-frequency content.
Figure 1: Three excerpts of the electric motor dataset
(b) generated by Diffusion-TS Yuuan & Qiao 2024
2
Related Work
2.1 Deep Generative Models for Time Series
Time-series generation has been explored across various deep generative paradigms, including GANs, VAEs, Transformers, and diffusion models. Early approaches focused on recurrent structures: C-RNN-GAN Mo-
gren 2016)) used LSTM-based generators and discriminators, while RCGAN (Esteban et al. 2017 introduced label- abel-conditioning for medical time series. TimeGAN (Yoon al.] 2019) combined adversarial training, su- pervised learning, and temporal embedding module to better capture temporal dynamics. Around the
3

---

Published in Transactions on Machine Learning Research (03/2026)
same time, WaveGAN Donahue et al. 2019 introduced convolutional GAN architecture for raw audio synthesis, illustrating that convolutional models can also be effective for generative tasks in the time domain. TimeVAE (Desai et al. 2021b) further explored this direction by proposing a convolutional variational au- tocncoder tailored to time-serics data. PSA-GAN (Paul et al.) 2022) employed progressive growing (Karras et al. 2018), incrementally increasing temporal resolution during training by adding blocks composed of convolution and residual self-attention to both the generator and discriminator. This fundamentally differs from our approach, which extends sequence length rather than resolution.
Recent advances in time-series generation have explored diffusion- and hybrid Transformer architec- tures. Diffusion-TS (Yuan & Qiao 2024) introduces a denoising diffusion probabilistic model (DDPM) tailored for multivariate time series generation. It employs an encoder-decoder Transformer architecture with disentangled temporal representations, incorporating trend and seasonal components through inter- pretable layers. Unlike traditional DDPMs, Diffusion-TS reconstructs the sample directly at each diffusion step and integrates a courier-basee loss term. Timo-Transtine Transformer (Liu al., 2024) presents hybrid ar- chitecture combining Temporal Convolutional Networks (TCNs) and Transformers in parallel design to simultaneously capture local and global features. A bidirectional cross-attention mechanism fuses these fea- tures within an adversarial autocncoder framework (Makhzani et al.) 2016). This design aims to improve the quality of generated time series by effectively modeling complex temporal dependencies.
A common limitation across all these approaches their focus on relatively short sequence lengths. Many models, including TimeGAN, TimeVAE, and Time-Transfone Transformer, are evaluated at I = 24. Only the transformer-base Diffusion-TSS and PSA-GAN extend this slightly, with ablations up to 256, leaving the performance significantly longer sequences largely unexplored.
2.2 Recurrent Variational Autoencoders
The Recurrent Variational Autoencoder (RVAF) was introduced by Fabius & Van Amersfoort (014)) com- bining variational inference with basic RNNs for sequence modeling. In this architecture, the latent space is connected to the decoder via a linear layer, and the sequence is reconstructed by applying a sigmoid acti- vation to each RNN hidden state We build on this framework by replacing the basic RNNs with LSTMs (or GRUs) and using a repeat-voctor mechanism that injects the same latent vector every time step of the decoder. This design encourages the latent code to encode global sequence properties, while the LSTM handles temporal dependencies. Instead of a sigmoid, we apply time-distributed linear layer, preserving approximate time-translation equivariance (see Section 3.1
Unlike dynamic VAEs (dVAE) that use a sequence of latent variables to increase flexibility (Girin et al., 2021), we opt for a single latent vector of fixed size across the entire sequence. This choice reflects our focus on the inductive bias of translational equivariance and stationarity, where the latent code is mcant to capture global properties of the sequence while allowing the decoder to model local temporal dynamics. This distinction means that, unlike in dVAF models, the latent code does not change over time, aligning with the assumptions of our model and the goal of preserving global structure while modeling temporal relationships.
3 Methods
3.1
Stationarity, Time-Shift Equivariance, and ESP
A central challenge in generative modeling of time series is how models handle temporal invariances. Real- world sensor data rarely satisfies strict stationarity. Instead, it often exhibits quasi-periodicity, characterized by similar repcating patterns whose amplitude or frequeney may vary slowly over time. Such data can be viewed as approximately stationary over limited horizons, since its statistical properties remain relatively stable under small temporal shifts. This raises the question of time-shift equivariance: whether model's predictive distribution treats the same local pattern consistently, independent of its absolute position within the sequence.

---

Published in Transactions on Machine Learning Research (03/2026)
Recurrent architectures such as LSTMs naturally encourage this behavior through their sequential update mechanism, but in practice true equivariance does not hold, as hidden states may retain information about initial conditions or absolute position. This effect can be studied through the Echo State Property (ESP), which describes the ability of recurrent networks to forget their initialization and converge to a state deter- mined solely by the input sequence.
While ESP is not equivalent to stationarity, it facilitates approximate shift equivariance by removing spurious dependencies on the imitial hidden state. After sufficient washout period, the network state is determined primarily by the recent imput sequence rather than by absolute position.
To avoid confusion, we briefly summarize the concepts used in this work:
Stationarity (data property):
process (x+) is strictly stationary if the joint distributions of any two windows Tre+e and F++ditt++d are identical for all shifts A. In practice, however, most real-world time series are only approximately stationary. A common and practically relevant case is quasi-periodicity, where the data exhibit recurring but not perfectly regular patterns, such oscillations with slowly vary- ing amplitude, phase, or frequency, that give risc to long-term statistical regularitics without strict invariance. Following the quasi-periodic anomaly-detection literature (Liu et al., 2022 Zangrando let al. 2022 Tang et al. 2023), we characterize this regime operationally as time series that admit a segmentation into cycles such that consocutive cycles are simílar after mild alignment (e.g., small time-warping or phase shift) and normalization, while residual components capture drift, noise, and anomalies. This perspective aligns with how pseudo-periodie streams handled in the data-st and segmentation literature (an Tang al. 2007 Yin et al. 2014
Time-shift equivariance (model property):
A model is lime-shift equivariant if it treats the same local pattern equivalently, regardless of its absolute position in the sequence. Formally, for strietly stationary data and small shifts A, the predictive distributions should satisfy
D(po(1e41 (as)), po(4+1+4 (s+ls+A)) ss 0,
where D denotes a divergence such as Kullback- Loibler or Censon-Shannor
Echo State Property (ESP, dynamical property of recurrent models):
ESP states that the influence of the initial hidden state vanishes over time: for any input sequence (ar) and any two initializations (ho, co) and (M) c)),
[e((za ho, ca) Fe(mr. No: d6)l - 0 as L + x0,
where Ft denotes the unrolled recurrence. ESP provides a mochanism for approximate timo-shift equivariance, since after sufficient washout period the hidden state depends only the input sequence and not absolute position.
To illustrate this relation more concretely, consider the recurrent transition of an LSTM cell,
(h441, f(z,, his ci),
which defines a discrete-time dynamical system on the hidden state. Now consider two partially overlapping input sequences X = [zo...., xn] and X' = [x,], where X' starts one step later but otherwise shares the same continuation. When both sequences are propagated through the recurrence f, their hidden trajectories initially differ due to the additional update step in X. However, under stable dynamics this difference diminishes over time, and
fxx, f(x.-1.... f10, h,))))) f(x,, f( M h,,i,,,000 h))),
(1)
for sufficiently long sequences. This convergence of hidden trajectories, often referred as state forgetting, is the operational manifestation of the Echo State Property and underlies approximate time-shift equivariance in recurrent models.
5

---

Published in Transactions on Machine Learning Research (03/2026)
3.2 Architectural Considerations for Quasi-Periodii Time Series
Given that our focus is on time series data with quasi-periodic behavior, other architectures such as con- volutional layers and transformers face specific limitations. Convolutional layers widely used to build trauslation-equiiariaatt networks, which makes them highly effective in domains like image processing where pattern recognition should be invariant to position. However, in the context of time series, convolution alone is not inherently designed for sequence generation: upscaling typically increases the resolution of a fixed time window rather than extending the sequence length itself (Paul et al., 2022). This distinction limits the ability of convolutional architectures to generate variable-length time series.
Transformers, on the other hand, excel at capturing long-rann rangeaange dependencies, but their self-attention mech- anism scales quadratically with sequence length (Katharopoulos et al. 2020), which makes them computa- tionally demanding for very long series. Moreover, translormers are not inherently translation--qquivariaal Instead, they are prrmutation-eqquiarian and therefore require explicit positional encodings to represent temporal order. While this ficxibility powerful for text or other symbolic sequences, it contrasts with the requirements of time series generation, where a consistent sense of order and time-shift equivariance are central.
By comparison, recurrent architectures such as LSTMs embed temporal order directly into their model dynamics. They maintain internal state that evolves sequentially with the data, naturally supporting the kind of approximate time-shift equivariance discussed above.
These properties motivate our choice of recurrent architectures for modeling quasi-periodic time series as studied in prior anomaly-detection work (Liu et al.] 2022 Zangrando et al. 2022 Tang et al. 2023). In such data, approximate time-shift equivariance encourages representations that are stable under phase shifts and cycle-to-cycle timing variability, matching the practical neod recognize the same pattern regardless of its temporal position. Our progressive training scheme (Section 3.4 complements this architectural choice by stabilizing learning long horizons where repetition exists but is not exact due to drift, noise, and anomalies. These conditions are characteristic of quasi-periodis benchmarks and industrial settings described in prior work (Zangrando el al.
2022 Yang et al. 2025
Encoder hidden states
Decoder hidden states
Time Distributed
2
h11
h1
Linear
Z1
.2
h2
h2
Linear
72
Stacked LSTM
Stacked LSTM
z
Repeat
hn
Linesr goon
Figure This figure illustrates the architecture of our model. Both the encoder and decoder consist of stacked LSTM layers. The encoder's final hidden state, denoted hn is used to compute the parameters I and log(o²), from which the latent variable z is sampled. The latent variable is then repeated across all time steps and used the imput to the decoder. The decoder generates the sequence step-by-step, with each time step's hidden state passed through a time-distributed lincar layer with shared weights. This is represented by the individual Linearshared blocks. Throughout the network, approximate equivariance is maintained with respect to time translation, and the number of trainable parameters remains constant regardless of the sequence length. For a detailed by-laer specification of the decoder with the explicit dimensions used in our experiments, see Appendix A.3

---

Published in Transactions on Machine Learning Research (03/2026)
3.3
Our model builds on the Variational Autoencoder (VAE) framework (Kingma & Welling 2014 Fabius & Van Amersfoort 2014), which minimises the negative evidence lower bound (ELBO):
(u.((r) = Blg + (xo(o po))))
(2)
PR
where LE is the reconstruction loss, La the KI-divergence to the prior po(=) ((0, I), and go(z|z) the approximale posterior (Murphy 2022
The inference network (z]x)) is implemented using stacked LSTM layers. Given the final point in time of a sequence, the output of the last LSTM layer is passed through two lincar layers to produce j and log(o). The latent variable is sampled via the reparameterisation trick, = f + a & e with ~ (0, /). The generative network (z(22) then reconstructs the data from 2. To achieve this, repeated across all time steps (using repeat vector), ensuring that it remains constant and shared throughout the entire sequence:
21 = 2 for allt E (1,2,...,,),
where denotes the total number of time steps. The repeat vector is followed by stacked LSTM layers. Finally, a time-distributed linear layer is applied in the output. This layer operates independently at each time step, applying the same linear transformation to the LSTM output at every time step, which can be viewed as a 1 x convolution across the time dimension, with shared weights across all time steps.
The time-distributed layer is inherently equivariant with respect to time-Iranslation, preserving temporal structure and shifts over time. Together with our LSTM-based approach and the repeat-vector mechanism, this design ensures that the number of trainable parameters remains independent of the soquence length, while also enabling an adapted training scheme that can accommodate increasing sequence lengths. The architecture is illustrated in Figure 2
Details and hyperparameters are provided in Appendix A.6
3.4 Training scheme for sequence lengths
Building the principles of time-shift equivariance and state forgetting discussed in Section 3.1 progressive training scheme that incrementally increases the sequence length during training. While the recurrent architecture introduced in Section 3.3 provides the necessary structural inductive bias, training on long sequences from the boginning often leads to unstable gradients and poor convergence. Our approach mitigates this by first training on short sequences and gradually extending the sequence length, allowing the model to incrementally adapt to longer temporal dependencies without sacrificing training stability.
we adopt
Training recurrent model such as an LSTM to generate consistent long sequences is challenging, as recurrent layers have a limited capacity lo preserve information over extended temporal ranges. To facilitate learning over longer horizons and to encourage stable hidden-state dynamics, we employ a progressive training scheme for the AEQ-RVAE-ST model. The dataset is initially divided into short sequences on which the model is first trained, stabilizing optimization and accelerating convergence. After this initial phase, we subsequently increase the sequence length: the dataset rebuilt into longer chunks, and training continues until the validation loss saturates. This process is repeated iteratively, enabling stable training over increasingly long horizons. Empirically, we find that this scheme improves both convergence stability and final performance compared to training directly on long sequences.
This scheme can be motivated probabilistically. For a time series of length 4, hidden features h of length 4 and a latent vector 2, we assume a recurrent generative structure:
p(z, h, z) p(2) I p(h z, hi-1 h2) p(a, h2).

---

Published in Transactions on Machine Learning Research (03/2026)
This process can be approximated by restricting dependencies to finite memory of t steps:
p(a, h 22
II p(h, ........... h1) p(z; h4)
II p(h/ 2, h.-...... h mmxx(1. p(a; h2).
(3)
Training on shorter sequences therefore corresponds to learning a truncated approximation of the full gen- erative process. Subsequently extending the sequence length during training relaxes this truncation and allows the model to gradually approximate the full time-shift invariant distribution po(x). This progressive extension of the training horizon operationalizes the approximate time-shift equivariance discussed carlier, allowing the model to learn stable long-term dynamics in quasi-periodic data.
Based on experience, recommend starting a short sequence length (e.g,, - 50 to = 100) and using moderate increments (Al < 150). Large increments (AI 2 300) tend to degrade performance. In the main experiments, we use increments of Al 100 as a robust default. A detailed sensitivity analysis is provided in Appendix A.1
4 Experiments
In our experiments, we compare the performance of AEQ-RVAE-ST to several baseline models. AEQ- RVAF-ST uses single, fixed configuration across all datasets and sequence lengths. For the baselines, we adopt official configurations from the respective repositories (e.g., for Diffusion-TT we use the authors' Sine configuration for the Sine datasct); see Appendix A.11.1 for details.
For the training procodure, we started with a sequence length of 100 and subsequently increased by 100 in each subsequent training phase, until reaching a maximum sequence length of 1000. We compare the performance of the models at sequence lengths of 100, 300, 500, and 1000. To evaluate performance, we employ combination of short-term consistency measures based on independently generated ELBOs, the discriminative score, and the contextual FID score. Additionally, we perform visual comparisons between the training and generated data distributions using dimennionnaiiyy reduction techniques such as PCA and t- SNE, All reported results were tested for statistical significance using the Wilcoxon rank-sum test (Wilcoxon) 1992
In cases where the difference was not statistically significant, multiple values are highlighted bold.
4.1
Data Sets
For experiments use three multivariate sensor datasets with typical semi-stationnry behavior, synthetic benchmark, and one less quasi-periodic dataset. We specifically selected datasets with a minimum size necessary for training generative models effectively.
This dataset was collected from a three-phase motor operating under constant speed and load conditions. We use only the file H1.5, selected arbitrarily among the available recordings. The data. recorded at a sampling rate of 16 Out of the twelve initially available channels, four were removed due to discrete behavior abrupt changes, leaving only smooth, continuous signals. The resulting dataset contains approximately 250,000 datapoints and represents highly quasi-periodic real-world time series.
Electric motor (EM) WiBbrock & Müllerr 2025: Mueller, 2024):
ECG data (ECG)? Goldberger et al. (2000): This dataset contains a two-channel electrocardiogram recording from the MIT-BIH Long-Term FCG Database. Tt. has nearly 10 million time steps of which we use the first 500,000 for training. The signals exhibit clear periodic structure corresponding cardiac cycles, yet show natural variability in frequency and morphology, including occasional irregularities such as arrhy Our objective is not to produce medically usable data; specialized models are likely more appropriate for medical applications (Neifar et al. 2023
8

---

Published in Transactions on Machine Learning Research (03/2026)
ETTm2 ( The ETTm2 dataset contains sensor measurements such load and oil temperature from electricity transformers, recorded over two-year period at coarse sampling rate of four points per hour. While originally proposed for long-term forecasting (Zhou et al., 2021), our analysis suggests that its temporal dynamics are weakly quasi-periodic at best, due to the limited temporal resolution, the short analysis horizon relative to the seasonal cycles, and the small dataset size (69,680 samples).
This dataset consists of five independent sine waves with randomly sampled frequencies and initial phases drawn from (0, 0.1). The resulting signals are smooth, noise-free, and nearly stationary, serving canonical benchmark for generative time-series models YYonn et al. 2019 Desai et al. 2021b
Synthetic Sine:
Yuan & Qiao 2024).
MetroPT3 (Davari et al. (2021) The MetroPT3 dataset consists of multivariate time-series data from analogue and digital sensors installed on a compressor, originally collected for predictive maintenance and anomaly detection. The data were logged at Hz. Similar to the Electric Motor datasct, we removed non- continuous or discrete signals. Out of the original 1.5 million lime steps, we used the first 500,000. Recurring patterns in this dataset are frequently interrupted by phases of flat signals, leading to irregular temporal dynamics.
4.2 Baseline Models
We compare against five established generative models spanning GANs, VAEs, diffusion models, and Transformer-basce architectures: TimeGAN (Yoon et al. 2019), a recurrent GAN; WaveGAN (Donahue et al. 2019), a convolutional GAN originally designed for raw audio; TimeVAE DDsaa et al. 2021 a con- volutional VAE; Diffusion-TT -T (Yuan & Qiao 2024], a diffusion model with Transformer-
trend-seasonal (Liu al. 2024), an adversarial autoencoder combining TCNs and
decomposition; and Time-Transfore Transformers. None of these baselines enforce approximate time-shift equivariance by design. A detailed description of each model's architecture and equivariance properties is provided in Appendix A.10
4.3 Emphasizing Inductive Bias with Echo State Property (ESP)
The ESP provides useful lens analyze inductive bias in recurrent generative models. Formally, ESP states that when driven by the same input sequence, hidden states forget arbitrary initial conditions and converge to a unique input-determined trajectory (Jaeger 2001: Manjunath & Jaeger 2013). Note that slandard LSTMs do not guarantee ESP in general. Our measurements are empirical indicators of contraction rather
than a formal guarantec (Yildiz et al. 2012 Buchner & Young 2006).
Interpreting ESP in our seiting leads three important insights:
ESP as forgetting irrelevant information: Strong ESP does not mean that the network indiscriminately forgets all information, but specifically that it suppresses dependence on arbitrary initializations. Once washout has occurred, the hidden states become determined primarily by the input. This aligns well with nearly stationary or quasi-periodic data, where invariance to absolute time is desirable.
ESP versus mcaningful long-term memory: A model without ESP may appear to "retain" information longer, but what is retained is often dependence on the random initialization rather than useful structure in the input sequence. Conversely, moderate ESP allows the model to forget initialization artifacts while still preserving long-term dependencies driven by the input. Thus, ESP should not be interpreted the opposite of memory capacity, but rather the ability separate meaningful input-driven memory from spurious initialization effects.
Inductive bias for stationarity: In generative modeling of time series, ESP encourages the network to emphasize relative temporal patterns over absolute time indices. This induces an inductive bias toward stationarity-like behavior: repeated patterns are treated consistently regardless of where they occur in the sequence. At the same time, the property is only approximate in our trained models, allowing ficxibility to retain non-stationary structure (e.g,, (rends or irregular variations) when present in the data.
9

---

Published in Transactions on Machine Learning Research (03/2026)
ESP across datasets (log scale)
Normalized hidden state distance (ratio. log scale)
10
10-1
10-2
10-3
10
10
10-7
10
200
400
600
800
1000
Sequence length (t)
ECG
EM
ETT
Sine
MetroPT3
Untrained
Figure 3: Echo State Property (ESP) analysis across datasets (log scale). The x-axis shows sequence length and the y-axis the normalized hidden-state distance r(t) - d(t)/d(0), where d(t) = E,, (hJ 3,155) 4, - [|2]. Here he denotes the hidden state of the top LSTM layer only (no cell states are used). Random initializations are applied only to the top layer; all lower layers identical fixed initializations across runs. By construction set r(0) = (the curve is normalized to the initial distance). The expeclation is approxinated by sampling 10 latent vectors 2; for each z one input sequence is generated and d(t) is averaged over 20 independent pairs of random top-layer initial states (AT (.). On the logarithmic scale, exponential contraction appears as straight lines. The untrained model indeed shows such monotonic exponential decay, with r(t) converging into the range of 10-9 and remaining stable thereafter, reflecting trivial washout due to random initialization. In contrast, trained models display weaker contraction with residual variability in the curves. ETTm2 reaches about 10-7, the synthetic sine dataset around 10-3, while ECG, EM, and MetroPT3 remain higher. This illustrates how training balances ESP with the preservation of long-term temporal structure. All hidden states are calculated based on the weights of the models with I = 1000.
As shown Fig. 3 we observe contraction across all datasets, though with varying strength. Trained models exhibit weaker contraction than the untrained baseline, with ETTm2 showing the strongest and ECG, EM, and MetroPT3 the weakest contraction among trained models. This illustrates how training counterbalances the architectural bias by preserving input-driven dependencies where useful, rather than enforcing unconditional washout.
The particularly strong contraction observed on ETTm2 is likely due to the combination a coarse temporal resolution of four samples per hour, an analyzed horizon of 1000 steps (approximately 10 days), and limited total span of about two years. Together, these factors make it difficult learn mcaningful long-term scasonal dependencies, so that the model instead forgets initial states rapidly, producing the appearance of strong ESP.
10

---

Published in Transactions on Machine Learning Research (03/2026)
4.4 Evaluation by Context-FID Score
To evaluate the distributional similarity between real and generated time series, we use the Context-FID score [Paul ct al. 2022), variant of the Fréchet Inception Distance (FID) commonly used in image generation. In this adaptation, the original Inception network is replaced by TS2Vec (Yue et al. 2022 : a self-supervised representation ion learning method for time series. The score is computed by encoding both real and generated sequences with a pretrained Vec model and calculating the Fréchet distance between the resulting feature distributions. Lower scores indicate that the synthetic data better matches the distribution of the real data.
Table reports the Context-FID scores across different sequence lengths and datasets.
Across the different sequence lengths, AEQ-RVAE-ST consistently outperforms all comparison models on the Electric Motor, ECG, and especially the Sine datasets starting from / 300. These datasets exhibit high quasi-periodicity, which aligns well with the inductive biases of our approach. On the lesser quasi-periodia datasets MetroPT3 and ETT, our model remains competitive, with TimeVAE surpassing it at l = 1000 for both datasets. Additionally, for MetroPT3, Diffusion-Ti TS outperforms our model at = 500.
4.5 Evaluations by Discriminative Score
metric for quality evaluation of synthetic time series data. For the discriminative score simple 2-layer RNN for binary classification is trained to distinguish between original and synthetic data. Implementation details are in the appendix [2] 12] It is defined as D = 10.5 al, where a represents the classification accuracy between the original test dataset and the synthetic Lest dataset that were not used during training. The best possible score of means that the classification network cannot distinguish original from synthetic data, whereas the worst score of 0.5 means that the network can easily do so.
The discriminative score DD was introduced by (Yoon et al. 2019
The discriminative score provides particularly meaningful insights when it allows for clear distinctions be- tween models, which is best achieved by avoiding scenarios where the score consistently reaches its best or worst possible values across different models. To ensure consistency, we used the same fixed number of sam- ples for training the discriminator across all experiments, regardless of sequence length. This fixed sample size was found to be suitable for our experimental setup.
As shown in Table 2 the Discriminative Score yields a less clear-cut picture compared other evaluation metrics. The Wilcoxon rank-sum test reveals that in several cases, performance differences between models are not statistically significant.
On the Electric Motor dataset, AEQ-RVAE-ST achieves the best performance from / = 300 onwards. For the ECG dataset, A AEQ-RVAF-ST antperforms all other models at 1= 1000, while for shorter sequence lengths, its performance is comparable to that of Diffusion-TS. On the ETT dataset, Diffusion-TS perform similarly well across all sequence lengths, with no statistically significant differences. The Sine dataset exhibits more nuanced behavior: Diffusion-TS performs best at 1 = 100; at / = 300, AEQ- RVAE-ST, TimeVAE, and Diffusion-ISs perform comparably; and from ( = 500 onwards, AEQ-RVAE-ST achieves the best results. For the MetroPT3 dataset, AEQ-RVAE-ST is best at / = 100, while Diffusion-TS slightly outperforms all other models at longer sequence lengths.
TimeVAE, and
4.6 Evaluation by PCA
In this section, we evaluate the quality of the generated time series using PCA (Hotelling) 1933 The idea is to train PCA the original data, project it into dover-dimensional space, and apply the same transformation to the synthetic data to assess distributional alignment. While widely used for identifying structural similarities, this technique does not account for temporal dependencies within the sequences. Additional 1-SNE Hiinoon & Van Der Maaten 2008) visualizations are provided in Appendix A.14
These common tochniques complement carlier methods that primarily assessed the sample quality of the models. For brevity, we present the results of four selected experiments in the main paper, as all experiments consistently yield the same findings. These four experiments include PCA plots the EM dataset. and on the
11

---

Published in Transactions on Machine Learning Research (03/2026)
Table 1: FID score of synthetic time series for six models (see 4.2 computed on the five datasets (see 4.1p at sequence lengths 100, 300, I 500, and I 1000. Lower scores indicate better performance. Each score based on 5000 generated samples, each evaluated (trained) 15 times, and reported with 1- sigma confidence intervals. AEQ-RVAE-ST consistently outperforms all baselines the highly periodical Electric Motor, ECG, and Sine datasets starting from / 300. On the less periodical MetroPT3 and ETT datasets, performance is more competitive, with TimeVAE and Diffusion- TS outperforming our model at certain sequence lengths.
Sequence lengths
Dataset
Model
100
300
500
1000
AEQ-RVAE-ST (ours)
0.35+10.04
0.12:0.01
0.10:0.01
0.24+0.02
TimeGAN
1.03:10.07
3.77:10.30
3.07:0.24
33.7:11.69
Flectric Motor
WaveGAN
0.55
0.75
0.87#0.14
1.41#0.24
TimeVAE
0.16+00.1
0.97+0.111
1.06#0.14
1.19++0.09
Diffusion-TS
0.04+0.00
0.69:10.06
1.10#0.11
1.93::0.13
Tine-Transformr
2.19
45.4+1.57
44.5+2.67
65.7+2.86
AEQ-RVAE-ST (ours)
0.08+0.01
0.09+0.02
0.14+0.02
0.46+0.06
TimeGAN
26.8
48.0+6.26
47.2
34.0
WaveGAN
1.54
1.56+0.14
1.54
1.51
ECG
TimeVAE
0.26+
0.89+0.007
1.07
1.30-L0.08
Diffusion-
0.16
0.28+0.03
0.52+0.03
3.74+0.22
Time- Trans-rransormmrr
1.34
29.7
33.0:22.28
40.3+2.44
AEQ-RVAE-ST (ours)
0.5810.05
0.6310.07
0.79+0.07
1.82::.116
TimeGAN
1.51
3.76+20.3
13.7
17.7+1.57
WaveGAN
3.49+0.22
3.90+0.37
4.38
4.94+0.42
ETT
TimeVAE
0.66
0.72+0008
0.97
1.56:0.14
Diflusion-TS
0.90
1.18:10.18
2.16
2.55-L0.27
Time-Tree Transformer
1.28
20.1+1.22
22.1
47.945.28
AEQ-RVAE-ST (ours)
0.33
0.3410.02
0.4610.03
0.42+0008
TimeGAN
7.70+0.32
6.01+0.34
7.96+
21.8+
WaveGAN
1.87
2.0910.13
2.81
3.36+L0.27
Sine
TimeVAE
0.24
0.55++0.05
1.26++0.14
3.03++1.00
Diffusion-TSS
0.0610.00
1.5240.13
0.74+0.04
2.66+00.00
Timo-Transforme
0.31
4.10:1021
51.2+1994
74.5+
AEQ-RVAE-ST (ours)
0.26:10.4
0.65
2.31
2.84+0.22
TimeGAN
5.79-H0.32
10.1
18.6+
35.74
WaveGAN
1.14++0.09
1.82+-012
2.04
2.43+-0.18
MetroPT3
TimeVAE
0.67+10.05
1.32.10.13
2.0240.29
2.08+0.31
Diffusion- TS
1.07+0.06
1.17+0.12
1.82:10.09
6.97+00.5
Time-Trrasfrrmer
2.28+
5.25++0.66
22.9+1.45
352-66.1
ECG dataset, each with sequence lengths of ( = 100 and 1000 (see figure 4 The full set of experiments is provided in Appendix A.14
The visual inspection of the PCA plots for the EM dataset with a sequence length of / = 100 reveals no significant differences in the distributions the models, with Time-Treee Transformer showing a slightly less 1000,
pronounced overlap compared the other models. However, as the sequence length increases the performance differences between the models become clearly visible. Interestingly, the PCA this length exhibits circular pattern, indicating the periodic characteristics of the dataset. Among the models.
12

---

Published in Transactions on Machine Learning Research (03/2026)
Table 2: Discriminative score of synthetic time series for six models (see 4.2 computed on the five datasets (see 4.1 at sequence longths 100, 300, I 500, and 1000. A lower score indicates better performance. Each score is based on 15 independent discriminator runs and reported with 1-sigma confidence intervals. AEQ-RVAE-ST performs best on the Electric Motor dataset from = 300 onward and significantly outperforms all models on ECG at 1000, while showing comparable performance to Diffusion-TSS at shorter lengths. For the ETT and Sine datasets, multiple models perform similarly depending on the sequence length. On MetroPT3, AEQ-RVAE-ST is best at I = 100, while Diffusion-TS dominates for longer sequences. In cases without statistically significant differences (Wilcoxon rank-sum test), multiple scores are highlighted in bold.
Sequencc lengths
Dataset
Model
100
300
500
1000
AEQ-RVAE-ST (ours)
.121#.021
.0321.018
.0382.018
.0852.015
TimeGAN
330+.030
.477+.018
.486+.013
.500+.000
Electric
WaveGAN
.322:0009
.416:.009
.425+.011
.444..011
Motor (EM)
TimeVAE
.268
.26
.185+.083
.152:.047
Diffusion-II TS
.1121.056
.377..130
.366+.085
.344.084
Time- Transfrraso
334+.098
.500+.000
500+.000
.500+.000
AEQ-RVAE-ST (ours)
.0121.011
.0091.008
.0161.014
.0091.010
TimeGAN
165..125
5500-0000
.500+.000
5500
WaveGAN
305
300
153
.288
ECG
TimeVAE
.034..066
0588:.120
.131
.153
Diffusion-TS
.0071.007
.0161.016
.0102.015
.382
Time-Transforma
.116..107
500++000
.496+.014
.499+.002
AEQ-RVAF-ST (ours)
.199+.034
.722..10
.189+.099
.132+.147
TimeGAN
.107..075
.160+.113
270
.20
WaveGAN
.362+.080
.413..113
.39
.355+.060
ETT
TimeVAE
.1181.110
.1404.053
.167..040
.0681.051
Diffusion-TS
.204+.086
.173+.063
.151+.055
.122+.051
Time-Transformee
.19
.1794.116
137
5500+.000
AEQ-RVAE-ST (ours)
.0691.015
.113..059
.080+.044
.021+.013
TimeGAN
165+.130
.57+.050
.491+.005
497+-005
WaveGAN
1874.036
.674.073
.499..025
.449..034
Sinc
TimeVAE
.611:.092
.1601.124
.2722.129
.377..14
Diflusion-TS
.035+.014
.182+.163
.294
.428
Time-Transfrrme
.131:.019
.911..004
.999.001
.500
AEQ-RVAF-ST
.098+.066
167
.23+.074
496+.004
TimeGAN
.284.041
.488:.002
.499..001
.4991.001
WaveGAN
4324.042
.94+.005
.977.0002
.497+.003
MetroPT3
TimeVAE
273
.43
.488+.024
.45++0044
Diffusion-TS
.139+.025
.251+.022
.319.0015
.486+.012
Time-Traasformer
.43++.007
.493+.005
.500++000
.500+.000
AEQ-RVAE-ST demonstrates the highest degree of overlap between the original and synthetic data, fitting the circular pallern without outliers. Diffusion-TS performs almost equally well, with slightly less overlap compared to AEQ-RVAE-ST (see figure I for visual comparison of the models). WaveGAN shows only few outliers near the circular pattern. TimeVAE synthetic points further fill the circle, leading to greater deviation from the original data distribution. The PCA plots for the ECG dataset provide detailed view of models' performances. At. 1 100, AEQ-RVAF-ST, TimeVAE, and Diffusion-TS perform equally well, showing
13

---

Published in Transactions on Machine Learning Research (03/2026)
PCA
PCA
PCA
AEQ- RVAE-ST (ours)
PCA
PCA
TimeGAN
PCA
PCA
PCA
WaveGAN
PCA
PCA
TimeVAE
KCA
PCA
PCA
PCA
Diffusion-TS
PCA
Time-
Transformer
EM, 101
EM, 1000
ECG, 1-100
ECG, 1-1000
Figure 4: PCA plots for the EM and ECG datasets at sequence lengths of ( = 100 and / = 1000. The higher the overlap between original and synthetic points, the better. For the EM dataset at I = 100, no significant showing slightly less overlap. At 1 = 1000, the circular
differences are observed, with Time--Tranforee pattern reflects the periodic characteristics of the dataset, with AEQ-RVAE-ST demonstrating the best overlap, closely followed by Diffusion-TSS TS. WaveGAN and TimeVAE show some deviations, while TimeGAN exhibits almost no overlap. For the ECG dataset at 100, AEQ-RVAE-ST, TimeVAE, and Diffusion- TS show strong overlap, while WaveGAN and Time-Transfan Transforrer exhibit less, and TimeGAN shows almost none. At 1 1000, RVA--ST performs best, followed by WaveGAN and TimeVAE, with Diffusion- TS performing worse and TimeGAN and Transforaoorr showing minimal variability and no significant overlap.
strong overlap with the original data. WaveGAN and Time-Transfonee Transformer show less overlap, and TimeGAN demonstrates almost no overlap at all. At 1000, AEQ-RVAF-ST achieves the best performance, with the original data being very well represented. This is followed by WaveGAN and TimeVAE, where the synthetic data points cluster together, but with less coverage of the original distribution. Diffusion-TS performs
14

---

Published in Transactions on Machine Learning Research (03/2026)
noticeably worse, while TimeGAN and Time- Transforme show almost overlap, with the generated data exhibiting minimal variability.
4.7 Training scheme ablations
In this experiment, compare the effectiveness of our proposed training approach against the conventional training method on the same network topology. Our comparison metric is the Evidence Lower Bound (ELBO), calculated for the original dataset X E =xxx where n, represents the numbers of samples, denotes the sequence length, and c the number channels. We calculate it as
1 Thg
E(X)
ELBO norm (Ca))())))
where La,s the loss of the trained model itself. Simply speaking, it is the typical model evaluation on a dataset, but converted ELBOnorm (sce Appendix A.7].. We run this comparison on all datasets with a sequence length of 1000, which is particularly long and challenging. It the maximum sequence length used in any of the previous experiments. For each of the following training schemes, we do 10 repetitions:
(i) Conventional train: One trains the model for predefined sequence length of = 1000
(ii) Subsequent train: The training procedure begins with sequence length of 1 100 and continues until the stopping criteria are met. Afterward, increase the sequence length by 100 and retrain the model, repeating this process until we complete training with sequence length of = 1000.
Table 3: Comparison of the effectiveness of our proposed training approach versus the conventional method. The performance metric is the ELBOnorm as described in Appendix A.7 On each dataset and model we repeated the experiments n = 10 times. The 1-sigma confidence intervals describe the results between the independently trained models.
Train method
EM
ECG
ETTm2
Sine
MetroPT3
conventional train
0.004 0.21811.004
0.103
0.174
0.837
0..140+0.001
subsequent train
0.20110.004
0.217:0.012
0.19410.010
0.14210.019
As shown in Table 3, the subsequent training scheme (ii) consistently outperforms the conventional training scheme (i) across all datasets, with statistically significant improvements (p < 0.002). The largest perfor- mance gain is observed on the Sine dataset, where the model's ability to capture sinusoidal patterns improves substantially. In Figure representative samples for cach model are shown for a sequence length of = 1000 on the sine dataset. A AEQ-RVAE-ST is the only model that can generate proper and consistent sine curves, which are characteristic of the dataset. The Sine dataset, as clear example a periodic and almost sta- tionary time scries, supports our hypothesis that the AEQ-RVAE-ST model benefits from an inductive bias towards periodicity that enables the model effectively generate consistent, high-quality long-range sequences in such scenarios. Further ablation studies the sensitivity of the subsequent training scheme to differ- ent sequence-length increments, as well as more detailed analysis of training schedules, provided in Appendix A.1
5 Discussion
In this paper, we present hypothesis-driven examination of modeling long time series using approximately time-shift-equivariant architectures. Our central hypothesis is that quasi-periodic time series benefit from an inductive bias that promotes temporal consistency and invariance to absolute time. Approximate time- shift equivariance enables a model to recognize and reproduce recurring temporal patterns across different positions in a sequence, which is particularly important for data with oscillatory repeating structures.
15

---

Published in Transactions on Machine Learning Research (03/2026)
56e
1003
1000
2000
(a) AEQ-EEEQAAAR RVAE-ST
(b) TimeGAN
(c) WaveGAN
mmy /
M
(d) TimeVAE
(e) Diffusion-TS
(f) Time Transforrorrrr
Figure 5: Representative samples for each model a sequence length of = 1000 on the sine dataset AEQ- RVAE-ST the only model capable of consistently generating accurate sinusoidal trajectories, demonstrating its ability to capture the strictly periodic characteristics of the data. For additional samples generated by our model on this dataset, we refer the reader to Figure in the appendix.
While the recurrent layers in our model provide only partial shift equivariance, the overall architecture maintains a consistent transformation behavior across time, leading two main advantages: (1) an inductive bias that aligns with the characteristics of quasi-periodic and slowly varying temporal dynamics, and (2) parameterization independent. of sequence length, which allows the model to scale efficiently to longer time horizons.
These properties allow the model to exploit temporal regularitics more effectively during training and support our interpretation that approximately equivariant recurrent architectures provide a suitable inductive bias for modeling quasi-periodic time series.
In our experiments, we compared AEQ-RVAE-ST with several state-of-the-art generative models across five benchmark datasets. Three of these (Flectric Motor, F.GG, and Sine) exhibit strong quasi-periodicin while ETT and MetroPT3 show greater temporal variability, though still containing recurring signal components typical of sensor-based data. On the quasi-periodic datasets, our model consistently outperformed all base- lines, especially as sequence length increased, as reflected by the Context-FID and Discriminative Score. On the more irregular datasets, remained competitive across most configurations. Latent-space visualizations using PCA and t-SNE further confirmed that our model captures the global structure of the data more laithfully than the baselines.
In Section A.2 samples of arbitrary length, illustrated for 1 = 5000. Together with the results on the Echo State Property (ESP) and state forgetting, these findings lend further support to our theoretical assumption (see Equation T that, for sufficiently long sequences, the hidden and cell states converge toward trajectories determined by the input dynamics rathor than by initial conditions.
demonstrate that model trained on sequences of length / = 1000 can generate coherent
Our findings confirm the effectiveness of the proposed approach and open several promising directions for fu- ture research. The methodology could be extended to other model classes, such diffusion- based generative architectures.
5.1 Limitations
Our approach is designed for quasi-periodic time series characterized in Section 3.1 data that exhibit recurring but imperfectly regular patterns, such as oscillations with slowly varying amplitude, phase, or frequency. While this inductive bias is advantageous for such data, it entails several limitations.
Time series with persistent trends, regime changes, or structural breaks violate the approximate stationarity assumption underlying our approach. Similarly, sparse event-driven time series (e.g., point processes, trans- action data) do not exhibit. the recurring motifs that our inductive bias exploits. Our model also requires
16

---

Published in Transactions on Machine Learning Research (03/2026)
sufficient training data to learn stable long-horizon dynamics. The ETTm2 dataset used in our experiments (69,681 time steps) represents borderline case, especially given that seasonal effects in this data unfold over longer time scales than the available data can capture.
Additionally, our model performs best on smoothly varying, high-resolution signals. Low temporal resolution can limit. the ability to capture fine-grained temporal structure (e.g., ETTm2 with 4 samples per hour). Time series with abrupt transitions or discontinuities also deviate from our assumptions. MetroPTT3 illustrates this challenge: several channels exhibit frequent sharp drops (see Figure [0] TP2, H1, Motor Motor_current) which dominate the dynamics and reduce the benefit of our inductive bias.
References
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruction probability. Special lecture IE 2(1):1-18, 2015.
Lv an Tang, Bin Cui, Hongyan Li, Gaoshan Miao, Dongqing Yang, and Xinbiao Zhou. Effective variation management for pseudo periodical streams. Proceedings of the ACM SIGMOD International Conference on Management of Data, pp. 257 268, Beijing, China, 2007. ACM. doi: 10. 11145//24744001247511. URL
https://doi. 10. 1111441/444000 1247511
Dzmitry Bahdanau. Neural machine translation by jointly learning to align and translate. ar preprint arXiv:1409.044 0478, 2014.
Shaojic Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.012 2018.
Michael Buehner and Peter Young. A tighter bound for the echo state property. IEEE Transactions on
Newral Networks, 17(3):820-82) 824, 2006.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in --vae. arXiv preprint 1844:8800005000 2018
Narjes Davari, Bruno Veloso, Rita P. Ribeiro, Pedro Mota Pereira, and João Gama. Predictive maintenance based anomaly detection using deep learning for air production unit in the railway industry. In 2021 IEEE 8th International Conference Dala Science and Advanced Analylics (DSAA), pp. 1 10, 2021. doi: AS1.DAAAA5....00....119
Abhyuday Desai, Cynthia Freeman, Zuhui Wang, and Ian Beaver. Timevae: variational auto-encoder for
multivariate time series generation. arXiv preprint arXiv:2111 11.021011000,, 2021a.
Abhyuday Desai, Cynthia Freeman, Zuhui Wang, and Tan Beaver. Timevae: variational auto-encoder for multivariate time series generation. https: httpp://github com//bbudesaiitiieeaE 2021b.
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis, 2019.
Cristóbal Esteban, Stephanie Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans, 2017.
Otto Fabius and Joost R. Van Amersfoort. Variational recurrent auto-encoders. arXiv preprint arXiv:1412.688, 2014.
Laurent Girin, Simon Loglaive, Xiaoyu Bic, Julien Diard, Thomas Hueber, and Xavier Hameda-Pinodas Dynamical variational autoencoders: A comprehensive review. Foundations and Trends0 in Machine Learning, 15(1 2):1-175, 2021. ISSN 1935-8245. doi: 10.1561 11561/2000000090 URL http://dxx doi. org/ 10. 10.1222020000
A. Goldberger, Amaral, L. Glass, J. Hausdorff, P. C. Ivanov, Mark, and H. E. Stanley. Physiobank, physiotoolkit, and physionet: Components of new research resource for complex physiologic signals. Circulation, 101 (23):e215 e-2200 2000. doi: 10.1161 wTw................ Online.
17

---

Published in Transactions on Machine Learning Research (03/2026)
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commuuicaatinss of the ACM, 63(11):
139 144, 2020.
Tom Hammerbacher, Markus Lange-Hegermann, and Gorden Platz. Including sparse production knowledge into variational autocncoders to increase anomaly detection reliability. In 2021 IEEE 17th International Conference Aulomalion Science and Engineering (CASE), pp. 1262 1267. IEEE, 2021.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mo- hamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with constrained variational framework. In International Conference Learning Representations, 2017.
G Hinton and L Van Der Maaten. Visualizing data using t-sne journal of machine learning research. Journal of Machine Learning Research, 9:2579-2600 2605, 2008.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Aduances in neural information processing systems, 33:6840 6851, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9:1735-80, 12 1997. doi:
Harold Hotelling. Analysis a complex of statistical variables into principal components. Journal of educational psychology. 24(6):417, 1933.
Herbert Jaeger. The 'echo state" approach to analysing and training recurrent neural networks. Technical Report GMD Report 148, German National Research Center for Information Technology (GMD), 2001. Updated 2010 with erratum.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation, 2018. URL http://aaxxiv..ggbbb//////a 10196
Angelos Katharoopolls, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rmns: Fast autoregressive transformers with linear attention. In International Conference Machine Learning, pp. 5156-5165. PMLR, 2020.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. 12 2014.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- basee learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278 22,4224, 1998.
Fan Liu, Xingshe Zhou, Jinli Cao, Zhu Wang, Tianben Wang, Hua Wang, and Yanchun Zhang. Anomaly detection in quasi-periodic time series based automatic data segmentation and attentional LSTM- CNN. IEEE Transactions on Knowledge and Data Engincering, 2466))2626-2640, 2022. doi: 10.1109/ 2O0DE.2020.3014800 URL
wwt.:////..........1111 1110/1TTDDDB 3014886
Yuansan Liu, Sudanthi Wijewickrema, Ang Li, Christofer Bester, Stephen O'Leary, and James Bailey. Time- transformer: Integrating local and global features for better time series generation, 2024. URL https:
//arxiv.org. www.:/rgg//bbbbboeooooo 1171 11714
Yonghong Luo, Xiangrui Cai, Ying Zhang, Jun Xu, et al. Multivariate time series imputation with generative adversarial networks. Advances in neural information processing systems, 31, 2018
Alireza Makhzani, Jonathon Shlens, Navdeop Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoen- coders, 2016. URL /https://arxi> org/
05644
Gopalakishna Manjunath and Herbert Jaeger. Echo state property linked to input: Exploring a funda- mental characteristic of recurrent neural networks. Neural Computation, 25(3):671-696, 2013.
Olof Mogren. C-rnn-gan: Continuous recurrent neural networks with adversarial training, 2016. URL
patps://arriis wwwpoo//g/.......o////ar/////aae 09904
18

---

Published in Transactions on Machine Learning Research (03/2026)
Philipp N. Mueller. Attention-enhaneed canditional-difusiinn-aane data synthesis for data augmentation in machine fault diagnosis. Engineering Applications of Artificial Intelligence, 131:107696, 2024. doi:
httpps//dooi.rgg/ h0t.:000///eegaappaa000000111110
Kevin P. Murphy. Probabilistic Machine Learning: An introduction. MIT Press, 2022. URL probml ai
Nour Neifar, Achral Ben- Ben-Hamadou, Afef Mdhhafaar and Mohamed Jmaiel. Diffeog: versatile probabilistic diffusion model for eog signals synthesis. arXiv preprint or arXi::2006.0 018.0, 2023.
Jeha Paul, Bohlke-Schmeidrr Michael, Mercado Pedro, Kapoor Shubham, Singh Nirwan Rajbir, Flunkert Valentin, Gasthaus Jan, and Januschowski Tim. Psa-gan: Progressive self attention gans for synthetic time series, 2022. URL
hwtpg//ab//2100000 00981
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodci, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenA/ blog, 1(8):9, 2019.
Sima Siami-Namini, Neda Tavakoli, and Akbar Siami Namin. The performance of Istm and bilstm in forecasting time series. In 2019 IEEE International conference big dala (Big Dala), pp. 3285 -292.. JEEE, 2019.
Xiaolan Tang, Desheng Zheng, Gebre S. Kebede, Zhengyu Li, Xiaoyu Li, Chao Lu, Lintao Li, Yong Zhou, and Shan Yang. An automatic segmentation framework of quasi- time series through graph structure. Applied Intelligence, 53:23482-234999 2023. doi: 10.1007 00.1007/s104800 020-000-222-04-4--- URL https:///oi. org/
10. 10.10077/10449-02--088811
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34: 24804 24816, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Frank Wilcoxon. Individual comparisons by ranking mothods. In Breakthroughs stafistics: Methodology and distribution, pp. 196 202. Springer, 1992.
P. Wißbrock and P. N. Miller Lonze motor bearing fault dataset (lenze-mb), 2025. URL https://do1. org/10.5281/zend
14762423
Kai Yang, Shaoyu Dou, Pan Luo, Xin Wang, and H. Vincent Poor. Robust group anomaly detection for quasi- periodic network time series. Xiv preprinl arXio:2506. 16815, 2025. doi: 10.48550/arXiv.221 156.11661.... URL https://arxit:
16815.
hogg/bbs/285 2506 16815
Yiyuan Yang, Ming Jin, Haomin Wen, Chaoli Zhang, Yuxuan Liang, Lintao Ma, Yi Wang, Chenghao Liu, Bin Yang, Zenglin Xu, et al. A survey on diffussio models for time series and spatio-temporal data. anviv preprinl Xiv:22404.. 18886, 2024.
Ismail Burak Yildiz, Herbert Jaeger, and Stefan J. Kiebel. Re-visiting the echo state property. Newral Networks, 35:1-9, 2012.
Ning Yin, Shanshan Wang, Shenda Hong, and Hongyan Li. A segment-wise method for pseudo periodic time series prediction. In Advanced Data Mining and Applications (ADMA 2014), volume 8933 of Lecture Notes in Computer Science, pp. 461-46088 664,47,, Guilin, China, 2014. Springer. doi: 10.1007 9700079977---19-11 URL https://doi. org/10 1007/970- 3-77788-311---1 3-77--7-7--117778----811 36
36.
Jinsung Yoon, Daniel Jarrett, and Mihaela Van der Schaar. Time-series generative adversarial networks. Aduances in neural information processing systems, 32, 2019.
Xinyu Yuan and Yan Qiao. Diffusion-TS: Interpretable diffusion for general time series generation. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreviiet
forum?id=4hhappji0
19

---

Published in Transactions on Machine Learning Research (03/2026)
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series, 2022. URL https://axxiv org/ abs/2106.
10466
Niccolò Zangrando, Piero Fraternali, Marco Petri, Nicolò Oreste Pinciroli Vago, and Sergio Luis Her- Gonzalez. Anomaly detection in quasi-periodic energy consumption data series: A comparison of algorithms. Eneryy Informalics, 5(Suppl 4):62, 2022. doi: 10.11866/442102--22---23233 URL
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Tn- former: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty-Pifth AAAI Conference Artificial Intelligence, AAAI 2021, Virtual Conference, volume 35, pp. 11106-11115. AAAI
Press, 2021.
Yuzhen Zhu, Shaojie Luo, Di Huang, Weiyan Zheng, Fang Su, and Beiping Hou. Dremn: decomposing residual convolutional neural networks for time series forecasting. Scientific Reports, 13(1):15901, 2023.
A Appendix
A.1 Training Scheme Ablations
Table 4: Sensitivity of subsequent training to different sequence-length increments on the Electric Motor and Sine datascts. Performance is mcasured via ELBOnorm (Appendix A.7). Each entry reports the mean over n = 5 independently trained models, with 1-sigma confidence intervals. Higher is better.
Subsequent schedule (sequence length 1)
EM
Sine
50, 100, 150. 1000 100, 200, 300, ..., 1000
0.22110.004
0.200
0.218 004
0.194+0.010
100, 250, 400, 550, ... 1000 1000
0.221
0.202=0.004
100, 400, 700, 1000
0.216=:0.008
0.156=:0.024
100, 550, 1000
0.216=+0.00%
0.141+00.009
1000 (no subsequent train)
0.094+0.004
-0.837=0.566
For fair comparison across schedules, start training at short sequence length (typically 1 = 100; for the +50 schedule we start at 1 -= 50) before applying larger sequence length increments. In our experience, starting directly with large sequence lengths (or making large jumps without this warm start) leads to substantially worse optimization and less stable training.
same warm-start protocol in all experiments: always
For each training schedule in Table we train five independently initialized models. Table 4 indicates that performance relatively robust for small-to-moderate increments, while large jumps degrade ELBO most notably on Sine. In particular, once the step size becomes large (roughly Al Z 300), performance degrades markedly. Moreover, on both Electric Motor and Sine, the schedules with increments of 50 and 150 outperform the 100-step schedule. Overall, the 150-step schedule yields the best performance across the two datasets considered.
Finally, Table f also includes the baseline that trains directly at / 1000 without subsequent training. This baseline performs substantially worse both datasets, highlighting that the subsequent train is critical for successful learning at long horizons.
A.2 Extended Time Series
In this section, we provide qualitative examples of generated time series by our model for each of the five datasets used in evaluation: Electric Motor, ECG, ETT, Sine, and MetroPT3. All samples were
20

---

Published in Transactions on Machine Learning Research (03/2026)
generated with fixed sequence length of 5000, using model weights trained on sequences up to / 1000. This allows to assess the model's ability to generalize and synthesize plausible data beyond the training horizon.
The results illustrate how well the model maintains the structure of the original data when generating extended sequences:
For time series with higher quasi-periodicity (Electric Motor, ECG, and Sine), the key patterns continue be synthesized plausibly beyond the training length. In these cases, a stable state emerges, characterized by repeating, but not identical, patterns (see Figures and 9h.
In the Sine datasel, sinusoidal curves are extended effectively, with only a slight reduction in ampli- tude observable in some channels. (Figure
For the less quasi-periodic time series (ETT and MetroPT3), clear degradation in synthesis quality is observed beyond the trained length. In both cases, the model produces repetitive, flatline-like like patterns with low variation, and characteristic structures are no longer preserved (Figures and 10
These qualitative results support the quantitative findings and further highlight the model's ability to gen- eralize well on quasi-periodical data, while revealing its limitations on more dynamic datasets.
pluse careet
phase
by plass Carrent
3050
3050
1,000
3002
4000
UNL.OLt
3.50
3002
Intermocdiate.
mntormotrrrrtt stator
3000
Figure 6: Example of generated time series sample of length / = 5000 from the Electric Motor dataset. The model trained on sequences up to = 1000. The main characteristies of the dataset continue to be well synthesized in the extended sample. During generation, the model reaches a stable state in which the output patterns kind of repeat. As a result, slower trends, especially visible in the leff motoreurr stator fad coord channel, are not fully reflected in the synthesis.
ECG2
1000 3000
2000
4000
5000
1033
2000
3000
4000
5000
Figure 7: Example of generated time scries sample of length = 5000 from the two-channel ECG datasct. The model was trained on sequences up 1000. The key characteristics of the data, particularly the heartbeat-like patterns across both channels, continue be well synthesized in the extended sequence. Still, stable state emerges, with periodic patterns that, while not identical, remain strongly similar over time.
21

---

Published in Transactions on Machine Learning Research (03/2026)
HUFL
HULL
NUPL
2820
2000
3050
4000
5003
1000
3000
3003
4020
5000
1000
2082
5000
4332
5000
MULL
IUPL
U.LL
Phonghhw
2020
2000
3050
4000
5002
1003
2000
3003
4020
5000
1000
2082
3000
4332
5000
OT
1000 2000 3000
4000
5000
Figure Example of a generated time series sample of length = 5000 from the ETT datasct. The model trained sequences up 1000. Up this length, the synthesis closely follows the patterns present in the original data. Beyond this point, stable state emerges. Most channels no longer reflect the dataset's characteristic patterns, though the "OT" channel still produces plausible structures.
2000
2000
3000
1000
5002
1000
3000
3000
4000
5000
1000
2000
2000
4000
5000
3.003 3220
3000
4020
5,000
1000 2000
3000
4000
2000
Figure 9: Example of a generated time scries sample of length 1 5000 from the Sine datasct. The model was trained on sequences up to I 1000. The sine curves are extended very consistently beyond the trained length, maintaining the dataset's 's structure. Upon closer inspection, slight decrease in amplitude can be observed in channels 2 and compared to the initial segment (up to 1000).
22

---

Published in Transactions on Machine Learning Research (03/2026)
TP2
TP3
3020
2000
3050
4000
5000
1000
3000
3003
4020
5000
1000
2002
5000
4332
5000
DV. pressure
Lempersiure
3820
2000
3050
4000
5002
1003
2000
3003
4020
5000
1000
3082
3000
4332
5000
Notor curfent
1000 2000 3000
4000 3000
Figure 10: Example a generated time series sample of length 5000 from the MctroPT3 datasct. The model was trained on sequences up to / 1000. While the generation follows the original data up this length, meaningful structure is preserved in the extended part. Still, stable state emerges, with the model settling into repetitive, low-variation patterns resembling noisy flatlines across all channels. This behavior is expected, as the MetroPT3 dataset exhibits low quasi-periodicity
A.3 Ablation: Decoder inductive bias
To disentangle the effect of recurrence from the effect of our decoder inductive bias, we compare AEQ-RVAF- ST against control decoder that keeps the same recurrent backbone but removes the key constraints of our design (length-indepenndent parameterization and approximate time-shift equivariance). We train this control variant on all datasets and sequence lengths used in the main paper (I E {100, 300, 500, 1000}) and report FID scores. For both decoders, we annotate each layer with its input and output dimensions to make the data flow explicit.
MEQ-RVAE-ST decoder (repeat-vector + shared per-time-step projection). For reference, the AEQ-RVAE-ST decoder broadcasts the latent code to all time steps via Repeat Vector, decodes with stack of LSTMs, and applies the same linear output projection independently at each time step:
# R [20}
(n=1) LSTM (256, return_ (eruuneessquenee==Trreee x TimeDistributed (De
RepeatVector (n=1)
R [20) -> R^ x 20}
R-{1 x 20} R- x 256)
(1 256) -> R x d_c}
where d_ 20 is the latent dimension, da 256 the LSTM hidden dimension, and de denotes the number of channels (dataset-dependert) since the RepcatVector merely copics z along the time axis and the TimeDistributed layer applics the same weight matrix W E 256xxd at every time step.
This design is length-independent:
layer's parameterization depends l,
Control decoder (recurrent, but without equurarrancee/enntt-iideeeeddde control decoder maps z through dense layer and reshapes to length-/ sequence with 256 features per time step, followed by the same four-layer LSTM stack. In contrast. to AEQ-RVAF-ST, the output is produced via global projection that flattens all recurrent states and predicts the full sequence jointly:
constraints). The
# R [20]
Dense (1*256, relu)
R [20} -> R^{1#256}
Reshape (1, 256)) LSTM (256, return_sequeiie return_sequences=TTuu)) 4 Flatten () Dense (1*d_c)
R (1+256) -> R^ x 256)
R* 256) R' x 256) R' 256) -> (1*226))
R=(1=256) (^(1dd_c)
23

---

Published in Transactions on Machine Learning Research (03/2026)
Reshape ((1, d_c))
R [1+d_c -> R' d_c}
After flattening, the final dense layer has weight matrix W ((1250))((-d)) which can implement position- specific bubsolute-time-deree )ependenn))))) mappings. Its parameterization explicitly depends on the target length 1. This makes it suitable control: it preserves recurrence, but removes the specific decoder structure that enforces our intended inductive bias.
Table reports FID scores (lower is better) for AEQ-RVAE-ST and the RVAE control decoder across all datasets and sequence lengths. The control decoder can produce reasonable results on short horizons (notably on ECG at 100, where both models are comparable within confidence intervals), but its performance degrades strongly the sequence length increases. This degradation is particularly pronounced on ETT, Sine and MetroPT3 at I = 1000, while AEQ-RVAE-ST remains substantially more stable. Overall, these results suggest that the decoder constraints in REQ-RVAE-ST become increasingly important for maintaining sample quality on longer horizons.
Table 5: FID score of synthetic time series for the decoder ablation, comparing AEQ-RVAE-ST (ours; repeat- vector + time-distributed output) against the RVAE baseline decoder (standard LSTM-VAE style decoder). Scores are computed on the five datasets (see 4.1 at sequence lengths 1 = 100, / = 300, = 500, and -= 1000. Lower scores indicate better performance. Each score is based on 5000 generated samples; results are averaged over 15 independently trained models and reported with 1-sigma confidence intervals.
Sequence lengths
Dataset
Model
100
300
500
1000
Electric
AEQ-RVAE-ST (ours)
0.35++004
0.12+0001
0.10+0000
0.24+0.02
Motor
RVAE Control
0.81
0.74+00.07
1.44:0.15
1.65#t0.11
AEQ-RVAE-ST (ours)
0.0810.01
0.0910.02
0.14+0.02
0.4610.06
ECG
RVAE Control
0.0710.01
0.39
0.65#0.06
0.04
AEQ-RVAE-ST (ours)
0.5810.05
0.65
0.79:10.07
1.82:0.116
ETT
RVAE Control
0.65
1.13::0.09
2.56+0.21
0.97
AEQ-RVAE-ST (ours)
0.3310.04
0.34100.02
0.46:0.03
0.42:00.03
Sine
RVAE Control
1.37
1.52++0.13
9.93++0.4
11.5++0.49
AEQ-RVAE-ST (ours)
0.26+0.04
0.65++0.07
2.81
2.84+0.22
MetroPT3
RVAE Control
0.60+10.06
1.10:10.11
2.34+0.27
13.31
A.4 Power Spectral Density Analysis
We use power spectral density (PSD) analysis to evaluate how well AEQ-RVAE-ST captures frequency characteristics. Specifically, we ask two questions: (1) Does the model learn to generate time series with the correct PSD? (2) Does it hallucinate quasi-periodic structures that are not present in the training data?
The training data consists of 30,000 synthetic sequences sampled from a target PSD with two closely spaced Gaussian peaks at fi 0.08 and fz 0.12 (normalized frequency). The peaks have amplitudes A 1.0 noise floor of e 10-4 To generate each sequence, we
and A 0.9, width o 0.002, and sit compute the magnitude spectrum from the target PSD, add uniformly random phases, and apply inverse FFT. We then min-max scale the entire dataset to [-1,1]. Figures 11 and 12 show the PSD and example samples for both the original training data and TEQ-RVAEESTT-geeerale data across sequence lengths E (100,500 900, 10003.
For 1= 100, the generated PSD shows oscillatory artifacts at frequencies above f2. These oscillations decay in absolute value with increasing frequency, but their amplitude remains constant. The model also suppresses spectral content below fi, removing the noise floor 6. At I 500, the oscillatory artifacts become ronger
24

---

Published in Transactions on Machine Learning Research (03/2026)
Original Data
Gencrated Data (AEQ-RVAE-ST)
1 = 100
Average (30,000 Samples)
Average Samples)
10
101
10
10-
10-
10-
10-9
Frequency
Frequency
Original Synthetic
Sampled
1.0
1.0
0.0
0.0-
-0.5
-0.5
-1.0
-1.00
[ = 500
Average Train Data PSD (30,000 Samples)
Awerage Generated Data PSD (30,000 Samples)
101
101
10-4
M
10->
10-5
10-5
13'7
10-
10n9
10-9
0.0
0.1
0.3
0.5
0.0
0.3
6.5
Frequency
Frequency
Original Synthetic
Sampled
1.0
1.0
-0.5
-0.5
-1.0
160
200
360
-1.0
100
200
300
400
Figure 11: PSD and sample comparison for I 100 and - 500. Top row per sequence length: PSD. Bottom example samples.
25

---

Published in Transactions on Machine Learning Research (03/2026)
Original Data
Gencrated Data (AEQ-RVAE-ST)
1 900
Average (30,000 Samples)
Average Samples)
10
101
10
10-
10-9
W
10-9
10-9
Froquency
Frequency
Original Synthetic
Sampled
1.0
1.0
0.0
0.0
-0.5
-0.5
-1.0
= 1000
Avrrage Train Data PSD (30,000 Samples)
Awerage Generated Data PSD (30,000 Samples)
101
10-4
MMn
10->
W
10-5
10-5
13'7
10
10n9
10-9
0.0
0.1
0.3
0.5
0.0
0.3
0.3
6.5
Frequency
Frequency
Original Synthetic
Sampled
1.0
1.0
www.
-0.5
-0.5
-1.0
200
600
-1.0
200
400
600
800
Figure 12: PSD and sample comparison for 900 and Bottom example samples.
1000. Top row per sequence length: PSD.
26

---

Published in Transactions on Machine Learning Research (03/2026)
and now also appear below fi. The spectral content between fi and f2 is clearly reduced compared to the original, showing that the model isolates the two dominant peaks while filtering out surrounding frequencies. At [ = 900, these effects become more pronounced: non-dominant frequencies are further suppressed and the oscillatory artifacts are stronger across the spectrum. At = 1000, something different happens: f2 becomes weaker relative to f1, indicating that the model has difficulty preserving both frequency components at this sequence length. The oscillatory artifacts high frequencies also change character and no longer maintain constant amplitude.
The visual inspection of the samples confirms these findings. At = 100, the generated sample looks nearly identical to the original. At 1 = 500, one can see slight smoothing in the high-frequency components upon close inspection, matching the noise floor suppression in the PSD. At = 900, the generated samples begin to show slightly compressed peaks compared to the originals. At = 1000, this compression becomes clearly visible: both frequencies are still present, but the amplitude range of the generated samples noticeably reduced.
To summarize: AEQ-RVAE-S learns to generate time series that approximately match the target PSD, successfully capturing both dominant frequencies up to t= 900, though increasingly filtering out weaker spectral content. At I = 1000, the model struggles to preserve the relative amplitude of both peaks. We observe no hallucinated quasi-periodic structures the oscillatory artifacts in the PSD represent spectral leakage rather than spurious periodicities and do not appear as visible patterns in the generated samples.
A.5 Wall-clock training time comparison
Table 6: Training time (wall-clock) comparison for the evaluated models on the Electric Motor dataset sequence length I = 1000. For Diffusion-TS we report training time and (separately) the time to sample 10,000 sequences.
Model
Training time
Notes
AEQ-RVAE-ST (subsequent train)
8 h
progressive training with +100 length increments standard training
AEQ- REQ-RRVA-ST (no subsequent)
2 h
h 20 min
TimeVAE
11 min
Diffusion-TSS
2 h 30 min + 1 h +1 h for sampling 10,000 samples
WaveGAN
40 min
TimeGAN
20 d
Table 0 reports indicative wall-clock training times the Electric Motor dataset sequence length / 1000. These measurements are intended order-of-magiituuk estimates of computational cost and may vary with implementation details and system load; moreover, the runs were not performed on identical hardware.
Specifically, AEQ-RVAE-ST, TimeVAE, and WaveGAN were trained on workstation (Ryzen 9 5950X, NVIDIA RTX 3080), while the remaining models were trained on a DGX system (EPYC 7742, NVIDIA A100). For Diffusion-TS, we additionally report the time required generate 10,000 synthetic sequences after training.
For AEQ-RVAE-ST. distinguish standard training from subsequent training. In subsequent training, the sequence length is increased progressively (from 1 = 100 to 1 = 1000 in increments of 100), which increases overall runtime compared to training directly at / = 1000.
Overall, the results suggest clear runtime differences our setup. The fastest models are the convolution- basod TimeVAE and WaveGAN, whereas the recurrent models (AEQ-RVAE-ST and TimeGAN) require substantially longer training times. In addition, diffusion-based models require additional time for sample generation after training.
27

---

Published in Transactions on Machine Learning Research (03/2026)
A.6 Hyperparameters and Loss Function
In all experiments, for the encoder well as the decoder, we stack 4 LSTM-layers each with 256 hidden units. The latent dimension is 2 20. We use Adam optimizer with learning rate = 10-4, P1 0.9, B = 0.999, c = 10-2. We perform min-max scaling with (-1,1). After scaling we do a train/validatton split with a ratio of 9:1.
We use the loss function
La. = a SSE + B DKL,
(5)
where the reconstruction loss, SSE, represents the sum of squared errors, computed for each individual sample within batch:
SSE
ΣΣ
(the Dta)²
(6)
T
where 7 is the sequence length and C is the number of channels. We then average the SSE over the entire batch. In our experiments we set 500
T and B 0.1.
The parameter follows the 3- VAE framework (Higgins et al. 2017 :: for < B 1 the reconstructed samples are less smoothed, while B encourages disentangled representations (Burgess al.) 2018). We
adjust antiproportional to the sequence length retain the ratio between the reconstruction loss and the KL-Divergence.
A.7 Loss ELBO conversion
Given the Gaussian likelihood with variance o², the likeliocl can be expressed in terms of the SSE:
SSE -2o log-likelihood o² log (2Tα²) -T -
Using ELBO = log- ligeliielhhodd - DKL MMurphy
2022.
together with 5 and 0² = 0.5 8 a we derive:
Ce.
a
SSE + DKL.
3
3
B
log-likelihood - 0.5 log T
T CCDL
= ELBO a. B,T.C = Ca. 0.5 loo T.
(8)
B
We normalize the ELBO by the product of sequence length and number channels to enable comparison across datasots with different dimensionalitics:
ELBOuorn (Ca a, B,T,C) ELBO(Lo,s, T,T,T,
(9)
T.C
A.8 Evaluation by Average ELBO
For completeness, we evaluate the average Evidence Lower Bound (ELBO) on a synthetic dataset X e R", xixe where n, represents the mumbers of samples, 1 denotes the sequence length, and e the number of channels. We refer to this metric as (X). In detail, we first train a VAF model on shorter sequence lengths & & I, which facilitates easier training. Since this metric reflects short-term reconstruction quality only, is not used for model ranking in our main evaluation.
We then calculate the average ELBO:
Eavg(X) nl -)
ELBO norm (Lo.o(
(10)
i=Ul E=0
28

---

Published in Transactions on Machine Learning Research (03/2026)
Table 7: Average ELBO score (X) of synthctic time series for six models (sce 4.2). computed on the five datasets (see 4.1 at sequence lengths = 100, -= 300, = 500, and / = 1000. Higher scores indicate better performance. Fach score based on 1000 generated samples evaluated with an ELBO model using the AEQ-RVAE-ST architecture, with I-sigma confidence intervals. Note that while the ELBO score is generally informative, it can overestimate quality certain datasets such Sine and ECG, where implausible outputs may undetected. For the Sine dataset in particular, uncorrelated channels and high sensitivity to local artifacts limit the reliability of this metric. Overestimated duc to local consistency effects (flat lines).
Sequence lengths
Dataset
Model
100
300
500
1000
ABQ-RVAE-ST(ours)
1.62:10.69
1.65:10.60
1.66:10.03
1.65:10.03
TimeGAN
1.20:10.59
1.33::0.48
1.13#0.56
-4.55#2.41
Electric Motor
WaveGAN
1.54+-0.11
1.54+-0.16
1.54+0.11
1.53++0.37
TimeVAE
1.49:0.888
1.38+1134
1.09
0.31
Diffusion-TS
1.58#0.06
1.36+0.26
1.38=#022
0.30
Tine-Transfrmmr
0.98+2.46
3...9
-21.7=0.91
-28.4+4.12
AEQ-RVAE-ST((Ury)
1.6410.13
1.6410.18
1.6310.20
1.5910.27
TimeGAN
114.6+1.87
-14.6+1.41
-13.7+6.67
-15.3+2.57
WaveGAN
1.12::0.1
1.11+0.87
1.10
1.10
ECG
TimeVAE
1.55+0.37
1.37+0.65
1.26+0.70
0.87
Diffusion-TSS
1.65+1.00
1.64+0.19
1.60+0.29
1.29
Time- Tame-rrasformer
1.07=00855
1.68
1.68+0.055
1.68+0.055
ABQ-RVAE-ST(ours)
1.4910.52
1.5010.40
1.52
1.53:00.63
TimeGAN
1.39+00.0
3.35
4..29
-0.38+
WaveGAN
1.40
1.39
1.2200.51
1.4240.48
ETT
TimeVAE
1.47+00944
1.20+1.554
0.89
0.42+2.44
Diffusion-IF TS
1.50:10.18
1.49+0.26
1.50
1.50
Time-Transformet
1.07+11.33
1.38-
1.49-
-39.9+5.84*
AFQ-RRAAF-ST(uus))
1.42+0022
1.19
1.28+0.48
1.41+0.27
TimeGAN
20.99
-1.25+22.2
-2.33+3.21
-4.73+5.64
WaveGAN
2..28
-1.0441..76
0..7+1..7
-0.94
Sine
TimeVAE
1.0640.66
-3.5548.18
-6.21
-8.81+12.1
Diffusion-TS
1.50-10.06
1.14+0.53
0.56+1.09
-0.30+1.600
Time-Transformer
1.23=0.49t
-0.12+1.45*
1.18+0.871
1.34+0.65
AEQ-RVAE-ST(ours)
1.41#1.74
0.76+3.44
0.57
0.60+3.75
TimeGAN
1.28+
0.61+4.39f
1.46+
-11.1+18.2
WaveGAN
-1.71
-1.62.44.90
-1.6444.83
-1.68
MetroPTT3
TimeVAE
-0.07
-2.0615.911
-5.6417.29
-9.03
Diffusion- TS
1.6310.92
1.4312.21
1.3612.53
0.7743.50
Time- Time-Tranfformr
-2.30+5.81
-3.05
-2.97+0.55
-302+14.4
where Le.0 is the loss of the ELBO Model and ELBO norm ELBO (cl)-1 is normalized ELBO, explained in Appendix A.7 By normalizing the ELBO, we get a fairer comparison of datasets with different dimensionalities and varying sequence lengths.
Eavg( gives us information about short term consistency over the whole synthetic dataset. We chose 50 which half of the lowest sequence length in the experiments. A well trained ELBO model (An & Cho 2015) allows us evaluate the (relative) short term consistency of synthetic data in high accuracy and low variance. To ensure reliable assessment of sample quality, prevented overfitting of the ELBO
29

---

Published in Transactions on Machine Learning Research (03/2026)
computed on the five datasets (see 4.1 at sequence lengths of 100, = 300, 500, and 1000. A higher score indicates better performance. For each score, 1000 generated samples were evaluated by an ELBO model (based on the TimeVAE architecture) and the results are reported with 1-sigma confidence intervals. The interpretation must follow analogously to the explanation provided in Section A.8 of the main paper, where the specifics and limitations of the ELBO score are discussed in detail. Overestimated due to local consistency effects (flat lines).
Table 8: Average ELBO score (X) of synthetic time serics for six models (scc 4.2)
Sequence lengths
Dataset.
Model
100
300
500
1000
AEQ-RVAE-ST (ours)
1.61:10.9
1.64:10.12
1.64+0.01
1.64:10022
TimeGAN
1.29+0.39
1.33::0.17
1.21
22.14
Electric Motor
WaveGAN
1.52++0.14
1.47++1.05
1.52+0.22
1.52+-0.15
TimeVAE
1.52:10.87
1.44
1.01
0.10
Diffusion-TS
1.56+0.45
1.35+0.36
1.39
1.30
Tine-TTrnsformer
1.25+1.88
22.997..5
-5.44+8161
-22.7+8.05
AEQ-RVAE-ST (ours)
1.62:00.77
1.62:00077
1.62:0.06
1.5910.06
TimeGAN
-..57=0222
2.26
-2.67+1.92
-..8
WaveGAN
1.32:10.29
1.33
1.32
1.32
ECG
TimeVAE
1.57+0.15
1,46:0.16
1.39
1.08
Diffusion-TS
1.63:10.06
1.6310.08
1.60
1.16+25.2
Tane-Tranffrmmr
1.22+0.50
1.67=0004
1.67
1.67=0.04
AEQ-RVAE-ST (ours)
1.5610.24
1.5710.09
1.5910.05
1.60:0.13
TimeGAN
1.49
1.20+1.49
0.83+0.91
-0.00
WaveGAN
1.50+..50
1.50+0.41
1.47
1.4940.43
ETT
TimeVAE
1.56:0.44
1.11:0.81
1.15
0.40+22.06
Diffusion-IS TS
1.53+0.07
1.52+0.13
1.52#0.13
1.52
Time- Time-Transformer
1.43++0.52
1.57+00.11
1.48+0.04
-39.6+45.63
AFQ-RRAAF-ST(ous)
1.46++0.07
1.44+0009
1.45+0.06
1.47+0.04
TimeGAN
0.66+1.04
0.39
-0.19+1.59
-2.16+3.77
WaveGAN
0.29
0.50
0.66
0.60
Sine
TimeVAE
1.42+0.12
0.6642.38
0.0413.04
-0.81
Diffusion-TS
1.48-0.02
1.44
1.33+0.11
1.23+0019
Time-Transforma
1.4490.09
1.27+0.22
1.44+0.13f
1.46=0.10*
AEQ-RVAE-ST (ours)
1.49+0.64
1.38+0.77
1.39
1.3610.81
TimeGAN
1.33++0.77
0.95+1.833
1.42+0.84
-0.07+2.94f
WaveGAN
0.35
0.18+11.67
0.23
0.22+1.64
MetroPTT
TimeVAE
1.06::1.14
-0.07+2.007
-3.81
-3.61
Diffusion- TS
1.63:10.24
1.5810.49
1.5910.41
1.04+2.08
Time- Transformer
0.06+1.73
-0.97+22.21
-1.20
-331+26.2
model by applying carly stopping after 50 epochs without improvement and restoring the best weights. In our experiments, we employed two distinct ELBO models for calculating (xvg(X). The first model is based on the RVAE-ST architecture, while the second utilizes the TimeVAE framework (Desai et al. 2021a). The use of a TimeVAE-basso ELBO model provides an additional evaluation to ensure that the AEQ AEQ-RVAE. ST-based model is not biased toward our own generated samples. As detailed in Appendix A.9 the results obtained using TAmeVAF. highly similar to those produced by the AEQ-RVAF-ST--een model.
30

---

Published in Transactions on Machine Learning Research (03/2026)
The average ELBO measures short-term consistency on subwindows of length E and can therefore overesti- mate models that reproduce local statistics while failing to capture global dynamics. This effect is visible for the Time-Transfrrmer on Sine, ECG and ETT datasets and also on for TimeGAN on the MetroPTT dataset: Both models produce flat segments that, when evaluated on short windows, appear locally consistent with the training data and therefore inflate aagg yet they do not reflect the characteristic dynamics of the dataset. The mismatch is evident in our other scores and in the PCA and t-SNE embeddings, where these samples cluster away from the real data. Interpreted with this caveat, AEQ-RVAE-ST produces the best samples on all datasets starting at I = 300, with the exception of MetroPT3, where Diffusion-TS is performing best.
A.9 Average Elbo with TimeVAE Elbo-Moder
Table S shows the results for the average ELBO score {X) using the base of TimeVAE as the ELBO model. However, instead of using the original loss function of TAmeVAE, we utilized the loss function of AEQ-RVAF- ST as it simplifies the conversion to the ELBO score as shown in (S Analogous to Table our model is outperforming all other models from I = 300 with the exception of the ECG dataset where our model is outperforming from / = 500. On the other side, our model is outperforming Diffusion-TSS on the MetroPT3 dataset on I= 1000.
A.10
Baseline Model Details
This section provides detailed descriptions of the baseline models, including their architectural properties and equivariance characteristics. Implementation details and hyperparameters for each model are provided in Appendix A.11.I
GAN-based model that is considered state-of-the-art in generation of times series data. TimeGAN's gemerator has a recurrent structure like AEQ-RVAF-ST. A key difference is that it's latent dimension is equal to the sequence length. Notably, equivariance this model is lost on the output layer of the generator which maps all hidden states at once through a lincar layer to a sequence. On its initial paper release, TimeGAN was lested and compared to other models on a small sequence length of = 24.
TimeGAN Yoon et al. 2019:
WaveGAN (Donahue et al., 2019): A GAN-based model developed for generation of raw audio waveforms. WaveGAN's generator is based on convolutional layers. It doesn't 't rely on typical audio processing techniques like spectrogram representations and is instead directly working in the time domain, making it also suitable for learning time series data. It is designed to exclusively support sequence lengths in powers of 2, specifically 214 to 216. Notably, WaveGAN loses it's equivariance on dense layer between the latent dimension and the generator, however the generator itself completely maintains equivariance with its upscaling approach. In our experiments, it was trained with the lowest possible sequence length of 2 and the generated samples were subsequently split. to match the required sequence length. In YYonn al. 2019). WaveGAN was outperformed by TimeGAN on low sequence length.
TimeVAE Desai al.) 2021bb: A VAE-based model designed for time series generation using convolutional layers. Analogous to WaveGAN, it loses equivariance between the latent dimension and the decoder and additionally it loses equivariance on the output layer where a flattened convolutional output is passed through a linear layer. It has demonstrated performance comparable to that of TimeGAN.
Diffusion-TS (Yuan & Qiao 2024): A generative model for time series based on the diffusion process framework. It combines trend and scasonal decomposition with a Transformer-basod architecture. A Fourier basis is used to model seasonal components, while a low-degree polynomial models trends. Samples are generated by reversing learned noise-injection process. While the model leverages the global structure of sequences, it lacks time-translation equivariance: this is due both to the use of position embeddings in the Transformer component and to the fixed basis decomposition, which breaks shift-invariance.
Time- Transformer [Lu et al. 2024): An adversarial autoencoder (AAE) model tailored for time series generation, integrating a novel Time- Transformer module within its docoder. The Time-Transformer employs a layer-wise parallel design, combining Temporal Convolutional Networks (TCNs) for local feature extraction and Transformers for capturing global dependencies. A bidirectional cross-attention mechanism facilitates
31

---

Published in Transactions on Machine Learning Research (03/2026)
effective fusion of local and global features. While TCNs are inherently translation-equiivriann, this property is overridden by the Transformer's positional encoding and attention structure, making the overall model not equivariant.
None of these baselines enforce approximate time-shift equivariance by design. Figure illustrates that AEQ-RVAF-ST does induce this bias, which particularly relevant. for long-hoo norihorizon training.
A.11 Implementation details of baseline models
A.11.1 Hyperparameters and model configs
To balance data divensity and computational efficiency, we used & dataset-specific step size when splitting time series into training sequences. This step size determines the offset between starting points of consecutive sequences, thereby influencing both the number of training samples and the memory requirements during training.
For the Electric Motor, ECG, and MetroPT3 datasets, chose step size of 0.1 where is the sequence length. For the ETT dataset, which exhibits more complex and longer-range temporal dependencies, we used a smaller step size of 0.04 I to increase the number of training samples. In contrast, for the synthetic Sine dataset, we fixed the number of training samples to 10,000 for each sequence length.
This approach reflects practical trade-off: while smaller step sizes increase training data diversity, they also lead to higher memory usage. Particularly for long sequences, using very small stop sizes (c.g., step size 1) can cause GPU memory overflow or even exceed system RAM, depending on the model architecture, implementation and dataset.
A.11.2 TimeGAN
We did all experiments with the same hyperparameters. Num layers=3, hidden dim=100, num iterations = 25000. The clockwise computation time on these hyperparameters were the highest of all models. We use the authors original implementation on Nvidia DGX A100 server in the 19.12-ffl-py3 container On sequence length 1000, the training took about 3 weeks wall-clock time.
A.11.3 WaveGAN
For WaveGan needed special preparation to be usable for training. First we min maxed scaled the dataset file, split it into training and validation parts and then converted each into a n-dimensional waav file, WaveGan is limited in configurability. In terms of sequence length the user decide between 244, 2 and 2 We chose 2 = 16384 because it is the smallest possible length. When we generate samples, cut them into equal parts which correspond to the desired sequence length 1. The rest of the hyperparameters were set to default. On the sine dataset training, wie created 10,000 samples with a length of 16,384. We used the ported pytorch implementation"
A.11.4 TimeVAE
We use TimeVAE with default parameters. We integrated components of the original TimeVAE implementa- tiou" such as the encoder, decoder, and loss function, into our own program framework. The reconstruction loss of TimeVAE
ΣΣ (Ut 0he)2
1 Σ (Te 0.)² 0
(11)
19.1 html
http:////gthbbb.cmmmmmmsstaeeeee http:///wwttttteelaaaaa///aaaana wavegan-pytorch
7 httper///ehhuboom///uudess///// ttme/AA
32

---

Published in Transactions on Machine Learning Research (03/2026)
TimeVAE includes hyperparameter a, which acts weighting factor for the reconstruction loss. The authors of the original paper recommend using value for a in the range of 0.5 to 3.5 to balance the trade-off between reconstruction accuracy and latent space regularization. In all of our experiments, we set a = 3.
A.11.5 Time- Transformer
We used the official implementation with default parameters. The encoder is 3-layer ID CNN (filters (64, 128, 256], kernel size 4, dropout 0.2). The decoder uses TimeSSoomer--asee architecture (head size 64, 3 heads, two transposed convolution layers with filters [128, 64), kernel size 4, dilations [1, 4], dropout 0.2). The discriminator is an MLP with hidden dimension 32. All three components use polynomial decay learning rate schedules (power = 0.5, 300 steps): autoencoder 0.005 - 0.0025, discriminator and generator 0.001 0.0001.
A.11.6 Diffusion-TS
We use the official implementation"" For all datasets except Sine, we use unified configuration: 3 encoder layers, decoder layers, model dimension d = 64, 500 diffusion timesteps, 4 attention heads, MLP expansion factor 4, kernel size 1, no dropout, L1 loss with cosine beta schedule. For the Sine dataset, we use the authors' provided configuration.
A.12 Discriminative Score
The 2-layer RNN for binary classification consists a GRU layer, where the hidden dimension set to [no/2], where ne is the number of channels. This is followed by a lincar layer with an output dimension of one. To prevent overfitting, early stopping with a patience of 50 applied. We cach discriminative score we repeated 15 training procedures. On each procedure, 2000 random samples were used the train dataset and 500 samples were used the validation dataset for early stopping monitoring. The discriminative score is then determined by validating further independent 500 samples.
A.13 PyTorch vs TensorFlow
Our experiments use TensorFlow. A PyTorch reimplementation initially showed worse performance, which we traced to differing default weight initializations in LSTM and Dense layers. After aligning both frameworks to use uniform initialization, results were consistent.
A.14 PCA and t-SNE Results
This section presents PCA and t-SNE plots for all datasets at sequence lengths ( 100 and / = 1000. The EM and ECG PCA plots shown in the main text (Figure are not repeated here. Since TimeGAN and WaveGAN show consistent performance across sequence lengths within a given dataset, these observations will not explicitly mentioned in each figure caption.
33

---

Published in Transactions on Machine Learning Research (03/2026)
ESNE
ESNE
CSME
CENE
AEQ- RVAF-ST (ours)
TimeGAN
WaveGAN
TimeVAE
ESME
1558
1544
Diffusion-TD
Time- Transformer
EM, /-100
EM, 1000
ECG, 1=100
ECG, 1=1000
Figure 13: t-SNE plots for sequence lengths 100 and 1000 on the Electric Motor (EM) and ECG datascts. At 1= 100, TimeGAN already performs worse than the other models on both datasets, similarly to Time-Transformer AEQ-RVAE-ST, WaveGAN, and Diffusion-TS perform similarly. TimeGAN and Time- Transformer fail to generate coherent samples at longer sequence lengths on both datasets.
At I 1000, REQ-RV shows the best performance
ECG, while on EM,
34

---

Published in Transactions on Machine Learning Research (03/2026)
PCA
PCA
CSNE
CNNE
AEQ- RVAE-ST (ours)
TimeGAN
PCA
CSRE
LSNE
WaveGAN
TimeVAE
PCA
PCA
1.54
ESEE
Diffusion-IT TS
Time- Transformer
PCA, 1=10
PCA, 1000
t-SEE 1=100
t-SNE, 1=1000
Figure 14: PCA and t-SNE plots for sequence lengths = 100 and 1 1000 on the ETT dataset. AEQ- RVAE-ST and Diffusion-TS consistently perform the best across all sequence lengths. WaveGAN fails to capture the full variance of the dataset. TimeVAE performs similarly to AEQ-RVAE-ST and Diffusion-TT - at 7 100, but its performance degrades with increasing sequence length. TimeGAN and Time Transfoorrr perform reasonably well at / = 100, though already worse than the other models, and their performance significantly drops at / 1000.
35

---

Published in Transactions on Machine Learning Research (03/2026)
PCA
PCA
SSME
CSNE
AEQ- RVAF-ST (ours)
TimeGAN
FFA
FFA
WaveGAN
TimeVAE
PCA
PCA
1:54E
Diffusion-Te TS
Time-
Transformer
PCA, 1-10
PCA, 10100
t-SNE, (=100 t-SNE, 1=1000
Figure 15: PCA and t-SNE plots for sequence lengths 100 and 1000 on the MetroPT3 dataset. AEQ-RVAE-T TimeVAE and Diffusion-TS perform similarly and the best across all sequence lengths. WaveGAN performs slightly worse, it does not capture the entire distribution of the dataset. Time- Transformer performs reasonably well at I = 100, but its performance degrades at longer sequence lengths. TimeGAN consistently performs the worst, failing to generate plausible samples.
36

---

Published in Transactions on Machine Learning Research (03/2026)
PCA
PCA
CSNE
CSNE
AEQ- RVAE-ST (ours)
TimeGAN
PCA
CSSE
CSNE
WaveGAN
TimeVAE
PCA
154E
SSE
Diffusion- TS
Time- Transformer
PCA, 1-10
PCA, 1000
t-N,E, 1-100
t-SNE, 1=1000
Figure PCA and t-SNE plots for sequence lengths 100 and 1000 on the Sine dataset. At == 100, all models perform similarly well, except TimeGAN which consistently performs less effectively, At
= 1000, Time- Transformer also shows decline in performance. AEQ-RVAE-ST, Diffusion-TS. WaveGAN and TimeVAF perform equally well at both sequence lengths. However, when looking at Figure 5 this does not fully reflect the models performance, as the limitations in accounting for temporal dependencies lead to significantly reduced elfectiveness.
37


Reviews:
[{"review_id": "iJI2lus5Ci", "reviewer": "Reviewer_8g3S", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors address the problem of training generative models for time series which may exhibit periodic, or approximately periodic, behavior. One of the main challenges of this problem is both the length and the amount of series required to effectively capture this structure, and the inductive biases that may help or hinder the ability to capture this type of temporal structure. The authors propose a Recurrent Variational Autoencoder Subsequent Train (RVAE-ST) method, which combine specific choices of inductive biases, network topologies, and a training scheme, to address this problem. To illustrate the effectiveness of their approach, they perform experiments on five benchmark datasets, and report both qualitative and quantitative metrics.", "claims_and_evidence_yn": "No", "claims_explanation": "One of my main concerns is the property of \"quasi-periodicity.\" Although the concept is fairly intuitive, the authors do not provide a proper, working mathematical definition for it, and thus it is difficult to assess to which degree the proposed architecture captures \"quasi-periodicity.\" The authors do perform comparisons between different methods in relevant datasets. However, these comparisons use metrics that may or may not reflect the successful approximation of quasi-periodicity. For this reason, I believe that it is necessary to propose a metric to assess \"quasi-periodicity\" to justify the claims of the paper. \n\nTo further explain my point, one could ask whether there are _degrees_ of quasi-periodicity. If there are, one could quantify what is the degree of quasi-periodicity exhibited in each dataset, and thus determine if some of them are more challenging to analyze. Another aspect is the relation between quasi-periodic structures and sequence length: when do quasi-periodic structures become noise-like as the sequence length increases? Finally, some quasi-periodic structures can be an artifact of sequence length. For instance, consider the function\n\n$$\n\tu(t) = \\cos(2\\pi f_0 t + \\epsilon \\cos(2\\pi f_1 t))\n$$\n\nfor $\\epsilon$ small. Its instantaneous frequency is\n\n$$\n\tf_0 - 2\\pi \\epsilon \\sin(2\\pi f_1 t)\n$$\n\nand therefore there are _periodic variations_ of the frequency around $f_0$. If one observed the time series over an interval of length smaller than $1/f_1$ then this periodic variation could plausibly be interpreted as quasi-periodicity. However, if it is observed over an interval of length much larger than $1/f_1$, then it becomes a periodic structure.\n \nAs an aside, the concepts of _quasiperiodic functions_ and _almost periodic functions_ do exists in the mathematical literature. Although their definition may be too abstract, perhaps they could be helpful to define suitable quantitative metrics to determine a degree of \"quasi-periodic-ness.\"", "audience_yn": "Yes", "audience_explanation": "I believe that the problem of modeling quasi-periodic structures is relevant both to the readers of TMLR and beyond. As the authors correctly point out, there are medical applications where the _deviation_ from a periodic behavior is of clinical interest. Although generating such time series does not seem, to my knowledge, to have a direct application, being able to successfully generate them suggests that one has identified a proper set of parameters that describe a quasi-periodic structure.", "requested_changes": "My requests are as follows.\n\n1. Perhaps my main request is to provide a workable definition for quasi-periodicity that enables principled comparisons between methods. Otherwise it is difficult to assess which structural features are being learned. \n2. Although the authors perform extensive experiments in relevant datasets, there is one simple synthetic experiment that perhaps could illustrate the behavior of the architecture: generating synthetic time series with prescribed Power Spectral Density (PSD). Although this type of process generates statistically stationary time series, a typical sample path may exhibit quasi-periodic structures. It would be interesting to see if the architecture, depending on sequence length, learns to generate time series with the same PSD or if it hallucinates quasi-periodic structures.", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "There are no additional comments.", "extra_scores": {}, "extra_text": {}}, {"review_id": "emYSTsCTw5", "reviewer": "Reviewer_jerd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a generative model for long, quasi-periodic time series called RVAE-ST (Recurrent Variational Autoencoder with Subsequent Training). The authors argue that while Transformers are currently popular, they lack the necessary inductive bias (time-shift equivariance) for modeling quasi-periodic sensor data (e.g., motors, ECGs).\n\nThe main contributions are:\n\n1. Architecture: A recurrent VAE topology that employs a \"repeat-vector\" mechanism and time-distributed output layers to enforce parameter independence from sequence length and approximate time-shift equivariance.\n2. Training Scheme: A \"Subsequent Training\" curriculum that progressively increases sequence length during training (e.g., from 100 to 1000) to mitigate the optimization instability of LSTMs on long horizons .\n3. Theoretical Analysis: An application of the Echo State Property (ESP) as a diagnostic tool to demonstrate that the trained model effectively balances forgetting initialization artifacts with retaining input-driven structure.\n4. Empirical Evaluation: Comparisons against five baselines (including TimeGAN, TimeVAE, and Diffusion-TS) across five datasets, showing superior performance on periodic data using Context-FID and Discriminative scores.\n\n## Strength\n\n1. The hypothesis regarding inductive biases (recurrence vs. positional encoding) is well-motivated and theoretically grounded.\n2. The progressive training scheme effectively addresses the well-known vanishing gradient problem in LSTMs for this specific task.\n3. The analysis of the Echo State Property provides interesting insights into how RNNs model stationarity.\n\n## Areas for Improvement:\n\n1. Baseline Configuration: The comparison would benefit from ensuring baselines are tuned to the same degree as the proposed method to fully validate the performance gains .\n2. Implementation Details: The specific configuration used for WaveGAN (training on very long sequences) may unintentionally handicap its performance.\n3. Metric Interpretation: There should be a clarification regarding discrepancies between the quantitative metrics (Discriminative Score) and the visual quality of the samples.", "claims_and_evidence_yn": "No", "claims_explanation": "The paper presents a compelling hypothesis, but the current empirical evidence does not yet fully support the claim of superiority over state-of-the-art baselines. I believe addressing the following points would make the evidence much more convincing:\n\n1. Baseline Hyperparameters: The authors note that they did not perform extensive hyperparameter tuning and used the default hyperparameters across models to ensure comparability . However, different architectures (e.g., GANs vs. Diffusion) often require very different settings to converge optimally. Without basic tuning of the baselines, it is difficult to determine if the performance gap is due to the proposed architecture or simply suboptimal baseline configurations.\n\n2. WaveGAN Implementation: The choice to train WaveGAN on sequences of length 2^14 (16,384) due to implementation constraints , while the target task is only length 1000, likely places this baseline at a severe disadvantage. This significantly increases the complexity of the learning task for WaveGAN compared to the other models.\n\n3. Metric Reliability: There appears to be a contradiction between the metrics and the visuals. For example, on the Sine dataset, Diffusion-TS achieves a significantly better Discriminative Score (0.021) than the proposed RVAE-ST (0.122), yet the visual samples in Figure 5 suggest RVAE-ST is far superior . This raises questions about whether the Discriminative Score is a reliable proxy for quality in this specific context.", "audience_yn": "Yes", "audience_explanation": "The paper tackles a relevant and practical problem of generating long-horizon sensor data.\n\n1. The argument for revisiting RNNs for specific data types (quasi-periodic) challenges the current dominance of Transformers and contributes valuable nuance to the field's understanding of inductive biases.\n2. The progressive training curriculum is a generalizable technique that could be very useful to practitioners working with recurrent models in other domains .", "requested_changes": "I believe the following adjustments would significantly strengthen the paper and ensure the claims are robust:\n\n1. Clarification on Baselines (Critical): To remove any doubt regarding the superiority of RVAE-ST, could the authors please perform a basic hyperparameter search (e.g., learning rate, latent dimension) for the key baselines (TimeGAN, Diffusion-TS)? Demonstrating that RVAE-ST outperforms even tuned baselines would make their claim stronger.\n\n2. Refining the WaveGAN Comparison (Critical): I strongly suggest adapting the WaveGAN implementation to support a sequence length closer to the target (e.g., 1024) rather than 16384. If this is not feasible, please add a discussion or ablation clarifying how this discrepancy might impact the results. This ensures the baseline represents the method's potential rather than a limitation of a specific codebase.\n\n3. Insight into Metrics (Critical): Could the authors please elaborate on why Diffusion-TS achieves a superior Discriminative Score on the Sine dataset despite the noisy visuals?. A brief discussion on the limitations of this metric for this specific task would help readers interpret the results more accurately.\n\n4. Curriculum Sensitivity (Strengthening): The \"Subsequent Training\" scheme is a valuable contribution, but the current step size (100) and start length (100) are heuristic. To demonstrate robustness, could you provide a sensitivity analysis? Specifically, does the method still work effectively if the step size is larger (e.g., +200 or +500)? This would confirm if the curriculum is a generalizable method or requires precise tuning.\n\n5. Architectural Inductive Bias (Strengthening): The paper claims the performance gain comes from the specific \"approximate equivariant\" architecture (Repeat Vector + Time-Distributed decoder). To isolate this benefit, could you perform an ablation comparing your proposed decoder against a standard LSTM-VAE decoder (where z initializes the hidden state)? This would verify that the improvement is indeed due to the specific inductive bias and not just the use of LSTMs.", "broader_impact_concerns": "No concerns. The work is methodological and focuses on sensor data generation, which presents minimal ethical risk.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "77mz1tq9Y4", "reviewer": "Reviewer_5aBU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes RVAE-ST, a recurrent variational autoencoder tailored for long, quasi-periodic time series generation. The core contribution is a principled combination of (i) an approximately time-shift-equivariant recurrent architecture, (ii) a sequence-length–independent parameterization, and (iii) a progressive training scheme that gradually increases sequence length to stabilize optimization and enable learning over long horizons.\n\nConceptually, the paper reframes recurrent VAEs through the lens of approximate stationarity and time-shift equivariance, arguing that many real-world sensor signals are best modeled by emphasizing relative temporal dynamics rather than absolute time indices. The authors further connect this inductive bias to the Echo State Property (ESP), using it as a diagnostic tool to analyze state forgetting and the balance between stability and long-term dependency retention.\n\nEmpirically, the model is evaluated on five benchmark datasets spanning highly quasi-periodic (sine, ECG, electric motor) to more irregular sensor data. Across multiple metrics—including ELBO, Context-FID, discriminative scores, and latent-space visualizations—RVAE-ST consistently matches or outperforms strong baselines, particularly at long sequence lengths where many competing models degrade.\n\nKey strengths\nA clear and well-motivated inductive bias aligned with quasi-periodic and approximately stationary time series.\nA simple yet effective progressive training strategy that substantially improves stability and performance on long sequences.\nComprehensive experimental evaluation across diverse datasets and architectural families.\nInsightful use of ESP as an analytical lens rather than a purely theoretical claim.\n\nPotential weaknesses\nThe progressive training schedule is heuristic and dataset-dependent, with limited guidance on principled selection.\nGains are most pronounced for quasi-periodic data, while advantages on more irregular datasets are sometimes marginal.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s central claims—namely that approximate time-shift equivariance combined with progressive sequence-length training improves recurrent generative modeling of long, quasi-periodic time series—are well supported by both empirical and analytical evidence.\n\nThe experimental section is thorough and carefully designed. Performance is evaluated using complementary metrics (ELBO, Context-FID, discriminative score, PCA/t-SNE), reducing reliance on any single measure. The ablation study comparing conventional training with the proposed progressive scheme is particularly convincing, showing consistent and statistically significant improvements across all datasets.\n\nIn addition, the ESP analysis provides a clear mechanistic explanation for the observed behavior, demonstrating how training balances state forgetting with preservation of meaningful temporal structure. Importantly, the authors avoid overclaiming: they explicitly acknowledge that ESP is used diagnostically rather than as a formal guarantee, which strengthens the credibility of the analysis.\n\nOverall, the evidence is accurate, clearly presented, and appropriately scoped to the claims made.", "audience_yn": "Yes", "audience_explanation": "The paper is likely to be of interest to a broad segment of the TMLR audience working on representation learning, generative modeling, and sequential data. In particular, researchers concerned with long-horizon time series, inductive biases in neural architectures, or stability issues in recurrent models will find the proposed approach and analysis valuable.\n\nBeyond the specific model, the paper contributes a conceptual framing—approximate equivariance and ESP-based diagnostics—that is applicable beyond VAEs and could inform future work on recurrent diffusion models or other generative frameworks. Its emphasis on simplicity and computational efficiency further increases its practical relevance.", "requested_changes": "1. Provide clearer guidance or heuristics for choosing the initial sequence length and increment schedule in the progressive training scheme, potentially supported by additional ablations or sensitivity analysis.\n2. Clarify and streamline the discussion of equivariance in baseline models to better distinguish architectural properties from empirical behavior.\n3. Expand discussion on failure modes or scenarios where the inductive bias may be mismatched (e.g., strongly non-stationary or event-driven time series).\n4. Consider reporting wall-clock training time or computational cost comparisons to further support claims of efficiency.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)