iHyperTime: Interpretable Time Series Generation with Implicit Neural Representations
Authors: Elizabeth Fons
Alejandro Sztrajman
Yousef El-Laham
Andrea Coletta
Alexandros Iosifidis
Svitlana Vyetrenko
Venue: Accepted by TMLR ()

Abstract: Implicit neural representations (INRs) have emerged as a powerful tool that provides an accurate and resolution-independent encoding of data. Their robustness as general approximators has been shown across diverse data modalities, such as images, video, audio, and 3D scenes. However, little attention has been given to leveraging these architectures for time series data. Addressing this gap, we propose an approach for time series generation based on two novel architectures: TSNet, an INR network for interpretable trend-seasonality time series representation, and iHyperTime, a hypernetwork architecture that leverages TSNet for time series generalization and synthesis. Through evaluations of fidelity and usefulness metrics, we demonstrate that iHyperTime outperforms current state-of-the-art methods in challenging scenarios that involve long or irregularly sampled time series, while performing on par on regularly sampled data. Furthermore, we showcase iHyperTime fast training speed, comparable to the fastest existing methods for short sequences and significantly superior for longer ones. Finally, we empirically validate the quality of the model's unsupervised trend-seasonality decomposition by comparing against the well-established STL method.

Published in Transactions on Machine Learning Research (08/2024)
Hyme Interpretable Time Series Generation with Im- plicit Neural Representations
Elizabeth Fons J.P. Morgan AT Research
Seiabbthh.ons@@immmppe
Alejandro Sztrajman
University of Cambridge
Yousef El-Laham J.P. Morgan Al Research
yousee.llalamm@@jjoorggg.co
Andrea Coletta Bank of Italy
andrea.coletta coletto Dolettt@honnaditalii....
Alexandros Iosifidis
abdcece. aa, dk
Aarhus University
Svitlana Vyetrenko
svillana svillanaa eyetrenko @jpmoryan.ccmm eycrrreeeeeeeooiooorrooooommm
J.P. Morgan Al Research
Reviewed on OpenReview: https: openreview. net/ forum? id= CSnGPgeo55
Abstract
Implicit neural representations (INRs) have emerged as a powerful tool that provides an accurate and resolution- inaatiionnndeeeddeent encoding data. Their robustness as general approximators has been shown across diverse data modalities, such as images, video, audio, and 3D scenes. However, little attention has been given leveraging these architoctures for time series data. Addressing this gap, we propose an approach for time series generation based two novel architectures: TSNet, an INR network for interpretable trend-seasonality time series representation, and iHyperTime, hypernetwork architecture that leverages TSNet for time series generalization and synthesis. Through evaluations of fidelily and usefulness metrics, we demonstrate that iHyperTime outperforms current state-of-the-art methods in challenging scenarios that involve long or irregularly sampled time series, while performing on par on regularly sampled data. Furthermore, we showcase iHyperTime fast training speed, comparable to the fastest existing methods for short sequences and significantly superior for longer ones. Finally, empirically validate the quality of the model's unsupervised trend-seasonality decomposition by comparing against the well-established STL method.
1 Introduction
Modeling time series data has boen a key topic of research for many years, constituting crucial component wide variety of areas such as climate modeling, medicine, biology, retail and finance (Lim & Zohren (021]) Traditional methods for time series modeling have relied on parametric models informed by expert knowledge.

---

Published in Transactions on Machine Learning Research (08/2024)
However, the development of modern machine learning methods has provided purely data-driven techniques to learn temporal relationships. In particular, neural network-based methods have gained popularity in recent. times, with applications to wide range of tasks, such as time series classification (Ismail Fawaz et al. 2020
clustering [Algahtani et al. 2021), segmentation (Zeng et al.] 2022), anomaly detection (Choi et al.) 2021
upsampling (Oh et al. 2020). impulation (Cao et al. 2018 Shukla & Marlin 2021). forecasting (Torres et al.
2021) and generation (Coletta et al. 2023). In particular, generation of synthotic time scries has recently gained attention due to the large number of applications in medical and financial fields, where data cannot be shared, either due to privacy reasons proprietary restrictions (Jordon et al.) 2021 Assefa et al. 2020 Moreover, synthetic time series can be used to augment training datasets to improve model generalization on downstream tasks, such as classification (Fons et al. 2021), forecasting and anomaly detection. In these fields, having a disentangled representation of time series can be critical for applications with regulatory focus, which often require transparency and interpretability of proposed machine learning solutions as well as injection of expert knowledge as constraints into the training process (Vyetrenko & Xu 2019).
The task of generating realistic time-series data poses considerable challenges, particularly due to the diverse nature of time series, which may vary along dimensions such univariate vs. multivariate, short vs. long, and regularly vs. irregularly sampled. Although numcrous solutions have been proposed to address the problem of time-series data generation (Alaa et al. 2021 Yoon et al. 2019 Esteban et al. 2017), the focus has predominantly been on generating data for well-structured scenarios, such as short and regularly sampled time series. Such an approach is often conflicting with the complexities of real-world time-series data, where irregularitiss missing values and diverse sequence lengths are commonplace (Fang & Wang 2020). Recent methods have addressed some of these shortcomings (Jeon et al., 2022 Zhou et al.] 2023 Coletta et al.) 2023 However, no single approach has shown consistent performance across all these challenging sccnarios. Furthermore, the scalability of many existing methods to longer sequences is constrained by escalating computational costs that correlate with series length.
In recent years, implicit noural representations (INRs) have gained popularity as an accurate and ficxible method to parameterize signals from diverse sources, such as images, video, audio and 3D scene data (Sitzmann et al. 2020b Mildenhall al.] 2020). Conventional methods for data encoding often rely on discrete representations, such as data grids, which are limited by their spatial resolution and present inherent. discretization artifacts. In contrast, INRs encode data in terms of continuous functional relationships between signals, and thus are uncoupled to spatial resolution. In practical terms, INRs provide a data representation framework that is resolution-independdee challenges in real-world time scries. While there have been a few recent works exploring the application of INRs to time series data (Jeong & Shin 2022 Woo el. al. 2023), , there no work on leveraging these architectures for generating synthetic time series.
which makes them ideally suited to deal with the aforementioned
In this paper, we propose novel method for time series generation based on two novel architectures: 1) TSNet, INR tailored for aesoostion-agnostic encoding of time series data, offering seead--sasonalltyyreesiii disentangling of single time series. 2) iHyper
hypernetwork architecture for generalization of time series datasets, that leverages TSNet to produce interpretable latent representations of the signals. Together, these architectures form a unified approach for disentangled representation and generation of multiple forms of time series data, including challenging cases such as multivariate, irregularly sampled, and long time series.
Generation quality Through empirical evaluations, we demonstrate that iHyperTime out performs existing thate-of-the-rtt methods for time series generation. Our method excels in complex scenarios such as irregularly sampled or long sequences, while performing on par with stato-of-the-art for regularly sampled time scries.
Efficiency We show that our architecture achieves rapid training speeds, comparable to the fastest methods for short sequences, and substantially faster for longer ones.
TSR Decomposition We validate the unsupervised decomposition capabilities of our method by bench- marking it against the widely-used STL decomposition technique.
2

---

Published in Transactions on Machine Learning Research (08/2024)
2
Related Work
Implicit Neural Reprosentations Implicit Neural Representations (INRs) provide continuous represen- lation of multidimensional data, by encoding a functional relationship between inpul coordinates and signal values, avoiding possible discretization artifacts. They have recently gained popularity in visual comput- ing [Meschoder et al. 2019 Mildenhall et al. 2020j duc to the key development of positional encodings (Tancik
ct al. 2020) and SIREN periodic activations (Sitzmann et al., 2020b), which have proven to be critical for the learning of frggufrrquency details. Whilst INRs have been shown to produce accurate reconstructions a wide variety of data sources, such as video, images and audio (Sitzmann et al.) 2020b Chen et al., 2021 Rott Shaham et al.) 2021), few works have leveraged them for time series representation (Jeong & Shin Woo et. al. 2023), and none have focused on interpretability and generation.
2022
Hypernetworks Hypernetworks are neural network architectures that are trained to predict the parameters of secondary networks, referred to as hyponctworks (Ha et al.] 2017 Sitzmann et al. 2020a).. In the last few years, some works have leveraged different hypernetwork architectures for the prediction of INR weights, in order to learn priors over image data (Skorokhodov et al. 2021) and 3D scene data (Littwin & Wolf 2019 Sitzmann et al., 2019: Sat.rajma et. al. 2021). Sitzmann al. (2020b) leverage a set encoder and hypernetwork decoder to learn prior over SIRENs encoding image data, and apply it for image in-painting.
Time Scries Generation Synthesis of time series data using deep generative models has been previously studied in the literature. Examples include the TimeGAN architecture (Yoon et al. 2019), GT-GAN (Jeon
lt al.) 2022), and QuantGAN (Wiese et al.) 2020). More recently, as an alternative to -aseb time series generation, Desai et al. (2021) proposed TimeVAE, based on a variational autoencoder, while Coletta et al. 2023) proposed a diffusion model architecture called DiffTime. Alaa et al. (2021) introduced Fourier Flows, a normalixing flow model for time series data that leverages the frequency domain representation, which is currently considered together with TimeGAN as state-of-the-art for time series generation. In image
the last few years, multiple methods have used INRs for data generation, with applications synthesis (Skorokhodov et al., (021), super-resolution (Chen et al. 2021) and panorama synthesis (Anokhin ct al. 2021). However, there are currently no applications of INRs on the generation of time series data.
Interpretable Time Series Seasonal-trend decomposition is a standard tool in time series analysis. The trend encapsulates the slow time-varying behavior of the signal, while seasonal components capture periodicity. These techniques introduce interpretability in time scries, which plays an important role in downstream tasks such as forecasting and anomaly detection. The classic approaches for decomposition are the widely used
STL algorithm (Cleveland et al., 1990), and its variants (Wen et al. 2019 Bandara et al.] 2022). Relevant to this work is the recent N-BEATS architecture (Oreshkin et al. 2020). a deep learning-best basdd model for univariate time scries forecasting that provides interpretability capabilities. The model explicitly encodes seaedd decomposition into the network by defining separate trend and seasonal blocks, which a low degree polynomial and Fourier series.
3 Formulation
In this Section, we describe the TSNet network architecture for time series representation and TSR decompo- sition, and the iHyperTime network leveraged for generalization and new data generation.
3.1 Time Series Representation
We consider a time series signal encoded by a discrete sequence of N observations y (Y1y..., NN) where y: RT is the m-dimensional observation at time 4, This time series defines a dataset D ((4, j=1 of time coordinates tÜ associated with observations y:. We want to find continuous mapping f f: - mm f(t) that parameterizes the discrete time series, so that y; (t.) for i = 1,..., N. The function f can be approximated by an implicit neural representation (INR) architecture conditioned on the training loss L C= Σ, Ily: - ?. Input and output of the INR are of dimensions 1 and m, corresponding to the time coordinate t and the prediction f(t). After training, the network encodes a continuous representation of the functional relationship f(t for a single time series.
3

---

Published in Transactions on Machine Learning Research (08/2024)
3.1.1 TSNet
We propose a novel INR. architecture, namely TSNet, that allows for interpretable time series representation with TSR decompasition. TSNet encodes time series as a continuous function as described the previous section. In particular, we assume that our proposed INR follows a classic time series additive decompasition, i.c.,
=(t)= ft) (t()) + fa(t),
(1)
where fr, fs: fx correspond to the trend, seasonality and residual components of f(t). Note that this is a standard assumption for time series decomposition techniques, such as STL and others (Cleveland et al. 1990j. We elaborate on our modeling of these three components the following.
Trend and Seasonality Blocks Following the work by Oreshkin et al. (2020), we model trend and seasonality via basis decompositions with coefficients learned by fully-connected networks. The trend component of time series aims model slow-varying (and occasionally monotonie) behavior, thus we consider polynomial regressor, i.c.,
P w th.
(t)
(2)
where denotes the degree of the polynomial, and w denotes the learned weight associated with the pth degree. In practice, P is chosen to be small (e.g., P 2) to capture low frequency behavior.
The seasonal component of the time series fs(() aims to capture the periodic behavior of the signal, and thus we leverage learnable Fourier decomposition:
N/2-1 Σ
fs(t) =
w cos (2Tit) + w()) N/2++ sin (2rit))
(3)
where w (5) are the weights predicted by the network.
Residual Block The residual of a time series comprises the high-frequency non-periodic components the signal. order to model it, we leverage a fully-connected network of K layers with sine activations (SIREN), as defined by Sitzmann et al. (2020b))
q&+1 = sin (Wow) qk + b))) (x(t) w qK + b(R)
O K
(4)
(5)
where w b/²h and qe are the weights, biases, and outputs of the k layer, with qo = t corresponding to the input of the nerwork. general factor wp multiplying the network weights determines the order of magnitude of the frequencies that will be used to encode the signal. As shown by Sitzmann et al. (2220b), SIRENs mitigate the spectral bias of regular fully-connceted networks, and thus are well suited for learning and representation of high-frequencies. We refer to Appendix D.2 for the TSNet model implementation details.
In summary, TSNet allows for the decomposition of time series into trend, seasonality, and residual components. While a standard INR, such as SIREN, can be used to encode time series as shown in Scction 3.1, it would lack the interpretable decomposition that our TSNet INR has. This interpretability is significant advantage over traditional INRs, which often lack such explicit decomposition capabilities.
3.2 Time Series Generation with iHyperTine Time
In Fig. I we show a diagram of our iHyperTime architecture, which can be used to learn prior over implicit neural representations (TSNet) of time series data. Next, we will detail its components and describe how iHyperTime can be used for time series generation. Additional details on the model implementation can be found in Appendix D.2
Set Encoder The set encoder composed of SIREN layers Sitzmann et al. 2020b) and takes as input an arbitrary set of tuples ((4..))) N where denotes the time-coordinate and y; the corresponding univariate
4

---

Published in Transactions on Machine Learning Research (08/2024)
Thural Werght
Trcul
Trand
TSNet
mmw
Pred Timee Swrtie
VMW
Set Eneoder
mma
HoperNes
Seavonal Black
[t. .(())
Besibual Hoperhet
Besidual
GT Spectrum
Figure 1: iHyperTimm architecture. The Set Encoder processes a set of tuples Y:)) representing a time series, and encodes it as embeddings Zr, Zg, ZR associated the components of the TSR decomposition. The hypernetwork decoders learn to predict the weights of their corresponding TSNet blocks from the embeddings. During training, the output of the hypernetworks is used to instantiate TSNet hyponetwork, and the loss is computed difference between y and the output of TSNet f(t), in terms of signal and spectral distribution.
multivariate time series value. Each tuple is encoded into a fixed-size embedding Z, g(1,, and the sample set is reduced to a single embedding Z by applying symmetric operation @ (e.g., averaging): Z + i=1 Z, where the function g R Rm + R used to determine the embedding of each tuple, is parameterized by the SIREN layers (model details in Appendix D.2). The use of set encoder introduces permutation invariance in the computation, and provides high degree of flexibility in terms of the input Zaheer et al. 2017). enabling the encoding of data with irregular sampling, which are common occurrences in time series.
Hypernetwork decoders The embedding is modeled as a concatenation of three sub-embeddings Zr, Zs, and ZR with Zr Rdr denoting the trend embedding, Zs E RRd denoting the seasonality embedding, and Z Rd denoting the residual embedding with dz dr + ds + dn. Each embedding component is pass through its own hypernetwork decoder, which outputs the weights of its corresponding block in the TSNet INR. For example, Zr is passed into the Trend Hypernetwork to output the weights of the trend block in TSNet. The output of TSNet sums the three signals from each block into a single predicted time series, which is compared against the ground truth signal via the reconstruction and spectral losses (Crec and LFFT) during training.
iHyperTime training During training, use the weights predicted by the hypernetwork decoders to instantiate TSNet hyponetwork and evaluate it the input time-coordinate t, to produce the predicted time series value f()). We then compare the TSNet hyponetwork prediction with the ground truth signal via the reconstruction and spectral losses Lee and LEFT.
The training of iHyperTime is performed three stages, in order to improve stability: 1) we train the Trend networks (Trend hypernetwork, Trend block) for 100 epochs, computing the MSE loss between the ground truth time series y and the output of the block: L = Σ, lly; - (..) This leads to smooth approximation of the time series, which we use as initial guess for the second stage. 2) We then train the Trend and Seasonality blocks together, computing the MSE reconstruction loss Eree and the FFT loss LEFT between the ground truth and the added output of both TSNet blocks. 3) Finally, we train the three blocks together.
Training Loss
The training of PlyperTime is done by optimizing the following loss, which contains an MSE reconstruction term Cree, a spectral loss LEFT and two regularization terms Eweights and Clatent. for the network weights and the latent embeddings, respectively:
W W
N-1 F
L
yi f(t))) +X1
Bill
(6)
7
Eree
Crontghte
Lrateat
4FFT
5

---

Published in Transactions on Machine Learning Research (08/2024)
where F5- = [Frff corresponds to the coefficient of the kth frequency in the discrete Fourier transform (DFT) of the time series. The term LFFT penalizes deviations of the signal's Is frequency spectrum with respect to ground truth. Thus, we ensure a high-fidelity reconstruction not only of the time series values, but also of its spectral composition. We refer to Appendices C and D for further details on LFFT and the implementation details of the iHyperTimm architecture.
Time Series Generation After training, we leverage the hypernetwork architecture to generate latent representations of the time series from our training set. Generation of new time series is produced by randomly selecting pairs of time series, and performing linear interpolation between their embeddings Z and (
2 = 2 + X (2 -201)
(7)
where A is also sampled randomly. Optionally, the interpolation can be performed on individual components ZT, Zs, ZR of the embeddings, enabling the conditional generation of time series.
4
Experiments
We present our evaluation of time series generation on regular and irregular data, covering time series of diverse lengths and numbers of channels. Additionally, perform analysis of our model's TSR decomposition, and we compare training and inference times with previous works.
4.1 Baselines and Evaluation
Datascts We test the performance of iHT using multiple datasets with varying characteristics such periodicity, level of noise, number of features and length of the series. Stock corresponds to Google stock price data from 2004 to 2019, where each observation has 6 features. Energy is UCI appliance prediction dataset (Candanedo et al.) 2017) with 28 features. Additionally, we also consider Monash dataset (Godahewa ct al. 2021). from which we choose FRED-MD, NN5 Daily, Temperature Rain, and Solar Weekly datasets. A complete description of the datasets can be found in Appendix B
Additionally, we conduct experiments on irregularly sampled time series, achieved by randomly removing fixed percentages of values from each time series. We create the datasets by removing 30, 50 and 70% of each time series.
Baselines We compare our method with TimeGAN Yoon et al. 2019). GT-GAN (Jeon et al. 2022 Fourier Flows (FF) (Alaa et al. 2021 : LS4 (Zhou et al. 2023), DiffTime (Coletta et al. 2023 and RCGAN [Esebaa et al. 2017). TimeGAN and GT-GAN have shown strong performance on multiariiat time series with short sequence lengths and are able to handle irregularly sampled data. LS4, DiffTime, and Fourier Flows have shown strong performance on time series with longer sequence length, generating distributions of frequencies that closely resemble the original data. We refer to Appendix D.I for further details on baselines and the adjusted DiffTime and RCGAN architectures, introduced to deal irregular data and longer time series, respectively.
Evaluation metrics To asses the quality of the synthesized data, we adopt the predictive and discriminative scores used in TimeGAN (Yoon et al. 2019 The predictive score measures the usefulness of the generated data by using a train on synthetic, test on real (TSTR) approach: a model is trained using the synthetic data to predict the next step in a sequence, and then it is evaluated using the real data. In the case of multivariate time series with m channels, - 1 channels are used to predict the next step of the mh channel. The mean absolute error (MAE) between the predicted values and the ground truth is used for the evaluation. In the case of irregular time scries, and in order to make a fair comparison with other methods for this type of data, we adopt the more challenging predictive task proposed by GT-GAN (Jeon et al., 2022), which involves predicting the entire future vector, instead of a single channel. Irregular time series often have randomly sampled time steps and channels, and predicting only one future element can make the evaluation less reliable.
The discriminative score serves as measurement of fidelity of the generated data, where the aim is assess if the synthetic data is indistinguishable from real data. For this purpose, discrininative model is
6

---

Published in Transactions on Machine Learning Research (08/2024)
trained to classify real and fake samples, and then used to test whether the original and generated data are correctly classified. The discriminative score is computed as AAccracyy 0.5|, where low value means that the classification is challenging, and therefore, the model cannot tell which samples are real and which are generated. For a qualitative evaluation, we analyze the synthesized and original time series by employing I-SNE visualizations which project the data into Iwo-dimensional space (van der Maaten & Hinton 2008). Additionally, we perform a kernel density estimation (Jeon et al., 2022 to compare the data distributions. For the experiments on longer real-world time series from the Monash dataset, we also consider the marginal score (Zhou et al.) 2023 that computes the absolute difference between the real and synthetic empirical probability density functions.
4.2 Experimental results on regular time series synthesis
Performance results for regularly sampled time series are provided in Table 1 for univariate and multivariate datasets of varying lengths. The results indicate that iHT outperforms all methods in terms of the predictive score. This highlights the usefulness of the data generated by our method as source of synthetic data for learning. In terms of discriminative score, iHT is competitive with state-of-the-art methods across all datasets, although no method emerges as a definitively superior approach. In Figure 2 the t-SNE visualizations show that the time series generated by iHT closely resemble the ground truth data distribution. We refer Appendix D for additional qualitative comparisons.
Table Regular time series generation performance Table 2: Irregular time series generation performance: predictive and discriminative scores. 30% missing data.
terms of predictive and discriminative scores.
Method
Energy24
Stock21
Stocks72
Stock360
Method
Energy24
Stock24
Stocks72
Stock360
.251
.000
.037
+ .000
.000
IHT
.019 + .001
.013 :: .001
.188 :: .000
.168 : .000
Predictve Scoe
GT-GAN
321 0022
000
.188 :
Peed Snow
GT-GAN
+
.021 .003
.000
.196
TimeGAN
273 + -034
.038 [X11
.226 + .002
.206 + .000
DifTTime
.022 + .011
.019
.006
.200
.00
.188
ROGAN
292 + 003
.000 4: 0001
.192 + 0001
.180 +: 000
T84
.033 + 001
.022 + .00
.198
000
.22 + .000
DiffTime
252 000
.038 001
+
.215
FF
.148 + 007
137 + 0229
.10 :: 000
.184 : 000)
L.S4
.168
.000
Original
+ 0011
.011
.002
0001
FP
.251 + .00
.076 001
.191 + 000
.360 + 0000
Original
250 + 003
.038 001
.186 + .001
.167 + .001
IHT
.452 + 003
.059
.046
.017
.007
.014 + .010
GT-GAN
.333 1: .063
.251 +: .097
.068 :: .007
.111 + 026
Doceinllattee Score
IHT
245
.044
.018
DifTTime
.298 1: .010
.215 4: .010
.100 1: .045
.057 1: .070
CT-GAN
.211 +.068
.077 + .031
055 + .017
.085 + .064
4:
.455 .004
.067
TimeGAN
.26 + .012
.102 .221
.073 + .047
.042 + .074
FF
.500
.497 .005
.082
.156 + .102
ROCAN
336 001
.027
.012
.007
DiffTime
445 0034
.016
.012
.301
LS4
498 000
.863
0227
089 + .081
1888 + .081
FF
.49 4: 001
.349
4 .113
.016 4 .018
.015 + .014
Table 3: Irregular time series generation performance: predictive and discriminative scores. 50% missing data,
Table 4: Irregular time series generation performance:
predictive and discriminative scores. 70% missing data.
Method
Energy24
Stock21
Stocks?2
Stock360
Method
Euergy24
Stock24
Stocks72
Stock360
(Ours)
= .002
.187
.000
.000
(Ours)
.000
.187
.168 .000
Pnd Stom
GT-GAN
061 = .01
.018 :: 002
.15 1: 000)
.196 1.000
GT--AN
.76 4 .001
120 :: .006
.03 i: 1000
.196 000
DiffTime
057
.012
.000
.000
188
.084
.024
.002
1000
000
LSI
063 .012
003 1: .015
212
1 /000
1977
.000
Fnd
DilTTaae
065 1 .001
.088 1: .063
.24 1: 000
.196 .00
FF
227 .014
.60 + .088
236
(00)
211 0000
PF
304 + .005
206 + 000
.66 000
.245 000
Original
045
0022
.001
Original
.045
001
.167
iHT (Oues)
472
.011
.004
.003
(Ours)
.182
.020
.011
Doc Same
GT-GAN
.31 = .010
.85 :: .073
.026
.012
.081
4: .022
GT-GAN
.325 4: .0447
.200 +: .053
.068 .002
.091 .013
DifTTine
422 0011
.332
1 .031
284
.137
.110
.061
L84
.099 1. .0022
.455 1. .011
.036
.026
.183 .017
LSI
300
A98
.000
.14
0034
J227
.015
DilFiinee
.444 4: .001
.421
4: .003
.436
0000
.148 .088
FF
500 .02
.498 4 .003
.376
4: 130
.422
1. .058
FF
.500 1. .003
.J88 4: .008
.424 1: .03
.369 1: .133
4.3 Experimental results on irregular time series synthesis
Tables 2
3 and show the results for the irregular time series generation with different percentages of dropped
values. iHT perferos all methods in terms of predictive score across all datasets. Furthermore, the
7

---

Published in Transactions on Machine Learning Research (08/2024)
(a) iHT (Ours) (b) GT-GAN
(c) TimeGAN
(d) ROGAN
(e) DiffTime
(f) LS4
(g) FF
Figure 2: t-SNE visualizations on Stock24 data, where a greater overlap of blue and red dots shows a better distributionna-sinllarty between the original and generated data, Our approach shows the best performance. (See Appendix 25 for high resolution charts)
performance of iHT does not degrade significantly with the percentage of dropped values, even in the extreme case of 70% missing data, demonstrating its usefulness in downstream tasks. Given that for irregular data we adopt the predictive score proposed by GTGAN (Joong & Shin, 2022). which prodicts the future values of all channels instead of a single one, we see a drop in predictive score with regards to the values presented in Table []] In the case of Energy24, which contains 28 channels, the more challenging task of predicting all channels instead of just one may result in a lower predictive score (better performance) compared to the TimeGAN evaluation. This is because the error is averaged over all 28 channels, potentially masking poor performance on individual channels. This effect is also observed, although to lesser extent, in the Stock24 dataset, which contains 6 channels. For the univariate datasets (Stock72 and Stock360), the predictive score remains largely unchanged as there are no additional channels to loverage. In regard to fidelity (discriminative score), our method shows the best performance in out of datasets. Specifically, iHT excels on long time series datasets, demonstrating its ability handle sequences beyond 24 time steps, even with high percentages of missing data. For the shorter datasets, shows a degradation in discriminative score when the percentage of dropped data increases. The top in Figure compares the distributions of original and synthetic data for the Stock24 dataset with 50% of removed values. iHT shows the best performance, with the closest match to the original data distribution. The bottom row shows the corresponding t-SNE visualizations where we can see that iHT and GT-GAN show the best overlap between original and generated data, with iHT showing more data diversity, covering a wider area across the original data. Additional plots for other missing rates are shown Appendix L where we observe similar behavior with iHT showing the best overlap in both distribution and t-SNE visualization.
Data Danaky Extinate
Original
Synthetic
Data Demaiyy cat mate
Original Synthetic
Data Densty Edtinate
10
Original Synthetic
Dete Dersity Entimate
- tu V * 1p
Original Synthetic
Denatyy Extimate u
Synthetic
Original
1
3
6
0.00
0.25
0.50
0.75
0.00
0.50
0.75
0.25
0.50
0.00
0.25
0.50
0.75
0.0
Data Value
Data Value
Data Value
Data Value
Data Value
ginall GT-GAN
-5
(a) iHT (Ours)
(b) GT-GAN
(c) DiffTime
(d) LS4
(c) FF
Figure 3: (Top) Data distribution on irregular Stock24 data (Missing 50%). (Bottom) t-SNE visualizations on irregular Stock24 data (Missing 50%), where greater overlap of blue and red dots shows a better distributional- similarity between the generated data and original data. Our approach shows the best performance.
4.4 Experimental results on longer real-world time series synthesis
Table 5 shows additional generation results for 4 real-world datasets from the Monash dataset, where 3 of the datasets contain time series with lengths 700 time steps. We report comparisons with LS4, which has
8

---

Published in Transactions on Machine Learning Research (08/2024)
shown state-of-art performance these datasets Zhou et al. 2023), while we leave the full evaluation table in Appendix F. In this scenario, the Classification (fidllit) and Prediction (usefulness) scores are computed using a 1-layer S4 model (Zhou et al. 2023). The results show the ability of iHT to deal with long time series, with superior performance in out of datasets w.r.t. the statc-of-art LS4.
Table 5: Generation results Monash datasets.
Data
Metric
LS4
IHT (Ours)
Data
Metric
LS4
IHT (Ours)
Marginal
0.0221
0.0177
Marginal
0.0834
0.2978
FRED-MD
Class f Prediction +
0.544
1.3278
Temp Rain
Class
0.976
11.2493
0.0373
0,0181
Prediction
0.521
0.132
Marginal
0.00671
0.00893
Marginal
0.0459
0.03273
NN5 Daily
Class f
0.636
0.4982
Solar Weekly
Class f
0.683
1.2413
Prediction 4
0.241
0.2349
Prediction J
4.5 Runtime
In Figure 4 show that the strong performance of iHT on long time series does not impact its computational time. We consider a set of synthetic datasets with lengths (80, 320, 1280, 5120, 20480) and we evaluate the training time for 100 iterations, and the inference time on one batch (Zhou et al. 2023 The figure shows that iHT has among the lowest computational times w.rtt. existing approaches. Moreover, iHT training times are almost unaffected by the length of the time series, with negligible changes even for 20,480 time steps, making it the fastest method for long sequences. Additional details and the overall training times are presented in Appendix G
30
107
Runtime
Runtime
320
1280
$120
20480
80
320
1260
5120
20400
Sequence Length
Sequence Length
TimaGAN GTGAN
DiffTime
FourierFlows
154
Figure 4: Training and inference time comparison for time series of different lengths.
4.6 Trend-Seasonality decomposition analysis
iHT provides controllable method for time series generation based an interpretable trend-scasonnlity- residual decomposition of latent embeddings. We explore the interpretable decomposition of iHT on analytically generated time series datasets that have trend (T), seasonal (S) and noise (R) components, or combination of two of them. The synthetic datasets are generated by uniformly sampling trend values, frequencies and levels of noise. We compute the decomposition error as the MSE between the output each block of iHT and the corresponding TSR analytic component. In table G we compare against the traditional STL method, which requires the period of scasonality as an additional parameter. In STL (eract), we compute STL with the exact period of the analytic signal. In STL (approx), we provide STL with an approximate period estimated by analyzing the Fourier spectrum of the signal, more realistic setting for time series decomposition. Our method shows the lowest trend error, with small standard deviation with respect to STL (approz), and shows comparable results with STL (ezact). Additionally, our method shows similar performance on the seasonality component when there is seasonality present in the dataset with respect to STL (appror)) and shows much better agreement when there no seasonality present (T+R dataset).
In Figure 5 (a) and (c), we show the distribution of the trend, and residuals outputs from iHT against the ground truth components for the T+S+R dataset. In the case of the seasonality component (b), we plot the
9

---

Published in Transactions on Machine Learning Research (08/2024)
GT: Trend HHT Thend
GT
GT: Res
High Low jou
IIT
Twed
(a) Trend
(b) Seasonality
(c) Residuals
(d) tSNE-trend
(e) tSNEaseasonality
Figure 5: (a,b,c): Distribution of the trend, seasonality and residual outputs of iHT vs ground truth. (d,e): t-SNE visualization of trend and scasonality embeddings in iHT. All cases correspond to the T+S+R datasct.
Table 6: Ablation study for srend-seasonalliy-reiii residuall decomposition of time series. We compare iHT against STL docomposition, for three datasct configurations: + (T+S), trend rrssdualll (T+R) and all three components (T+S+R).
MSE
T+S+R
T-R
T+S
(x10-
(exuct)
STL (approx)
iHT (Ours)
STL (esacct
STL (apprux)
iIT (Ours)
STL (excuct
STL (approx)
iIT (Ours)
Trend
0.46 1 2.08
1.04 1. 4.69
0.60 1 0.50
0.07 1. 0.07
1.26 5.70
0.20 1 0.27
0.46 1 2.10
1.17 4 5.12
0.46 1: 0.48
Seasouality
0.57 + 2.10
1.19 + 1.73
1.21 + 0.68
0.03 : 0.02
1.35 = 5.71
0.17 = 0.29
0.41 # 2.09
1.18 + 5.11
1.25 : 0.77
Residuals
0.17 0.15
0.18 0.12
1.29 + 0.62
0.15 0.09
0.18 0.13
0.41 0.26
0.03 + 0.10
0.04 + 0.07
1.25 0.71
histogram for the frequencies of the time series, where we estimate the frequency of each time series by finding the dominant frequency component in the discrete Fourier transform. The trend and seasonality histograms show very good agreement with the ground truth, while the residuals show slightly wider tails. Finally, we visualize the learned representations via t-SNE of the embeddings. Figure 5 (d) and (e) shows that iHT is able to learn the trend and seasonal patterns from the dataset. In plot (d) the color separation corresponds to positive and negative trend, while plot (e) shows the separation in frequencies. The effectiveness of this decomposition is further showcased in our additional experiments on synthetic and real-world datascts (Appendices H and [], where each hypernet. accurately captures its corresponding component. Additional evidence is provided in Appendix J through controlled generation experiments demonstrating the ability of the trend-hypernet to manipulate the trend of generated time series.
4.7 Ablation studies
In Table 7 we change the architecture of iHT to create simpler ablation models, and report predictive and discriminative metrics for multiple datasets. iHT corresponds to our full proposed model. In iHT (no FFT) we have removed the FFT loss from the training. In iHT-SIREN we remove the TSR docomposition from iHT, replacing TSNet with a SIREN network. We observe that the predictive scores are comparable for all configurations, while discriminative scores improve for all datasets when we incorporate the interpretable decomposition. The error reduces further when we add the FFT loss to the training process. The strong predictive scores across all variants indicate their effectiveness in capturing the underlying patterns and dynamics of the time series data. This suggests that even models with lower discriminative scores, such as iHT (no FFT) and iHT-SIREN, are capable of generating useful predictions. However, the discriminative scores for these variants reveal that they may struggle to produce samples that are statistically indistinguishabl from real data. In particular, iHT-SIREN shows the worst discriminative score across all datasets, showcasing the importance of TSNet, while the improvement in discriminative score from the Fourier loss is more modest. We show additional ablation studies for irregularly sampled data in Section K.1 in the Appendix.
In Table g we study the effect of wo in iHT, with wo E [5, 10, 30, 100, 100,300), and we report the generation metrics for the Stock24 and Stock72 datasets. In Stock24, the best results are obtained with wo = 30. In the case of Stock72, the discriminative and predictive scores show a slight improvement for wo = 100 with win = 30 showing the second best performance.
Finally, we conduct an ablation study on iHT using Random Fourier Features (FFN) instead of a SIREN network for the residuals. Given the importance of the scale factor in FFN as explained in (Woo al. 2023). evaluate FFN with multiple scale factor values: 5, 10, 100. Table ssoows the discriminative and predictive
10

---

Published in Transactions on Machine Learning Research (08/2024)
Table 7: Ablation study for model architecture: comparison of iHT against simpler configurations: ilfT (no FFT) without FFT loss, and iHT-SIREN without TSR decomposition.
Energy24
Stock24
Stock72
Stock360
Energy24
Stock24
Stock?2
Stock360
Prodictive Score
Disc. Score
0.168
0.014
IHT (no FFT)
0.046
0.014
0,188 0.188
0.168
IHT (no FFT)
0.278
0.073
0.015
0.011
IHT-SIREN
0.018
0.013
0.169
iHT-SIREN
0.3/11
0.108
0.022
0.021
Table 8: iHT with different values of wi E [5, 10, 30, 100, 300] for Stock24 and Stock 72 datasets.
We = 5
Wlg = 10
Wig =
30
100
= 300
Stock24
Discr-score
0.159
0.212=0.044
0.054=0.028
0.087=00066
0,489:10.007
Pred-score
0.04110.001
0.040=0.001
0.037=0.000
0.037=00000
0.059#000002
Stock72
Discr-score
0.045+0.034 0.058+0.038
0.028=0.027
0.026:0.011 0.183::00001
0.250:00.031
Pred-score 0.188:0.0002
0.18610.001
0.185=0.001
0.196::0.000
scores for the Stock24 and Stock72 datasets. For the Stock24 dataset, SIREN achieves the best discriminative score. In the Stock72 dataset, FFN with scale factor 10 shows slightly better performance in discriminative score than SIREN network. In all cases the predictive score shows similar performance.
Table iHT with SIREN layers and with FFN with different scales (a [5, 10, 100) for Stock24 and Stock72 datasets.
STREN (wo 30)
FFN (a = 5)
FFN (a = 10) FFN (a 100)
Stock24
Discr-score
0.054+0.028 0.037+10.000
0.244=0.162 0.036+10000
0.346=0.169 0.036=0.000
0.36:+0.183
Pred-score
0.036+00000
Stock72
Discr-score
0.028-10.027
0.02810.014 0.184+0.001
0.024+0.028
0.0541:0.029
Pred-score
0.185+0.001
0.185=0.002
0.184+0.001
5 Conclusions and Limitations
In this work, we presented iHyperTime, a versatile and efficient framework for generating time series with a wide range of characteristics. Unlike existing generative models that excel either in short or long sequences, our model demonstrates superior performance across both types of datasets. Our evaluations reveal its efficacy in handling irregularly sampled data, where it consistently surpasses current benchmarks. For regularly sampled sequences, iHyper'Time's performance is competitive with the best available models, regardless of time series length. One of the model's notable strengths is its rapid training speed, which is not only comparable to the quickest existing methods for short sequences but also significantly faster for longer ones. Importantly, our architecture incorporates inductive biases that facilitate unsupervised decomposition of time series into trend, scasonality, and residual components, capability we validated against the established STL decomposition method. Additionally, we illustrated iHyperTime' ability learn semantically meaningful representations, opening the door for applications that. involve generating time series conditioned interpretable factors.
11

---

Published in Transactions on Machine Learning Research (08/2024)
Limitations and Future Work Despite its strengths, iHT exhibits limitations, particularly its reliance conditional generation mechanisms, which may restrict diversity in the generated sequences. This limitation becomes particularly evident when two closely similar time series are used for generation, especially with a small interpolation paramcter (A), potentially resulting in a narrower range of diversity than desired. Such a scenario underscores the need for a more robust stralegy to ensure a wider exploration of the embedding space without compromising the integrity and relevance of the generated data.
To address this limitation and enhance iHT's generative capabilities, possible approach is to incorporate convex combination of multiple embeddings into the framework. This approach, akin to strategies employed for data augmentation with PCA (Serrano et al. 2016),, promises to expand the diversity of generated time series. By adopting a method that allows for random sampling within the convex hull of the embedding space, iHT could achieve a more comprehensive exploration of potential time series characteristics while maintaining consistency with the underlying data distribution. Moreover, inspired by competitive results obtained by diffusion models in generation tasks, integrating diffusion model principles into iHT could pave the way for the development of an advanced generalive model capable of unconditional generation. This extension would not only broaden the scope of iHT'ss applicability but also contribute the evolution of time series generation methodologies, offering new opportunities for rescarch and application in various domains. Our rescarch demonstrates promising results in time series generation across a wide range of datasets and highlights the potential of using implicit neural representations (INR) for time series generation with interpretable components. We hope our work could inspire and foster novel research in this area, which has received limited attention.
References
Ahmed Alaa, Alex James Chan, and Mihacla van der Schaar. Generative time-series modeling with fourier flows. In Proceedings of the International Conference on Learning Representations (ICI.), 2021.
Ali Alqahtani, Mohammed Ali, Xianghua Xie, and Mark W. Jones. Deep time-serics clustering: A review. Electronics, 10(2)), 2021.
Ivan Anokhin, Kirill V. Demochkin, Taras Khakhulin, Gleb Sterkin, Victor S. Lempitsky, and Denis Korzhenkov. Image generators with conditbonally-indeeendedd pixel synthesis. Proceedings of the IEFF Conference on Computer Vision and Pattern Recognttion (CVPR), pp. 14273-14282, 2021.
Samuel A. Assefa, Danial Dervovic, Mahmoud Mahfouz, Robert E. Tillman, Prashant Reddy, and Manuela Veloso. Generating synthetic data in finance: Opportunities, challenges and pitfalls. In International Conference on Al in Pinance, 2020.
Kasun Bandara, Rob J. Hyndman, and C. Bergmeir. Mstl: A scasonal-trend decomposition algorithm for time series with multiple scasonal patterns. International Journal of Operational Research, 2022.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous modeling of sporadically-obsered time series. In Advances in Neurol Information Processing Systems (NeurIPS), 2019.
Luis M Candanedo, Véronique Feldheim, and Dominique Deramaix. Data driven prediction models of energy use of appliances in low-energy house. Energy and buildings, 140:81-97, 2017.
Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: Bidirectional recurrent imputation for time series. In Advances in Neural Information Processing Systems (NewrIPS), volume 31, 2018.
Ricky T. Q Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems (NeurIPS), 2018
Yinbo Chen, Sifci Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8628 8638, 2021.
Kukjin Choi, Jihun Yi, Changhwa Park, and Sungroh Yoon. Deep learning for anomaly detoction in time-scries data: Review, analysis, and guidelines. IEF. Access, 0-120043-120000 2021.
12

---

Published in Transactions on Machine Learning Research (08/2024)
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In Newrips 2014 Workshop on Deep Learning, December 2014, 2014.
Robert B. Cleveland, William S. Cleveland, Jean E. McRae, and Trma Terpenning. Stl: A seasonal-trend decomposition procedure based on loess (with discussion). Journal of Official Statistics, 6:3-73, 1990.
Andrea Coletta, Sriram Gopalakrishan, Daniel Borrajo, and Svitlana Vyetrenko. On the constrained time-series generation problem. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Abhyuday Desai, Cynthia Freeman, Zuhui Wang, and Ian Beaver. Timevac: A variational auto-encoder for multivariate time series generalion. arXiv preprint (xXi::2111000000 2021.
Cristóbal Esteban, Stephanic L. Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans. Xiv preprint Xiee:: 1766. 02633, 2017.
Chenguang Fang and Chen Wang. Time series data imputation: survey on deep learning approaches. arXiv preprint arXiv:2011.113/7 2020.
Elizabeth Fons, Paula Dawson, Xiao-jun Zeng, John Keane, and Alexandros losifidis. Adaptive weighting scheme for automatic time-scries data augmentation. orXiv preprint arXiv:2102008310 2021.
Rakshitha Wathsadini Godahewa, Christoph Bergmeir, Geoffrey 1 Webb, Rob Hyndman, and Pablo Montero- Manso. Monash time scries forecasting archive. In Advances in Newral Information Processing Systems (NeurIPS), 2021.
David Ha, Andrew M. Dai, and Quoe V. Le. Hypernetworks. In Proceedings of the International Conference Learning Representations (ICLR), 2017.
Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F. Schmidt, Jonathan Weber, Geoffrey T. Webb, Thassane Tdoumghar, Pierre-Alain Muller, and François Petitjean. Inceptiontime: Finding alexnet for time series classification. Data Mining and Knowledge Discovery, 2020.
Jinsung Jeon, Joonghak Kim, Haryong Song, Seunghyeon Cho, and Noseong Park. GT-GAN: General purpose
time series synthesis with generative adversarial networks. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Kyeong-Joong Jeong and Yong-Min Shin. Time-scries anomaly detection with implicit neural representation. arXiv preprint arXic:2201.1100 11950, 2022.
James Jordon, Daniel Jarrett, Evgeny Saveliev, Jinsung Yoon, Paul Elbers, Patrick Thoral, Ari Ercole, Cheng Zhang, Danielle Belgrave, and Mihaela van der Schaar. Hide-and-seek privacy challenge: Synthetic data generation vs. patient ro-identification. In Advances in Neural Information Processing Systems (NeurIPS), volume 133, pp. 206-215 215, 06-12 12 Dec 2021.
Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Phylosophiccal Transactions of the Royal Society A, 2021.
Gidi Littwin and Lior Wolf. Deep meta functionals for shape representation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) pp. 1824-1833, 2019.
Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific Reports, 8(1): 6085, 2018.
Yonghong Luo, Xiangrui Cai, Ying ZHANG, Jun Xu, and Yuan xiaojie. Multivariate time series imputation with generative adversarial networks. In Advances in Neural Information Processing Systems (NewrIPS), volume 31, 2018
M.J. Menne, C.N. Williams, Jr., and R.S. Vose. Long-term daily and monthly climate records from stations across the contiguous united states (u.s. historical climatology network). 2016.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Lars Mescheder, Michael Occhsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as ncural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Cheolhwan Oh, Seungmin Han, and Jongpil Jeong. Time-series data augmentation based on interpolation.
Procedia Computer Science, 175:64-71, 2020.
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion amalysis for interpretable time series forecasting. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Tamar Rott Shaham, Michael Gharbi, Richard Zhang, Eli Shechtman, and Tomer Michaeli. Spatially-adaptive pixelwise networks for fast image translation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Yulia Rubanova, Ricky T. Q. Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. In Advances in Neurol Information Processing Systems (NeurIPS), 2019.
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning internal representations by error propagation. 1986. URL
org//orruusD::6225742
Ana Serrano, Diego Gutierrez, Karol Myszkowski, Hans- Peter Seidel, and Belen Masia. An intuitive control space for material appearance. ACM Transactions Graphics (TOG) Proceedings of ACM SIGGRAPH, 35(6), dec 2016.
Satya Narayan Shukla and Benjamin Marlin. Heteroscedastic temporal variational autoencoder for irregularly sampled time series. In International Conference on Learning Representations, 2021.
Tkaro Silva, George Moody, Daniel Scott, Leo Celi, and Roger Mark. Predicting in-hospital mortality of icu patients: The 2012.
in cardiology challenge 2012. Computing in cardiology, 39:245 248, 01
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation nerworks: Continuous 3d-structure-aware neural scene representations. In Advances in Neurol Information Processing Systems (NeurIPS), 2019.
Vincent Sitzmann, Eric R. Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta- learning signed distance functions. In Advances in Neural Information Processing Systems (NewrIPS), 2020a.
Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Advances in Neural Information Processing Systems (NeurIPS), 2020b.
Ivan Skorokhodov, Savva. Ignatyev, and Mohamed Elhosciny. Adversarial generation of continuous images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10753-10764, June 2021.
Alejandro Satrajman, Gilles Rainer, Tobias Ritschel, and Tim Weyrich. Neural brdf representation and importance sampling. Computer Graphics Forum, 40(6):332 346, 2021.
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems (NeurIPS), 2020.
14

---

Published in Transactions on Machine Learning Research (08/2024)
José F. Torres, Dalil Hadjout, Abderrazak Scbaa, Francisco Martinez- Alvarez, and Alicia Troncoso Lora. Deep learning for time series forecasting: A survey. Big dala, 2021.
Laurens van der Maaten and Geoftry Hinton. Visualizing data using t-SNE. Journal of Machine Learning 2008.
Research,
Svitlana Vyetrenko and Shaojie Xu. Risk-sensitive compact decision trees for autonomous execution presence of simulated market response. arXie preprint arXix: 02311,, 2019.
Qingsong Wen, Jingkun Gao, Xiaomin Song, Liang Sun, Huan Xu, and Shenghuo Zhu. Robuststl: A robust seasonal-trend decomposition algorithm for long time series. AAAI Conference on Artificial Intelligence, 33(01):5409 5416, Jul. 2019.
Magnus Wiese, Robert Knobloch, Ralf Korn, and Peter Kretschmer. Quant gans: deep generation of financial time series. Quantilative Finance, pp- 1 1-22, Apr 2020.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. Hoi. Learning deep time-index models for time series forecasting. In Proceedings of the International Conference on Machine Learning (ICML), 2023.
Jinsung Yoon, Daniel Jarrett, and Mihaela van der Schaar. Time-series generative adversarial networks. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in Neural Information Processing Systems (NewrIPS), 2017.
Li Zeng, Baifan Zhou, Mohammad Al-Rifai, and Evgeny Kharlamov. Segtime: Precise time series segmentation without sliding window, 2022.
Linqi Zhou, Michael Poli, Winnie Xu, Stefano Massaroli, and Stefano Ermon. Deep latent state space models for time-series generation. In Proceedings of the International Conference on Machine Learning (ICML), pp. 12625-42643 PMLR, 2023.
15

---

Published in Transactions on Machine Learning Research (08/2024)
A Additional Related Work
Implicit Neural Reproscntations INRs (or coordinate-based neural networks) have recently gained popularity in computer vision applications. The usual implementation of INRs consists of a fully-connected neural network (MLP) that maps coordinates (e.g. xyz-coordinates) to the corresponding values of the data, essentially encoding their functional relationship in the network. One of the main advantages of this approach for data representation, is that the information is encoded in a contimnous/ grid-free representation, that provides a built-in non-linear interpolation of the data. This avoids the usual artifacts that arise from discretization, and has been shown combine flexible and accurate data representation with high memory efficiency (Sizzmann et al. 2020b; Tancik et al. 2020). Whilst. INRs have been shown work on data from diverse sources, such as video, images and audio (Sitzmann et al., 2020b Chen et al.) 2021 Rott Shaham ct al. 2021). their recent. popularity has been motivated by multiple applications in the representation of 3D
scene data, such as 3D geometry (Park et al. 2019 Mescheder et al. 2019 Sitzmann et al.,
2020a 2019
and object appearance [Mildenhall et. al.) 2020: Sztrajman et al.) 2021). ). In early architectures, INRs showed
a lack of accuracy in the encoding of figh-freeuency details of signals. Mildenhall et al. (2020) proposed
positional encodings to address this issue, and [aaciik et al.] (2020) further explored them, showing that by
using bourier-based features in the input layer, the network is able to learn the full spectrum of frequencies from data. Concurrently, [Sizzmann et al. (2020b) tackled the encoding of figh-freqeency data by proposing the use of sinusoidal activation functions (SIREN: Sinusoidal Representation Networks), and [Benbarka et al. (2022) showed the equivalence between Fourier features and single-layer SIRENs. Our INR architecture for time series data (Section 3) is based on the SIREN architecture by Sitzmann et al.
Hypernetworks A hypernetwork is a neural network architecture designed to predict the weight values of a secondary neural network, denominated a hyponetwork (Sitzmann et al., 2020a). The concept of hypernetwork was formalized by Ha et al. (2017), drawing inspiration from methods in evolutionary computing (Stanley lt al. 2009 Moreover, while convolutional encoders have been likened to the function of the human visual system (Skorokhodov et al. 2021), the analogy cannot be extended to convolutional decoders, and some researchers have argued that hypernetworks much more closely match the behavior of the prefrontal cortex (Russin et al. 2020). Hypernetworks have been praised for their expressivity, compression due to weight sharing, and for their fast inference times (Skorokhooo et al. 2021). They have been leveraged for multiple applications, including few-shot learning (Rusu et al. 2019 Zhao et al.
2020 continual lcarning (von Oswald et al., 2020) and architecture search (Zhang et. al. 2019 Brock et al. 2018). Moreover, in the last.
two years some works have started to leverage hypernetworks for the training of INRs, enabling the learning of latent encodings of data, while also maintaining the flexible and accurate reconstruction of signals provided by INRs. This approach has been implemented with different hypernetwork architect ures, to learn priors over
image data. (Sitzmann et al.) 2020b Skorokhodov et al. 2021 0 3D scene geometry (Littwin & Wolf 2019 Sitzmann et al. 2019 2020a) and material appearance (Sztrajman et al. 2021). Tancik et al.
(2021) leverage hypernetworks to speed-up the training of INRs by providing learned initializations of the network weights. Sitzmann et al. (020b) combine a set encoder with a hypernetwork decoder to learn a prior over INRs representing image data, and apply it for image in in-painting. Our hypernetwork architecture from Section 3 is similar to Sitzmann el al.'s, however we learn a prior over the space of time series and leverage it for new data synthesis through interpolation of the learned embeddings.
Interpretable Time Series Seasonal-trent decomposition techniques are standard tools in time series analysis used to decompose time series into trend, seasonal, and remainder components. The trend component encapsulates the slow time-varying behavior of the time series, while seasonal components capture recurring (i.e., periodic) fluctuations in the data. These techniques enable an intuitive and interpretable analysis of time series data which play an important role in variety of downstream tasks, including forecasting and anomaly detection. The classic approach for performing the decomposition is the widely used STL algorithm (Cleveland et al., 1990). To account for outliers and distributional shifts, a robust version of the algorithm, called Robust STL, has also been proposed (Wen et al., 2019). Additional challenges in seasonal-trend decomposition involve dealing with complex time series data that exhibit multiple scasonal components, to which techniques such as multiple STL (MSTL) have been proposed (Bandara et al., 2022). The ability to break time series into interpretable components has been topic of recent interest in the
16

---

Published in Transactions on Machine Learning Research (08/2024)
context of anomaly detection, forecasting, and generation. Relevant to this work is the recently proposed N-BEATS architecture (Oreshkin et al. 2020), a deep learning-based univariate time series forecasting solution that provides time series interpretability capabilities without considerable loss in predictive performance. The N-BEATS architecture explicitly encodes seasonal-trend decomposition into the network by defining two blocks: a trend block which uses a small ordered polynomial capture slow varying behaviors, and a scasonality block which uses Fourier series to capture cyclical patterns. Little work, however, has been done in the design of generation schemes that allow for decomposition of time series data into interpretable components. While TimeVAE (Desai et al. 2021 proposes VAE architecture where the decoder has trend and seasonality blocks to allow for interpretable generation, no results highlighting the advantage of this capability were demonstrated.
Time Series Imputation Time series imputation (or interpolation), which aims to fill in missing values
estimate values at specific time points within existing series (Cao et al. 2018 Liu 2018 Luo et al. 2018 Shukla & Marlin 2021), is related but distinct area of research. While both time scries imputation and generation involve producing new data, they differ significantly in their goals and evaluation metrics. Time series generation focuses on creating entirely new sequences that replicate the patterns and characteristics of the original dataset, often assessed using discriminative and predictive scores. In contrast, time series imputation focuses on accurately filling gaps within cxisting scries, typically evaluated using metries like mean squared error. Furthermore, the methodological approaches for imputation and generation are not always directly transferable. A model designed for effective generation may not excel at imputation, and vico-versa. In Section[]] we have adapted our time series generation approach to also handle the task of imputation, demonstrating its flexibility and broad applicability.
Datasets
Table 10: Main characteristies of the datasets used.
Dataset
Number of Samples
Length of Time series
No of Features
Source
Stock24
3661
24
Stock72
3613
72
Link
Stock360
3325
360
Energy
19635
24
28
Link
FRED-MD
107
728
Link
NN5 Daily
111
791
Link
Temp Rain
32072
725
Link
Solar Weekly
137
52
Link
PhysioNet
8000
< 2880
41
Link Link
USHCN
1218
< 1491
Link
Almospheric CO2
288
60
Link
Airline Passenger
108
36
Link
Here we introduce in detail the datasets used in evaluation. We use publicly available Google stocks data from Yahoo finance, and the UCI Energy dataset (Candanedo et al. 2017j. Google stock datasct contains daily observations from 2004 to 2019 with 6 features, namely open, high, low, close, adjusted close, ond volume. The energy data contains 28 features with 10-minute resolution. Finally, we consider 4 datasets with longer real-world timo-scrics from Monash repository (Godahewa et al. 2021), namely FRED-MD, NN5 Daily, Temperature Rain, and Solar Weekly. The first three datasets have time-scries of length of around 700, while the latter has time-series with length of 52. The datasets characteristies summarized in Table 10
17

---

Published in Transactions on Machine Learning Research (08/2024)
In order to make fair comparison with current state-of-the-at methods, process the Stock and Energy in two different ways: in line with Yoon et al. (2019) and Jeon et al. (022). we slice the data using a window of 24 time steps, corresponding to datasets Stock24 and Energy24. Following Coletta et al. (2023), we select one feature per dataset (univariate) and slice it using windows of 72 and 360 time steps, which correspond to Stock72 and Energy360.
In Section extend the evaluation of our model on two real-world irregularly sampled datasets: PhysioNet, which corresponds to health measurements from ICU patients, and USHCN, climatology network dataset reflecting environmental conditions such precipitation and temperature. In Section we evaluate the docomposition capability of iHT on real-world datascts that exhibit strong trend and scasonality: Atmospheric CO2 and Airline Passanger datasets. The Atmospheric CO2 dataset corresponds to monthly data (Jamuary 1959 to December 1987) with a total of 348 observations with a seasonality period of 12 (Cleveland et al.) 1990 We sliced the data into time series of sequence length 60, obtaining total of 288 time series. The Airline Passenger dataset corresponds to monthly data (January 1948 to December 1960) with total of 144 observations with seasonality period of 12. We sliced the data into time series of sequence length 36, obtaining a total of 108 time series.
C Fourier-based loss
As part of the training of our ilyper'Time architecture, propose a Fourier spectrum reconstruction loss. For discrete-time signal f = (/o (()). ((),..,, [(N),, the N-point discrete Fourier transform (DFT) is utilized to obtain the corresponding frequency domain representation of f through the following operation:
N-1 Σ fne 2mi 2mi( n=0
F -
(F), O
where = I corresponds the imaginary unit of a complex number. The coefficient Fg E quantifies the strength in representation of the kth frequency component of the signal. The DFT has a time complexity of O(N2). In practice, an algorithm called the fast Fourier transform (FFT) is used to compute the DFT due to its lower time complexity (i.e., ON log N)). Using the FFT obtain the frequency domain representations of two discrete-tine signals f and f, we introduce Fourier-based reconstruction loss as follows:
N-1 LFFT
F - Pall-
K==0
Here, we utilized the PyTorc implementation of the FFT to obtain the DFT for each signal. is important note that the DFT is only well-defined for regularly sampled signals. In the case of this work, the discrete-time signal obtained by deterministically sampling the function f() via discretized grid of time steps t 10, NN.
D Implementation & Reproduuibility Details
D.1 Baselines
We use the following methods with publicly available code benchmark for our method:
Fourier Flows Alaa et al.
2021):
TimeGAN Yoon et al.
2019
GT-GAN Jeon et al.
2022
httpe//gstaabbcccn///nuuggleeaea
RCGAN Esteban
2017
https://gtthuub.ccc//777///////2
LS4 LS4 Zhou et al.
2023
https: https://ggtttaaccc////aaa//////0
DiffTime Coletta et
http:://antiv.oor//aaa//303
18

---

Published in Transactions on Machine Learning Research (08/2024)
We adapted TimeGAN, RCGAN, and GT-GAN for longer time-series by setting the hidden dimensions to be equal to the time-series length, as suggested by authors and empirically evaluated. Moreover, we improve ROGAN discriminator to handle longer time-series more effectively using the CSDI transformer architecture Tashiro et al. 2021). For DiffTime we reach out the authors to get access to their code, and to handle missing data we dynamically mask the input lime-series to let. the model learn to reconstruct the whole original time-series, similarly to CSDI approach for imputation (Tashiro et al. 2021
D.2 Implementation Details
MllpperT'ime is composed of set encoder, three decoder (hypernetworks) whose outputs corresponds to the weights of cach of the blocks in TSNet. Below we explain each component in detail.
Set Encoder The set encoder is a SIREN with two hidden layers of 128 neurons and an output layer (embedding) of 40 neurons. Tt takes as input an arbitrary set of tuples time-coordinato and y; the corresponding univariate or multivariate time scries value. We use a single floating point value as temporal coordinate (lime f). As data pre-processing, the values are scaled to the interval [],, with a common global factor for all time series of a dataset. MinMax scaling also applied to the time scries amplitudes, although in the interval [0, 1]. In all cases, regardless of sequence length, the time series is fed to the set encoder single set, and is hence converted into single embedding Z.
where t denotes the
Decoder (Hypernetwork) Each decoder block (hypernetwork) is one-layer MLP with ReLU activations, with hidden layer of dimension 128. The output of each hypernetwork is vector that contains the weights of its corresponding decomposition block. Table 11 shows the dimension details, where Yt ns and nn correspond to the number of weights in the trend, seasonality and residuals blocks, which form TSNet.
Table 11: urchitecture of the hypernetworks in the decoder.
Hypernet block
Design
Input size
Output size
Trend Hypernet
Relu
10
128
Linear
128
fT
Season Hypernet
Relu
15
128
Linear
128
ng
Res. Hypernet
Rehu
15
128
Linear
128
nR
TSnet Architecture TSnet is an implicit neural representation of univariate It is composed of three distinctive blocks that perform trend-seasonalltyy-ree residull additive decomposition of the time series signal. Table 12 shows the network details of each component of TSNet. In the trend block, corresponds to the degree of the polynomial, L corresponds to the max length of the time series, and m corresponds to the number of features.
univariate/mullivrritae mullivariaae time series data.
TSNes
Trend
Pred Tinee Series
VMw
{t}
Seasonal
Residwal Block
Figure 6: Diagram of the TSnet architecture.
19

---

Published in Transactions on Machine Learning Research (08/2024)
Table 12: Architecture TSNet.
Layer
Design
Input size
Output size
Trend block
Linear
p
m
Seasonality block
Linear
L
m71
Residual block
Sine(T.neea))
60
2
Sine(Linear)
60
60
3
Sine(Linear))
60
60
4
Sine(Linear)
60
60
5
Lincar
60
m
Training The training of iHT is performed in three stages to improve stability: 1) we train the Trend HyperNetwork, Trend Block for 100 epochs, computing the MSE loss between the ground truth time series y and the output of the block: Li = > Il (0)l.. This leads as initial guess for the second stage. 2) We then train the Trend and Seasonality blocks
smooth approximation of the time
series, which together, computing the MSE reconstruction loss Liee and the FFT loss LFFT between the ground truth and the added output of both TSnet blocks. 3) Finally, we train the three blocks together.
Stage 1 training:
Number of epochs: 100
Learning rate: le -
Stage 2 training:
Number of epochs: 150
Learning rate: 5e - 5
Stage 3 training:
Number of epochs: 150
Learning rate: 5e 5
Batch size: 256
Å1 = 1.0 x 10-3
A2 = 1.0
A = 1.0 x 10-2
We train Energy24, Stock24, Stock72, Stock360 and Solar Weekly datasets for 400 epochs, with Adam optimizer. For the NN5 daily, and Fred MD datasets we trained for 500 epochs, and for the Temperature Rain dataset train iHT for 1500 epochs.
Hardware and Software We implement our method in Python and the experiments are ran using 24dn.22larg AWS instance with NVIDIA T4 GPU, 8 CPU and 32gb of RAM.
E
Additional experimental results on real-world time series synthesis
We show additional comparisons of time series synthesis the four Monash datasots in Table 13 Out method still shows competitive results across most datasets, with best predictive score in three cases, only loosing against Latent ODE in the FRED-MD dataset.
20

---

Published in Transactions on Machine Learning Research (08/2024)
Table 13: Generation results on FRED-MD, NN5 Daily, Temperature Rain, and Solar Weekly,
Data
Metric
RNN-VAE
GP-VAE
ODEVVAE
Latent ODE
TimeGAN
SDEGAN
SaShiMi
LS4
iHT (Ours)
FRED-MD
Marginal
0.132
0.152
0.122
0.0416
0.0813
0.0841
0.0482
0.0221
0.0177
Class, f
0.0362
0.0188
0.0282
0.327
0.0294
0.501
0.00119
0.544
1.3278
Prodiction +
1.47
2.05
0.567
0.0132
0.0575
0.677
0.232
0.0373
0.0181
NNG Daily
Marginal
0.137
0.117
0.211
0.107
0.0396
0.0852
0.0199
0.00671
0,00893
Class. f
0.000339
0.00246
0.00102
0.000381
0.00160
0.0852
0.0446
0.636
0.4982
Prodiction 4
0.907
1.109
1.19
1.04
1.34
0.849
0.241
0.23499
Temp Rain
Marginal
0.0144
0.0106
0,198
0.758
0.0831
Class, f
0.00000122
0.0000123
0.0000319
0.0000419
0.00271
0.0100
0. 0.0000167
0.976
11.2493
Prediction 4
159
2.305
1.133
145
1.96
2.46
2.12
0.521
0,132
Solar Weekly
Marginal 4
0.0903
0.308
0.153
DJ0853
0.0496
0.147
0.173
0.0459
0.03273
Claes,
0.0521
0.000731
0.0998
0.0521
0.64189
0,00102
0.683
Prodiction
1.25
1.47
0.761
0.973
0.237
0.976
0.578
0.141
0.0739
F
Additional experiments with real irregularly sampled data
In this section, we present additional results on real-world datasets that are inherently irregularly sampled, such as PhysioNet and USHCN. Given that the established evaluation of time series generation (e.g., through discriminative and predictive scores) is not possible in these datasets, we focus the task of interpolation, following the approach taken in prior works (e.g., De Brouwer et al. 2019, Zhou et al. 2024, Rubanova et al. 2019). In particular, we adopt the experimental setup outlined in Zhou et al. (2023).
Data Following [Zoou et al. 2023 we use PhysioNet and USHCN as our datasets of choice. The United States Historical Climatology Network (USHCN) (Menne et al. 2016) is a climate dataset containing daily measurements from 1,218 weather stations across the US for precipitation, snowfall, snow depth, minimum, and maximum temperature. PhysioNet (Silva al., 2012) is a dataset containing health mcasurements of signals from 8,000 ICU patients in their first 48 hours. We follow the code provided by (Zhou al. [023] to process PhysioNet and the code provided by (Browwer et al. 2019 to pre-process USHCN. We mean squared error (MSE) to evaluate.
Table 14: MSE scores x x10-3 for interpolation on irregularly sampled time series data.
Data
RNN
RNN-VAE
ODE-RNN GRU-D
Latent ODE
LS4
iHT (Ours)
PhysioNet
2.92
5.93
2.23
3.33
8.34
0.63
3.35
USHCN
4.32
7.56
2.47
3.40
6.86
0.06
1.46
We compare iHT with RNN (Rumelhart al. 1986..
RNN-VAF (Chung et al. 2014
Rubanova et al.
2019), ODE-RNN (Rubanova et al.) 2019), GRU-D (Rubanova al. 2019). Latent ODE (Chen al. 2018 Rubanova et al.) 2019) and LS4. Table 14 presents the interpolation performance of various models on irregularly sampled time series from the PhysioNet and USHCN datasets. While LS4 consistently achieves the best performance, our proposed model demonstrates strong competitiveness. On PhysioNet, iHT's MSE of 3.35 x 10-3 outperforms several advanced models, including Latent ODE and RNN-VAE. On USHCN, iHT achieves the second-best performance with an MSE of 1.46 x 10 10-3, These results point towards iHT's capability to effectively handle irregular sampling across diverse domains, further demonstrating its potential for real-world applications.
G Training time comparison
Table 15 shows the training time of iHT and all the other baselines for the Energy and Stock datasets. iHT has the lowest training time across all datasets, with Fourier Flows having similar performance in the Stock
http:://gthubb.cm//eebrrrrottt
21

---

Published in Transactions on Machine Learning Research (08/2024)
datasets. In the case of Energy, given that Fourier Flows trains each feature separately, this sequential training increases the computational time because of the large number of features present in the dataset. The training times of TimeGAN and GTGAN are orders of magnitude larger for the datasets with the longest time series, the case of TimeGAN because it based on RNNs, whilst GTGAN's needs to solve various differential equations.
Table 15: Comparison of training time. iHT shows the shortest training time on all datasets.
Training Time (HH.MM)
Energy24
Stock24
Stock72
Stock360
iHT (Ours)
00:15
00:03
00:03
00:04
GTGAN
10:39
12:20
04:32
21:23
TimeGAN
12:28
11:40
34:30
65:00
FourierFlows
02:48
00:07
00:03
00:05
LS4
04:19
00:57
01:16
02:09
DiffTime
17:03
02:52
01:42
02:13
G.1 Additional Analysis of Computational Times
Our experiments shown in Figurd use the synthetic data-set of 8192000 data-points (e.g. number of sam- ples sequence length), organized in 5 datasets of different sequence lengths / E {80, 320, 1280, 5120, 20480J. Thus, we obtain 5 datasets of {102400, 25600, 6400, 1600, 400} number of samples, which we process using {1024, 256, 64, 16, 4} as batch-size, respectively. Thanks to this batch-size, we can fairly train each model for 100 iteration and always loading to GPU the same amount of data-points, for each sequence length and iteration.Fo inference, we only sample a single batch of data.
Impact of data loading All the times are computed directly in the training and testing loop, after the model has been alrcady loaded in the GPU memory, using NVIDIA T4 GPU. Morcover, we found that the data loading (both for training and inference) is negligible w.r.t. model computational time. We measure the loading being around 0.0145 sec for training (100 iterations) and inference 0.001 sec (1 iteration).
Fourier Flow Finally, we add Table 16 the computational times Fourier Flows, which trend still has some couvexing. We believe the slightly drop likely comes from not optimized code to handle large batch-sizes: the longer time-series (e.g., 320 and 1280) worsen the computational time, but the model also relieved by the smaller batch-size (e.g., 256 and 64).
Table 16: Left: Characteristics of datasets to assess computational time. Right: Details of training and inference times of Fourier Flows.
Time series length
Batch size
FFlows train time (sec)
FFlows inference time (sec)
80
1024
12.5
0.34
320
256
8.14
0.10
1280
64
6.24
0.08
5120
16
10.19
1.15
20480
93.72
39.24
22

---

Published in Transactions on Machine Learning Research (08/2024)
H Additional Trend-Seasonen decomposition analysis
In this section we provide further details of the analysis of iHT docomposition.
Synthetic dataset. We generated time series datasets that have trend (T), seasonal (S) and noise (R) components, or a combination of two of them. The trend was generated by randomly choosing the degree of
the polynomial, a sign and a slope. code example with the parameters is shown in Code Snippet [ To model the seasonality component we use a Sine function with the frequency sampled uniformly within [1,10]. Finally, for the residual component used Gaussian noise, with the standard deviation sampled between O and 0.2. For each dataset, generated 2000 time series of 200 time steps. Figure 7 shows examples of each dataset.
Code Snippet 1: Trend generation
def generate_trend ()
trend_slope = np random uniform (1. (1)
degree up random choice [1,]
sign np. random. choice 1))
trend = sign (trend_sloped wltpsssslppeeeeeee wttpeererllaaaemeeeeeeeeeeeee (*pplle **degree
return trend
Table 17
we compare iHT with STL (ezact) and STL (approx), in the previous dataset TSR, T+R, T+S the additional dataset of S-R R dataset. We can observe that in the S+R dataset iHT shows the worst performance in all three components. Interestingly, both STL (exact) and STL (approx) show higher errors than in the other datasets, showing that this is more challenging case.
T+S+R
SIR
-4.3
T+S+R
T+R
1.25
13
6.0
11 TM
4.0
35
10 30 15
Figure 7: Example of synthetic datasots with trend (T), scasonal (S) and noise (R) components, combination of two of them.
Table 17: Ablation study for trend--easoonalityyreddded decomposition of time series. We compare iHT against STL decomposition, for three dataset configurations: trend seasonality (T+S), trend residual (T+R), seasonality (S+R) and all three components (TSR).
MSE
TSB ST,, (oppexs)
T&R STL leppere))
T+8
SiR
(x10-33
STI, (ecaet)
HFT (Qura)
STL, (rcwet)
HFT (Ourz)
STL (TLwe))))
STI, (symme)
IRT (Ours)
STI.
STI. (apprre)
IHT (Ours)
Throd
0.06 a 1.08
1.04 = 4.0
U. 0.5
5U7 à 0.07
1.30 a 5.7
03 0.57
0.554:21
1.17 5.32
0.06 = U.45
015 019
071 1.8
5.11 8.31
Seuscmaltt
0.77 a 2.1
1.30
= 478
121 = 0.08
200 4 0.22
1.55 = 6.4
0.17 = 029
044 2,00
118 2:1.1
L.5 0.7
254 268
289 3.0
14.85 11.49
Restinls
0.17 a 0.15
0.38 = 0.12
1.9 a 0:2
2.15 a 0.39
0.18 = 0.13
0.41 = 025
001 001
0.0 6.07
125 = 0.1
284
2.45 3.07
12.8 a: ATL
we show the evaluation of the output of iHT for each dataset. The first three plots on each row correspond the distribution of trend, scasonality and residuals outputs from iHT against the ground truth components. We can see in Figures and 10 that for the T+S and T+R datasets, the trend shows good agreement, and in the case of no residuals, observe narrow distribution close to zero, while
In Figures g
9 10
and
[1
23

---

Published in Transactions on Machine Learning Research (08/2024)
in the case of no seasonality, the frequency histogram is also very narrow around zero. In Figure I which corresponds to S+R we can observe that the distribution of trend is quite broad, and this is in line with the results on Table 17, Even in the seasonality distribution we can observe a slightly worse match with regards to the other cascs. The plots on the right show the learned representations via t-SNE the embeddings. We can still observe a separation in trend up and down on the T+R datasel, although the separation high and low frequency the S+R plot is less obvious.
GF: Trend
GT:
Int: Ren
250
CH
64
(a) Trend
(b) Seasonality
(c) Residuals
(d) tSNE-trend
(e) tSNT-seasonality
Figure & (a,,,)): Distribution of the trend, scasonality and residual outputs of iHT vs ground truth. (d,c):
t-SNE visualization of trend and scasonality embeddings in iHT for the TSR dataset.
GT
IHT: Rcs
Tend up
IHE Trend
in1
Twis cean
rightng Lon foot
(a) Trend
(b) Seasonality
(c) Residuals
(d) uSNE-trend
(e) ISNT-seasonalitt
Figure 9: (a,b,c): Distribution of the trend, scasonality and residual outputs of iHT vs ground truth. (d,c): I-SNE visualization of trend and seasonality embeddings in iHT for the T+S dataset.
GT: Trend
IHT
GT: files
Bend Teis
Trend
(a) Trend
(b) Seasonality
(c) Residuals
(d) tSNE-trend
Figure 10: (a,b,c): Distribution of the trend, seasonality and residual outputs of iHT vs ground truth. (d):
t-SNE visualization of trend embeddings in iHT for the T+R dataset.
HIT: Trend
GT: Res
Hun free
i11
(a) Trend
(b) Seasonality
(c) Residuals
(d) tSNE-seasonality
Figure 11: (a.): Distribution of the trend, scasonality and residual outputs of iHT vs ground truth. (d):
I-SNE visualization of seasonality embeddings in iHT for the S+R dataset.
24

---

Published in Transactions on Machine Learning Research (08/2024)
Real-Data Trend-Seasonen decomposition analysis
In this section we report an analysis on three real world datascts to show that iHT is able to disambiguate between trend, season and residual components after training. We used two well known real-world time series datasets that exhibit strong seasonality and trend, Atmospheric CO2, and Airline Passenger, and use the Stock72 dataset that exhibits strong trend and weak seasonality, as described section
The Atmospheric CO2 dataset corresponds to monthly data (January 1959 to December 1987) with a seasonality period of 12 CCeveland et al. 1990). We sliced the data into time series of sequence length 60.
The Airline Passenger dataset corresponds to monthly data (January 1948 to December 1960) with a seasonality period of 12 8 We sliced the data into time series of sequence length 36.
To estimate the presence of trend and scasonality of each datasct we use two metrics: the strength of trend and strength of seasonality (Hyudıan & Achanasopoulos 2018 Wang et al., 2006). As these datasets do not include ground-truth tttnt trend and seasonality, we estimate them using STL (Cleveland et al., 1990). Given a time series with additive decomposition in trend, scasonality and residual: y/- T, S R2 define the strength of trend as:
Fr = max (0, 10.1- Var(R.)
Var(T, + Rτ)
This measures the relative variance of the remainder component R2 the variance of the trend with the remainder component. The value ranges from 0 to 1, with 0 indicating trend and 1 indicating strong trend.
Strength of seasonality is defined by:
Var Re) Fs = max 0, Var(S, + Rτ)
Which mcasures the relative variance of the remainder component R₂ to the variance of the scasonality with the remainder component. A value close to 0 indicates little to no seasonality, and a value close to 1 indicates strong seasonality.
Table 18 shows these metrics for the three datascts. As expected, Stock72 doesn't exhibit strong scasonality, while the other two datasets exhibit strong trends and seasonality, with all values on average above 0.98.
Table 18: Strength of trend and seasonality in the CO2, Air Passanger and Stock72 datasets.
CO2
Air Pass
Stock72
Trend strength
0.99
0.01
0.98 +0.01
0.99 + 0.01
Seasonal strength
0.99 = 0.00
0.99
#0.01
0.32 + 0.09
We trained iHT on each dataset and we show in Figures 12 and 13 the reconstructed time series and the output of each of the individual blocks for the CO2 and Air Passanger datasets, respectively. For the output of the trend, seasonality and residual block, we compare with the STL method. We observe that there is a good agreement in the trend and seasonality components in both datasets. The STT. method requires parameter the period of the seasonality, which is known for both datasets, while our approach is able to estimate the seasonality from the data. In the case of the Stock72 dataset, for the STL comparison, we estimated the period using Fourier transform and retrieving the dominant frequency. Figure L4| shows the decomposition generated by iHT and compared against STL. We can see in this case that the scasonal component quite small magnitude compared to the residuals.
Downloaded from https:://ww..atteeeeess htwpe////woo.ooooooo///e//a/aaae
25

---

Published in Transactions on Machine Learning Research (08/2024)
0.45
Soound Stuth
5TL Tend
0.755
STL Recibal
lecdan
0.32
0.13
0.477
0.73
0.000
0.92
-2.00%
177 Seasonatity
37C SeasoraPty
30
60
32
10
10
20
0.33
Aeconatruteed
BC- Block
0.073
0.25
0.20
0.075
0.02
0.35
0.10
0.16
-2 5.005
0.53
0.4
-2.075
LIT- Sepsonaity
Terestopp
0.75
Groundd muthh
Reconatructed
3TL Tund
B5- block
0.733
STL Rhsibual
Residuad blact
0.55
0.6
0.000
0.077
Sezsensity
0.39
2.000
0.92
0.15
60
23
00
20
22
70
60
10
20
Figure 12: iHT decomposition on CO2 dataset.
0.32
2.13
Secuonallty BiKE
0.03
HI Hoythul Lloc
511 Tosd
5/L- Sospurallyy
0.45
0.50
0.34
2.000
0.92
2.00
0.50
0.23
0.25
0.15
0.200
-0.31
10
Aecosstucte
Funn
Saateraliy
03.35
13.
0.2
0.04
0.7
0.19
0.00
-0.2
51000
1992
1491
0.752
Droundl traii
Seocersityy
Hoskal
0.7
0.525
0325
0852
0.2
0.000
201
-0.
0.23
30
20
P91 51000
1893 51000
1992 513000
St885
Figure 13: iHT decomposition on Air travel dataset.
26

---

Published in Transactions on Machine Learning Research (08/2024)
0.33
Thend
6.000
Sensrrllt
0.313
0.27
5.000
6332
0.00
0200
0.19
0365
1.000
0.332
0.18
0350
STL Der
10
20
30
80
600
10
23
10
10
10
70
30
13
10
20
30050
0.54
H4T ver teach
Sepsorailty LlIce
6.000
01312
0.000
0.333
0.63
0.60
0.000
0.32
0.7
0.50
-2 010
-0.315
0.58
Thme
0.26
Cround
0.24
0.010
0.22
0.22
1.000
11:33
0.23
0.20
5.000
0333
100
0.31
-2.000
-0333
0.19
0.38
-2.000
-0333
STL Del cual
10
20
20
83
50
60
10
23
16
40
10
10
20
30
23
52
60
10
32
stege
27087
stepe
Figure 14: iHT decomposition on Stock72 dataset.
27

---

Published in Transactions on Machine Learning Research (08/2024)
J Time Series Generation using Trend Component
Here consider peculiar capability of iHT that enables the user provide an input trend component to generate time-series accordingly. iHT generates new time series by performing interpolation in the embedding space between two time series, and then generating the novel weights of TSNet that represents the novel time series. This allows us to control the gemeration by projecting a desired trend pattern in iHT to use in the generation process. To evaluate the performance of this guided generation, we use iHT trained on Stock24. We provide an imput trend and generate 1000 time series using iHT. In this experiment we consider the following baselines: DiffTime, RCGAN, TimeGAN, GT-GAN. While DiffTime (Coletta al. 2023) naturally designed for constrained time-series generation, adapted RCGAN, TimeGAN, GT-GAN architectures to deal with the input trend. In detail, we re-trained them conditioned models using an additional input trend, computed as polynomial interpolation from original data during the training. In Figure 15 show how the generated time-series follow the trend, For each approach we generate 1000 samples and we plot their 5-95th percentile values as the light-bluc shaded arca, while trend is the dotted orange linc. In Table 19 wc report the quantitative metrics, which evaluate how much each generaled lime-series deviate from the input trend by computing the L2 distance and Dynamic- Dynamie-Time- Warping (DTW) distance between the generated sample and the input trend. The results show that iHT has among the best performance and it is competitive w...t. to DiffTime, which is specifically designed to incorporate such trend constraints.
Table 19: Time-Series Trend Generation Stock24 dataset.
Algo
L2 Distance
DTW Distance
iHT (Ours)
23.68+18.6
14.43+13.60
DiffTime
19.83=5.400
15.42114.79
GT-GAN
1304.2:1126.99
1303.9+1303.1
TimeGAN
88.18+12.10
17.29
RCGAN
60.56+19.20
32.94+66005
Figure 15 shows additional qualitative results using iHT trained on Stock72 with a wide diversity of input trends. We can that in all cases, the input trend is within the 5-95th percentile values of the generated time series, showing good agreement of the synthetic time series with the input trend.
IHT
GT-GAN
TimeGAN
ROGAN
DifTTime
1000
330
340
5-95th percontile
320
Trend
300
170
.00
300
300
(a) HHT
(b) GT-GAN
(c) TimeGAN
(d) ROGAN
(e)
DiffTime
Figure 15: A visualizations of time-series generated according the input Trend. The orange dotted time-series is the trend, and the shaded blue area shows the 5% and 95% percentiles of the generated synthetic time-series. Our approaches show among the best performance with time-scries closer to the input trend.
K Additional Ablation Studies
In this section we provide further ablation studies. Firstly we report the impact in performance of the architecture of iHT on irregularly sampled data. We then study the impact in performance of the size of the latent dimension. Furthermore, we report generation metrics when splitting the dataset into train-test, including diversity metric.
28

---

Published in Transactions on Machine Learning Research (08/2024)
1020
800
790
1100
1000
780
1030
1080
1020
760
0
20
40
60
20
40
60
20
40
20
40
60
960
750
1200
730
940
720
740
710
920
730
20
60
60
60
20
40
Figure 16: A visualizations of time-series generated by iHT according to an input Trend, on the Stock72 dataset. The orange dotted time-series is the trend, and the shaded blue area shows the 5% and 95% percentiles of the generated synthetic time-series.
K.1 Analysis of model architecture
22 we report the predictive and discriminative scores for two simpler iHT models as explained in Section 4.7 for the case of irregularly sampled data with different levels of dropped values (30, 50 and 70, respectively). In all cases of missing values, don't observe difference in performance for the predictive score in the case of iHT (no FFT), but we do observe some loss in performance for the case of the simpler architecture without TSR decomposition for the Stock datasets (iHT-SIREN), In the case of the discriminative score, Energy24 dataset shows a degradation in performance for both cases of iHT-SIREN and iHT (no FFT). For Stock24 we see a degradation in performance for the iHT-SIREN model, and the case of small percentage of missing values there degradation performance when not including the FFT loss. In the case of larger amounts of missing values, we see the opposite, that not using the Fourier loss yields higher discriminative scores. Similarly for the longer time series datasets of Stock72 and Stock360 we observe a loss in performance in discriminative score when using the simpler iHT-SIREN but in the case of iHT (no FFT) there is no clear trend.
In Tables 20 21
Table 20: Ablation study for model architecture: comparison of iHT against simpler configurations: iHT (no FFT) without FFT loss, and iHT-SIREN without TSR decomposition, for irregularly sampled data with 30% dropped values.
Energy24
Stock24
Stock72
Stock300
Energy24
Stock24
Stock72
Stock360
Prodictivn Score
Dise. Score
HHT (no FFT)
0.049
0.013
0.188
0.168
iHT
0.452
0.059
0.017
0.014
0.167
(no FFT)
1HT-SIREN
0.050
0.015
0.189
0.169
iHT-SIREN
0.473
0.089
0.036
0,008
Table 21: Ablation study for model architecture: comparison of iHT against simpler configurations: iHT (no FFT) without FFT loss, and iHTSSIENN without TSR decompasition, for irregularly sampled data with 50% dropped values.
Energy24
Stock24
Stock72
Stock360
Energy24
Stock24
Stock72
Stock360
Predictive Score
Dise. Score
IHT
0.051
0.014
0.187
0.168
IHT
0.472
0.102
0.011
0.004
HHT (no FFT)
0.051
0.012
0.167
iHT (no FFT)
0.482
0.065
0.019
0.011
AHT-SIREN
0.050
0.014
0.189
0.169
HHT-SIREN
0.481
0.106
0.017
0.005
29

---

Published in Transactions on Machine Learning Research (08/2024)
Table 22: Ablation study for model architecture: comparison of iHT against simpler configurations: iHT (no FFT) without FFT loss, and iHT-SIREN without TSR decomposition, for irregularly sampled data with 70% dropped values.
Energy24
Stock24
Stock72
Stack360
Energy24
Stock24
Stock72
Stock360
Predictive Score
Disc. Score
IHT
0.053
0.014
0.187
0.168
IHT
0.482
0.115
0.020
0.011
HHT (no FFT) 1HT-SIREN
0.062
0.015
0.188
0.168
iHT (no FFT)
0.493
0.062
0.013
0.008
IHT-SIREN
K.2
Analysis of the dimension of Zr, 2s and Z
The results of varying the value of the three latent dimensions shown in Table 23 and Table 24 where the
report Z T S R (e.g., Z 4 6 8 represents ZT 4, Zs 6 and Z = 8)
Table 23: Results for different values of Z on Stock 24 dataset.
Z 2 3
3
2
4
6
6
Z
10
15
15
2 40
60
60
2
100
150
150
Pred score
0.037++0.000 0.0614:0.334
0.038++0.000
0.037=00.000
0.036++0.000
0.037
Discr score
0.143:00009
0.054=0.022
0.215+:00032
0.382:10.034
Table 24: Results for different values Z on Stock 72 datasct.
7 2.233 3
7_ 6
10 15 15
7_40__ 60_ 60
100
150 150
Discr-score
0.046:+0.022
0.044:0.017 0.184+0.001
0.028=0.027 0.185-+0.001
0.025#0.019 0.185++0.00
0.035=0.034
Pred-score
0.186-+0.002
0.185++0.001
K.3 Train and Test Split on Stock Data
In Table 25 we show the evaluation of the approaches while randomly splitting data between training set (i.e., 80% of dataset) and test set (i.e., 20% of dataset). The table reports the quantitative metrics for the regular Stock data with 24 length, while Figure shows the t-SNE visualization (Top) and the data distribution (Bottom). These results confirm the results from the main paper, with iHT achieving the best performance in discriminative and predictive scores and second best performance in Marginal Score. Additionally, given that the diversity of generated time scries is an important aspect of the generation task, we use a novel metric to quantitatively evaluate the diversity of time series, namely sym-RRecall (Alaa et al.) 2021) which extends the 3-R Recall metric from (Khayatkhoei AbdAlmageed 2023). This metric quantifies the diversity of time series as the extent to which synthetic samples cover the full variability of real samples. The results are shown in the last row of Table 25 We observe that iHT shows the second best performance with regards to diversily, with DiffTime showing the best performance in diversity.
Table 25: Generation performance on regular Stock24 dataset
Metric
iHT (Ours)
GT-GAN
TimeGAN
DiffTime
LS4
FFlows
Discr-score $
0.05411.002
0.27310.046
0.068110.018
0.079
0.1541100080
0.426 0.2
Pred-score
0.03740.000
0.046+0.001
0.043+40.001
0.044=0.001
0.039:00.000
0.055+40.008
Marginal Score +
0.355
0.403
0.434
0.359
0.513
0.335
sym-Recall
0.594
0.500
0.413
0.787
0.206
0.000
30

---

Published in Transactions on Machine Learning Research (08/2024)
(a) iHT (Ours)
GT-GAN
TimeGAN
(d) DiffTime
LS4
(f) FF
Cola Canary Diime
Datn Dareity Fnthoes
Cats Caratty Datmas
Dosky Estmore
Tireca
Diflime
Cusa Cerad
0.0
1.0
0.3
Duta
Data
(g) iHT (Ours)
(h) GT-GAN
TimeGAN
(j) DiffTime
(k) LS4
(1) FF
Figure 17: (Top) I-SNE visualizations regular Stock24 data where greater overlap of blue and red dots shows a better Iaaributional-similann between the generated data and original data. (Bottom) Data distribution on regular Stock24 data. Our approach shows the best performance.
L Visualizations with tSNE and data distributions
In this scction we report an additional evaluation of the synthetic and real data distributions. We evaluato the synthetic distributions on Stock24, including missing data from 30% to 70%; then we analyse synthetic data longer stock time-series, i.e., stock72 and stock360; and finally we evaluate the synthetic distributions for Energy data, which has 28 dimensions.
L.1 Data distribution
First we plot the real (blue) and synthetic (orange) distributions empirically evaluated through kernel-density estimation of real and generated data. Figure 18 shows the empirical distributions for Stock24, where iHT has among the closest match with original data. The superior performance of iHT is more evident with irregular data, from Figure to Figure 21 where iHT is always able to closely resemble the real data distributions.
The performance of iHT is consistent with longer time-series (Figure 22 and Figure 23 and highly dimensional
data like Energy Figure
L.2 tSNE visualization
We now evaluate the real (bluc) and synthetic (red) distributions through t-SNE visualizations. Figure 25 shows the I-SNE plots for Stock24, where iHT has among the best performance (i.e., the synthetic data almost completely overlap with real data). As mentioned in the previous section, the superior performance of iHT is more evident with irregular data, from Figure 26 to Figure 28 where iHT is the only method to closely resemble the real data distribution. While GT-GAN reproduces similarly the real data distributions, the synthetic distributions are more condensed around the original data, and don't cover the full space.
For longer time-scrics (Figure 29 and Figure 30 the t-SNE plots show that iHT is able to better reproduce the original data distributions. Finally, Figure shows the t-SNE plots for energy data where with consistent performance for iHT.
31

---

Published in Transactions on Machine Learning Research (08/2024)
Data Density Estimate
5
Original Synthetic
Data Density Estimate
Original GT-GAN
Data Density Estimate
Original TimeGAN
4
4.
3
2-
2
1
0.00
0.25
0.50
0.75
0.0
0.5
1.0
Data Value
Data Value
Data Value (c)
(a) iHT
(b) GT-GAN
TimeGAN
Data Density Estimate
Original
Data Density Estimate
Original
Data Density Estimate
Original Synthetic
Synthetic
4-
Synthetic
4-
3-
2.
2/
1-
0.0
0.5
1.0
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
Value
Data Value
Data Value
(d) DiffTime
(e) LS4
(f) FF
Figure 18: Data distribution on Stock24 data.
Data Dondit Extimati
5
Original Symiheei
Data Diceety Entimate
Originel Synthetic
Data Densty Extemats
12.5
Original Symthetic
Dat Denaty betmate
Driginal Synthetic
Data Demaity netimara
Original Synthwtic
x
10.0
2
2:5
0:25
0.50
0.75
0.50
1.50
0.75
Data Value
Data Value
Data Valuee
Data Value
Data Value
(a) iHT
(b) GT-GAN
(c)
DiffTime
(d) LS4
(e) FF
Figure 19: Data distribution on irregular Stock24 data (Missing 70%).
Data Denatyy Estimate
Original Synchetic
Date Deraaty batimate
Original Synthetic
Data Densty Estimate
10
Data Derecty Extimete
. of & up
Original Synchetic
Denstty Extimate U
Oriiinal Synthetic
Synchatic
3.
2u
0.00
0.25
0.50
0.75
0.00
0.25
0.30
0.75
1.00
0.00
0.25 Value
0.30
0.75
0.00
0.25
0.50
0.75
0.0
0.5
Doto Value
Value
Value
Value
(a) iHT
(b) GT-GAN
(c)
DiffTime
(d) LS4
(e) FF
Figure 20: Data distribution on irregular Stock24 data (Missing 50%).
Data Denutt Estinaraa
Original Synthetic
Date Denalty Estimecee
Original Syythhttic
Data Denutty Estimaze
0
Original Synthetic
Density Estimene
Original synthetic
Denatty Estemore
Original bymbheic
4.
0.00
0.55
0.50
0.75
0.0
0.5
1.0
8:00
0.25
0.50
0.75
0.00
0.25
3.50
0.75
2035
0.60
0.25
0.50 0.75
Data Value
Data Value
Data Volue
Data
Value
Date Value
(a) iHT
(b) GT-GAN
(c)
DiffTime
(d) T.S4
(e) FF
Figure 21: Data distribution on irregular Stock24 data (Missing 30%).
32

---

Published in Transactions on Machine Learning Research (08/2024)
Sata Denuty Est make
Griginal Symtheti
Sete Density Ext moke
Driginal Synthetic
200 Conutt E1t: maka
origral Synthetic
Deta Cersity Estimare
Original Synchetic
Original Symtheic
d
Jeta
0.0
3.5
1.0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Data Value
Data Value
Data Value
Date Value
Data Value
(a) iHT
(b) GT-GAN
(c)
DiffTime
(d) LS4
(e) FF
Figure 22: Data distribution on regular Stock72 data.
Data Dardty Estiatt
Data Extinate
Original Syntheticc
Datia Dersiy Exiinnate
Original Synthotic
Data Demiyy Exiimete
Original
Dara Derery Estimate
Orignall Synthetic
Synthetic
20.0
Synthetic
7.5
5.0
2.5
0.00
0.25
1.00
0.0
1.0
0.00
0.25
0.50 0.75
1.00
0.00
0.25
0.50
0.75
Data Value
Data Value
Data Value
Dasa Value
Data Value
(a) iHT
(b) GT-GAN
(c) DiffTime
(d) LS4
(e) FF
Figure 23: Data distribution on regular Stock360 data.
Data Density Estimate
6
Original Synthetic
Data Density Estimate
12:
Original GT-GAN
Data Density Estimate
12
Original TimeGAN
10:
10
4
8-
6-
2
2-
8.2
0.4
0.6
0.2
0.4
0.6
0.2
0.4
0.6
Data Value
Data Value
Data Value
(a) iHT
(b) GT-GAN
(c) TimeGAN
Data Density Estimate
Original Synthetic
Data Density Estimate
10-
Original Synthetic
Data Density Estimate
10:
Original Synthetic
a-
6-
4-
2
.2
0.2
0.4
8.2
0.4
0.6
8.2
0.4
0.6
Data Value
Data Value
Data Value
(d) Difflimee
(e)
LS4
(f) FF
Figure 24: Data distribution on Energy data.
33

---

Published in Transactions on Machine Learning Research (08/2024)
Dew
-25
-1.3
2004
(a) iHT
(b) GT-GAN
(c) TimeGAN
12ME pt1
(d) RCGAN
(e) Difflimm
(f) LS4
(g) FF
Figure 25: t-SNE visualizations on Stock24 data, where greater overlap of blue and red dots shows a better distriuttional-smmilartty between the generated data and original data. Our approach shows the best performance.
Original
-10
GE-MM
(a) IHT
(b) GT-GAN
(c) DiffTime
(d) LS4
(e) FF
Figure 26: t-SNE visualizations on irregular Stock24 data (Missing 70%), where a greater overlap of blue and red dots shows a better distributional-smmilaaii between the generated data and original data. Our approach shows the best performance.
Original
-5
-10
(a) IHT
(b) GT-GAN
(c) DiffTime
(d) LS4
(e) FF
Figure 27: L-SNE visualizations on irregular Stock24 data (Missing 50%), where a greater overlap of blue and red dots shows a better distributional-similaiii between the generated data and original data. Our approach shows the best performance.
34

---

Published in Transactions on Machine Learning Research (08/2024)
Driginal GE-BAN
IHT
(b) GT-GAN
(c) DiffTime
(d) TS4
(e) FF
Figure 28: t-SNE visualizations on irregular Stock24 data (Missing 30%), where a greater overlap of blue and red dots shows a better distributional-ssmilare between the generated data and original data. Our approach shows the best performance.
CT.OW
H0.6
(a) iHT
(b) GT-GAN
(c) TimeGAN
(d) RCGAN
(e) DiffTime
(1) LS4
(g) FF
Figure t-SNE visualizations on Stock72 data, where greater overlap of blue and red dots shows a better distribuutionalssiiilrtty between the generated data and original data. Our approach shows the best. performance.
35

---

Published in Transactions on Machine Learning Research (08/2024)
TeaSSH
-44
-23
-04
-00
IHT
(b) GT-GAN
(c) TimeGAN
-80
(d) RCGAN
(e) Diff
(I) LS4
(g) FF
Figure 30: tE visualizations Stock360 data, where greater overlap of blue and red dots shows better distributional-ssmilarty between the generated data and original data. Our approach shows the best performance.
Original GT-GAN
Vine
-10
10
(a) iHT
(b) GT-GAN
(c) TimeGAN
!
(d) DiffTime
(e) LS4
(f) FF
Figure 31: t-SNE visualizations Energy data, where greater overlap of blue and red dots shows better distributionaalsiiiilriyy between the generated data and original data. Our approach shows the best performance.
36

---

Published in Transactions on Machine Learning Research (08/2024)
M Financial Returns and Autocorrelation
We now evaluate the distributions of two well known properties (i.c., stylized facts) of financial time-scries, namely the returns and autocorrelation. We consider stock uni-variate data with length of 72. Figure 32 and Figure 33 show the refurns and the audocorvelation of returns distributions, respectively. The two figures confirm the ability of iHT to learn the real data properties (i.e., the real and synthetic distributions mostly overlap).
Retums IIT
Returne DifMiie
Retums GTTG0N
Returns TimecANN
mal
17000
mal
2.000
10000
33302
3000
23000
4400
10000
3602
4000
2000
1000
2900
-0.00
0.22
0.2
Dary Paturr
Daiyy Partrm
Daiyy Pattom
Dady Returm
(a) iHT
(b) DiffTime
(c) GT--GAN
(d) TimeGAN
11500
3330
km
3300
12300
2000
6333
40000
2000
400
3950
2500
3300
Daily
Daiky
(e) RCGAN
(f) Fourier Flow
(g) LSS
Figure 32: Returns distribution of stock timo-scrics with length 72.
37

---

Published in Transactions on Machine Learning Research (08/2024)
Auto Cor IIT
Auto Corr Difflime
Auto Com GT-GAN
Auto Corr TimeGAN
2500
3000
2000
2000
4000
2331
13:00
3000
01.0
14C2
750
2003
1100
360
230
3000
500
1.2
4.5
6.0
8.5
1.0
-1.0
-0.
3.0
3.5
2.2
-2.5
00
05
3.00
-0.5
60
05
15
oftoConducinnl Roturoo
SutdSonddiion Ricuure
AutuCerddtiin Pictart
AddiDerrdaaan of Pettare
(a) iHT
(b) DiffTime
(c) GT-GAN
(d) TimeGAN
Auto-Car ROGAN
Acto-Comm Corf FF
Auto L54
3552
rot
km
Wm
2500
1000
1500
2300
5000
1000
AatoComearoon Reture
AuteGermiitten of Tafarm
AdtoCerdaaonn of Ranuro
(c) RCGAN
(I) Fourier Flow
(H) LS4
Figure 33: Autocorrelation of returns distributions of stock time-series with length 72.
38

---

Published in Transactions on Machine Learning Research (08/2024)
N Visualizations of Generated Samples
Finally we report examples of generated time-serics. It is worth to mention that in Figure 34 the synthetic time-series from iHT effectively respect the open-higgh-low-clsee relationship from the real data high (low) is the highest (lowest)) series. Such data property is preserved also when the model is trained on missing data, as shown in Figure 35 Figure 36 and Figure 37 With the exception of DiffTimm, which is specifically designed for constrained time-scries generation, most of the existing approaches do not preserve such property.
Energy data is shown in Figure 38 for regular time-series, and in Figure 39 Figure 40 and Figure 41 for irregular time-series. Considering that Energy has 28 features, with different scales, plot only the first 5 normalized features.
Finally, we plot Longer-times for regular stock72 in Figure 42 Figure and Figure 45
While we plot the irregular stock72 in Figure 43
Difflime
Original
1569
2250
640
2000
1540
2000
1750
1900
1500
6.0
16
20
10
20
10
20
16
1.0
675
3800
34:0
320
615
1340
1400
1350
300
1200-
16
20
10
20
10
20
10
1.0
330
1800
1175
Open
1159
1380
1600-
2300
1125
2250
1100
10
20
10
20
10
20
20
20
Figure 34: An example of Regular Stock24 samples.
L.54
DiffTima
IHT
Original
1600
1325
1500
2000
6440
2350
1475
1480
1500-
1500
1450
1460
1000-
35
20
10
20
10
20
20
1.0
2250
20004
6.75
1650
1420
1500
2150
3825
1800
1400
1000-
2050
1389-
10
20
1.
20
10
20
30
100
2200
1860
Low
L150
page
1640
1125
1520
1000
10
16
10
Figure 35: An example of Irregular Stock24 samples (70% missing data).
39

---

Published in Transactions on Machine Learning Research (08/2024)
DiffTime
Original
6.0
1540
3600
1.600
2000
1440
620
15.00
1420
1500
2350
1760
3600
2000
2300
1740-
1500-
600-
1550
1720
1000-
2200
1750
1400
2000-
Clede
1460
Low
1700
1440
1540-
1500
Plgh
14200
1529
1850
1400
1500
30
Figure 36: An example of Trregular Stock24 samples (50% missing data).
DiffTime
Original
1520
2500
600
1040
1380
2000
3420
620-
1480
1369
1500
1460
1000
2000
2010
1500
2205
630
2000
2375-
2150-
1950
1.400
2125
575
1000
2000
L175
Cican
Low
Pagh
1560
1500
1503
1000
1520
1482
10
10
30
26
Figure 37: An example of Trregular Stock24 samples (30% missing data).
40

---

Published in Transactions on Machine Learning Research (08/2024)
DiffTime
11T
Original
9.6
0.1
0.3-
0.2
02-
0.34
9.0
10
25
3.6
0.3
0.4
0.4
0.2
0.2
0.0
10
25
0.75
0.3
1sa2
0.4
0.25
0.1
0.0
3.0
0.25
10
Figure 38: An example of Regular Energy24 samples.
DirlTime
INT
Original
3.6
0.4
0.4
0.2-
0.3
0.0
20
1b
10
10
0.4
0.5
0.2
0.2
0.3
20
1b
10
0.4
0.3
Tea-3
0.1
02
20
20
10
Figure 39: An example of Irrogular Energy24 samples (70% missing data).
41

---

Published in Transactions on Machine Learning Research (08/2024)
L54
Difffime
11T
Original
0.6
0.4
0.4
0.0
0.0
0.4
0.5
0.41
0.4
0.2
-0.5
0.0
15
20
0.3
900-2
0.5
0.2
02
3.0
0.0
16
20
Figure 40: An example of Trregular Energy24 samples (50% missing data).
DirfTime
INT
Original
0.4
02
0.2
26
1b
20
10
10
0.8
0.4
0.5
0.3
0.4
0.0
0.3
-0.5
20
15
20
10
0.75
Tea-3
0.30
0.2
01
0.25
0.0
0.00
20
15
10
Figure 41: An example of Irrogular Energy24 samples (30% missing data).
42

---

Published in Transactions on Machine Learning Research (08/202)
L54
DiffTime
IIT
GT-GAN
TimeeAN
ROGAN
Original
1520
M
1005
275
15:0
1550
15:00-
1540
1540
1873
225
200
300
V/
50
16770
50
275
15755
38:40
1850
1880
200
500
1822
1775-
390
1525
1750
1500
1620
1775
1800
1725
1:0
400
3475
1700
350
56
50
1.655
1820
14.00
1.640
750
500
1700
2230
1620
M
34:00
1600-
1685
2350
1580
1600
650
262
1350
50
50
Figure 42: An example of Regular Stock72 samples.
DiffTime
GT-GAN
Original
750
550
302-
480
1.000
280
300
11
300
440
140
2.00
453
40
do
20
20
40
40
220
66
220
10000
7.50
AA/ M
280
300
5000
1.000-
260
1300-
200
850
40
d0
20
40
60
300
105-
50
1.00
850-
520
150
500-
-30
100
425
20
40
60
20
40
Figure 43: An example of Trregular Stock72 samples (70% missing data).
43

---

Published in Transactions on Machine Learning Research (08/2024)
L.44
DiffTime
IIT
GT-GAN
Original
102
300-
M
525
300
100-
475-
750
40
60
20
40
60
1/1
20.00
150
400
400
475
750
1300-
740
362
1024
340
20
1000
60
400
20
40
320
400
4000
235
200
302-
100
100-
650
6-
80-
200
186
200
40
60
20
40
Figure 44: An example of Trregular Stock72 samples (50% missing data).
DiffTime
GT-GAN
Original
400
330
250
322-
VM
350-
225
1 PM
310
870-
70-
60-
300
175
60
20
60
26
46
60
240
240
600
230-
502-
530
342-
700-
200
230
2.0
20
20
60
60
1050
520
790
500
780
300
700-
226-
480-
770
100-
460
782
740
146
60
20
69
40
Figure 45: An example of Trregular Stock72 samples (30% missing data).
44

---

Published in Transactions on Machine Learning Research (08/2024)
Supplemental References
Ahmed M. Alaa, Boris van Breugel, Evgeny S. Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Nuri Benbarka, Timon Höfer, Hamd ul Moqect Riaz, and Andreas Zell. Seeing implicit neural representations fourier series. pp. 2292, 2022.
Andrew Brock, Theo Lim, J.M. Ritchie, and Nick Weston. SMASH: One-shol model archilecture search through hypernetworks. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Robin John Hyndman and George Athanasopoulos. Forecasting: Principles and Practice. OTexts, Australia, 2nd edition, 2018.
Mahyar Khayatkhoci and Wacl AbdAlmageed. Emergent asymmetry of precision and recall for mcasuring fidelity and diversity of generative models in high dimensions. In Proceedings of the International Conference on Machine Learning (ICML), 2023.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Jacob Russin Russin, Randall O'Reilly, and Yoshua Bengio Bengio. Deep learning needs prefrontal cortex. In Bridging AI and Cognitive Science ICLR 2020 Workshop, 2020.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learuing with latent embedding optimization. In Proceedings of the Internalional Conference on Learning Representations (ICLR), 2019.
Kenneth O. Stanley, David B. D' D'Ambrosio, and Jason Gauci. Hypercube-Based Encoding for Evolving Large-Scale Neural Networks. Artificial Life, 15(2):185 212, 04 2009.
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural reprosentations. In Proceedings of the IEEE Conference on Compuler Vision and Pullern Recognition (CVPR), 2021.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neurol Information Processing Systems (NeurIPS), 2021.
Johannes von Oswald, Christian Henning, Benjamin F. Grewe, and João Sacramento. Continual learning with hypernetworks. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Xiaozhe Wang, Kate Smith, and Rob Hyndman. Characteristic--ased clustering for time series data. Data Min. Knowl. Discov., 13(3):335 364, nov 2006.
Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture scarch. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Dominic Zhao, Seijin Kobayashi, João Sacramento, and Johannes von Oswald. Meta-learning via hyper- networks. In 4th Workshop on Meta-Learning Learning at NewrIPS 2020 (MetaLearn 2020). Advances in Neural Information Processing Systems (NeurIPS), 2020.
45


Reviews:
[{"review_id": "8vYHgRvDc5", "reviewer": "Reviewer_hq5b", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The paper focuses on an important problem of time series generation which can be useful in several practical use cases.\n- The paper proposes an interesting idea of disentangling trend, seasonality and residuals, and learning them separately.\n- Experiments show that the framework is able to outperform baselines on multiple datasets.\n- The paper is easy to follow.\n\n\n### Weaknesses\n- One of the major motivation and focus of the paper seem to revolve around irregularly sampled time series (as mentioned in abstract and introduction), but the paper lacks rigorous experimentation to support the claim that the proposed approach works better on irregularly sampled time series. The paper creates synthetic dataset to replicate `irregular sampling` instead of using widely available irregularly sampled time series datasets such as MIMIC-III, PhysioNet. \n- The paper misses some important line of work for irregularly sampled time series and fails to compare with some recent SOTA approaches in this space [1].\n- Also the paper claims that the proposed approach outperforms current state-of-the-art methods in challenging scenarios that involve long or irregularly sampled time series, while performing on par on regularly sampled data. It's hard to be convinced that the claim is true since the paper misses SOTA comparison and rigorous experimentation.  \n\n\nReferences:\n[1] S. N. Shukla and B. M. Marlin. Heteroscedastic Temporal Variational Autoencoder for Irregularly Sampled Time Series. In ICLR 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes iHyperTime, a framework for generating synthetic time series data. The proposed architecture is based on two key components: TSNet, an Implicit Neural Representation (INR) tailored for encoding time series data in a resolution-agnostic manner, facilitating the disentanglement of trend, seasonality, and residuals within a single time series, and iHyperTime, a hypernetwork architecture that utilizes TSNet to generate interpretable latent representations of time series data, enhancing the generalization capabilities across various types of time series, including multivariate, irregularly sampled, and long sequences. The evaluations demonstrate that iHyperTime outperforms existing state-of-the-art methods in generating synthetic time series, particularly excelling in challenging scenarios such as irregularly sampled or long sequences. The approach achieves rapid training speeds, comparable to the fastest existing methods for short sequences and significantly faster for longer sequences.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Rigorous experimentation on irregularly sampled time series data and comaprison with SOTA approaches in this space, e.g. [1]/", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZFzrxYsAqa", "reviewer": "Reviewer_2zXs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. Using INR’s resolution agnostic property for handling irregularly sampled or long series data.\n\n2. Presentation and readability are good.\n\n3. The proposed method is capable of handling irregular as well as long time series.\n\n**Weakness:**\n\nW1. The authors mention that “The use of a set encoder introduces permutation invariance in the computation”. Permutation invariance: Is it desirable for time series data where serial dependencies are present?\n\nW2. In Table 1, why does the predictive score drops by a magnitude of 10 (tables 2, 3, 4) in case of missing data for “Energy24” dataset whereas for all other datasets the values remain comparable to the values in Table 1?\n\nW3. The insights and trends from Table 7 are not clear to me. It appears that all variants are equally useful (indicated by the predictive scores) however, at the same time the discriminative scores indicate that the iHT (no FFT) and iHT-SIREN variants are not realistic enough? This seems to be contradictory. Please clarify.\n\nW4. I think the authors should perform the ablation of Table 7 with missing data as well. I believe there the difference between will be more significant allowing us to analyze the role of these contribution in case of missing data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a novel method of time series data generation using a hypernetwork iHyperTime that is responsible for predicting the weights of Implicit Neural Representation of the time series data, namely TSNet. Through extensive experimentation the authors show that the proposed method performs better than the existing method specifically for the time series which contain missing data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In general, I found that the Experiment section discusses the setup used for experimentation in details along with the tables for the various experiments that have been carried out. But the discussion lacks the insights that one can draw from these tables. In other words the interpretation of the tables is left to the reader (As pointed out in W2 and W3 above). I would request authors to make these inferences explicit in the text discussing the tables wherever possible.\n\nPlease fix or explain the weaknesses I mentioned above.", "broader_impact_concerns": "No ethics review needed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TDUcNLn1n8", "reviewer": "Reviewer_Gp9n", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-This paper proposes a new architecture, that combine the advantage of both TSNet and INR for the  trend-seasonality time\nseries representation.  the proposed method has stronger interpretability, as it points the information flow process in different networks.\n\n-The authors conduct extensive experiments on various benchmarks.  On all these datasets, it shows superiority on multiple benchmarks. The method also has strong generalization ability considering the diversity of these datasets.\n\nWeaknesses:\n\n-The proposed method combine the existing architectures to improve performance. It is better to further discuss the difference compared with TSNet and INR, which can further enhance the quality of this paper.\n\n-The authors propose three modules, trend hypernet, seasonal hypernet, and residual hypernet, which have different functions. How to validate that the practical function in a trained network? More evidences are required.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Based on TSNet, and INR, this paper proposes a new method for time series generation.  The experiments are conducted on multiple dataset with varying characteristics. The results show that the proposed method achieves better performance than the existing methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I would strongly encourage the authors to address **all** the concerns of the reviewers and particularly the consideration of the related work (and differences from those) and also to strengthen the comparisons with standard datasets, such as MIMIC-III, PhysioNet.
Decision: Accept (poster)