Time Series Continuous Modeling for Imputation and Forecasting with Implicit Neural Representations
Authors: Etienne Le Naour
Louis Serrano
Léon Migus
Yuan Yin
Ghislain Agoua
Nicolas Baskiotis
patrick gallinari
Vincent Guigue
Venue: Accepted by TMLR ()

Abstract: We introduce a novel modeling approach for time series imputation and forecasting, tailored to address the challenges often encountered in real-world data, such as irregular samples, missing data, or unaligned measurements from multiple sensors. Our method relies on a continuous-time-dependent model of the series' evolution dynamics. It leverages adaptations of conditional, implicit neural representations for sequential data. A modulation mechanism, driven by a meta-learning algorithm, allows adaptation to unseen samples and extrapolation beyond observed time-windows for long-term predictions. The model provides a highly flexible and unified framework for imputation and forecasting tasks across a wide range of challenging scenarios. It achieves state-of-the-art performance on classical benchmarks and outperforms alternative time-continuous models.

Published in Transactions on Machine Learning Research (04/2024)
Time Series Continuous Modeling for Imputation and Fore- casting with Implicit Neural Representations
Etienne Le Naour"
cliemne.lennoouraaee
EDF R&D, Palaiseau, France Sorbonne Université, CNRS, ISIR, 15005 Paris, Prance
Louis Serrano"
Sorbonne Université, CNRS, ISIR, 75005 Paris, Prance
Léon Migus
Sorbonne Université, CNRS, ISIR, 75005 Paris, Pronce Sorbonne Université, CNRS, Laboraloire Jacques-Louss Lions, 75005 Paris, France
Yuan Yin
yuen. www..yiiBBarrooomeemmiieerssreh
Sorbonne Université, CNRS, ISTR, 15005 Paris, Prance
Ghislain Agoua
ghislain.agoua
EDF R&D, Palaiseau, France
Nicolas Baskiotis
Sorbonne Université, CNRS, ISTR, 75005 Paris, France
Patrick Gallinari
Sorbonne Universilé, CNRS, ISIR, 75005 Paris, France Crileo AI Lab, Puris, France
Vincent Guigue
AgroParisTech, Palaiseau, Prance
Reviewed on OpenReview: https: // openreview. net/ forum? id= PivaXDlaal
Abstract
We introduce a novel modeling approach for time series imputation and forecasting, tailored to address the challenges often encountered in real-world data, such irregular samples, missing data, or unaligned measurements from multiple sensors. Our method relies seddimoous-iiee--eeeedde
model of the series' evolution dynamics. It leverages adaptations of conditional, implicit neural representations for sequential data. A modulation mechanism, driven by a meta-learning algorithm, allows adaptation to unseen samples and extrapolation beyond observed time-windows for long-term predictions. The model provides a highly flexible and unified framework for imputation and forecasting tasks across a wide range of challenging scenarios. It achieves state of the-art performance on classical benchmarks and outperforms alternative tine-contimuous models.
Equal contribution

---

Published in Transactions on Machine Learning Research (04/2024)
1 Introduction
Time series analysis and modeling are ubiquitous in a wide range of fields, including industry, medicine, and climate science. The variely, heterogeneily and increasing number of deployed sensors, raise new challenges when dealing with real-world problems for which current methods often fail. For example, data are frequently irregularly sampled, contain missing values, or are unaligned when collected from distributed sensors (Schulz and Stattegger, 1997; Clark and Bjornstad, 2004). Recent advancements in deep learning have significantly improved state-of-the-art, performance in both data imputation (Cao et al., 2018; Du el. al., 2023) and forecasting tasks (Zeng et al., 2022; Nie et al., 2022). Many prate-o--thearrs models, such as transformers, have been primarily designed for dense and regular grids (Wu et al., 2021; Nie et al., 2022; Du et al., 2023). They struggle to handle irregular data and often suffer from significant performance degradation (Chen et al., 2001; Kim et al., 2019).
Our objective is to explore allernatives to state-of-the-arl (SOTA) transformers able to handle, in a unified framework, imputation and forecasting tasks for irregularly, arbitrarily sampled, and unaligned time series sources. Time-dependent continuous models (Rasmussen and Williams, 2006; Garnelo et al., 2018; Rubanova et al., 2019) offer such alternative. However, until now, their performance has lagged significantly behind that of models designed for regular discrete grids. A few years ago, implicit neural representations (INRs) emerged as a powerful tool for representing images as continuous functions of spatial coordinates (Sitzmann et al., 2020; Tancik et al., 2020) with recent new applications such as image gencration (Dupont et al., 2022) or even modeling dynamical systems (Yin el al., 2023).
In this work, we leverage the potential of conditional INR models within meta-learning approach introduce TimcFlow: a unified framework designed for modeling continuous time series and addressing imputation and forecasting tasks with irregular and unaligned observations. Our key contributions are:
We propose a novel framework that excels in modeling time series as continuous functions of time, accepting arbitrary time step inputs, thus enabling the handling of irregular and unaligned time series for both imputation and forecasting tasks. This is one of the very first attempts adapt INRs that enables efficient handling of both imputation and forecasting tasks within unified framework. The methodology which leverages the synergy between the model components, evidenced in the context of this application, is a pioneering contribution to the field.
We conducted an extensive comparison with statc-of-tho-art continuous and discrete models. It demonstrates that our approach oulperforms continuous and discrete SOTA deep learning ap- proaches for imputation. As for long-term forecasting, it outperforms existing continuous models both on regular and irregular samples. It is on par with SOTA discrete models on regularly sam- pled time series while allowing for a much greater fiexibility for irregular samplings, allowing to cope with situations where discrete models fail. Furthermore, we prove that our method effortlessly handles previously unseen time series and new time windows, making it well-suited for real-world applications.
2
Related work
Discrete methods for time series imputation and forecasting. Recently, Deep Learning (DL) methods have been widely used for both time series imputation and forecasting. For imputation, BRITS (Cao et al., 2018) uses a bidirectional recurrent neural network (RNN). Alternative frameworks were later explored, e.g., GAN-based (Luo et al., 2018; 2019; Liu et al., 2019), VaE-based (Fortuin et al., 2020), diffusion-based (Tashiro et al, 2021), matrix factorization-basse (TIDER, Liu et al., 2023) and transformer-based (SAITS, Du et al., 2023) approaches. These methods cannot handle irregular time series. In situations involving multiple sensors, such as those placed at different locations, incorporating new sensors necessitates retraining the entire model, thereby limiting their usability. For forecasting, most recent DL SOTA models are based on transformers. Initial approaches apply plain transformers directly to the series, each token being a series element (Zhou et al., 2021: Liu et al., 2022; Wu et al., 2021; Zhou et al., 2022). These transformers may underperform linear models as shown in (Zeng et al., 2022). PatchTST (Nie
2

---

Published in Transactions on Machine Learning Research (04/2024)
et al., 2022) significantly improved transformers SOTA performance by considering sub-series tokens of the series. However, all these models cannot, handle properly irregularly sampled look-back windows.
Continuous methods for time series. Gaussian Processes (Rasmussen and Williams, 2006) have been a popular family of methods for modeling time series as continuous functions. They require choosing an appropriate kernel (Corani et al., 2021) and may suffer limitations in large dimensions settings. Neural Processes (NPs) (Garnelo al., 2018; Kim et al., 2019) paramcterize Gaussian processes through an encoder- decoder architecture leading to more computationally efficient implementations. NPs have been used to model simple signals for imputation and forecasting tasks, but struggle with more complex signals. Bilos et al. (2023) parameterizes a Gaussian Process through diffusion model, but the model has difficulty adapting to a large number of timestamps. Other approaches such as Brouwer et al. (2019) and Rubanova et al. (2019) model time series continuously with latent ordinary differential equations. mTAN (Shukla and Marlin, 2021), a transformer model, uses an attention mechanism to impute irregular time series. While these approaches have shown significant progress in continuous modeling for time series, we observed that their performances on imputation and forecasting tasks are inferior compared to the aforementioned discrete models (Table 1, Table 2).
Implicit neural representations. The recent development of implicit neural representations (LNRs) has led to impressive results in computer vision (Sitzmann et al., 2020; Tancik et al., 2020; Fathony et al., 2021; Mildenhall et al., 2021). INRs can represent data as a continuous function, which can be queried at any coordinate. While they have been applied in other fields such as physics (Yin et al., 2023) and meteorology (Huang and Hoefler, 2023), there has been limited research on INRs for time series analysis. Prior works (Fons et al., 2022; Jeong and Shin, 2022) focused on time series generation for data augmentation and on time series encoding for reconstruction but are limited by their fixed grid input requirement. Deep'Time (Woo et al., 2022) is the closest work to our contribution. DeepTTime learns set of basis INR functions from a training set of multiple time series and combines them using a Ridge regressor. This regressor allows it to adapt to new time series. It has been designed for forecasting only. The original version cammol handle imputation properly and was adapted to do so for our comparisons. In our experiments, we will demonstrate that TimeFlow siguificantly outperforms Deep'Time in imputation and also forecasting tasks when dealing with missing values in the look- back window. TimeFlow also shows a slight advantage over DeepTime in forecasting regularly sampled series.
3
The TimeFlow framework
3.1 Problem setting
We aim to develop a unified framework for time series imputation and forccasting that reduces dependency fixed sampling scheme for time series. We introduce the following notations for both tasks. During training, in the imputation setting, we have access to time series in an observed temporal grid denoted as Ti, which is a subset of the dense temporal support T. In the forecasting setting, we observe time series within a limited past time grid, referred to as the 'look-back window' and denoted as Tan (a subset of T), as well as a future grid, the 'horizon', denoted as Tout (also a subset of TT) At test time, in both cases, and given observed values in a temporal grid Te included in a possibly new temporal window T", our objective is infer the time series values within T T T if we infer values in the training temporal support (e.g. in the classical imputation scenario, see Section 4.1), T # # if we infer for new temporal support(e.g. in the forecasting setting, see Section 4.2).
3.2 Key components
Our framework is articulated around three key components:
(i) INR-based time-continuous functions: a discrete time series x = (Ity Ttg...., can be represented by an underlying time-contimouus function x: E R4 + 24 E Rd (in our experiments d d= 1). We want to approximate the ground-truth by employing implicit neural representations
3

---

Published in Transactions on Machine Learning Research (04/2024)
(INRs), which are neural networks capable of learning parameterized continuous function fo from discrete data by minimizing the reconstruction loss between observed data and network's outputs.
(ii Conditional INRs with modulations: An INR can represent only one function, whether it's an image or a time series. To effectively represent collection of time series (zl)) using INRs, we improve their encoding by incorporating per-sample modulations, which we denote as voti) These modulations condition the parameters 0 the INRs. We use the notation fa to refer to the conditioned INR with the modulations 2b())
Optimization-based encoding: the conditioning modulation parameters eاهي) are calculated as a (ii) function of codes 20) that represent the individual sample series. We acquire these codes 2(1) through a meta-learning optimization process using an auto-decoding strategy. Notably, auto-decoding has been found be more efficient for this purpose than set encoders (Kim et al., 2019).
In the following sections, we will elaborate each component of our method. Given that the choices made for each component and the methodology developed to enhance their synergy are essential aspects, we provide a discussion the various choices involved in Section 3.4.
INR-based time-continuous functions. We implement our INR with Fourier features and a feed- forward network (FFN) with ReLU activations, i.e. for time coordinate I E T, the output of the INR fe is given by ft( (t)( The Fourier Features 7()) are frequency embedding of the time co- ordinates used to capture high-frequencies (Tancik et al., 2020; Mildenhall et al., 2021). In our case, we chose 7() == (s)n, cos(rt), sin(2 cos(22 (()), with N the number of fixed frequencies. For an INR. with layers, the output computed as follows: (i) we get the frequency embedding do = 7(), (ii) we update the hidden states according to Φi = ReLU(( + bi) for 1,..., (iii) we project onto the output space fo(t) 0 + DL+1-
Conditional INRs with modulations. As indicated, sample conditioning of the INR is performed through modulations of its param- eters. In order to adapt rapidly the model to new samples, the conditioning should rely only on small number of the INR parameters. This is achieved by modifying only the biases of the INR through the introduction of ad- ditional bias term w((i) for each layer also known as shift modulation. To further limit gener- the versatility of the conditioning, ale the instance modulations gb) from com- pact codes z(5) through a linear hypernetwork with parameters w, i.c., s5)) = (=(1))) Consequently, the approximation of a time se- ries x00), denoted globally as (z(n), will depend on shared parameters and w that are common among all the INRs involved in modeling the series family and the code 200) specific to series æ(0), The output of the Itth layer of the modulated INR is given by Φi ReLUU(00r-11 + by w(("),, where wa()) network hw. This design enables gathering information across samples into the common parameters of the INR and hypernetwork, while the codes contain only specific information about their respective lime-series samples. The architecture is illustrated Figure 1.
Wi
hypernetwork. hy
Clode
Autodecaded
Wi 200 W/z
W.
modulations
Frequency
4 a(0)di- +, W92
embedding
02
01.
16 1E L)
fonstev ER.
4/2
INR
Figure 1: Overview of TimeFlow architecture. Forward pass to approximate the time series ,06), stands for the ReLC activation function.
Wiz and = (Wi),, are the parameters of the hyper-
Optimization-basee encoding. We condition the INR using the data from T., and learn the shared INR and hypernetwork parameters 0 and w using Ti for both imputation and forecasting, and Tout for forecasting
4

---

Published in Transactions on Machine Learning Research (04/2024)
only. We achieve the conditioning on Tin by optimizing the codes 200 through gradient descent. The joint optimization of the codes and common parameters is challenging. In TimeFlow, it is achieved through a meta-learning approach, adapted from Dupont et al. (2022) and Zintgraf et al. (2019). The objective is to learn shared parameters so that the code 20) can be adapted in just a few gradient steps for a new series x0)) For training, we perform parameler optimization at two levels: the inner-loop and the outer-loop. The inner-loop adapts the code z(0) to condition the network on the set T while the outer-loop updates the common parameters using T and also Tull for forecasting (sce Appendix H for more detailed intuition.). We present our training optimization in Algorithm 1. At each training epoch and for each batch of data B composed of time series z(3) sampled from the training set, the inner loop, before updating the common parameters in the outer loop using a. loss over the whole batch. We introduce a parameter À to weight the importance of the loss over Tour w.r.t. the loss over Tin for the outer-loop. In practice, when Tout exisls, i.e. for forecasting, we set. À = 1 and X = 0 otherwise. We use an MSE loss over the observations grid (T ot) == E - = We denote a and v) the learning rates of the inner-loop and outer-loop. Using K 3 steps for training and testing sufficient for our experiments thanks to the use of second-order meta-learning as explained in Section 3.4.
first update individually the codes 20ح in
Algorithm 1: TimeFlow Training
while no converyence do
Sample batch B of data (x())))eB:
Set codes to zero 2 + 0, Vj E B; // inner loop for encoding:
for j E B and step E {l,..., , K} do
26) - 200) aV ztn Luy (eenn x01);
// outer loop step:
10, w] - [0, w| 10 |8 Ejeol (So (an,, x00) + >L (Jonn out
3.3 TimeFlow inference
During the inference process, we aim infer the time series value for each timestamp in the dense grid 7*)
based on the partial observation grid T,r6)) C CTG)) We can encounter two scenarios: (i) One where we observe the same time window as during training (T*()) T() as in the imputation sctting Section 4.1.
(ii) One, where we are dealing with newly observed time window (T*6) j TÚ)), as in the forecasting setting in Section 4.2. At inference, the parameters 0 and are kept fixed to their final training values. We
optimize the individual parameters 200 basod on the newly observed grid T:(() using the K inner-steps of the meta-learning algorithm described in Algorithm 2. We are then in position to query Se (I) for any given timestamp t E T-06),
Algorithm 2: TimeFlow Inference with trained 0, w
For the j-th series (I(1),, set code to zero =*()) +- O; for step E (1,..., K} do
2-1 + (i) a 3-(3) 2-()) Cop (e
Query fa (e (t) for any E (i)
3.4 Discussion on implementation choices
As indicated before, adapting the components and enhancing their synergy for the tasks of imputation and forecasting is not trivial and requires careful choices. We conducted several ablation studies to provide a comprehensive examination of key implementation choices of our framework.
Our findings can be summarized as follows:
5

---

Published in Transactions on Machine Learning Research (04/2024)
Choice of INR: An FFN with Fourier Features outperformed other popular INRs for the tasks considered in this study. Unlike SIREN (Sitzmann et al., 2020), which does not explicitly incorporate frequencies but uses sine activation functions, the Fourier features network can more effectively capture a wider range of frequencics, especially at low sampling rates. This is crucial for accurately capturing high frequencies sparsely observed time series. Our experiments, detailed in Section B.2.1 and Table 4, demonstrate this superiority across various datasets.
Choice of encoding / meta-learning: TimeFloo with set encoder for learning the compact conditioning codes 2 in place of the auto-decoding strategy used here, proved much less effective on complex datasets. This is further elaborated in Section B.2.4 and Table 11. Additionally, replacing the 2nd-order meta-learning optimization for a 1st-order one, such as REPTILE (Nichol et al., 2018), led to unstable training, as shown Table 10.
Choice of modulations: Complexifying the modulation by introducing scaling paramcters in addition to shift parameters did not provide performance gains. Our experiments on the Electricily dataset, detailed in Section B.2.5 and Table 12, indicate that shift-only modulation is more efficient.
For TimeFlow, across all experiments, we used a code dimension of 128, an FFN with a depth of 5 and a width of 256, and 64 Fourier features. We used inner steps and a learning rate of 0.01 for the inner-loop, and a learning rate of 5 10- for the outer-loop. We performed comprehensive analysis to understand notably the influence of the dimension: latent code dimension of 128 was suitable for our tasks; this is supported by results in Section B.2.2 and Table and the influence of the number of inner steps: using 3 inner steps for training and inference struck a favorable balance between reconstruction capabilities and computational efficiency, as detailed in Section B.2.3.
4 Experiments
We conducted a comprchensive evaluation of our TimeFlow framework across three different tasks, comparing its performance to state-of-the-ard continuous and discrete baseline methods. In Section 4.1, we assess TimeFlow's capabilities to impute sparsely observed time series under various sampling rates. Section 4.2 focuses on long-term forecasting, where we evaluate TimeFlow over standard long-term forecasting horizons. In Section 4.3, we tackle a challenging task forecasting with incomplete look-back windows, thus combining the challenges of imputation and forecasting. The code for the experiments is available at this link.
Datasets. We tested our framework three extensive multivariate datasets where single phenomenon is measured at multiple locations over time, namely Electricity, Thaffic and Solar. The Electricity dataset comprises hourly electricity load curves of 321 customers in Portugal, spanning the years 2012 to 2014. The Truffic dataset is composed of hourly road occupancy rates from 862 locations in San Francisco during 2015 and 2016. Lastly, the Solar dataset contains measurements of solar power production from 137 photovoltaic plants Alabama, recorded 10-minute intervals in 2006. Additionally, we have created an hourly version, SolarH, for the sake of consistency in the forceasting section. These datascts exhibit diversity in various characteristics: They exhibit. diverse temporal frequencies, including daily and weekly seasonality observed in the Thaffic and Electricity datasets, while the Solar dataset possesses only daily frequency. There is individual variability across data samples and more pronounced trends in the Electricity datasct compared to the Traffic and Solar datasets.
4.1 Imputation
We consider the classical imputation setting where n time series are partially observed over a given time window. Using our approach, we predict for each time series the value at any timestamp in that time window based on partial observations.
Setting. For time series z(f), we denote the set of observed points as T.(1) and the ground truth set of points as TU), The observed time grids may be irregularly spaced and may differ across the different time
6

---

Published in Transactions on Machine Learning Research (04/2024)
series (T T Vji f2). The model is trained for each x00 following Algorithm 1. Then, we aim to infer for any unobserved E TV) the missing value (i) conditioned on Til) according to Algorithm 2. For this imputation task, the TimeFlow training and inference procedures are detailed in Section and illustrated in Figure 2. For comparison with the SOTA imputation baselines, we assume that the ground truth time grid is the same for each sample. The subsampling rate is define as the rate of observed values.
Training
20 8 and u are optimized according to Algorithm I on Lyo (5, ,7,)
Observed series x
x0i
Observed grid @
10
1) Only 200 optimized through steps gradient
2) Then, we can infer
Inference
descent on T to condition the network per time series x
(ay,) for 7
E T
x
Infer values on the dense grid T
Observed grid T
(i) During training, for each time series r(61, our observations (red dots )) are restricted to the sparscly sampled grid, denoted as T:()) (ii) During inference, our objective is to infer the values over the dense grids T(0, on the unobserved data points (such as the blue dots on the figure).
Figure Training and inference procedures of TimeFlow for imputation.
Baselines. We compare TimeFlow with various baselines, including discrete imputation methods, such CSDI (Tashiro et al., 2021), SATTS (Du et al., 2023), BRITS (Cao et al., 2018), and TIDER (Liu et al., 2023), and continuous ones, such as Neural Process (NP, Garnelo et al., 2018), mTAN (Shukla and Marlin, 2021), and Deep'Time with slight adjustmente (Woo et al., 2022) (details cf. Appendix D.3). See Appendix D.1.1 for the baseline training procedure and hyperparameter selection. For each dataset, we divide the series into five independent time windows (consisting of 2000 timestamps for Electricity and Traffic, and 10,000 timestamps for Solar), perform imputation on each time window and average the performance obtain robust results. We evaluate the quality of the models for different subsampling rates, from the easiest T 0.5 to the most difficult 0.05. All the scores presented in the experiments are reported as Mean Absolute Error (MAE).
We show in Table 1 that TimeFlow outperforms both discrete and continuous models across al- most all r's for the given datasets. The relative improvements of TimeFlow, as defined in Appendix C, over basclines are significant, ranging from 15% 50%. Especially for the lowest sampling rate t 0.05, Time- Flow out perforrss all discrete baselines, ratonntrating the advantages of continuous modeling. Additionally, it achieves lower imputation errors compared to continuous models in all but one cases. Qualitatively, we see on example scries in Figure 3 that our model shows significant imputation capabilities, with subsampling rate at = 0.1 on the Electricity dataset. It captures well different frequencies and amplitudes in a chal- lenging case (sample 35); although it underestimates the amplitude of some peaks. In a more challenging scenario (sample 25), where the series exhibit additional trend changes and frequency variations within the data, TimeFlow correctly imputes most timestamps, outperforming BRITS, which the best-performing method for the Electricily dataset.
Results.
Imputation on previously unseen time scries. In more practical scenarios, such as cases involving the installation of new sensors, we often encounter new time series originaling from the same underlying phenomenon. In such instances, it becomes crucial to make inferences for these previously unseen time series. Thanks to efficient adaptation in latent space, our model can casily be applicd these new time series (as shown in Appendix D.2, Table 18), contrasting with SOTA methods like SAITS and BRITS, which require full model retraining the whole set of time series.

---

Published in Transactions on Machine Learning Research (04/2024)
Table 1: Mean MAE imputation results on the missing grid only. Each time series is divided into time windows onto which imputation is performed, and the performances are averaged over the 5 windows. In the table, 7 stands for the subsampling rate, i.e. the proportion of observed points considered for each time window. Bold results are best, underlined results second best. TimeFlow improvement represents the overall percentage improvement achieved by TimeFlow compared to the specific method being considered.
Continuous methods
Discrete methoods
TimeFlow
DorpTime
TTAN
Neusal
Process
CSDI
SAITS
BRITS
TIDER
0.427
0.10
0.250
0.010
0.333 1 0.034
0.412 1 0.947
0.417 1: 0.067
0.398 1: 0.072
0,308 1: 0.011
0.267 1: 0.015
0.399
0.000
Electricity
0.225
0.008
0.133
0.017
0.089
0.261
0.008
0.011
0.391
0.30
0.212
0.007
0.240
0.014
0.335
0.015
0.300
0.022
0:277
0.058
0.236 + 0.008
0.221
+ 0.008
0.384
0.009
0.50
0.191
0.007
0.227
0.012
0,340
0.022
0.016 0.297
0.168
0.003
0.209 : 0.008
0.193 4: 0.008
0.386
0.009
0.05 0,095
0.015
0,190
0.020
0.241
0.102
0.115
0.015
0.374
0.033
0.142 4: 0.016
0.165 4: 0.014
0.291
0.009
0.10
0.083
0.015
0.159 + 0.013
0.251
0.081
0.014 0.114
0.375
0.038
0.124 + 0.018
0.132 + 0.015
0:276
0.010
Solar
0.30
0.061 0.012
0.135 + 0.014
0.338 = 0.06
0.108 1: 0.016
0.156 + 0.0002
0.100 + 0.015
0.098 + 0.012
0.266
0.010
0.013
0.013
0.262
0.16 0.283 0.016
0.246
0.010
0.406
0.074
0.318
0.014
0.337
0.046
0.203 + 0.007
0.261 + 0.010
0.363
0.00T
0.10 0.211 = 0.012
0.214 + D.007
0,319
0.025
0,288
0.018
0.288
0.017
0.237 : 0.006
0.215 2 0.009
0.362
0.006
Trafic
0,20 0,166
0.006
0,216
0.006
0,270
0.012
0.271
0.011
0.269
0.017
0.197 1 0.005
0.224
0.008
0.361
0.006
0.30
0.151
0.007
0.172 + 0.008
0.006 0.251
0.012 0.259
0.037 0.240
0.150
: 0.006
0.197
4: 0.007
0.355
0.006
0.139
0.100
0.008
0.060
TimeFlor low improvecenn:
21.14% %
30.53 %
31.61 St
36.12 %
20.33 %
18.90 %
53,10 %
Individual 35: TimeFlow MAE 0.316 BRITS MAE 0.488
2.5
0.0
0.00
0.02
0.04
0.06
0.08
0.10
Individual 25: TimeFlow MAE 0.404 BRITS MAE 0.737
0.00
0.02
0.04
0.06
0.08
0.10
Ground Truth
TimeFlow imputation
BRITS imputation
Observed points
Figure 3: Electricity dataset. TimeFlow imputation (blue line) and BRITS imputation (gray line) with 10% of known point (red points) on the eight first days of samples 35 (top) and 25 (bottom).
4.2 Forecasting
4.2.1
Forecasting for known time series
In this section, we are interested the conventional long-term forecasting scenario. It consists predicting the phenomenon in a specific future period, the horizon, based on the history of limited past period, the look-back window. The forecaster is trained on a set of n observed time series for a given time window (train period) and tested new distinct time windows.
Setting. For a given time series _((i), (()) denotes the look- back window and T-5) the horizon of H points. During training, at each epoch, we train Sa,na following Algorithm with randomly drawn pairs of look-back window and horizon (T U ((i)) jee within the observed train period. Then, for a distinct new time window T*6), given look-back window T,(() we forecast future values any E T=(6) the horizon interval, following Algorithm 2. We illustrate the training and inference of TimeFlow for the forecasting
8

---

Published in Transactions on Machine Learning Research (04/2024)
task in Figure 4. For further insight into the training window and inference periods, well as additional experiments conducted under different inference scenarios, see Appendix F.1.
Training
Draw 1
Draw k
Ú (f Draw :
vr E in
u
T
x pD =
Sommecesa (bl)
{r Draw :
out
Series
0 uniquely optimized Tioe
0 and w optimized on Tiu and our
Grid
Train period
Inference
Only ztdl optimized through steps gradient descent T
Then, infer (onateinn(() for any r e T
Series
Grid
Infer values T T(())
Train period
New time window T T
Figure 4: Training and inference procedure of TimeFlow for forecasting. (i) During training (top-figure), for each time series rQ01, we observe some look-back window/horizon drawing pairs in the trained period. TimeFlow is trained with Algorithm 1 to predict all observed timestamps in this drawing pairs while being conditioned by the observed look-back window. (ii) Once TimeFlow is optimized, the objective during inference (bottom-figure) is to infer the horizon over new time windows (blue dots o) while being conditionod by the newly observed look-back window (red dots )).
Baselines. To evaluate the quality of our model in long-term forecasting, we compare it to the discrete basclines PatchTST (Nie al., 2022), DLincar (Zeng et al., 2022), AutoFormer (Wu et al., 2021), and Informer (Zhou et al., 2021). We also include continuous baselines DeepTime and Neural Process (NP). See Appendix E.3.1 for the baseline training procedure and hyperparameter selection. In Table 2, we present the forecasting results for standard horizons in long-term forecasting: H E (96, 192, 336, 720}. The look-back window length is fixed to 512.
Table 2: Mcan MAE forecast results averaged over different time windows. Each time, the model is trained one time window and tested on the others (there are 2 windows for SolarHl and 5 for Electricily and Traffie). H stands for the horizon. Bold results are best, and underlined results are second best. TimeFlow improvement represents the overall percentage improvement achieved by TimcFlow compared to the specific method being considered.
Continuous methods
Discrete methods
H
TimeFlow
DeepTimm
Neural Process
Patch-TST
DLinear
AutoPormer
Informer
96
0.228 1: 0.028
0.244
0.026
0.382 1 0.045
0.221 + 0.023
0.241 1: 0.030
0.546 1 0.277
0.603 1. 0.255
Electricity
192
0.020
0.019
0.101
0.016
0.229
0.252
0.025
0.500
0.190
0.690
0.291
336
0.270
+ 0.031
0.284 + 0.034
0.434 + 0.076
0.251
0.027
0.288
0.038
0.523
+ 0.188
0.736
+ 0.271
720
0.316
0.055
0.359
0.051
0.607 1 0.150
0.297
0.039
0.365
0.059
0.631
1 0.237
0.746
1. 0.265
96
0.190 = 0.013
0.190
0.020
0.221 + 0.048
0.262
0.070
0.208
0.014
0.245 + 0.045
0.248 + 0.022
SolarH
192
0.202
0.020
0.204
0.028
0.244 0.048
0.253
0.051
0.217
0.022
0.333
0.107
0.270
0.031
336
0.209
= 0.017
0.199
1 0.026
0,200 + 0.006
0.259
= 0.071
0.217 = 0.026
0.3334
1: 0.079
(L328
+ 01,448
720
0.218
0.041
0.229
0.024
0.103 + 0.147
0. 2.267
0.0G1
0.249
0.034
0.351 + 0.055
0.337 - 0.037
0.032
0,283 0.027
0.033
0.059
0.372
0.078
Traflie
192
0.212
0.028
0.220
0.022
0.292 + 0.024
0.197
0.030
0.221
0.023
0.368 + 0.057
0.511 = 0.247
336
0.238
0.034
0.245
0.038
0.305 + 0.039
0.222
0.039
0.250
0.040
0.434 + 0.061
0.561 + 0.263
720
0.050
0.052
0.339
0.057 0.269
0.037 0.300
0.462
0.062
0.067
TlmeFlow improvement
3.7 X
29.06 %
3.23 %
6.92 %
42.00 %
48.57 %
9

---

Published in Transactions on Machine Learning Research (04/2024)
Results. The results in Table 2 show that approach ranks in the top two across all datasets and horizons and is the overall best continuous method. TimeFlow's performance is comparable to the current SOTA model PatchTST, with only 2% relative difference. Moreover, TimeFlow shows consistent results across the three datascts, whereas the other best discrete and contimuous basclines, i.c. PatchTST and DeepTime, performance drops lor some datasels. We also nole that, despite the greal performance of the SOTA PatchTST, other transformer-based basclines (discrete methods in Table 2) perform poorly. We provide a detailed insight on these results in Appendix E.1. Overall, although this evaluation setting favors discrete methods because the time series observed at evenly distributed time steps, TimeFlow consistently performs as well as PatchTST and outperforms all the other methods, whether discrete or continuous. It is the first time that continuous model has achieved the same level of performance as discrete methods within their specific selting.
4.2.2 Forecasting on previously unseen time series.
This section discusses how TimeF'low adapts to unseen time series, which is critical in forecasting, Indeed, in many real-world applications, forecasters are trained on limited subset of available samples and applied to wider range of samples during inference. Informer, AutoFormer, or DLinear original architectures directly model the relationships between time series (chamnel-eependence)) limiting their adaptability to new samples. contrast, TimeFlow takes a different approach by considering the observed series at different locations as distinct samples, similar to PatchTST, Neural Process, and DeepTTmme This independence allows TimeFlow to effectively generalize to previously unseen time series of the same phenomenon.
Setting. In this setting, we propose to evaluate how TimeFlow performs previously unseen time series, We compare it to the best forecaster, PStchTST. We train TineFlow and PatchTST on 50 % of the samples and consider the remaining 50 as the new time series. The training procedure is the same as described in Figure 4. In Figure 5, we present the results of TimeFlow and PatchTST for both known and new samples (for periods outside the training window).
Results. The results in Figure highlight. two key observations. First, both approaches show robust adaptability new samples, evidenced by the minimal difference in mean absolute error between known and new samples at inference. Second, TimeFlow and PStchTST exhibit comparable performance in this context, with negligible differences across horizons and datasets.
0.40
0.25
0.35
0.30
6.0
0.5
MAC error
0.20
0.30
0.15
6.23
0.00
0.35
132
326
720
96
102
359
720
302
355
720
Horizon
TimeFlow Known samples
TimeFlow New samples PatchTST New samples
PatchTST Known samples
Figure 5: Mean MAE forecasting task results over different horizons in the context of generalization to new time serios. Comparison of TimeFlow and PatchTST performances on the Electricity, Troffic and SolarH datasets.
10

---

Published in Transactions on Machine Learning Research (04/2024)
4.3 Challenging task: Forecast while imputing incomplete look-back windows
In real-world scenarios, it is common to encounter missing or irregularly sampled series when making pre- dictions on new time windows (Cinar et al., 2018; Tang et al., 2020). Contimuous methods can handle these cases, as they are designed to accommodate irregular sampling within the look-back window. In this sec- tion, we formulate a task to simulate these real-world scenarios. It's worth noting that this task often encountered in practice but is rarely considered in the DL literature.
Setting and baselines. This scenario is similar to the forecast setting in Section 4.2 and illustrated in Figure 4. The difference is that during inference, the look-back window is subsampled at a rate T smaller than the one used for the training phase. This simulates situation with missing observations in the look back window. Consequently, two distinct tasks emerge during the inference phasc: imputing missing points within the sparsely observed look-back window, and forecasting over the horizon with this degraded context In Table 3, we compare to the two other continuous baselines, Deep'Time and NP on Electricity and Traffic for different T's and horizons.
Table 3: MAE results for forecasting with missing values in the look-back window. stands for the percentage of observed values in the look-back window. Best results are in bold.
TimeFlow
DeepTime
Nenral Process
H
Imposation
Forecass,
Inpulation
Foreccatt
Impulation
Forecass
0.5
0.151
0.003
0.239
0.013
0.200
0.001
0.270 1: 0.019
0.460
0.088
0,480 0.078
16
0.2
0,208 0.006
0.260
0.015
0.249 + 0.006
0,296 + 0.023
0.644
0.079
0.630 + 0.095
Electricity
0.272
0.006
0.295
0.016
0.284 0.007
0.324
0.026
0.740
0.083
0.106
0.5
0.149
1 0.004
0,235
1 0.011
0.201 1 0.001
0.265 1 0.018
0.461 1 0.045
0.498 1 0.070
0,209
0.006
0.257
0.007
0.290
0.023
0.075
0.101
0.1
0,274
0.010
0.289
+ 0.016
0.282 + 0.007
0.315
+ 0.025
0.461
+ 0.045
0.724 + 0.090
0.5
0,180
0.016
0.219
0.026
0.272
0.028
0.243
0.000
0.436
0.025
0.444 0.047
96
0.2
0,239
0.019
0.243 1 0.027
0.335 4: 0,028
0.293 1: 0.027
0.506
4: 0.049
0.597 1: 0.075
Traffle
0.1
0.020 0.512
0.290
0.027
0.385 1 0.025
0.311
0.027
0.731
0.102
0.731
0.182
0.176
0.014
0.217
0.017
0.027
0.021
0.042
0.043
192
0,2
0.233
1: 0.017
0,236
4 0.021
0.286 4: 0,027
0.276 4: 0.020
0.685
4: 0.109
0.678 1: 0.108
0.304
0.019
0,277
0.021
0.025
0.021
0.178
0.174
Tinel low Imprrveeeent
18.97 %
11.87 9
61.88 %
58.11 %
Results. In Table 3, the results show that TimeFlow consistently outperforms other methods in imputation and forecasting for every scenarios. When comparing with the complete look-back windows observations scenario from Table 2, one observes that at a 0.5 sampling rate, TimeFlow presents only a slight reduction in performance, whereas other baseline methods experience more significant drops. For instance, when we compare forecast results between a complete window and a T = 0.5 subsampled window for Electricily with a forecasting horizon of H = 96, TimeFlow's 's error increases by a mere 4.6% (from 0.228 to 0.239). In contrast, DeepTime's error grows by over 10% (from 0.244 to 0.270), and NP experiences a rise of around 25% (from 0.392 to 0.486). For lower sampling rates, TimeFlow still delivers correct predictions. Qualitatively, we see on the series example in Figure 6 that despite observing only 10% of the look-back window, the model can correctly infer both the complete look-back window and the horizon. Both quantitative and qualitative results show the robustness and efficiency of TimcFlow on this particularly challenging setting.
Ground truth
Imputation
F Forecast points
Observed points
Figure 6: Traffic dataset, sample 95. Tn this figure, TimeFlow simultaneously imputes and forecasts horizon 96 with 10% partially observed look- back window of length 512.
11

---

Published in Transactions on Machine Learning Research (04/2024)
5 Limitations
While TomeFlow shows promising performance across various tasks and settings, it is important to recognize some limitations. First, due to its auto-decoding process, TimeFlow tends to be significantly slower at inference time compared to other baselines by one to two orders of magnitude (Table 23). In addition, although TomeFlow effectively handles sets of homogeneous time series, additional mechanisms are required to handle heterogencous time series with different frequencies effoctively. The per-context shift modulation mechanism does not allow TimeFlow to fit time series with drastically different structures. This also explains why TimeFlow is particularly well adapted to regular frequency patterns.In the experiment section, all considered datasets exhibit pronounced periodicities. Finally, it should be noted that effective training of TimeFlow requires a relatively large number of samples (typically 2 100) to allow the model to accurately distinguish between individual patterns and shared information.
6 Conclusion
We have introduced a unified framework for continuous time series modeling leveraging conditional INR and meta-learning. Our experiments have demonstrated superior performance compared to other continuous methods, and better or comparable results to SOTA discrete methods. One of the standout features of our framework its inherent contimuity and the ability to modulate the INR parameters. This unique flexibility lets TimeFlow effectively tackle wide array of challenges, including forecasting in the presence of missing values, accommodating irregular time steps, and extending the trained model's applicability to previously unscen time serics and new time windows. Our empirical results have shown TimeFlow's effectiveness in handling homogeneous multivariate time series. As a logical next step, exlending TimeFlow's capabilities to address heterogeneous multivariate phenomena represents a promising direction for future research.
References
M. Bilos, K. Rasul, A. Schneider, Y. Nevmyvaka, and S. Günnemann. Modeling temporal data continuous functions with stochastic process diffusion. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML, volume 202 of Proceedings of Machine Learning Research, pages
PMLR, 2023.
F. D. Brouwer, J. Simm, A. Arany, and Y. Moreau. cru--ddebaaeess continuous modeling of sporadically- observed time series. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 7379 7390, 2019.
W. Cao, D. Wang, J. Li, H. Zhou, Y. Li, and L. Li. Brits: bidirectional recurrent imputation for time series. In Proceedings of the 32nd International Conference on Neural Information. Processing Systems, pages 6776 6786, 2018
H. Chen, S. Grant-Muller L. Mussone, and F. Montgomery. A study of hybrid neural network approaches and the effects of missing data on traffic forecasting. Newral Computing & Applications, 10:277 286, 2001.
T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd aem sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
Y. G. Cinar, H. Mirisaee, P. Goswami, É. Gaussier, and A. Ait-Bachir. Period-aware content attention rnns for time series forecasting with missing values. Neurocomputing, 312:177-186, 2018
J. S. Clark and O. N. Bjørnstad. Population time series: process variability, observation errors, missing values, lags, and hidden states. Ecology, 85(1)):3140 01500, 2004.
G. Corani, A. Benavoli, and M. Zaffalon. Time series forecasting with gaussian processes needs priors. In Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part IV 21, pages 103 117. Springer, 2021.
12

---

Published in Transactions on Machine Learning Research (04/2024)
W. Du, D. Côté, and Y. Liu. Saits: Self-attention-baseed imputation for time series. Expert Systems with Applications, 219:119619, 19619, 2023.
E. Dupont, H. Kim, S. M. A. Eslami, D. J. Rezende, and D. Rosenbaum. From data to functa: Your data point is a function and you can treat like one. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5694 5725. PMLR, 2022.
Sahu, D. Willmott, and Kolter. Multiplicative filter networks. In International Conference on Learning Representations, 2021.
Fathony,
C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126 1135. PMLR, 2017.
E. Fons, A. Sztrajman, Y. El-Laham, A. losifidis, and S. Vyetrenko. Hypertime: Implicit neural representa- tion for time scries. CoRR, abs/ abs/2208.05886, 2022.
V. Fortuin, D. Baranchuk, G. Rätsch, and S. Mandt. Gp-vac: Deep probabilistic time series imputation. International conference on artificial intelligence and statistics, pages 1651 1661. PMLR, 2020.
M. Garnelo, D. Rosenbaum, C. Maddison, T. Ramalho, D. Saxton, M. Shanahan, Y. W. Teh, D. J. Rezende, and S. M. A. Eslami. Conditional neural processes. In Proceedings of the 35th International Conference on Machine Learning, ICML, volume 80, pages 1690-1699, PMLR, 2018.
T. J. Hastie. Generalized additive models. In Statistical models in S, pages 249 307. Routledge, 2017.
L. Huang and T. Hoefter. Compressing multidimensional weather and climate data into neural networks. In International Conference on Learning Representations, ICLR, 2023.
R. J. Hyndman and G. Athanasopoulos. Forecasting: principles and practice. OTexts, 2018.
K. Jeong and Y. Shin. Time-series anomaly detection with implicit neural representation. CoRR, abs/2201.111500 2022.
T. Kim, W. Ko, and J. Kim. Analysis and impact evaluation of missing data imputation in day-ahead pv generation forecasting, Applied Sciences, 9(1):204, 2019.
S. Liu, H. Yu, C. Liao, J. Li, Lin, A. X. Liu, and S. Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
S. Lin, X. Li, G. Cong, Y. Chen, and Y. Jiang. Multivariate time-series imputation with disentangled temporal representations. In The Eleventh International Conference on Learning Representations, ICLR, 2023.
Y. Liu, R. Yu, S. Zheng, E. Zhan, and Y. Yue. Naomi: Non-autoregressive multiresolution sequence impu- tation. Advances in neural information processing systems, 32, 2019.
Y. Luo, X. Cai, Y. Zhang, J. Xu, al. Multivariate time series imputation with generative adversarial networks. Advances in neural information processing systems, 31, 2018.
Y. Luo, Y. Zhang, X. Cai, and X. Yuan. F2gan: End-to-end generative adversarial network for multivariate time series imputation. In Proceedings of the 28th international joint conference on artificial intelligence, pages 3094-3100. AAAI Press, 2019.
B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
A. Nichol, J. Achiam, and J. Schulman. On first-order meta-learning algorithms. arXiv:1833.020 1809.02999, 2018.
arXiv preprint
13

---

Published in Transactions on Machine Learning Research (04/2024)
Y. Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam. A time scries is worth 64 words: Long-term forecasting with transformers. CoRR, abs/2211.14730, 2022.
C. E. Rasmussen and C. K. I. Williams. Gaussian processes for machine learning. Adaptive computation and machine learning. MIT Press, 2006.
Y. Rubanova, R. T. Q. Chen, and D. Duvenaud. Latent odes for irregularly-sampled time series. CoRR, abs/1907.000 10001997.03907, 2019.
M. Schulz and K. Stattegger. Spectrum: Spectral analysis of unevenly spaced palcoclimatic time scries. Computers & Geosciences, 28(9):929-99) 945, 1997.
S. N. Shukla and B. M. Marlin. Multi-time attention networks for irregularly sampled time series. In 9th International Conference on Leurning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
V. Sitzmann, J. N. P. Martel, A. W. Bergman, D. Lindell, and G. Wetzstein. Implicit noural repre- sentations with periodic activation functions. In Advances in Neural Informalion Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NewrIPS 2020, December 6- 6-12, 2020, virtual, 2020.
M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
X. Tang, H. Yao, Y. Sun, C. C. Aggarwal, P. Mitra, and S. Wang. Joint modeling of local and global temporal dynamics for multivariate time series forecasting with missing values. In The Chirby-Fourthe AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Secone Innovative Applications of Artificial Intelligence Conference, 1AA1 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI, pages 5956-5963. AAAI Press, 2020.
Y. Tashiro, J. Song, Y. Song, and S. Ermon. Csdi: Conditional based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34:24804 24816, 2021.
S. J. Taylor and B. Letham. Forecasting at scale. The American Statistician, 72(1):37-45 -5.8774-,111 2018.
G. Woo, C. Liu, D. Sahoo, A. Kumar, and S. C. H. Hoi. Deeptime: Deep time-index meta-learning for non-stationary time-series forecasting. CoRR, abs/2207.06046. 2022.
H. Wu, J. Xu, J. Wang, and M. Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 94: Annual Conference on Newral Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pages 22419 22430, 2021.
Y. Yin, M. Kirchmeyer, J.-Y. Franceschi, A. Rakotomamonjy, and P. Gallinari. Contimuous pdc dynamics forecasting with implicit neural representations. In International Conference on Learning Represenlations, ICLR, 2023.
A. Zeng, M. Chen, L. Zhang, and Q, Xu. Are transformers effective for time series forecasting? CoRR, abs/2205. 13504, 2022.
H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, pages 11106-11115, 2021.
T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, and R. Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In K. Chaudhuri, Jegelka, Song, C. Szepesvári, G. Niu, and S. Sabato, oditors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Reseorch, pages 77268-27286. PMI.R, 2022.
14

---

Published in Transactions on Machine Learning Research (04/2024)
L. Zintgraf, K. Shiarli, V. Kurin, K. Hofmann, and S. Whiteson. Fast context adaptation via meta-learning.
In International Conference on Machine Learning, pages 7693-7702. PMLR, 2019.
15

---

Published in Transactions on Machine Learning Research (04/2024)
A Reproductiblity statement
Our work entirely reproducible, and all the references to the information in order to reproduce are in this section.
Code.
The code for all our experiments is available this link.
Data. A subset of the processed data is available with the code at this link. The dataset description, processing and normalization are presented in Appendix C.
Model. The model and the training details are presented in Section and the hyperparameter selection is available in Appendix B.1.
GPU. We used NVIDIA TITAN RTX 24Go single GPU to conduct all the experiments for our method, which is coded in PyTorch (Python 3.9.2).
B Architecture details and ablation studies
B.1 Architecture details
For all imputation and forecasting experiments we choose the following hyperparameters:
z dimension: 128
Number of layers: 5
Hidden layers dimension: 256
(t) e 22x64
2 code learning rate (a in Algorithm 1): 10-2
Hypernetwork and INR learning rate: 5 x 10-4
Number of steps in inner loop: K 3
Number of epochs: 4 10a
Batch size: 64
It is worth noting that the hyperparameters mentioned above remain consistent, across all experiments ducted in the paper. We chose to maintain fixed set of hyperparameters for our model, while other impulation and forecasting approaches commonly fine-tune hyperparanneeers based validation dataset. The obtained results exhibit high robustness across various settings, suggesting that the selected hyperpa- ramoters are already effective achicving reliable outcomes.
B.2
Ablation studies
B.2.1 Fourier features vs SIREN on imputation task
Baseline The SIREN network differs from the Fourier features network because it does not explicitly incorporate frequencies input. Instead, it is multi-layer perceptron network that utilizes sine activation functions. An adjustable parameter, denoted wp, is multiplied with the input matrices of the preceding layers lo capture broader range of frequencies. For this comparison, we adopt the same hyperparameters described in Appendix B.1, selecting wp = 30 to align with Sitzmann et al. (2020). Furthermore, we set the learning rate of both the hypernetwork and the INR 4, compare the imputation results obtained by the Fourier features network and the SIREN network, specifically focusing the first time window from the Electricity, Traffic and Solar datasets.
10-5 enhance training stability. In Table
16

---

Published in Transactions on Machine Learning Research (04/2024)
Table 4: MAE imputation errors on the first time window of each dataset. Best results are bold.
TimeFlow
TimeFlow SIREN
0.05
0.323
0.466
Electricity
0:20
0.224
0.242
0.30
0.211
0.50
0.194
0,200
0.05
0.105
0.114
0.10
0.083
0.094
Solar
0:20
0.065
0.079
0.30
0.061
0.0122
0.50
0.056
0.066
0.05
0.292
0.333
0.10
0.220
0.252
Traffcc
0.200
0.168
0,191
0.30
0.152
0.163
0.50
0.141
0.154
Results According to the results presented in Table 4, the Fourier features network outperforms the SIREN network the imputation task on these datasets. Notably, the performance gap between the two network architectures are more pronounced at low sampling rates. This disparity can be attributed the SIREN network's difficulty in accurately capturing high frequencies when the time scries is sparscly observed. We hypothesize that the MLP with ReLU activations correctly learns the different frequencies of time series with multi-temporal patterns by switching on or off the Fourier embedding frequencies.
B.2.2 Influence of the latent code dimension
The dimension of the latent code is a crucial parameter in our architecture. If it is too small, underfits the time series. Consequently, this adversely affects the performance of both the imputation and forecasting tasks. On the other hand, if the dimension of is too large, it can lead to overfitting, hindering the model's ability to generalize to new data points.
Baselines To investigate the impact of x dimensionality on the performance of TimeFlow, we conducted experiments on the three considered datasets, specifically focusing on the forecasting task. We varied the sizes of 2 within (32, 64, 128, 256). The other hyperparameters are set presented in Appendix B.1. The obtained results for each dimension are summarized in Table 5.
Table 5: MAE error for different 2 dimension.
H
32
64
128
256
0.018
0.215
Electricity
192
0.243 + 0.720
0.230
0.018
0,230
0.026
0:233
+ 0.017
0.029
0.031
0.243
0.032
720
0.293 + 0.0277
0.284
0.028
0.303
0.041
0.283 + 0.029
0.182 + 0.009
0.181
0.012
0.179
0.003
0:225 + 0.047
SolarH
192
0.195 4 0.014
0.196
0.016
0.193
0.015
0.197 : 0.029
336
0.181 + 0.011
0.182
0.011
0.189
0.013
0.153 + 0.012
720
0.201 + 0.027
0.199
0.025
0,209
0.029
0:200 : 0.030
0..223 + 0.024
0.215
0.028
0.215
0.087
0.210 1 0.033
Traffic
192
0.214 1. 0.018
0,217
0.025
0.206
0.023
0,203 1 0.024
336
0.2238 +: 0.029
0.281
0.029
0.226
0.030
0:229 + 0.029
720
0.272
0.040
0.269
0.035
0,259
0.038
0.262 0.010
Results
The results presented in Table suggest that a 2 dimension of 128 is reasonable compromise but only optimal for some settings. Moreover, even though the choice of z dimension seems important, it doesn't critically impact the MAE error for the forecasting task.
17

---

Published in Transactions on Machine Learning Research (04/2024)
B.2.3 Influence of the number of gradient steps
As can be seen in Table 6, using three gradient steps at inference yield an inference time of less than 0.2 seconds. The latter can still be reduced by doing only one slep at the cost of an increase in the forecasting error, As observed in Table 6, increasing the number of gradient steps above 3 steps during inference does not improve forecasting performance.
Table 6: Inference time (in seconds) and MAE error on the forecasting task on the Electricity dataset for a horizon of length 720, a look-back window of length 512, and a varying number of adaptation gradient steps. The statistics are computed over 10 runs using an NVIDIA TITAN RTX GPU.
Gradient despent atege
10
50
500
5000
Inferennce Line (x)
0.003
0.127
3.517
MAE
0,351 A 0.038 0.303 0.041
0.300 1 0.040
0.299 A: 0.039
0.302 = 0.038
0,308 +: 0.037
Table 7: MAE error on the forecasting task using inner-step during training and a varying number of adaptation gradient steps at inference. Best results are in bold and / symbol mcans that the MAE score is very high (2 1).
96
0.244 + 0.017
0,246 + 0.017
0.261 - 0.016
Electricity
0.253
0.024
0.253
0.022
0.020
0.019
336
0.267
0.032
0,268
0.000
0.277 0.028
0.281 + 0.027
720
0.302
0.030
0,306
0.029
0.310
0.028
0.301 : 0.029
0.192 + 0.023
0.623
0.307
SolarH
0.175
0.006
0.068
336
0.192
0.016
0.471
0.029
720
0.216
0.034
0.165 4: 0.063
0.550 + 0.187
0.215
0.029
0.329 + 0.089
Tralic
192
0.208
0.019
0,310 0.033
0.312
0.032
336
0.237
0.028
0.307
0.038
720
0.263
0.038
0.320 1: 0.040
Table 8: MAE error on the forecasting task using 3 inner-steps during training and varying number of adaptation gradient steps at inference. Best results are in bold.
96
0.250 + 0.020
0.222
+ 0.018
0.222 :: 0.017
0.228 :: 0.018
Electricity
0.269
0.020
0:232
336
0,273
+ 0.033
0.262
+ 0.031
0:264 :: 0.032
0.268 +: 0.032
0.351
0.038
0.303
0.299
96
0.487 : 0.196
0.179 :: 0.003
0.181 : 0.003
0.188 1 0.003
SolarIl
0.411
0.088
0.195
336
0.135
+ 0.153
0.189
: 0.013
0,203
: 0.006
0.223 1 0.012
0.173
0.209
0.029
0.006
96
0.320 1: 0.038
0.215
+: 0.037
0.219 1: 0.043
0.228 1: 0.046
Traffle
0.023
0.206
0.023
0:209
0.027
336
0.345 4: 0.038
0.226
+: 0.030
0.228
1: 0.081
0.233 1: 0.082
0.321 0.034
0.038
0.260
0.038
0.039
Results We conduct more extensive experiments in Table 7, Table 8, Table 9 to quantify the MAE score variation according to different number of gradient steps during training and inference. The tables show that using the same number of steps in training and inference leads to better results. This is expected since using different gradient steps for training and inference makes the inference model slightly different from the training model. In addition, using 3 gradient steps instcad of clearly improves the performancos, but using 10 instead of does not. Indeed, it usually leads to overall better results for longer horizon, but the gain is not clear for smaller horizons. Hence using 3 gradient steps is suitable choice.
B.2.4 TimeFlow variants with other meta- techniques
Baselines Before converging the current architecture and optimization of TimeFlow, we explored dif- ferent. options to condition the INR with the observations. The first. one was inspired by the neural process
18

---

Published in Transactions on Machine Learning Research (04/2024)
Table 9: MAE error on the forecasting task using 10 inner-steps during training and a varying number of adaptation gradient steps at inference. Best results are in bold.
II
30
96
0.381
0.030
0.249
0.024
0.236
0.024
0.238 4: 0.024
Eloctricity
192
0.448
+: 0.045
0.273
0.019
0.244
# 0.014
0.244 # 0.013
336
0.514
0.053
0.253
0.033
0.241
0.025
0,242 4: 0.024
720
0.647
+ 0.068
0.400
0.051
0.286
+ 0.023
0.287 # 0.021
96
0.605
0.029
0.350
0.018
0.188
0.012
0,199 +: 0.015
SolecH
192
0.382
D.0F2
0.250
0.012
0.202
0.034
0.204 + 0.035
336
0.745
0.105
0.431
0.221
0.201
0.033
0,208 0.032
720
0.745
D.082
0.477
0.039
0.205
0.030
0.205
0.029
96
0.450
0.023
0.273
0.026
0.225
0.028
0.230
0.034
Tealic
192
0.506
+: 0.028
0,318
0.021
0.283
+ 0.022
0.236 # 0.026
336
0.021
0.247
0.028
0.240 0.031
720
0.511 + 0.035
0. 32323 0.022
0.266 + 0.027
0.272 + 0.024
architecture, which uses a set encoder to transform a set of observations (fi, )izz into a latent code 2 by applying a pooling layer after a feed forward network. We observed that this encoder in combination with the modulated fourier features network was able to achieve relatively good results on the forecasting task but suffered of underfitting on more complex datasets such as Electricity.
This led us to consider aulo-decoding methods instead, i.e. encoder-less, architectures for conditioning the weights of the coordinate-based network. We trained TimeFlow with the REPTILE algorithm (Nichol et al., 2018), which is a first-order meta-learning technique that adapts the code in a few steps of gradient descent. In contrast with a second-order method, we observed that REPTILE was less costly to train but struggled to escape sub optimal minima, which led to unstable training and underfitting.
From an implementation point of view, the only difference between second order and first order, is that in the latter the code is detached from the computation graph before taking the outer-loop parameter update. z(0, w), which means
When the code is not detached, it remains a function of the common parameters that the computation graph for the outer-loop also includes the inner-loop updates to the codes. Therefore the outer-loop gradient update involves a gradient through a gradient and requires an additional backward pass through the INR. to compute the Hessian. Please refer to Finn et al. (2017) for more technical details.
Comparison of second-order and first-order (REPTILE) meta learning for TimeFlow on the im- putation task. Mean MAE results on the missing grid over five different time windows. 7 stands for the subsampling rate. Bold results are best.
Table 10:
TimeFlow
TimeFlow REPTILE
0.05 0.324
0.013
0.363
0.062
0.10
0.010
0.343
Flectricity
0.20
0.225
0.008
0.312
0.043
0.30
0.212
0.007
0,508 0.085
0.50 0.194
0.007
01305
0.046
(UB6 0.095 0.015
f1125 =: 01022
0.10 0.083 0.015
0.123 0.032
Solor
[130 0.061 1: 0.012
81100 1: 0.027
0.50 0.054 0.013
0.102 0.421
0.05 0.283
0.304 0.025
0.10 0.211 0.012
0.264 1: 0.009
Trafic
(1.0) 0.168 1: 0.006
[12222 0.019
(1300)
[12118
0.50 0.139 0.007
0.216 0.017
Results In Table 10, we show the performance of first-order TimeFlow on the imputation task. In low sampling regimes the difference with TimeFlow is less perceplive, but its performance plateaus when the number of points increases. This is not surprising. Indeed, as though the task is actually simpler when 7 increases, the optimization is made more difficult with the increasod number of obscrvations. We provide the performance of TimeFlow with a set encoder on the Forecasting task in Table 11. We observed that this version failed to generalize well for complex datasets.
19

---

Published in Transactions on Machine Learning Research (04/2024)
Table 11: Comparison of optimization- based and set-encoder- based meta learning for TimeFlow on the forecasting task. Mean MAE forecast results over different time windows. H stands for the horizon. Bold results are best.
H
TineFlow
TameFlow set encoddr
96
0.228 1: 0.026
0.362 1: 0.032
192
0.238
1 0.020
0.360 4: 0.028
Electricity
336
0.270
1: 0.031
0.382 + 0.038
720
0.316
0.055
0.431 0:059
96
0.190 + 0.013
0:251 + 0.071
192
0.202
0.020
0.239 0.058
SolarH H
336
0.209
+ 0.017
0:235 + 0.040
720
0.218
0.048
0.231 0.002
96
0.217 1 0.036
0.276 + 0.031
192
0.212
0,281
Trafic
336
0.258
+ 0.084
0.297 +: 0.012
720
0.279
0.333
B.2.5 Influence of the modulation
In TimeFlow, we apply shift modulations to the parameters of the INR, i.e. for each layer we only modify the biases of the network with an extra bias term of We generate these bias terms with a linear hypernetwork that maps the code 2 to the modulations. The output of the 1-th layer of the modulated INR is thus given by 61+1 = ReLUU00rri + by + v,()), where w/()) Wiz(s) and (Wi) are parameters of the hypernetwork. However, another common modulation is the combination of the scale and shift modulation, which leads to the output of the 1-th layer of the modulated INR being given by d ReL(((())20)) (iod--+b0 +v)))))))) where vt W)2 and (Wi)f., and (8i) are parameters of the hypernetwork and o is the Hadamard product.
In Table 12, we conduct additional experiments on the Electricity dataset in the forecasting setting with different time horizons. In these experiments, we compare two sccnarios: one where the INR is modulatod only by a shift factor and the other where the INR is modulated by both shift and scale factor. We kept the architecture and hyperparameters consistent with those described in Appendix B.1. The experiments shown in Table 12 indicate that the INR is longer to train with shift and scale modulations due to the increased number of parameters involved. Furthermore, we observe that the shift and scale modulated INR performed similarly or even worse than the INR with only shift modulation. These two drawbacks, namely an increased computational time and similar or worse performances, motivate modulating the INR only by a shift factor.
Ablation on modulations for the forecasting task on Electricity dataset for different horizons. Models are trained on given time window and tested four new time windows. Models are trained on single NVIDIA TITAN RTX GPU.
Table 12:
96
192
336
720
Training
Training
MAE
Training
Training time
Shift 0.233 # 0.014
2h300
0.245 + 0.016
2h31
0.264 #: 010020
2h33
0.303
0.041
2h46
Shift and scaaee 0.267 = 0.019
3L,29
0. 2623334 0.014
3132
0.208 :: 0.025
3lv15
0.308
0.087
4h14
B.2.6 Discussion on other hyperparameters
While the dimension of = is indeed a crucial hyperparameter, it is important to nole that other hyperpa- rameters also play a significant role in the performance of the INR. For example, the number of layers in the FFN directly affects the ability of the model fit the time scries. In our experiments, we have observed that using five or more layers yields good performance, and including additional layers can lead slight improvements in the generalivation settings.
20

---

Published in Transactions on Machine Learning Research (04/2024)
Similarly, the number of frequencies used in the frequency embedding is another important hyperparameter. Using too few frequencies can limit the network's 's ability to capture patterns, while using too many frequencies can hinder its ability to generalize accurately.
The choice of learning rate is critical for achieving stable convergence during training. Therefore, in practice, we use low learning rate combined with a cosine annealing scheduler ensure stable and effective training.
C Datasets, scores and normalization
For the complete datasets, Electricity dataset available here, Traffic dataset here and Solar data set here. Table 13 provides a concise overview of the main information about the datasets used for forecasting and imputation lasks.
Table 13: Summary of datasets information
Dalasel mame Number sumples Number lime steps Sumpling frequency
Location
Years
Electricity
321
26 304
hourly
Portugal
2012 2014
Traffle
hourly
Francisro bay 2016
Solar
137
52 560
10 minutes
Alahama
2006
SolorH
700
bourly
Alshama
How TimeFlow relative improvement score is computed? In many paper tables, the TimeFlow improvement score appears in the last row of the table. Its purpose to quantily the average marginal gain of TimeFlow over the method under consideration. It computed as follows:
st(baseline) - sy (Timeeloo)
TimeFlow improvement I.
st (baseline)
s stands for the Mean Absolute Error score of the considered method against the ground truth line I (for instance in Table 1, $1 (TimeFlow)) 0.324, 82 (Tome)low)) 0.250 etc).
stands for the number of line the table (for instance 15 in Table 1).
z-normalization To preprocess each dataset, we apply the widely used z-normalization technique per- (z) m2) sample j on the entire series: Tnorm
std(2(2))
D Imputation experiments
D.1
Baselines details
D.1.1 Baselines training and hyperparameters
The baselines underwent meticulous training and extensive testing, involving thorough exploration of hyper- parameters. We used SAITS repository (code) for BRITS and SAITS. The adopted setting results from a hyperparameters search, which yields marginally superior results for both methods compared the default settings. The marginal difference in scores underscores the robustness of BRITS and SAITS. In addition, mTAN and TIDER did not perform optimally with the recommended configurations, requiring extensive search. Details of the parameters explored are provided in Table 14 and Table 15. While the hyperparameter scarch led to performance improvements, overall results remained sub-optimal. For CSDI, the recommended settings proved inadequate for the considered datasets, prompting a comprehensive search. Among various parameters, the number diffusion steps emerged as crucial, significantly enhancing performance, partic- ularly higher draw ratios. However, superior performance was attained with more diffusion steps, albeit increased computational cost. The chosen parameters are detailed in Table 16. In addition, the orig- inal DeepTime implementation did not perform well imputation. Hence, adapted the Deep'Time
21

---

Published in Transactions on Machine Learning Research (04/2024)
training procedure (see Appendix D.3). Lastly, the vanilla Neural Process baseline underperformed, customized its architecture to conduct fair comparison with TimeFlow. We used the INR and hypernet- work from TimeFlow to align the Neural Process with our temporal frequency bias and shift modulation tochnique.
Table 14: mTAN hyperparameter search.
Dimeersio
linesr schecluler
NumRelPointe
k-iwae
Target malio
50
1
1 10-
32
5
0.2
0.95
0.5
0.001
128
Table 15: TIDER hyperparameter search.
Disnersion size
Agr
Acrend
Season number
0.1
0.01
0.0001
0.1
0.001
0.005
Table 16: CSDI chosen hyperparameters.
Epocha
Layers
Channels
Nheads
Diffusion cmbbeddin dimmssion
NSteps
Scherhlee
Time embboddinng
Feature embolldinng
Quad
D.1.2 Models complexity
We can in Table 17 that method has 10 times less parameters than BRITS and 20 times less than SAITS. It is mainly due to their modelisation of interaction between samples. SAITS, which is based on transformers has the highest number of parameters when mTAN has the lowest number of parameters.
Table 17: Number of parameters for each DL methods on the imputation task on the Electricity dataset.
TimeFlow
DeepTime
NeuralProcess
TTAN
SAITS
BRITS
TIDER
Number of parameters
60%
1515k
248k
113k
137k
6 220k
034k
D.2 Imputation for previously unseen time series
Setting In this section we analyze in details the imputations results for previously unseen time series described in Section 4.1. Spocifically, TimoFlow is trained on a given set of time series within a defined time window and then used for inference on new time series. We train TimeFlow on 50 % of the samples and consider the remaining
as the new time series.
We compare in Table 18 observed grid fit scores and missing grid inference scores for time series known training and time series unknown at training.
The results presented in Table 18 indicate that the inference MAE for missing grids shows consis- tency between known and new samples, regardless of the data or sampling rate. However, is worth noting that there is a slight drop in performance compared the results in table Table This decrease is because in Table 18, the shared architecture is trained on only half the samples, affecting its overall performance.
Results
22

---

Published in Transactions on Machine Learning Research (04/2024)
Table 18: TimeFlow MAE imputation errors results for imputation previsouly unseen time series.
Known
New
Fit
Infereace
Fis
Inference
0.10
0.046
0.006
0.302
0.010
0.144
0.088
0.309
0.016
Elcctriiity
0.30
0.093
0.022
0,206
0.010
0.163
0.073
0.271 0.017
0.05
0.014
0.002
0,104
0.015
0.000
0.037
0.109
0.016
0.10
0.017
0.002
0,082
0.015
0.052
0.036
0.099 0.017
Solar
0.20
0.22
0.008
0.078
0.014
0.088
0.031
0.89
0.017
0.30
0.038
0.009
0.072
0.013
0.063
0.028
0,081 0.018
0.00
0.011 0.045
0.066
0.013
0.007
0.025
0,050 0.019
0.05
0.041
0.003
0.291
0.013
091
0.051
0:291
0.012
0.033
0.001
0,200
0.010
0.091
0.080
0.216
0.012
Trafic
0.20
0.037
0.006
0.175
0.008
0.095
0.058
0.156 0.013
0.30
0.048
0.005
0,164
0.006
0.098
0.051
0.175 0.013
0.50
0.068
0.001
0.139 1 0.007
0.042 0.110
0.169
0.012
D.3 Details on ReepTime adaptation for imputation
As DeepTime was proposed to address the forecasting task with a deeptime-index model, the authors did not tackle the task of imputation and left it out for future work. Given the success of this method and the motivation of our work, we wanted to explore its capabilities to impute time series with several subsampling current. framework, we first. tried to train the model in a self-supervised way, i.e. trying
rates. Following to reconstruct observations 2 E T after the INR has been conditioned with the Ridge Regressor the same set of observations, but discovered failure cases for < 0.20. To be faithful to the original supervised training of DeepTime, we therefore randomly mask out 50% of the observations that we use as context for the Ridge Regressor and try to infer the other 50% (the targets) to train the INR.
We provide a qualitative comparison of the model's performance with these two different training procedures in Figure 7. We can notice that the model that results from the self-super training perfectly fits the observations but completely misses the important patterns of the series. On the other hand, when DeepTTime is trained to infer target values based on observations, it is able to capture the general trends. We think that in the small subsampling regime (r < 0.20), the Ridge Regressor easily fits very well all the observations which hinders the training of the INR's basis.
bndiidie MAC sullouaey MAZ 30d1
6.0
ESC
Truth
Deeptime self-supervised
Deeptime supervised
Observed points
Figure 7: Electricity dataset. Self supervised Deep'Time imputation (blue line) and supervised Deep Time imputation (black line) with 5% of known point (red points) on the eight first days of samples 11 (top) and (bottom).
23

---

Published in Transactions on Machine Learning Research (04/2024)
D.4 Imputation against non deep learning methods
Setting In addition the deep learning methods presented in Table 1, we evalute TimeFlow against two classic machine learning baselines, K-Nearest Neighbours (KNN) and linear interpolation, which valuable for getting idea of the complexity of the problem.
Table 19: Mean MAE imputation results on the missing grid only over five different time window. 7 stands for the subsampling rate. Bold results are best, underline results are second best.
TimeFlow
Linesr interpolation
KNN (k=3)
0.06
0.324 1 0.013
0.828 à: 0.045
0.531 à: 0.033
0.10
0.250
è 0.010
0.716
0.030
0.416 0.020
Electricity
0.20
0.225
0.008
0.518
0.020
0.363 0.019
0.30
0.212
0.007
0.396
0.022
0.342 0.017
0.50 0,194
0.007
0.275
0.015
0.323 0.016
0.66
0.095
0.015
0.339
0.031
0.151 0.017
0.10
0.083
0.015
0.170
0.014
0.128 + 0.017
Solar
0.20
0.072
0.015
0.088
0.010
0.110 0.016
0.30
0.061
0.012
0.063
0:000
0.103 0.017
0.50
0.064
0.013
0.044
0.008
0.086
0.016
0.06
0.283
0.016
0.813 0.027
0.387 0.014
0,10
0.211 0.012
0.701 0.026
0.293 0.012
Traffic
0.20 0.168
0.006
0.508 0.022
0.249 0.010
0.30
0.151 0.007
0.387
0.228 0.009
0.50 0.139 0.007
0.263
0.204
TimeFlow improvement
Results KNN imputation uses information from other individuals and gives satisfactory results at all sampling rates. On the other hand, the purely univariate approach of linear interpolation struggles at low sampling rates but performs well at high sampling rates. TimeFlow significantly outperforms both baselines by a large margin.
E
Forecasting experiments
E.1
Distinction between adjacent time windows and new time windows during inference
In Section 4.2, we presented the forecasting results for periods outside the training period. Those periods can be classified into two types: adjacent to disjoint from the training period. Figure illustrates these distinct test periods for the Electricity dataset. The same principle applies to the Thaffic and SolarH datasets, with one notable difference: the number of test periods is smaller in these datasets compared to Electricity dataset due to the fewer time steps available.
In Table 2, we presented the results indistinctly for the two types of test periods: adjacent to and disjoint from the training window. Here, we aim to differentiate the results for these two types of window and emphasize their significant impact on Informer and AutoFormer results. Specifically, Table 20 showcases the results for the test periods adjacent the training window. In contrast, Table 21 displays the results for the test periods disjointed from the training window
Results TimeFlow, TST DLincar and DeepTime maintain consistent forecasting results whether tested on the period adjacent the training period or on disjoint period. However, AutoFormer and Informer show significant drop in performance when tested new disjoint periods.
24

---

Published in Transactions on Machine Learning Research (04/2024)
Adjacent time window
New windows
Test period n*1
Test period n²2
Test period
Test period n14
Test period n'5
4000
8000
12000
16000
20000
test look-back window
Train period
test horizon
Figure 8: Distinction between adjacent time windows and new time windows during inference for the Elec- Iricily dataset
Table 20: Mean MAF forecast results for adjacent time windows. H stands for the horizon. Bold results are best, underline results are second best.
Coulinmous wetbode
Discrete methods
H
TimeFlon Flom
Time
Neusal Proness
Pateh-TST
Di.luow
AntoRormer
Informer
96
0.218 + 0.017
0,210 = 0.027
0.392 i 0.045
0.214 = 0.020 0.236 = 0.035 0,310 0.081
0.293 :: 0.0184
Eleetricity
192
0.238
:: 0.012
0.251 = 0.023
0,401 i 0.040
0.225 =: 0.017
0,248 = 0.082
0,322
0.046
0.336 = 0.082
336
0.265
+: 0.036
0.290
0.034
0,431 1 0.075
0.242 + 0.024
0,284 = 0.643
0.330
0.019
0,405
= 0.041
720
0.318
4: 0.073
0.356
0.060
0.605 1 0.149
0,291 + 0.040
0,370 +: 0.086
0.456
0.062
0.489
0.072
96
0.172 + 0.017
0.197 = 0.002
0.221
0.048
0.232 + 0.008
0.204
0.002
0.261
0.053
0.273
0.023
SolarIl
192
0.198 + 0.010
0.202
0.014
0.244
0.048
0.231
0.027
0,211
0.012
0.312
0.085
0.256
0.026
336
0.2007 + 0.01g
0.200
0.012
0.241
0.005
0:254
0.048
0,212
0.019
0.341
0.107
0.287
0.006
720
0,215 +: 0.016
0.011 0.240
0,403
0.147
0:271
0.036
0.246
0.015
0.368
0.006
0.341
0.049
96
0.210 +: 0.033
0.229
0.032
0,283
0.028
0,201
1: 0.031
0.225
0.031
0,299
0.080
0.321
0.113
192
0.2808
+ 0.021
0.220
0.020
0.023 0.202
0.196
0.024
0.215
0.022
0.320
0.136
0.321
0.062
336
0.237
+ 0.040
0.247
0.033
0.305
0.030
0,220
0.086
0.244
0.835
0,450
0.127
0.304
0.066
720
0,266
0.048
0.045 0,290
0.037 0,339
0.268
0.050
0,290
0.047
0,630
0.043
0.411
0.065
TimeFlow improvemenn
6.56 %
30.79 %
2.64
7.30
35.43 %6
33.07 96
Table 21: Mean MAE forecast results for new time windows. H stands for the horizon. Bold results are best, underline results are second best.
Continnous mothods
Discrete methods
H
TimeFlow
DeepTine
Neural Process
Patch-TST
DLinear
AutoPormer
Informer
96
0.230
0.012
0.245 0.028
0.392 4 0.045
0.222
0.023
0.240 0.025 0,806 +: 0.281 0.605
0.227
Electricity
0.246 0.025
0.252 0.018
0.401
0.046
0.231
0.020
0.257
0.027
0.545 + 0.186
0.776
0.257
336
0.271
0.029
0.285
0.034
0.434
0.076
0.253
0.027
0.298
0.051
0.571 + 0.181
0.823
0.241
720
0.316
0.051
0.350
0.048
0.007
0.15
0.299
0.008
0.373
0.075
0.674
0.245
0.811
0.225
96
0.208
0.005
0.206
0.026
0.221 à 0.048
0.293
0.089
0.212
0.019
0.238 # 0.027
0.234
0.011
0.206
0.012
0.207
0.037
0.244
0.048
0.274
0.060
0.223
0.029
0.356
0.122
0.250
0.033
SolacH
336
0.211
0.199
0.035
0.240
0.006
0.264
0.223
0.327
0.029
0.366
0.039
0.222
0.020
0.147 0.403
0.262
0.012
0.218
0,283, 0.0275
0.039
0.229
0.326
0.099
0.065
Traffle
0.213
0.292, 0.0236
0.031
0.223
0.575
0.254
0.049
0.239
0.305, 0.0392
0.040
0.252
0.598
0.286
0.065
0.280
0.339, 0.0375
0.270
0.059
0.304
0.641
0.072
0.064
TimeFFoo Flow improvement
2.50
27.75 %
%
6.90 %
46.26 %
45.53 %
25

---

Published in Transactions on Machine Learning Research (04/2024)
E.2 Plots comparison: TimeFlow vs PatchTST
Table 2 demonstrates the similar lorecasting performance of TimeFlow and PatchTST across all horizons. To visually represent their predictions, the figures below showease the forecasted outcomes of these methods for two samples (24 and 38) and two horizons (96 and 192) on the Electricity, SolarH, and Traffic datasets.
Sample: 24 Horizon 96
Sample 38 Horizon 96
-45
-2.
3.5
oba
0.00
Lio
Sample:
1.92
Sample
0.5
5.0
3.50
Li0
Ground Truth
PatchTST
TimeFlow
Figure 9: Qualitative comparisons of TimeFlow vs PStchTST on the Electricity dataset for new time windows
Sample: 24 Horizan: 96
Sample 38 Horizon 96
0.00
6.55
aãe
1.00
0.60
0.85
6.00
6.05
x.o
Sample: 24 Horizon 192
Sample
38 Horizon 192
350
840
3.45
1.6
Ground Truth
PatchTST
Time
Figure 10: Qualitative comparisons of TimcFlow vs PatchTST on the SolorH dataset for new time windows
Sample
Sample
0.25
x
Horizon
0.70
3/7
3.0
1/0
6.20
6.35
0.5
0.5
0/5
1.50
Ground Truth
PatchTST
Time FlonFlow
Figure 11: Qualitative comparisons of TincFlow vs PatchTST on the Traffic datasct for new time windows
Results The visual analysis of the figures above reveals that the predictions of TimeFlow and PStchTST remarkably similar. For instance, when examining sample 24 and horizon 192 of the Traffic dataset, both
26

---

Published in Transactions on Machine Learning Research (04/2024)
forecasters exhibit similar error patterns. The only noticeable distinction emerges in the SolorH dataset, where PStchTST tends to overestimate certain peaks.
E.3 Baseline details
E.3.1 Baselines training and hyperparameters
We provide a detailed breakdown of the hyperparameters and our training approach for the forecasting baselines. We took an in-depth approach, testing each method under a range of configurations to ensure they were well-suited to the unique characteristics of the datasets and tasks hand. For DLinear and transformer baselines, including PatchTST, AutoFormer, and Informer, detailed in the PatchTST baselines (code) and adhered to the best practices recommended for our particular tasks. Notably, our implementation of PatchTST was combined with ReVIN, enhancing the robustness of the results. Regarding Deep'Time, we followed the recommended hyperparameters. opting for a structure with 5 layers, each 256 units wide, and using 4096 Fourier features spanning a diverse set scales. As for the Neural Process, the standard model did not train as expected. So, we customized its architecture to align
utilized the implementations
conduct a fair comparison with TimeFlow. We used the INR and hypernetwork from TimeFlow the Neural Process with our temporal frequency bias and shift modulation technique. We also meticulously scarched for the optimal hyperparametors, like the Kullback Leibler (KL) divergence weight and learning rale. Moreover, extended the training duration to ensure thorough convergence.
E.3.2 Models complexity
In this section, we present the parameter counts and the inference time for the main forecasting baselines. Except for TimeFlow and DeepTime, the number of parameters varies with the number of samples, the look- back window, and the horizon. Thus, report the number of parameters for two specific configurations, including a fixed dataset, a fixed look-back window, and a fixed horizon. In Table 22, we see that for PStchTST and DLinear, the larger the horizon, the more the number of parameters increases. In Table 23, it is shown that all methods' computational time increases with the horizon, which is expected. Moreover, TimeFlow is slower than the basclines that use forward computations only. Still, on the Electricity datasot, for example, the method can infer for 321 samples a horizon of 720 values with a look-back window of 512 timestamps in less than 0.2s, which does not look prohibitive for many real-world usages. This is mainly due to the small number of gradient steps at inference.
Table 22: The number of parameters for main basclines the forccasting task the Electricity dataset for horizons 96 and 720. The look-back window size is 512.
Flow DeepTiaee
Process Patch: TST
Informee Autoformer
3.5kk
602k
6.30kk
Table 23: Inference time (in seconds) for the forecasting task on the Electricity dataset with horizons 96 and 720 and a look-back window of length 512. The statistics are computed over 10 runs using an NVIDIA TITAN RTX GPU.
Time Flow
Patch-TST
DLinear
DoepTime
AutoPormer
Informer
0.117
0.007
0.003
0.176
0.020
0.008
0.001
0.002
0.034- 0.001 0.0251 0.002
27

---

Published in Transactions on Machine Learning Research (04/2024)
E.4 Sparsely observed look-back window: comparison with Patch- TST
Sctting and bascline. Let's consider setting where at inference time, the look-back window is sparsely observed. Models such as PatchTST must proceed in Iwo steps: (i) completing the look-back window a dense regular grid using imputation; (ii) apply the model on the completed window to predict the future. We compared TimeFlow with the following two-step processing baselinc: lincar interpolation handling the missing values within the partially observed look-back window, and PatchTST handling the forecasting task. We conducted experiments on the Traffic and Flectricity datasets, focusing on the 96 and 192 horizons. Tn Table 24, we present the results at different sampling rates 0.2, within the look-back window.
Table 24: MAE results for forecasting on new samples and new period with missing values in the look-back window. Best results in bold.
ThmeFlow
Linoer interpo
PatekTST
II
Imputation erior
Foroust error
Imputation crfor
Poeorost ceror
1.
0.000 : 0.000
0.228 = 0.028
0.000 +: 0.000
0.221 :: 0.023
0.5
0.151 1: 0.003
0.239 :: 0.013
0.257 1: 0,008
0.279 1: 0.026
96
0.451
Electricity
0.1
0,272
0.006
0.295
0.016
0.663 +: 0.029
0.631 :: 0.053
1.
0.000 à 0.000
0.238 = 0.020
0.000 4: 0.000
0.229 4: 0.020
0.5
0.149 +: 0.001
0.235 = 0.011
0.258 + 0,006
0,280 1: 0.032
192
0.1
0,274
0.010
0.289
0.016
0.689 4: 0.030
0,650 0.080
1.
0.000
0.000
0.217
0.032
0.000 + 0.000
0.203 + 0.037
0.5
0.219
0.017
0.224 + 0.033
0.276 +: 0.012
0.255 4: 0.041
96
0.029
0.532 + 0.017
0.483
Trallic
0.738
0.721
1.
0.000 + 0.000
0.212 + 0.028
0.000 + 0.000
0.197 + 0.030
0.5
0.176
+ 0.014
0.217 = 0.017
0:276 + 0:011
0.245 +: 0.029
192
0.2
0.233
: 0.017
0.236 = 0.021
0.532 + 0,020
0,480 : 0.050
0.734
0.787 0.172
Results. Although PatchTST performs slightly better with a dense look-back window, its performance significantly deteriorates as the value of t decreases. In contrast, the performance of TimeFlow only minimally affected by the reduction in the sampling rate.
E.5 Influence of the look-back window for forecasting
In Figure 12, it is shown that both excessively short and overly long look-back windows can harm TimeFlow forecasting performance. More precisely, the performances increases with the look-back window size up to certain size, where the performances then drop slowly.
0.30
MMb forecadd
0.28
40
96
192
336
512
720
820
1220
book-buckkoo Window langh
Figure 12: MAE forecast error per look-back windows length for the Electricity dataset (horizon window length is 336). The model is trained a given time window and tested on four new time windows.
28

---

Published in Transactions on Machine Learning Research (04/2024)
E.6 Influence of the horizon length for forecasting
In Figure 13, it is shown that the performances decrease with the length of the horizon. This is to be expected, since the longer the horizon, the harder the task.
0.45
0.40
0.20
48
96
152
336
720
1440
2880
Horizon window length
Figure 13: MAE forecast error per horizons length for the Electricity dataset (look- back window length is 512). The model is trained given time window and tested on four new time windows.
29

---

Published in Transactions on Machine Learning Research (04/2024)
F Discussion on using frequency embedding as input to regression models
F.1 Related work
In forccasting univariate time series, several models have explored the integration of frequency embedding of limestamps for regression purposes. Taylor and Letham (2018) approached the time series as a continuous function of time using a general additive model (Hastie, 2017). They represented seasonality components as learnable Fourier series while explicitly specifying the ground truth seasonalities (e.g., weekly, monthly). Similarly, Hyndman and Athanasopoulos (2018) proposed embedding the timestamp with the ground truth frequencies and applying a regression model to predict the series value the timestamp t. Both methods rely on the explicit specification of seasonalities and are tailored for purely univariate time series, where information is not shared between samples.
In contrast, other models, such as TimeFlow or DeepTime (Woo et al., 2022), based on deep learning techniques, offer more flexibility. These approaches can autonomously learn relevant frequencies and ef- lectively share information between samples through backpropagation. This enables a more dynamic and adaptable approach to time series forecasting, particularly in scenarios with complex temporal patterns and inter-sample dependencies.
F.2
Experiments
Given the scasonal patterns observed in the Solar, Electricity, and Truffic datasets, an alternative to doep learning forecasting methods is individually regress timestamp embedding on the corresponding value using robust regressor. This approach exploits the inherent periodicity in the data, using timestamp embeddings to capture temporal dependencies and accurately predict the target values.
Baselines. We compare TimeFlow against two regression baselines:
This baseline uses the same frequency embed- ding as TimeFlow and applies an XGBoost regressor (Chen and Guestrin, 2016) on top. The aim to assess whether the XGBoost regressor can effectively identify the correct frequencies, filter out irrelevant ones, and establish the appropriate mapping between timestamps and values.
TimeFlow frequencics embedding + XGBoost:
XGB Explicit Seasonal encoding + XGboost: in this baseline, we give explicitly the right frequencies of each datasets to the model. It important to highlight that this method uses information that other baselines don't. For instance for the Traffic dataset, the Explicit Seasonal encoding is 7t) (t, cos (2ml) sin (2a!) cos 21x7 integers trend, daily frequencies and weekly frequencies. We apply the same type of frequencies embedding for Solar and Electricity with the appropriate seasonalitics.
2nt (21xT) It allows explicitly 2rt .sin 21x7
Experimental Setup. For each dataset and sample, we applied the frequency embedding individually and then trained an XGBoost regressor on each observed timestamp (the look-back window in forecasting and the observed grid in imputation). This approach results in model per sample. The XGBoost regressor is configured with the following hyperparameters:
estimators: 500
max depth: 4
learning rate: 0.1
lambda: [0.1, 1, 10]
The optimal regularization parameter lambda is determined through cross-validation The imputation and forecasting results presented in detail in Table 25 and Table 26, respectively.
30

---

Published in Transactions on Machine Learning Research (04/2024)
Table 25: Mean MAE imputation results on the missing grid only. In the table, T stands for the subsampling rate, i.c. the proportion of observed points considered for each samples. Bold results are best, underlined results are second best.
Thneelow
TimeFlow froquencies emboodding XGB Explicit Sessomal cncoding
XGDI
0.95 0.324 0.013
0.834 0.092
0.365 0.051
0.10 0.250 0.010
0.761 0.074
0.918 0.049
Electricity
0.20 0.225
0.008
0.632 0.006
0.278 0.044
0.30 0.212 0.007
0.536 0.041
0,259 0.048
0.50 0.194 0.007
0.418 0.042
0,238 0.022
0.05 0.095 = 0.015
0.603 + 0.035
0.231 0.021
0.10 0.083 = 0.015
0,178 +:2 0.021
0.190 0.022
Solar
0.20 0.072 = 0.015
0,350 4: 0.022
0.150 4: 0.019
0.30 0.061 = 0.012
0,286 à: 0.018
0.134 d: 0.011
0.50 0.054 0.013
0.227 +: 0.015
0.123 d: 0.015
0.05 0,288 0.016
0.739 + 0.140
0.344 0.036
0.10 0.211 0.012
0.676 + 0.129
0:200 + 0.029
Traffic
0.20 0.168 0.006
0.562 :: 0.108
0.245 0.027
0.30 0.151
0.007
0.487 + 0.095
0:223 0.015
0.50 0.139 0.007
0.393 + 0.083
0.198 0.021
Time low improvement
69.5 %
33.7 %
Imputation results. TimeFlow performs better than the other two methods. Although the second base- line explicitly incorporates ground truth frequencies and provides decent results, its inability to share infor- mation between samples leads to the loss of valuable insights that TimeFlow effectively exploits. In addition, the first baseline struggles to Iearn the correct frequencies and overfits observed data points, resulting in excessive high-frequency noise. As a result, its performance degrades significantly compared to the second baseline, where frequencies are explicitly provided. These findings underscore TimeFlow's ability to iden- tify the underlying frequencies, and leverage shared information across samples to improve accuracy during imputation.
Table 26: Mean MAE forecast results for adjacent time windows averaged over different time windows. Each time, the model is trained on one time window and tested the others (there are 2 windows for SolarH and 5 for Electricity and Traffic). H stands for the horizon. Bold results are best, and underlined results are second best
H TimeFlow
TimeFlow frequencies embeediing XGB
Explicit Seasounal uncoding XGB
96
0.218
0.017
0.662 0.102
0.282 0,020
Flectricity
192
0.238 0.012
0.750 0.128
0.279 0.021
336
0.265
0,036
0.509 1.1
0.294 0.041
720
0.318 0.073
0.852 0.144
0.357 0.092
96 0.172 0.017
0.792 0.062
0.244 0.023
192
0.198 0.010
0.933 0.055
0.236 0.018
SolazH
336
0.207
0.019
1,033 0.022
0.229 0.022
720
0.215 0.016
1.116 0.057
0,262
96 0.216 0.033
0.655 0.150
0,288
Teaflle
192
0.208 0.021
0.678 0.139
0.206
336
0.237 0.040
0.719 0.143
0.202
720 0.266 0.048
0.741 0.140
Tlneelow improvement
70.8
%
Forecasting results. TimeFlow also outperforms the other two baselines in forecasting. However, the improvement over the sccond bascline, where the correct froquencies are explicitly provided, is more modest compared to the gains observed in imputation. Nevertheless, the relative improvement achieved by TimeFlow remains significant. (exceeding 15 %). Similar to the imputation scenario, the XGBoost baseline with Time-
31

---

Published in Transactions on Machine Learning Research (04/2024)
Flow timestamps encoding, which attempts to learn the correct frequencies, fails to discern them accurately and introduces excessive high-frequency components.
G Latent space exploration
G.1 Latent space interpolation between two learned codes
It is interesting to understand how the latent space behaves between two learned codes, and 22, which are representations of 21 and We propose in Figure 14 to visualize how new 2A = Az1 + A)22 are decoded the time-series domain (Jo. (xx) values).
Setting. We choose two 21 and 22 learned from the Electricity dataset and interpolate these two latent codes for E {0.0, 0,1, 0.25, 0.50, 0.75, 1.).
Results. In Figure 14, we observe that the interpolation path between two codes yields a smooth transition in the time series domain. This suggests that the latent space is smooth and well-structured, and that the learned representations captured meaningful features of the time series, which could explain TimeFlow's generalization property.
G.2 TimeFlow sensitivity to modulations perturbation
In the preceding section, observed the smoothness of the latent space. A crucial question arises: can interpret each dimension of the latent space independently ?
Setting In this setting, we perturbed specific dimension of the modulation (by adding Gaussian noise) for only one particular layer of the INR for the Electricity dataset. Then, we observe the difference in the time domain between the non-perturbated TimcFlow and the perturbated onc. For instance in Figure 15, we add noise only for the third layer of the INR and the 50th channel.
Results. In Figure 15, we obscrved that adding a small perturbation added a smooth daily frequency pattern. In Figure 16, observed that adding small perturbation induces bias that impacts the high frequencies but does not affect the low frequencies. Finally, in Figure 17, adding small perturbation induces very local and slight bias (the effect is almost mull). In conclusion, the impact of the small perturbation depends on the channel and the layer, but it is hard to interpret each dimension independently.
G.3 Visualization of two code distributions in the latent space
Examining the behavior of the latent space at the instance level is of particular interest. It provides insights into how individual time series evolve within the latent space. However, exploring the latent space between two time series distributions is also crucial.
Setting. We propose encoding all samples (321 samples) from the Electricity dataset for two distinct time periods (each period is about 25 days Rs 600 timestamps). This results in two distributions of latent codes, each representing different temporal support. Then, we employ Principal Component Analysis (PCA) visualize these two latent code distributions in two-dimensional space, as illustrated in Figure 18. This visualization allows to explore the structural differences, similaritics, and temporal variations in the latent space representations across the specified time intervals. In Figure 18a, the two compared time period are separated by approximately 3 months (+= 2000 timestamps). In Figure 18b, the two compared time period are soparated by approximately 6 months (=s 4000 timestamps).
Results. Figure 18a shows that when the temporal periods are not too far from each other, the distributions of codes can largely overlap in the 2D visualization from the PCA. Conversely, as illustrated Figure 18b, when the temporal periods are far from each other, the distributions of codes between the two periods become more distinct in the 2D visualization. This observation suggests that the proximity disparity in temporal
32

---

Published in Transactions on Machine Learning Research (04/2024)
wrtida-0.00
-20
-.1
120
unb
-45
-11
040
umbod-00.2
3.2
680
65
136
lambbaa -0.75
55
52p
lamodaaw0 >
Iamodal11.
Figure 14: Visualization of the reconstructed time series for different linear interpolation of the two codes 21 and learned from the Electricity datasct.
33

---

Published in Transactions on Machine Learning Research (04/2024)
Vridel CUINE pRTcuu DUEAE GAYE
Figure 15: Effect of adding small perturbation to the modulation shift of the third layer and 50th channel.
Figure 16: Effect of adding a small perturbation to the modulation shift of the third layer and 51th channel.
cure
Figure 17: Effect of adding a small perturbation to the modulation shift of the fourth layer and 50th channel.
distribution shift influences the separability latent representations in the 2D PCA space. However, this presumed separability in the latent space does not seem to siguificantly impact the generalization performance of TimeFlow across time, as evidenced by the results presented in Table 2 and Table 21. This suggests that our INR can handle relatively diverse code distributions.
34

---

Published in Transactions on Machine Learning Research (04/2024)
Time Window
Time Window 1 Time Window 3
Time
10-
Princppl Component
-10
10
-75
-5.0
-25
0.0
23
3.0
7.5
Princppa Component
Prindpal Component
(a) The temporal shift between the two codes distribu-(b) The temporal shift between the two codes distribu- tion is months.
tion is months.
Figure 18: Visualization of the two first PCA axes for two distributions the latent codes (temporal distribution shift).
35

---

Published in Transactions on Machine Learning Research (04/2024)
H The intuition behind the meta-learning optimization in time series forecasting
The concept of inner and outer loops is to be reformulated within the broader general framework of model- agmostic meta-learning (Finn et al., 2017), where the authors seek to enable rapid adaptation of the model to unseen tasks. Tn TimeFlow, we adapt the general idea of agnostic meta-learning to our tasks. We propose an efficient way to achieve this goal by splitting the parameters into two parts: context parameters (learned in the inner loop, responsible for the adaptive part of the model) and meta-pos veta-parameters (or 'parameters sharod across tasks') (learned in the outer loop, responsible for the generic part of the model).
In the context of time series forecasting. TimeFlow aims to have subset of parameters that adapts to specific. context ual factors (e.g., the look-back window of particular sample) and another subset that performs the forecasting task according to this learned context (e.g., forecasting any point within the forecast horizon as well as within the look-back window). To achieve this, we seek to adjust the codes 200) exclusively on the contexts (e.g., the look-back window) for each sample j, while the shared parameters between samples and w characterize a shared function capable of forecasting based on given 200) This function could be represented fa (1, (x))))) This concept entails adapting 201 by sample j and training w by batch.
36


Reviews:
[{"review_id": "Y538lHnsTy", "reviewer": "Reviewer_gTKA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The paper is mostly well-written and easy to understand.\n2. The method is simple and their experiments demonstrate its effectiveness.\n\nCons:\n1. I do not find much novelty in their proposed method. While their introduction of bias terms at each layer of the network for their \"Conditional INRs with modulations\" seems to be an interesting technique, I did not any other significant contribution in their proposed method. Also, this seems to be their significant contribution, I find the description provided in this section seems to be a bit blunt.\n\n2. In section 3.3, the description of their meta-learning algorithm is not clear and well-motivated. (a) I request the authors to provide a detailed intuition and the meta-learning algorithm (2) to obtain z*(j). (b) Also, is it possible to think of any other (perhaps simpler) techniques to achieve the same goal? Even if the authors find that such a technique is not very effective, can the authors provide a discussion and additional ablation studies to demonstrate the effectiveness? \n\n3. Full form of mean absolute error (MAE) was missing. Please define the performance metrics and why it is better. I suggest providing a separate subsection/paragraph to describe these error metrics. I felt that MAPE should have been a better measure for the experiments provided in this section.\n\n4. In the related work section, the authors mentioned that DeepTime is the closest to their proposed method. However, they \"cannot handle imputation properly\" and the authors adapted this model for their comparison. Can the authors provide a detailed description of how they adapted DeepTime for their empirical comparison? Such description is needed to ensure their experimental comparisons remain fair.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a time series modelling framework using implicit neural representations (INRs) that basically incorporates the intermediate representations of the neural networks. The goal is to model \"irregular samples, missing data, or unaligned measurements from multiple sensors\" for time series data analysis.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see above.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sxBh0eDs2v", "reviewer": "Reviewer_3vms", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n---------\n\n+ approach seems simple yet effective \n+ inference for new time series appears to be fast (only 3 gradient steps)  \n+ experiments appear to have sufficient coverage: 3 datasets and plenty of representatitive methods across discrete and continuous time paradigms\n+ inductive bias of the frequency embedding seems well-matched to the chosen datasets\n\nWeaknesses\n----------\n\n- unclear how carefully each alternative method had its hyperparameters tuned to the provided datasets\n\n- technical presentation could use some refinement, especially early in Sec 3 providing more precise definitions of key notation \n\n- need more substantive description of why BRITS or similar baselines require \"whole model retraining\"\n\n- The claim of using \"second-order optimization\" for the meta-learning seems to need further explanation. Throughout Alg 1 and Alg 2, I only see first-order gradient descent updates to codes $z^j$ and parameters", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper develops an approach to time series representation learning intended for applications that need to *impute* and *forecast* over irregularly-sampled continuous time.\n\nIt appears tailored to tasks involving many univariate time series that exhibit periodic-like trends homogeneously across the sequences. For example, the evaluations study:\n\n- electricity demand at every hour for many customers in the same region over the same time period\n- traffic (road occupancy) at many locations in one city over the same time period \n\nThe proposed approach produces a representation vector for each time t for each sequence j as illustrated in Figure 1. The key steps are:\n\n1) scalar time t is first encoded as a frequency embedding vector (sin and cos values at N fixed frequencies)\n2) each sequence has a (learned) code vector $z^j$\n3) an L-layer feed forward architecture transforms the frequency vector via a series of linear + ReLU layers, with each linear transform including an additive \"shift\" formed via a weight matrix times $z^j$. This produces a customized representation of time t in sequence j.\n\nAt training time, Algo 1 is used, alternating between:\n- updating each code vector $z^j$\n- updating feed-forward weight matrix parameters\n\nEach update uses gradient steps to minimize reconstruction error. \n\nPost training, the model can be applied to new sequences via Algorithm 2 (where just a code vector is learned).\n\n### Comparison to other work\n\nThis is quite a different approach than other dominant paradigms in time series:\n\n- regular-grid RNNs like BRITS\n- ODE-based continuous time models.\n- models like mTAN that use attention\n\nI think it's a refreshingly different approach", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## C0: Improve (or better describe) handling of hyperparameters for alternative methods \n\nI appreciate that there are a few tables in the appendix describing the sizes of models overall. But I feel that much more care is needed to describe how hyperparameters are set for each alternative method.\n\n- Are you purely using recommended defaults?\n- Are you at least examining learning curves to verify that methods are converging?\n- Have you tried different architecture sizes to better manage over/under fitting tradeoffs?\n\nMy current worry is that some baselines might do substantially better in evaluations if more effort was put into tuning their hyperparameters in a task-specific way.\n\nPerhaps I missed something in the appendix (if so, please just point me there!).\n\n## C1: How much of your reported gains in evaluation come from good inductive bias due to frequency embedding?\n\nFrom inspection of Fig 3 and Fig 5, these datasets have strong periodic trends and the proposed approach's use of frequency embedding seems well-posed to handle that well.\n\nCan you clarify which (if any) of the alternative methods you compare to in Table 1-2 might have a favorable inductive bias for such periodic behavior? For example, doesn't seem like BRITS has a natural inclination for periodicity, and it shows in Fig 3 bottom row. One wonders if a GP with periodic kernel (or similar) might be better for this kind of data.\n\n\n\n## C2: improve technical presentation in Sec 3.1 and 3.2\n\nThe problem setting description in 3.1 and 3.2 needs some work. Here are a few questions that I don't think are sufficiently clear after a close read:\n\n- Is each $\\mathcal{T}$ (with superscripts) meant to be a set of timestamps? a set of observations? or a \"window\"? \n- Why do we need a \"complete\" time series observation set $\\mathcal{T}$? Can't we just define what's observed at training and the heldout values needed at test?\n\n- Is $x^j$ a univariate time series? Or could it be multivariate? How does the size of the output vector $d$ produced by the TimeFlow network compare to the size of $x^j$?\n\n- In figure 1, is $\\sigma$ just a ReLU?\n\n## C3: why would BRITS necessarily need \"full model retraining\"\n\nAt top of page 8, the text implies that BRITS or other approaches need \"full model retraining\" if a new sensor is added.\n\nAre you assuming that all locations at a single time (e.g. all electricity loads across the whole region) are modeled jointly?\n\nCouldn't you instead have a parameter sharing approach, where the BRITS RNN is designed to consume/produce a single univariate time series, using shared parameters that are jointly trained on many related sequences? And thus if you get a new sensor just use the existing parameters of the RNN? Not sure if I'm missing something.\n\n## Minor requests\n\n### c4: Improve presentation of \"second-order\" meta-learning\n\nI don't understand what is \"second-order\" about the proposed approach. Perhaps I'm missing something obvious, but please clarify.\n\n\n### c5: Several questions about method design need clarification\n\n* In Algo 1: Why use both L_in (imputation loss) and L_out (forecasting loss) to update parameters but only L_in to update codes? \n\n* Is 3-step inference really enough? Or is it just a useful compromise between speed and quality? Would you get better performance with more steps? (or with an adaptive number of steps customized for each sequence?)", "broader_impact_concerns": "Not applicable in my view.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IwPZG9gu5T", "reviewer": "Reviewer_kiRB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper proposes a simple and elegant approach that can simultaneously perform imputation and forecasting.\n- Despite its simplicity, the proposed model achieves high predictive accuracy in both time series imputation and forecasting tasks. Showing that such a simple method can outperform the more sophisticated imputation/forecasting approaches is an important contribution.\n- The appendix provides a very thorough description of the experimental setup. The code for the experiments is well-documented, making the results of the paper easy to reproduce. The appendix also includes various ablations, studying the effect of different modeling choices.\n\n**Weaknesses**\n- *Discussion of prior work*: The paper does not discuss a closely-related idea of using Fourier features as input to regression models in forecasting (see, e.g.,[1](https://otexts.com/fpp3/useful-predictors.html#fourier-series), [2](https://pkg.robjhyndman.com/forecast/reference/fourier.html)). It would be important to at least discuss this connection or, ideally, use one of such models as a baseline in the forecasting evaluation (e.g., Fourier features + LightGBM).\n- *Evaluation*: The empirical evaluation is restricted to 3 time series datasets (Electricity, Traffic, Solar) that all exhibit strong hourly/daily seasonality. This setting strongly favors a model based on Fourier features - such as the proposed TimeFlow model. Ideally, the evaluation should involve at least some datasets without strong seasonality. At the very least, this limitation of the setup should be highlighted in the paper.\n- *Notation & clarity*: Some sections of the paper (especially Section 3) are hard to follow;  some notation and abbreviations are used without being properly introduced first. Some specific examples:\n  - The difference between $\\mathcal{T}$ and $\\mathcal{T}^*$ at the start of Section 3 are unclear until Figure 4 later in the paper. Also, it would be helpful to state that $\\mathcal{T}$ only represents the grid points / times, when the time series is observed, and not the observed values.\n  - In Section 3.2 (i), the difference between $f$, $x$ and $f_\\theta$ is unclear. Is $f$ a learned model or the ground-truth continuous signal, where $x$ is its discretized version?\n  - Abbreviations SIREN, REPTILE are used without any references or definition.\n  - It's unclear, why in Algorithm 1 the embeddings $z$ cannot be learned jointly with the parameters $\\theta, w$  and require a separate loop.\n\n**Minor suggestion**\n- The proposed model is trained by minimizing the MSE loss, but then evaluated using the MAE loss. Training the model with MAE loss could improve its performance in the experiments even further.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new deep learning model for time series forecasting and imputation called TimeFlow. The model consists of two main components:\n1. Each time series in the dataset is associated with a learnable embedding vector $z^{(j)}$. \n2. A neural network $f(t | z^{(j)})$ predicts the value of time series with embedding $z^{(j)}$ at time $t$. More specifically, the time $t$ is embedded into a vector using Fourier features and then fed to an MLP. The embedding $z^{(j)}$ is linearly projected to adjust the biases of the MLP. \n\nThe model can also be applied to new time series not encountered in the training set. In this case, new embeddings $z^{(j)}$ should be learned for these time series while keeping the weights of $f(t|z^{(j)})$ fixed.\n\nIn the experimental evaluation, TimeFlow is compared to other time series forecasting and imputation approaches. The proposed model outperforms the existing continuous-time time series models, and achieves comparable performance to strong forecasting approaches such as PatchTST.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to see the weaknesses mentioned above addressed in the revised version of the manuscript.\n1. Discussion & comparison to forecasting regression models based on Fourier features.\n2. Extending the benchmark evaluation to non-seasonal time series datasets (e.g., see any dataset from https://forecastingdata.org/ with weekly frequency).\n3. The claim in the last sentence of paragraph \"Continuous methods for time series\" in Section 2 needs a justification or references to be supported.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QJTA0lQfEh", "reviewer": "Reviewer_T9Ej", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Pro:**\n* The proposed model is to the best of my knowledge novel. It is sensible and addresses shortcomings of previous models in a systematic manner. It builds on previous work but makes original and sound modifications.\n* Experiments are conducted with care - there is a large number of important baselines, and results include a measure of statistical variation, which paints a much more reliable picture.\n* The ablations, though mostly in the appendix, really add to the paper, and answer many questions I had about the main model.\n* The paper is very well written, the supportive figures are nice and really help, and overall the work seem well polished.\n\n**Con:**\n* The biggest weakness to me is that the main experiments are performed on sequences used during training, and results for novel sequences are only shown in the appendix (and are missing comparison against all models). Since the datapoints shown are sampled randomly, the main-paper experiments do not test precisely on the training sequences, rather, they test a very strict case of in-distribution generalization. While this case is not unimportant, I personally would have liked to see more emphasis on generalization to novel sequences (from a comparable, roughly in-distribution, source). The results shown (in the appendix) are not fully conclusive but suggest that the method proposed in the paper still does quite well in this case, but might lose its edge over some of the strong competitor methods.\n* Evaluations are nice and well executed, but only on 3, relatively small-scale datasets (that are important and highly relevant for the problem setup that the paper wants to tackle). I think it is enough to support the main claims, but a more sophisticated example could also be interesting.\n* (very minor) some of the main tables might be more readable as a (bar) plot, though having the exact numbers (maybe in the appendix) is good in general.\n\n**Verdict:**\nOverall, I think the work is well executed, well polished, and contains more than just the necessary baselines and ablations. I am currently in favor of accepting the paper, and only have minor comments/suggestions. My biggest criticism is that generalization performance on novel sequences, which is practically very relevant, is currently quite de-emphasized, and might even deserve some more space in the main paper. Having said that, I currently believe the main claims are sound and supported by the experimental evidence and I think the work is interesting to the TMLR audience.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary:** The paper tackles the problem of time-series interpolation (imputation) and extrapolation (forecasting) in a setting where data is potentially sampled non-uniformly and sparsely across time. The approach is to treat this as a continuous-time modeling problem, and use INRs (implicit neural representations) combined with meta-learning to do so. INRs are a differentiable parametrization of a continuous-time function that uses a neural network to iteratively refine (by weighted combination and passing through a non-linearity) an initial set of continuous basis functions (parameterized in the Fourier domain). MAML-style Meta-learning is used to learn a family of such functions and a mechanism to rapidly select within this family with very few steps of gradient descent at inference time. Evaluations on imputation and forecasting and the combination of both, with extensive comparison against baseline and SOTA methods, show good performance of the method. Extensive ablations in the appendix justify the design choices and are insightful w.r.t. alternative modeling techniques.\n\n**Main contributions:**\n \n* Proposal of a model for time series modeling that can handle missing, irregularly sampled data, and supports both interpolation and extrapolation simultaneously.\n* Comparison against a number of important and relevant baselines, with results in favor of the novel method. The comparison is performed for interpolation and extrapolation separately, and both combined.\n* Extensive ablations to justify design choices and evaluate alternatives.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please indicate early in the paper, and with good emphasis, that the main results shown are from the same data that was used during training (with other observed points sampled, but the same underlying functions). I first realized this when the generalization ablations in the appendix were mentioned.\n2. Not a must, but consider discussing the generalization to novel sequences a bit more in the paper. It currently creates the impression that the method performs slightly worse and this is a bit shoved under the carpet. I don’t think there is any reason to do this - these results are interesting and I personally would like to see them in the main paper (perhaps as a bar plot to save space…). If the authors would rather leave them in the appendix, I will not protest.\n3. It would be nice to see comparison against (more/all) baseline methods for the novel sequence evaluations too. Basically, Tables 1, 2, and 3 but for the case when evaluating on the 50% novel sequences. Like 2. I don’t consider this a must for publication, but an improvement for a stronger paper.\n4. Please add a limitations section of the current approach and evaluations. The section should also mention that the advantage of TimeFlow over PatchTST disappears on novel sequences (Table 21) with both methods performing equally well. Additionally please mention that TimeFlow is an order or two of magnitude slower at inference time compared to the other baselines.\n5. Figure 2 - bottom left panel: if I understood correctly z(j) is optimized using the sparse/irregular inputs (i.e. the red time-steps and red function values only), which would look similar to the left on the upper panel. Only after optimizing z, and in step 2) (right panel) are the blue dots actually used. Is this correct? If yes, consider changing the bottom left panel to make this more obvious (remove blue dots, and add observed x(j)).\n6. Typo P7: “with on a subsampling rate“\n7. Fig. 3 legend: “learned points” should be “observed points” (same as Fig 5).\n8. Fig. 4.: On my first read it was not entirely clear to me whether the new time windows used for evaluation overlap with the range used during training or not (the figure visually indicates that this is not the case, but it wasn’t clear from the text). Please add a sentence or two and point to the section in the appendix where this is clarified (with a nice figure even).\n9. Please state at/before Table 1 that bold scores are best (I guess without considering overlapping error bars?) and underline scores are second best.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper was reviewed by three reviewers. After rebuttal, all three reviewers were positive, with two Accept and one Leaning Accept. Initial concerns of the reviewers included clarity of presentation, discussion of related work, evaluation metrics of the experiments, etc. These concerns were satisfactorily addressed by the authors. 

Reviewer 3vms had remaining concerns about presentation of meta learning and dataset regarding seasonality. Hope these can be addressed or discussed in the minor revision before publication.
Decision: Accept (poster)