Neural Functions for Learning Periodic Signal
Authors: Woojin Cho
Minju Jo
Kookjin Lee
Noseong Park
Venue: ICLR 2025 Poster (2025)
Keywords: Periodicity
Neural function
Primary area: learning on time series and dynamical systems

Abstract: As function approximators, deep neural networks have served as an effective tool to represent various signal types. Recent approaches utilize multi-layer perceptrons (MLPs) to learn a nonlinear mapping from a coordinate to its corresponding signal, facilitating the learning of continuous neural representations from discrete data points. Despite notable successes in learning diverse signal types, coordinate-based MLPs often face issues of overfitting and limited generalizability beyond the training region, resulting in subpar extrapolation performance. This study addresses scenarios where the underlying true signals exhibit periodic properties, either spatially or temporally. We propose a novel network architecture, which extracts periodic patterns from measurements and leverages this information to represent the signal, thereby enhancing generalization and improving extrapolation performance. We demonstrate the efficacy of the proposed method through comprehensive experiments, including the learning of the periodic solutions for differential equations, and time series imputation (interpolation) and forecasting (extrapolation) on real-world datasets.

Published as a conference paper at ICLR 2025
NEURAL FUNCTIONS FOR LEARNING PERIODIC SIGNAL
Kookjin Lee 'TelPIX, LG CNS, AArizona State University,
Woojin Cho MinjuJoo
Noseong Park KKIIST
(woojin. py, minnju42, fj1ee8344 / @ggmalll com,
noseongekaist.ac. .c.
ABSTRACT
As function approximators, deep neural networks have served as an effective tool represent various signal types. Recent approaches utilize multi-layer perceptrons (MLPs) to learn nonlinear mapping from coordinate to its corresponding signal, facilitating the learning of continuous neural representations from discrete data points. Despite notable successes in learning diverse signal types, coordinate- based MI.ss often face issues of overfitting and limited generalizability beyond the training region, resulting in subpar extrapolation performance. This study addresses scenarios where the underlying true signals exhibit periodic properties, either spatially or temporally. We propose a novel network architecture, which extracts periodic patterns from measurements and leverages this information to represent the signal, thereby enhancing generalization and improving extrapola- tion performance. We demonstrate the efficacy of the proposed method through comprehensive experiments, including the learning of the periodic solutions for differential equations, and time series imputation (interpolation) and forecasting (extrapolation) on real-world datasets.
INTRODUCTION
Coordinate-bautt based multi-layer perceptrons (MLPs) or implicit neural representations (INRs) learn continuous representation of signal, which are collected in discrete measurements, not necessarily sampled uniform mesh grid. Being independent on a regularly-samples data and learning the continuous signal naturally address difficulties in model training on irregularly sampled data or data that are not on Cartesian system. These unique features lead to many successes in applications that have historically posed difficulties for conventional approaches, including learning solutions of complex nonlinear differential equations (DEs) (Sitzmann et al. 2020 Raissi et al. 2019). 3D scene representations (Mildenhall et al. 2021). etc.
Recognizing the potential of INRs, extensive efforts have been dedicated to improving various aspects of INRs. Notably, in Sitzmann et al. (2020). new activation function has been proposed to facilitate the learning of freuurry signals. l... In Dupont et al. (2022): : Lee et al. (2021): 1): Cho et al. (2024).
meta-learning-bet -baseed training algorithms have been studied for efficiency. In De Luigi et al. (023)
Zhou et al. (2023), research has explored methods for navigating learned latent space and extracting
hidden representations enhance downstream task performance.
Despite these recent advancements, even in its state-of-the-art (SoTA) formulation, INRs surpris- ingly fall short in effectively representing signals that exhibit sequential nature (such as time series). This deficiency becomes particularly evident in their inability to accurately predict signals -uttof-distribution (OoD) inputs, leading catastrophic failures in extrapolation forecasting scenarios. Figure visually illustrates example of learning a simple periodic function sin (50x)) with recent INR architectures including SIREN (Sitzmann al. 2020), FFN (Tancik al.) 2020). and WIRE (Saragadam et al. 2023); the example essentially demonstrates that those existing architectures struggle in extrapolation.
"Equal Contribution

---

Published as a conference paper at ICLR 2025
In this study, we focus on scenarios where the ground truth signal is exhibiting, periodic- ity, which is indeed common in real-world scenarios; for example, in natural science, pe- riodic behaviors are observed in many applications from simple example, such as the oscillation of a pendulum, which is a classic example of periodic motion governed by principles of physics, to a complex example such Earth's 's periodic and cyclical variations of weather and temperature. Consequently, capturing periodicity from given range of data (e.g., historical data) is critical in accurate prediction in extrapolation (e.g.. future evolution).
2,0
2.5
3.3
50
Dun, meitingg wmon 30w05, tess wine 2 2.6600a
To this end, we propose a novel architectural design of INRs. The design involves two key aspects: i) learning Fourier feature mappings for transforming a spatial and/or temporal coordinate to facilitate captur- ing of periodicity and ii) decomposing the learning of periodic and scale components of an observed sig- nal. This separation introduces inductive bias the existence of the periodicity and the trend (dictated by the scale component), enabling precise prediction in extrapolation settings. Moreover, the proposed model inherits all desirable traits of conventional INRs: the data, per-
2.0
3.5
50
training
tead
2.0 WIRE. briinng emor: 1.1e-01, test eriie. 2.20e-00
52
Figure 1: Regression of target function sin (0xz)) and extrapolation tests. The training (yellow) and the test (red) regions are sepa- rated by the vertical bar (a 2.2).
capability to operate on pregularly-samplee data-ine ine-inntttnnaasstt (eliminating the need for large dataset for training), and a lightweight structure (no need for sliding windows or heavy numerical computations during inference).
In summary, our contributions include:
1. We design a novel INR architecture employing two separate internal networks for learning periodic and scale components of an observed signal.
2. We propose algorithm, which requires only single trained model to perform both interpolation (imputation) and extrapolation (forecasting).
3. We demonstrate that our design outperform SoTA baselines in interpolation and extrapolation tasks several well-known real-world benchmark problems.
2 DESIDERATA FOR INR-BASED MODELING FOR PERIODIC FUNCTIONS
Now, identify desired characteristics required for INRs in handling periodic sequential data, and based on the identified characteristics we design an efficient and effective network.
2.1 DECOMPOSITION SEQUENTIAL SIGNAL INTO FACTORS
Decomposing time series signal into interpretable factors, e.g., seasonality and trend, is common practice for improving the performance models in the field time series (Cleveland et al. 1990) (cf. Section 5). Separately predicting them and later combining them into signal is effective in improving the time series model accuracy (Hamilton) 2020). We propose a new INR design for separately modeling the periodic factor and the scale factor and later combining them to reconstruct the signal. This factor-by-facte
processing approach alleviates the modeling burden of INRs for periodic sequential data. Also, do not explicitly supervise the learning of the two individual factors; instead we train the proposed model with the original undecomposed signal to reduce the training overhead and implicitly learn the two factors. Our theoretical analysis (cf. Appendix D shows that learning time series through this design allows effectively extracting periodic factors.
2.2 LEARNING FOURIER FEATURES
Fourier features of signals introduce angle. interpreting the signals in the frequency domain, which is known to be effective in time series modeling. Thus, using Fourier features in the network design can be beneficial for sequential data modeling. Recent INRs, such as SIREN (Sitzmann et al.
2

---

Published as a conference paper at ICLR 2025
Sequential index
encoder
0
p/r(c;; 04)
Scale factors t/ s/yl
Scale decoder 0,
* scale
C
Learnable Fourier mapping
period
Feature encoder
))(
Perlodlc facturs [WF, styl
Petiodic decodax
8g
speriod
f
Figure 2: NeRT architecture. The input coordinate (cf, of) is converted to periodic/scale factors through bhe thr, and v followed by two decoders os and Up to effectively infer the signal intensity XI, at the input coordinate.
2020) and FFN (Tancik et al. 2020 also aggressively use Fourier features. However, those existing Fourier feature extraction methods are optimized mainly toward images (e.g., hand-tuned frequency in the input layer of SIREN) and they often do not show satisfactory performance for sequential data modeling (e.g., poor extrapolation shown in Figure DT. Taking a cue from Li et al. (202I),
overcome the limitation by learning Fourier feature mapping layer (like FFNs but learn Fourier mapping instead of the FFNs' hand-craft mapping) and utilizing the sinusoidal activation (like SIRENs) the details of our model design for NeRT will be described shortly in the next section and our theoretical analyses justify the appropriateness of our design to sequential data processing.
3 MODEL ARCHITECTURE
We describe our proposed INR-based framework (cf. Figure 2 NeRT, for interpolation and ex- trapolation functions with periodicity. As we intend to learn periodic sequential signal in factored form, a periodic factor and a scale factor, we design an encoder-decoderr-t architecture, where i) the encoders generate embeddings for input coordinates, and ii) the decoders take the embeddings as input and produce a periodic factor and a scale factor, individually. The two factors are then multiplied produce the signal at the input coordinate.
neural network
3.1 ENCODER
The encoder of NeRT reads an input coordinate of multi-dimensional sequences, C2 (c),
N, where N and M denote the length and the dimensionality of the sequences. The first coordinate ci refers to the sequential index and the second coordinate of refers to an one-hot vector with one in only the jth element, which is used to index the jth feature. Depending on the application, the first coordinate of is typically temporal index in case of learning time series data or a spatial index æ in case learning periodic signal defined over spatial domain. further describe other encoding schemes for processing raw coordinate input in Appendix C
Embedding of the Input Coordinate For input coordinate (c/, c/)..)) the following FC layers with ReLU activation functions:
(r(c 0,) = FCI, o pr -.C,.-10...000 o FC(()))
(I)
vy (c(c:00))= FCr, pr FC1,-10 o (C)
where L, and L, mean the number of layers in each encoder respectively, and is ReLU. Since of an one-hot vector, one can consider that (c/;0,) generates an embedding for the one-hot vector.
Learnable Fourier Feature Mapping The previous two embeddings, v/: and v.y. do not generate Fourier features and therefore, use the following method to generate them. In particular, our method is to learn an optimal Fourier mapping layer (instead of hand-crafted ones):
(r(; 0r) (Am Osin(w (ww.......... + B Pet mm=
(2)
3

---

Published as a conference paper at ICLR 2025
where ci.m. is m-th value of cf and @ denotes element-wise multiplication. wm E R is the m-th frequency of the Fourier feature dy, which is sampled from uniform distribution [a, 6l. learnable vector Am E I indicates the amplitude, initialized to 1. Vectors Bm E R 6m E R denote the phase shift and the bias respectively, and they are initialized to Optionally, we can employ additional FC layer after Equation 2 to compress the embedding. All together, we use three types of embeddings, the. y, and up given the coordinate.
Remark 3.1. According to the NTK theory (Jacot al.) 2018), our kernel satisfies the stationary and the shift-invariant properties - thus, one can consider that the learnable Fourier feature mapping of NeRT performs ID convolution-base processing of signal with learnable kernel. In addition, our proposed mapping has an additional property that resorts to the extreme value theorem to perform prediction in OoD (extrapolation) p is i) continuous, and ii) confined the min/max values defined by A.
3.2 DECODER
The proposed decoder architecture separately generates the periodic and the scale factors, denoted Øp and 6s. and multiply them infer the signal intensity:
6p( by: Ps FCL, o Ps FCL,-10.... FCi @ wy), (v'F @ 1 1 th 0,) FCL, Pr FC
(3)
FCi @ by @ vr),
where L, and Lg are the numbers of layers in the decoders, Ps is the sinusoidal function, and denotes concatenation of vectors. The output of op and Φ, correspond xperiod and secale in Figure We constrain & period [-1, 1] by using the Sine activation and & inferred periodic and scale factors, respectively. Note that the decoder to infer the periodic factor, ie., 9p. mainly rely on our Fourier feature thy: since y is the embedding of the spatial coordinate, only the contains the temporal information infer. The scale decoder reads all available embeddings for the input coordinate (c/, cf) Our final inference outcome, i.e., the signal intensity, at the input coordinate is & xperiod &scale
to denote the
The advantages of our design are twofold: i) Our learnable Fourier feature mapping layer is specialized to extract periodic features according the periodic factor of the inferred signal intensity based on the Fourier feature and ii) the periodic and scale factors give interpretable predictions for sequential signals both in interpolation and extrapolation.
NTK- analysis and, thus, it is sensible to model
3.3 TRAINING ALGORITHM
To train, we employ regular gradient-based optimizer to minimize mean-squared error (MSE) over data and predictions, MN Σ E x2 Here, we consider M-variate sequences measured at N collocation points (See Appendix Forr a pseudo-code like algorithm). We emphasize again that the proposed model requires single model training and use the same trained model to perform both interpolation and extrapolation while other non-INR baselines require training of different models for each task.
Regularizing Scale Component Imposing extra constraint the scale component such penalty on the magnitude of the 3rd-order derivative of the scale w.r.t. the input can regularize the output of the scale component, preventing from learning oscillatory patterns from the signal. Thanks to the inherent nature of INRs, any order of derivative can be computed via automatic differentiation. In the experiments section, we present results with such penalty.
For training INRs for multiple sequence instances, we provide two different options: a vanilla mode modulated mode. For the modulation, we adopt an idea of latent modulation to our NeRT in Appendix K] To summarize, in the vanilla mode INRs are trained individually and in the modulated mode the shared part is trained via meta-learning algorithm.
Modulation
4

---

Published as a conference paper at ICLR 2025
Pus
Eth
WIT
Tue
Fra
Pue
Nite vear
Scaclede
1.5
1.5
0.5
0.5
0.5
-0.
4.5
1.0
1.6
-1.5
1.2
1.4
1.8
2.0
-1.5
1.0
1.4
2.0
(a) INR basclines
(b) NeRT (Ours)
(c) Periodic/scale factors by NeRT
Figure 3: Damped oscillator. Extrapolation results (Figures 3 training (Figure (c). The left side the solid vertical line represents the training range, while the right side represents the testing range.
(a)-(b) and extracted factors during
4 EXPERIMENTS
In this section, we evaluate the performance of NeRT several benchmark problems. We have largely two sets of experiments: Synthetic data and real-world datasets. the first set experiments, consider synthetic periodic data obtained from solutions of ODEs/PDEs, showcasing the model's applicability to different domains, i.e., spatial (PDE) or temporal (ODE) domains. In the next set of experiments, we demonstrate the performance of models well-known real-world time series
datasets, which can be further categorized into periodic time series (Ziyin et al. 2020 Fan et al. 2022) and long-term time series (Zeng et al., 2022).
To assess whether underlying signals are learned effectively, we formulate tasks as either interpolation or extrapolation tasks; that is, given discrete measurements of the underlying signal, train the proposed model and test it in either interpolation (imputation) or extrapolation (forecasting) settings. Unless otherwise specified, the input the signal either spatial coordinates x or temporal coordinates for spatial or temporal signals, respectively (see Table the summary). We use MSE for the evaluation metric and conduct the experiments with three different random seeds and present their mean and standard deviation of evaluation metrics. More detailed descriptions of experiments and additional analyses such as ablation studies are listed in Appendix.
Experimental Environments We implement NeRT and baselines with PYTHON 3.9.7 and Py- TORCH 1.13.0 (Paszke et al.) 2019) that supports CUDA 11.6. The experiments are conducted on systems equipped with Intel Core-19 CPUs and NVIDIA RTX A6000, A5000 GPUs. All imple- mentations our proposed method can be reproduced by referring to the attached README.nd. To benefit the community, the code will be relcased online.
Baselines for Comparison As baseline models, we consider SIREN (iizmmann et al. 2020). FEN (Tancik et al.) (020) and WIRE (Saragadam et al.) (202)), the three representative INR models. For fair comparison, we set the model sizes to be the same. All models are trained with Adam (Kingma Ba 2014) with learning rate of 0.001. addition, we use eight existing time series models including Transformer-basee and NODE-based models non-INR baselines (cf.. Appendix []. See the full description of the experimental setup in Appendix G
4.1 SCIENTIFIC DATA - SIMULATED DATASETS
In the scientific domain, datasets often exhibit periodic characteristics, and analyzing these pattern is crucial in the processing of understanding the data. We train and evaluate NeRT and the other existing baselines using a two scientific dataset: harmonic oscillation and 2D-Helmholtz equations (MeClenny & Braga-Neto 2020). In all benchmark datasets, NeRT can represent the data more delicately compared other models, and demonstrates a strong capability learn the inherent periodicity within the data.
4.1.1 HARMONIC OSCILLATION
To analyze the performance NeRT, we use the harmonic oscillation trajectory data. Simply reusing existing INRs, however, results in poor performance even for learning simple and noise-less
5

---

Published as a conference paper at ICLR 2025
(a) SIREN
(b) FFN
(c) WIRE
(d) NeRT (Ours)
(e) Ground truth
Figure 4: Extrapolation task on the 2D-Helmholtz equation. Results of the extrapolation task with the training range of x [1, 1.5) and [1,1.5] i.e., the left-lower square, (Figures (a)-(d) and the ground-truth solution (Figure (e)).
30
Ground Truth
Tanh
Truth WiRE
25
Snake
NelTT
25
NelT
RetU
SIREN
20
Temperature
20
25
15
10
-5
2004
2006
2008
2010
2012
2014
2016
2017
-5
2004
2006
2008
2010
2012
2014
2016
2017
(a) Snake baselines
(b) INR baselines
Figure 5: Comparison with Snake and INR baselines. The middle area between the two solid vertical lines represents training period.
simulated ODE data (cf. Figure 3. SIRENs, FFNs and WIREs are known to learn very complex (high-frequency) signals (e.g., images with details) by using either the sinusoidal activation, Fourier features or complex Gabor wavelet activation. However, as shown in Figure 3 these models perform poorly in the extrapolation task, while NeRT produces accurate predictions. (See Appendix for more details on experimental setting.)
4.1.2 PDES WITH PERIODICITY
Here, we demonstrate that NeRT can be applied another domain, namely learning solutions of PDEs matter of fact, PDEs are implicit functions describing dynamics in the spatiotemporal coordinate system. Thus, we test NeRT on learning the solutions of 2D-Helmholtz equation, which is known to produce periodic behaviors. We refer readers Appendix H for details.
Experimental Setups We perform extrapolation tasks over the spatial domain, where the model is trained by using a set of collocation points (z, y) E [L, 1.5]² and is tested on (z, y) E [1, 2 [1, 1.5 The 2D Helmholtz equations The form the PDE as follows:
used benchmark problem in McClenny & Braga-Neto (2020).
8 Ou(r,v) v)
Puu(z, v) 122 (,(( v) g(2, v) =
Ox.² +
Oy²
g(x, v) = (-(a1x)² (227)² + 12 sin(ar T.) sin(a)))))
where k is constant, and q 3) is source term. NeRT and baselines are trained to predict the solution given location (Ir, 3). Unlike the multi-variate time series, uni-variate, NeRT uses only 1/, as its encoder we feed the raw coordinate (x. v) instead of c/.
As shown in Figure 4 all three -Rsbased models have no difficulties in being overfitted to the training data. However, only the proposed method extrapolates accurately in the test region, while the other two baselines fail. We provide the analyses additional experimental results with in Appendix
Experimental Results

---

Published as a conference paper at ICLR 2025
Table Experimental results on periodic time series. The experimental results of the same model, presented on the same row, are from single training, and the best results are reported in boldface. Errors measured Mean Squared Error (MSE) are reported.
Interpolation
Extrapolation
Dataset
# blocks
SIREN
FFN
WIRE
NeRT
SIREN
FFN
WIRE
NeRT
0,0200
0.0189
0,0170
0,0061
0.0256
0.0144
0.0244
0.0057
Electricity
0.0182
0.0144
0.0157
0.0057
0.0233
0.0142
0.0241
0.0077
0.0183
0.0148
0,0168
0,0056
0.0231
0.0142
0.0251
0.0088
0.0174
0.0140
0.0187
0,0057
0.0176
0.0113
0.0200
0.0050
Traffic
0.0169
0.0121
0.0191
0,0062
0.0181
0.0127
0.0199
0.0057
0.0169
0.0114
0,0180
0,0055
0.0185
0.0130
0.0200
0,0115
0.0230
0.0179
0.0182
0.0047
0.0596
0.0495
0.0466
0,0131
Caiso
0.0206
0.0179
0.0165
0.0041
0.0427
0.0399
0.0485
0.0148
0.0337
0.0326
0,0270
0.0099
0.0260
0.0196
0.0473
0,0051
0.0510
0.0502
0.0458
0.0149
0.0326
0.0346
0.0442
0,0235
NP
0.0464
0.0415
0.0424
0.0186
0.0329
0.0313
0.0443
0.0273
0.0476
0.0431
0.0434
0.0196
0.0334
0.0304
0.0453
0.0274
0820925202
(a) SIREN
(b) FFN
WIRE
(d) NeRT
(c) SIREN
(f) FFN
WIRE
(h) NeRT
Figure 6: Forecasting and imputation [Top] Imputation results in NP (())-dd), the middle area between the two vertical lines represents testing block. [Bottom] Forecasting results in Traffic ((e)-(h)), where the area on the right of the vertical line represents testing range.
4.2 REAL-WORLD TIME SERIES DATA
Now, we compare the performance of NeRT and baselines on real-world time series: Two periodic and long-term ones. Experimental results show that the proposed method outperforms in both scenarios, even for the long-term time series, which typically has much weaker periodicity than the periodic time series. On top of that, better represent the real-world time series data, suggest an additional module for encoding time series (cf. Appendix (): in short, this module leverages calendar representation of time indices, which readily available the dataset. In Sections 4.2.2 apply the encoding module to every model, including baselines, and also provide ablation studies on the spatiotemporal coordinate.
4.2.1 ATMOSPHERIC DATA
Experimental Setups We use the atmospheric data from Minamitorishima Island as benchmark dataset known to exhibit periodicity (Ziyin et al.) 2020). The daily maximum temperature data from April 2008 April 2014 is used the training dataset, while the periods from April 2004 April 2008 and from April 2014 to April 2017 are used for test. All other experimental settings follows the Ziyin et al. 2020
7

---

Published as conference paper at ICLR 2025
Linear DLinear
NLinear RNN
LSTM Autoformer
Informer
Latent ODE
NeRT
FEDformer
Neural CDE
0.10
0.10
961 336
720
96 192 336
720
96192
336
720
96192 336
720
(a) Electricity
(b) Traffic
(c) Caiso
(d) NP
Figure 7: Comparisons with time series baselines for varying = {96, 192, 336, 720} (out-
put/prediction window size) with fixed input window size m 48.
Comparison with Existing Baselines To verify the performance of NeRT, first construct INR baselines using SIREN, FFN and WIRE. Additionally, we utilize various activation functions used as baselines in Ziyin
2020), including MLPs with ReL.U, Tanh, and Snake.
As shown in Figure ML.s with Snake, INR baselines and NeRT successfully represent the training period. However, baselines applying ReLU and Tanh activation functions in simple feedforward networks struggle to represent the data during the training interval. In the test period, all baselines, including Snake, SIREN, FFN and WIRE fail in predictions, while only NeRT successfully captures periodicity and predicts the test interval.
4.2.2 PERIODIC TIME SERIES
Experimental Setups For periodic time series experiments, we select four uni- time series datasets, i.e., Electricity, Traffic, Caiso, and NP, which are all famous benchmark datasets used in [an et al. (2022) and are known to have some periodic patterns. To demonstrate the efficacy learning periodic time series, we conduct interpolation and extrapolation tasks on missing blocks, each of which with length of 500 observations
for hourly observations, 500 observations correspond to three weeks. We consider up to three missing blocks, i.e., 9 weeks. Since INR models are able to solve interpolation and extrapolation simultaneously, both tasks are tested by single trained model. Detailed experimental settings such as the location of the blocks and hyperparameters are described in Appendix
Comparison with the Existing INRs First we present the comparisons against the INR baselines, SIREN, FFN, and WIRE in Table (see Table II for standard deviations of the metric), which essentially shows that NeRT outperforms in every task in all scenarios. Especially for Caiso, NeRT exhibits significantly lower errors both the interpolation and extrapolation tasks, which are around quarter of those of baselines. Notably, NeRT shows reasonably small MSFs even for extrapolating the last 1,500 observations. Figure visualizes how three models interpolate in NP (Figures (a)-(d) and extrapolate in Traffic (Figures 6 (c)-(h).. In both cases, only NeRT shows good predictions while two other baselines struggle. An additional set of experiments with modulated versions all INRs be found in Appendix
Comparison with Non-INR Baselines To provide more comprehensive assessment, we conduct experiments comparing NeRT eight existing time series models, including popular Transformer- based baselines. (1) Accuracy: In these experiments, we measure forecasting results by varying input/output window size, m and n and the full results are reported in Tables 12 13 and [4 in Appendix. Figure illustrates the changes in MSE with varying with = 48. Notably, NeRT consistently achieves lower MSE compared to other baseline models; particularly, the slope of NeRT's MSE is much lower than those of other methods' MSE. We emphasize that NeRT does not require retraining while the other methods need to be retrained for varying n. For detailed experimental setups and results, refer to Appendix J (2) Computationall/eemoyy costs: Table 15 in
8

---

Published as a conference paper at ICLR 2025
Table 3: Experimental results of ablation studies on Fourier feature mapping
NeRT (Fixed Fourier mapping)
NeRT (Learnable Fourier mapping)
Dataset
Interpolation
Extrapolation
Interpolation
Extrapolation
Electricity
0.0079=00015
0.0069=0.0009
0.006
0.0057:000007
NP
0.0190=0.0016
0.0313=0.0044
0.0149=0.005
0.0235:10.002
Appendix assures that NeRT has advantages in terms of the overall computatioonal/meroory efficiency than the baselines.
4.2.3 LONG-TERI TIME SERIES
Experimental Setups We conduct experiments on general real-world long-term time series datasets to show the scalability NeRT. The benchmark datasets the long-term series forecasting task (Wen) et al. 2022) are used for our experiments. Since the periodicity is typically weak in those datasets, we consider this task is much more challenging. We randomly drop 30%, 50%, and 70% observations and evaluate the interpolating performance using MSE as metric.
Experimental Results Due to space reasons, we list the experiments only with two datasets, ETTh and National Illness, in conjunction with their detailed experimental settings. Full results are in Appendix [ According to Table in every drop ratio, NeRT beats other baselines by a large margin. For example, in ETThI, NeRT shows an MSE of 0.0911, while SIREN and FFN shows 0.2173 and 0.3407, respectively. Moreover, in National Illness, NeRT outperforms other baselines by 489%.
Table 2: Long-term time series
Dataset
Drop ratio SIREN
FTN
NeRT
30%
0,1945
0,2522
0.0828
ETTh1
0,2173
0.3407
0.0911
70%
0,2605
0,4256
0.1257
30%
0,3502
0,1110
0.0239
National Illness
0.1716
0,2319
0.0291
70%
0.3564
0.4453
0.0871
4.2.4 ABLATION STUDIES
Ablation Studies on the Spatiotemporal Coordinate For all models (SIREN, FFN, and NeRT), test the effect of the proposed coordinate system. In all three models, using the proposed coordinate system is critical in achieving better performance, which is reported in Appendix C.I
Ablation Studies Learnable Fourier Feature Mapping To evaluate the effect of learnable Fourier feature mapping, we design experiments comparing its performance with a fixed Fourier feature mapping. The fixed Fourier feature mapping fixes the learnale parameters of the learnable Fourier feature mapping described in Equation 2 specifically, Wm. Am. Bm, and 8m. The fixed values are to be identical the initial values of the learnable Fourier feature mapping, as discussed in Section 3.1 These experiments are conducted on the "one missing block" setting from Table T We use electricity and NP datasets for this evaluation, and the results are summarized Table 3 As shown in Table 3 using learnable Fourier mapping in NeRT yields better performance compared to using fixed Fourier mapping in both interpolation and extrapolation tasks. That is, by adapting learnable factors to Fourier feature mapping, NeRT successfully learns set of frequencies that better represents the specific time series.
4.3 DISCUSSION
Although the main goal here learn periodic signal effectively, the proposed approach can be also considered as one alternative method for time series modeling. The method has many unique and desirable aspects compared to traditional time series modeling approaches such as RNNs and their
variants (e.g., LSTM) (Connor et al. 1994; Hochreiter & Schmidhuber 1997 Qin et al.] 2017 Lai
et al.
2018. Sherstiisy 2020 Hess et al. 2023
Koppe et al.| 2019 Mikhaei et al. 2022 Brenner
al.
2024). Transformers Vaswani et al.
2017 Zhou al. [202 Wu al.) 202 I Liu et al. 2021
Wen et al. 2022; Zhou et al. 2022). Neural ODEs (Chen et al. 20118) and Neural CDEs (Kidger et al.
2020). As shown in the sets of experiments, the proposed algorithm is capable of efficiently handling

---

Published as a conference paper at ICLR 2025
irregularly measured time series data, operating in a non-autoregresssve way (both in training and test time), and forecasting based on the entire history (i.e., not relying on sliding windows). While promising, the proposed method needs improvements in various aspects to be considered as a serious competitor in the time series modeling business; for example, how to emphasize more on recent trend to make precise predictions, how adjust model parameters efficiently based on new incoming observations, how to extract meta-knowledge of time series data from multiple time series instances of the same kind, etc. The concept of "modulation" and an accompanying meta-learning algorithm (as shown in Section K) offer preliminary insights into addressing these questions. However, a comprehensive exploration of these aspects is beyond the scope of this study. Our primary objective here is to effectively learn periodic signals by designing the new architecture for INRs.
5 RELATED WORK
Implicit Neural Representations INR approaches, including physics-informed neural networks (PINN) (Raissi et al. 2019) for solving PDE problems and neural radiance fields (NeRF) (Mildenhall et al. (021) for 3D representation, are quickly permeating various fields. In addition, countermea- sure to the spectral bias [Rahaman et al. 2019) in vanilla multi-layer perceptrons has been proposed. enabling more sophisticated INR-based representations (Sitzmann et al. 2020; Saragadam et al. 2023). According to Tancik et al. (2020), it is shown that the infinitely wide ReLU based neural equivalent the
kernel method in the perspective of NTK (Jacot et al. 2018). Further extended from Fourier-mapped input networks, the work in [Benbarka et al. (2022) draws connection Fourier series, leading to MLPs with integer lattice mapping input. In the time series domain, there exist INR-based studies on unsupervised anomaly detection (Jeong Shin (022) and time series forecasting (Woo et al. 2022). Relevant works in computer vision domain includes NPP (Chen et al. 2022), which learns periodic signal by developing a periodicity detector and patch-wise matching.
network with random Fourier features
Decomposition Methods Used in Time Series Modeling Decomposition methods aim to separate a time series sample into multiple components, often including a trend, scasonality, and residual component (Cleveland et al. 1990). These components can then be modeled separately, allowing for more accurate predictions and insights. There exist various decomposition- have been used for time series, ranging from traditional time series models to recent deep learning models. For example, the wavelet decomposition decomposes time series into different frequency bands (Pereival & Walden] 2000; Wang et al.) 2018) and the singular spectrum analysis (SSA) decomposes time series into a set of eigenvectors to extract various oscillatory patterns which are interpretable (Vautard & Ghil 1989 (Sulandari al. 2020).
methods that
CONCLUSIONS
Due to the strength in learning coordinate-based systems. INR has high potential for various fields in natural sciences and engincering. However, despite the promising nature of INR, it has been rarely applied to functions with periodic behavior (e.g., time series), and no existing unified models are based on the INR paradigm. Therefore, we aim to address the limitations that conventional time series models possess and propose NeRT which resorts to the advantages of INR to effectively resolve existing challenges in the field. Based on the INR framework, NeRT effectively learns and predicts sequential data by separating the periodic and scale factors. Additionally, we suggest a method for embedding the spatiotemporal coordinate of multivariate time series. Through this approach, NeRT clearly outperforms existing INR models and can represent the sequential data more accurately.
Limitations and Future Works Compared existing time series models which rely on a concept of sliding-windows, windows, INR-based modeling approaches proposed in this paper requires only coordinate- based queries for predictions during inference. This is extremely computationally efficient, but the same time could cause a lack of adaptivity when new set of data is available to fine-tune the model. To mitigate this issue, we adopt the idea of modulation from Dupont et al. (2022). which is shown to be effective in resolving such issue some extent. Enabling the model to adapt to shift modulation can be challenging even with the latent modulation; however, we consider this as the future direction for this research and we focus on learning functions with periodic behaviors in this study.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea governmen((MSIT) (No. RS-2024-00457882,, AI Research Hub Project). K. Lee acknowledges support from the U.S. National Science Foundation under grant 2338909. K. Lee also acknowledges Research Computing at Arizona State University for providing HPC resources that have contributed to the partial research results reported within this paper.
REFERENCES
Nuri Benbarka, Timon Höfer, Andreas 7ell, al. Seeing implicit neural representations fourier series. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2041-2050, 2022.
Erik Lien Bolager, Ana Cukarska, Iryna Burak, Zahra Monfared, and Felix Dietrich. Gradient-free training of recurrent neural networks. arXiv preprint arXiv:2410.2344 2024.
Manuel Brenner, Christoph Jürgen Hemmer, Zahra Monfared, and Daniel Durstewitz. Almost-linear mns yield highly interpretable symbolic codes in dynamical systems reconstruction. arXiv preprint arXiv:2410. 14242440..14400 14240,
2024.
Bowei Chen, Tiancheng Zhi, Martial Hebert, and Srinivasa G Narasimhan. Learning continuous implicit representation for near-pen sarr-perooiic patterns. European Conference on Computer Vision, pp. 529-546. Springer, 2022.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Woojin Cho, Kookjin Lee, Donsub Rim, and Noseong Park. Hypemetwork-base meta-learning for
low-rank physics-informed neural networks. Advances Neural Information Processing Systems, 36, 2024.
Robert B Cleveland, William Cleveland, Jean McRae, and Irma Terpenning. Stl: A seasonal-trend decomposition. J. Off. Stat, 6(1):3-73, 1990.
Jerome T Connor, R Douglas Martin, and Les E Atlas. Recurrent neural networks and robust time series prediction. IEEE transactions on neural networks, 5(2):240-255 1994.
Luca De Luigi, Adriano Cardace, Riccardo Spezialetti, Pierluigi Zama Ramirez, Samuele Salti, and Luigi Di Stefano. Deep learning on implicit neural representations of shapes. arXiv preprint SIi::2302.05438, 2023.
Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data functa: Your data point is a function and you can treat it like one. 39th International Conference Machine Learning (ICML), 2022.
Wei Fan, Shun Zheng, Xiaohan Yi, Wei Cao, Yanjie Fu, Jiang Bian, and Tie- Yan Liu. Depts: deep expansion learning for periodic time series forecasting, arXiv preprint arXhv:2203.07681, 2022.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Tch, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International conference on machine learning, pp. 170--1713. PMLR, 2018.
James Douglas Hamilton. Time series analysis. Princcton university press, 2020.
Florian Hess, Zahra Monfared, Manuel Brenner, and Daniel Durstewitz. Generalized teacher forcing for learning chaotic dynamics. arXiv preprint arXiv:2306.04400 2023.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
11

---

Published as a conference paper at ICLR 2025
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel:
Convergence and
generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Kyeong-Joong Jeong and Yong-Min Shin. Time-series anomaly detection with implicit neural representation. arXiv preprint urXiv:221.11195. 2022.
Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. Time2vec: Learning vector representation of time, arXiv preprint arXiv:/907.0532. 2019.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696-677 67077 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
Georgia Koppe, Hazem Toutounji. Peter Kirsch, Stefanie Lis, and Daniel Durstewitz. Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fmni. PLoS computational biology, 15(8)=e007263, 2019.
Guokun Lai, Wei-Cheng Chang, Yiming Yang. and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp. 95-104, 2018.
Jacho Lee, Jihoon Tack. Namhoon Lee, and Jinwoo Shin. Meta-learning sparse implicit neural representations. Advances in Neural Information Processing Systems, 2021.
Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, and Samy Bengio. Learnable fourier features for multi- dimensional spatial positional encoding. Advances Neural Information Processing Systems, 34: 15816-15829, 2021.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex Liu, and Schahram Dust- dar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations, 2021.
Edward N Lorenz. Deterministic nonperiodic flow. Journal of atmospheric sciences, 20(2):130-441, 1963.
Levi McClenny and Ulisses Braga-Neto. Self-adaptive physics-informed neural networks using soft attention mechanism. arXiv preprint arXiv:2009.04544 2020.
Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self dofulaodug multivariate point process. Advances in neural information processing systems, 30, 2017.
Jonas Mikhaeil, Zahra Monfared, and Daniel Durstewitz. On the difficulty of learning chaotic dynamics with rnns. Advances in Neural Information Processing Systems, 35:1 1597-119711112 2022.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, peghrmerformance deep learning library. Advances in neural information processing systems, 32, 2019.
Donald B Percival and Andrew Walden. Wavelet methods for time series analysis, volume 4. Cambridge university press, 2000.
Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison Cottrell. A dual-stage attention- based recurrent neural network for time series prediction. arXiv preprint arXiv:1704.022 2017.
12

---

Published as a conference paper at ICLR 2025
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pp. 5301-5310. PMLR, 2019.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physies, 378:686-7017 2019.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sapple time series. Advances in neural information processing systems, 32, 2019.
Vishwanath Saragadam, Daniel LeJeune. Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G Baraniuk. Wire: Wavelet implicit neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp- 18507-18516, 2023.
Alex Sherstinsky, Fundamentals of recurrent neural network (mn) and long short-term memory (Istm) network. Physica D: Nonlinear Phenomena, 404:132306, 2020.
Satya Narayan Shukla and Benjamin M Marlin. Multi-time attention networks for irregularly sampled time series. arXiv preprint arXiv:2101. 10318, 2021.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473. 2020.
Winita Sulandari, S Subanar, Muhammad Hisyam Lee, and Paulo Canas Rodrigues. Time series forecasting using singular spectrum analysis, fuzzy systems and neural networks. MethodsX, 7: 101015, 2020.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Robert Vautard and Michael Ghil. Singular spectrum analysis in nonlinear dynamics, with applications to paleoclimatic time series. Physica D: Nonlinear Phenomena, 35(3):395-422 1989.
Jingyuan Wang, Ze Wang, Jianfeng Li, and Junjie Wu. Multilevel wavelet decomposition network for interpretable time series analysis. Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 2437-2446, 2018.
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey. arXiv preprint arXiv:2202.07125, 2022.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Deeptime: Deep time- index meta-learning for non-stationary time-series forecasting. arXiv preprint arXiv:2207.0666 060466 2022.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto- for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419-2222 19222430 2021.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205. 13504, 2022.
Allan Zhou, Kaien Yang, Yiding Jiang, Kaylee Burs,, Winnie Xu, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Neural functional transformers. arXiv preprint arXiv: 1354,, 2023.
Haoyi Zhou, Shanghang Zhang. Jiegi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11 11106-11115, 2021.
13

---

Published as a conference paper at ICLR 2025
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequeney enhanced decomposed transformer for long-term series forecasting. In International Conference Machine Learning. pp. 27268-27286. PMLR, 2022.
Liu Ziyin, Tilman Hartwig, and Masahito Ueda. Neural networks fail to learn periodic functions and how to fix it. Advances in Neural Information Processing Systems, 33: 33:1558-1594, 2020.
14

---

Published as a conference paper at ICLR 2025
A
SUMMARY OF DATASETS
Table 4: Summary of datasets used in the experiments
Problem
Ingui
Physical meuning
Toy example (Section
sin(30x)
ID spatial coordinate
Scientilic data (Section
4.1
2D Ilelmboliz equations
Harmonic escillator
f
2D spatial courdinate
temporal coordinate
Atmospheric data
temporal coordinate
Time series data (Section
4.2
Periodic time series data
temporal cocrdinate temporal coordinate
Long-term time scries data
calendar information
The summary of the datasets used in our experiments is presented in Table 4 wave as a toy example and two ideal scientific datasets obtained from ODE and PDE as benchmark datasets. Additionally, we evaluate NeRT's performance on real-world time series data, using total of seven real-world benchmark datasets: four types of periodic time series datasets and three types of long-term time series datasets. In every dataset, NeRT outperforms other basclines.
We employ a sinusoidal
B COMPARISONS TO THE EXISTING TIME SERIES MODELING APPROACH
Although the proposed method mainly focuses on learning periodic signals, it is highly relevant to time series modeling and we discuss the proposed method as an alternative model for time series models.
As time series processing has been one fundamental task of machine learning, many different deep- learning (DL) algorithms have been studied. Recurrent neural networks (RNNs) and their variants (c.g., long short-term memory, or LSTM) (Connor et al. 1994; Hochreiter & Schmidhuber, 1997]
Qin et al. 2017 Lai et al., 2018 Sherstinsky (2200) have been (one of) the first DL algorithms for processing time series data. Transformers ( (Vaswani et al. 2017: Zhou et al.) 2021 Wu et al., 2021 al., 202T Wen et al. [022]] Zhou et 20.2) quickly superseded RNNs thanks to its higher representation learning capability aided by the self-attention. Another line of modeling approaches include the differential equation- based DI. paradigm, also known as continuous-tin methods, which time series neural ordinary differential equations (NODEs) (Chen et al. 2018)) and neural controlled differential equations (NCDEs) (Kidger two exemplary works in this line of research.
exhibit favorable features for handling arregularlyssamppe
al. 2020
INR-based time series modeling as proposed in the current manuscript exhibits some advantages over previous time series modeling approaches. First, the proposed method is naturally capable of processing irregular time series, which is often challenging to typical time series models. Some remedies include time series embedding (Kazemi al., 2019), positional encoding (Vaswani et al.) 2017). padding, or likelihood based approaches (Mei & Eisner 2017). which partially resolves the difficulty at increased cost. Secondly, the proposed method operates without the concept of sliding window (i.c., reading m past observations and predicting n observations ahead, where m and are hyperparameters) The performance of traditional approaches relying on the sliding window is often highly sensitive to the hyperparameters (cf. Appendix B.I). Lastly, the proposed method can be trained even in the case where only a single time series instance is available.
The weaknesses or the limitation of the proposed method are highly related to some items of the listed advantages. The vanilla INR-based modeling does not provide means to place more emphasis on recent measurements for forecasting (c.g., sliding windows), which is typical practice in other time series modeling approaches. Also, it typically trains a single INR for an individual time series instance, making it challenging to extract features that are common to dataset level.
As mentioned above, to address such weaknesses, the concept of "modulation" and an accompanying meta-leane leetnng algorithm (as shown in Section K)) can be utilized. However, it requires further significant investigations, which has not been pursued in this study as the main goal of this study is to learn periodic signal effectively with the new architecture design.
15

---

Published as a conference paper at ICLR 2025
RNN(100)
LSTM(100) LSTM(10,080)
RNN(10,000)
0.2
Table 5: Experimental results of time
series models varying window-sizes
-0.4
0.6
Model
100
10000
RNN
0.5999
0.4478
-0.8
LSTM
0.6927
1.0530
10
20
30
40
50
Time
Figure 8: Visualization of results according to window- size of existing time series models
B.1 SENSITIVITY OF WINDOW- BASED TIME SERIES MODELS
The commonly used approach time series models, called shifting window, assumes fixed window size. However, this method has several critical drawbacks. First, it requires finding an optimal input window size. Using a too small window size may result in small and inference time, but the model sees short patterns and struggles to infer effectively. On the other hand, employing a too large input window size can enable capturing long-term patterns but at the same time can lead to long training and inference time. Thus, the window size is highly sensitive hyperparameter, and finding balance between these two settings is a challenging task.
To empirically show how the size of window affects the model in forecasting time series, we compare results of forecasting 50 future observations by varying the input window size in {100, 10000} with other conditions fixed. For the experiment, LSTM, and train them for 1,000 epochs on ETTh1. As shown in Table for both RNN and LSTM, their model predictions show big differences depending on the window size, in terms of MSE. Note that models with a small window size, i.c., a window size of 100, shows better predictions when using RNN, while LSTM shows lower MSE when using longer window size, i.e., window size of 10000. predicted values exhibit significant
choose two typical time series models, RNN and
The same trend can also be observed in Figure 8 In Figure (8) differences. Therefore, the window size a critical hyperparameter modeling time series.
C SPATIO-TEMPORAL COORDINATE SYSTEMS OF TIME SERIES DATA
Feature index
{ef
Year
Yeaz
(stact your butt you)
(Suna - fasu))
1
0
0
0
1
Moath
Month
(5ite $ane)
nd
[1
Day
Scaling
(Sish farw)
Day
3md
encoding
One hot
0
1
0
Hour (0 2a)
Hour
(Sxe ~ Sate)
:
Miluut
Minutt
(Suee Sata)
[M (M-1)
0
1
0
0
0
Stomd
Seomnd (Stss ~ Sar.))
(0--
M
1
0
0
(Mia. Man.)
(a) Time stamp encoding
(b) Feature index encoding
Figure 9: Spatio-temporal coordinate construction. Our method to define temporal coordinates [ (Figure (a)) and spatial coordinates {c/ JM (Figure (b)).
Unlike domains of applications where the coordinate system relatively well-defined (i.e., 2d Cartesian coordinate systems for images solutions of partial differential equations, PDEs), multi- variate time series modeling needs new definition of a coordinate system. Given an M- -i-iimensional multi-variate time series [x where x; x(4+) Ω-M naive way of building INRs is to
16

---

Published as a conference paper at ICLR 2025
directly use the time stamps, f,, coordinate and x; signal intensity. However, such coordinate system is too primitive to properly represent multi-variate time series data in INRs.
Instead, propose to manufacture refined coordinate system, providing fine-gres tine-rrnullrrty in indexing temporal domain and feature domain (See Figure D.. We interpret the feature domain, i.c., the M- dimensional space, Ω- spatial domain and, thus, propose novel spatio-temporal coordinate system for multi-variate time series data. In our new proposed coordinate system, a coordinate can be expressed as {c; == c (fc, 133 ) where cf E R. and cx E RD Here, of and of denote the temporal and the spatial coordinates of t, (i E N) and the j-th feature (j M). respectively.
Spatial coordinates For the spatial coordinate, we employ one-hot encoding, which translate an integer index into one-hot vectors. Unlike images or PDE solutions, the spatial locality in the feature domain less clear. Moreover, imposing certain order based on the integer index is likely to impose undesirable biases. Thus, we propose to use one-hot encoding, which removes reliance on a specific ordering and general enough to represent represent many different multi-variate time series.
Temporal coordinates For the temporal coordinate, when the calendar information is readily available (as in Section 4.2.3), we fully utilize the information; we apply simple min-max scaling to put different quantitative values (year, month, day, etc) into the same numerical scale [Smin Smax]. effectively resolving numerical issues.
C.1
ABLATION STUDIES ON THE SPATIO-TEMPORAL COORDINATE
3.0
1.0
0.8
58
54
52
5500
10000
12300
20200
13300
10400
13508
3900
20000
12300
10200
13200
10c0o
10590
(a) SIREN (w/o coordinates mapping)
(b) SIREN (coordinates mapping)
3.0
3.0
0.8
0.6
64-
0.4
022
02
50
50.
-02
2010
m9to
2000
10200
20100
FFN (w/o coordinates mapping)
FFN (coordinates mapping)
2.0
7.0
Then
Tiee
60
0.2
5900
19300
1330d
10400
$900
10000
10200
L3300
10535
(c) NeRT (w/o coordinates mapping)
(f) NeRT (coordinates mapping)
Figure 10: Experimental results of ablation study according to the spatiotemporal coordinates mapping introduced in Appendix C The figures in left column are the results without coordinates mapping and the figures in right column are the results using coordinates mapping.
As an ablation study, we aim to investigate the impact of the proposed coordinates mapping in Figure on INR models. The overall experimental setup follows that of section 4.2 To ensure fair comparison. we set the model size to be the same across all models. In addition, the initial frequency of SIREN and NeRT is fixed at 30, which is known to work well in Sitzmann et al, (2020). The results presented in Figure a0)a). (c). (c) correspond to the models trained without coordinates mapping, where one-dimensional scaled time-stamp used input the model. For the baselines,
17

---

Published as a conference paper at ICLR 2025
SIREN and FFN, it can be observed that they struggle to accurately represent the overall domain, while NeRT exhibits a more refined representation. However, NeRT still exhibits discrepancies from the ground truth in extrapolation. As shown in Figure b),,, (d), (f), which is the result of using coordinates mapping, it can be observed that INR models can depict time series more accurately when input coordinates mapped. In particular, NeRT can even perform sophisticated extrapolation inference.
D ANALYSES ON NERT
D.1 FOURIER FEATURE MAPPING IN TEMPORAL COORDINATE TRANSFORMATION
The Fourier feature mapping introduced in Tancik et al. 2020) transforms the input coordinates using periodic functions, allowing the neural networks solve the spectral bias MLPs. This approach is simple yet powerful way to address the problem. Moreover, the Fourier feature mapping exhibits shift-invariant properties from an NTK perspective. Our learnable Fourier feature mapping enjoys all the advantages of the Fourier feature mapping and moreover, it addresses the difficulty of finding a lask-specific fixed set of frequencies in the Fourier feature mapping since we learn them as well. As explained in Section 2 NeRT maps the temporal coordinate onto desired closed finite interval [Smin, Smaz] Therefore, NeRT can approximate discrete pordinatee-base time series continuous function in the closed domain and thus, the extrapolation in the original temporal coordinate can be somehow considered as an interpolation in the learned coordinate, e.g., everyday 12pm has Smin regardless of year and month. Under these conditions, when coordinates outside the training range are inputted, the domain of NeRT converted to the maximum and minimum values of F by the extreme value theorem. In other words, by Equation [ wide range of the temporal coordinate, and this approach works very effectively, especially when performing extrapolations.
NeRT can operate in
E ADDITIONAL EXPERIMENTS WITH AN ODE-BASED SYNTHETIC TIME SERIES
E. EXPERIMENTAL SETUPS
To evaluate the effectiveness of our proposed NeRT in an ideal setting without any noise, we conduct experiments (cf. Figure 3). We use the damped oscillation ODE, which can represent harmonic and oscillatory motion, as the benchmark dataset. The specific equation and analytic solution are as follows:
dxx
+ ba dr
m.d dt2
+ higw O,
dt
(5)
2tt) Are 2 cos(wat + sp).
Equation 5 represents the motion equation that accounts for all the forces acting on the object, where denotes the mass of an object, and and ka are physical constants. We set = 1, ba 0
(undamping) or 4 (damping), wa 50, and A = 10. All experiments employ 2,000 epochs.
E.2 n1th ORDER DERIVATIVE PENALTY
We propose an additional penalty term for use in the training process INR to represent smooth dynamics and prevent overfitting. Since NeRT directly takes coordinates input, it can compute the 2h partial derivative the output with respect to the input coordinates. In the harmonic oscillation experiment, aim refine the periodicity/scale by adding the third-order partial derivative term with respect to time coordinate of the output to the loss.
E.3 EXPERIMENTAL. RESULTS
In this section, we do additional experiments on undamping/damping oscillatory signals. For each ODEs, we do interpolation, extrapolation, and mixed task, which do both interpolation and
18

---

Published as a conference paper at ICLR 2025
extrapolation at the same time. Since extrapolation task for damping oscillatory signal is in the main paper, we list results for other five experiments, and the results are summarized Table 6
Table 6: Additional results with an ODE-based synthetic time series
Task
SIREN
FFN
WIRE
NeRT
Interpolation
96.6879=31.001 0011
0.0856#0.045 55.0295+224795
0.0183=+00025
Undamping
Extrapolation 23.9945
1.0117407321
52.8508:22.014
0.4109::04335
Interp+Extrap 93.38601138.3111 0.127010.0888
50.6520.15.5992
0.0336110.220
Interpolation 0. 0.166.:09
0.00171.0.006
0.2258
0.0011::0.000
Damping
Interp+Extrap 0.2758:+0.0077
0.003420.0018
0.1967:0.009
0.0004::.002
As shown in Table 6 NeRT outperforms SIREN, FFN and WIRE in every task by big margin, and visualizations of the results are summarized in Figures
I2
and 15
19

---

Published as a conference paper at ICLR 2025
20
20
Tue
Tue
10
10
10
-10
-10
-10
1.2
3.6
1.0
1.4
1.2
3.6
2.0
(a) INR basclines
(b) NeRT (Ours)
(c) Periodic/scale factors by NeRT
Interpolation task on an undamping oscillatory signal. Results of interpolation task with an ODE (Figures [[a) and extracted factors during training (Figure [T.... The inside of the two solid lines testing range (t E [1.2, 1.8) and the outside is a training range (t [1.0, 1.2), [1.8, 2.0)).
Figure 11:
20
20
20
WIRE
Scale
10
10
-10
1.0
1.2
1.4
1.6 Time
1.8
2.0
1.0
1.2
1.4
1.6
1.8
2.0
1.0
1.2
1.4
1.6
1.8
2.0
Time
Time
(a) INR baselines
(b) NeRT (Ours)
(c) Periodic/scale factors by NeRT
Figure 12: Extrapolation task on an undamping oscillatory signal. Results of extrapolation task with ODE (Figures] 12 (b)) and extracted factors during training (Figure e)). The left side of the solid line represents the training range (f [1.0,1.4) 1.4)). while the right side represents the testing range [1.4, 2.0)).
20
WHz
Tur
Nale
-10
-10
Time
(a) INR baselines
(b) NeRT (Ours)
Periodic/scale factors by NeRT
Figure 13: Interpolation and Extrapolation task undamping oscillatory signal. Results of interpolation and extrapolation tasks with an ODE (Figures [3] a)-()) and extracted factors during
training (Figure c)). The training range / E [1.0, 1.2], [1.5, 1.8), and the testing range I E [1.2, 1.5, [1.8, 2.0).
20

---

Published as a conference paper at ICLR 2025
Pue
kth
MM
WIRE
Nww Tue
Fred
Tuet
Nute voa
Scde lndo
1.5
0.5
0.5
0.5
-1.5
4.5
-1.5
1.0
1.6
2.0
1.2
1.4
1.8
2.0
1.0
1.4
2.0
(a) INR basclines
(b) NeRT (Ours)
(c) Periodic/scale factors by NeRT
Figure 14: Preliminary study with damping oscillatory signal. Results of interpolation task with an ODE (Figures] [4(a)-b) and extracted factors during training (Figure c)). The inside of the two solid lines represents the testing range (1 E [1.2, 1.8), while the outside represents the training range (t E [1.0, 1.2], [1.8, 2.0).
1.5
1.5
1.5
0.5
00.5
0.5
0.55
-1.5
-1.5
1.0
1.2
1.4
1.6
1.8
2.0
1.0
1.2
1.4
1.6
188
2.0
1.0
1.2
1.4
1.6
1.8
2.0
Time
Time
Time
(a) INR baselines
(b) NeRT (Ours)
(c) Periodic/scale factors by NeRT
Figure 15: Preliminary study with a damping oscillatory signal. Results of interpolation and ex- trapolation tasks with an ODE (Figures 15 (b)) and extracted factors during training (Figure 15 c)).
The training range are E [1.0, 1.2), [1.5, 1.8], and the testing range are E [1.2, 1.5), [1.8, 2.0].
F ALGORITHM
To provide detailed explanations of the training process of NeRT, we present the following training Algorithm
Algorithm Training of the proposed method
** Training */
Training datasets: M-variate sequence ((x,, fi)
Input: set of training sampled coordinate: {c/, (fc/) LM 1
Initialize the parameters of NeRT {0,, 0r. 0g, 0g, 0p) for epoch to ep do
Compute forward pass: (ir((ct, of); (0,, 0r 0, 0p)) Compute MSE: (œt.) - xi,)²²
Compute backward pass and update the parameters
Compute the validation error with validation data
end for
Output: Optimal parameters of NeRT (Lowest validation error)
21

---

Published as a conference paper at ICLR 2025
G DETAILED DESCRIPTION OF DATASETS
G.1 2D-HELMHOLTZ EQUATION
The 2D Helmholtz equation, used in Section 4.1 is differential equation utilized for modeling wave and electromagnetic phenomena. We experiment with the condition of the Equation 4 and are able to directly obtain the analytical solution u(x, y) sin(a1 TT)) sin(a)
G.2 TIME SERIES DATASETS
Table 7:
Dataset statistics. Max. length (resp. Min. length) is the longest (resp. shortest) timestamp length among the timestamps the features the samples.
Periodic time series
Long-term time series
Dataset
Electricity
Traffic
Caiso
NP
ETTh1
ETTh2
National Illness
Frequency
hourly
hourly
hourly
hourly
hourly
hourly
weekly
Start date
2012-01-01
2008-01 02 2013-01-01
2013-01- -01 2016-07-01
2016-07-01
2002-01
End date
2015-01-01
2009-03-31
9020-12-31
2018-06-26
2018-06-26
2020-06-30
# features
1
7
7
7
Max. length 26,304
10,560
74,079
70,120
17,420
17,420
966
Min. length
26,271
10,512
41,547
70,076
17,420
17,420
966
Experiments on time series datasets consists of two parts: i) periodic time series, and ii) long-term time series. We use four datasets used as benchmark datasets in Fan al. (2022) for the periodic time series task and three datasets from Wen et al. (2022) for the long-term time series task. As shown in Table 7 in order to show how NeRT predicts in various scenarios, choose datasets with a wide range of length and frequency, including both uni-variate and multi-variate datasets. Detailed descriptions of datasets
follows:
Periodic time series
Electricity contains hourly records of electricity consumption from 2012 to 2014.
Traffic consists of hourly data from the sensors in San Francisco freeways, providing information on the road occupancy rates between 2015 and 2016.
Caiso comprises hourly actual electricity load series in various zones across Californa from 2013 to 2021.
-N contains a collection of hourly energy production volume from 2013 to 2020 in several European countries.
Long-term time series
ETTh1 and ETTh2 (Zhou et al., 2021 are hourly collected ETT (Electricity Trans- former Temperature) datasets from July 2016 to July 2018.
- National Illness is a weekly collected medical dataset from the Centers for Disease Control and Prevention of the United States. It contains the information of patients with influenza-like illness spanning from 2002 to 2021.
22

---

Published as a conference paper at ICLR 2025
H
DETAILED EXPERIMENTAL RESULTS ON 2D HELMHOLTZ EQUATION
11 12
(a) ReLU
(b) Tanh
(c) SIREN
(d) FFN
(e) WIRE
(1) NeRT (Ours)
(g) ReLU
(h) Tanh
(i) SIREN
(i) FFN
(k) WIRE
(l) NeRT (Ours)
Figure 16: Detailed experimental results
Table 8: Experimental results of Helmholtz equations
Epech
ReLU
Tanh
SIREN
FFN
WIRE
NeRT
0.339
0.317020.0406 0.3325:0.1177
0.4625-000820
0.2753=000270
0.0025=0.0066
10,000
14.1617=14.1040
1.4257+1.9118
0.250420.0366
0.4455=0.070
0.2540=0.0232
0.009900.005
In the extension of Section 4.1 to provide more comprehensive analysis, we increase the number of epochs and add baseline. Figures (6]a)-(f) represent the results after training for 1,000 epochs, while Figures [l)) depict the results after training for 10,000 epochs. Furthermore, keeping all hyperaarameters and model sizes the same, add baselines by changing the activation function to ReLU and Tanh. This is depicted in the first and second columns of Figure [6 As shown in Figures [6] and (b), models using ReLU and Tanh activation functions struggle even to learn the training range. On the other hand, all INR models demonstrate successfully precise learning of the training range (ar [1.0, 1.5], y/ E [1.0, 1.5)) within just 1, 1,000 epoch. When training extended to 10,000 epochs, all models managed to learn the training range, but only NeRT successfully predict the test range (z [1.5, 2.0]. y E [1.5, 2.0)).
23

---

Published as a conference paper at ICLR 2025
1 EXPERIMENTS ON PERIODIC TIME SERIES
1.1 DETAILED EXPERIMENTAL SETUPS
Length 500
Length 500
Length 500
Block 1
Block
Block 3
Block 2
Block Block 2 Block 1
Time
Training block
Interpoiation block
Extrspolation block
Validation block
Figure 17: Experimental setup on periodic time series.
Table 9: Best hyperparameter configurrttion in the periodic time series task
Whit
inner
dim(s
dim(t/)
dim(hp)
dim(h,)
Electricity
5.0
1.0
30
30
10
30
Traffic
10.0
1.0
30
50
10
Caiso 10.0
1.0
10
30
50
10
NP
3.0
30
We design the experiment using the first 10 samples, each of which consists of 12 blocks (cf. Figure 17p. from each dataset and conduct an experiment to fill in the values of missing blocks. In sample, we perform both the interpolation and the extrapolation tasks. Detailed locations and constructions are summarized in Figure 17 As shown in Figure 17. in a sample there are three interpolation blocks colored in red, three extrapolation blocks colored in blue, one validation block colored in green, and the remaining yellow parts represent the training dataset. Each block has length of 500, and the missing blocks for each task, as specified as "# blocks" in Table I are as follows:
# blocks means that perform the interpolation and extrapolation tasks for Block 1.
# blocks = 2 means that we perform the interpolation and extrapolation tasks for Block I and Block 2.
# blocks 3 means that we perform the interpolation and extrapolation tasks for Block 1, Block 2 and Block 3.
A validation block is not used for training in every task, and used for the purpose of validation. Each model trained for 2,000 epochs.
For hyperparameters, set Smax to 1 and for the fair comparison, for all methods and share the frequencies across the models. There are two frequencies used hyperparameters, init and inner init is used in learnable Fourier feature mapping and corresponds to in Equation inner denotes the frequency of the sinusoidal function p. in Equation 3 For the number of layers, we set Le, Le,Ls and L, to and L, 5. The best hyperparameter configurations of NeRT in the periodic time series task are summarized in Table 9 We use an additional FC layer after Equation and let h, and h, be the hidden vectors of the period and scale decoders, respectively.
similar model sizes
1.2 ADDITIONAL EXPERIMENTAL RESULTS
We conduct periodic time series experiments on two numerical methods, Linear (linear interpolation) and Cubic (cubic interpolation), and report the results in Table [00 As shown in Table [0] NeRT beats two numerical methods. We compare NeRT to Linear and Cubic only for the interpolation task, since those numerical methods not able extrapolate.
24

---

Published as a conference paper at ICLR 2025
19 20 and 21 show the visualizations of the remaining datasets' interpolation and extrapolation results that are not included the main paper. In Figures 18 and 20 show the results at the best epoch, i.e., the lowest validation error, and in Figures 19 and 21 show results at the last epoch, i.e., after 2,000 epochs. NeRT avoids overfitting to training data in both cases while other two baselines significantly overfitted to training data and fail in the testing range.
Additionally, in Figures 18
In Figure 22 propose how the periodic and scale factors of NeRT work in the periodic time series interpolation and extrapolation tasks. Figures 22 (a)-(d) correspond to the interpolation task and Figures 22 (e)-(h) correspond the extrapolation task.
Table 10: Additional experimental results on periodic time series. The best results are reported in boldface.
Interpolation
Dataset
blocks
Linear
Cubic
NeRT
0.0126
61 61.4654
0.0061=10.0006 -10000066
Electricity
0.0155
47.4215
0.0057:10.0005
0.0182
41,9424
0.0056.10.0006
0.0220
4.4286
0.0057
Traffic
2
0.0191
3.0941
0.0062:+0.000
3
0.0191
4.1447
0.0055.10.or
0.0223
8.8545
0.0047 11000012
Caiso
2
0.0171
6,3525
0.00411:0.0002
0.0171
6.4894
0.009911000007
0.0426
1.3798
0.0149::0.005
NP
2
0.0477
2.2489
0.0186:100.0m
3
0.0454
3.8557
0.0196 19196.1..0000
Table 11: Standard deviations of experimental results on periodic time series. Each experiment repeated three times.
Interpolation
Extrapolation
Dataset
blocks
SIREN
FFN
WIRE
NeRT
SIREN
FFN
WIRE
NeRT
10.0007
10.0021
20.0014
10.0006
10.0005
+0.0009
10.0004
10.0007
Electricity
2
10.0009
10.0012
10.0018
10.0005
10.0006
10.0006
10.0027
10.0019
3
+0.0004
#0.0013
=0.0017
=0.0066
+0.0001
+0.0005
+0.0020
#0.0008
+0.0007
+0.0007
+0.0011
+0.0002
+0.0009
+0.0007
+0.0003
+00000
Traffic
2
10.0003
10.0001
10.0010
10.0000
10.0004
10.0003
10.0004
10.000 0003
10.0006
10.0001
10.0014
10.0002
10.0006
10.0002
10.0003
10.0026
+0.0008
10.0013
10.001
20.0012
+0.0018
+0.0045
+0.0010
+0.0014
Caiso
2
+0.0002
+0.0012
+0.0019
+0.0002
+0.0005
+0.0017
+0.0029
+ +0.0066
3
10.0004
10.0017
10.0016
10.0007
10.0007
10.0008
10.0022
10.0008
10.0005
10.0013
10.0019
10.0005
10.0014
10.0021
10.0033
10.0023
NP
2
10.0004
+0.0002
10.0019
-0.0008
+0.0010
+0.0015
+0.0020
+0.0022
3
++0.0001
+0.0005
+0.0010
#0.0006
+0.0007
+0.0017
+000041
+0.0030
25

---

Published as a conference paper at ICLR 2025
(a) SIREN (Electricity)
(b) FFN (Electricity)
(c) WIRE (Electricity)
(d) NeRT (Electricity)
C01020832
SIREN (Traffic)
( FFN (Traffic)
WIRE (Traffic)
(h) NeRT (Traffic)
GERERENL
SIREN (Caiso)
FFN (Caiso)
WIRE (Caiso)
() NeRT (Caiso)
Figure 18: Interpolation results missing intervals at the lowest validation error checkpoint. Interpolation results in Electricity (Figures T8 (a)-dd), in Traffic (Figures (e)- and in Caiso (Figures 18 (i)) The space between the solid lines represents the testing range, while the outer two parts represent the training range.
(a) SIREN (Electricity)
(b) FFN (Flectricity)
(c) WIRE (Electricity)
(d) NeRT (Flectricity)
SIREN (Traffic)
( FFN (Traffic)
WIRE (Traffie)
(h) NeRT (Traffie)
(i) SIREN (Caiso)
( FFN (Caiso)
(k) WIRE (Caiso)
(I) NeRT (Caiso)
Figure 19: Interpolation results on missing intervals after the last epoch. Interpolation results in Electricity (Figures 19 (a)- in Traffic (Figures (e)-()), and in Caiso (Figures 19 (i)-()). The space between the two solid lines represents the testing range, while the outer two parts represent training range.
26

---

Published as a conference paper at ICLR 2025
0000007
(a) SIREN (Electricity)
(b) FFN (Electricity)
(c) WIRE (Electricity)
(d) NeRT (Electricity)
(e) SIREN (Caiso)
(1) FFN (Caiso)
WIRE (Caiso)
(h) NeRT (Caiso)
SIREN (NP)
FFN (NP)
(k) WIRE (NP)
NeRT (NP)
Figure 20: Extrapolation results missing intervals at the lowest validation error checkpoint. Extrapolation results in Electricity (Figures 20 (a)-(d)), in Caiso (Figures 2 (c)-(h)), and in NP
(Figures 20 (i)- The right area after the solid vertical line is testing range.
(a) SIREN (Electricity)
(b) FFN (Electricity)
(c) WIRE (Electricity)
(d) NeRT (Electricity)
(e) SIREN (Caiso)
(1 FFN (Caiso)
WIRE (Caiso)
(h) NeRT (Caiso)
0O00000
andi an WHC 7006 new
m0x dest
7900 mow
1910
1986 ros
Jece 19t3 1950 1910 1912 NOS
SIREN (NP)
FFN (NP)
(k) WIRE (NP)
NeRT (NP)
Figure 21: Extrapolation results missing intervals after the last epoch. Extrapolation results in Electricity (Figures 21 (a)- in Caiso (Figures 21 (e)-h)),, and in NP (Figures 21 (i)-()). The right area after the solid vertical line is testing range.
27

---

Published as a conference paper at ICLR 2025
0.75
Tree
factor
Scale factor
1.4
True
factor
Scale factor
1.2
0.25
MMr 14 MM
1.0
-0.50
0.2
-1.00
0.0
2200
3350
3400
2300
3500
3700
2000
3900
1200
1300
1400
1500
1800
1700
1800
1990
(a) Electricity (Interpolation)
(b) Traffic (Interpolation)
Perlod factor
Scale factor
1.4
Parlod
Scale
0.5
PAA
0.0
VVY
-0.5
0.4
-1.0
0.2
5500
7000
7105
7200
7300
7400
7500
7600
8700
2600
5500
5000
9100
2200
9300
(c) Caiso (Interpolation)
(d) NP (Interpolation)
True
Peried factor
Scale factor
True
Period factor
Scale factor
2.5
2.0
0.6
1.9
0.4
0.5
0.2
0.0
(e) Electricity (Extrapolation)
(1) Traffic (Extrapolation)
1.5
1.4
True
Period factor
Scale factor
True
Peried
1.2
10000000
1.9
0.6
-0.5
0.4
-1.0
0.2
49120
49200
10300
<5100
49320
4060S
49700
49800
55100 89500
20622
59752
89800 69900
73000
70156
(g) Caiso (Extrapolation)
(h) NP (Extrapolation)
Figure 22: Periodic and Scale factors trained on periodic time series. Interpolation results (Figures 22 (a)-(d)), where the space between the two solid lines represents the testing range, while the outer two parts represent the training range, and extrapolation results (Figures 22 (e)-(h), where right area after the solid vertical line is testing range.
28

---

Published as a conference paper at ICLR 2025
J ADDITIONAL COMPARISON WITH TIME SERIES BASELINES
J.1 EXPERIMENTAL SETUPS
In this section, we conduct comprehensive analyses between our model and existing time series baselines to support the necessity of adapting INR specifically to time series data (cf. Appendix [B]. We use four datasets used in Section 4.2. and we have meticulously arranged the experimental setup in the subsequent manner for the purpose of this study. To be specific, we compare the forecasting performance by varying the input window size of (48, 96, 192) and the output window size n of (96, 192, 336, 720), following the overall experimental configuration of (Zeng et al. 2002).. For each data sample, we fix the total length 2,880 with train size of 1,440 and validation and test sizes of 720. To assess the model performance, we use mean-squared error (MSE) of the test range at the epoch where the best MSE on the validation range is achieved.
Baselines To evaluate the performance of NeRT, we compare with eight existing time series baselines. As representatives of traditional time series models, we use RNN and LSTM baselines. Additionally, we compare NeRT against models in the long-term time series forecasting domain, including Linear-based models, i.e., Linear, DLinear, and NLinear from Zeng al. (2022), , and
terssformer-basee baselines, i.c., Autoformer (Wu et al.] [202]), Informer (Zhou et al. 2021 and
FEDformer (Zhou et al. 2022). Furthermore, we employ Latent ODE (Rubanova al. 2019 and
Neural CDE (Kidger et al. 2020 as Neural ODE-base models.
Training Methodological Differences between NeRT and Other Baselines Existing time series baselines are highly sensitive to varying input and output window sizes (cf. L.2 of Appendix [B). so they need repeated experiments for each window size change. In contrast, since our NeRT operates independently of window size, we need to train NeRT only once for each dataset. During the single training, NeRT makes single set of predictions for all 720 points, and then calculate MSEs for all the output window sizes of 96, 192, 336, and 720, respectively.
J.2 EXPERIMENTAL RESULTS
We summarize experimental results of time series forecasting depending on the input/output window sizes in Tables 12 and 14 Note that while other time series baselines are trained individually for each combination of the input/output window sizes, NeRT employs only a single model for each dataset. As shown in the tables, NeRT consistently shows the best MSE regardless of the dataset and window size. On top of that, the results of the baselines are highly affected by the input/output window sizes (cf. Appendix B.I), making hard choose an appropriate window size setting. Figure 23 depicts how MSE changes as n varies, with m fixed to 96 and 192. In every case, NeRT shows the lowest MSE with the lowest slope, compared other baselines. Additionally, Figure 24 shows how the models predict n values given the input window size m on the Traffic dataset, where = 96 and m 48 in this setting.
Computational Cost In Table 155 we describe models' computational complexity in terms of time and memory (ef. L4 of Appendix B). We report the complexity for all window combinations of the baselines individually, whereas with the NeRT's capacity to provide results for all combinations at once using a single model, we report the training complexity of the single NeRT model. The results in Table 15 correspond to an input window size of 96, and each value represents the complexity required during the training of a single data sample. For NeRT, we record the average time/memory complexity needed to train a single data sample, with the values in parentheses indicating the total complexity required for training single model.
As shown in Table 15 the total memory complexity of NeRT is notably smaller than the memory complexity of all MINTraasformee/Neurll bDE-based models when training just one data sample. This reduction in complexity ranges from being 5.5 times smaller to as much as 1,730 times smaller and in certain instances, it is even smaller than those of Linear- NeRT outperforms Transformer/Neurall ODE-d models, but it is slower than Linear/RNN- based models. In summary, considering the forecasting results from Tables 12 [3] and 14] NeRT demonstrates significant forecasting performance improvements, approximately 2 to times better
models. For the time complexity,
29

---

Published as a conference paper at ICLR 2025
than existing time series models, while still maintaining sufficiently fast training time complexity and memory complexity similar to Linear- based models.
Informer
Latent ODE
NeRT
DLinear
RMN
Autotormer
FEDtormer
Neural CDE
0.10
0.10
0.01
96192 336
720
96192
336
720
96192 336
720
96192 336
720
(a) Electricity (m 96)
(b) Traffic (m 96)
(c) Caiso (m 96)
(d) (m 96)
0.10
0.10
0.01
192
96192
336
96192 336
96192 336
(c) Electricity (m 192)
(f) Traffic (m 192)
(g) Caiso (m 192)
(h) NP (m 192)
Figure 23: Comparisons with time series baselines for varying {96. 192, 336, 720} (out- put/prediction window size) with a fixed input window size m 96 ((a)-(d)) and m 192 ((e)-hh)).
Table 12: Comparision with time series baselines (m = 48)
Jnner-hased
RNN-hased
Transtomer-hisee
Neural ODF-baao
INR-bused
Dataset
Lincur
DLinear
NLineur
RNN
LSTM
Autoformer
Informer
FEDiomer
Latest ODE
Neural CDE
NeKT (Ouns)
0.0671
0.0485
0.0673
0.0174
Elecmicity
0/0747
0.0629
0.0351
0.0411
0.0144
0.0100
0.085
0.0186
336
0.0843
0.0689
0,0387
0.0444
0.0547
0.0520
0.0411
0.460
0.1500
0.0403
0.0196
720
0,1461
0.1326
0.0482
0.0890
0.1601
0.0741
0.100
00668
0.1565
0.0220
0.0196
96
0.0278
0,0240
0.0215
0.0191
0.0229
0.0395
00112
0.0900
0.0349
0.0192
0.0086
Trafiic
0.0275
0.0225
0.0202
0.0368
0.062
0,0288
0,0078
0.0001
720
0.0416
0.0418
0.0289
0.0297
0.0377
0.0652
0.0670
0.1199
0.1036
0.0320
0.0W7
96
0.0999
0.0843
0.0448
0.156
0.0671
0.881
0.917
00455
0.108
0.001
0.0236
Caiso
192
0.0972
0.0799
0.0450
0.0007
0.0647
0.0539
0.3322
0,410
0.1538
0.053
0.0362
336
0,1154
0.0927
0.0548
0.0658
0,0711
0.0683
0.0624
0.0022
0.1841
0.0573
0.0293
0.3257
96
0.1111
0.1085
0.0600
0.0897
0.0790
0.0784
0.703
0.1060
0.0370
NP
192
0.1007
0.0990
0.0795
0.0649
0.0814
0.0733
0.000
0.047
0.1522
0.0862
0.0409
336
0,1072
0.1085
0.0932
0.0006
0.0794
0.0860
0.0727
0.0499
0.1465
0.0699
0.0413
720
0.2223
0.2217
0.1414
0,1093
0.2534
0.1073
0.0981
0.0820
0.3414
0.1120
0.0478
30

---

Published as a conference paper at ICLR 2025
Table 13: Comparision with time series baselines (m 96)
Liner-hased
RNN-hised
Transfomer-hsed
Neural ODDF-basd
INR-tosed
Datnset
Linear
DLinear
NLineur
RNN
Autoformer
Informer
FEDiomer
Lateat
Neural
NetTT (Ouns)
96
0.0.74
0.0257
0,0214
0.0182
0.0496
0,0430
0.0150
0.0132
0,0468
0.0320
0.0174
Elecnicity
192
0,0433
0,0293
0.0369
0,406
0.0503
0.0533
0.620
00451
0,7223
0,0300
0.0186
336
0.0470
0,0329
0,0306
0.0439
0.0544
0.0476
0.0556
0,0900
0.1064
0.0295
0.0196
720
0.0853
0.0670
0.0376
0.0890
0.1601
0.0728
0.9177
0.5855
0.1726
0.0716
0.0196
0/0167
0.0108
0.0133
0.0100
0.0997
0.0228
0.0186
0.0086
Traffic
0:0148
0.0103
0.0120
0.0778
0.0191
0.0381
0,0293
0.0078
336
0.0166
0.0115
0.0134
0,0200
0.0223
0.0277
0.0375
0.404
0.0439
0.0202
0.0090
720
0,0280
0.0221
0.0196
0.0298
0.0377
0.0619
0.0871
0.0515
0.0792
0.0316
0.0097
96
0.049%
0.0343
0.0284
0.0566
0.0663
0.0569
0.1112
0.0511
0.0611
0.0478
0.0236
0.0487
0.0368
0.0274
0.0594
0.1086
0,0500
0.0515
0.0262
0:0557
0,0400
0.0346
0.0647
0,0700
0.0634
0.1268
0.1249
0.0293
720
0.1138
0,0983
0,0644
0.1273
0.2432
0.1279
0.1296
0.1413
0.2397
0.1086
0.0324
96
0,0767
0.0693
0,0464
0.0687
0.0807
0,1138
0.0787
00384
0.0003
0.0541
0.0370
NP
192
0.0879
0,0790
0.0626
0.0650
0.0828
0.0868
0.0961
0.0766
0.1152
0.0635
0.0409
336
0.0847
0,0757
0.0607
0.0604
0,0794
0.0895
0.0633
0.0625
0.1084
0.0646
0.0413
0,1188
0.1126
0,0806
0.1290
0.1056
0.1136
0.1424
0.0478
Table 14: Comparision with time series baselines (m 192)
RNN- bused
Transformrr-be based
ODE-
33R-bueed
Dataset
21
Linear
DLinear
NLinear
RNN
LSTM
Autoformer
Informer
FEDfomer
Latent ODE
Neural CDE
NeKT (Ouns)
96
0.0312
0,0217
0.0193
0.0374
0.0493
0.0481
0.0609
00453
0.348)
0.034)
0.0174
Electriiiy
192
0.0358
0,0268
0.0241
0.998
0.0499
0.0433
0.527
0.4455
0.0244
0.0403
0.0186
336
0,0370
0,0290
0.0281
0.0435
0,0542
0.0476
0.0266
0.0507
0.0634
0.0414
0.0196
720
0,0474
0,0327
0,0313
0.886
0,0711
00951
0,2029
0.014
0.0196
96
0.0114
0.0099
0.1005
0.0191
0.0228
0.0410
000300
00335
0.033
00197
0.0086
Traffic
192
0.0105
0.0096
0,0092
0.0178
0.0191
0.0272
0.0371
0.0400
0.0268
0.0192
0.0078
336
0,0108
0.0101
0.0096
0.0198
0.0222
0.0306
0.0360
0.105
0.0387
0.0204
0.0090
720
0.0180
0.0152
0.0141
0.0299
0,0377
0.0634
0.0756
0.0167
0.1163
0.0310
0.0997
0.0443
0.0848
0.0273
0,0550
0.0604
0.1342
00464
0.0951
0.0469
0.0236
0.0188
0,0378
0.0274
0.0597
0,0678
0.0551
0.1082
0.0828
0.0262
Caiso
336
0.0528
0,0428
0,0317
0.0638
0.0725
0.0577
0.1183
0.0592
0.0971
0.0553
0.0293
720
0.0642
0.0928
0.0923
0.1270
0.2451
0.1048
0.1533
0.1334
0.2253
D. 0.1077
0.0324
96
(30743
0,0670
0.0448
0.0672
0.0786
0.0976
0.000
0.0022
0.0852
0.0535
0.0370
192
0.0912
0.0855
0.0589
0.0661
0,0834
0.0905
0.0729
0.0675
0.1579
0.0645
0.0409
0.0995
0.0928
0.0606
0.0724
0.1261
0.0652
0.0413
720
0.0960
0.0883
0,0711
0.1283
0.2530
0.1214
0.1254
0.1029
0.2017
0.1079
0.0478
Table 15: Computational cost with m at 96. Each value is measured during training one data sample. For NeRT, since it is not trained for each window combination, we report the average cost required for training one sample, with total amount in the parentheses.
Lincar-busel
RNN-based
Tranformer- based
Neural ODE-hused
INR-bused
Complexity
Linear
Dlinear
NLinear
RNN
LSTM
Autoformer
Informer
FEDformer
Lateni ODB
Neural CDB
NeRT
96
0.0338
0.04449
0.353
0.0864
0.8951
0.9333
0.8525
9.3507
150144
9.947
Time (sec)
192
010329
0.0446
0.347
0.0836
0.0930
1.6733
0,7802
9.4909
17:2511
9.9699
D.2345
335
0.030
0.0443
0.338
0.0808
0.0930
1.2866
09100
10,7902
20.2
9.8612
(2.8151)
720
0.0022
0,02755
0.0266
0.0335
0,0387
0.945
0.700
7,7796
9.6910
5,2095
96
1.0757
L.1816
1.0911
54,2966
92.9467
9889066
30499854
2254,9459
24.2246
9.6597
Memory (MB)
192
1.1519
1,3109
1.1753
54,5799
93,0425
1292
679.2822
2252.90 23559
26.4951
9.6597
0.1318
336
1.2769
1,5300
1.3003
54,5508
93.2148
1768.7114
908.2749
2890.7788
29.9116
9.91668
(1.8820)
720
1.4126
1,8613
1.4992
46.4707
78.8451
2735.4766
1273.8196
3913.7681
33.2192
8,7642
31

---

Published as a conference paper at ICLR 2025
Linear
(b) DLinear
(c) NLincar
0.8
0.8
0.7
0.2
0.8
9.8
05.
0.4
03
022
0.1
0.1
20
340
320
146
(d) RNN
(c) LSTM
(f) Autoformer
0.4
033
33
0.2
02
0.
0.
0.0
3.0
0.6
20
g Informer
(h) FEDformer
(i) Latent ODE
Fads
(i) Neural CDE
(k) NeRT (ours)
Figure 24: Forecasting task on Traffic. We set m to 48 and n to 96. The left side of the solid line represents the input window, while the right side represents the output window.
32

---

Published as a conference paper at ICLR 2025
K ADDITIONAL COMPARISON WITH MODULATED INR ON UNSEEN SAMPLES
Table 16: Comparison with Modulated INRs
Modulated NeRT
Modulated NeRT (scale and period)
Dataset
Task
Modulated SIREN
Modulated FFN
(scalc)
Imputation
0.1840
0.0355
0,0322
0.0315
Electricity
Forecasting
0.1844
0.0188
0.0161
0.0149
Imputation
0.0457
0.0067
0.0009
0.0040
Traffic
Forecasting
0.0373
0.0116
0.0042
0.0091
Imputation
0.0306
0.0390
0.0375
0.0264
Caiso
Forecasting 0.0869
0.0907
0.0663
0.0629
Imputation 0.0361
0.0257
0.0224
0.0229
NP
Forecasting 0.0238
0.0449
0.0441
0.0166
K.1 LATENT MODULATION
Fundamentally, single INR model tends to overfit to single data sample, making it challenging to represent unseen data samples effectively. To overcome this limitation, recently developed modulation techniques involve i) sharing model parameters and ii) learning sample-specific parameters, enabling INR models capable introduced in Dupont et al. (2022). is one of the most effective training methods for INRs to infer unseen samples after learning multiple samples - we strictly follow this training method this subsection. It is a meta-learning based modulation approach that allows the representation of diverse data samples by adding an additional learnable bias to each shared MLP layer and for each sample, the biases in all the MLP layers changed at the end, all data samples can be somehow learned by the combination of the shared MLP parameters and the sample-specifi additional biases. By adopting this concept to all INR-based models used in the paper, which are SIREN, FFN, and NeRT, they are able to predict values of unseen samples.
representing various data samples. In particular, the latent modulation,
K.2 EXPERIMENTAL SETUPS
To ensure a fair comparison, NeRT, SIREN, and FFN employ the same latent modulation approach. The baselines, referred as modulated SIREN and modulated FFN, are SIREN and FFN models with latent modulation applied to all layers except the first and last. In the case of modulated NeRT, we propose two variants. Firstly, we apply latent modulation exclusively the scale decoder, denoted Modulated NeRT denoted Modulated NeRT (scale). Secondly. latent modulation is applied to both the scale decoder and the periodic decoder of vanilla NeRT, (scale and period).
The datasets used the experiments are the periodic time series datasets discussed in Section 4.2 and the experiments conducted in environment identical to that described in Appendíx [ The dimensionality of the modulation vector remains consistent at 256 throughout the training of all models. Additionally, testing is carried on unseen block of unseen samples that are not part the training process. Both imputation and forecasting tasks are simultaneously inferred within a single model.
K.3 EXPERIMENTAL RESULTS
All experimental results are summarized in Table T6 and it can be observed that modulated NeRTs outperform the modulated INR baselines significantly across all benchmark datasets. Particularly, for Traffic dataset, Modulated NeRT (scale) exhibits MSE values that approximately one-seventh the magnitude of the baseline for interpolation and half the magnitude for extrapolation. Consequently, NeRT shows its scalability unseen samples with commendable performance.
33

---

Published as a conference paper at ICLR 2025
L EXPERIMENTS ON LONG-TERM TIME SERIES
L.1 DETAILED EXPERIMENTAL SETUPS
Table 17: Hyperparameters of long-term time series
Smox
Drop ratio
fnit
inner
dim(v/)
dim(pr)
dim(hp)
dim(h.)
30%
5.0
1.0
50
30
200
10
ETThl
100
50%
5.0
3.0
30
30
100
50
70%
10.0
3.0
30
30
100
30
30%
5.0
1.0
10
30
200
10
ETTh2
100
50%
5.0
3.0
30
30
100
50
70%
10.0
3.0
30
30
30%
5.0
1.0
50
10
100
10
National Illness
50%
5.0
3.0
30
50
30
10
70%
10.0
3.0
10
10
10
10
For fair comparison, we share انن and inner and employ similar model sizes across the tested models. We note hyperparameter configurations used in long-term time series experiments in Table 17 In terms of the number of layers NeRT, we set L,, L and Lg to 2, and Lp to
L.2 ADDITIONAL EXPERIMENTAL RESULTS
Table 18: Full table on long-term time series. The best results are reported in boldface.
Drop ratio
Linear
Cubic
SIREN
FFN
NeRT
30%
0.0892
0.1268
0.45500000030
0.2522+:0.3992
0.0828++0.022
ETTh1
50%
0.1178
0.1662
0.2173+0.0216
0.3407:t00013
0.0911+0.0097
70%
0.1978
0.2902
0.2605
0.4256.100119
0.1257.10.005
30%
0.0407
0.0655
0. 0.101040.0075
0.1863++00011
0.0344::0.0020
ETTh2
50%
0.0473
0.0847
0.0023+000001
0.2351
0.0423:+0.0022
70%
0.0596
0.1250
0.0964100002
0.3178110:0660
0.0575::0.001
30%
0.0266
0.0248
0.3502++0.0200
0.1110+:0.0059
0.0239++000009
National Illness
50%
0.0567
0.0484
0.17164400376
0.23194000737
0.02911:0.0045
70%
0.0902
0.0876
0.356440.0202
0.4453+000042
0.0871+..0255
We report the full experimental results of Table 2 in Table 18 As shown in Table 18
NeRT shows the lowest MSE in every dataset, regardless of the drop ratio. For example, NeRT shows an MSE of 0. 1257 in ETTh with drop ratio of 70%, while baselines exhibit errors from 0.1978 in minimum to 0.4256 in maximum. Figures 25 26 and 27 show how models learn and represent the spatiotemporal coordinates of ETThl, ET TTTh2, and National Illness, respectively. In those figures, the top row ((a)(c)) distinguishes the training and the testing sets in the learned coordinate systems where the X-axis refers the temporal information and the Y-axis the spatial information. While training. models only see the white-colored coordinates, i.e., training samples, and then predict values in the black-colored coordinates. i.e., validating and testing samples. Other rows are the results the first 50 timestamps by each method in each dataset. Unlike other baselines, NeRT successfully learns the spatial coordinate systems to embed features and accurately represents the temporal pattern in each feature. Surprisingly, NeRT demonstrates remarkable predictions in extreme scenarios with a drop ratio of 70%. and it maintains its performance well compared to other baselines challenging situations, i.e., high drop ratios.
34

---

Published as a conference paper at ICLR 2025
Festire
Trrm
(a) Coordinate (70%)
Coordinate (50%)
(c) Coordinate (30%)
Fasture
Fasture
FactUn
0
(d) SIREN (70%)
(e) SIREN (50%)
(f) SIREN (30%)
Reatun
Time
Time
Time
(g) FFN (70%)
(h) FFN (50%)
FFN (30%)
Festure
Festure
50
50
0
10
40
(i NeRT (70%)
(k) NeRT (50%)
(I) NeRT (30%)
Raaure
Time
(m) Ground truth
Figure 25; Experimental results of long-term time series (ETThI). In (a)-(c), white (resp. black) cells mean training (resp. validating/testing) samples.
35

---

Published as a conference paper at ICLR 2025
Festire
Fature
Tirm
(a) Coordinate (70%)
Coordinate (50%)
(c) Coordinate (30%)
Fasturo
Fasture
(d) SIREN (70%)
(e) SIREN (50%)
(f) SIREN (30%)
Fesure
Reatune
Time
Time
Time
(g) FFN (70%)
(h) FFN (50%)
FFN (30%)
Festure
Festure
50
50
10
(i NeRT (70%)
(k) NeRT (50%)
(l) NeRT (30%)
Raaur
43
Time
(m) Ground truth
Figure 26: Experimental results of long-term time series (ETTh2). In (a)-(c), white (resp. black) cells mean training (resp. validating/testing) samples.
36

---

Published as a conference paper at ICLR 2025
Festire
Fature
(a) Coordinate (70%)
Coordinate (50%)
(c) Coordinate (30%)
Fasture
Fasture
0
(d) SIREN (70%)
(e) SIREN (50%)
(f) SIREN (30%)
Reature
Time
Time
Time
(g) FFN (70%)
(h) FFN (50%)
(i) FFN (30%)
Festure
50
Festure
53
50
50
10
(i) NeRT (70%)
(k) NeRT (50%)
(l) NeRT (30%)
Raaure
Time
(m) Ground truth
Figure 27: Experimental results of long-term time series (National Illness). In (a)-(c), white (resp.
black) cells mean training (resp. validating/testing) samples.
37

---

Published as a conference paper at ICLR 2025
M EXPERIMENTS ON COUPLED MASS-SPRING SYSTEM
1.00
1.00
Displacement (m)
0.75
Ground Iruth (x1)
Prediction (d)
Ground Truth (x2)
Prediction (x2)
Displacement (m)
0.75
Ground Iruth
Prediction [x1]
Prediction (x2)
Ground Truth
0.50
0.50
0.25
0.25
0.00
0.00
-0.25
-0.25
-0.50
0.50
-0.75
0.75
-1.00
11.00
2
4
6
8
10
10
Time (s)
Time (s)
(a) Extrapolation result of NeRT
(b) Extrapolation result of Sampled RNN
Figure 28: Coupled mass-spring system. Extrapolation results (Figures 28 (a)-(b)). The left side of the solid vertical line represents the training range, while the right side represents the testing range.
In this section, we present an additional experiment to evaluate NeRT's performance using a coupled mass-spring system. This system demonstrates the dynamics of two masses interacting strongly through springs, providing classical physics benchmark for studying oscillatory motion. For simple mass system, the form of the governing equation is follows:
m1 - =1 k2( F22) c1
(6)
m2 (2(222 x1) kz C232,
where æ and 22 represent the displacements of masses m and me from their equilibrium positions, t and V2 denote their accelerations, and d and it represent their velocities. The terms k1, k2 and k3 are the spring constants, and ej and co are the damping coefficients. Mass mi is connected via coupling spring with constant kz-
to a wall via a spring with constant k1. and to mass Similarly mass m2 is connected to another wall through spring with constant kz. The systems initialized with the following parameters: both masses m., are set to 1.0kg, the spring constants are 6 = 10.0N /m, K22 = 15.0N/m. and k3 = 10.0.V
//m, /m, and the damping coefficients are = 0.5kg/s s and e2 0.5kg/s. The initial displacements are z (0) 1.0m and (0) - 1.0mm, with initial velocities set to zero.
For this experiment, train NeRT using data from Os to t 3s and evaluate its ability to extrapolate the system dynamics from = 3s to t = 10s. As shown in Figure [8] the experimental results demonstrate that NeRT generalize effectively modeling the dynamics of the coupled system over extended time periods. To validate performance of NeRT, we compared it with Sampled RNN, a state-of-the-art RNN-based method. NeRT achieved lower MSE during extrapolation (0.0012) compared to Sampled RNN (0.2632), demonstrating stronger generalization beyond the training range.
38

---

Published as a conference paper at ICLR 2025
N EXPERIMENTS ON CHAOTIC DYNAMIC: LORENZ SYSTEM
MMM M. MMMM
M.M.
M_M
AL
M
MAW
Prediction
Truk
WWY
0-
Prediction Wirmon
True
WWW WWA
10
20
30
40
50
0
10
20
30
40
50
h2
NNM
AM
h2
MHM
WY
Wiil
www NON
WWT W/
10
20
30
40
50
10
20
40
50
2.5
2.5
WAll
0.0
0.0
10
20
30
40
50
10
20
30
40
50
(a) Extrapolation result of NeRT
(b) Extrapolation result of Sampled RNN
Figure 29: Lorenz system. Extrapolation result of NeRT (Figure 29 (a)) and extrapolation result of Sampled RNN(Figure 29 (b)). The left side of the solid vertical line represents the training range, while the right side represents the testing range.
To further evaluate the broader applicability of NeRT, we conducted additional experiments the chaotic Lorenz system. The Lorenz system, originally devised to model atmospheric convection in Lorenz (963), is governed by the following set of equations:
hi a (h2 h1)
lig = h ha)) - 1t2
hg h2 Bh3,
where a, p, are parameters that control the dynamics of the system. For our experiments, we set = 10, B 8/3, and = 28, which corresponds to the chaotic regime of the Lorenz system. Specifically, the dataset is generated by solving the initial value problem for t [0, 5] with time step of At = 0.01. Initial conditions are sampled randomly from a uniform distribution, hg Uniform (I- 20, 20) [0)5
For baseline comparison, we include the state-of-the-art RNN-based model, Sampled RNN (Bolager) let 20244, which has shown strong performance in modeling chaotic dynamic systems. We use the data from 0 to = 40s for training, and the data from = 40s to = 50s for testing. As shown in Figure 29 within the training region, NeRT successfully models the system's 's dynamics, demonstrating its ability to capture chaotic behavior over shorter time intervals. However, both NeRT and Sampled RNN, struggle with the extrapolation task for the Lorenz system.
The Lorenz system, as chaotic system, poses significant challenges in extracting periodicity. which highlights the limitations of both models. This dataset serves as valuable tool for identifying these limitations and emphasizes the need to extend the underlying framework to handle non-periodic signals. Expanding this methodology to model non-periodic systems remains a critical avenue for future work, offering potential for broader applicability to complex, real-world dynamics.
39

---

Published as a conference paper at ICLR 2025
O ADDITIONAL COMPARISON WITH GAUSSIAN PROCESS
1.5
59%
55%
1.0
1.0
0.5
0.0
0.0
0.5
-1.0
-10
-10
1.5
1.0
2.0
Time
Time
Time
(a) Interpolation
(b) Extrapolation
(c) Inter. & Extra.
Figure 30: Results of the Gaussian Process (GP) with rational quadratic kernel the damped oscillation ODE. Figures (a). (b) and (c) show the results for interpolation, extrapolation, and both interpolation and extrapolation, respectively.
1.5
10
1.0
1.0
0.5
0.0
0.0
0.0
-1.0
-10
-10
-1.5
1.6
1.8
1.0
2.0
Time
Time
Time
(a) Interpolation
(b) Extrapolation
(c) Inter. & Extra.
Figure 31: Results of the Gaussian Process (GP) with RBF kernel on the damped oscillation ODE. Figures (a), (b) and (c) show the results for interpolation, extrapolation, and both interpolation and extrapolation, respectively.
1.5
1.0
1.0
1.0
0.0
0.0
0.0
-1.0
-1.0
-1.0
1.0
1.2
1.4
1.6
1.8
2.0
1.0
1.4
1.6
1.8
2.0
1.0
1.6
1.8
2.0
Time
Time
Time
(a) Interpolation
(b) Extrapolation
(c) Inter. & Extra.
Figure 32: Results of the Gaussian Process (GP) with RBF+ White kernel on the damped oscillation ODE. Figures (a), (b) and (c) show the results for interpolation, extrapolation, and both interpolation and extrapolation, respectively.
1.0
1.0
1.0
0.0
0.0
0.0
-1.0
-1.0
1.0
1.0
1.2
1.4
1.6
1.8
2.0
1.0
1.4
1.6
1.8
2.0
1.0
12
1.6
1.8
20
Time
Time
Time
(a) Interpolation
(b) Extrapolation
(c) Inter. & Extra.
Figure 33: Results of the Gaussian Process (GP) with exponential sine squared kernels the damped oscillation ODE, Figures (a), (b) and (c) show the results for interpolation, extrapolation, and both interpolation and extrapolation, respectively.
40

---

Published as a conference paper at ICLR 2025
Toe
ON?
Pue
01
Tue
1.5
1.5
0.5
0.5
0.5
-0.
4.5 1.0
1.6 1.5
1.5
1.2
1.8
2.0
1.0
1.4
2.0
(a) Interpolation
(b) Extrapolation
(c) Inter. & Extra.
Figure 34: Results of the Conditional Neural Processes (CNP) (Garnelo et al., 2018 on the damped oscillation ODE. Figures (a), (b) and (c) show the results for interpolation, extrapolation, and both interpolation and extrapolation, respectively.
conduct additional experiments to compare the performance of Gaussian Process (GP) with different kernels and the Conditional Neural Process (CNP) Garnelo et al. (2018) the damped oscillation dataset. The GP experiments utilize four types kernels: i) rational quadratic, ii) RBF, iii) RBF+white, iv) and squared exponential periodic. Figures 30 31
32 and 33 illustrate the
interpolation and extrapolation results for each kernel. Across all cases, the GPs struggle to accurately capture the dynamics. Additionally, we evaluate the CNP model under the same experimental settings (please see Section I.I. and Appendix [E) and results are summarized in Figure 34 Similar to GP, CNP faces significant challenges in performing both interpolation and extrapolation.
P COMPARISON WITH BASELINE FOR IMPUTATION TASK
20
2.5
Thoe
Thoe
10
0.5
-0.5
-1.5
1.0
1.2
1.4
1.6
1.8
2.0
1.0
1.2
1.4
1.8
2.0
(a) Interpolation (Undamped)
(b) Interpolation (Damped)
Figure 35: Results of the Multi-time Attention Networks (mTAN) Shukla & Marlin 2022) on the damped/undamppdd oscillation ODEs. Extrapolation results (Figures 33 (3] (a)- The inside of the two solid lines represents the testing range (t E [1.2, 1.8], while the outside represents the training range (t [1.0, 1.2), (1.8,2.0) 2.0).
We conduct additional interpolation experiments using the multi-time Attention Networks (mTAN) (Shukla & Marlin 2021) undamped and damped oscillation datasets. As shown in Figure 35 mTAN struggles to perform interpolation under these conditions. All experimental settings consistent with those used for NeRT, ensuring fair comparison. Detailed experimental settings are provided in Appendix E
41


Reviews:
[{"review_id": "Z01WfNco2k", "reviewer": "Reviewer_Ta2X", "summary": "The paper introduces a novel framework called NeRT, designed to model periodic signals effectively. It separates periodic and scale factors in the data, enabling improved predictions for both interpolation and extrapolation tasks using a single model. The proposed method demonstrates its effectiveness through extensive experiments, which include learning periodic solutions for differential equations and performing time series tasks such as imputation (interpolation) and forecasting (extrapolation) with real-world datasets.", "questions": "I) In Section 3.1, why was the activation function chosen as ReLU in Eq. (1)? Are there any guidelines or heuristics for selecting appropriate activation functions for different datasets or tasks?\n\nII) Could the authors provide a more comprehensive and clearer comparison of their method with RNNs? Please see above (\"Weaknesses\").", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Originality**: The NeRT framework models periodic signals by effectively separating periodic and scale factors. This method creatively combines concepts from Fourier analysis and neural network architectures, leading to a more nuanced understanding of periodic behaviors in data.\n\n**Clarity**: The paper is almost well-written and clear. The authors explain the motivation for their approach, the architecture of NeRT, and the experimental results. Diagrams and examples support key points throughout.\n\n**Significance**: The significance of this work lies in its potential to advance the field of time series and sequential data analysis, particularly in applications involving periodic signals/patterns.", "weaknesses": "1. Although the proposed method can model/forecast periodic signals, I am not convinced it outperforms state-of-the-art RNNs [2] or other successful methods for analyzing and modeling time series data, such as Koopman-based methods [1]. These methods can model and forecast not only periodic time series but also more challenging time series data related to chaotic systems, see for instance [1,2,3,4,5]. \n- Could the authors compare more aspects, for example, **training time** and the **hyperparameters** (the number of hyperparameters and computationally consuming hyperparameter search) for their method and the state-of-the-art methods (e.g. shPLRNN in [2]) for simple benchmark systems like the van der Pol oscillator?\nAlso, please note that while MSE seems an appropriate evaluation measure for periodic time series, for non-periodic time series data, one should consider other measures like Geometrical Measure (D_stsp) and/or Temporal Measure (D_H) as mentioned in [2].\n\n2. While periodic signals are indeed significant in various applications, real-world datasets often exhibit a wide range of behaviors, including non-periodic, chaotic, and irregular patterns. Therefore, designing a method that exclusively targets periodic signals may limit its applicability in practical scenarios where the underlying dynamics of the time series are **unknown** or **complex**.\n- Could the authors test NeRT on non-periodic time series data, for example, on chaotic systems like the Lorenz or Rössler systems, to demonstrate the broader applicability of their method?\n\n3. The paper mentions comparisons with RNNs, but it does not specify which types of RNNs were used in the experiments (in addition to LSTM). In recent years, various RNNs with specialized training algorithms have been developed to capture, model, and forecast time series data. Thus, to ensure a fair and comprehensive evaluation of NeRT's performance, the authors should include comparisons with these advanced RNN architectures, which are designed to capture **different types of temporal dependencies** in time series data. By benchmarking against **state-of-the-art RNNs** (e.g. [1,2,5]), the authors could provide clearer context for the advantages of their approach, thereby strengthening the validity of their claims regarding NeRT's superiority in handling periodic signals.\n- Could the authors compare NeRT with some state-of-the-art RNNs, e.g., shPLRNN in [2], AL-RNNs in [5] or/and Sampled RNN in [1]?\n\n---------------------------------------------------------------------\n**References**\n\n[1] Bolager, Erik Lien, Ana Cukarska, Iryna Burak, Zahra Monfared, and Felix Dietrich, \"Gradient-free training of recurrent neural networks\", arXiv preprint arXiv:2410.23467 (2024).\n\n[2] Florian Hess, Zahra Monfared, Manuel Brenner, and Daniel Durstewitz, Generalized Teacher Forcing for Learning Chaotic Dynamics, In Proceedings of the 40th International Conference on Machine Learning. PMLR, (2023). \n\n[3] Georgia Koppe, Hazem Toutounji, Peter Kirsch, Stefanie Lis, and Daniel Durstewitz, Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fmri, PLoS computational biology, 15(8):e1007263, (2019).\n\n[4] Jonas Mikhaeil, Zahra Monfared, Daniel Durstewitz, On the difficulty of learning chaotic dynamics with RNNs, Advances in Neural Information Processing Systems, 35:11297-11312, (2022).\n\n[5] Brenner, Manuel, Christoph Jürgen Hemmer, Zahra Monfared, Daniel Durstewitz, Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstruction, NeurIPS 2024, arXiv preprint arXiv:2410.14240 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c7GzBwJ90c", "reviewer": "Reviewer_NJRt", "summary": "The paper presents an approach to improving the performance of coordinate-based multi-layer perceptrons (MLPs), also known as implicit neural representations (INRs), particularly in scenarios where the underlying signals exhibit periodic properties. The authors identify limitations in existing INR architectures, such as overfitting and poor generalization beyond the training region, which lead to suboptimal extrapolation performance, especially for sequential data like time series. To address these challenges, the paper proposes a new network architecture that enhances the ability of INRs to capture periodic patterns. The proposed approach involves two separate internal networks to learn the periodic and scale components of an observed signal.  The proposed algorithm requires only a single trained model to perform both interpolation (imputation) and extrapolation (forecasting). The proposed method demonstrates superior performance over state-of-the-art (SoTA) baselines in interpolation and extrapolation tasks across several synthetic and real-world benchmark datasets.", "questions": "Listed in the weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel architecture for implicit neural representation, consisting of two separate internal networks to learn the periodic and scale components of an observed signal. \n- The proposed approach enables forecasting and imputation using a single trained model. \n- Experiments show that the framework achieves better performance than baselines across multiple tasks and datasets.\n- The paper is well written and easy to follow.", "weaknesses": "- Could the authors provide insights into why an INR-based approach excels in forecasting and imputation of time series compared to RNN-style approaches that use time series sequences as input? Is the superior performance primarily due to the ability of the proposed method to better capture periodicity present in the datasets?\n- The paper currently lacks comparisons with state-of-the-art approaches for the imputation/interpolation task.\n- While the paper claims that the proposed approach can model irregularly-sampled time series, there are no experiments on real-world datasets to substantiate this claim. I would like the authors to compare their method with state-of-the-art approaches [1,2] for irregular time series on datasets such as MIMIC-III or PhysioNet to support their argument.\n\n### References\n1. Multi-Time Attention Networks for Irregularly Sampled Time Series, International Conference on Learning Representations, 2021.\n2. Latent ordinary differential equations for irregularly-sampled time series, Neural Information Processing Systems, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ar2q3NZoPX", "reviewer": "Reviewer_DxCf", "summary": "This paper introduces a simple but effective strategy to model periodic time-series data. The main contribution of this paper incorporates a Fourier mapping layer to encode the scale and periodic factors and reconstructs the signal via (scale * periodic signals). In the experiments, this work demonstrates robust and good performances on all datasets including synthetic data and real-world atmospheric data.", "questions": "Even though the paper is very well-written, the reviewer would still like to ask the following clarifying questions during the rebuttal phase. The reviewer is very likely to change the final overall score. \n\n1. Could the authors define what NeRT stands for? The reviewer tried to find a clear definition but was unable to.\n\n2. The authors cited Fan et al. (2020) but did not include their method as a baseline. What might be the reason for this?\n\n3. Could the authors fix the order/placement of the figures? It can be confusing to match the figures with their corresponding sections.\n\n4. The reviewer noticed that the authors mainly conducted experiments on 1D time-series data. Are there any limitations preventing NeRT from being applied to 2D time-series data (e.g., coupled systems)?\n\n5. Can NeRT be generalized to signals with mixed frequencies? In this case, the signals may combine low-, mid-, and high-frequency components.\n\n6. Although methods like RNN and LSTM may require a window, they could be more effective for data with shifting periods (where spikes do not occur at fixed times). How might NeRT adapt to such data? The reviewer is concerned that NeRT may be overly biased toward signals with exact periods, which could be a significant limitation of this approach.\n\n7. (minor suggestion) It might be helpful to further elaborate on Sec. 3.1 when defining the c_i^t and c_i^f. A better wording here could help the audiences to understand the framework even better.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is very well-written with extensive experimental study to demonstrate the proposed method (NeRT). The method is simple to understand and clear to interpret. The Fourier mapping layer can be very helpful to model this kind of periodic behavior from time-series. The experimental section is also very strong, which outperforms all baseline methods. The detailed visualization and ablation study further show how NeRT is a very effective strategy for complex signals with periodic behaviors.", "weaknesses": "The weaknesses of this paper are also clear which is mainly coming from the fact that NeRT could only potentially able to model periodic signals. The Fourier embedding/feature may be a huge constraint since the signal must have a fixed period. Yet this is understandable since there are many natural phenomena that has fixed frequency (e.g. the global temperature, traffic). \n\nThere are also some issues in terms of presentation that may potentially need to be fixed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ByatOa8hYz", "reviewer": "Reviewer_CBC1", "summary": "The authors propose to approximate periodic signals using a carefully designed encoder + decoder architecture which can be interpreted as learning a decomposition of the input signal into its frequency and scale components. The authors demonstrate their approach provides improved performance on some INR and time-series forecasting benchmarks.", "questions": "- Rather than replacing traditional approaches for time-series forecasting, can you see your approach being used in combination with autoregressive style approaches?\n\n- How does the shift invariance of your approach impact forecasting for time-series problems?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed architecture could be useful in a number of domains where the underlying data is known to exhibit strong periodic structure.\n- The proposed architecture is mathematically sound\n- The proposed architecture performs well on the numerical studies chosen by the authors demonstrating that the proposed architecture is a useful addition.", "weaknesses": "- While the proposed architecture is a useful contribution, I struggled to understand the specific challenges your approach is trying to address as opposed to classical methods like GP regression with periodic kernels, compressive sensing of periodic signals or even older works for learning periodic signals with neural networks [1]. It would be helpful to be specific about the application domains you are interested in and why classical approaches are not fit for these tasks.\n\n- I think the discussion on how your approach can be used for time-series forecasting could do with some refinement. Typical approaches for time-series forecasting choose to learn both a trend / seasonality component + an autoregressive component because we assume many time-series forecasting problems contain both long term trends + seasonalities and autoregressive dynamics (such as local momentum and mean reversion). I think it would be helpful to be more explicit about the theoretical limitations of using your approach over standard autoregressive models and their variants in this setting.\n\n[1] Ruiz, A., D. H. Owens, and S. Townley. \"Learning periodic signals with recurrent neural networks.\" (1996): 1131-1136.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)