Jumpy Recurrent Neural Networks
Authors: Samuel James Greydanus
Stefan Lee
Alan Fern
Venue:  (2021)
Keywords: RNNs
temporal abstraction
planning
intuitive physics

Abstract: Recurrent neural networks (RNNs) can learn complex, long-range structure in time series data simply by predicting one point at a time. Because of this ability, they have enjoyed widespread adoption in commercial and academic contexts. Yet RNNs have a fundamental limitation: they represent time as a series of discrete, uniform time steps. As a result, they force a tradeoff between temporal resolution and the computational expense of predicting far into the future. To resolve this tension, we propose a Jumpy RNN model which does not predict state transitions over uniform intervals of time. Instead, it predicts a sequence of linear dynamics functions in latent space and intervals of time over which their predictions can be expected to be accurate. This structure enables our model to jump over long time intervals while retaining the ability to produce fine-grained or continuous-time predictions when necessary. In simple physics simulations, our model can skip over long spans of predictable motion and focus on key events such as collisions between two balls. On a set of physics tasks including coordinate and pixel observations of a small-scale billiards environment, our model matches the performance of a baseline RNN while using a fifth of the compute. On a real-world weather forecasting dataset, it makes more accurate predictions while using fewer sampling steps. When used for model-based planning, our method matches a baseline RNN while using half the compute.

Under review as a conference paper at ICLR 2021
JUMPY RECURRENT NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Recurrent neural networks (RNNs) can learn complex, long-range structure in time series data simply by predicting point time. Because of this ability, they have enjoyed widespread adoption in commercial and academic contexts. Yet RNNs have fundamental limitation: they represent time series of discrete, uniform time steps. As a result, they force a tradeoff' between temporal resolution and the computational expense of predicting far into the future. To resolve this tension, we propose Jumpy RNN model which does not predict state transitions over uniform intervals of time. Instead, it predicts sequence of linear dynamics functions in latent space and intervals of time over which their predictions can be expected be accurate. This structure enables our model to jump over long time intervals while retaining the ability to produce fine-graine or continuous-time predictions when necessary. In simple physics simulations, our model can skip over long spans of predictable motion and focus on key events such collisions between two balls. On a set of physics tasks including coordinate and pixel observations of small-scale billiards environment, our model matches the performance of a baseline RNN while using a fifth of the compute. On a real-world weather forecasting dataset, it makes more accurate predictions while using fewer sampling steps. When used for model-based planning, our method matches a baseline RNN while using half the compute.
INTRODUCTION
is said that change happens slowly and then all at once. Billiards balls move across a table before colliding and changing trajectories; water molecules cool slowly and then undergo rapid phase transition into ice; and economic systems enjoy periods of stability interspersed with abrupt market downturns. That is to say, many time series exhibit periods of relatively homogeneous change divided by important events. Despite this, recurrent neural networks (RNNs), popular for time series modeling, treat time in uniform intervals potentially wasting prediction resources long intervals of relatively constant change.
One reason for this that standard RNNs are sequence models without an explicit notion of time. Instead, the amount of time represented by a single RNN update is implicitly set by the training data. For example, a model trained on sequences of daily average temperatures has implicit time step of a day. For a fixed computational budget, this introduces a trade-off between fidelity and temporal range. A model trained resolution of one time step per minute would require over 10K iterations to make prediction for one week in the future. At the other end of the spectrum, one-week resolution model could achieve this in single step but could not provide information about the intervening days. As such, selecting point on this spectrum is troublesome design decision.
In this work, we present Jumpy RNNs, a simple recurrent architecture that takes update steps at variable, data-dependent time-scales while being able to provide dense predictions at intervening points. The core innovation is to define the hidden state as a continuous, piece-wise linear function of time. Specifically, each Jumpy RNN step predicts not only a hidden state h4, but also hidden velocity h; and a span of time A over which the linear latent dynamics h(t)) h4 ha(t i) should be applied. Our model then jumps forward in time by A before updating again. Any intermediate time step can be produced by decoding the corresponding hidden state h(t).
During training, our model learns to use these functions to span the non- con-uniform time durations between key events, where key events emerge as time points where linear latent extrapolation is

---

Under review as a conference paper at ICLR 2021
True Dynamics
Baseline RNN
Prediction
Jumpy RNN
Prediction
A
A
111
1.00
FC
1.00
1.00
A
0.75
0.25
0.50
2 9.50
0.25
0.25
GRU
6.25
0.00
FC
0.00
10
Figure 1: Predicting the dynamics of two billiards balls (left) using bascline RNN cell (center) and a Jumpy RNN cell (right). Whereas the baseline model produces a hidden state h, at each time step, jumpy model predicts continuous- tittimmmuu---eeeeeeeee hidden state, over predicted interval A,. This allows to skip over long spans of predictable motion and focus on key events such as collisions.
ineffective. In Figure I for example, we see that our model updates at the collision points between the two balls and the walls. During time spans when the balls are undergoing constant motion, our model does not perform cell updates. In contrast, a standard RNN must tick uniformly through time.
We demonstrate our proposed model in several physical dynamics prediction tasks. We show Jumpy RNNs achieve comparable performance to the baseline while being between three and twenty times more efficient to sample. This includes settings with non-linear pixel-based observations. Further, we show that our model outperforms RNNs with any fixed step length, showing the importance of data-dependent step sizes. Finally, we demonstrate that a learned Jumpy RNN dynamics model can be leveraged as an efficient forward predictor in planning domain. Our key contributions are to:
Identify trade-off between temporal resolution and the computational expense of RNNs, Propose Jumpy RNNs, which make jumpy predictions and interpolate between them, Show empirically that Jumpy RNNs are efficient and effective jumpy time series prediction.
2 JUMPY RECURRENT NEURAL NETWORKS
Consider a continuous-time function x(() sampled at uniform time steps to form the sequence Tp. .1....,, Pr. We study the problem of generative modeling, where given an initial set of observa- tions, the goal is to auto-regressively predict a likely continuation.
Standard RNN. The per-step RNN computation during auto-regressive sequence generation is:
he RNNCell( h ((+:) ), f.-1 f(he)
where (-)) and /(-) are observation encoders and decoders respectively, and hy is the hidden state. when the dynamics of xtt) are simple (or constant) for long stretches of time. One way to linearly reduce the computation required to predict into the future is to increase the time-span A between RNN ticks. Standard RNNs, however, then lose the ability to predict at clock times in between the time steps. This introduces a trade-off between predictive resolution and the computational cost of predicting far into the future.
This computation is performed each time-step
2.1 JUMPY RNN ARCHITECTURE
Continuous Hidden Dynamics with Constant Jumps. Our first step toward resolving the trade-off upgrade the standard RNN that can learn to linearly interpolate a continuous-time hidden state htt) between updates. Let A be the time between RNN ticks such that RNN tick i (starting at i 0) corresponds to continuous time point 7; iA. For update i, the RNN predicts both a hidden state h; and hidden velocity h; that describes how the hidden state h(t) evolves over the time interval [T. + A). Specifically, the operation of this linear-dynamic, RNN with constant jump is given by:
h, hi
RNNCel ( [ML-L. h h @ (á(TT)
h(t) h4 h (t-(tt)))) Tr) for t [..
(2)
a(t) f(h(t))
(3)
where [] denotes concatenation and fttt) is the continuous time output prediction that can be immediately produced on demand for any time in between ticks. Under this model, the hidden state
2

---

Under review as a conference paper at ICLR 2021
x(t)
R1
lb2
R4
lts
h6
Agg
hg
As
A11
112
RNN
Jumpy RNN
het
+ halt -d,)
ha(t d,
42)
Mg thatt
A2
Ag
Figure 2: Our Jumpy RNN defines a continuous-time, piece-wiss wise linear hidden state.
evolves as a piece-wise linear function of time, with transitions between linear functions occurring RNN updates. This linearity constraint provides strong and natural inductive bias and allows for fast interpolation. Importantly, the hidden-state linearity does not translate to lincarity in output space, the learned decoder f-)) can be arbitrarily complex non-linear function.
While the above model can interpolate between time steps, it is still constrained by constant jump width A. Manually selecting A difficult since the ideal value depends on the complexity of a<t) and the encoder and decoder capacity. Further, rrtt may vary in complexity over time such that single jump width insufficient.
Jumpy RNNs. The Jumpy RNN upgrades the above model to dynamically adjust A throughout the sequence as shown in Figure 2
Specifically, we now predict a duration A; in addition to linear which the corresponding linear latent dynamics accurately approximate xx).. RNN ticks now occur at variable points in time that RNN step occurs at time point T; Σj Aj. Now our update is:
latent dynamics [hi h4 at each RNN step. This duration encodes the time span
-i-1
hig h4 RNNCell ( [[L-I, h Ai-1Ai1)) (i(T.)
A: LeakyReLU(Whh +1
(5)
((t) h, + (t- for t [Ts, A.]
where again (tt) - f(h(t)). In practice, this approach can implemented top of any existing RNN variant, for example, GRUs. Training the jumpy model, however, introduces new complications since the optimal dynamic steps A are unknown and interact with the rest of the hidden state. This partly addressed by the inductive bias of Eq. 5 which encourages model to always jump forward time by least one time step. The next section details our approach to effective training.
2.2 TRAINING JUMPY RNNS
Even though our model is defined over continuous time, the training data for time-series models typically consists of observation sequences of the form Fy. T1, .2.... ........ TT. For the remainder we will assume that there is one time unit between each sample, i.e. = =()). We now describe how to train Jumpy RNNs in this setting by constructing loss function specifically suited to our jumpy model.
Supervising Predictions. Supervising for prediction accuracy simply requires evaluating the model at each discrete time step and applying the loss: L. Σ ((x,, (t)), where ((-, is loss between predicted and actual values. In concept, this loss alone may be sufficient for optimizing accuracy, however, does encourage the model take large jumps (predict large A,) when possible. In fact, it may encourage small jumps as an easier optimization path. Thus, below we augment the loss to encourage jumpiness.
Supervising Dynamic Jumps. We interpret the the jump width A; as the duration that the linear dynamics can accurately approximate x(t). To formalize this, we say training point x, accurately approximated if é(1,, a(t)) < where is hyperparameter. Ideally, we would like the Jumpy RNN produce the maximum jump width that maintains this constraint.
More formally, given tixed network weights the optimal jump width A at step is the solution to:
max A; A,21
s.t.
Exx, k(t) <e v {x, E [,,,,, A.]]
3

---

Under review as a conference paper at ICLR 2021
which seeks the largest jump that does not violate the loss threshold for any intervening observation. This can be solved using a simple forward line search that starts at T2 and moves forward through the samples until our model's prediction loss under the current dynamics exceeds e. If this occurs the first step (implying no time passes between ticks), we set A to 1. In Appendix 1 we show this can be efficiently vectorized for entire batch as part of a single pass through the sequence.
We compute the optimal A on-the-fly during training and jump forward accordingly - effectively ignoring the predicted A; as in standard teacher-forcing style training. To update the model predict the correct A. we augment the prediction loss L with jump size loss La Σ, ||a - A At inference, the predicted A is used to determine jump length.
Interpreting Error Threshold e. The error threshold e is a key hyperparameter introduced in our approach. Examining Equation 7 e acts as trade-off between jumpiness and approximation error. In the extreme of setting O. the model predictions will likely never be below c, A; will remain at 1, and our model will reduce back standard RNN that updates at each time step. At the other extreme, choosing transition. In practice, we found that setting e to the final training loss of the baseline model yields 3-10x jumpy models with the same test error as the baseline model.
forces our model to predict the entire time series with single linear latent
3 EXPERIMENTS
In this section, we evaluate our model on a suite of tasks including both coordinate and video representations of a small-scale billiards simulation. Our model attains test errors and autoregressive sample errors comparable to that of the baseline while doing so in 3-20x fewer steps. Lastly, we show how our model can be leveraged for more efficient model-based planning.
Implementation Details. We use the same architecture and training regime across all experiments unless otherwise specified. We use a GRU cell with 128 hidden units and our encoder and decoder networks are four-layer residual MLPs with 128 hidden units and ReLU activations. We train model with ADAM for 10, 000 steps with batch size of 256, a learning rate of 1 x 10 and a decay schedule of 0.9 every 1000 steps. We used mean square error loss function to train both teacher forcing predictions and A predictions, and scaled the latter by x 10-5 relative to the former. For model selection, used early stopping.
Jump Bootstrapping, The initial transition to jumping more than one step is challenging in that requires latent dynamics extrapolate essentially without supervision. To bootstrap the process, we force a jump even when error is greater than in 1% of cases. This leads to more stable results; for more discussion see Appendix A.2
3.1 MODELING CONSTANT MOTION
To begin, we consider simple settings that describe constant motion of a moving object. These settings demonstrate the limitations of standard RNNs, our model's 's ability to translate linear latent dynamics to non-linear motion, and the plotting strategies will use for visualizing all models.
Lines. First, we consider the simple task of modeling linear motion. We generate 10,000 training trajectories corresponding to linear motion along one dimension in (x, 3) Cartesian coordinates. Specifically, { ,o,,,.., (I20, y20)} where (0, 1). Plotted out, this corresponds a flat, straight line starting our Jumpy RNN model quickly learns to jump to the end of the sequence (mean AS of 20) while the standard RNN operates at fixed rate. We trained on 90% of the data and evaluated on the remainder and both models trivially achieved very low error (10-8). See the Appendix A.4 for samples.
and y/+ c where e is a random constant in random height. Unsurprisingly,
Circles. Our second task also involves modeling constant motion; but in this case in terms of constant tangential velocity while moving in circle. While highly non-linear in the Cartesian observation space, the motion is linear in polar coordinates. This experiment examines our model's ability to encode non-linear observations with linear latent dynamics effectively learning Cartesian-to-Pin Polar conversion. As before, our dataset contained 10, 000 examples, this time with 25 time steps each. Each sequence started from random angle in (0, 2 and a random radius in (1, We fixed the tangential velocity to a constant value across all trajectories based on the intuition of a particle undergoing constant velocity along 1D manifolds of various curvatures.
4

---

Under review as a conference paper at ICLR 2021
Baseline RNN
jumpy RNN
Mean
Jumpiness vs. Error
10
Task 2: Circles
yareo Bav
20
Jupp ball
15-
Prediction error
10-1
5
10t1
5000
10000
Train step
Mean
Figure 3: A comparison between a Jumpy RNN and baseline RNN trained the circles task. Columns and show samples from the models. The Jumpy model summarizes almost the entire tra- jectory with a single step. Columns 3 shows how jumpiness increases throughout training. Column shows that the jumpy model is able to produce autoregressive samples of comparable quality to the baseline while using twenty times fewer cell updates to do so.
As in the lines task, our model learned to jump over significant spans of time (mean A; of 17). Figure 3 provides our primary visualization of these results. The first two columns show representative samples from standard RNN (left) and our Jumpy RNN (right). Dots correspond to RNN updates with black dots corresponding to priming observations. Recall that our model can make a prediction at any time point between updates. We plot these intermediate evaluations with dashed line and they follow the ground truth trajectory closely. The third column shows the average optimal jump width A during training. As we can see, it rises over the course of training as the model learns.
The fourth column presents a key summary plot of our experiment - a comparison of mean squared error vs. efficiency terms of jumpiness. In this chart, further lower-right is better - corresponding to low error with large jumps. In red, we show the progression of error for the standard RNN over the course of training (light to dark). Likewise, we show our model's in blue. Unlike the standard RNN which is at a fixed A (and thus a vertical line in this plot), our approach changes both in error and jumpiness. We see that our approach significantly jumpier while maintaining a similar error rate.
The gray curve represents the performance of the jumpy model when forced take constant jumps of size A. In this task, the jumpy model converges to a point only slightly beyond this curve, indicating limited utility in dynamic A, prediction compared to the optimal fixed time step. This is perhaps unsurprising in this case because the dynamics are uniform throughout each se- quence. However, our model still was able to automat- ically identify this point without manual search. As we show in the following section, dynamically predicting A, can be impactful in more complex settings.
Mean
jangel
J.mpy?
20.0
jarpy223
17.5
paspy255
janget
15.0
12.5
10.0
7.5
0
5000
10000
15000
Train step
Finally, we vary the dimensionality of our model's hid- den state and retrain. In figure 4 we find positive relationship between model capacity and jumpiness.
Figure 4: Jumpiness increases as we in- crease model capacity (hidden state size). We observed this effect across all datasets and experiments.
3.2 MODELING INTERACTION DYNAMICS
To examine more complex dynamics, we consider a billiards-likk simulation in which two balls are bouncing around within a walled enclosure. We examine this setting for 1- and 2-dimensional versions from both coordinate and image-based inputs. Ball trajectories in this setting exhibit periods of essentially linear motion punctuated by collisions. As such, we expect our Jumpy RNN model to adaptively adjust its step size accordingly.
Datasets. In each setting, we generate 10, 000 trajectories of 45 time steps each. The environment lacks friction, collisions are perfectly elastic, and walls placed along 0 and 1 for each coordinate. For coordinate settings, we record trajectories as x, y coordinates for each ball (or just y in the ID case). For image settings, render 28 28 grayscale images depicting top-down view of the balls.
Billiards From Coordinates. The first two rows of Figure 5 visualize the result of ID and 2D billiards respectively. To visualize samples in the 1D case, treat time as the x-axis. For 2D,
5

---

Under review as a conference paper at ICLR 2021
Baseline RMN
Jumpy ANN
Mear
Jumpiness vs. Error
Task 3: Billiards
1.0
1.0
Juap Sect
Prediction error (MSL))
> 0.6
>. 0.6
10"
2-
0.2
0.2
10:2
10
20
20
10000 20000 30000
Train step
Baselino ANN
Jumay RNN
Mean At
Jampiness vs. Error
10
Task Billiards2D
1.0
bil
Prediction ertor (MSE)
0.8
0.8
4-
0.6
0.6
3-
3 1x000
0.2
x xx10-2
0.0
0.0
0-
0.00
0.25
0,50
0.75 1.00
0.00
0.25
0.50
0.75
1.00
20000 40000
Train step
Mean At
Daseline RNN
Jumpy RNN
Mean At
Jumpiness vs. Error
Task PixBilliards11
Prediction eetoar (MSF)
-003
0
20000
40000
1
Mean At 2
Train
Task PixBilliarddDD
jumpiness Error
Baseline
Jumpy
True path
Prediction (MSE)
Junpy
4
3-
6 1110%
2-
x000
3x 10-
20000
Train stop
Mean de
Figure 5: Analyzing a jumpy RNN and baseline RNN trained on four billiards tasks. Columns 1 and illustrate qualitatively that the jumpy model's predictions were at least good as the bascline's. Columns 3 and 4 illustrate that our jumpy model's jumpiness varied with data modality (less jumpy behavior the more difficult pixels task), but that it matched the baseline model's performance while being more efficient to sample from. Forcing the jumpy model to use constant A gives much worse performance; indicating the learned adaptive stepping skips over the most predictable time spans and concentrates on the less predictable ones, like collisions. Note: Supplementary Materials also contains videos for comparing model predictions.
fade from light to dark to indicate time. In samples from both cases, we can see the model learns to concentrate cell updates near collisions. However, we do see instances where collisions are skipped entirely and filled in with linear latent interpolation alone as in the ID example. The Jumpiness vs. Error plots (last columns) also reflects the impact of this adaptive jumping. Our approach is able to significantly improve in both accuracy and jumpiness compared to using fixed step size (darkest blue triangle vs. gray curve). For ID. the baseline model achieves lower error but requires 6x the compute. In the 2D case, our model outperforms the bascline while executing 4x fewer RNN updates.
Billiards From Pixels. Learning from pixels directly is challenging setting. While the underlying motion of the balls is the same before, the way pixels change the balls move is extremely non-linear turning on or off abruptly. For these experiments, we increase the hidden layer size to 512 units and train for 50, 000 steps. The third and forth rows of Figure show result summaries for ID and 2D pixel- billiards. In the ID setting, we find that both the baseline and our model

---

Under review as a conference paper at ICLR 2021
Train MSE
Test MSE
Sample MSE
Mcan A
Task
MSE Scale
RNN
Jumpy
RNN
Jumpy
RNN
Jumpy
RNN
Jumpy
1: Lines
x10
1.08
39.9
1.77
41.2
13.9
21.9
20
2: Circles
=10-
0.03
3.66
0.03
3.64
3.98
5.25
17
3: Billiards1D
x 110-5
2.40
2.86
2.44
2.93
1.77
4.37
5.1
4: Billiards2D
x10
2.06
3.55
3.09
3.73
2100
1760
4.5
5: PixBillID
x10
1.26
1.49
1.56
1.74
2.37
2.61
4.1
6: PixBil12D
x10
1.31
3.00
2.08
3.90
64.5
37.1
3.2
7: Weather
x10
1.53 / 1.61
1.64
1.29 / 1.43
1.24
21.2 / 13.8
10.8
2.22
Table I: Summary of train, test, and auto-regressive MSE and mean jump length for all settings.
There are two RNN baselines listed in the last row, corresponding to models with A; fixed to 1 and 2 respectively.
achieve similar error, but our model does so with 4x fewer steps. In the 2D setting however, the baseline RNN yields a 50% higher error rate and generates unrealistic trajectories where balls bend along their trajectory like the sample shown. Reviewing training curves, we noted that the baseline RNN overfits more severely fails to generalize. One hypothesis for our model's success is that the bias towards linear latent dynamics acts as regularizer in this setting.
Quantitative Summary. All experimental settings are summarized with dataset level metries at convergence in Table We find our Jumpy RNN model performs at near the same sample error rate while making to 20 times fewer RNN steps. Note that absolute scale error terms varies by orders of magnitude across tasks and is specified only in the MSE Scale column for compactness.
3.3 THE JENA WEATHER DATASET
Having determined that the Jumpy RNN could learn useful behaviors on synthetic dynamics tasks of considerable complexity, we were also interested to see how it would perform on real-world data. With this in mind, trained it the Jena weather timeseries dataset recorded by the Max Planck Institute for Biogeochemistry (Kolle) 2016). The dataset consists of 14 different features such as air temperature, atmospheric pressure, and humidity. They were collected every ten minutes between the years of 2009 and 2016. This dataset is common baseline for sequence prediction and forecasting, being used in Deep Learning with Python and the TensorFlow time series forecasting tutorial (Abadi al. 2015 Francois 2017).
We subsampled the dataset in order to obtain observations every 30 minutes and then divided into sequences of 80 time steps each. Additional preprocessing steps followed previous work and can be found in Appendix AA.3 In addition to training our Jumpy model and a baseline RNN model, we also trained another baseline RNN with fixed A, set to 2. Like the gray lines that correspond to fixed A; in Figure 5 this baseline can be used to infer the advantage of adaptive jumpiness over simply subsampling dataset at different rate.
We found that our model improved over both baselines terms of test error and autoregressive sampling error. The difference in sampling error most significant, nearly halving that of the baseline model. Qualitative results the Appendix support this finding, with most of the Jumpy RNN predictions being being somewhat better than those of the baseline models. We suspect that part of this improvement in performance comes from the fact that our model performs RNN cell updates less often, and thus does well as sequences increase in length. We also suspect that our model's bias towards learning locally-linear dynamics acted as regularizer and reduced overfitting to local "inconsequential chaos" (Neitz et al., 2018)
3.4 MODEL-BASED PLANNING VIA
Our last experiment aimed to leverage the dynamics models we trained in the previous section to measure the value of jumpiness for model- -baed planning. Specifically, we considered a planning task in the 1D Billiards setting where agent must set the initial conditions (position and direction) of one ball (ball A) such that the other ball (ball B) reached fixed goal position (y = 0.33) at time t 20. A plan was successful if ball B's final position was within 0.1 of the goal.
7

---

Under review as a conference paper at ICLR 2021
We apply simple planner that starts from a given configuration and iteratively updates ball A's initial conditions by: (1) simulating the forward dynamics via our trained RNN / Jumpy RNN model to estimate the distance between ball B and the goal, and then (2) back-propaseee propagating this error through the dynamics model to estimate gradients for the initial conditions and then taking a step in that direction to update the initial conditions.
This procedure exposes interesting characteristics of our approach. Its increased efficiency at predicting the far-future directly impacts planning speed. By taking fewer steps, it may also be less susceptible to
exploding or vanishing gradients common long-horizon sequence models. Even with access to perfect simulator, this sort of gradient-base bbased planning is not guaranteed to
Tasks Tasks solved (%)
60
50
Beseline RNN
return an optimal plan. However, it does provide a setting for relative comparison.
40
Jumpy ditiio
30
We evaluate an ENN-Planner and Jumpy-Planner 1000 episodes, running the planning loop for 20 iterations each. Success rates are shown in Figure 6 While the baseline RNN succeeds slightly more often, it also takes twice as long our jumpy model (1.22 10 vs. 6.04 x 10 seconds trajectory). This suggests that Jumpy RNNs have potential efficient dynamics models for planning.
20
10
20
Planning step
Figure 6: Success rates for RNN- and Jumpy RNN-based planners.
4 RELATED WORK
In this section, review set of related works that, like own, aim model sequences at different timescales or across variable durations. The first and simplest way to model sequences at different timescales is to introduce multiplicative gating system in order to read, write, and erase memory vectors at different rates (Hochreiter & Schmidhuber) 1997; CChung et al.] [014 Graves et al. 2014). This gating system lets models retain information for longer than they would otherwise, but since the gates are leaky, over time, they still allow noise to dilute the stored information.
Hierarchical Recurrent Models. In order to solve the problem of leaky gating, there is line of work which replaces differentiable gating mechanisms with hard reads, writes, and erases and performs them different frequencies (Lin al., [996] Ling al., (015). Relatedly, Koutnik et al.) (2014) proposed dividing an RNN internal state into groups and only performing cell updates on the 2t group after 2 time steps. More recent works have aimed to make this hierarchical structure more adaptive, either by data-specific rules (Ling et al. 2015) or by a learning mechanism (Chung cct al. 2019).
Skip RNN Campos al. (2018) is related to Chung al. 2019) in that it the skip operation is "applied to the whole stack of RNN layers at the same time" thereby shortening the effective size of the computational graph. The Phased LSTM (Neil et al.) 2016), meanwhile, adds learnable periodic time gate to LSTMs so that different neurons can update themselves at different frequencies. More recently, Gooal et al. (2099b have proposed Recurrent Independent Mechanisms, which consist of a set of recurrent models which make dynamics predictions comi-independently and compete with one another when making global prediction by way of an attention mechanism. This leads to specialization of individual RIMs and permits some of them to operate at different timescales. We should note that there is a wealth of other works in this general domain including (Voelker al. 2019; Jernite et al., 2016). A common thread relating these hierarchical recurrent models is that they can all model data different timescales, but parts of them (the encoders at the very least) all must tick at at each time step in sequence, and hence cannot jump over regions of homogeneous change. Additionally, they are all rest on the notion of discrete "time steps" and not treat time as a continuous variable.
Modeling Irregular Timeseries. There is a parallel lineage of RNN models which, like hierarchical RNNs, permits hidden state updates over variable durations of time. The motivation for these models to handle sequences with irregular or missing time steps. The simplest approach is to include the time gap between observations as input to the RNN cell. In practice, though, preprocessing the data by interpolating Che et al. (2018) or masking (Lipton et al. (2016) and then training a regular RNN tends work better. More recently, several works have proposed applying exponential
8

---

Under review as a conference paper at ICLR 2021
decay function to the RNN hidden state in proportion to the duration of time between observations
(Che et al. 2018 Cao et al. 2018 Rajkomar et al. [0188] Mozer et al. (017). This approach makes
hidden state dynamics continuous time in a limited sense: hidden states are constrained to decay
towards zero at a particular rate.
Neural Ordinary Differential Equations. One way to learn more complex hidden state dynamics is to predict the derivative of an RNN hidden state and then integrate them over time. That is the core idea of Neural Ordinary Differential Equations, class of models which has existed since the 1988 Pearlmutter 1995), but which has recently seen a resurgence of interest (Chen et al. 2018). Recent work suggests that, like exponential decay models, neural ODEs can learn effective hidden state dynamics for irregular timeseries models (Rubanova et al. 2019). Like our jumpy model, these models can be integrated adaptively over time.
1990's (LeCun et al.
One challenge in training neural ODEs is that they must be continuously solved even when no observations occur, making forward passes take 60% 120% longer than standard RNNs (Rubanova et al. 2019). Since the integration speed of Neural ODE is proportional to the curvature of the hidden state dynamics (higher curvature requires more steps), one can improve speed and efficiency by regularizing that curvature. |Filay et al. (2020) take step in this direction. Even so, the number of time steps per function evaluation tends to be less than one. Our model resembles Neural ODEs in that it parameterizes the derivative of hidden state. However, we make the simplifying assumption that the hidden state dynamics are linear for long stretches of time. This makes our model extremely efficient integrate over long spans of time- more efficient, in fact, than baseline RNN. This jumpy behavior is what our model is designed for, and where it excels compared to Neural ODEs.
Other Related Works. Recent work by Gregor al. (018)) in the context of reinforcement learning develops a jumpy planning model which does not use an RNN cell or perform continuous interpolation of latent states. Another relevant work Embed to Control by Watter et al. (2015) which, like our model, assumes that dynamics is linear in a latent space. As with our work, they train their model on several dynamics problems, some of them from pixels. Unlike our work, their model performs inference over uniform time steps and does not learn jumpy behavior.
Some related works like Adaptive Skip Intervals by Neitz et al. (2018) and Time-Agnoste Prediction Jayaraman et al. (2019) use a "minimum-over-tinne
loss across horizon of future states. This effectively allows them to predict the next predictable state, regardless of how distant in time might be. In doing so, they decouple dynamics prediction from time and permit temporally-abstran planning. But in doing so, they are unable to reconstruct the dynamics which occur between these predictable frames, they are unable estimate the "rate of change" of the system, and they do not treat time as a continuous quantity. One way to look at these "minimum-ove connimmm--vve-ttmee approaches to temporal abstraction is method of finding bottleneck states states that can be expected to occur with a high degree of certainty McGovern & Barto (200T): : [Stlle & Precup (0002). Another means of achieving temporal abstraction is look for "decision states," which occur in situations where the agent diverges from its default behavior in order to "make a goal-dependent decision." Infobot, by Goyal al. (2019a), investigate this notion of temporal abstraction and propose a model which leverages model -ased planning only when it is in such a state. Unlike our model, parts of Infobot still "tick" at every time step and time is not treated as a continuous variable.
5 CONCLUSIONS
Having achieved widespread use in commercial and academic settings, RNNs are already a useful tool. But even though they are useful tools, they still have fundamental limitations. In this paper, we reckoned with the fact that they can only forecast the future in discrete, uniform time steps. order to make RNNs more useful in more contexts, it is essential to find solutions to such restrictions. With this in mind, we proposed a Jumpy RNN model which can skip over long durations of comparatively homogeneous change and focus on pivotal events the need arises. We hope that this line of work will expand the possible uses of RNNs and make them capable of representing time in a more efficient and flexible manner.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL tensorflow.org.
org/ Software available from
Victor Campos, Brendan Jou, Xavier Giro-i Nieto, Jordi Torres, and Shih-Fu Chang. Skip rnn: Learning to skip state updates in recurrent neural networks. International Conference on Learning Representations, 2018.
Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: Bidirectional recurrent imputation for time series. In Advances in Neural Information Processing Systems, pp. 6775-6785, 2018.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):1-12, 2018.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in neural information processing systems, pp. 6571-6583, 2018.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Workshop on Deep Learning, December 2014, 2014.
Junyoung Chung. Sungjin Ahn, and Yoshua Bengio. Hicrarchical multiscale recurrent neural networks. In 5th International Conference on Learning Representations, ICLR 20/7, 2019.
Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam M Oberman. How train your neural ode: the world of jacobian and kinetic regularization. 2020.
Chollet Francois. Deep learning with python, 2017.
Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Yoshua Bengio, and Sergey Levine. Infobot: Transfer and exploration via the information bottle- neck. International Conference Learning Representations, 2019a.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. arXiv preprint 19Xiv:1909.1089) 2019b.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5400 2014.
Karol Gregor, George Papamakarios, Frederic Besse, Lars Buesing, and Theophane Weber. Temporal difference variational auto-encoder. In International Conference on Learning Representations, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Dinesh Jayaraman, Frederik Ebert, Alexei A Efros, and Sergey Levine. Time-agnostic prediction: Predicting predictable video frames. International Conference on Learning Representations, 2019.
Yacine Jernite, Edouard Grave, Armand Joulin, and Tomas Mikolov. Variable computation in recurrent neural networks. arXiv preprint arXiv:1611.06118, 2016.
Olaf Kolle. The jena weather dataset. http::/www.gggeiimm.m..lllleeee 2016.
10

---

Under review as a conference paper at ICLR 2021
Jan Koutnik, Klaus Greff, Faustino Gomez, and Juergen Schmidhuber. A clockwork mn. In International Conference on Machine Learning, pp. 1863-1871, 2014.
Yann LeCun, D Touresky, G Hinton, and Sejnowski. A theoretical framework for back kack-propagation. In Proceedings of the 1988 comnectionist models summer school, volume 1, pp. 21-28. CMU, Pittsburgh, Pa: Morgan Kaufmann, 1988.
Tsungnan Lin, Bill G Horne, Peter Tino, and C Lee Giles. Learning long-term dependencies in narx recurrent neural networks. IEEE Transactions on Neural Networks, 7(6): 1996.
Wang Ling, Isabel Trancoso. Chris Dyer, and Alan W Black. Character-basee neural machine translation. arXiv preprint arXiv: 2015.
Zachary C Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with rnns: Improved classification of clinical time series. Machine Learning for Healthcare Conference, pp. 253-270, 2016.
Amy McGovern and Andrew G Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. 2001.
Michael C Mozer, Denis Kazakov, and Robert Lindsey. Discrete event, continuous time rnns. arXiv preprint arXiv: 1710.447 2017.
Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. Phased Istm: Accelerating recurrent network training for long or event-based sequences. Advances in neural information processing systems, 29:3882-3890, 2016.
Alexander Neitz, Giambattista Parascandolo. Stefan Bauer, and Bernhard Schölkopf. Adaptive skip intervals: Temporal abstraction for recurrent dynamical models. In Advances in Neural Information Processing Systems, pp. 9816-9826, 2018.
Barak A Pearlmutter.
Gradient calculations for dynamic recurrent neural networks: A survey. IEEE Transactions Neural nelworks, 6(5):1212-1228, 1995.
Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M Dai, Nissan Hajaj, Michaela Hardt, Peter J Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, et al. Scalable and accurate deep learning with electronic health records. NPJ Digital Medicine, 1(1):18, 2018.
Yulia Rubanova, Ricky TQ Chen, and David Duvenaud. Latent odes for series. Advances in Neural Information Processing Systems, 2019.
time
Martin Stolle and Doina Precup. Learning options in reinforcement learning. International Symposium abstraction, reformulation, and approximation, pp. 212-223. Springer, 2002.
Aaron Voelker, Ivana Kajić, and Chris Eliasmith. Legendre memory units: Continuous-time represen- tation in recurrent neural networks. In Advances in Neural Information Processing Systems, pp.
15570-15579, 2019.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed control: A locally linear latent dynamics model for control from raw images. In Advances in neural information processing systems, pp. 2746-2754, 2015.
A APPENDIX
A.1 EFFICIENT BATCH-WISE LINE SEARCH WITH MASKING.
h4 = RNNCEll ( [[LLLI hi-1 (*(T.) A: LeakyReLU(Why + +1
(8)
(9)
A(t) = h, +t (t- T,)h; for t E [T. + A
(10)
In Section 2.2 we defined the optimal jump width, A and described how to find it for a given step i using the constrained optimization objective in Equation Then we described how might solve
11

---

Under review as a conference paper at ICLR 2021
that equation by starting from T; and iteratively moving through the sample sequence, decoding model's prediction under the current dynamics, until the loss exceeds c. In this section, we describe how to vectorize this process efficiently over batch of examples.
At a high level, we will compute the RNN cell update batch at each time stop, but only update stored hidden state vectors for instances that exceed the threshold. Let the H; be B d matrix of hidden state vectors corresponding to a batch of B instances. Likewise for H, and the hidden velocity vectors.
Let M be a B dimensional, binary vector where the 6th entry denotes whether the loss for the 6th instance is greater than at the current step. With these definitions, we can write the batched update of H2 via a dynamic masking operation follows:
H:+ (t- H, H.
if E (mz, ((t)) <e otherwise
RNNCell ( [H, H,A.-1l, & (*(())
-MO He + (1- & RNNCell(.) where
<E
Since must compute M dynamically as we progress over the time dimension, cannot fuse the RNN cell updates over time, a common technique used to speed up teacher forcing. This, along with the additional cost of decoding h, and evaluating prediction error, increases the runtime of a training step of our model by factor of three over a baseline RNN. Unlike reference RNN implementations. however, our code is not optimized for speed. Furthermore, we are more interested in our model's s performance during evaluation than during training, and thus are willing to tolerate this constant slowdown.
A.2 DISCUSSION OF JUMPY BOOTSTRAPPING
In early experiments, we noticed that our model had difficult time transitioning from no jumpiness (selecting A = 1 at every time step) to some jumpiness (selecting A = 2 at rare intervals). Upon further investigation, we found that, having never needed to linearly extrapolate dynamics for more than one time step, our model was very bad doing so early in training. In fact, the prediction errors at A 2 were consistently much higher than c. We considered two methods of solving this problem. One was to anneal starting from large value to promote jumpy behavior early in training. The other was to allow the model to take jumpy step 1% of the time, regardless of the value of e. The second option gave the best and most stable results so we used our experiments.
A.3 ADDITIONAL RESULTS FROM JENA WEATHER DATASET
We preprocessed the data recommended by previous works: first we added sin transforms of the time feature corresponding day and year frequencies. Then we normalized all features by subtracting the mean and dividing by the standard deviation along each feature channel. The only case in which our methods diverge from previous approaches was in the number of features used: instead of using all 16 features, we used just five: day and year time features, temperature, pressure, and density (p). During training, we used the same hyperparameters as described previous experiments except: we used hidden states of size 256, we added weight decay of x 10 The additional weight decay was necessary because this dataset was smaller than previous datasets and overfitting was a significant problem.
A.4 VISUALIZING THE LINES TASK
12

---

Under review as a conference paper at ICLR 2021
Kooee sibipel)
22.5
Rouder 11
20.0
(deggC)
17.5
T T(degC)
(degC)
20.0
17.5
15.0
10
12.5
40
80
40
80
20
40
80
feunle 2 -.- New 4m
(degC)
(degC)
(ddegC)
-2
Textrr 915 12--2
FANI
21
0
20
40
60
80
0
20
40
60
80
20
40
60
80
Time
Time
Time
Figure 7: A comparison between jumpy RNN and two baseline RNNs trained on the Jena weather dataset. Each subplot shows a different test set trajectory and the corresponding predictions of the models. They warmup period occurs during the 20 time steps preceding the vertical gray lines. We see that the Jumpy model consistently produces trajectories that match or improve upon either of the baseline models. We less variability jump sizes compared the synthetic tasks; this is because the inherent noise in the data prevented our model from ever taking jumps larger than A; = 4. In future work, we hope to investigate alternative criteria for jumpiness, such as cunulative prediction error, which could mitigate this effect.
Jumay
Jumpiness
25
1.00
100
Liber
0.75
15
100
0.50
0.50
Task
0.25
0.25
10"4.
0.00
0.00
5
10-4
1
100
200
0
10
20
Train
Mean At
Figure 8: A comparison between jumpy RNN and a baseline RNN trained on lines task. Columns and 2 show samples from the models. Here we see that the Jumpy model can summarize entire trajectory with single linear function, whereas the bascline model cannot. Columns 3 and report key quantitative statistics of the two models. particular, Column 3 shows how jumpiness in- creases throughout training. Column shows that the Jumpy model is able to produce autoregressive samples of comparable quality to the baseline while using twenty times fewer cell updates to do so.
13


Reviews:
[{"review_id": "lPeGuurvsv5", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Main problem: The paper tries to tackle the problem of deciding how to make jumpy predictions or more precisely event based updates in a RNN as compared to clock based updated. Normal RNNs/LSTMs update their hidden states at every time step. Predicting the hidden states at each time step often lead to compounding errors i.e., small errors accumulate while making predictions for many time steps in the future. I really like this problem.\n\nMethod: The paper proposes a method to update hidden states in an event driven manner. The paper defines the hidden state as a continuous, piece wise linear function. The paper proposes to predict the jump interval as well a hidden velocity which basically signifies the \"change\" in hidden dynamics over the jump interval, thus the update function of the proposed method is a function of both the jump interval as well as hidden velocity.  In order to encourage the jump_interval of more than 1, the paper defines an auxiliary parameter such that the prediction loss b/w the actual input and predicted input is less than \\epsilon, where epsilon is a hyper-parameter. \n\nStrong points:\n\n- I really like the underlying idea, irrespective of the results.\n- I like the preliminary results in figure 3 which basically justifies the core behind the proposed method.\n- I also like the idea, and preliminary results for jumpy planning. it was also studied in [a, b].\n\nQuestions:\n\n1.  A main drawback of the proposed method is the reliance on the \\epsilon parameter. In many cases, it would be the case that their are multiple entities in the environment, like for ex 2 bouncing balls as compared to one, and then the model has to adaptively decide as to for which part of the input, the model should skip the update. So, I'm curious as to how the model with perform when their are multiple bouncing balls (or multiple entities in the env.) against each other.\n\n2. It would also be interesting to study the generalization performance of the proposed method. Does the ability to skip update gives the ability to generalize better to out of distribution examples.\n\n3.  \"Finally, we vary the dimensionality of our model’s hidden state and retrain. In figure 4, we find a positive\nrelationship between model capacity and jumpiness\" . The paper mention that they find a positive correlation b/w the model capacity and jumpiness.  Does this behaviour exists in all the datasets which the paper explore ?\n\n4. Related work: I think, many of the important references are missing. There are various important references. Like there's some work done where the idea is to learn how to skip updates in RNN like in SkipRNN (c), or in Adaptive Skip Intervals (d), or in the context of RIMs (e), where each module decides whether to update it's hidden state or not update their hidden state. There's also some work where the idea is to learn event based representations like in PhaseLSTM (f). It would be nice to see how the proposed method compares to any of these methods.\n\nReferences:\n\n- (a) Time-Agnostic Prediction: Predicting Predictable Video Frames, https://arxiv.org/abs/1808.07784\n- (b). InfoBot, https://arxiv.org/abs/1901.10902\n- (c). Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks, https://arxiv.org/abs/1708.06834\n- (d). Adaptive Skip Intervals: Temporal Abstraction for Recurrent Dynamical Models, https://arxiv.org/abs/1808.04768\n- (e). Recurrent Independent Mechanisms, https://arxiv.org/abs/1909.10893 \n- (f). PhaseLSTM, https://papers.nips.cc/paper/6310-phased-lstm-accelerating-recurrent-network-training-for-long-or-event-based-sequences.pdf\n\n5. I'm not sure what's the take away message from the paper as of now. In the current form the paper shows computational benefits i.e., the proposed method can achieve similar results, by updating dynamically based on events, and hence establishes a trade-off between the temporal resolution of the input and the computational expense. The results for planning are also very interesting. So, one suggestion could be to do extensive experiments and formulate the entire paper in the context of planning because as of now, the bouncing ball results for test mse are not better as compared to the baseline. \n\n=======\n\nAfter Rebuttal: I have read the rebuttal, as well as reviews by other reviewers. I really like the idea, but its important to evaluate the idea with respect to a downstream task to get a better idea on how to use the learned structure. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "99BgauCQ5j", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This work presents Jumpy RNNs, a recurrent network that learns to take variable length steps based on time-scales of the data. The core idea of the paper is to learn a hidden velocity and time span, along with the standard hidden state. The hidden velocity is then used to linearly interpolate the hidden state within the learned time span.  This leads the proposed model to update the hidden state only at the end of time span thus allowing the model to jump over by certain steps proportional to time span. The model has flexibility to produce fine-grained or continuous-time predictions as well as predicting far into the future. Since, Jumpy RNNs do not update the hidden state at every time step, they are computationally efficient than standard RNNs. \n\nPositives:\n1. The idea of building a model to provide short term and long term prediction using just one hidden state update is interesting. The problem addressed is important in time series predictions in many domains.\n2. The paper is well written and easy to follow.\n3. Experiments show that the model works well on simulated datasets.\n\nConcerns:\n1. The key concern about the paper is the lack of rigorous experimentation to study the usefulness of the proposed method. The current experiments are mostly based on simulations and are too simple to evaluate the effectiveness of the proposed model. In almost all the cases, the rate of change is constant with few sharp changes. In particular, it would be interesting to see how the model performs on real world datasets, where the hidden state dynamics is unknown, for short-term and long-term time series predictions.\n2. Despite the paper stating that there have been some recent work on Neural ODEs (Chen et al., 2018), the paper does not compare with them. Neural ODEs are one of the possible candidates for solving the given problem and it would be interesting to see how fast/slow they are compared to the proposed model.\n3. The linear-dynamics of hidden state is too simple and can only handle constant hidden state dynamics or the one with constant slope. Although the authors mention that the hidden-state linearity does not translate to linearity in output space as the learned decoder can be an arbitrarily complex non-linear function, but the experiments are only based on constant or linear hidden state/output dynamics.\n4. What's the difference between Test MSE and Sample MSE? Based on the results, standard RNNs achieve lower MSE scores than the proposed model in most cases but use more computations. An interesting comparison would be to compare with a standard RNN with similar computational power as the final learned Jumpy RNN. Possible baseline could be training standard RNN with reduced sampling rate and do linear interpolations in between the predictions.     \n\nOther comments:\n1. What does baseline model refer to here: 'setting $\\epsilon$\u000f to the final training loss of the baseline model'?\n2. In the 3rd line of 2nd para in section 3.1, shouldn't it be 'c' instead of 'x'?\n3. Could the authors comment on the possibility of using the proposed approach to learn from irregularly sampled time series?\n\n==========Post-rebuttal comments================\n\nBased on other reviews and authors response, I have decided to keep my score. I still feel this paper need more work such as experiments on real world datasets and more comparisons as pointed out in the review. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "7fPfbNKL9fZ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Paper Summary\n\nThis paper proposes a recurrent network architecture for future prediction where the hidden states (and the outputs) aren't updated step by step as done traditionally. Instead, the network models the hidden state dynamics as being piecewise linear over varying time spans. It learns to produce the linear dynamics together with each time span, and can \"jump\" to the next time span according to its own predictions. Hidden states for any time step within a span can be easily obtained by interpolation using the predicted linear dynamics. Experiments on a series of synthetic benchmarks are used to demonstrate that the model can learn to utilize this structure to reduce the amount of computation.\n\n## Strengths\n\n- The proposed approach is simple and intuitive. The study is likely to be valuable to researchers in sequence modeling, by showing that these simple ideas can work surprisingly well. One might expect that the non-stationarity of the \"jump size\" targets might make training such models too difficult, but the authors show that other than requiring a couple of tricks (setting $\\epsilon$ using a baseline and \"jump bootstrapping\"), training works well.\n\n- The paper is an enjoyable read due to its clarity and presentation. The claims made are modest and clear without over-reaching, the experiments use tasks that are directly motivated by the goals of the study, and the related work is clearly and fairly discussed.\n\n## Weaknesses\n\n- A clear weakness of the paper is that all experiments were conducted on synthetic tasks of low complexity. Although these experiments were illustrative and helpful, it is unclear if the approach works reasonably well on more realistic problems. One \"failure\" mode could be that the Jumpy RNN tends to jump almost every step for noisy and real-world data. At this stage this study does not really say whether this architecture is also likely to work well on such problems, but is more of a proof of concept.\n\n## Review Summary\n\nI think the strengths of this paper outweigh the weaknesses. While the experiments were not conducted on common benchmarks or real-world data, I think this paper is well-written as a proof of concept, does not overclaim and should motivate further work on similar ideas. My recommendation is to accept.\n\n## Questions for Authors\n\n1. Have you tried more complex datasets or RL problems? Are there any negative results you can report?\n2. Have you found the non-stationarity of the targets to be issue for training under any conditions?\n\n## Post-rebuttal\n\nI thank the authors for their hard work, and for incorporating my suggestions into the paper. I believe the paper has improved.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-he9wJkXxOM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes Jumpy Recurrent Neural Network, an RNN model with non-uniform time steps. To train this model, the authors propose to use a greedy supervision to determine optimal time intervals. The experiments on linear dynamics prediction and planning show comparable performance of proposed model against standard RNNs. The advantage is that the proposed model can significantly speed up RNNs under relatively linear dynamics prediction tasks. Even though non-uniform time step RNNs have been studied in many literature, this proposed training supervision method seems novel.\n\n+ves: \n\n+ Overall, the paper is well written. In particular, the proposed model and its training methods are clearly explained.\n\n+ The results section is well structured. The hyperparameters, architecture, and experimental settings are well demonstrated. \n\n \nConcerns: \n\n- The proposed method, non-uniform time interval RNN, is not novel. For example, Che et al 2018, Lipton et al 2016, have very similar contributions. This paper needs to clarify the difference.\n\n \n- The proposed greedy training method seems have significant flaw. Clearly it will have Delta=1 at the beginning of the training. Then it can only improve corresponding Delta one by one at each timestep. The loss corresponding to Delta is highly non-continuous and it will definitely not learn to find the real optimal Delta.\n\n\n- The model does not seem to learn meaningful Delta. See Figure 3 and 4. For such clean data, the model should learn to predict only turning points but it seems it just predict random intervals. Also, there are no verification of learning meaningful Delta.\n\n \n- The experiment is clearly not fair for a standard RNN. Since Delta=1 is always true for standard RNN, clearly it will be slower. The speedup ratio will only be the average of predicted Delta in the proposed model. However, what if the RNN just sets a larger Delta, what if the RNN just set a random predicted Delta? \n\n \n- The last experiment is not convincing. First, it's a simple supervised task, not real \"model-based planning\". Second, Figure 6 seems not converged. No experimental detail is given for this task. Third, twice as fast mean the model generally just predict Delta=2.\n\n \n- All the experiments have internal bias towards linear dynamics. This favors the model with corresponding inductive bias. To prove the model is useful, I would recommend to run standard RNN synthetic task like copying task, adding task etc, to prove its basic functionality.\n\n \nMinor comments: \n\n* Figure 1, in basic RNN, it should be an arrow instead of a line from h_i to GRU.\n\n* Equation 3, people usually don't write output as x, otherwise it can only be a sequence prediction task like language modeling. Please write it y. Also, it creates confusion to equation 7 whether it's input or output.\n\n* The \"Jump Bootstrap\" seems very important technique for the proposed method. But there's no experiment or discussion on it.\n\n=====POST-REBUTTAL COMMENTS========\n\nI thank the detailed response from the authors. The authors addressed the novelty of this paper. The experimental results on toy tasks are convincing. However, the way this method increases Delta still seems very problematic to me and not seem robust in complex real world cases.\n\nI increased my score.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a "jumpy RNN" to adaptively change the step size of an RNN to match the time scales of the system dynamics. Reviewers found merit in the simple and intuitive idea, but were less enthusiastic about the experimental results and the comparison to existing work. (Adaptive step size methods have been a subject of recent work in RNNs, not to mention in numerical methods for ODE solvers.) Overall, I think the additions the authors made in the discussion phase did strengthen the paper, but further work is necessary before publication. 
Decision: Reject