Flipped Classroom: Effective Teaching for Time Series Forecasting
Authors: Philipp Teutsch
Patrick Mäder
Venue: Accepted by TMLR ()

Abstract: Sequence-to-sequence models based on LSTM and GRU are a most popular choice for forecasting time series data reaching state-of-the-art performance. Training such models can be delicate though. The two most common training strategies within this context are teacher forcing (TF) and free running (FR). TF can be used to help the model to converge faster but may provoke an exposure bias issue due to a discrepancy between training and inference phase. FR helps to avoid this but does not necessarily lead to better results, since it tends to make the training slow and unstable instead. Scheduled sampling was the first approach tackling these issues by picking the best from both worlds and combining it into a curriculum learning (CL) strategy. Although scheduled sampling seems to be a convincing alternative to FR and TF, we found that, even if parametrized carefully, scheduled sampling may lead to premature termination of the training when applied for time series forecasting. To mitigate the problems of the above approaches we formalize CL strategies along the training as well as the training iteration scale. We propose several new curricula, and systematically evaluate their performance in two experimental sets. For our experiments, we utilize six datasets generated from prominent chaotic systems. We found that the newly proposed increasing training scale curricula with a probabilistic iteration scale curriculum consistently outperforms previous training strategies yielding an NRMSE improvement of up to 81% over FR or TF training. For some datasets we additionally observe a reduced number of training iterations. We observed that all models trained with the new curricula yield higher prediction stability allowing for longer prediction horizons.

Published in Transactions on Machine Learning Research (10/2022)
Flipped Classroom:
Effective Teaching for Time Series Forecasting
Philipp Teutsch Technische Universität Amenau
Shalipp.teutsch@iie www.-atttttmmmmddeemm de
Patrick Mäder
patrick. Sadricckmaedder@@tt-uteeee
Technische Universität Imenna
Saat--Shhhhlle--nnnestitte Jena
Reviewed on OpenReview: /ttps://openw: opw.r/vee......e..oo net/ formm? id= d=u2220YEc
Abstract
seppuence-to-equeeee models based on LSTM and GRU are most popular choice for fore- casting time series data reaching state-of-the-ae the--arttt performance. Training such models can be delicate though. The two most common training strategies within this context are teacher forcing (TF) and free rumning (FR). TF can be used to help the model to converge faster but may provoke an exposure bias issue due to a discrepancy between training and infer- ence phase. FR helps to avoid this but does not necessarily lead better results, since it tends to make the training slow and unstable instead. Scheduled sampling approach tackling these issucs by picking the best from both worlds and combining it into curriculum learning (CL) strategy. Although scheduled sampling seems to be convincing alternative to FR and TF, we found that, even if parametrized carefully, scheduled sampling may load to premature termination of the training when applied for time scries forecast- ing. To mitigate the problems of the above approaches we formalize CL strategies along the training as well as the training iteration scale. We propose several new curricula, and systematically evaluate their performance in two experimental sets. For our experiments, utilize six datasets generated from prominent chaotic systems. We found that the newly proposed increasing training scale curricula with probabilistic iteration scale curriculum consistently outperforms previous training strategies yielding an NRMSE improvement of to 81% over FR or TF training. For some datasets we additionally observe a reduced number training iterations. We observed that all models trained with the new curricula. yield higher prediction stability allowing for longer prediction horizons.
the first
1 Introduction
Advanced Recurrent Neural Networks (RNNs) such as Long Short Term Memory (LSTM) (Hochreiter & Schmidhuber 1997) and Gated Recurrent Unit (GRU) (Cho et al. 2014) achieved significant results predicting sequential data (Chung al. 2014 Nowak et al. 2017 can for example be textual data as processed for Natural Language Processing (NLP) tasks where RNN models were the method of choice for long time, before feed-forward architectures like transformers showed superior results in processing natural language data (Devlin et al. 2018 Yang et al. 2019 Radford et al, 2019 Brown et al. 2020). Shifting the view to the field of modeling dynamical or even chaotic systems, encoder-decoder RNNs are still the method of choice for forecasting such continuous time series data (Wang
et al., 2017). Such sequential data
et al.]
2019 Thavarajah et al., 2021: Vlachas et. al. 2018 Sehovac el. al. 2019
Sehovac & Grolinger 2020
Shen et al. 2020 Walther et al. 2022 Pandey et al.
2022
Nevertheless encoder-decoder RNNs do have their difficulties, especially when it comes to effectively training them. Firstly, there is the exposure bias issue that can appear when teacher forcing (TF) is used for training

---

Published in Transactions on Machine Learning Research (10/2022)
the model. TF is the strategy that is typically applied when training RNNs for time series sequenco-to- sequence tasks (Williams & Zipser 1989), regardless of the type of data. To understand the problem, we first give a quick side note about the motivation behind TF. Its main advantage is that it can significantly reduce the number of steps a model needs to converge during training and improve its stability (Miao ct al. 2020p. However, TF may result in worse model generalization due to a discrepancy between training and testing data distribution. It is less resilient against self-induced perturbations cansed by prediction errors the inference phase (Sangiorgio & Dercole, 2020). However, simply training the model without TF in free running (FR) mode instead does not necessarily provide convincing results, can be seen in Section 5.6 Rather more sophisticated strategy is needed to guide the model through the training procedure.
Several authors propose methods to mitigate the exposure bias or improve training stability and results in general inventing certain strategies (Bengio et al. 2015;
Nicolai & Silfverberg, 2020 Lamb et al.] 2016 Liu
ct al.
2020: Dou et al. 2019 Hofmann & Mäder
2021 Even though most of these methods address the
training for NLP tasks, such as text translation, text completion or image captioning, we focus on time series forecasting. Our datasets consist of sequences sampled from approximations of different chaotic systems. With these systems, the next state can always be derived from the past state(s) deterministically, but they also tend to be casily irritated by small perturbations. Thus the trained model needs to be especially resilient against small perturbations when reto-segryyyvvely predicting future values, otherwise those small errors will quickly accumulate to larger errors (Sangiorgio & Dercole, 2020). These aspects make the kind of data we use especially challenging. Since TF avoids the error accumulation only during training, we argue the models are even more vulncrable to issues like the exposure bias and will thus more likely fail to forecast larger horizons.
Around the interest of predicting particularly (chaotic) dynamical systems with RNNs, another field has formed that very intensively studies several specialized RNNN-bsed approaches proposed to stabilize the training process and prevent exploding gradients. Most of propositions include architectural tweaks or even new RNN architectures considcring the specifics of dynamical systems and their theory (Lusch ct al. 2018
Vlachas el al.)
2018 Schmidt et al.
2019 Champion el al. 2019 Chang el al., 2019 Rusch & Mishra, 2020
Erichson et al.
2020 Rusch et al.
2021 Li et al.) 2021).
Monfared et al.
2021), , for example, performed theoretical analysis relating RNN dynamics to loss gradients and argue that this analysis is especially insightful for chaotic systems.. With this in mind they suggest a kind of sparse teacher forcing (STF), inspired by the work of Williams & Zipser 1989) that uses information about the degree of chaos of the treated dynamical system. As a result, they form a training strategy that is applicable without any architectural adaptations and without further hyper-parameters Their results using a vanilla RNN, piecewise linear recurrent neural network (PLRNN) and an LSTM for the Lorenz (Lorenz) 1963) and the Rössler (Rössler 1976 systems show clear superiority of applying chaos-dependent STF.
Reservoir computing RNNs were successfully applied to chaotic system forecasting and analysis tasks well. For example, Pathak et al. (017) propose a reservoir computing approach that fits the attractor of chaotic systems and predicts their Lyapunov exponents.
In this paper, we focus on CL training strategies like scheduled sampling that require no architectural changes of the model and thus can be applied easily for different and existing sequence-to-sequeee (seq2seq) models. All presented strategics will be evaluated across different benchmark datasets. Our main contributions are the following: First, assembling set of training strategies for encoder-decoder RNNs that can be applied for existing seq2seq models without adapting their architecture. Second, presenting collection of strategies' prrer-par configurations that optimize the performance of the trained model. Third, proposing a "flipped classroom" like strategy that outperforms all existing comparable approaches on several datasets sampled from different chaotic systems. Fourth, proposing a method that yields substantially better prediction stability and therefore allows for forecasting longer horizons.
The course of the paper continues with Section 2 where we provide the background of soqueecc-to-sequeeee RNNs and the conventional ways to train them. We also give short introduction to chaotic behavior of our data. In Section 3 we examine existing approaches dealing with the difficulty of training seq2seq RNNs in the context of different applications. Also, we give an overview of work treating RNNs for chaotic system prediction in particular. Section 4 describes how we designed our training strategies and how they
2

---

Published in Transactions on Machine Learning Research (10/2022)
are applied. Further information about the experimental setup and our results present in Section In Section ① we discuss the results, the strengths and limitations of the different strategies. Finally, in Section we conclude our findings.
2
Background
Within this paper, we study Curriculum Learning (CL) training strategies for sequencc-to-ssequenncc RNNs. We are using an recoderd architecture (Chung al. 2014) as model forecast time series data. The rough structure of the encoder-decoder architecture is shown in Figure Ta It consists of two separate RNNs, encoder and a decoder. The encoder is trained to build up a hidden state representing the recent history of the processed time scries. It takes an input sequence 2, E Rd is a d dimensional vector. For each processed step, the encoder updates its hidden state to provide context information for the following steps. The last encoder hidden state (after processing Tn)) is then used as imitial hidden state of the decoder. Triggered by a sequence's preceding value, the decoder's task is predicting its next future value while taking into account the sequence's s history via the accumulated hidden state. For a trained network in the inference phase, that means that the decoder's preceding prediction is auto-rogressively fed back as input into the decoder (aka auto-regrossive prediction) (cp. Fig. 1a All outputs yj RR logether form the output sequence (y1, ..... mm).
In) of input values where cach value
$1
9
RNNN Call
RNN Coll
RNN Cell
RNN Cell
RNN Call
RNN
RNN Cell
RNN Call
RNN Cell
RNN Cell
Cell
Encoder
Decoder
32
(b)
Figure 1: RNN encoder-decoder architecture in inference phase (a) and in training phase using teacher forcing (b)
2.1 Training Sequence Prediction Tasks
While training, the decoder's inputs may either be previously predicted outputs (free running) or known previous values stemmingg from the given dataset (teacher forcing). Training in free running mode might be an intuitive approach, but slows down the training. This is due to accunulated error throughout multiple prediction steps without the support of teacher forced inputs especially in the early training epochs (Nicolai & Silfverberg. 2020). In contrast., teacher forcing aims avoid this problem by utilizing the corresponding previous ground truth value as decoder input rather than the previously predicted value. This way, the model learns from the beginning of the training noticeable faster (Miao et al. 2020) (cp. Fig. Ib]. However,TH TF also bears significant drawback, since during training the model is never exposed to the noisy predicted values it will later face in the inference phase. Therefore does often not generalize very well. A model trained with TF solely learned to predict on basis of perfect values and is thus vulnerable for small perturbations on its input, this effect called exposure bias (Ranzato et al. 2015).
adapt its weights to perfect input values and converges
2.2 Chaotic Systems
The data we are using for our experiments later in Scction generated from chaotic systems. Whether or not a dynamical system is chaotic, can be confirmed by considering its Lyapunov exponents x for k [1, d]. Given initial perturbation én the exponential rate with which the perturbation will increase (or decrease)
in the direction of dimension the Lyapunov exponent d = lim 1 In ([lع:)))) DDngwell 2006 That
t
3

---

Published in Transactions on Machine Learning Research (10/2022)
is, the Lyapunov exponents denote how sensitive the system is to the initial conditions (initial state). A deterministic dynamical system with at. least one positive Lyapunov exponent while being aperiodic in its asymptotic limit is called chaotic (Dingwell 2006). Analogously, dynamical systems with at least two positive, one negative, and one zero Lyapunov exponent are called hyper-chaotic systems. Dingwell points out that the largest Lyapunov exponent (LLE) can be used as a measure to compare the chaotic behavior of different systems.
3 Related Work
Schmidt (2019) defines exposure bias as describing "a lack of generalization with respect to an usually implicit and potentially task and domain dependent measure other than maximun-likelhoood" meaning that when the exclusive objective is to maximize the likelihood between the output and the target sequence, one can use TF during training [Goodfellow et al. 2016). However, Goodfellow et al. argue that the kind of input the model sees while testing will typically diverge from the training data and the trained model may lack the ability of correcting its own mistakes. Thus, in practice, the TF can be a proper training strategy but may hinder the model to learn compensating its inaccuracies. He et al. study exposure bias for natural language generation tasks (He et al., 2019). They use sequence- and word-level quantification metrics to observe the influence of diverging prefix distributions distributions are generated. One with and one without induced exposure bias. Those two distributions are then compared on basis of the corresponding corpus-bleu scores (Papineni et al. 2002). The study concludes that for language generation, the effect of the exposure bias is less serious than widely believed.
the distribution of the generated sequences. Two
Several studies propose approaches to overcome the exposure bias induced by TF. The carliest of these studies proposes scheduled sampling (Bengio al., 2015). Scheduled sampling tries to take the advantages of training with TF while also acclimating the trained model to its own generated data error. It does that by using ground truth values as input for subset of the training steps and predicted values for the remaining. & denotes the TF probability at step i. Accordingly, the probability of using the predicted value is 64 During training 6 decreases from 6s to 6e. This procedure changes the input data while training, making a curriculum learning approach, as which scheduled sampling was proposed and works without major architectural adaptions. Originally proposed for image captioning tasks, scheduled sampling was also applied, e.g., for sound event detection (Drossos et al., 2019 Nicolai and Silfverberg consider and study the TF probability
a hyper-parameter. Rather than using decay function that determines the decrease of é; over the course of training epochs, they use a fix TF probability throughout the training [Nicolai & Silfverberg. 2020). They observed a moderate improvement compared to strict TF training. Scheduled sampling is not restricted to RNN-based requence-to-sequenee models though, it has also been studiod for transformer architectures (Mihaylova & Martins 2019). Mihaylova and Martins tested their modified transformer on two translation lasks but could only observe improved lest results for one of them.
Apart from scheduled sampling (Bengio et al., 2015), number of approaches have been proposed typically aiming to mitigate the exposure bias problem by adapting model architectures beyond encoder-decoder design. Professor forcing (Lamb et al. 2016) is one of these more interfering approaches that aims to guide the teacher forced model in training by embedding it into a Generative Adversarial Network (GAN) framework (Goodfellow et al., 2014). This framework consists of two encoder-decoder RNNs that respectively form the generator and the discriminator. The generating RNNs have shared weights that are trained with the same target sequence using their respective inputs while at the same time they try to fool the discriminator by keeping their hidden states and outputs as similar as possible. The authors conclude that their method, compared to TF, provides better generalization for single and multi-step prediction. In the field of Text-To- Speech (TTS), the concept of professor forcing has also been applied in the GaN-based training algorithm proposed by Guo et al. (2019). They adapted professor forcing and found that replacing the TF generator with one that uses scheduled sampling improved the results of their TTS model in terms of intelligibility. As another approach for TTS, (Liu et al., 2020) proposed teacher-student training using a training scheme to keep the hidden states of the model in FR. mode close to those of a model that was trained with TF. It applies a compound objective function to align the states of the teacher and the student model. The authors observe improved naturalness and robustness of the synthesized speech compared to their baseline.
4

---

Published in Transactions on Machine Learning Research (10/2022)
Dou et [Dou et al. 2019) proposed attention forcing as yet another training strategy for sequence- to-sequence models relying on an attention mechanism that forces a reference attention alignment while training the model without TF. They studied TTS tasks and observed a significant gain in quality of the generated specch. The authors conclude that attention forcing is especially robust in cases where the order of predicted output is irrelevant.
The discussed approaches for mitigating exposure bias were proposed in the context of NLP and mainly target speech or text generation. For time scries forecasting, (Sangiorgio & Dercole (2020) suggest to neglect TF completely and solely train the model in FR mode, thereby, sacrificing the faster convergence of TF and potentially not reaching convergence at all.
In the context of RNNs for forecasting and analyzing dynamical systems, the majorily of existing work deals with exploding and vanishing gradients as well as capturing long-term dependencies while preserving the expressiveness of the network. Various studics rely on methods from dynamical systems theory applied to RNN propose new network architectures.
Lusch et al. 2018)) and Champion et al. (2019) use modified autoencoder to learn appropriate eigenfunctions that the Koopman operator needs to lincarize the nonlinear dynamics of the system. In another study, Vlachas et al. (2018) extend an LSTM model with a mean stochastic model to keep its state in the statistical steady state and prevent it from escaping the system's attractor. Schmidt et al. (2019) propose a more generalized version of a PLRNN (Koppe et al.) 2019 by utilizing a subset of regularized memory units that hold information much longer and can thus keep track of long-term dependencies while the remaining parts of the architecture are designated to approximate the fast-scale dynamics of the underlying dynamical system. The Antisymmetric Recurrent Neural Network (AntisymmetriccNN)) introduced by Chang et al. (2019) represents RNN designed to inherit the stability properties of the underlying ordinary differential equation (ODE), ensuring trainability of the network together with its capability of keeping track of long- term dependencies. A similar approach has been proposed as Coupled Oscillatory Recurrent Neural Networks (coRNNs) (Rusch & Mishra 2020) that are based on secondary order ODEs modeling a coupled network of controlled forced and damped nonlinear oscillators. The authors prove precise bounds of the RNN's state gradients and thus the ability of the coRNN being a possible solution for exploding or vanishing gradients. Erichson ct al. (2020) propose the Lipschitz RNN having additional hidden-to-hidden matricos enabling the RNN lo remain Lipschitz continuous. This stabilizes the network and alleviales the exploding and vanishing gradient problem. In Li et al. (2020; 2021), the authors propose the Fourier respectively the Markov neural operator that are built from multiple concatenated Fourier layers that directly work on the Fourier modes of the dynamical system. This way, they retain major portion of the dynamics and forecast the future behavior of the system. Both, the incremental Recurrent Neural Wetwork (IRNN) (Kag et al. 2019) and the time adaptive RNN (Kag & Saligrama 2021) use additional recurrent iterations on each input to enable the model of coping different input time scales, where the latter provides time-varying function that adapts the model's behavior to the time scale of the provided input.
All of this shows the increasing interest in the application of machine learning (ML) models for forccasting and analyzing (chaotic) dynamical systemss. To meet this trend, Giilpin (2021) recently published a fully featured collection of benchmark datasets being related to chaotic systems including their mathematical properties.
A more general guide of training RNNs for chaotic systems is given by Monfared et al. 20221) ). They discuss under which conditions the chaotic behavior of the input destabilizes the RNN and thus leads to exploding gradients during training. As solution they propose STF, where every 7-th time step true input value is provided (teacher forced) as input instead of the previous prediction.
4 Teaching Strategies
Within this soction, we systematically discuss existing tcaching strategics for sequenec-to-ssqueeee prediction models and propose new strategics. All of these will then be evaluated in an experimental study with different time series reported in the following section.
5

---

Published in Transactions on Machine Learning Research (10/2022)
Teacher Forcing (TF)
Free Running (FR)
TF
FR
Curriculum Learning (CL.
Decreasing Teacher Forcing (DTF)
Constant Teacher Forcing (CTF)
Inereasing Teacher Forcing (ITF)
Probubilistic (P)
far parapick
CL-DTF-P
CL-CTF-P
CL-ITF-P
Deterministic (D)
CL-DTF-D
CL-ITF-D
Figure 2: Overview of the proposed and evaluated training strategies where teacher forcing (TF) and free running (FR) refer to the two extreme cases that are combined to different CL strategies.
4.1 Free Running (FR) vs. Teacher Forcing (TF)
A widely used training stralegy for RNN sequeence-to-sequence models is to use TF throughout the whole training. Thereby, data is processed as shown in Fig. 2 (top left), i.e., the model is never exposed to its own predictions during training. A single forward step of the decoder during TF training is denoted as
3' = - c,,c"'),
(1)
where yi is the ground truth value for time step i, G is the predicted value for time step 0 denotes the trainable parameters of the model, and c' is the decoder's hidden state at time step t.
The other extreme form training FR, i.., only the model's first output value predicted on basis of ground truth input and all subsequent output. values the sequence are predicted on basis of previous predictions throughout the training (cp. Fig. (top right)). A single forward step of the decoder during FR training is denoted as
0' f( 0,
(2)

---

Published in Transactions on Machine Learning Research (10/2022)
2.35
2.30
-2
2.25
-4
2.35
4
2.30
2.25
-2
-2
2.30
2.25
2 4
4
2.35
(a)
(b)
Figure 3: 30 000 time steps sampled with time delta of dt = 0.1 of Thomas' cyclically symmetric attractor in (a) chaotic parametrization and (b) periodic parametrization
TF: chaotic
0.004
FR: chaotic
TF: pertodic
FR: periodic
0.003
NRMSE
0.002
0.001
0.000
20
40
60
80
100
step
Figure 4: Test NRMSE over 100 predicted time steps of the chaotically and periodically parametrized Thomas attractor (cp. Fig. 3b predicted by GRU models trained with teacher forcing (TF) and free running (FR).
A claimed major benefit of TF training is faster model convergence and thus reduced training time (Miao et al. 2020), while a major benefit of FR training is avoided exposure bias arising from solely training with ground truth data, yielding a model that performs less robust on unseen validation data (Ranzato et al. 2015). To illustrate these benefits and drawbacks, we utilize the Thomas attractor with two parametrizations. the first resulting in a periodic (cp. Fig. Bb) and the second resulting a chaotic attractor (cp. Fig. 3a By sampling from the attractors, we build two corresponding datasets of 000 samples each. For both datasets, we train single layer encoder-decoder GRU following the FR and the TF strategy. Figure 4 shows the Normalized Root Mean Squared Error (NRMSE) per trained model over 100 predicted time steps. All models have been initialized with 150 ground truth values to build up the hidden state before predicting these 100 time steps. We observe that the chaotic variant is harder to predict for the trained models (cp. blue and green line in the figure) and that those trained with TF tend to predict with a smaller error at the first steps, which then grows relatively fast. In contrast, the prediction error of the FR-trained models starts on higher level but stays more stable over the prediction horizon, arguing that time series forecasting represents especially challenging task for sequence-to-sequeen models when it exhibits chaotic behavior.

---

Published in Transactions on Machine Learning Research (10/2022)
The more precise forecasting capabilities of TF-trained network at the carly prediction steps vs. the overall more stable long-term prediction performance of a FR-trained network observed in the Thomas example (cp.
Fig 3a
3bJ, motivates the idea of combining both strategies into CL approach.
describes the exposure bias in natural language generation as a lack generalization. This argumentation motivates an analysis of training with FR and TF strategies when applied to forecasting dynamical systems with different amounts of available training data. Figure shows the NRMSE when fore- casting the Thomas attractor using different dataset sizes and reveals that increasing the dataset size yields generally improved model performance for TF well as FR, while their relative difference is maintained.
Schmidt 2019
TF
FR
NRMSE
10-1
5000
10000
15000
20000
25000
30000
dataset size
Figure 5: The NRMSE for different dataset sizes when using TF and FR during training for forecasting the Thomas attractor
4.2 Curriculum Learning (CL)
Within the context of our work, we denote the curriculum learning concept as combining TF and FR training, i.c., starting from the second decoder step, the curriculum prescribes per decoder step whether to use the ground truth value or the predicted value of the previous time step as input. We formalize a single training
step of a CL approach follows:
8' [ ,,c-1 1
(3)
1100-1,0,c otherwise
where the TF decision & governs whether the decoder input is teacher forced not. Figure 2 illustrates the data flow of sequence-to-sequenec model training with CL in between the conventional strategies. In naming scheme CL-DTF-P resembles the scheduled sampling approach proposed by [engiio et al. (2015). Below, we discuss the different lypes of curricula on training and iteration scale resulting in dilferent. ways for determining +
4.3 Curriculum on Training Scale
The TF ratio 6 per training iteration i determined by a curriculum function C: N - [0, 1]. We distinguish throe fundamental types of curriculum on training scale. First, constant curricula, where a constant amount of TF is maintained throughout the training denoted as
Second, decreasing curricula, where the training starts with a high amount of TF that continuously declines throughout the training. Third, increasing curricula, where the training starts at low amount of TF that
8

---

Published in Transactions on Machine Learning Research (10/2022)
continuously increases throughout the training. Both follow transition function C' : N [starts Cend denoted as
C(i,,
(5)
where Estart I C S 6++1 I Cend for increasing curricula, Cstart 2 G, c++1 2 cend for decreasing curricula and Extart F Fend for both. The following equations exemplary specify decreasing curricula. (cp. Eqs. following differing transition functions inspired by those used to study the scheduled sampling approach [Bengio et al. 2015
i max(Eends Cend + (Estart Cend) (1 {.).
Cin (i)
1 <<t, L, iEN, F. -
with Cend
(6)
k Cend + (Estart Cend)
Cinesig(t)
k ket
with fend < Estarts 1 1 k, i iEN,
(7)
Cezp(i) = Cend + (Estort Cend) k'
with Cend Cstart: 0 k 1, ieN,
(8)
where the curriculum length parameter L determines the pace number of iterations which the curiculum Ckin needs to transition from Cstart to Cend- The curricula CiavSig and Ceap have no such parameter since the functions never completely reach fend in theory. In practice though, we adapt the curriculum-speefic parameter to stretch or compress these curricula along the iteration axis to achieve the same effect. Figure (ia exemplary visualizes three decreasing and three increasing curricula, following differing transition functions C and being parametrized with Estart 1 and Cend and Cstart and Cend respectively. Furthermore, each is parametrized have a curriculum length of L 000. Figure Bb shows examples of decreasing and increasing Clin with different L.
1.0
1.0
0.8
0.8
125
linear sigmoid
0.6
250
500
0.4
expomential
1000
2000
0.2
0.2
0.0
200
400
600
800
1000
0
500
1000
1500
2000
iteration
iteration
(a)
(b)
Figure 6: Examples of different decreasing curricula and their corresponding increasing versions (a) and multiple linear curricula with different pace L (b).
4.4 Curriculum on Iteration Scale
6; prescribes a ratio of TF vs. FR steps for given training iteration Based on that solely prescribes the amount of TF for iteration, we
develop micro curricula for distributing the TF and FR steps, eventually providing a TF decision & per training step. We propose two ways to distribute TF and FR steps within one training iteration: (1) probabilistic where is interpreted as the probability of being a TF step, and (2) deterministic where as a rate that determines the number of TF steps trained before moving to FR for the rest of the training sequence. For a probabilistic CL, we denote the TF decision Φ., which is a discrete random variable that drawn from a Bernoulli distribution:
Bernoulli(e).
(9)

---

Published in Transactions on Machine Learning Research (10/2022)
For deterministic CL, + depends not only on but also the current position j within the predicted sequence of length m. Therefore, in this case we denote the TF decision Be.j as:
1, i 1
(10)
ofheewise.
5
Evaluation
To compare the training strategies described in Section 4 we evaluate each with varying parametrization on six different datasets. Our experiments aim to answer the following six research questions:
RQ1 Bascline teaching strategics. How well and consistent do the current bascline strategics FR and TF train model for forecasting dynamical systems?
RQ2 Curriculum learning strategies. How do the different curriculum learning strategies perform in comparison to the baseline strategies?
RQ3 Training length. How is training length influenced by the different tcaching strategies?
RQ4 Prediction stability. How stable model's prediction performance longer prediction hori- when trained with the different strategies?
RQ5 Curriculum parametrization. How much does the curriculum's parametrization influence model performance?
RQ6 Iteration scale curriculum. How do iteration scale curricula differ in resulting model perfor- mance?
5.1
Evaluated Curricula
In lotal, define eight strategies to evaluate in this study (cp. Fig. 2p. For comparison, train the two baseline methods TF and FR that "teach" throughout the entire training do not "teach" all respectively. All other methods prescribe teaching curriculum CL throughout the training and we distinguish these strategies along two dimensions: (1) the overall increasing (ITF), constant (CTF), or decreasing (DTF) trend in TF throughout the training curriculum and (2) the probabilistic (P) or deterministic (D) TF distribution within training steps.
5.2 Parametrization of Training Curricula
Table shows all training-ttrateeg--speefii parameters and their values for the evaluated strategics. Wo subdivide our experiments into three sets: baseline, exploralory, and essential experiments. The baseline strategies FR and TF do not have any additional parameters. The CL-CTF-P strategy has the e parameter configuring the strategy's TF ratio. The increasing and decreasing strategies CL-DTF-x and CL-ITF-x are configured by Estart and eend referring to the initial and eventual amount of TF and the function C transitioning between both. Additionally, F. determines the number of training epochs in between extart and fend- For the exploratory experiments, we utilize a fix L = 1 0000 while for the essential experiments, we evaluate all strategics solcly using a lincar transition Ctinear in the curriculum (cp. Eq. with either Estart and 6and = (increasing) or estart = and éend = (decreasing).
5.3 Performance Metrics
We use the NRMSE and the R² metrics as well as two derived of those to evaluate model performance. NRMSE is a normalized version of the Root Mean Squared Error (RMSE) where smaller values indicate better prediction performance. For a single value of sequence, NRMSE calculated as:
V Ei 0
NRMSE(y. 9)
(11)
10

---

Published in Transactions on Machine Learning Research (10/2022)
Table Curriculum strategy parameters used during the baseline, exploratory, and the essential experiments
Curriculum
Strategy
paramcter
evaluated values
C
FR
0.0
baseline
L
C
TF
1.0
L
C
CL-CTF-P
(.,,
exploratory
C
{linear, inverse sigmoid, exponential] [0.25, 0.5, 0.75, 1.0}
CL-DTF-P, CL-DTF-D
Estart - Cand
0.0
L
1000
C
(lincar, inverse sigmoid, exponential]
CL-ITF-P, CL-ITE-D
Estart + Eend
{0.0, 0.25, 0.5, 0.75)
1.0
T.
1000
C
linear
essential
CL-DTF-P, CL-DTF-D, CL-ITF-P, CL-ITF-D
Estart + Cend
{0.0 - 1.0, 1.0 - 0.0}
L
{62, 125, 250, 500, 1 1000, 2000, 4000, 8 000, 100,,3200000
where y is ground truth vector, is the corresponding prediction, dis the standard deviation across the whole datasct, and d is the size of the vectors and 9. For model evaluation, calculate the mean NRMSE over all forecasted steps of a sequence. Additionally, compute and report the NRMSE only for the last 10 forecasted steps of a sequence to specifically evaluate model performance at long prediction horizons.
The R² score lies in the range (-o0, 1] with higher values referring to better prodiction performance. A score of means that the prediction is as good as predicting the ground truth sequence's mean vector y. The R² score is computed as:
ΣS
- (1) 03)²
R² R2-1-
(12)
Ej (1) 02)
We use the R² score to compute another metric LTR > 0.9 measuring the number of Lyapunov Time (LT)s that model can predict without the Ri2 score dropping below certain threshold of 0.9. Sangiorgio and Dercole (Sangiorgio & Dercole 2020) proposed this metric while applying less strict threshold of 0.7.
5.4 Evaluated Datasets
We use six different time series datasets that we built by approximating six commonly studied chaotic systems (cp. Tab. I i.e., Mackey-Glass (Mackey & Glass 1977) Rössler (Rössler 1976). Thomas' cyclically symmetric attractor (Thomas] 1999). Hyper Rössler (Rossler 1979. Lorenz (Lorenz 1963 and Lorenz's '96 [Lorenz] 1996.. Table shows the differential equations per system and how we parametrized them. These systems differ from cach other in the number of dimensions d and the degree of chaos as indicated by the largest Iyapunov exponent in the LLE column of Tab. [ The LLEs are approximated values that were published independently in the past (Brown et al. 1991 Sprott & Chlouverakis 2007 Sano & Sawada 1985) Sandri 1996 Hart] 2003 Brajard al.) 2020). We generate datascts by choosing an initial state vector of size d and approximate 100000 samples using the respective differential equations. We use SciPy package's implementation of the Livermore solver for ordinary differential equations (LSODD)) (Radhakrishnan &
11

---

Published in Transactions on Machine Learning Research (10/2022)
Table II: Details the chaotic systems that were approximated to generate the data used for our experiments
System
ODE/DDE
Parameters
d LLE
Mackey-Glas
dr 8
1tx7 AE
t 17, 10, 7 = 0.1
0.006
with 7/ B,n
β 0.2, di 1.0
Thomas
de dy 3
sin(y) be
b = 0.1, dt = 0.1
3 0.055
sin(z) by
sin(x) bz
Rössler
de de
(y + 22
a 0.2, b = 0.2
3 0.069
dy
dE
r + az
c 5.7, dt = 0.12
dz
b + 2(x c)
Hyper Rössler
dE
1/ Z
a 0.25,
0.14
My
I + ay + w
C = 0.5, d = 0.05
di
DE
5 T2
dt = 0.1
ZE
die
C2 + due
dE
Lorenz
d
оy
0 10, 3
0.905
d
Tz
+ pi
p = 28, di 0.01
ry
de
Lorenz'96
dek
I 2k-12k41 Ik F
F= 8, dt 0.05
40 1.67
for k 1... d and x- Fd
Hindmarsh 1993 except for Mackey-Glass which we approximate through the Python module JiTCDDE implementing the delayed differential equation (DDE) integration method, as proposed by (Shampine & Thompson 2001). Thereby, defines the time difference ween two sampled states per dataset and is shown in Table II Where available, we chose similar previous studies aiming for comparability of results. We split each dataset into 80% training samples and 10% validation and testing samples respectively. All data is normalized following z-transform.
8m
Pully Connected Lager
Fally Counected Layer
Connertnd Layer
GRU Cell
GRU Cell
GRU Cell
GRU Call
GRU Cell
Encoder
Decoder
Ymr1
Figure 7: Structure of simple encoder-decoder GRU used for training with teacher forcing
5.5
Training Procedure
All evaluated models follow an encoder-decoder GRU architecture with an additional fully connected layer after the decoder (cp. Fig. 7 We performed a full grid search for the eyper-parameter learning rate, batch size, lcarning rate reduction factor, loss plateau, input length n and hidden state size determine suitable configurations for the experiments. Based on this optimization, we used the Adam (Kingma et al., (0055) optimizer with batch size of 128 and apply Reduce Learning Rate Plateau (RLROP) with initial
12

---

Published in Transactions on Machine Learning Research (10/2022)
Table 1II: Results of the exploratory tests with the best hyper-parameetr configuration per strategy and system. The arrow besides each metric's column title indicates whether smaller (4) or larger (1) values are favored. The best result values per dataset are printed in bold and the best baseline NRMSEs are underlined. Together with each dataset, we put the corresponding LLE in parenthesis.
Best performing curriculum
Trained epochs
NRMSE over 1LT
Strategy
C
absolut 4
rel. impr.
@BL epoch
last 10% +
FR TF
constant constant
0.00
427
0.03416
0.047222
Thomas (0.055)
1.00
163
0.34545
0.607954
CL-CTE-P
0.25
450
0.05535
62.03%
0.05675
0.082443
CL-DTTE-
inverse sigmoid
0.75
> 0.00
598
0.01858
45.61%
0.02120
0.034325
CL-DTEED
exponential
0.25
0.00
557
0.03229
5.47%
0.03792
0.039749
CL-ITE-P
exponential
0.00
1.00
620
0.01403
58.93%
0.02026
0,026014
CL-ITE-D
exponential
0.25
/ 1.00
944
0.01126
67.04%
0.02179
0.018571
FR
constant
0.00
363
0.00098
0.000930
Rössler (0.009)
TF
constant
1.00
500
0.00743
0.016119
CL-CTF-P
constant
0.25
2081
0.00084
14.29%
0.00084
0.001333
CL-DTF-P
linear
1.00
Yy 0.00
2 2751
0,00083
15.31%
0.00083
0,000931
CL-DTEED
inverse sigmoid
0.25
0.00
4113
0.00064
34.69%
0.00066
0.000578
CL-ITE-P
inverse sigmoid
0.00 7 1.00
7194
0.00025
74.49%
0.00034
0.000358
CL-ITE-D
linear
0.75 / 1.00
5 132
0.00024
75.51%
0.00031
0.000390
FR TF
constant
0.00
918
0.01209
0.013166
Lorenz (0.905)
constant
1.00
467
0,00152
0.002244
CL-CTE-P
constant
0.75
297
0.00167
-9.87%
0.00167
0.002599
CL-DTTE
inverse sigmoid
0.75 > 0.00
522
0.00168
-0.53%%
0.00162
0.002425
CL-DTF-D
inverse sigmoid
1.00 > 0.00
204
0.00187
23.03%
0.00187
0.002823
CL-ITF-P
linear
0.00 7 1.00
750
0.00149
1.97%
0.00217
0.002233
CL-ITF-D
inverse sigmoid
0.75 / 1.00
803
0.00124
18.42%
0.00132
0.002084
FR TF
constant
0.00
8 125
0.07273
0.126511
Lorenz '96 (1.67)
constant
1.00
4 175
0.03805
0.075583
CL-CTF-P
constant linear
0.50
2615
0.07995
-110.12%
0.07995
O. 0.140700
CL-DTE-P
0.75 > 0.00
939
0.04654
-22.31%
0.04654
0.087228
CL-ITTED
linear
0.75
0.00
177
0.04381
-15.14%
0.04381
0.08 1025
CL-ITE-P
inverse sigmoid
0.25 / 1.00
4787
0.01854
51.27%
0.02016
0.086651
CL-ITE-D
inverse sigmoid
0.00 / 1.00
3/263
0.02093
44.99%
0.02196
0.040356
learning rate of le-3 and a reduction factor of 0.6, i.e., 40% learning rate reduction, given a loss plateau of 10 epochs for all datasets except Lorenz'96, where we use a reduction factor 0.9 and a 20 epoch plateau respectively. Furthermore, we found an input length of n 150 steps and a hidden state size of 256 to be most suitable. We use early stopping with patience of 100 epochs and a minimum improvement threshold of 1% to ensure the convergence of the model while preventing from overfitting. We train all models with dataset-specific prediction length defined as:
(13)
dt
dt
LLE
The reason being that we aim train for the same forccasting horizon that we mainly evaluate a trained model with. We adapt this horizon to the dataset's LT, thereby aiming for performance measures that are comparable across datasets.
We provide plots of the training and validation loss curves of the final parametrization strategy and dataset in Appendix A Based on these loss curves, we observe for ITF in contrast. to DTF strategies that the training loss tends to move away from the validation loss faster. This explainable by the fact that with increasing training time, ITF strategics deliver an incrcasing amount TF inputs, counteracting the accumulation of error along the forecasted sequence and therefore further reducing training loss. For DTF strategies observe opposing behavior. Regarding training iterations, observe that ITF strategies
13

---

Published in Transactions on Machine Learning Research (10/2022)
typically train for a larger number of epochs. Following the TF ratio curve (c) we see that even when the increasing curriculum already arrived at the final 6. 1.0, which means that it is using solely TF training from there on, it can improve for many more epochs than by simply using TF from the beginning. Since the termination of the training is determined by the carly stopping criterion this shows that ITF may facilitate a longer and (regarding the validation loss) lypically more successful training process compared to DTF and bascline strategies. We show the corresponding curves exemplary for the Rössler training in Figure
TF
CL-ITF-P
retio
training validation
-12
treining
12
validation
10
10-1
1.0
teacher forcing rabio
0.8
0.8
0.6
0.6
10-1
10-2
teacher
0.4
0.4
10-7
0.2
10-7
0.2
10-1
0.0
0.0
100
200
300
400
500
1000
2000
3000
4000
epoch
epech
(a)
(b)
Figure 8: Loss and teacher forcing ratio curves for Rössler
5.6
Results
Tablle III shows results for the exploratory experiments. Per evaluated strategy and dataset, the table reports resulting model performance in terms of NRMSE. We only report that curriculum configuration in terms of transition function C and c schedule that yiclds the highost NRMSE per strategy and dataset. Each model was used to predict a datasst-sppteeicc horizon LT. The best result per datasel and performance metric is highlighted in bold. First, we study the baseline strategies FR and TF and observe that for two datasets, i.e., Thomas and Rössler, the FR strategy outperforms the TF bascline, while TF outperforms FR for the other two. We select the one that performs best per dataset to measure relative improvement or deterioration gained by training with the respective curriculum learning strategy (cp. column "NMRSF. rel. impr."). We observe that across all datasets and performance metrics, the CL-ITF-P and CL-ITF-D strategies yield the best and second best-performing model with a relative improvement of 1.97 - 80.61% over the best performing baseline strategy. The other curriculum learning strategies perform less consistent across the datasets. The CL-DTF-x strategies yield an improved NRMSE for half of the datasets, while the constant CL-CTF-P only yields an improvement for the Thomas attractor. We separately report the NRMSE of the last 10% predicted values of 1 LT test horizon per dataset to assess how robust prediction is over time (cp. column "NRMSE last 10%"). We observe that the CL-ITT-P and CL-ITF-D strategies also reach the best performance in terms of this metric, meaning that they yield the most robust models. We further observe diverse set of curriculum configurations yielding the best performing model per strategy and dataset. That means that all available transition functions, i.e., linear, inverse sigmoid, and exponential, have been discovered as best choice for at least one of the trained models. Further, we obscrve all evaluatod c as best choice for the CL- CTF-P strategy and one of the datasets respectively. Similarly, the best-performing initial for the increasing and decreasing transitions per dataset spans all evaluated values except for 0.5. The table also reports the number of training iterations until reaching the early stopping criterion (cp. column "training epochs"). We observe that the two bascline strategies utilize strongly differing numbers of iterations across all datasets. For the Thomas and the Rössler attractor, the TF strategy does not allow for proper model convergence, being characterized by a low number of iterations and a high NRMSE compared to the other strategies.
AA reproduction package for the experiments is available on Githuh https://sthhub.co//pp/////////// classroom. The datasets used in this paper are published on Dataverse https://bnitrg///1999/////V
14

---

Published in Transactions on Machine Learning Research (10/2022)
Table IV: Best curriculum length per strategy and system for all six datasets. The arrow besides a metric's column title indicates whether smaller (4) or larger (t) values are favored. The best result values per dataset are printed in bold and the best baseline NRMSEs are underlined. Together with each dataset, we put the corresponding LLE in parenthesis.
Best performing curriculum
Trained epochs
NRMSE over 1LT
#LT with
Strategy
absolut.
rel. impr.f
@BL epoch..
last 10%4
R > 0.9
Mackey Glass (0.006)
FR
0.00
4713
0.00391
0.00391
0.004101
4.50
TF
1.00
44
0.09535
0.09535
0.171945
1.64
CLCTTFP
0.25
2918
0.00632
-61.64%
0.00632
0.006544
4.51
CL-DTF-P
1.00 V 0.00
2000
3 773
0.00215
45.01%
0.00215
0.003010
4.95
CL-DTF-D
1.00
0.00
1.00
431
0.00585
-49.22%
0.00585
0.011022
3.91
CL-ITF-P
0.00
7 1.00
500
1 1566
0.00104
73.40%
0.00104
0.001793
5.18
CL-ITF-D
0.00
1.00
500
1 8008
0.00211
46.03%
0,00211
0.003032
4.99
FR
0.00
427
0.03416
0.034166
0.047222
2.04
Thomas (0.055)
TF
1.00
163
0.34545
0.34545
0.607954
1.73
CL-CTF-P
0.25
450
0.05535
62.03%
0,05675
0.082443
1.53
CL-DTF-P
1.00 S. 0.00
1 000
356
0.05084
-4.83%
0.05084
0.105585
2.13
CL-DTF-D
1.00
0.00
1 0.00
326
0.10712
213.58%
0,10712
0,206923
1.53
CL-ITF-P
0.00
1.00
500
677
0.00930
72.78%
0.01645
0.016729
3.99
CL-ITE-D
0.00 7 1.00
500
649
0.01819
46.75%
0.03934
0.030589
2.05
FR
0.00
3 863
0.00098
0,00098
0.000930
9.46
Róieller (0.009)
TF
1.00
500
0.00743
0,00743
0.016119
4.75
CL-TT-PP
0.25
2081
0.00084
14.29%
0.00084
0.001333
7.51
CL-DTF-P
1.00
0.00
10000
2 751
0.00083
15.31%
0.00083
0.000931
8.46
CL-DTF-D
1.00
5. 0.00
125
4 888
0.00100
-2.04%
0,00116
0.000947
9.28
CL-ITF-P
0.00
7 1.00
500
4 123
0.00019
80.61%
0,00022
0.000303
10.23
CL-ITE-D
0.00
/ 1.00
4000
7 267
0.00027
72.24%
0.00051
0.000368
9.41
Hyper Rössllr (0.14)
FR
1.00
6 461
0.00599
0.00599
0.007011
6.57
TF
0.00
2 788
0.00435
0,00762
0.011194
5.24
CL-CTE-P
0.25
2.999
0.01450
233.33%
0.01450
0.015944
5.21
CL-DTF-P
1.00
0.00
2000
3773
0.00560
28.74%
0.00560
0.007052
6.32
CL-DTE-D
1.00
0.00
16:000
1 793
0.00490
12.64%
0.00490
0.007471
6.30
CL-ITF-P
0.00
1.00
125
2 2002
0.00366
15.80%
0,00366
0.005802
6.50
CL-ITF-D
0.00 / 1.00
250
3317
0.00326
25.06%
0,00326
0.004639
6.72
FR
0.00
918
0.01209
0.01319
0.013166
3.31
Lovenz (0.9055)
TF
1.00
467
0.00152
0.00152
0.002244
6.72
CL-CTF.P
0.75
297
0.00167
-9.87%
0.00167
0.002599
6.43
CL-DTF-P
1.00
0.00
4.000
450
0.00121
18. 42%
0.00124
0.001925
6.64
CL-DTE-D
1.U
0.00
160000
587
0.00111
26.97%
0.00127
0.001650
6.53
CL-ITE-P
0.00
7..0
250
137
0.00060
60.53%
0.00124
0.000883
7.19
CL-ITF-D
0.00 7 1.00
250
578
0.00135
11.18%
0.00189
0.001725
4.33
0.00
125
0.07273
0,08362
0.126511
2.34
Lorenz'S (1.07)
TF
1.00
4 175
0.03805
0.03805
0.075583
3.01
CLCTTF.P
0.50
2615
0.07995
-110.12%
0.07995
0.140700
CL-DTF-P
1.0 N, 0.00
1000
988
0.052278
-38.71%
0.05278
0.098130
CL-DIF-D
1.00
0.00
1000
4083
0.07119
-87.10%%
0.07119
0.126636
2.34
CL-ITE-P
0.00
1.00
250
3 3886
0.01680
55.85%
0.01680
0.032439
4.01
CL-ITF-D
0.00 / 1.00
250
3379
0.01628
57.21%
0.01628
0.031464
4.18
Among the curriculum teaching strategies across all datasets, the strategies with increasing TF ratio CL- ITF-x utilize the most training iterations. These CL-ITF-x strategies also utilize more training iterations than the netter-performing baseline strategy across all datasets. To better understand whether the longer training is the sole reason for the higher performance of the CL-ITF-x trained models, we additionally report the performance in terms of NRMSE of all curriculum-trained models after the same number of training iterations as The better performing baseline model, i.e., after 427 epochs for Thomas, after 3 863 epochs for Rössler, after 467 epochs for Lorenz, and after 4 175 epochs for Lorenz'96 (cp. column "Performance @BL epochs"). We observe across all datasets that the cest-performing teaching strategy still remains CL-ITF-P or CL-ITF-D. In conclusion, the exploratory experiments demonstrated that well-parametrized CL-ITF-x strategy yields 18.42 75.51% performance increase across the evaluated datasets.
15

---

Published in Transactions on Machine Learning Research (10/2022)
1.00
FR
1.00
FR
TF
CL-CTF-P
CL-CTT-P
0.96
CL-DIF-P
0.96
CL-DIF-P
CL-DIT-D
CL-DIF-D
0.94
CL-ITF-P
0.91
CL-ITF-P
0.82
CL-ITF-D
0.92
CL-ITF-D
thresbold
thresheld
0.86
0.86
4 LT
2
LT
(a) Mackey-Glaase
(b) Thomas
1.00
1.00
0.96
0.96
FR
TF
0.94
0.94
FR
CL-CTT-P
CLCCTFP
0.92
CL-DIF-P
0.92
CL-DIF-P
CL-DIF-D
CL-DTF-D
CL-ITF-P
CL-ITF-P
CL-IIF-D
0.88
CL-ITF-D
0.86
threshold
0.86
thresbold
10
12
LT
LT
(c) Rössler
(d) Hyper Rössler
1.00
1.00
FR
TF
CL-CTE-P
0.96
0.96
CL-DTF-P
FR
CL-DTF-D
0.94
0.94
CL-ITF-P
CL-CTF-P
0.92
CL-DIF-P
0.92
CL-ITF-D
thresheld
CL-DIF-D
CL-ITF-P
0.88
CL-IIF-D
0.88
0.86
threshold
0.86
8
5
LT
(e) Lorenz
(f) Lorenz' 96
Figure R2 score over multiple LTs for the six studied datasets using eight different training strategies
However, this improvement comes at the cost of an intensive parameter optimization of the respective cur- riculum. Therefore, we run second series of essential experiments in which we simplify the parametrization
of the curriculum by utilizing a linear transition from either 0.0 + 1.0 (CL-ITF-x) or 1.0 - 0.0 (CL-DTF-x). that is solely parametrized by the length of this transition in terms of training epochs E. Table IV reports results in terms of the previously introduced performance metrics again, measured over a prediction horizon of LT and across the same teaching strategies for six datasets, including those studied for the exploratory experiments. Since the changes of the essential over the exploratory experiments solely affect teaching strate- gies with TF as well the constant curriculum CL-CTF-P, which still report in Table IV for direct comparison. Overall, we observe that CL-ITF-P outperforms all other strategics for four out of six datascts, while it performs second best for the remaining two datasets where the deterministic version CL-ITF-D performs best. These strategies yield relative improvements ranging from 25.06 80.61% and are, thus, even higher than those observed for the exploratory experiments. Beyond that we obscrve that for all datasets treated in both experimental sets, the training curricula used in the essentiol experiments yield better performing models. For three out of four datasets, the training even requires substantially less training iterations than
curriculum, they have effect the baseline strategies FR and
16

---

Published in Transactions on Machine Learning Research (10/2022)
in the explorative experiments. Additionally, we report in column "#IT with R > 0.9" the prediction horizon in terms of I,T that a trained model can predict while constantly maintaining an R² > 0.9. We observe that the ranking between the different strategies mostly remains the same as those observed when prodicting LT. That means that the best-performing strategy consistently remains the same for the longer horizon. Figure 9 more concretely depicts how the R2 score develops across the prediction horizon for the different teaching strategics and datascts.
5.7 Additional Experiments
Judging from the essential experiments, CL-ITF-x are our winning strategies on the datasets we tested. However, as mentioned in Section B there are many other approaches that predict chaotic systems with adapted RNNs architectures that take theoretical insights of dynamical systems into account. STF [Monfared ct al. 2021 does not require any architectural modifications but instead provides an adaptod training strategy. It determines time interval т LLE the next TF value is used within one sequence. We find this strategy comparable to the CL approaches we study. Therefore, we execute another set of experiments where we use STF during the training of our incoder-decoder GRU for all chaotic systems in Table II Since our data is sampled with different dt, we have to redefine the time interval as T LLE-di
in2 that denotes how many FR steps are processed before
In2
The results (cp. Tab. V show that STF provides improved performance compared to the best baseline for three of six datasets ranging from 26.21 46.75% relative improvement. For this it requires no additional prper-parameters if the system's LLE is known. It also beats the best-performing CL strategy on the Hyper- Hyper-Rössler dataset by margin of 1.15%. For the rest of the datasets, the results stay behind those of the CL-ITF-x strategies, showing a worse, i.., increased, NRMSE by 7.00- 236.18% We assume that where STF systematically induces TF to catch chaos- prevsttieenniing exploding gradients before they appear, using knowledge about the processed data, CL helps the model to find more consistent minima while not taking the degree of chaos into account. Originally, STF was not proposed to work for the kind of sequence- 2-sequence architecture we are using, which may limit its effectiveness here. We further hypothesize that the GRU is able to keep the risk of exploding gradients low in many cases, due to its gating mechanism and thus prevents STF to really show its full strength here.
For further investigation on CL for non-chaotic systems and to enrich our experiments, we conduct additional experiments that include the application of the baseline strategies TF and FR together with CL-ITF-P and CL-ITF-D on a periodic system and a measured real-world dataset. We use CL-ITF-P and CL-ITF-D, since they provide the most consistent relative improvements in the essential experiments. As periodic system, we study the Thomas attractor (Thomas, 1999) with parameter b = 0.32899, which ensures a periodic behavior. Extending our evaluation empirical data, we selected a time series used in the Santa Fe Institute competition (Weigend & Gershenfeld 1993
The results in Table VI support our assumption that CL-ITF-x strategics are also successfully applicable for time series data originating dynamical systems with periodic behavior, achieving relative improvements of 21.05 - 42.11%. Regarding the Santa Fe dataset we observe less impact by our strategies. Only having an improvement by 5.90% for CL-ITF-P and a worsening by 2.86% for CL-ITF-D on the empirical real- world data. When increasing the prediction horizon from 20 to 200 steps, we observe an increased relative improvement of 26.27% for CL-IITFPP. At the same time, we observe relative decrease of -5.89% for CL-ITF-D. To investigate the radical difference between these results, we study the in relation to the appearance of TF steps in the respective training curriculun. Figure 10 shows two different probabilities along the training epochs. There is the TF probability e that is directly determined by the curriculum function and Pe200, which denotes the probability of a training sample to contain at least one TF step. It is computed as P52000 Em (1 e) for the probabilistic strategy. For CL-ITFFD, this probability is either 0 or 1, meaning that it only has an effect on training epochs > 160. If we would use the same L as for CL-ITF-P here, the training would converge after 208 epochs just as in the plain FR case. CL-ITF-P though, due to its probabilistic nature, can sporadically induce TF steps for some of the batches in the
17

---

Published in Transactions on Machine Learning Research (10/2022)
Table V: Results of STF with those of the baseline and the CL-ITF-x strategies
System
Strategy
Epochs
NRMSE +
Rel. impr. 1
FR
473
0.00391
TF
44
0.09535
Mackey-Glas (0.006)
CL-ITE-P CL-ITF-D
1 1566
0.00104
73.40%
1 1088
0.00211
46.03%
STF
5517
0.00254
35.04%
FR
427
0.03416
TF
163
0.34545
Thomas (0.055)
CL-ITE-P
677
0.00930
72.78%
CL-ITF-D
649
0.01819
46.75%
STF
432
0.03655
-7.00%
FR
3 3663
0.00098
TF
500
0.00743
Rössler (0.069)
CL-ITF-P CL-ITF-D
4 523
0.00019
80.61%
7667
0.00027
72.24%
STF
796
0.00065
33.67%
FR
6461
0.00599
TF
2788
0.00435
Hyper-Rössler (0.14)
CL-ITF-P CL-ITF-D
2802
0.00366
15.86%
3317
0.00326
25.06%
STF
2645
0.00321
26.21%
FR
918
0.01209
TF
466
0.00152
Lorenz (0.905)
CL-ITF-P
1137
0.00060
60.53%
CL-ITF-D
578
0.00135
11.18%
STF
1 153
0.00511
18%..18%
FR
8 125
0.07273
TF
4155
0.03805
Lorenz '96 (1.67)
CL-ITF-P
3866
0.01680
55.85%
CL-ITF-D
3379
0.01628
57.21%
STF
1478
0.09030
-137.32%
earlier phase of training and thus provides a much smoother transition towards TF phase. We argue that this smoothness crucial for the given task, which appears to be very sensitive TF training steps.
We also conducted experiments for other RNNN architectures, i.e., we selected a vanilla RNN, an LSTM, unitary evolution RNN (uRNN) (Arjovsky et al. 2016) and Lipschitz RNN (LRNN) (Erichson et al. 2020), in the same encoder-decoder setup applied for the previous experiments with the GRU architecture. In the Table VII we compare the NRMSE and relative improvement of these architectures on the four chaotic dalasets from the exploratory experiments (cp. Tab. II We compare TF, FR and previously best- performing training strategies CL-ITF-P and CL-ITF-D (cp. Tab. IV Except for one case, we observe that both CL strategics outperform the respective best-performing bascline stratogy on the vanilla RNN well the LSTM architecture. The only exception is a vanilla RNN trained forecast the Lorenz '96 system using CL-ITF-P. This setup suffers performance decrease by 83.45%, while the NRMSE in all other experiments
18

---

Published in Transactions on Machine Learning Research (10/2022)
Table VI: Comparing baseline strategies and CL-ITE-P on periodic Thomas and measured Santa Fe laser dataset
Strategy
Epochs
NRMSE +
Rel. impr. 1
FR
542
0.00057
Per. Thomas
TF
542
0.00107
CL-ITE-P
800
794
0.00033
42.11%
CL-ITF-D
125
326
0.00045
21.05%
FR
500
0.02170
Santa Fe (m 20)
TF
22
0.04793
CL-ITE-P
32000
469
0.02042
5.90%
CL-ITF-D
4000
536
0.02232
-2.86%
FR
208
0.2786
Santa Fe (m 200)
TF
0.7104
CL-ITE-P
64 000
1506
0.2054
26.27%
CT-ITF-D
32 000
201
0.2950
-5.89%
1.0
strategy
0.006
CL-ITF-P
0.8
CL-ITF-D
0.005
0.6
0.004
P5300
0.003
0.002
0.2
50
100
150
200
epoch
Figure 10: The probability course of a sample to include at least one TF step (solid) and the corresponding TF probability (dashed). Exemplary shown for the Santa Fe setup with 200 output steps (cp. Tab. VTb.
improves by 37.83 78.28% RNN and 26.04 69.75% LSTM respectively when using the CT. strategies. The uRNN and the LRNN do adapt their weight matrices directly while training, but rather update several structured building block matrices construct the actual weight matrices. We observe improvements from 0.4 to 77.75% and from 0.21 to 83.74% for the two architectures respectively. We provide more detailed information about these experiments in appendix C
In our experiments on the Lorenz '96 data, the performance of the encoder-decoder RNN remains an out- lier. Especially when we consider that for the deterministic variant CL-ITF-DD there a clear NRMSE improveeent by 75.28%. We use this setup to take closer look on the training gradients and process. In Figure 1a plot the mean of the decoder RNN gradient norm (I/VWall) over the training epochs for the CL-ITF-P and the CL-ITF-D case. We also add the TF probability and the learning rate 7 to the plot. For comparison with the two successful setups, we also give overview of the gradient norm of the GRU and the LSTM- model under the same conditions (cp. Figs
1c and l1b
For all models, we observe an oscillation of the gradient norm during CL-ITF-P training. CL-ITF-D avoids this behavior with a constant iteration scale curriculum that keeps the TF-FR-pattern fix for several epochs between the changes. The crucial epochs are indicated by green dashed vertical lines. In contrast, CL-ITF-P provides probabilistic iteration scale curriculum that leads intra-epoch changes. The resulting variation
19

---

Published in Transactions on Machine Learning Research (10/2022)
in the gradient norm only stops when the training scale curriculum hits the final c == 1. Preceding that, the oscillating gradient norm originates from an unsteady training loss. Most of the time, the unsteady loss can be compensated due to the increasing amount of TF over the epochs. For the Lorenz 96 data, however, it scems the unstcady loss together with the used scheduler lead to a rapid decrease of the learning rate. This effect observable during the curriculum training of the basic RNN, the LSTM and the GRU. The latter two models seem to handle the resulting minimal learning rate of 3e-6 much better and manage to continue their training successfully.
CLIFDD
10
10a
10a
CLITF-P
10a
10a
10a
10a
epoch
(a) RNN
10"
107
CL-IT-D
CL-ITFD
100
10
10"
10a
CL-TF-PP
VW
10-
10a
104
10a
10a
10
10a
109
10a
10*
epoch
epoch
(b) LSTM
(c) GRU
Figure 11: Course of the average decoder gradient norm VWal| during training on Lorenz data logether with TF probability e and learning rate 7).
10a
wt Ir schedulerr
10:4
10a
10a
wio ir scheduler
10:2
train,
val. loss
10
10:4
101
10a
10a
10a
epoch
Figure 12: Course of the training and validation loss for the RNN on Lorenz '96 data using CL-ITTF-P with and without learning rate scheduler.
20

---

Published in Transactions on Machine Learning Research (10/2022)
To get an impression of the RNN's behavior without the unwanted learning rate drop, we conducted the same CL--ITF-P experiments again without applying any learning rate scheduler. This time we plot the training loss, that exhibits the same oscillating behavior as the gradient norm, together with the validation loss, to see the change lcads to improved loss values. The plot in Figure 12 shows that without the unwanted learning rate drop, the basic RNN remains stable during the transient. phase lowards = and ends up with a validation loss 0.012 instead of 0.373. For the more sophisticated RNNs, we see clearly improved validation losses after much less epochs well. The corresponding plots are shown in appendix B
Table VII: Forecasting performance of additional RNNs (vanilla RNN, LSTM, unitary evolution RNN (uRNN) and Lipschitz RNN (LRNN)) on the different chaotic datasets. Given is the NRMSE with rel- ative improvement in parenthesis for the CL strategies.
System
Strategy
RNN
LSTM
uRNN
LRNN
FR
0.48117
0.43698
0.74991
0.11697
TF
0.41274
0.05265
0.85679
0.05714
Thomas
0.17955
0.01892
0.61889
0.05237
CL-ITF-P
(56.50%)
(64.06%)
(17.47%)
(8.35%)
0.25659
0.01181
0.73331
0.03926
CL-ITE-D
(37.83%)
(77.57%)
(2.21%)
(31.22 (1..99)
FR
0.00747
0.00210
0.03726
0.27122
TF
0.50174
0.00139
0.03317
0.90844
Rössler
0.00283
0.00063
0.02473
0.05185
CL-ITF-P
(62.12%)
(54.68%)
(25.44%)
(80.8%%)
0.00319
0.00085
0.02420
0.04409
CL-ITF-D
(57.30%
(38.85%)
(27.04%)
(83.74%)
FR
0.11389
0.06526
0.13021
0.06675
TF
0.00913
0.00169
0.96357
0.02830
Lorenz
0.00603
0.00075
0.03098
0.01732
CL-ITF-P
(33.95%)
(55.62%)
(76.21%)
(38.80%)
0.00378
0.00125
0.02897
0.02417
CL-ITF-D
(58.60%)
(26.04%)
(77.75%)
(14.59%)
FR
0.31002
0.13010
0.26359
0.62710
TF
0.57870
0.22757
0.61159
0.65122
Lorenz '96
0.56872
0.03935
0.26253
0.62170
CL-ITT-P
(-83.45%)
(69.75%)
(0.4%)
(0.86%)
0.07663
0.06855
0.18483
0.62581
CL-ITF-D
(75.28%)
(47.31%)
(29.88%)
(0.21%)
6 Discussion
Baseline teaching strategies (RQ1). Considering the baseline teaching strategies FR and TF, we observe that per dataset, of the strategies performs substantially better than the other. We also observe that for the upper, based on their LLE, less chaotic datasets in TV performs better, while for the lower, more chaotic datasets, TF yields the better-performing model. However, a larger study with more datasets would be required to justify this claim. Our takeaway that none of the methods can be universally recommended again motivating curriculum learning strategies.
Curriculum learning strategies (RQ2). Among the curriculum learning strategies, we observe that blending FR with constant ratio of TF, i.e., CL-CTE-P, almost consistently yields worse results than
21

---

Published in Transactions on Machine Learning Research (10/2022)
the best- best-performing baseline strategy and we therefore consider the strategy not relevant. The decreasing curricula. CL-DTF-x that start the training with a high degree of TF and then incrementally reduce it to pure FR training partly perform better than the CL-CTF-P strategy and for a few datasets even substantially better than the baseline. We also proposed and studied inereasing curricula CL-ITF-x that start the training with no or a low degree of TF, which is then incrementally increased over the course of the training. We observe that these strategics consistently outperform not only the baseline strategies, but all other tested curriculum learning strategies as well. However, it was not foreseeable when this would be the case, making their application not suitable for new datasets without lot of experimentation and tuning.
Training length (RQ3). Choosing an improper teaching strategy can result in an carly convergence on a high level of generalization error, e.g,, TF strategy for Mackey-Glass, Thomas, and Rössler. Models that yield better performance typically train for more iterations (cp. Tab. I and [V]. However, a longer training may not necessarily yield a better performance, e.g., FR vs. TF for Lorenz. When considering the best-performing CL-ITF-x strategics compared to the best-performing bascline strategy, we observe moderately increased training ilerations for some datasets, i.e., Thomas, Rössler, Hyper Rössler, Lorenz, but also decreased training iterations for other datasets, ie., Mackey Glass and Lorenz'96. To better understand whether the longer training is the true reason for the netter-performing CL-ITF-x models, we compared their performance when only trained for as many iterations as the baseline model and still observe superior performance over it. In conclusion, we observe that the CL-ITF-x strategies facilitate a robust training, reaching a better generalizing model in a comparable training time on all six tested datasets.
Prediction stability (RQ4). We evaluated the generalization as NRMSE for all models trained with the different training strategies per dataset, while forecasting dataset-specific horizon of 1 LT. However, this metric reflects only an average. When we strive for higher model performance task, we often aim for a longer prediction horizon at an acceptable error. To compare prediction stability, we report the NRMSE metric separately, solely computed on the last 10% of the LT horizon. Additionally, we computed how many LTs per datascts can be predicted before falling below an R² of 0.9. We found that the CL-ITF-x strategies yielded the lowest NRMSE of the last 10% predicted values across all datasets and even more promising that these strategies facilitated the longest prediction horizon without falling below R² = 0.9. We conclude that the CL-ITF-x strategies may help models find more stable minima. in training, substantially improving their long-term forecasting performance.
multi-step prediction
Curriculum parametrization (RQ5). Initially, we evaluated curriculum learning strategies with a variety of different transition functions and individual start and end TF rate (cp. exploratory experiments). In these experiments, we observed high prediction performance of the CL-ITF-x strategies, but with a diverse dataset- specific best-performing curriculum, meaning that the application of these strategies for new datasets would have necessitated an extensive prper-parameter search. In a second set of essential experiments, we therefore explored whether we could identify curricula with less parametrization and a similar performance. We found those by using a linear transition that is solely configured by a single parameter L that determines the pace with which the TF increases or decreases over the course of the training. We found that these curricula were not only comparable to the previous transition functions and their parametrization but performed better for all four datasets that we evaluated in both experimental sets and yielded the cest-performing model across all six datasets in the second experiment.
Iteration scale curriculum (RQ6). Having the CL-ITF-x strategies outperforming the CL-DTF-x strate- gies leads to rethinking the hitherto common intuition of supporting the early phases of training by TF and moving towards FR in the later stages of training. Rather, we hypothesize that this lures the model into regions of only seemingly valid and stable minima in which they stay even during the more FR-heavy epochs, resulting in a premature termination of the training. This hepothesis is supported by the resulting loss curves for the respective experiments. For example, for the increasing curriculum case, the training duration and the course of the e (c.p., Fig. show that a training that starts off with FR can enable the model to train much longer in the pure TF zone later. This is compared to the case where solely TF is used. The difference between the two CL-ITT-xx strategies is how the prescribed amount of TF is distributed across the prediction steps of one training iteration (aka epoch). While the CL-ITF-D strategy distributes them as one cohesive sequence, the CL-ITF-P strategy distributes them randomly across the training sequence. We found that in the essential experiments with the linear transition, the CL-ITF-P strategy performed overall best for four
22

---

Published in Transactions on Machine Learning Research (10/2022)
of the six datasets and would have also been a good choice with substantial gain over the best-performing baseline training strategy for the other two datasets. In conclusion, we observe that the CL--ITF-P strategy trains models that yield 16 81% higher performance than a conventional training with FR or TF. Apart from that, the essential results do not lead to a clear conclusion whether to use CL-ITF-P or CL-ITF-D in a given case. The above-ment ioned most obvious difference in the distribution of TF steps firstly may lead to a more coherent backpropagation in the deterministic variant, but it also results in a different behavior regarding maximum number of consecutive FR steps (TF-gap) for a given e. Having the same curriculum function applied for CL-ITE-P and CL-ITF-D makes the TF-gap decrease much faster in the early training stage for the probabilistic variant. Further, it changes the TF-gap in a logarithmic, rather than a linear fashion as for CL-ITF-D. This difference cannot be compensated by parametrizing the curriculum length, demonstrating the need for both strategies. Plus, this only affects the mean TF-gap produced by CL-ITT-P, which has a variance of 1 due to its geometric distribution. Therefore, the TF-gap also varies a lot in the carly training stage. In the context of the result Tables V and V we see the probabilistic and the deterministic lype of CL-ITF-x can exhibit disfunctions in some specific cases. More thorough investigation of such cases is required to further improve CL strategies, making them more generally applicable and fully identifying their limitations.
Limitations of this work. Our observations allow us to draw conclusions regarding appropriate curricula for the training of sequence-to-sequence RNN on contimuous time series data. More precisely, the data in our study originates from dynamical systems that predominantly impose chaotic behavior. Consequently, all observations made here are currently limited to these kind of models and data. We acknowledge that more research is necessary to clarify several uncertain points. First, regarding the question why an increasing curriculum tends to find more stable minima throughout all studied datasets. Leading to the question, what determines a proper curriculum and parametrization. A first step could be to shift the focus on the baseline strategies again to extract primary conditions under which trainings get unstable. Therefore, a closer look at the weights and behavior of the model gradients during training, the statistics of the gradient of the processed time series and the used sampling rate is required at least. Finally, a more thorough investigation on empirical real-world data improving on the early and inconclusive results on the Santa Fe dataset (cp. Tab. VI) is needed to reveal the capabilities and limitations of different CL strategies in this regard.
A Application in other domains. In this paper we focus exclusively on the field of time series forecasting. However, the proposed strategies could easily be applied to MT. models exhibiting autoregressive predic- tions trained for other domains. For example, eequence-to-sqquen Machine Translation (AMT). As mentioned before, schoduled sampling, that is included in the CL-DTF-x strategies, was originally proposed, and successfully used to improve the training of RNNs for NLP, but, on the other hand, performed rather poorly for time series forccasting in our experiments. Thus, we cannot draw any conclusions about the usefulness of CL-ITF-x strategies in other domains based on our results either. Therefore, evaluating the effectiveness of increasing curricula in other domains remains a task for future work and, at this point, would go beyond the focus of our study.
RNNs and transformers for Automatic
7 Conclusions
While training encoder-decoder RNNs to forecast time series data, strategies like TF produce a discrepancy between training and inference mode, i.e., the exposure bias. Curriculum learning strategies like scheduled sampling or training directly in FR mode are used to reduce this effect. We run an extensive series of experiments using six chaotic dynamical systems as benchmark and observed that not for all of them the exposure bias yields a crucial problem. We see that even an FR training that is per definition free of exposure bias may not be optimal. In fact, we observed that for half of our datasets, TF performs better, while for the other half, FR yields better results. With the focus on curriculum lcarning strategies like scheduled sampling, we proposed two novel strategies and found that those yield models that consistently outperform the best-performing baseline model by a wide margin of 15 15-80% NRMSE in a multi-step prediction scenario over one Lyapunov time. We found that these models are robust in their prodiction, allowing to forecast longer horizons with a higher performance. We found it sufficient to parametrize the strategy with a single additional parameter adopting the pace of the curriculum.
23

---

Published in Transactions on Machine Learning Research (10/2022)
Acknowledgments
This rescarch was funded by the project P2018-02-001 "DeepTurb - Deep Learning in and of Turbulence" of the Carl Zeiss Foundation; the German Federal Ministry for the Environment, Nature Conservation, Nuclear Safety and Consumer Protection (BMUV) grant: 67KT2086A; and the German Ministry of Education and Research (BMBF) grant: 011S20062B.
References
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In Inter- national conference on machine learning, pp. 1120-1128. PMLR, 2016.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 1171-1179) 2015.
Julien Brajard, Alberto Carrassi, Marc Bocquet, and Laurent Bertino. Combining data assimilation and machine learning to emulate dynamical model from sparse and noisy observations: A case study with the lorenz 96 model. Journal of Computational Science, 44:101171, 2020.
Reggie Brown, Paul Bryant, and Henry DI Abarbanel. Computing the Iyapunov spectrum of dynamical system from an observed time series. Physical Review A, 43(6))2787, 1991.
Tom В Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint Xr:e::2005. 14165, 2020.
Kathleen Champion, Bethany Lusch, J Nathan Kutz, and Steven L Brunton. Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, 116(45):22445- 22451, 2019.
Bo Chang, Minmin Chen, Eldad Haber, and Ed H Chi. Antisymmetricrm: A dynamical system view recurrent neural networks. arXiv preprint orxiv:1902. 190::1990.00689, 2019.
Kyunghyun Cho, Bart Van Merriënboer, Damitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXie preprint 1.X.i:1400.11200 2014.
Junyoung Chung, Caglar Gulcehre, KyungHyuun Cho, and Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXic:1412.355: 12 2014.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0448 2018.
Jonathan B Dingwell. Lyapunov exponents. Wiley encyclopedia of biomedical engincering, 2006.
Qingyun Dou, Yiting Lu, Joshua Efiong, and Mark JF Gales. Attention forcing for sequence-to-sxqqeene model training, arXiv preprint XrXi::1909. 12289, 2019.
Konstantinos Drossos, Shayan Gharib, Paul Magron, and Tuomas Virtanen. Language modelling for sound event detection with teacher forcing and scheduled sampling. arXiv preprint arXiv:1907.0885 2019.
N Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W Mahoney. Lipschitz recurrent neural networks. ar preprint ar Xrx:::2006.1 12070, 2020.
William Gilpin. Chaos as an interpretable benchmark for forecasting and data-driven modelling. arXiv preprint ar arXiv:2110.05200 2021.
Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27:2672 2-2202226680 2680, 2014.
24

---

Published in Transactions on Machine Learning Research (10/2022)
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Haohan Guo, Frank K Soong, Lei He, and Lei Xic. A new gan-based end-to-end tts training algorithm. arXiv preprint Xim: 1904.94444440 04775, 2019.
Michael D Hartl. Lyapunov exponents in constrained and unconstrained ordinary differential equations. arXiv preprint (oysics/03030777 2003.
Tianxing He, Jingzhao Zhang, Zhiming Zhou, and James Glass. Quantifying exposure bias for neural language generation. Xiv preprint arXis:1905.16611 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-178) 1997.
Martin Hofmann and Patrick Mäder. Synaptic scaling-an artificial neural network rogularization inspired by nature. IEEE Transactions on Neural Networks and Leurning Systems, 2021.
Anil Kag and Venkatesh Saligrama. Time adaptive recurrent neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 15149 11149-15158, 2021.
Anil Kag, Ziming Zhang, and Venkatesh Saligrama. Rnns incrementally evolving on equilibrium manifold: A panacea for vanishing and exploding gradients? In International Conference on Learning Representa- tions, 2019.
DP Kingma, LJ Ba, et al. Adam: A method for stochastic optimization. 2015.
Georgia Koppe, Hazem Toutounji, Peter Kirsch, Stefanie Lis, and Daniel Durstewitz. Identifying nonlincar dynamical systems via generative recurrent neural networks with applications to fmri. PLoS computational biology, 15880ee107726, 2019.
Alex M Lamb, Anirudh Goyal Alias Parth Goyal, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. Professor forcing: new algorithm for training recurrent networks. In Advances in neural information processing systems, pp. 4601 4609, 2016.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anaudkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXie:2010. 08895, 2020.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stu- art, and Anima Anandkumar. Markov neural operators for learning chaotic systems. arXio preprint arXiv:2106.0688 06898, 2021.
Rui Liu, Berrak Sisman, Jingdong Li, Feilong Bao, Guanglai Gao, and Haizhou Li. Teacher-student training for robust tacotron- basod tts. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6274 1278. IEEE, 2020.
Edward N Lorenz. Deterministic nonperiodic flow. Journal of the atmospheric sciences, 20(2):130-141, 1963.
Edward N Lorenz. Predictability: A problem partly solved. In Proc. Seminar on predictability, volume 1, 1996.
Bethany Lusch, J Nathan Kutz, and Steven T. Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9(1):1-10, 2018
Michael C Mackey and Leon Glass. Oscillation and chaos in physiological control systems. Science, 197 (4300)) 1977.
Chenfeng Miao, Shuang Liang, Minchuan Chen, Jun Ma, Shaojun Wang, and Jing Xiao. Flow-tts: A non-autoregressive network for text to speech based on flow. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (TCASSP), pp. 7209 7213. IEFF, 2020.
25

---

arXiv preprint Published in Transactions on Machine Learning Research (10/2022)
Tsvetomila Mihaylova and André FT Martins. Scheduled sampling for transformers. arXiv:190.007651 2019.
Zahra Monfared, Jonas M. Mikhaeil, and Daniel Durstewitz. How to train ris on chaotic data? arXiv preprint arXiv:2110.0723 07238, 2021.
Garrett Nicolai and Miikaa Silfverberg. Noise isn't always negative: Countering exposure bias in sequence- to-sequence inflection models. In Proceedings of the 28th International Conference Computational Linguislics, pp.
2020.
Jakub Nowak, Ahmet. Taspinar, and Rafal Scherer. Lstm recurrent neural networks for short text and sentiment classification. In International Conference on Artificial Intelligence and Soft Computing, pp. 553-562. Springer, 2017.
Sandeep Pandey, Philipp Teutsch, Patrick Mäder, and Jörg Schumacher. Direct data-driven forecast of local turbulent heat flux in rayleigh- bénard convection. Physics of Fluids, 34(4):045106, 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation machine translation. In Proccedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311 318, 2002.
Jaideep Pathak, Zhixin Lu, Brian R Hunt, Michelle Girvan, and Edward Oit. Using machine learning to replicate chaotic attractors and calculate Iyapunov exponents from data. Chaos: An Interdisciplinary Journal of Nonlinear Science, 27(12)122110,, 2017.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Nya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Krishnan Radhakrishnan and Alan C Hindmarsh. Description and use of Isode, the livermore solver for ordinary differential equations. 1993.
Mare' Aureloo Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.00 1.06322, 2015.
0E5889510996 Rossler. An equation for hyperchaos. Physies Letters A, 71 1979.
Otto E Rössler. An equation for continuous chaos. Physics Letters A, 57(5): 397-333933333 -38, 1976.
T Konstantin Rusch and Siddhartha Mishra. Coupled oscillatory recurrent neural network (cornn); An accurate and (gradient) stable architecture for learning long time dependencies. ar preprint STEi::00000095:: 2020.
T Konstantin Rusch, Siddhartha Mishra, N Benjamin Erichson, and Michael W Mahoney. Long expressive memory for sequence modeling. arXie preprint 1xXiv:2100007// 2021.
Marco Sandri. Numerical calculation of lyapunov exponents. Mathematica Journal, 6(3):78 84, 1996.
Mattco Sangiorgio and Fabio Dercole. Robustness of Istm neural networks for multi-stop forecasting of chaotic time series. Chaos, Solitons & Fractals, 139:110045, 2020.
Masaki Sano and Yasuji Sawada. Measurement of the lyapunov spectrum from chaotic time series. Physicaa review letters, 15810) 1985.
Dominik Schmidt, Georgia Koppe, Zahra Monfared, Max Beutelspacher, and Daniel Durstewitz. Identify- ing nonlinear dynamical systems with multiple time scales and long-range dependencies. arXiv preprint arXiv:1910.08 03471, 2019.
Florian Schmidt. Generalization in generation: A closer look at exposure bias. arXiv preprint 1rX1v:1910.000 00292, 2019.
26

---

Published in Transactions on Machine Learning Research (10/2022)
Ljubisa Schovac and Katarina Grolinger. Deep learning for load forecasting: Sequence to sequence recurrent neural networks with attention. IEEF Access, 8:36411-3000 134413,666, 2020.
Ljubisa Schovac, Cornelius Nesen, and Katarina Grolinger. Forecasting building energy consumption with deep learning: A sequence to sequence approach. In 2019 IEEF International Congress Internet of Things (ICIOT), pp. 108-116. IEEE, 2019.
Lawrence F Shampine and Skip Thompson. Solving ddes in matlab. Applied Numerical Mathematics, 37() 441-458, 2001.
Guorui Shen, Jürgen Kurths, and Ye Yuan. Sequence-to-sequunec prediction of spatiotemporal systems. Chaos: An Interdisciplinary Journal of Nonlinear Science, 30(2):023102, 2020.
Julien Clinton Sprott and Konstantinos E Chlouverakis. Labyrinth chaos. International Journal of Bifurea- lion and Chaos, 17(06):2097-22 2108, 2007.
Rohan Thavarajah, Xiang Zhai, Zheren Ma, and David Castineira. Fast. modeling and understanding fluid dynamics systems with encoder decoder networks. Machine Learning: Science and Technology, 2(2): 025022, 2021.
René Thomas. Deterministic chaos seen in terms feedback circuits: Analysis, syuthesis," labyrinth chaos'. International Journal of Bifureation and Chaos, 9(10):1889 1905, 1999.
Pantelis R Vlachas, Wonmin Byeon, Zhong Y Wan, Themistoklis P Sapsis, and Petros Koumoutsakos. Data- driven forecasting ing of high-dimensional chaotic systems with long short-term memory networks. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, (F4(2213)201100255 2018.
Dominik Walther, Leander Schmidt, Klaus Schricker, Christina Junger, Jean Pierre Bergmann, Gunther Notni, and Patrick Mäder. Automatic detection and prediction of discontinuities in laser beam butt welding utilizing deep learning. Journal of Advanced Joining Processes, 6:100119, 2022.
Rui Wang, Eugenia Kalnay, and Balakumar Balachandran. Neural machine-basel forccasting of chaotic dynamics. Nonlinear Dynamies, (8(4))2903-2911) 2019.
Andreas Weigend and Neil A Gershenfeld. Results of the time series prediction competition at the santa fe institute. In IEEE international conference on neural networks, pp. 1786-1793, IEEE, 1993.
Ronald J Williams and David Zipser. A learning algorithm for continually rumning fully recurrent neural networks. Neural computation, (2):270-28M) 1989.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoe V Le. Xlnet: Gen- eralized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Wenpeng Yin, Katharina Kann, Mo Yu, and Hinrich Schütze. Comparative study of enn and rmn for natural language processing. arxiv preprint 1rXiv:1702.0m 01923, 2017.
27

---

Published in Transactions on Machine Learning Research (10/2022)
A Training and Validation Loss Curves
10a
122
12
10"4
10"1
rdde
Learsa rale
10-9
40-2
8
10
10-4
10-4
0.1
10-4
10
10a
10
9.0
1011
600
20
10000
2000
3000
spech
osech
(a)
(b)
traiinng
1.2
valdation
10.
10-
10-
10
0.6
16-
10-1
0.0
spach
(c)
CL-OTF-P
CL-ITF-P
Imiiing vilderion
traiiing
10-1
forcinn ratto
MT Forche mdu
10-4
10-4
10
30
10
10-4
10-9
10"1
13a1
0.0
(d)
(e)
CL-OTF-D
CL-ITF-D
rf/ado
tmining validaion
rato
traiiing
voliduiio
13:4.
MAaCHE forcing raboo
10-9
19-9
10-3
10-3
10-4
10-4
0.2
10-7
15-7.
150
200
350
300
730
(f)
(g)
Figure 13: Training and validation loss for Mackey-Glass
28

---

Published in Transactions on Machine Learning Research (10/2022)
maining validation
122
trahing
1.2
10"4
1014.
farcing
10'
1911.
0.0
spoch
spoch
(a)
(b)
CL-CTF-P
meton
traiing
valdation
10-4
16-
10a
0.2
10a
(c)
CL-DTF-P
CL-ITF-P
taiiiing Varidaiion Sella
rato
training validation
1.2
TMAKC Concing ratio
saacher forcing
1.
1019
1319.
10'7
0.2
15TT.
0.
250
330
350
200
730
(d)
(e)
CL-DTF-D
CL-ITF-D
19 1ed0
tmiiing volldarion
If 120
training volidition
1.0
tekcher frrcng wio
10-3.
cuacher forcing
4
10-4
18-1.
10-7
0.2
10-7.
02
100
155
300
500
opoch
(f)
(g)
Figure 14: Training and validation loss for Thomas
29

---

Published in Transactions on Machine Learning Research (10/2022)
12
trahingg veteation
1.2
valddriinn
10"4
1014
farcing ratio
TaaC HET Gercing taloo
10.
10"
19
0.0
spoch
spoch
(a)
(b)
CL-CTF-P
mein
treiingg valdation
10-2
16-
10a
0.2
10a
(c)
CL-DTF-P
CL-ITF-P
rada
traiiing vaidacion
raco
trahiigg validaiio
1.2
1.0
saaccer Forcing ratio
TMAKC Cinging ratio
0.6
10n0
10
0.2
107T.
0.
1500
2002
2503
1000
4000
(d)
(e)
CL-DTT-D
CL-ITF-D
tf rado
Imaning volidarion
If Fad
training volidation
1.0
furcag wilo
10-3
caacher forciig
13-3.
10-1
18-1.
10-7
0.2
19-T.
02
1000
2000
3003
2000
4000
5000
1000
7058
opoch
(f)
(g)
Figure 15: Training and validation loss for Rössler
30

---

Published in Transactions on Machine Learning Research (10/2022)
13"
validation
12
10"
trahingg
1.2
farcing ratlo
HeT Gercing taloo
10
19^4
0.8
10
10a4
0.0
spoch
spoch
(a)
(b)
CL-CTF-P
10"
metnn
treiiigg valdation
10
10-1
16-4
16a
0.2
10a
(c)
CL-DTF-P
CL-ITF-P
15a
rfrado
traiiing vaidacion
rato
trahiigg validtiio
1.2
10-
10-1
IMAK Cing ratio
Saaccer Forcing
10-1
10-
0.6
10-2
19-
10-9
0.2
10-9
02
1300
2500
3000
1000
13:00
2000
2500
(d)
(e)
CL-DTT-D
CL-ITF-D
10a
1f redo
Imaming volidarion
If 1aci
training volidition
10a1
10a1
4
10
Oa
13
caacher
10-
0.2
18-1
02
750
500
750
1000
1250
1500
1750
5.0
1550
2000
3000
opoch
(f)
(g)
Figure 16: Training and validation loss for Hyper- Rösseer
31

---

Published in Transactions on Machine Learning Research (10/2022)
1.2
trahiigg
1.2
validation
farcing ratlo
10-3-
10-h.
0
1019
1919.
10a
10"1.
0.0
spoch
spoch
(a)
(b)
CL-CTF-P
metin
treiing
valdation
16
0.2
10
0.00
opoch
(c)
CL-DTF-P
CL-ITF-P
rfrado
traiiing varidaiin
rat
trahiigg validation
1.2
10-1.
IMAK We Congngg ratio
saaccer forcing
5-
0
10-1
18-9.
10.
0.2
15-7.
02
300
200
500
1003
1200
(d)
(e)
CL-DTF-D
CL-ITF-D
17 re00
Imaming volidarion
If Fnod
training volidation
10"4
1.0
10"4
cuacher forcing
4
Oa
10.
0.2
1377.
0.
100
600
1.0
500
opoch
(g)
Figure 17: Training and validation loss for Lorenz
32

---

Published in Transactions on Machine Learning Research (10/2022)
FR
tmiming velidetion
trehing valiicioo
10-1
19a1
forcin
1014
10"4
talld
10-a
0.2
10
10
1903
2005
1000
4000
1000
2000
3000
4000
50.00
KI000
7000
R000
spech
epech
(a)
(b)
CL-CTF-P
training
1.2
10a4
16-
16a
10a
10
1..0
500
1000
1500
2000
spoch
CL-OTF-P
CL-IT-PP
109
12
farcing rale
10-1
ing Galo
10-
10-4
10-2
10-4
0.6
10-4
10
0.0
10-4
0.0
200
400
600
1000
500
1.000
1500
2500
3500
4000
spoch
spoch
(d)
(e)
CL-TF-D
304
Uirato
trining
1.2
10
1f rede
traning validation
5
10-1
tolchra frrch ouoo
10-4
10
4.88
10
0.0
30
02
0.0
20
1000
2000 spech
3000
4000
1030
1550
2300
3000
3530
opoch
(f)
(g)
Figure 18: Training and validation loss for Lorenz '96
33

---

Published in Transactions on Machine Learning Research (10/2022)
B CL-ITF-P With and Without LR Scheduler
100
104
10'
10
train, loss
10
10a
10a
10F
10a
10
epoch
(a)
10"
schecller
20
10c
irschedklr
10a
train, loss
10a
10
10a
10 opoch
(b)
100
trachedkuerr
10a
10a
10a
scheduler
102
train,
10a
10:
10
20
epach
Figure 19: Course of the training and validation loss for the basic RNN (a), LSTM (b) and GRU (c)
Lorenz'9e 99 data with and without using a learning rate scheduler.
34

---

Published in Transactions on Machine Learning Research (10/2022)
C
Different RNN Models
Table VIII: Forecasting performance of the vanilla RNN on four different. chaotic datasets
System
Strategy
Epochs
NRMSE
Rel. impr.
FR
51
0,48117
TF
249
0,41274
Thomas
CL-ITE-P
1000
266
0,17955
56.50%
CL-ITF-D
250
183
0.25659
37.83%
FR
2
0.00747
TF
0.50174
Rässler
CL-ITF-P CL-ITF-D)
500
2063
0.00283
62.12%
1000
3075
0.00319
37.30%
FR
506
0.11389
TF
572
0.00913
Lorenz
CL-ITE-P
1000
746
0.00603
33.95%
CL-ITE-D
125
782
0.00378
58.60%
FR
1710
0.31002
TF
637
0.57870
Lorenz'96
CL-ITE-P
1000
505
0.56872
-33.45%
CL-ITE-D
1000
6.573
0.07663
75.28%
Table IX: Forecasting performance of the LSTM on four different chaotic datasets.
System
Strategy
L
Epochs
NRMSE
Rel. impr.
FR
0.43698
TF
818
0.05265
Thomas
CL-ITF-P CL-ITE-D
758
0.01892
64.06%
125
859
0.01181
77.57%
FR
2417
0.00210
TF
1 650
0.00139
Rässler
CL-ITE-P CL-ITF-D
1 1000
3422
0.00063
34.68%
250
2367
0.00085
38.85%
FR
1154
0.06526
TF
398
0.00169
Lorenz
CL-ITF-P CL-ITF-D
250
806
0.00075
35.62%
31
419
0.00125
26.04%
FR
4019
0.13010
TF
3721
0.22757
Lorenz's '96
CL-ITF-P CL-ITE-D
500
9 164
0.03935
69.75%
62
4 4509
0.06855
47.31%
35

---

Published in Transactions on Machine Learning Research (10/2022)
Table X: Forecasting performance of the unitary evolution RNN on four different chaotic datascts.
System
Strategy
L
Epochs
NRMSE
Rel. impr.
FR
354
0.74991
TF
53
0,85679
Thomas
CL-ITF-P CL-ITE-D
0.61889
17.47%
64.000
340
0.73331
2.21%
FR
911
0.03726
TF
1887
0.03317
Rössler
CL-ITT-P
1000
126
0.02473
25.44%
CL-ITE-D
500
402
0.02420
27.04%
FR
758
0.13021
TF
0.96357
Lorenz
CL-ITT-P
1000
758
0.03098
76.21%
CL-ITF-D
500
1 096
0.02897
77.75%
1043
0,26359
TF
398
0.61159
Lorenz 96
CL-ITF-P
2000
697
0.26253
0.4%
CL-ITF-D
2000
1335
0.18483
29.88%
Table XI: Forecasting performance of the Lipschitz RNN on four different chaotic datasets.
System
Strategy
Epochs
NRMSE
Rel. impr.
FR
879
0.11697
TF
928
0.05714
Thomas
CL-ITE-P CL-ITE-D
250
879
0.05237
8.35% 31.29%
500
191
0.03926
FR
3 704
0.27122
TF
79
0.90844
Rössler
CL-ITE-P CL-ITE-D
250
2304
0.05185
80.88%
500
3449
0.04409
83.74%
FR
298
0.06675
TF
625
0,02830
I Lorenz
CL-ITE-P CL-ITF-D
62
914
0.01732
38.80%
250
686
0.02417
14.59%
FR
489
0.62710
TF
469
0.65122
Lorenz
CL-ITF-P
477
0.62170
0.86%
CL-ITF-D
4000
489
0.6258
0.21%
36


Reviews:
[{"review_id": "gNxjKCpvX0", "reviewer": "Reviewer_EVtJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n-----------\n\nSimplicity of the proposed curriculum specially the probabilistic strategies that allow switching between teacher forcing or free running. \n\nExtensive empirical evaluation on different chaotic dynamical systems.\n\n\nWeaknesses (Earlier version):\n-----------\n\nAlthough the proposed method is simple, and it sheds some light on the different curriculum learning strategies, the fact remains that its an incremental update on the proposed methods in the literature. \n\nThe paper mainly focuses on GRU based auto-encoders and uses some ablations to study the impact of the proposed scheme on LSTMs. \n\n-----\n\nGiven the above listed weaknesses, I do think the paper merits acceptance mainly due to the fact that the work aggregates various CL strategies and compares (with some reservations) on a series of meaningful dynamical systems that simulate time-series forecasting. Even though the models are restricted to GRUs mainly, it still shows a simple scheme can be meaningfully adopted to yield non-trivial results. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the risk of exposure bias in training sequential models with teacher-forcing vs free-running, i.e.,\n\n(a) Teacher-forcing (TF): Each timestep uses the correct input from the teacher \n(b) Free-running (FR): Each timesteps uses the model prediction from the previous step as the input. \n\nTeacher-forcing leads to faster training convergence but risks exposure bias since the teacher always provides the correct inputs during the training stage, the model is not exposed to its predictions as input from the previous stages. Since there is no teacher availability during the inference stage, the model risks exposure bias. \n\nFree-running removes the exposure bias during the inference stage as the model has been trained with its predictions from the past (and this is inline with the test time setup as the teacher is not available during the inference stage)\n\n\nThis paper's proposal is to provide a set of curriculum that decides what fraction of the input timesteps uses the teacher inputs and what fraction uses the model predictions from previous stages. In contrast to previous works, this paper proposes curriculum that switches between TF and FR on two different scales: \n(i) depending on training iteration ( ex. start with TF in the initial iterations and then decay  to use  FR )\n(ii) within a training iteration ( ex. initial timesteps use TF while later timesteps use FR)\n\nPaper evaluates the proposed curriculum learning strategies on various dynamical systems including chaotic as well as periodic behavior. \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Given the above listed weaknesses, I do think the paper merits acceptance mainly due to the fact that the work aggregates various CL strategies and compares (with some reservations on various architectures) on a series of meaningful dynamical systems that simulate time-series forecasting. Even though the models are restricted to GRUs mainly, it still shows a simple scheme can be meaningfully adopted to yield non-trivial results. \n\nTo strengthen their work,  I would recommend the authors to keep the same model structure but use couple of RNNs ( say LSTMs, ODE-RNNs, Unitary RNNs, etc. ) and show that their proposed scheme works across architectures. \n\nFurther more, I had commented in previous submission w.r.t. general applicability of the scheme extending beyond chaotic systems. I acknowledge that the authors included the comparison on the periodic dynamical systems. It would be interesting to see if the similar curriculum extends to other domains such as seq-to-seq text generation or audio generation. Or the authors to comment as to what might be the limitations of extending the proposed scheme to other domains. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ODZm3C2m9q", "reviewer": "Reviewer_9Rk6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\ndemonstrate exposure bias is the issue in chaotic time series.\n\nprovide extensive experiments to study different training strategies showing that the probabilistic iteration scale curriculum works better.\n\nWeaknesses:\n\nLimited novelty. These training strategies are highly similar to ones from scheduled sampling (Bengio et al., 2015).\n\nIntroduced more free hyper-parameter needed to tune.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This article studies the exposure bias in chaotic time series forecasting. Exposure bias, which is commonly available in sequence prediction for many NLP tasks, has been widely explored in literature. Scheduled sampling (Bengio et al., 2015), which is one of most effective solutions to migrate this issue, has been adopted by authors. Their main contribution is two folds:\n\ndemonstrate that exposure bias is also a big issue in chaotic time series.\nprovide extensive empirical experiments of these training strategies on chaotic time series.\n\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The revised version has addressed most of my concerns.  \nI'd like to give an acceptance. ", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WdbOg3GOYv", "reviewer": "Reviewer_EFQM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Experiments that clearly show exposure bias exists in chaotic time series forecasting, especially when pure TF strategy is used\n2. Systematic evaluation setup, with metrics to measure deterioration of forecast quality over prediction horizon\n\nWeakness:\n1. Directionally the CL-ITF-X strategies seem to outperform the baseline & other curriculums but there is no analysis on why this is the case. (Some hypotheses & future directions are mentioned in the limitations section.)\n2. All results are empirical, limiting generalizability.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to study methods that can alleviate exposure bias while training sequence-to-sequence models using GRU networks, specifically for the task of time series forecasting. The authors focus on combining the two most commonly used training strategies - Teacher Forcing(TF) & Free Running(FR) using various schemes. TF always uses actual input during the training phase, running a higher risk of exposure bias during the inference stage. FR on the other hand always uses the output from the model while training, which usually results in longer training periods. The authors propose a number of ways to combine TF & FR (called curriculums) to take advantage of both the methods while tackling exposure bias. The curriculums are framed by using both deterministic & probabilistic strategies that switch between TF & FR. The switches can happen after a certain iteration of the training stage (training scale) or after a certain timestamp of a given iteration (iteration scale). \n\nEvaluation (using NRMSE & R2) at various parts of the prediction horizon, on dynamical systems with varying degrees of chaos suggests mixed results\nBut the suggested curriculums outperform the pure TF or FR settings. Also curriculums starting out with FR and increasing TF are observed to be dominant & most stable.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "There has been no or only minimal attempts to explain some of the counterintuitive results observed:\n1. Superior performance of the CL-ITF-X strategies against the common intuition of strategies with TF heavy early epochs. There is mention of a hypothesis but can this be expanded further?\n2. Non-trivial performance decrease of the CL-ITF-P strategy using a vanilla RNN on Lorenz’96. \n3. Underwhelming performance of the CL-ITF-X strategies on the Santa Fe data set.\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines teaching strategies for sequential data, highlighting challenges with teacher forcing and with free running, and introduces a curriculum learning approach that is the subject of a thorough empirical investigation. Relative to the last submission, this version has significantly improved the narrative and clarity of argument, and the overall message is now clear and well supported by evidence. There remains some hesitation about the novelty and potential impact of the results, as the curriculum learning schedule is similar to prior work. However, the relatively small modifications suggested here do seem to have significant impact, and the empirical analysis is thorough. Some reviewers do suggest a broader set of experiments, with additional architectures etc., but given the breadth of the analysis already undertaken, all reviewers and I agree that the current version merits acceptance.
Decision: Accept (poster)