Active Tuning
Authors: Sebastian Otte
Matthias Karlbauer
Martin V. Butz
Venue:  (2021)
Keywords: Signal Filtering
Recurrent Neural Network
Time Series
Denoising
Temporal Gradients

Abstract: We introduce Active Tuning, a novel paradigm for optimizing the internal dynamics of recurrent neural networks (RNNs) on the fly. In contrast to the conventional sequence-to-sequence mapping scheme, Active Tuning decouples the RNN's recurrent neural activities from the input stream, using the unfolding temporal gradient signal to tune the internal dynamics into the data stream. As a consequence, the model output depends only on its internal hidden dynamics and the closed-loop feedback of its own predictions; its hidden state is continuously adapted by means of the temporal gradient resulting from backpropagating the discrepancy between the signal observations and the model outputs through time. In this way, Active Tuning infers the signal actively but indirectly based on the originally learned temporal patterns, fitting the most plausible hidden state sequence into the observations. We demonstrate the effectiveness of Active Tuning on several time series prediction benchmarks, including multiple super-imposed sine waves, a chaotic double pendulum, and spatiotemporal wave dynamics. Active Tuning consistently improves the robustness, accuracy, and generalization abilities of all evaluated models. Moreover, networks trained for signal prediction and denoising can be successfully applied to a much larger range of noise conditions with the help of Active Tuning. Thus, given a capable time series predictor, Active Tuning enhances its online signal filtering, denoising, and reconstruction abilities without the need for additional training.

Under review conference paper ICLR 2021
ACTIVE TUNING
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce Active Tuning, novel paradigm for optimizing the internal dynam- ics of recurrent neural networks (RNNs) on the fly. In contrast to the conventional sequeece-to-scqueee
mapping scheme, Active Tuning decouples the RNN's s re- current neural activities from the input stream, using the unfolding temporal gra- dient signal tune the internal dynamics into the data stream. As consequence, the model output depends only on its internal hidden dynamics and the closed- loop feedback of its own predictions; its hidden state is continuously adapted by means of the temporal gradient resulting from backpropagating the discrep- ancy between the signal observations and the model outputs through time. In this way. Active Tuning infers the signal actively but indirectly based on the originally learned temporal patterns, fitting the most plausible hidden state sequence into the observations. We demonstrate the effectiveness of Active Tuning on several time series prediction benchmarks, including multiple super-imposed sine waves, a chaotic double pendulum, and spatiotemporal wave dynamics. Active Tuning consistently improves the robustness, accuracy, and generalization abilities of all evaluated models. Moreover, networks trained for signal prediction and denoising be successfully applied to much larger range of noise conditions with the help of Active Tuning. Thus, given a capable time series predictor, Active Tuning enhances its online signal filtering, denoising, and reconstruction abilities without the need for additional training.
INTRODUCTION
Recurrent neural networks (RNNs) are inherently only robust against noise to a limited extent and they often generate unsuitable predictions when confronted with conrupted missing data (cf., e.g.. Otte et al. 2015). To tackle noise, an explicit noise-aware training procedure can employed, yielding denoising networks, which are targeted to handle particular noise types and levels. Recur- rent oscillators, such echo networks (ESNs) (Jaeger] 2001 Koryakin al. 2012 Otte et al.) 2016), when initialized with teacher forcing, however, are highly dependent on a clean and accurate target signal. Given an overly noisy signal, the system is often not able to tune its neural activities into the desired target dynamics at all. Here, we present method that be to regular teacher forcing and, moreover, as a general tool for more robustly tuning and thus syn- chronizing the dynamics of a generative differentiable temporal forward model such as a standard
an alternative
RNN. ESN, or LSTM-like RNN (Hochreiter & Schmidhuber 1997 Otte et al. 2014 Chung et al., 2014
Otte et al. 2016) into the observed data stream.
The proposed method, which we call Active Tuning, uses gradient back prok-propngation through time (BPTT) (Werbos 1990). where the mack-propagated gradient signal is used to tune the hidden actiy- ities of neural network instead of adapting its weights. The way utilize the temporal gradient signal is related to learning parametric biases (Sugita et al.] (20TT) and applying dynamic context inference (Butz et al., 2019).. With Active Tuning, two essential aspects apply: First, during signal inference, the model not driven by the observations directly, but indirectly via prediction error- inducted temporal gradient information, which is used to infer the hidden state activation sequence that best explains the observed signal. Second, the general stabilization ability of propagating signal hypotheses through the network exploited, effectively washing out activity components (such as noise) that cannot be modeled with the learned temporal structures within the network. As a result, the vuinerable internal dynamics are kept within system- consistent activity milieu, effectively de- coupling it from noise or other unknown distortions that present in the defective actual signal.

---

Under review as conference paper ICLR 2021
In this work we show that Active Tuning elicits enhanced signal filtering abilities, without the need for explicitly training distinct models for exactly such purposes. Excitingly, this method allows for instance the successful application of an entirely noise-unaware RNN (trained on clean, ideal data) under highly noisy and unknown conditions.
the following, we first detail the Active Tuning algorithm. We then evaluate the RNN on three time series benchmarks-muliin wave dynamics. The results confirm that Active Tuning enhances noise robustness in all cases. The mechanism mostly even beats the performance of networks that were explicitly trained to handle a particular noise level. It can also cope with missing data when tuning the predictor's 's state into the observations. In conclusion, we recommend to employ Active Tuning in all time series prediction cases, when the data is known to noisy, corrupted, or to contain missing values and the generative differentiable temporal forward model-typically a particular RNN urchitecture-Knoww about the potential underlying system dynamics. The resulting data processing system will be able to handle a larger range of noise and corrupted data, filtering the signal, generating more accurate predictions, and thus identifying the underlying data patterns more accurately and reliably.
superimposed sine waves, a chaotic pendulum, and spatiotemporal
2 ACTIVE TUNING
Starting point for the application of Active Tuning is a trained temporal forward model. This may be, as mentioned earlier, an RNN, but could also be another type of temporal model. The prerequisite is, however, a differentiable model that implements dependencies over time, such that BPTT can be used to reversely route gradient information through the computational forward chain. Without loss of generality, we assume that the model of interest, whose forward function may be referred to fas. fulfills the following structure:
(o' xx'x')) SM (a"+1 xx**!)
(1) where o' is the current latent hidden state of the model (e.g. the hidden outputs of LSTM units, their cell states, or any other latent variable of interest) and x' the current signal observation. Based on this information Sas generates a prediction for the next input x++1 and updates its latent state ++1 accordingly.
Following the conventional inference scheme, we feed a given sequence time step by time step into the network and receive a one-time step ahead prediction after each particular step. Over time, this effectively synchronizes the network with the observed signal. Once the network dynamics are initialized, which is typically realized by teacher forcing, the network can generate prediction and its dynamics can driven further into the future in a closed-loop manner, wherehy the network feeds itself with its own predictions. To realize next time step- and closed-loop predictions, direct contact with the signal is inevitable drive the teacher forcing process. In contrast, Active Tuning decouples the network from the direct influence the signal. Instead, the model is permanently kept in closed-logg mode, which initially prevents the network from generating meaningful predictions. Over a certain time frame, Active Tuning keeps track of the recent signal history, the recent hidden states of the model, as well as its recent predictions. We call this time frame (retrospective) tuning horizon or tuning length (denoted with R).
The principle of Active Tuning can best be explained with the help of Figure I and Algorithm I The latter gives a more formal perspective onto the principle. Note that for every invocation of the procedure a previously unrolled forward chain (from the previous invocation or an initial unrolling) is assumed. L refers to the prediction error of the entire unrolled prediction sequence and the respec- tive observations, whereas L refers to the local prediction error just for a time step t'. With every new perceived and potentially noise-affected signal observation x'. one or multiple tuning cycles are performed. Every tuning cycle hereby consists of the following stages: First, from the currently believed sequence of signal predictions (which is in based on sequence of hidden states) and the actual observed recent inputs, a prediction error is calculated and propagated back into the past reversely along the unfolded forward computation sequence. The temporal gradient travels to the very left of the tuning horizon and is finally projected onto the seed hidden state o which is then adapted by applying the gradient signal in order to minimize the encountered prediction error. This adaption can be done using any gradient- pradient-based optimizer. Note that in this paper, we exclusively Adam (Kingma & Ba 2015), but other optimizers are possible well, Second, after the adap-

---

Under review conference paper ICLR 2021
x R=1
x²
R-2
x
x
fM
fM
fu
x
x R--1
x R-1
R+22
x²
fx
SM
fx
fu
x--R
x R-1
8 R+2
x2-2
g--
X
x
x R43
x²
x
gt. R41
fM
fAr
fM
14-R--1
8/ R443
R+2++2
x R+3
R-1
Figure I: Illustration of Active Tuning principle over two consecutive world time steps. fM refers the forward pass of the base model. x', x' etc. are the recent potentially defective signal observations, whereas X x etc. are respective predictions (outputs of the model's forward function fM)) R denotes the length of the retrospective tuming horizon, that is, the number of time steps the prediction error is projected into the past using BPTT. o'-R refers to the latent (hidden) state of .M the beginning of the tuning horizon, which essentially seeds the unfolding prediction sequence (black lines). o is actively optimized based on the back-projected prediction eroor gradient (red lines).
Algorithm Active Tuning procedure
Input Current observation xf
Output: Prediction (filtered output), predictive hidden state a²
x,, of (x f __==2)
fe Generate current prediction bused on previous forward chain */
for to do 1e Perform multiple luning cycles */
for 1' 11o Rdb do
Srucpppropggute prediction error
Oo"
g" ac Oc
B
++1
if <
0o
O0" Oo
0
otherwise
end for
R- update( "" Bg
1 Perform taning (e.g. with Adam update rule) +/
for
e' + R+ to do
em Roll out forward chain again based adapted hidden
x",0" 0 [x
end for
end for
return x', o²
tion of this seed state (and maybe the seed input as well) the prediction sequence rolled out from the past into the present again, effectively refining the output sequence towards better explanation of the recently observed signal. Each tuning cycle thus updates the current prediction x and the current hidden state of from which closed-loop future prediction transition into the next world time step. one forward step has to be computed. The formerly leftmost seed states can be discarded and the recorded history is shifted by one time step, making o R+1 the seed state that will be unnd within the next world time step. From then on, the procedure
be rolled out, if desired. To

---

Under review as conference paper ICLR 2021
repeated, yielding the continuous adaptive tuning process. As result, the model is predominantly driven by its own imagination, that is, its own top down predictions. Meanwhile, the predictions themselves are adapted by means of the temporal gradients based on the accumulated prediction error, but not by the signal directly. In a nutshell, Active Tuning realizes gradient-base mini- optimization procedure on any of the model's latent variables within one world time step. While it needs to be acknowledged that this process draws additional computational resources, in this paper we investigate the resulting gain in signal processing robustness.
Intuitively speaking, Active Tuning tries to fit known temporal patterns, as memorized within the forward model, to the concurrently observed data. Due to the strong pressure towards consistency maintenance, which is naturally enforced by means of the temporal gradient information in combi- nation with the repeatedly performed forward passes of the hidden state activities, the network will generate adaptations and potential recombinations of patterss that it has learned during training. Oc- currences that cannot be generated from the repertoire of neural dynamics will therefore not appear (or only in significantly suppressed form) the model's 's output. As a consequence, there is much smaller need to strive for noise robustness during training. Our results helow indeed confirm that the model may be trained clean, idealized target signals. However, imprinting slight denois- ing tendency during training proves to useful when facing more noisy data. Enhanced with our Active Tuning scheme, the model will be able to robustly produce high-quality outputs even under extremely adverse conditions- as long (some of) the assumed target signals are actually present. Our scheme is thus a tool that can be highly useful in various application scenarios for signal recon- struction and flexible denoising. Nevertheless, it should be mentioned that with Active Tuning the computational overhead for inference scales with the number of tuning cycles and the tuning length.
3 EXPERIMENTS
In order to investigate the abilities of Active Tuning we studied its behavior at considering three different types of time series data, namely, one-dimensional linear dynamics, two-dimensional non- linear dynamics, and distributed spatiotemporal dynamics. For all three problem domains we used a comparable setup except for the particular recurrent neural network architectures applied. We trained the networks as one time step ahead predictors whose task is to predict the next input given both the current input and the history of inputs aggregated in the latent hidden state of the models. The target sequences were generated directly from the clean imput sequences by realizing shift of one time step. Moreover, we trained networks under six different denoising conditions (normally distributed) per experiment, where we fed potentially noisy signal into the network and provide the true signal (one time step ahcad) as the target value (Lu al. 2013 Otte et al.] 2015) Goodfellow let al. 2016). These conditions are determined by their relative noise ratios: 0.0 (no noise), 0.05, 0.2, 0.5, and 1.0, where the ratios depend on the respective base signal statistics. For instance, a noise ratio of 0.1 mcans that the noise added to the input has standard deviation of 0.1 times the standard deviation of the base signal. As a result we obtained predictive denoising experts for each of these conditions. All models were trained with Adam (Kingma & Ba 2015) using its default parameters (learning rate 1 = 0.001, 3, 0.9 and 32- 0.999) over 100 (frst two experiments) or 200 (third experiment) epochs, respectively.
3.1 MULTI-SSPPERIPPSEER OSCILLATOR
The first experiment is variant of the nultiple superimposed oscillator (MSO) benchmark (Schmid- huber et al. 2007 Koryakin et al. 2012. Otte et al. (016). Multiple sine waves with different frequencies, phase-shifts, and amplitudes are superimposed into one signal (ef. [Eq. 2 the [Sec- tion A. where n gives the number of superimposed waves, f, the frequency, a, the amplitude, and ; the phase-shift of each particular wave, respectively. Typically, the task of this benchmark is to predict the further progression of the signal given some initial data points (e.g. the first 100 time steps) of the sequence. The resulting dynamics are comparably simple as they can, in principle. be learned with linear model. It is, however, surprisingly difficult for basT- models. namely LSTM-like RNNs, to precisely continue given sequence for more than few time steps (Oite ct al. 2019). For this experiment we considered the MSO, dynamics with the default frequencies f1 0.2. J2 0.311, fa = 0.42, f4 0.51, and fs = 0.63. An illustration of an exemplary the ground truth signal can be found in Figure 2

---

Under review as conference paper ICLR 2021
For training. we generated 10 000 examples with 400 time steps each, using random amplitudes o, [0, 1] and random phase-shifts pp; ~ [0, 2 For testing, another 1 1000 examples were generated. As base model, we used an LSTM network with one input, 32 hidden units, one línear output neuron, and no biases, resulting in 256 parameters. Additionally, to contrast our results with another state- of-theeart sequence-to-seeeee se--quuence model, temporal convolution networks (TCNs) (Kalchbrenner et al. 2016) were trained. Preliminary experiments showed that seven layers with 1, 8, 12, 16, 12, 8, and I feature maps, kemel size of 3, and standard temporal dilation rate- rale-yiclding temporal horizon of 64 time steps-tended to generate the best performance with a comparable number of parameters (i.e. 4 682). Code was taken from Bai et al. (2018).
3.2 CHAOTIC PENDULUM
The second experiment is based on the simulation of a chaotic double pendulum. As illustrated in Figure 5) (cf. [Section A.2), the double pendulum consists of two joints whose angles are specified by 0₁ and 0g and two rods of length I and /2. Besides the length the rods, the masses my and my affect the behavior of the pendulum. The pendulum's end-effecto (where my is attached) generates smooth, but highly non-linear trajectories. More precisely. exhibits chaotic behavior, meaning that already slight changes of the current system state can quickly cause major changes of the pendulum's ' state over time (Korsch et al., 20088 Patthak et al.) 2018). It is thus typically difficult precisely predict the dynamics of such system for more than a few time steps into the future. making it a challenging benchmark problem for our purposes.
In the literature, the double pendulum's dynamics are typically described using the equations of motion, given by Eq. 3 and Eq. 4 (ef. ), respectively, which are derived from the Lagrangian of the system and the Euler-Lagrange equations; see Korsch et al. (2008) for details. For simulating the double pendulum, we applied the fourth-order Runge-Kutt Kutta (RK4) (Press) 2007) method to numerically integrate the equations of motion. All four parameters 41, 12, mi, and my were set to 1.0. A temporal step size of the pendulum is described by its two angles, which were selected randomly for each sample to be within 01 ~ [90": 270"| and 02 [01 + 30 to ensure sufficient energy in the system. One out of ten sequences was initiated with zero angle momenta, that is 0,, 02 0.0. The number of train and test samples, as well as the sequence lengths were chosen analogously to experiment one. As base model we used an LSTM network with two inputs, 32 hidden units. two linear output neurons, and again no biases. Again, we trained TCNs on this data by changing the number of input and output feature maps to two. Otherwise, the settings were identical to the ones used in experiment one.
0.01 was chosen for numerical integration. The initial state of
3.3 SPATIOTEMPORAL WAVE DYNAMICS
the third experiment we considered a more complex spatiotemporal wave propagation process, based the wave dynamics formalized by [Eq. (cf. [Section A.3). Here, and correspond to a spatial position in the simulated field, while denotes the time step and = 3.0 the propagation speed factor of the waves. The temporal and spatial approximation step sizes were set to h4 = 0.1 and h2g = hy = 1.0, respectively. No explicit boundary condition was applied, resulting in the waves being reflected at the borders and the overall energy staying constant over time.
We generated sequences for a regular grid of 16 x 16 pixels. See Figure 4 or Figure 6 for illustra- tions of the wo-dimensional wave. In contrast to the previous two experiments, 200 samples with a sequence length of 80 were generated for training. whereas 20 samples over 400 time steps were used for evaluation. As base network we used distributed graph-RNN called DISTANA (Karl: bauer et al. 2020)., which is essentially a mesh of the same RNN module (an LSTM, which consists here of four units only). which is distributed over the spatial dimensions of the problem space (here two-dimensional grid), where neighboring modules benchmark, and this recurrent graph network in particular, to demonstrate the effectiveness of Active Tuning in a setup of higher complexity. Moreover. we again trained TCNs, here with three layers, having 1, 8, and feature maps, respectively, using x 3 spatial kernel sizes and standard dilation rates for the temporal dimension. Noteworthy, while the applicd DISTANA model counts 200 pa- rameters, the applied TCN has an order of magnitude more parameters (2 306). Less parameters yielded significant drop in performance.
laterally connected. We chose this wave

---

Under review conference paper ICLR 2021
4 RESULTS AND DISCUSSION
The quantitative evaluations are based on the root mean square error (RMSE) between the network outputs and the ground truth. All reported values averages initialized and trained models. order elaborate on the applicability each denoising expert on unseen noise conditions, we evaluated all models using the noise ratios 0.0, 0.1, 0.2, 0.5, and 1.0, compared on all noise
ten independently randomly
resulting in 25 baseline scores for each experiment. These baselines tios against eight Active Tuning setups, which were based on models trained without any noise (0.0) or with only small portion of input noise (0. (0.05). The individual parameters of Active Tuning used produce the results reported in [Section A. the appendix. Note that in all experiments. the latent hidden outputs of the LSTM units (not the cell states) were chosen as optimization target for the Active Tuning algorithm. Furthermore, these hidden states were initialized normally distributed with standard deviation 0.1 in all cases, whereas the cell state were initialized with zero.
4.1 MSO RESULTS
The results of the MSO experiments are summarized in Table Active Tuning improves the results for the weakest model (0.0) in all cases (column 3 vs. 7). partially almost by order of magnitude. Noteworthy, for the inference noise ratio 0.1, driven with Active Tuming the noise-unaware model becomes better than the actual expert. Recall that the model was no retrained, only the paradigm how the model is applied was changed. On the other hand, there is no advantage for Active Tuning when the base network encountered minimal noise (0.05) during training in this experiment. For comparison, a noise-umme uniieuuniffommed TCN (0.0) performs better than the respective RNN (cf. column 2 vs. column 3 in Table 1}. Active Tuning reverses this disparity. On this benchmark, however, denoising expert TCNs clearly outperform the expert RNNs (cf. Table (ection A.4).
Table 1: MSO noise suppression results (RMSE)
Training (signal noise)
Inference (signal noise)
TCN
Regular inference RNN
Active Tuning
0.0
0.0
0.1
0.2
0.5
1.0
0.0
0.05
0.0
0.0411
0.0039
0.04998
0.0781
0.1526
0.2383
0.1
0.1341
0.5880
0.0966
0.0993
0.1579
0.2397
0.0912
0.0947
0.2
0.2580
1.0336
0.1734
0.1454
0.1728
0.2446
0.1682
0.1550
0.5
0.6189
1.8713
0.4265
0.3190
0.2538
0.2751
0.3583
0.3611
1.0
1.1676
2.6241
0.9502
0.6437
0.4396
0.3639
0.5699
0.6101
Table 2: MSO missing data results (RMSE)
Missing data probability
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Regular inference
0.1206
1.0357
2.2410
2.9601
3.3488
3.7070
4.0039
4.0065
3.4861
Active Tuning
0.0321
0.0381
0.0581
0.0826
0.1462
0.3874
0. 0.6209
1.0757
1.4064
Ground truth
Network autput
Netwart output
-2
-4
50
125
200
Time
Figure 2: Visual comparison of regular inference (orange) Active Tuning (light blue) an MSO example with strong noise (noise ratio 1.0) using a noise-unaware LSTM.

---

Under review conference paper ICLR 2021
To get an impression of the actual improvement of the output quality, DoasiderFiguuee The noise- unaware model (0.0) produces poor predictions when confronted with strong signal noise (1.0). When driven with Active Tuning instead of regular inference (teacher forcing), the output of the same model becomes smooth and approximates the ground truth reasonably well. Active Tuning thus helps to catch most of the trend information while mostly ignoring noise.
As an additional evaluation, [Table 2 demonstrates the ability of Active Tuning to cope with missing data. The results based the noise-unaware model. While tuning into the signal, particular observation are missing (dropped out) with a certain probability ranging from 0.1 to 0.9. In case a missing observation, the prediction of the model is used instead. Already with dropout chance of 20 % the RNN struggles tune its hidden neural states into the signal, thus generating error larger than I. In contrast, exactly the same RNN model remains significantly more stable when driven with Active Tuning. Even with a dropout chance of 50 - 60 % the RNN still produces errors clearly below the reference error of approximately 0.9, which is the RMSE error generated when always predicting zero. Note that with regular inference, the error decreases slightly with highest dropout rate. This is the case because here the network receives so few inputs such that starts to produce outputs for some sequences.
It seems that during regular inference the network dynamies overly driven by the input data. When parts of the input is missing, the internal dynamics do not synchronize with the true dynamics because multiple consecutive time steps of consistent signal observations appear necessary. Active Tuning enables the network to reflect on the recent past including its own prediction history. While it attempts to maintain consistency with the learned dynamics, infers a hidden state sequence that best explains the (even sparsely) encountered observations and thus effectively fills the input gaps retrospectively with predictions that seem maximally plausible.
4.2 PENDULUM RESULTS
For the pendulum experiment, the potential of Active Tuning becomes even more evident. The results presented Table 3jndicate that for noise ratios Active Tuning outperforms the respective expert RNNs, particularly when applied to the model that was trained on small noise (0.05). With increasing noise level, the problem becomes progressively impossible to learn. For example, the 1.0-expet-model does not seem to provide any reasonable function at all, indicated by the worse RMSE score compared to other models (1.0 inference noise row). In contrast, Active Tuning can still handle these extremely unfavorable conditions surprisingly well. [Figure 3/sows exemplary case. The unknown ground truth plotted against the noisy observations (shown the left image). The center image shows the prediction of the reference LSTM (trained with 0.05 noise) when regular inference is applied. It is clearly difficult to recognize coherent trajectory reflecting the dynamics of the double pendulum. With Active Tuning, on the other hand, the same network produces a mostly clean prediction that is relatively close to the ground truth sequence.
Analogously to MSO, we evaluated the robustness against missing data on the pendulum. The results reported in Table 4 In contrast to the previous scenario (ef. Table 2) it is noticeable that the base model is intrinsically more stable this experiment. Still, Active Tuning yields a significant improvement in all cases. For the mid-range dropout rates, it decreases the prediction error by approximately
order of magnitude. Even with dropout rate of 80 % still somewhat accurate predictions are generated. Please note again that the comparison here uses exactly the same RNN (the same structure as well as the same weights) for both regular inference and Active Tuning.
Table 3: Pendulum noise suppression results (RMSE)
Training (signal noise)
Inference (signal noise)
TCN
Regular inference RNN
Active Tuning
0.0
0,0
0.1
0.2
0.5
1.0
0.0
0.05
0.0
0.0135
0.0091
0.1475
0.2471
0.4459
0.5700
0.1
0.1155
0.2097
0.1537
0.2489
0.4463
0.5702
0.0880
0.0865
0.2
0.2272
0.4021
0.1711
0.2545
0.4474
0.5710
0.1423
0.1284
0.5
0.5572
0.8458
0.2702
0.2945
0.4563
0.5764
0.2954
0.2460
1.0
1.0959
1.2753
0.5308
0.4444
0.4918
0.5954
0.4868
0.4030

---

Under review conference paper ICLR 2021
Table 4: Pendulum missing data results (RMSE)
Missing data probability
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Regular inference
0.0158
0.0377
0.1493
0.3286
0.6100
0.9370
1.2831
1.6068
1.8457
Active Tuning
0.0111
0.0148
0.0190
0.0278
0.0518
0.0977
0.1698
0.3206
0.7932
Ground truth
Observed signal
Networt
Network output
-2
0
2
2
X
X
X
Figure 3: Exemplary comparison of regular inference (orange) Active Tuning (light blue) the double pendulum's end-effector trajectory; the black dot denotes the start position. Here, the second strongest noise condition (0.5) is shown, using a 0.05-noise LSTM for inference and Active Tuning.
4.3
WAVE RESULTS
The results of the wave experiment (Table 5) consistently support the findings from the pendulum experiments. When driven with Active Tuning, the considered models produce better results than the explicitly trained denoising experts on all noise levels. Figure 4 shows in accordance with the previous experiments that the noisy signal observations (1.0) is filtered effectively and latency-free exclusively when using Active Tuning, yielding smooth signal prediction across the entire spa- tiotemporal sequence. While wwo-dimensional output the network operating in conventional inference mode is hardly recognizable as a wave, the network output of the same model combined with Active Tuning clearly reveals the two-dimensional wave structure with hardly perceivable de- viations from the ground truth. More qualitative results can found Figure 6 (Section A.4).
We furthermore compared performance with a common, non-recurrent, sequence-to -oroonncce-t--suueneee learn- ing architecture. Here we considered standard TCN architecture (Bai et al. 2018), which we also trained to focus all considered denoising levels. The full performance table is shown in Table B (Section A.4p. The first result column in Table S shows that even the best TCN performance al- ways outperformed by Active Tuning. Importantly, DISTANA with Active Tuming outperforms the best TCN results all noise levels when the DISTANA model
trained for denoising.
We also performed experiments with other noise distributions (e.g. salt-and- peppeeppprrr noise). Some- what surprisingly this manipulation affected the quality of the output only marginally. Thus, in contrast to deep convolutional networks (Geirhos al. 2018), the denoising RNNs applied here did not overfit the noise type.
Table 5: Wave noise suppression results (RMSE)
Training (signal noise)
Inference (signal noise)
TCN
Regular inference RNN
Active Tuning
(best)
0.0
0.1
0.2
0.5
1.0
0.0
0.05
0.0051
0.0007
0.0021
0.0042
0.0096
0.0175
0.1
0.0138
0.0268
0.0073
0.0064
0.0100
0.0176
0.0073
0.0062
0.2
0.0252
0.0533
0.0142
0.0106
0.0113
0.0178
0.0097
0.0087
0.5
0.0581
0.1295
0.0362
0.0262
0.0180
0.0197
0.0173
0.0150
1.0
0.1133
0.2368
0.0784
0.2467
0.0345
0.0261
0.0283
0.0213

---

Under review conference paper ICLR 2021
0.8
2.6
-0.2
-0.4
25
75
1:00
125
175
200
Time
Figure Exemplary comparison of regular inference (orange) vs. Active Tuning (light blue) on wave examples with strong noise (1.0) using DISTANA trained without noise. The four top rows visualize ground truth, noisy observations (network input), network output without, and network output with Active Tuning. The plot below shows the wave activities the center position.
5 CONCLUSION
this work we augmented RNN architectures with Active Tuning, which decouples the internal dynamics of an RNN from the data stream. Instead of relying on the input signal to set the internal network states, Active Tuning retrospectively projects the dynamic loss signal onto its internal latent states, effectively tuning them. We have shown that RNNs driven with Active Tuning can reliably denoise various types time series dynamics, mostly yielding higher accuracy than specifically trained denoising expert RNNs. In all cases, however, the augmentation with Active Tuning has beaten the reference RNN with teacher foreing. Moreover, creases the tolerance against missing data by large extent allowing the models to generate accurate prediction even if more than 50 % of the input are missing. Comparisons with TCN have shown that Active Tuning yields superior performance. outperformed by the similarly trained recurrent graph neural network DISTANA (Karlbauer et al.) 2020). When adding Active Tuning, even the noise uniformed DISTANA version outperformed the best TCN networks. Note that even though we used Active Tuning exclusively for RNNs in this paper, is in general not restricted to such models. We are particularly interested adapting the principle to other sequence learning models such TCNs.
have shown that Active Tuning in-
the wave experiments. TCNs are consistently
While the presented results all very encouraging, it should be noted that in experience Active Tuning is slightly slower to tune the network clean signal. Seeing that Active Tuning can principle be mixed with traditional teacher forcing, we are currently exploring switching teacher forcing and off in adaptive manner depending the present signal conditions.
Another concern lies in the applied tuning length and numher of tuning cycles. In the presented experiments, we used up to 16 time steps with partially up to 30 tuning cycles. Additional ongoing research aims at reducing the resulting computational overhead. Ideally, Active Tuning will work reliably with a single update cycle over a tuning length of very few time steps, which would allow perform Active Tuning along with the regular forward pass of the model fused computation step. Additionally, we aim at applying Active Tuning to real- worldw denoising and forecasting challenges, including speech recognition and weather forecasting.

---

Under review conference paper ICLR 2021
REFERENCES
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv: 2018.
Martin V. Butz, David Bilkey, Dania Humaidan, Alistair Knott, and Sebastian Otte. Learning, planning, and control in a monolithic neural event inference architecture. Neural Networks, 117: 135-144, 2019. doi: 10. 101 10j nwwmee...etmmm......m
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv: 1412.3555 [cs], December 2014. arXiv: 1412.3555.
Robert Geirhos, Carlos R. M. Temme, Jonas Rauber, Heiko H. Schütt, Matthias Bethge, and Fe- lix A. Wichmann. Generalisation in humans and deep neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Binn Bianchi, and R. Gameut (eds.), Advances in Neural Infor- mation Processing Systems 31, pp.
Curran Associates, Inc., 2018
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, Cambridge, MA, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term memory. Neural Computation, 9(): 1735-1780, November 1997. doi: 10. 1621neo/o... 1997.9.8.11 1735..
Herbert Jaeger. The "echo state" approach to analysing and training recurrent neural networks. Technical Report GMD Report, 148, Fraunhofer Institute for Analysis and Information Systems AIS, Sankt Augustin, Germany, 2001.
Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu. Neural machine translation in linear time. arXiv: 1610. 70099, 2016.
Matthias Karlbauer, Sebastian Otte, Hendrik P. A. Lensch, Thomas Scholten, Volker Wulfmeyer, and Martin V. Butz. Inferring, predicting, and denoising causal wave dynamics, 2020. arXiv: 2009.09187
Diederik P. Kingma and Jimmy L. Ba. Adam: A method for stochastic optimization. 3rd Interna- tional Conference for Learning Representations, abs/1412.6980, 2015.
Hans Jürgen Korsch, Häns-Jörg Jodl, and Timo Hartmann. Chaos: a program collection for PC. Springer, Berlin; ; New York, 3rd rev. and enlarged ed edition, 2008. ISBN 978-3-540-74866-3.
Danil Koryakin, Johannes Lohmann, and Martin Butz. Balanced echo state networks. Neural Networks, 36:35-45, 2012. doi: 10. 201.et.201...088
Xugang Lu, Yu Tsao, Shigeki Matsuda, and Chiori Hori. Speech enhancement based on deep denoising autoencoder. In Interspeech, volume 2013, pp. 436 440, 2013.
Sebastian Otte, Marcus Liwicki, and Andreas Zell. Dynamic Cortex Memory: Enhancing Recur- rent Neural Networks for Gradient-Based Sequence Learning. In Artificial Neural Networks and Machine Learning ICANN 2014, number 8681 in Lecture Notes in Computer Science, pp. 1-8. Springer International Publishing, September 2014.
Sebastian Otte, Marcus Liwicki, and Andreas Zell. An Analysis of Dynamic Cortex Memory Net- works. In International Joint Conference on Neural Networks (IJNN), pp. 3338-3345, Killarney, Ireland, July 2015. ISBN 978- 108-1-4499--999 1959-8.
Sebastian Otte, Martin V. Butz, Danil Koryakin, Fabian Becker, Marcus Liwicki, and Andreas Zell. Optimizing recurrent reservoirs with neuro-evolution. Neurocomputing, 192: 122-188-338. June 2016. ISSN 0925-2312. doi: 1...0166..nuccmm2222 20..000..88.
Sebastian Otte, Patricia Rubisch, and Martin V. Butz. Gradient-based learning of compositional dynamics with modular mns. In Artificial Neural Networks and Machine Learning - ICANN 2019, number 11727 in Lecture Notes in Computer Science, pp. 484-496. Springer International Publishing, September 2019. ISBN 978-3-030-30487-4.
10

---

Under review conference paper ICLR 2021
Jaideep Pathak, Alexander Wikner, Rebeckah Fussell, Sarthak Chandra, Brian R Hunt. Michelle Girvan, and Edward Ott. Hybrid forecasting of chaotic processes: Using machine learning in conjunction with knowledge-hased hased model. Chaos: An Interdisciplinaryy Journal of Nonlinear Science, 1814) 2018.
William Press. Numerical recipes : the art of scientific computing. Cambridge University Press, Cambridge UK ;:New York, 3rd ed. edition, 2007. ISBN 978-0-521-88068-8
Jürgen Schmidhuber, Daan Wienstra, Matteo Gagliolo, and Faustino Gomez. Training recurrent neural networks by evolino. Neural Computation, 19:757-779, 2007.
Yuuya Sugita, Jun Tani, and Martin Butz. Simultaneously emerging braitenberg codes and com- positionality. Adaptive Behavior, 19:295-316, 2011. doi: 10.1177/10597123 1416871.
Paul Werbos.
IEEE, Backpropagation through time: what does and how to do it. Proceedings of the 1990. ISSN 0018-9219. doi: 10.1 10.//595558337
A
APPENDIX
A.I
MSO EXPERIMENT
The MSO experiment were based on the following equation:
MSO., Σ a sinnsfft + sp:)
(2)
A.2
CHAOTIC PENDULUM EXPERIMENT
61
mT1
1712
02
Figure 5: The double pendulum used for data generation of the second experiment and resulting nonlinear trajectory of the pendulum's end-effector.
The pendulum experiments were based on the following equations:
P9i sfn( cas(fz A1) + nuf sin 01) cos(O2 01) sin(A:) P d2 sin(A; -81)
0
(3)
ccoss (02 01)
0, 92 siin(:))oss() - 01) 108 sin (0:2 0:) cos(02 01) 32 sin(O,) 203 sin(Oz A.)
(4)
μ cos" (02 01)
where
9 92
mg
91
1
my
m2
and g 9.81 being the gravitational constant.

---

Under review conference paper ICLR 2021
A.3 WAVE DYNAMICS EXPERIMENT
The wave experiments were based on the following equation:
Sa
18au
8'u
(S)
0F2
Ozx
This equation was solved numerically using the method of second order central difference, yielding
8hu an
u(e, h2) R2 1o
2u(x, y, () u(x, v. lu)
(6)
0y²
with, after solving 0 u/00x and analogously /uu/Oyy via the same method,
u(x + 11,, y. 2u(x, y. t) + u(æ 11.. y.
(7)
2x
h
0uu
u( hy 2n xr, v.t) + u(x ty
(8)
0y²
A.4 FURTHER RESULTS
The performance of temporal convolution networks (TCNs) at predicting and denoising the MSO. pendulum and spatiotemporal wave dynamics are reported in the following tables (Table o Table and Table 8). An additional qualitative evaluation of the benchmark larger grid shown in Figure
Table 6: TCN MSO noise suppression results (RMSE)
Inference
Training (signal noise)
(signal noise)
0.0
0.1
0.2
0.5
1.0
0.0411
0.0352
0.0633
0.0968
0.1255
0.1341
0.0936
0.0789
0.1016
0.1289
0.2580
0.1764
0.1206
0.1151
0.1386
0.6189
0.4367
0.3239
0.1898
0.1921
1.1676
0.8744
0.7431
0.3926
0.3140
Table 7: TCN pendulum noise suppression results (RMSE)
Inference (signal noise)
Training (signal noise)
0.0
0.1
0.2
0.5
1.0
0.0135
0.0367
0.0637
0.1369
0.2281
0.1155
0.0759
0.0817
0,1416
0.2298
0.2272
0.1407
0.1214
0.1541
0.2339
0.5572
0.3733
0.2952
0.2278
0.2624
1.0959
0.8193
0.6748
0.4144
0.3491
Table 8: TCN noise suppression results (RMSE)
Inference (signal noise)
Training (signal noise)
0.0
0.1
0.2
0.5
1.0
0.0057
0.0051
0.0064
0.0109
0.0173
0.0168
0.01433
0.0138
0.0158
0.0207
0.0321
0.0271
0.0252
0.0253
0.0284
0.0790
0.0666
0.0611
0.0581
0.0587
1.0
0.1551
0.1320
0.1216
0.1146
0.1133
12

---

Under review conference paper ICLR 2021
Ground truth
Observations
Network output
Network output (AT)
Figure 6: Four consecutive snapshots (from left right, with distance of 50 time steps ) of the wave propagating through large 40 grid comparing regular inference and Active Tuning on a noise-unaware network (0.0).
A.5 ACTIVE TUNING PARAMETERS
The tables below report all parameters of Active Tuning including the parameters for state adaptation with Adam for all experiments.
Table 9: Active Tuning parameters MSO noise suppression experiment
Training noise
Signal noise
Tuning length (R)
Tuning eyeles (C)
1)
B
0.0
0.1
10
0.005
0.9
0.99
0.0
0.2
10
0.005
0.9
0.99
0.0
0.5
14
10
0,006
0.9
0.99
0.0
1.0
16
10
0.004
0.5
0.99
0.05
0.1
10
0.008
0.9
0.99
0.05
0.2
12
0.005
0.5
0.999
0.05
0.5
10
0.007
0.9
0.99
0.05
1.0
16
10
0.006
0.5
0.9

---

Under review conference paper ICLR 2021
Table 10: Active Tuning parameters MSO missing data experiment
Missing data probability
Tuning length (77)
Tuning cycles (C)
S1
Ba
0.5
20
0.005
0.9
0.99
0.6
10
0.005
0.9
0.99
Table 11: Active Tuning parameters pendulum noise suppression experiment
Training noise
Signal noise
Tuning length (I2)
Tuning cycles (C)
1/
B1
0.0
0.1
10
0,005
0.9
0.99
0.0
0.2
10
0,005
0.9
0.99
0.0
0.5
10
0.004
0.5
0.99
0.0
1.0
12
10
0.004
0.5
0.9
0.05
0.1
10
0.008
0.9
0.99
0.05
0.2
10
0.005
0.5
0.99
0.05
0.5
10
0.004
0.5
0.99
0.05
1.0
12
10
0.005
0.5
0.9
Table 12: Active Tuning parameters- - peedulum missing data experiment
Missing data probability Tuning length (I?)
Tuning cycles (C)
31
82
0.1
20
0.005
0.9
0.7
20
0.005
0.9
Table 13: Active Tuning parameters
wave noise suppression experiment
Training noise
Signal noise
Tuning length (R)
Tuning cycles (C)
8
Ba
0.0
0.1
10
0.01
0.9
0,999
0.0
17
10-5
0.0
0.5
20
8 x 10
0.0
0,999
0.0
1.0
30
x 10-5
0.0
0.999
0.05
0.1
0.012
0,999
0.05
0.2
17
1 10 4
0.0
0,999
0.05
0.5
20
1 10
0.0
0.999
0.05
1.0
30
x 10
0.0
0.999


Reviews:
[{"review_id": "fhawELqBPqq", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper proposes a way to adapt an autoregressive model (RNN in examples) to the incoming noisy signal to generate noise-free data output. The approach is interesting due to applying updates to the hidden state of the past observation. The proposed approached is named Active Tuning and evaluated on 3 toy tasks. The idea sounds interesting, however the lack of comparisons with other approaches and theoretical justification of why this approach is superior makes it hard to convince reader. \n\nQuality: Paper is well written and most of the concepts is clear. However, paper will benefit from a better explanation of the method, simpler diagram and equations to remove uncertainty on implementation. \n\nOriginality: I believe the idea is novel and interesting for community. It has a potential to outperform meta-learning and sequence-to-sequence models on the task of model adaptation to noisy samples. \n\nPros:\n- Idea is interesting and has potential. \n- Explanation is clear, but still can be improved. \n- Provided experiments show benefits of the proposed method with respect to direct regression task (same model trained with less or more noise amount)\n\nCons:\n- Comparison with other techniques such as meta-learning, sequence-to-sequence models is required to understand the potential of the method.\n- Same comparisons might be interesting for tuning weights instead of hidden states. Or having only a small part of the model to be tuned (like the last layer). \n- Application to more practical problems could benefit the paper. For example image denosing task could be relevant (works like Noise2Noise, Noise2Self etc)", "extra_scores": {}, "extra_text": {}}, {"review_id": "Urp6dKu-zXu", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an interesting paper on an idea introduced by the authors as active tuning. This paper is well-written and clearly explains the proposed active tuning scheme. I read the paper carefully multiple times, and feel that a few inclusions will help the readers better understand the proposed method.\n\nAt the base level, this paper builds on optimizing the internal dynamics of a recurrent neural network unlike optimizing internal weights in traditional sequence-to-sequence mapping. This is achieved by decoupling the recurrent neural activities from the input temporal signal and propagating the error (the difference between the estimated input value and the observed input value of the input signal) to tune the internal dynamics of the network. To demonstrate the effectiveness of active tuning the authors trained a distributed graph recurrent neural network (DISTANA) on three datasets with increasing complexity. Datasets included: multiple super-imposed sine waves, a chaotic double pendulum, and spatiotemporal wave dynamics.  On average ten independent experiments were performed and the effectiveness of active tuning was evaluated using root mean square (RMS).  Samples for the experiments were generated using five different noise ratios between 0 and 1 to measure the effectiveness of the proposed method for noisy data scenarios. The network was also trained on no noise to 0.05 noise induced into training data to see if it would help the models better generalize. The results as depicted in graphs show that active tuning is not only robust but generalizes well on noisy data. \n\nRecommendations:\n1. The active Tuning algorithm itself is missing from this paper. Even though the explanation is clear, it would help the readers to see the algorithm itself for better understanding. The reviewer referred to Hidden Latent State Inference in a Spatio-Temporal Generative by karlbauer et. al. 2020 (arXiv:2009.09823) for the algorithm. \n\n2. The authors confirm that 10000 and 1000 samples have been generated for all the problem domains tested. However, it is not clear if steps were in place to make sure that no bias was introduced during this sample generation. \n\n3. While the tuning length and tuning cycles were fixed for all three datasets, it is important to see how these values can be optimized based on the complexity of the time series data. Experimental results using a range of values for tunning length and tuning cycles would be beneficial.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7t7g6Xk4Og_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a propagation method to estimate RNN dynamic parameters during the learning process. The algorithm is introduced well and the paper is clearly written.\nThe paper misses a related work section on other tuning methods or absence there of under special circumstances. \nFor the same reason, I am not convinced on the extent of comparisons in the simulation results. A large amount of the focus of the experiments is on the robustness to noise which is fine if there was an equal amount of comparisons against other tuning methods. Otherwise, if the focus of the paper is supposed to be only on noise robustness, I think the motivation in abstract and introduction needs to be clearer.\nWhile the motivation of the paper can be to some extent taken from the results, the introduction does not substantially motivate the problem. \nLastly, I think that majority of details on pages 4 and 5 are unnecessary. Instead, I think a more detailed discussion on comparing additional computational cost of active tuning to other traditional methods would be very useful.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a method to estimate dynamics parameters in recurrent structured models during the learning process. All three reviewers agreed that the idea is interesting and the proposed method could be potentially useful. However, two of the three reviewers have a serious concern about the lack of comparison with other approaches. I agree with these two reviewers; due to the lack of discussion and comparison with existing studies, I cannot recommend accepting this submission in its current form. 
Decision: Reject