Online Continual Learning with Feedforward Adaptation
Authors: ABULIKEMU ABUDUWEILI
Changliu Liu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Online adaptation
Online Learning
Continual Learning

Abstract: Recently deep learning has been widely used in time-series prediction tasks. Although a trained deep neural network model typically performs well on the training set, performance drop significantly in a test set under slight distribution shifts. This challenge motivates the adoption of online adaptation algorithms to update the prediction models in real-time to improve the prediction performance. Existing online adaptation methods optimize the prediction model by feeding back the latest prediction error computed with respect to the latest observation. However, feedback based approach is prone to forgetting past information. 
In this work, we propose an online adaptation method with feedforward compensation, which uses critical data samples from a memory buffer, instead of the latest samples, to optimize the prediction model. We prove that the proposed feedforward approach has a smaller error bound than the feedback approach in slow time-varying systems.  The experiments on several time-series prediction tasks show that the proposed feedforward adaptation outperforms conventional feedback adaptation by more than 10%. In addition, the proposed feedforward adaptation method is able to estimate an uncertainty bound of the prediction that is agnostic from specific optimizers, while existing feedback adaptation could not. 

Under review as a conference paper at ICLR 2023
ONLINE CONTINUAL LEARNING WITH FEEDFORWARD ADAPTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Recently deep learning has been widely used in time-series prediction tasks. Al- though trained deep neural network model typically performs well on the train- ing set, performance drop significantly in test set under slight distribution shifts. This challenge motivates the adoption of online test-time adaptation algorithms to update the prediction models in real time to improve the prediction performance. Existing online adaptation methods optimize the prediction model by feeding back the latest prediction error computed with respect the latest observation. How- ever, the feedback based approach is prone to forgetting past information. In this work, propose an online adaptation method with feedforward compensation, which uses critical data samples from a memory buffer, instead of the latest sam- ples, to optimize the prediction model. We prove that the proposed approach has a smaller error bound than previously used approaches in slow time-varying sys- tems. The experiments on several time-series prediction tasks show that the pro- posed feedforward adaptation outperforms previous adaptation methods by 12%. In addition, the proposed feedforward adaptation method is able estimate an un- certainty bound of the prediction that is agnostic from specific optimizers, while existing feedback adaptation could not.
INTRODUCTION
Time-series prediction (or forecasting) has been widely studied in many fields, including control, energy management, and financial investment Box et (2015) ( BBrokwell & Davis (2002). Among the research applications, acquiring future trends and tendencies of the time-series data is one of the most important subjects. With the emergence of deep learning, many deep neural network models have been proposed to solve this problem| Lim & Zohren (202I). e.g., Recurrent Neural Networks Lai et al. (2018) and Temporal Convolutional Networks Bai et al. (2018). Inspired by the great suc- cess of Transformer in the NLP and CV community Vaswani et al. 2017 Dosovitskiy et al.) 2020), Transformer-style methods have been introduced to capture long-term term dependencies in time series prediction tasks Zhou et al. (202T). Benefiting from the self-attention mechanism. Transformers ob- tain a great advantage in modeling long-term dependencies for sequential data Brown et al. (2020). Although trained Transformer model (or other big deep neural network models) typically per- forms well on the training set, performance can significantly drop in slightly different test domain or under a slightly different data distribution Popel & Bojar 2018): Si et al. (2019).
In practical time-series prediction problems, there are often significant distributional discrepancies between the offline training set and the real-time testing set. These differences may be attributed to multiple factors. In some cases, is too expensive to collect large unbiased training datasets, e.g., for weather prediction or medical time-series prediction. In other cases, it may be difficult to obtain the training instances from specific domain. For example, human-robot collaboration, it is hard to collect data from all potential future users. In these cases, adaptation techniques applicd to deal with the distribution mismatch between offline training and real-time testing Blum (999). Besides, some tasks require the system to adapt itself after every observation. For example, in human-robot collaboration, the robot needs to continually adapt its behaviors to different users. In these scenarios, online adaptation techniques
often embraced Abbudweell et al. (2019).
Online adaptation is a special case of online continual learning, which continually learns from real-time streaming data. In online adaptation, a prediction model receives sequential observations,

---

Under review as a conference paper at ICLR 2023
and then online optimization algorithm (e.g. SGD) updates the prediction model according to the prediction loss measured by the observed data. The goal of online adaptation is to improve prediction accuracy in subsequent rounds. Online adaptation is currendy applicd to many kinds of research like time-series. prediction Pastor et al. (201 AAuddweiii & Liu (2020), image recognition Lee & Kriegman (2005); (Chen et al. (2022), and machine translation Martinez- Gömez et al. (2012). In this paper, we mainly focus on time-series prediction tasks, but the proposed methods also can be used for other online adaptation (or online learning) tasks.
Most existing online adaptation approaches are based on feedback compensation Tonioni et al. (2019), analogous feedback control. In feedback adaptation. a prediction model only utilizes the latest received data. After observing a new sample, the online optimization algorithm updates the prediction model according to the prediction loss measured between the last prediction and the latest ground truth. However, this kind of passive feedback compensation is not efficient.
In this work, we propose feedforward compensation online adaptation to maximize information extraction from existing data, especially those that are more critical. A critical sample is more helpful to reduce the objective (loss) of the model when the sample is selected for training. In the proposed feedforward adaptation, we will not only have forgetting as is done in conventional online adaptation Palcologu et al. (2008). but also enable recalling to compensate for potential short- sighted behaviors due to forgetting. There is a balance between forgetting and recalling. On the one hand, to rapidly learn the new function value in a time-varying system, we need to forget some of the old data. On the other hand, too much forgetting may cause unstable and incorrect predictions when we encounter a similar pattern with historical data. To achieve the balance between forgetting and recalling, we design a novel mechanism for feedforward compensation using memory buffer similar to the functionality of the Hippocampus in the human brain Barron (202T). I). We will maintain the memory buffer by storing recent L-steps observations (or hidden features) of samples. When the prediction model experiences similar observations, it will pull the corresponding data (critical sam- ple) from the memory buffer to enhance learning. For example, in human behavior prediction tasks, human subject may exhibit similar behavior patterns on different days. These would be extremely difficult to discover if we only learn from the most recent data like conventional online adaptation but can be identified using the feedforward adaptation methods with memory buffer. We can also use the related information between the current sample and critical samples to measure the uncertainty bound the current prediction. Our main contributions can be summarized in the following points.
By summarizing feedforward and feedback adaptation methods, we provide a general online test-time adaptation framework and prove its error bound.
We propose feedforward compensation for online test-time adaptation problems. We prove that the proposed feedforward adaptation method has smaller error bound than previously used feedback methods.
We propose an uncertainty- bound estimation related to the feedforward approach, which is ag- nostic from specific optimizers.
We conduct extensive experiments to show that the proposed feedforward adaptation is superior to conventional feedback adaptation.
2
PROBLEM OVERVIEW
2.1 TIME-SERIES PREDICTION
The time series prediction problem is to make inferences on future time-series data given the past and current observations. We consider a multi-step prediction problem: using recent I steps' ob- servations to predict future O steps' data. Assume the transition model is composed of a feature extractor (or Encoder) E and predictor (or decoder) f. At time step f, the input to the model is [z]] which denotes the stack of I-step recent observations. The output V142,
X, of the model is Y1+1 = 1b++1.9422.-. ... Me+o], which denotes the stack of O-step future predictions. The observations I₁, yr are vectors that may contain trajectory or feature, and x, = y, for some cases (e.g. univariate prediction). The transition model for time series prediction be formulated
Z₁ = E(X)):
(1)
Y141 f*(),
(2)
where Z, is a hidden feature representation of input X,. Feature extractor E does not change over time, while predictor fa changes over time. Let fe denote the ground-truth predictor, that generates
2

---

Under review as a conference paper at ICLR 2023
ground-truth output Y1+1- In online adaptation, we use the parameterized model (e.g. Neural Net- works) f(O,, Z+) with learnable parameters 0g to estimate the ground-truth predictor ft (Z,). This paper assumes that the encoder E' is fixed in online adaptation, which could be trained offline before online adaptation or non-parametric feature extractor
2.2 ONLINE ADAPTATION
Due to the temporal nature of time series prediction, the output (future) space in prediction tasks is not fixed. Since the train set and test set are split in chronological order, train-test distribution mismatch is very common in time-series prediction. Thus, online test-time adaptation is crucial to overcome the distribution mismatch problems and make the prediction robust to time-varying and heterogeneous behaviors.
Online adaptation also can be called adaptable prediction, since it makes an inference concurring with updating model parameters. Online adaptation explores local overfitting to minimize the pre- diction error: at time step i, the prediction error c++1 is to be minimized. The optimization objective shown below
Lerr min er+1 = min |Y++ ʃ(0,, Ze)lep>
P:
8:
where fe(Z,) Y++1 [1++1 1/ ... Vt+o] is the ground truth observation (to be received in the future) and f0, 7,)) Y1+1 [ 0142.'.. ... :l++ is the predicted outcome from the be in any Ep norm.
learned model parameter 0g. The adaptation objective
In conventional (feedback) online adaptation, the objective of minimizing the prediction error the future Eq. (3 can be approximated by minimizing the fitting error in the past, as shown below
f, min
I| f(ee, Z)llps
where 0 A < is forgetting factor. The model parameter O, is updated iteratively when new observations are received. Then new prediction is made using the new estimate. In the next time step, the estimate will be updated again given a new observation, and the process repeats. It is worth noting that the observation we received at time t is Vr- The other terms in Y, remain unknown. This paper focused on adaptation methods using one-step-ahead observation. is also possible to conduct online adaptation with multi-step ahead observations Abuduweili & Liu (202T)
In this paper, we propose feedforward adaptation method, whose objective is different from feed- back adaptation. In the feedforward adaptation, the objective of minimizing the prediction error in the future Eq. 3 can be approximated by minimizing the upper bound of the prediction error the future, as shown below
Eff min Bound[er41l = min Bound|]| f(0,, Z.)lpp]
We will show the algorithms and effectiveness of the feedforward adaptation in section 3 In specific adaptation algorithms, the feedback and feedforward approach differ by sample selection strategy.
Algorithm 1 General Online Adaptation Framework (Adaptable Prediction)
Input: Initial predictor f(oo: :) with parameters dg. Feature Extractor E, Optimizer (l,;;:)
Output: Sequence predictions (1141)/-1
1: for t= 1, 2, ... I'' do
2: Receive the ground truth observation values Zt. V/:
3: Find the critical input-output pairs (Ze, Mo+1) for 1 < s < t, where Z, = E(X,)
4: Adaptation: 0: (0,-11 Det1, Vo+1), where Y,41 [8+11 -9+00 f(0-1,, Zz)
5: Prediction: Y 10+1,"' Dc+0] = s(0,, Zi), where Z. E(X+)
6: end for
II special case, the feature extractor can be an identity mapping F F(X) X, then we adapt the end-to- end neural network model in online adaptation
3

---

Under review as a conference paper at ICLR 2023
We provide general online adaptation framework shown in algorithm [ by incorporating ventional feedback with the proposed feedforward adaptation. At time step t. after receiving the current observations (z,, We), we select the critical imput-output pair (2,. from historical ob- servations. The critical pair was used to adjust the parameters of the prediction model by an online optimizer (e.g. SGD). Then obtain the current prediction result with the adapted model. The main difference between different adaptation algorithms lies in the critical pair selection strategy (line in algorithm D In feedback adaptation, the critical input-output pairs are composed by the latest observations Z, = Zt-1- In random adaptation, the critical input-output pairs are ran- domly sampled from historical observations Z 11..... Zt-1]- We will show in section 3 in feedforward adaptation, the critical input-output pairs are the most similar samples to the current
observation Zμ = arg ming || 2 Zl1.
3
FEEDFORWARD ADAPTATION APPROACH
3.1 ERROR BOUND FOR GENERAL ONLINE ADAPTATION
The theoretical analysis of the paper based on two basic conditions about the local smoothness property of ground-truth predictors ft.
K-Lipschitz continuity condition. For a time-step and Vs E [t L, 1). we have local K
Lipschitzness for ground-truth prediction function Sr and recent L steps input data:
(Z.) fz(2)))l - K Zull:
(6)
where K is the bound (real number) for the change of the value of the function over input space. Intuitively, a Lipschitz continuous function is limited in how fast function value can change over input space. It is proven that every function that has bounded first derivatives is Lipschitz continuous Sohrab (2003). Similar to Eq. 6). we assume our parameterized function (O,,;) (e.g. Neural Networks) has Lipschitz continuity with constant value K:
[0,, Z.) f(0,, 2ill s RI - ZuIl-
K-Lipschitz continuity common in machine learning because neural networks have bounded first derivatives by proper training.
time-varying condition. For a time-step t and Vs E [t -,, 1]. assume the ground-truth prediction functions has bounded changes within recent T. steps under the same input Z5:
Il f. (Z.)
where 8 is the bound (real number) for change of the value of the function sequences over time fixed input. A bounded time-varying condition is a common condition in time- series tasks because obtain similar future time-series signals with the same input in most cases.
Theorem (Error Bound of Online Adaptation). Under the K K-Lipschitz continuity condition (Eq. 66) and (7) and ó time-varying condition (Eq. (8). the (prior) prediction enror c++1 of general has the following upper bound:
online adaptation (algorithm [
c S K Z. + 5 |Y f(0.. Z.)l [1f Z.) - f(0,, ZJ)l
(9)
< (K + K)|IZ, Z.1 8 | - Zo)l
(10)
Please check the proof appendix C.T
3.2 FEEDFORWARD ADAPTATION
As discussed in section 2.2 the goal the online adaptation is to minimize the prediction error in the future Eq. 33. Due to the lack of ground-truth value in the current steps, it is not feasible to directly minimize the prediction error. Conventional online adaptation methods approximate the original objective by minimizing the fitting error in the past Eq. 4h. In this work, approximate the original objective by minimizing the bound of the prediction error Eq. (5). Which is equivalent to optimizing the worst-case scenarios. The worst-case performance analysis is really useful in
4

---

Under review as a conference paper at ICLR 2023
real-world applications Roughgarden 2021 We will show that, under the following assumption, feedforward adaptation provides better results than feedback adaptation.
Temporarily Slow Time-varying Assumption. For the 8 time-varying function fi. assume 6 is iimeiindependent or it slowly changes within recent L-steps: 8t O.
as
Note that, only assume the ground-truth function fa varying slowly (locally) within recent L- steps not (globally) for every step. Thus the assumption is reasonable because the major changes in function value caused by the change of imputs instead of the tine-dependency in many real- world prediction problems. In addition in some cases, 8 is caused by random noises, then which still follow our assumptions. Under the above assumption, Then the optimization objective for the error bound is shown below:
ignore the term in optimization.
fs min Bound[es41] = min (K + R Z: - Z. + |1 Z.)ll 0..eE[t-LA-11)
(11)
However, we still can not directly minimize the above objective, because we not know K, R. Then a simplification is applied to the above objective to make it applicable. We change the joint minimization over and 0g to bi-level optimization which first minimizes the first term of objective over sampling time-step s, then minimizes the second term of objective over parameter O,. Thus, the simplified objective function is shown below:
final min 8:
-1 f(e., Z,+)ll
-
(12)
s.t. arg min 1z Z.
(13)
eE(t-L,-11]
In a summary, the proposed feedforward adaptation method selects the most similar samples to the current observation as critical pair (Z vsr) by Eq. (13). Then using the critical pair to optimize the prediction model by Eq. (2)
3.3 UNCERTAINTY ESTIMATION
The error bound Eq. ( provides uncertainty estimation of the prediction results. Here we use estimation of K, and 8 to approximate real K and Eq. (9) We use confidence factor (0, 1] to decay the error bound. The uncertainty estimation Ú,41 for prediction Yc+1 1 is shown below:
Ū4+1 = [ Z. Z. + | f(0,, Z.)1l If((,, Z.) f(A, Z.)l)
(14)
Where = 1.-1 for feedback adaptation and s = s* (Fq. (13) for feedforward adaptation. The con- fidence factor is predefined hyperparameter, e.g. 0.9 for 90% confidence of uncertainty.
and K, are also predefined hyperparameters, e.g. 8 0.001, K = 1. For more accurate uncertainty estimation, can set small fixed value, and iteratively update K, according to the estimated uncertainty U. and real error et. If the previous uncertainty estimation is much larger than the real error then we shrink the K, value. If the previous uncertainty estimation is much smaller than the real error then we enlarge the K, value. The K, estimation criteria are shown below.
K1-1 if Ú, 2e,, K, 1.5K, if Ú, & 0.5e,, K, Kr- if 0.5e, < Ü. 2e, (15) K,
1.5
In adaptation and prediction, feedforward adaptation does not require K and values. In the uncer- tainty estimation, we need to approximate K and 8 or serve these as predefined hyperparameters, but an accurate approximation is not necessary. Since mainly consider the relative uncertainty estimation between different samples or different optimizers. As shown in Eq. (4). the inaccuracy of K and e mainly raise a data- indepeddddnn error, which is no impact on the data-selection strategy and optimizers. The overall feedforward adaptation algorithm is shown in algorithm 2 The ~ lines of algorithm is corresponding to the 1 lines of algorithm Buffer is used to store recent Z-step observations.
5

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Online Adaptation with Feedforward Compensation
Input: Initial predictor f(oo:)) :) with parameters 8g. Feature Extractor E, Optimizer (:,;;:)
Input: Empty L-size buffer B
Output: Sequence of predictions {Y++1 and estimated uncertainty {0 1
do 2: Receive the ground truth observation values 21, yri Construct input X = [ Tzl
1: for 1,
3: Find the critical (similar) input-output pairs (Zar. Vs*
from buffer B by Eq. 13
4: Adaptation by Eq. 12 0, = O(0e: +1: +1)+ where (1s :.. +ol = [0,, 7.
5: Prediction: Ye+1 1 -D++ol [l] = [O2,, Ze where Z. = E(X.)
6: Uncertainty Ü.+1 Estimation by Eq. (14 and Eq. 15
7: Add current data to buffer: B.append (2..)) yb.
8: if size(B) > L then
9: B 4 geep_morr_reed reaeererceentttee samples (B, L)
end if
11: end for
3.4 EFFECTIVENESS OF FEEDFORWARD ADAPTATION
As described in section 3.2 under the temporarily slow time-varying assumption, feedforward adap- tation achieves smaller error bound than feedback adaptation. In this section, we analyze more theoretical results on the online adaptation of some specific functions.
Linear time-variant function. Consider function fa which can be separated into time-invariant function g(2)) and linearly time-dependent part at. Input 2 is a random variable from the uniform distribution (0, 1).
f((2,) = g(x)) + at,
2, ~ ((011)
(16)
Lemma 1. We use neural network f(,::) to learn the linear time-varian function Eq. 16p. Let K
be a Lipschitz constant for the ground-truth function fc, and K is a Lipschitz constant for neural network [O):)). Let 0 and B!/ denote the prediction error bound for feedback and feedforward adaptation. Then we have the following results.
(a) Expectation of the error bound for feedback adaptation is E[B K+R 3
+ a.
(b) Expectation of the error bound for feedforward adaptation is
B [ s (K + R) max() 2 + min(,,
L+1 2
With I. = V 2(K+K) 1, we achieve
the smallest final error bound for feedforward adaptation: B|B!1 S 22x(K + R).
(c) If time-varye varying factor a is smaller, specifically a S a* = 1-v12 KK + K), feedforward adapta- tion has smaller error bound than feedback adaptation.
Time-invariant polynomial function. Consider a time-invariant polynomial ground-truth function f(())) with the input of random variable sampled from the uniform distribution (1 1).
De+1 fe(ze) = Σ
Wiet
2, 1)
(17)
1=1
Lemma 2.
We use linear projection f(V,, 2x) V&a, to learn the time-invariant polynomial function
Eq. [1 Then we have the following results.
(a)
Expectation of the prediction error for feedback adaptation is E[e/"1 2
1
NN11 N2
Σ Li=1
W4+1-
3
(b) Expectation of the prediction error for feedforward adaptation is
[leffi) < max( 7 LFI Enr
N-1 W The final error converges lim
E (((41)
= 0.
(c) Feedforward adaptation is provably better than feedback adaptation.
The proof of Lemma and Lemma are provided in appendix C.2 Thus feedforward adaptation can be used to learn time-invariant functions and slow teme-varying systems. These kinds of systems very common in the world. For example, the exchange rate and incidence rate for common diseases have slow time-dependent shift over time.

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
4.1 SYNTHETIC EXPERIMENTS
To evaluate Lemma in section 3.4 we consider the following time-varying function.
Ve+1 fx(=)) sin 24 + ot, 21
(0,
(18)
According to Lemma 1, we derive the error bound for feedback adaptation B and feedforward adaptation B shown below (please check details appendix D.I.
L+1 2
B[B{"]
+ a, 12
BIB 5 4(L +
+
(19)
5 V 2 with L V
5
E
E
(20)
2ax
Then calculate the threshold a* If S a², feedforward adaptation has smaller error bound.
0.1, L
R:
3
(21)
2a*
Target le 0.001)
(o:1)
Grounstruth Feedfoowwrd
[b.3]
Groundtruuh Feedback
feedformand
Reedforward (L-5)
0.4
feedforwand (1:100]
2.0
Focdforward (L=100)
Awerage
0.0
Target le 0.011
(b.2)
(b.4)
0.2
2.5
0.1
0.5
10
10
10
0.0
40
600
100
40
60
80
Time Step
Slep
(a) Average error
(b) Prediction output
Figure 1: (a) Average eигог feedback and feedforward adaptation with different buffer size L and time-varying factor a. (b) Prediction output of feedforward (blue curve) and feedback adaptation (green curve) with buffer size 100 and time-varying factor 10-3 and a 10-2
Figure Ta shows the average prediction error of feedback and feedforward adaptation under different vare-iarying factors Note that, feedforward adaptation with L is the same feedback adaptation. We will show that the results in Fig. carorresdond to the theoretical results from Lemma 1. (1) As be seen, for smaller time-varying factor a, feedforward adaptation with larger buffer size achieves smaller prediction error. Besides, if a larger, the performance of the feedforward adaptation drops. These are consistent with Eq. (20). For example, feedforward adaptation with buffer size L = 100 achieves the smallest prediction еттоr when a < 10-3 (2) If < 0.1, feedforward adaptation with buffer size L 3 performs better than feedback adaptation. But > 0.1, feedback adaptation is better. The threshold a* z 0.1 is consistent with Eq. (2
Figure db illustrate the comparison between feedforward and feedback adaptation. The first row shows the prediction results for a 10 target function of feedforward adaptation with L = 100 (b.1 subfigure) and feedback adaptation (b.3 subfigure). As can be seen, feedforward adaptation can learn the ground truth more precisely. The second row shows the prediction results for 10-2 target function. For this case, prediction with feedforward adaptation (b.2 subfigure) has significant shift from the ground truth, although it is still able to learn some input-dependent details. Compared to feedback adaptation, feedforward adaptation is more focused on the input-dependent part of the ground truth function and pays less attention to the time-var varein shift of the function. These empirical results validate the conclusions in section 3.4 The experimental evaluation of Lemma 2 shown in appendix D.2

---

Under review as a conference paper at ICLR 2023
4.2 EXPERIMENTS ON REAL WORLD DATA
In this section, we evaluate the proposed feedforward adaptation on four real-world benchmarks. including ETT (Electricity Transformer Temperature), Exchange-Rate, ILI (Influenza-like Illnes)), and THOR human motion trajectory dataset. We include two models as our learnable prediction function f(0,:) :): Informer Zhou et al. (2021) and simple MLP (multi-layer perception).
Experimental Design. First we offline train models (Informer and MLP) on a train set. After training. we will incrementally receive the data point from the test set like in real-world applications. At each time step, we conduct an online adaptation to optimize the model with selected previous observations by feedback or feedforward compensation. Then the prediction output is inferred from the updated model. We evaluate the prediction results with mean squared error (MSE) and mean absolute error (MAE). In the experiments, we only adapt the decoder of the model and make the encoder fixed. Please check the detailed experimental design in appendix D.3
Baselines. We compare the proposed method with four baselines. 1) w/o adapt directly conduct pre- diction without adaptation. Which is a lower bound for adaptation methods. 2) Feedback adaptation is the most important baseline to us. 3) Random adaptation is method that selects the critical pair from the L-size buffer with random sampling. 4) Full adaptation is a method that uses all samples from the buffer to adapt the model, which is similar to offline training.
The detailed results of the experiment on SGD optimizer are shown in table I In table I] the last row denotes the performance gain of the proposed feedforward adaptation over feedback adapta- tion in terms of average result. Feedforward adaptation achieves the best results on all four datasets. Specifically. feedforward udaptation outperforms feedback adaptation by 12.6% 7. 7.4%, 22. 22.34% and 8.1% in terms of average MSE on four datasets respectively. Feedforward adaptation is better than full adaptation, which means adapting models with more samples is not effective, because online adaptation focuses on the ability to rapidly learn and adapt in the presence of non-stationarity in- stead of generalization ability like offline learning. Feedforward adaptation is better than feedback and random adaptation, which means the sample selection strategy based sample similarity in feedforward adaptation is more critical than the time-based sample selection and random sampling.
Results.
Table 1: Performance comparison between the proposed feedforward adaptation method and other baselines. Avg denotes the average results of two models (MLP and Informer). The last row denotes the performance gain of feedforward adaptation over feedback adaptation. We use boldface and underline for the best and second-best average results.
Method Dataset
ETTh1
Exchange
ILI
THOR
Adaptation
Model
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MLP
0.195
0.371
0.549
0.540
4.348
1.413
0.135
0.117
w/o Adaptation
Informer
0.211
0.389
1.128
0.858
4.942
1.531
0.137
0.171
Avg
0.203
0.380
0.839
0.699
4.645
1.472
0.136
0.144
MLP
0.142
0.311
0.362
0.443
3.844
1.340
0.127
0.150
Full
Informer
0.146
0.307
0.503
0.567
3,413
1.252
0.213
0.287
Adaptation
Avg
0.144
0.309
0.433
0.505
3.628
1.296
0.170
0.219
MLP
0.145
0.312
0.365
0,445
3,865
1.347
0.132
0.148
Random
Informer
0.131
0.283
0.497
0.553
3.867
1.380
0.204
0.271
Adaptation
Avg
0.138
0.298
0.431
0.499
3.866
1.364
0.168
0.210
MIP
0.153
0.317
0.349
0.442
3.868
1.334
0.112
0.119
Feedback
Informer
0.116
0.269
0.326
0.461
4.383
1.411
0.134
0.173
Adaptation
Avg
0.135
0.293
0.338
0.452
4.123
1.373
0.123
0.146
MLP
0.128
0.286
0.347
0.432
3,041
1.193
0.102
0.108
Feed Forward
Informer
0.109
0.259
0.276
0.399
3.363
1.312
0.125
0.156
Adaptation
Avg
0.118
0.272
0.311
0.415
3,202
1.252
0.113
0.132
Gain of Foedforward Over Foodback
12.6%
7.2%
8.0%
8.2%
22.3%
16.1%
8.1%
9.6%
We found another interesting result by comparison between Informer and MLP on table Without adaptation, the simple MLP outperforms the Informer average. This is caused by the overfitting of the Informer on the training set. However, by online adaptation, especially with feedforward adaptation, the performance of Informer is greatly increased. For example, the performance of Informer on the ETThl dataset increased from 0.211 to 0.109 by 48% and outperform the MLP model. That shows Informer has greater representation ability but may perform poorly the test
8

---

Under review as a conference paper at ICLR 2023
set. Considering this case, it is embarrassing to directly use the big Transformer-lik models real-world time-series prediction. But online adaptation may make the big Transformer- models powerful again. This phenomenon emphasizes the importance of online adaptation in real-world time-series prediction.
(8)
Cor (perern 15
(C)
Vethel
(e)
Rarden Poolian
12
532
1833
1500
2000
1500
Time dhitt se exted samp c-
120
(b)
1 15 8
(d)
Matrod
Put
(1)
Coant (pencat))
Feedlact Feedoorac
Period
291
433 ECE
selocted
200
1059
Time ship
Figure 2: Experiments on ETTh1 data. (a) Time series output; (b) FFT period analysis; (c) Time shift / 8 between current sample Ze and selected sample Z, in feedback and random adaptation; (d) Time shift t s in feedforward adaptation; (e) Online prediction error. (f) Sample difference 1| Z, between current sample Ze and adapted sample Z..
Discussion.
We will show that the sample selection strategy in the proposed feedforward adaptation method could intrinsically mine the periodicity of the input data. Figure 2 shows the ETTh time- series data, and Fig. (b)) is the FFT (Fast Fourier Transformation) period analysis of the ETTh1 data. As can be seen ETThl has roughly T Rs 500 repetition periods. Random adaptation randomly selects samples from the buffer, then the time shift between current time-step / and the time-step for selected samples is (L -
LJ: Feedback adaptation only selects the last time sample to optimization, then (t-s) = This can be found from the distribution of time shift (t - s)
in Fig. c). For the proposed feedforward adaptation in Fig. 2 d), many samples were selected from (t Rs 500 steps earlier, which corresponds to the repetition period of T zs 500. Because feedforward adaptation selects the most similar samples the current sample, which helps to extract the hidden periodicity of the input signal over time. Thus the distribution of t - s is similar to FFT period analysis. Experiments on different datasets are shown appendix D.5 According to theorem 1 (Eq. (10), the error bound is related to the sample difference || - Z=ll- Figure shows the real prediction error for different adaptation methods over time. As can be seen, feedforward adaptation has the smallest prediction error, because feedforward adaptation has the smallest sample difference || Z,| during adaptation, as shown Fig.
One of the advantages of the proposed feedforward adaptation is that it could provide uncertainty estimation. The results of prediction output and uncertainty are shown in appendix [.6
5 CONCLUSION
This paper studies an effective feedforward adaptation algorithm for time-series prediction tasks. Firstly we propose the general framework for online adaptation which includes feedback and feed- forward adaptation. Then we propose the feedforward adaptation algorithm by selecting the most similar critical samples for optimization. We prove that, in a time-invariant or slow time-varying system, the feedforward adaptation has a smaller error bound than conventional feedback adapta- empirically validate the effectiveness of the proposed algorithm both
tion. In the experiments, on synthetic and real-world data. In the end, we show that online adaptation can greatly improve the performance of Informer other models.
REFERENCES
Abulikemu Abuduweili and Changliu Liu. Robust online model adaptation by extended kalman filter with exponential moving average and dynamic multi-epoch strategy. In Learning for Dynamics and Control, pp. 65-74, PMLR, 2020.

---

Under review as a conference paper at ICLR 2023
Abulikemu Abuduweili and Changliu Liu. Robust nonlinear adaptation algorithms for multitask prediction networks. International Journal of Adaptive Control and Signal Processing, 35(3): 314-341, 2021.
Abulikemu Abuduweili, Siyan Li, and Changliu Liu. Adaptable human intention and trajectory prediction for human-robot collaboration. arXiv preprint 1rXiv:199,,05089, 2019.
Nesreen K Ahmed, Amir F Atiya, Neamat El Gayar, and Hisham El-Shishiny. An empirical com- parison of machine learning models for time series forecasting. Econometric reviews, 29(5-6): 621, 2010.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterization. In International Conference Machine Learning, pp- 242-252. PMLR, 2019.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.0127/ 2018.
Helen C Barron. Neural inhibition for continual learning and memory. Current opinion in neurobi- ology, 67:85-944 94, 2021.
Shubhendu Bhasin, Rushikesh Kamalapurkar, Huyen T Dinh, and Warren E Dixon. Robust identification- based state derivative estimation for nonlinear systems. IEEE Transactions on Au- tomatic Control, 58(1):187-192, 2012.
Avrim Blum. On-line algorithms in machine learning. Online algorithms, pp. 506-325, 1998.
George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
Peter Brockwell and Richard Davis. Introduction to time series and forecasting. Springer, 2002.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurul information processing systems, 13:1877-1901, 2020.
Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 295-305, June 2022.
Yujiao Cheng, Weiye Zhao, Changliu Liu, and Masayoshi Tomizuka. Human motion prediction using semi-adaptable neural networks. In 2019 American Control Conference (ACC), pp. 4884- 4890, 2019. doi: 00.29119AACC2201998884444
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-3355 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
Andrew H Jazwinski. Stochastic processes and filtering theory. Courier Corporation, 2007.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Grabska-Barwinska. al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-335 2017.
Jyrki Kivinen, Alexander Smola, and Robert C Williamson. Online learning with kernels. IEEE transactions on signal processing, 52(8):2165-22 221277666766, 2004.
Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al. Universal source- free domain adap- tation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4544 4553, 2020.
10

---

Under review as a conference paper at ICLR 2023
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. The 41st international ACM SIGIR conference research & development in information retrieval, pp. 95-104, 2018.
Kuang- Chih Lee and David Kriegman. Online learning of probabilistic appearance manifolds for video-based recognition and tracking. In 2005 IEEE Computer Society Conference on Computer Vision and Pullern Recognition (CVPR (C), volume pp. 852-859. IEEE, 2005.
Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Diaz-Rodríguez. Continual learning for robotics: Definition, framework, learning strategies, op- portunities and challenges. Information fusion, 58:52 68, 2020.
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Wu. Model adaptation: Unsuper- vised domain adaptation without source data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patern Recognition, pp.
2020.
Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A, 379/2194):202002099 2021.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-ccmpyexity pyramidal attention for long-range time series modeling and fore- casting. In International Conference on Learning Representationss, 2021.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey, Neurocomputing, 469:28-51, 2022.
Pascual Martínez-Gern Gómez, Germán Manchis-Trilles and Francisco Casacuberta. Online adaptation strategies for statistical machine translation in post-editing scenarios. Pattern Recognition, 45(9): 3193-3203, 2012.
Constantin Paleologu, Jacob Benesty, and Silviu Ciochina. A robust variable forgetting factor re- cursive least-squares algorithm for system identification. IEEE Signal Processing Letters, 15: 597-600, 2008.
Peter Pastor, Ludovic Righetti, Mrinal Kalakrishnan, and Stefan Schaal. Online movement adap- tation based on previous sensor experiences. In 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 365-371. IEEE, 2011.
Martin Popel and Ondřej Bojar. Training tips for the transformer model. arXiv:1804.002477 2018.
arXiv preprint
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Tim Roughgarden. Beyond the worst-case analysis of algorithms. Cambridge University Press, 2021.
Dennis W. Ruck, Steven K. Rogers, Matthew Kabrisky, Peter S. Maybeck, and Mark E. Oxley. Comparative analysis of backpropagation and the extended kalman filter for training multilayer perceptrons. IEEE transactions on pattern analysis & machine intelligence, 14(06))686-691, 1992.
Andrey Rudenko, Tomasz P Kucner, Chittaranjan S Swaminathan, Ravi T Chadalavada, Kai O Arras, and Achim Lilienthal. Thör: Human-robor navigation data collection and accurate motion trajectories dataset. IEEE Robotics and Automation Letters. 5(2):676-682 2020.
Wenwen Si, Tianhao Wei, and Changliu Liu. Agen: Adaptable generative prediction networks for autonomous driving. 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 281-286. IEEE, 2019.
11

---

Under review as a conference paper at ICLR 2023
Houshang H. Sohrab. Basic real analysis, volume 231. Springer, 2003.
Alessio Tonioni, Fabio Tosi, Matteo Poggi, Stefano Mattoccia, and Luigi Di Stefano. Real-time self-adaptive deep stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 195-204, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation entropy minimization. arXiv preprint (XXv:2006.10726 2020.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans-
formers with auto-correlation for fong-term series forecasting, Advances in Neural Information Processing Systems, 34:22419-22430, 2021.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205. 13504, 2022.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang- Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 11106-11118 2021.
12

---

Under review as a conference paper at ICLR 2023
A
RELATED WORKS
Time series prediction. Time-series prediction historically has very diverse applications across various domains Lim & Zohren (202T). These applications trigger many methods for time-series prediction. Modern machine learning methods provide a purely data-driven manner without do- main knowledge Ahmed et al. (2010). In recent years, deep neural networks becomes the domi- nant approach inspired by notable achievements of deep learning in computer vision [Dosovitskiy] et al. (2020) and natural language processing Vaswani et al. (2017). Among deep neural networks, Transformer-styy models are becoming more and more popular in time-series prediction tasks, since which has the potential to capture long-term dependencies [Brown et al. (2020). Most Transformer- style models focus on long-teri time series forecasting problem. including Informer Zhou et al. (02T), Autoformer Wu et al. (02T), and Pyraformer Liu et al. (02T).
Continual Learning also known as Lifelong learning, aims at providing incrementally updated knowledge in an ever-changing environment. From the data statistics perspective, continual learning studies the problem of learning from non-ii.d data, with the goal of preserving and extending the acquired knowledge. Conventional continual learning assumes the new data arrive one task at a time, and the data distribution for each task is stationary De Lange et al. (202T) Ib. Which is also called offline continual learning. Different from the batch training on offline continual learning, Online Continual Learning focuses on the more realistic problem, where data arrive one tiny batch at a time and previously seen batches from the current or the previous tasks are not accessible Mai et al. (2022). Therefore, online continual learning required to efficiently learn from a single sample from the online data stream in single step. Recent works of continual learning can be roughly divided into four families: dynamic architectures, regularization-based rehearsal, and generative replay approach Lesort et al. (2020). Among these methods, Rehearsal-based methods are more related to our work, which use memory replays to enhance the knowledge from the previous data or processes such as iCaRL |Rebufi et al. (2017). [opez & Ranzato (2017). However, most of these methods focus on offline continual learning.
Test-time Adaptation is special setting of domain adaptation and continual learning where trained model on the training domain has to adapt to the testing domain without accessing the train- ing set set| Kundu al. (2020). Most test-time adaptation methods for image data utilize unsupervised domain adaptation algorithms to improve the performance on the test set such as entropy minimiza- tion Wang et al. 2020), pseudo-labeling Li et al. (2020). For the time-series prediction task, the input signal at the current step might be the ground-truth truth label of the previous step. So (partially) supervised learning can be used for test-time adaptation on time-series tasks Abbudwweili & Liu (2021). In many real-world applications, online test time adaptation (online adaptation for short) is more reasonable than offline adaptation. For example, in automotive driving, we only receive an observation (sample) in an online manner at each time step, instead of batch data of offlline setting.
Online Adaptation continually learns from real-time streaming data. Online adaptation is a special case of online continual learning in that the model is always offline trained on training dataset (which can be very small), and then incrementally adapt its models in an online test dataset (Chen et al. (2019). Unlike works that focus avoiding catastrophic forgetting in offline continual learn- ing Kirkpatrick et al. (2017), online adaptation focuses on the ability to rapidly learn and adapt in the presence of non-stationarity. Most existing online adaptation approaches are based feedback compensation with SGD (Stochastic Gradient Descent) or EKF (Extended Kalman Filter) optimizers
Bhasin al. (2012).
B ONLINE OPTIMIZERS
In algorithm [ an online optimizer was used to adjust the model with the selected critical samples: 0, = (0,..1. 10.1 Me+1). Unlike offline machine learning, online optimizers need to handle iter- ative onput-output pairs with single sample. We assume objective function (loss function) has suitable smoothness properties (e.g. differentiable and locally Lipschitz continuous). SGD and EKF two widely used optimizers for online adaptation.
Stochastic Gradient Descent (SGD) is one of the widely used optimizers in online adaptation Kivinen al. (2004). SGD is memory efficient and stable. The update formula of the SGD is shown
13

---

Under review as a conference paper at ICLR 2023
below:
( 20 11++1 f(0, -
lo-0...
(22)
where 7) is the learning rate.
Extended Kalman Filter (EKF) is another strong optimizer online adaptation Jazwinski (007)) In the original EKF, the object being estimated is the state value of dynamic system. In online adap- tation, we can apply the EKF approach to adapt model parameters by regarding model parameters as system states. The EKF approach has faster convergence rate than SGD, but is very memory- exhaustive. When optimizing small neural networks, EKF approach has been demonstrated to be superior to the SGD-based algorithms Ruck et al. (T992); (Abuduweili [Abduuwelll model size is not big, EKF is strong optimizer for online adaptation. The update formula of the EKF shown below:
Liu (202I). If the adapted
H, 0f(0, Zs)
10=b.-1
(23)
08
G. = Pt-1 7-1.117 -1, (H+ Pt-1 H1 + A1)
(24)
0. 0,-1 + +G (Yo41 f(0,-1, Za))
(25)
P. = x -He PP-1)
(26)
where À E (0, is forgetting factor, H, is the gradient matrix, G2 is the Kalman gain, and Pe is matrix representing the uncertainty in the estimates of parameters. The initial value of the matrix Pa can be diagonal matrix Po = po I, ,1) for po > O.
C THEORETICAL RESULTS ON SECTION 3
C.I THEOREM (ERROR BOUND ONLINE ADAPTATION)
Bound of ground-truth difference. If the sereis of ground-trume truth functions ft within recent L steps follows K -LLipschitz continuity Eq. O and time-varying conditions Eq. (8). then the ground-truth value Y++ and Y has the following property:
[| - Ys+1ll S KIIZ: - Zall -
(27)
The proof is shown below:
YY+1 1s4l
[[((Z)) fs(Z.)l
[[((Z)) S((Z.) + fx
[[f((Z)) -f((Z.) +l]((())))) - (triangle inoquality)
< K| 2,l1+ + (lf((Z)) (K Lipschitzness)
< K| Z,| + (8 time varying)
(28)
Error Bound of Online Adaptation. For time step t, the (prior) prediction et+1 has the following inequality:
c111 |Y41 -Y-1ll | f(0s, Z.)ll
||e+1 - -Y,-1 Ys+ - f(0,, Z.) + f(0,, Z.) f(Or, Z.)l
< || Y,-1ll + |Y i(0,, 7.)1 + (1, 7.)) (10,,2)) (triangle inequality)
KK Z, 5 -(a. Z.)ll |l + IIf(0,, f(0,, Z:)ll (Eq. (7 IIl
(29)
The first terms come from the difference between ground-truth Y++1 Y the third term is (posterior) fitting error for input-output tuple (Z., Y,+1). and the last term is the difference between
two predictions. Combining Eq. (29) with Eq. 77). we obtain the error bound for general online adaptation is shown below:
e+11 (K + K) 2.1 + 8 |Y /(0,,2.)1
(30)
14

---

Under review as a conference paper at ICLR 2023
C.2 EFFECTIVENESS FEEDFORWARD ADAPTATION
C.2.1 ERROR BOUND OF ONLINE ADAPTATION ON LINEAR TIME-VARIANT FUNCTION
In this section, we compare the error bound of feedforward and feedback adaptation methods on linear time-variant systems (functions). Consider a function ft which can be separated into time- invariant function (xx)) and línearly time-dependent part at. Input is a random variable from the uniform distribution ((() 1).
~ (0, ====)=(<>) at
(31)
(32)
We use neural network f(0,:) to learn the above function. Let K be a Lipschitz constant for fo (it is also equal to the Lipschitz constant for the function g), and K is Lipschitz constant for neural network (0):). The 8 time-varying condition becomes (z,) f=(2,)ll = allt - a|1, and
time-varying factor 8 = aill s||. Then we have an error bound from Eq. (10):
c++1 (K + R) 2.11 + a s|1 + |1ys+1 f(e. 2g
(33)
Now we consider the comparison of the error bound between feedforward and feedback adaptation. Note that, the last term of the above equation ||y++1 - f(0,. | is a fitting error on input-output tuple (Ee, 3/-+))+)).... ). and the fitting error is irrelevant to feedback feedforward compensation strategy. Besides, in over-pan pher-parameterized neural networks, the fitting error is very small, even can be zero Allen-Zhu al. (2019). Thus we ignore the fitting error in comparison. We now compare the expectation of the error bound. Let Be denote the prediction error bound for online adaptation, B. Bound[er+i]. Then we have the expectation of the error bound for online adaptation as shown below.
E[B.] = (K + B = + aBEl[t s|]
(34)
Then we use Eq. 34 to derive the expectation of the error bound for feedback and feedforward adaptation methods. The core of the proof is estimate [[[] =,Il] and E[[ sl].
In the following sections, we compare the error bounds of four methods. 1) Feedback adaptation is the most important baseline that selects the latest observations to optimize the model. 2) Random adaptation is a method that randomly selects samples from the L-size buffer to optimize the model. method that uses all samples from the buffer to adapt the model, which is similar to offline training. 4) Feedforward adaptation is the proposed method, that selects the most similar samples to optimize the model.
3) Full adaptation
Expectation of the error bound for feedback adaptation. In feedback adaptation, the selected input-output pairs are the latest observations =, = 2t-1 and 8 = 1. The current sample =, and last sample ž are independent random variables from (100) 1). The expectation of the distance between these two independent variables is 3. then [[]| - ==Il] = [B[| 2-1lll = 3. We have
the expectation for the error bound of feedback adaptation /& as shown:
E B = (K + E] - =1 + aBll =|]
= =(K + R) 1 + -k
K + K
+
(35)
Expectation of the error bound for feedforward adaptation. In feedforward adaptation, the selected input-output pairs are the most similar samples to the current observation 2 ain,in,, || =,|l from I-size buffer, and s* = arg min 1 7. The expecta- tion E z=ll] represents the average minimum distance between current sample 2, and previous samples. If < L. 2 24 2z]] considers the minimum distance for samples, which is no greater than :. Ift L, Ei| 2 considers the minimum distance for L + samples in the buffer, which no greater than Zt.. Similarly, El]l sIl] is an average distance between indices for
t < L, which is 4. or average distance from L + 1 indices for 2 I, which is L+1 2 expectation for the error bound of feedforward adaptation B as shown:
We have the
BIB!11 (K + K]][[ z+ll| Ill + a'Ellt s|]
t L+1 1
< (K + R) max 7. L+ + min
(36)
2 2
15

---

Under review as a conference paper at ICLR 2023
We mainly consider the final performance t L, then the above equation can be simplified as:
E|Bl'
K +
++1 2
(37)
L+1
With L = 2( 1, we achieve the smallest error bound for feedforward adaptation:
E BBB V /2a(K + R)
(38)
Expectation of the error bound for random adaptation. In random adaptation, the critical input- output pairs are randomly sampled from historical observations 2, [. , from L size buffer. The current input 2, and selected input 2n are independent random variables from 2((0, 1). Similar to feedback adaptation Bill - 2, |I] = 3' For the term Elilt s|]]. it is similar to feedforward adaptation, because is randomly sampled from [t- I, 1 1]. We have that Bi[K *Il] = &
if t S L, and [i[]i- s|l]
L+1 if L. We have the expectation for the error bound of random 2
adaptation Brnd as shown:
E B = (K + R)El 2 - 2=ll + aEE(t s|l
S (K K): + a min( L
1
(39)
We mainly consider the final performance 1 L, then the above equation can be simplified
K+
R
L+1
B.Bndd
<
3 + a
(40)
2
If 2:
K+
R
L+1
K+
L+1
+ a
2 S
3 + a
(41)
L
we have E[B!"] < E[Brnd]. Actually if L 1, feedfor- ward adaptation is the same as random adaptation, because the buffer only has one sample. Thus, feedforward adaptation has no greater error bound than random adaptation.
According Eq. 37
40 and
Expectation of the error bound for full adaptation. The full adaptation uses all samples from the buffer -Z.- 2x-1] to adapt the model. Considering > L, for an arbitrary sample 2, |1 = 5. because all samples in the buffer are independent
[žt_L, ... -1], we have [[]|2 -2, uniformly-randee variables. Considering the final performance I,, we derive the expectation for the error bound of full adaptation Bf as shown:
B[B! = (K + ROEE 7
Σ 8=1-L
1 - -alll + aB
1 T.
Σ
s|]
s=t-L L
1
-K (R)+ Σ
Bill2t zelll
+ +aE
L
s
= (K (K+K) KK)
Σ
Blll2 20ll +
L + 1
2
s
(-1
L++1 2
SK R)
Σ
L
3
s=t-1
L+1 2
< (K
(42)
As can be seen, the expected error bound of full adaptation is the same as the error bound of ran- dom adaptation. It is reasonable because full adaptation usually averages out individual differences between samples. Thus we have a similar conclusion as random adaptation, that feedforward adap- tation has greater enror bound than full adaptation. We have conclusion below.
16

---

Under review as a conference paper at ICLR 2023
Lemma 3. Given linear variaat function (may not be slowly varying function) Eq. (32 the expected error bound of the feedforward adaptation is smaller than the full adaptation and random adaptation, with the proper buffer size (e.g. L >> 2).
Therefore, in the following comparison, we only compare the feedforward and feedback adaptation.
When is feedforward better than feedback. Similar the worst-case analysis, derive the conditions that feedforward adaptation has a smaller error bound than feedback adaptation. To find the condition for E [ $ "[ solve the following inequality
K K+K
2(K K) K)
V 2a(K + K)
+a and
-121
(43)
Then we obtain a time condition on linear time-variam function for feedforward adaptation:
a a 4 V12 K R) x 0.089( K + R)
(44)
As summary, if time-varying factor of linear avariat function Eq. (32) is smaller, specifically S 1-y12 K + K), feedforward adaptation has smaller error bound than feedback adaptation. at 1- (K + K) a threshold for selecting whether feedforward or feedback. Thus, we proved the Lemma in section 3.4.
We can extend the above conclusion to a slightly larger set of functions. a can be time-dependent, but the supremum of it needs to be small. Consider the function fa(z)) = g(z)) h(t), and let
a, 2f. @h Similar the above analysis on linear time-variant function Eq. 32 we have of
conclusion below.
Lemma 4. Given a function fr, let a;
0h. Let K and K denote the Lipschitz constant for ground- 8t
truth ft and learnable function (e.g. neural networks) f(0). If sup(a:) < 1-yI + K), with the
proper buffer size, feedforward adaptation achieves smaller error bound than feedback adaptation.
Combing Lemma with Lemma 4, have conclusion about the effectiveness of the feedforward adaptation method.
Lemma 5. Given a function fe, let de = 0f: Let K and K denote the Lipschitz constant for ground-truth ft and learnable function (e.g. neural networks) f(0).
8t
(a) With buffer size > 2, feedforward adaptation always has smaller error bound than full adaptation and random adaptation.
(b) If sup(a:) 1-و (K + R), with the proper buffer size L adaptation achieves a smaller error bound than feedback adaptation.
2(K+K) 1, feedforward
C.2.2 PREDICTION ERROR OF ONLINE ADAPTATION ON TIME INVARIANT POLYNOMIAL
In this section, compare the eπror (not eπror bound) of feedforward and feedback adaptation methods on time-invariant polynomial systems (functions). Consider a time-invariant polynomial ground-truth function f(()) with the input of random variable sampled from the uniform distribution (4(0 1).
Zt 1(0,1)
(45)
N 9t-1 fe(zt) Σ
W
(46)
i=1
Assume our parameterized prediction model is a linear projection: Út f(Vi, (V,,,2)) 2x) V,2e
(47)
In online adaptation, we use the critical pair (1+)))+1) to optimize the linear prediction model at time step I. Then we have parameter of
V2 = arg min [[ 2, Notill
Us+1
Σ 1i=1
W
W
W1+12
(48)
Zg
Zg
==0
17

---

Under review as a conference paper at ICLR 2023
Then the absolute prediction enror (4 norm) of online adaptation is shown below:
N-1
6++1
1ue+1 f(V,, 2x)
194+1
Vi2i
W 1
Σ
W411 zcl
1=0
N-1
N-1
i-1
= |
Σ
Wit1 - = -1z1|-1
Σ
W1+1
2
2/ - 2x)
j=0
|
|zt
2z|
Σ
Wi+1
Σ
4
(49)
0=0
j=0
We consider the expectation of the prediction error. We have the following property in statistics:
B(2p) ++1
for ~ 1)),
(50)
For simplicity, assume W; > 0. The expectation of the prediction error is shown below.
B(ex-1) = -(2)) B[ 2,
W
(51)
t=U
Then we use Eq. (11) to derive the expectation of the error for feedback and feedforward adaptation methods. The core measurement of E[| 2n ||| and E[2--1-1 2g].
Expectation of the prediction error for feedback adaptation. In feedback adaptation, 2 the current sample 2, and the selected sample 2, are independent random variables from 24 1). We have [[[= 2=l) (Eq. (35)). Then we derive an expectation of the prediction error for feedback adaptation:
N-1
N-1
1
B(e
W
1+1
B[z
W
++1
Bi B1
B[=]]
2
i=0
1=0
1=0
1=
1
W
6
i-jj+1
1=0
N-1
W
Wet
1i+1)2
3
(i+
i=U
1=
1=0
N-1
W4+1
(52)
3 N2
As can be seen, the prediction error of feedback adaptation cannot converge. The expected prediction error E(eft) ) in each step is lower bounded by non-zero constant.
Expectation of the prediction error for feedforward adaptation. In feedforward adaptation, the selected samples 2, are the most similar samples to the current observation 2, We have E[[=, 2=() max( z-T) (Eq. (37). Then we derive expectation of the prediction error
18

---

Under review as a conference paper at ICLR 2023
for feedforward adaptation:
N-1
B(e(H1
max
W111
B
2
t
L
1=U
N-1
1
max
W
Blzt
1' I
max
Wi+
B
I.+1
N
max
Wi+1
2
L
N-1
2 max
W4+1
(53)
When we consider the final performance,
lim B(e(+1) S 1000000000
lim 1-00 max
Σ =0
(54)
1' I'++1
As can be seen, the expectation of the prediction error of feedforward compensation converges to zero with a large buffer size T.. Thus, we proved the Lemma in section 3.4
Expectation the prediction error for full (or random) adaptation. For full adaptation or random adaptation, we have E(|2; z=]) (Eq. (42)), which is the same as feedback adaptation.
Then we can see that the expectation of the prediction error for full (or random) adaptation the same as feedback adaptation:
N
1
B(e =
W4+1
3
1=0
j=0
1N -1
Σ
W++1
(55)
N2
As can be seen, the prediction error of full adaptation cannot converge. Thus, feedforward adaptation is provably better than full adaptation in the time-invariant polynomial system.
When is feedforward better than feedback. In the previous appendix C.2.1 we already know that feedforward adaptation has smaller error bound than feedback adaptation approximate time-invariant system (a R 0). this section, we furtherly prove that for the time-invariant poly- nomial system, feedforward adaptation has smaller expected prediction error (not error bound) than feedback adaptation, full adaptation, and random adaptation methods. As the result, feed- forward adaptation is better than feedback adaptation (or full adaptation, random adaptation) for time-invariant or slowly time-variant systems.
D ADDITIONAL. EXPERIMENTS
D. SYNTHETIC EXPERIMENTS: LINEAR TIME-VARYIE FUNCTION
To evaluate Lemma in section 3.4 consider the following time- varying function.
2, U(00 1)
(56)
Vt+1 fe(z)) = sin 2, + at
(57)
Our parameterized prediction model is one-layer perception with Sigmoid activation function.
Ú = f(V,, by; 2)) S(V.2)) + 1+e-V Vizi b
(58)
19

---

Under review as a conference paper at ICLR 2023
We have The Lipschitz constant K and K for the ground- truth function fi and the one-layer percep- tion
0f: 02,
K = sup
= sup (2))
(59)
0f
K sup(1
dz: ) = sup VV S(Vi2)) -1- 5(V)z)) 0.25 sup [V]
(60)
We use SGD as an optimizer in feedback and feedforward adaptation. During training, we keep the V&ll bounded, i.c. ||Vi]| < 1, then K = 1.4 We use Lemma 1 to calculate the error bound for feedback and feedforward adaptation:
5
+
(61)
12
5
L+1
E
EBB
+
(62)
4(L
+
2
In feedforward adaptation, when V
1, we get the minimum error bound:
2n
EIB!' 5 2 with
5
(63)
20
Then we calculate the threshold a², If a S a*, feedforward adaptation has smaller error bound.
V12
α 6
K
R) R 0.089(K R) Rz 0.11
(64)
5
L'
1
(65)
2x
The experimental evaluation of the above theoretical results is shown in section 4.1
D.2 SYNTHETIC EXPERIMENTS TIME-INVARIANT QUADRATIC FUNCTION
To evaluate Lemma in section 3.4 we consider the following quadratic function with random input.
~ ((,
(66)
V1+1 fe(z)) 23
(67)
Our parameterized prediction model is linear projection:
Út = f(V,, =x) = Ve=
(68)
Set the buffer size L 99. According to Lemma 2, the expected prediction error for feedback and feedforward adaptation are:
1N N-1
1
E Blefh
W
(69)
N2
12
N-1
B(eff) < 2 t' L'+11
max
W = max 7. L +
= max
(70)
2t 200
Fig. 3al shows the prediction error for feedback and feedforward adaptation. As can be seen, the emror curve for feedback adaptation is not converged. While the error curve for feedforward adap- tation converges by the trend of 2: The experimental results are consistent with Eq. 69) and 77). Fig. shows the prediction results for feedforward adaptation. The dashed blue region denotes the estimated uncertainty in section (3. In the figure, the ground truth is always within the estimated uncertainty, which validifies the proposed uncertainty estimation.
1.
20

---

Under review as a conference paper at ICLR 2023
0.7
Peedback feedforward 122
Ground Buult Prediction
0.6
1.0
W Man Ma t 1
M
M MLL
9.4
Taraet
0.6
Prediction
0.1
0.0
Stop
Time Stop
(b) Prediction feedforward adaptation Figure Error curve and prediction results on quadratic function.
(a) Error Curve
D.3 EXPERIMENTAL DESIGN FOR REAL-WORLD EXPERIMENTS
Dataset.
ET et al. (2021) dataset contains the data collected from electricity transformers, including
load and oil temperature that recorded every 15 minutes between July 2016 and July 2018. Which consists of two hourly-level datasets (ETTh) and two 15-minute-level datasets (ETTm). In our experiments, we used the first hourly-level dataset ETTh1 univariance prediction task.
Exchange-Rate Lai al. (2018) records the daily exchange rates of eight countries from 1990 to
2016.
ILI describes the ratio of patients with ILI and the total number of patients. Which includes the weekly recorded influenza-like illness (IL.) patients data from the Centers for Disease Control and Prevention of the United States between 2002 and 2021.
THOR Rudenko et al. (2020) is a public dataset of human motion trajectories, recorded controlled indoor experiment. Which includes the motion trajectories with diverse and accurate social human motion data in a shared indoor environment. In our experiments, we use No. agent's 's trajectory as a test set.
train set, No. 7 8 as validation set, and No. 9 ~ 10 agent's trajectory Zhou et al. (2021) and simple MLP (multi-layer perception).
Backbone models. We include two models as our learnable prediction function f(0,:): Informer
Informer Zhou et al. (2021) is a widely used transformer-basee time-series prediction model. Which extends the Transformer with KL-divergence based ProbSparse attention.
MLP is a simple but robust baseline for time-series prediction. Our MI. consists of 2 layers. The first layer can be considered Encoder Z, W Xx, After the encoder, MLP has layer normalization, activation function and final linear projection Y++1 = Relel(ayyeeeoom Z,)). The layer normalization and the final projection can be served decoder. Note that, did not flatten the input for MLP, the expression Z₁- = W X, a linear layer along the temporal axis.
Hyperparameters. For offline training. we follow the strategy in Zeng et al. (2022). In adaptation, set the learning rate of SGD as 1) 0.1 and set the EKF hyperparameters Buffer size for feedforward adaptation
po 0.1, X 1. = 1000. For ETTh1 dataset, we use / 192 step recent observations to predict future O 192 step output data. For the Exchange-Rate dataset, = 96 step recent observations to predict future O = 192 step output data. For the ILI dataset, use 36 step recent observations predict future O 36 step output data. For the THOR dataset, we use = 20 step recent observations to predict future O 20 step output data. For uncertainty estimation, we set = 0, K 1.
D.4 EXPERIMENTAL RESULTS ON EKF OPTIMIZERS
We conduct experiments on SGD and EKF as optimizers in adaptation. The descriptions of SGD and EKF are shown in appendix B In this section, report the detailed results of the experiment on the
http:////...c..gg.ggg/////////// 1asettaalasshhaaaa dashboard rdt htt.a/////a////////////////////n
21

---

Under review as a conference paper at ICLR 2023
EKF optimizer, shown in table Similar to results the SGD table feedforward adaptation still achieves the best results on all four datasets with EKF optimizer. Specifically, feedforward adaptation outperforms feedback adaptation by 5.9%, 15.1%, 24.1%, and 2.6% in terms of average MSE on four datasets respectively. The performance gain on the IIII dataset is the most obvious. It is reasonable because the incidence rate of common diseases in region is slow time-varying system, the future value is mostly dependent on the current input.
Table 2: Performance comparison between the proposed feedforward adaptation method and other baselines with EKF optimization. Avg denotes the average results of two models (MLP and former). The last row denotes the performance gain of feedforward adaptation over feedback adap- tation.
Method (Dataset
ETThi
Exchange
IIII
THOR
Adaptation
Model
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MLP
0.195
0.371
0.549
0.540
4.348
1.413
0.135
0.117
w/o
Informer
0.211
0.389
1.128
0.858
4,942
1.531
0.137
0.171
Adaptation
Avg
0.203
0.380
0.839
0.699
4.645
1.472
0.136
0.144
MLP
0.174
0.335
0.404
0.468
3.318
1.262
0.123
0.149
Random
Informer
0.191
0.352
0.847
0.684
3.573
1.328
0.277
0.313
Adaptation
Avg
0.182
0.343
0.625
0.576
3.445
1.295
0.200
0.231
MLP
0.129
0.288
0.356
0.459
3.873
1.341
0.107
0.106
Feedback
Informer
0.109
0.249
0.359
0.462
4.320
1.401
0.124
0.168
Adaptation
Avg
0.119
0.268
0.357
0,460
4.110
1.371
0.116
0.137
MLP
0.123
0.280
0.352
0,450
2.847
1.154
0.102
0.103
Feed Forward
Informer
0.102
0.248
0.254
0.327
3,395
1.244
0.124
0.167
Adaptation
Avg
0.112
0.264
0,303
0,388
3,121
1.199
0.113
0.135
Gain of Feedforward Over Feedback
5.9%
1.5%
15.1%
15.6%
24.1%
12.5%
2.6%
1.5%
D.5 STUDY OF THE SAMPLE SELECTION STRATEGY OF DIFFERENT ADAPTATION METHODS
We will show that the sample selection strategy in the proposed feedforward adaptation method could intrinsically extract the periodicity of the input data. The experiments on the ETTh1 dataset were shown in Fig. 2 In this section, we show the results of ILI, Exchange, THOR datasets. In our experiment, we set the buffer size L = 1000. We use SGD optimizer to adapt the MLP model with different adaptation methods.
(a)
50
(c)
Method
Count (percent)
40
Feodback Random
Output yt
10
40
60
80
100
120
140
20
40
60
100
Time step t
time shift of selected sample
0.8
(b)
(d)
Melhod
EFT intersity
0.8
Count (percent)
40
Feedfowwrd
30
04
20
10
20 40
60
80
100
20
40 60
100
Period
time shift of selected sample t-
Figure 4: Experiments on ILI data. (a) Time series output; (b) FFT period analysis; (c) Time shift (-s between current sample Z, and selected sample Z, in feedback and random adaptation; (d) Time shift s in feedforward adaptation; (e) Online prediction error. (f) Sample difference || Z. between current sample Z, and adapted sample Z..
ILI dataset. Figure a) shows the ILI time-series data (Ist dimension of the output), and Fig. the FFT (Fast Fourier Transformation) period analysis the ILI data. As can be seen ETTh has roughly T 50 repetition periods. Random adaptation randomly selects samples from the
22

---

Under review as a conference paper at ICLR 2023
buffer, then the time shift between current time-step and the time-step for selected samples is () ~ [1, L]; Feedback adaptation only selects the last time sample to optimization, then st = 1. This can be found from the distribution of time shift (t s) in Fig. 4 c). For the proposed feedforward adaptation in Fig. d), many samples were selected from (f 8) R 50 steps earlier, which corresponds the repetition period of T 50. Because feedforward adaptation selects the most similar samples to the current sample, which helps to extract the hidden periodicity of the input signal over time. Thus the distribution of is similar to FFT period analysis.
(a)
50
(c)
Method
Count (percent)
-40
Feedbark
Output yt
2
30
20
10
0
200
400
600
800
1000
1200
26
50
75
100
125
150
175
230
Time stop
time shift of ected sample
110
(b)
Method
Intersily
0.08
Count (parcent)
Faedlonmrrd
0.06
80
0.04
40
0.02
0.00
25
50
75
100
125
150
175
200
0
100 125
150
Pariod
time
sample
Figure 5: Experiments on Exchange data. (a) Time series output: (b) FFT period analysis; (c) Time shift 8 between current sample Zr and selected sample Zgg in feedback and random adaptation; (d) Time shift /. - s in feedforward adaptation; (e) Online prediction error. (f) Sample difference || - -2a|l between current sample Ze and adapted sample Z..
Exchange dataset. Figure and Fig. () is the FFT (Fast Fourier Transformation) period analysis of the Exchange data. As can be seen Exchange has no apparent periodicity. (The intensity of the FFT signal is too low, for example, the intensity of at the T Rs 90 is about 0.02). In this case, Feedforward adaptation in Fig. d), is likely select the recent samples like feedback adaptation most recent sample is a similar sample itself. Even for this kind of no- no-periodic system, feedforward adaptation still outperforms feedback adaptation, because it has more freedom to select samples. At least feedforward adaptation can select samples exactly the same as feedback adaptation.
shows the Exchange time-series data (6th dimension of the output),
Fig. ((c).. Because the
7.5
(c)
Method
Count (percent)
40
Feedbook
Outputyy yt.
2.5
30
iteescom
20
-2.5
10
-5.0
2000
4000
6000
8000
10000
200
1000
Time step t
time shift of selected sample t S
1.5
(b)
80
(d)
Wethod
FFT Intenssty
Count (percent)
1.0
40
0.5
20
250
500
750
1000
1250
1500
1750
2000
0
200
400 600 800
1000
Period
time shift of selected sample C C-
Figure 6: Experiments on THOR data. (a) Time series output; (b) FFT period analysis; (c) Time shift f - 8 between current sample Z, and selected sample Z, in feedback and random adaptation; (d) Time shift - s in feedforward adaptation; (c) Online prediction error. (f) Sample difference || Z.I| between current sample Z, and adapted sample Z..
THOR dataset. Figure a) shows the THOR time-series data (Ist dimension of the output), and Fig. 6 bb) is the FFT (Fast Fourier Transformation) period analysis of the THOR data. As can be seen
23

---

Under review as a conference paper at ICLR 2023
THOR has roughly T 1200 repetition periods. But the buffer size is L 1000 < T. In this case, we only store the recent 1000 samples in the buffer, but the period more larger'' 1200 > L, so the feedforward adaptation cannot extract the periodicity. In this case, Feedforward adaptation Fig. d). is likely to select the recent samples like feedback adaptation most recent sample is similar sample itself within the buffer. In the future, will investigate the more efficient sampling and buffer storing strategy to extract the very long time-dependency and periods.
Fig. [()). Because the
D.6 PREDICTION OUTPUT AND UNCERTAINTY
0.0
nuth Prediction
-0.5
Predictin Output
-10
Prodiction
-15
-2.0
-2
GroundTruth
Prediction
-2.6
500
1000
1500 Time stop
2000
2800
20
60
80
100
120
140
Time stop
Prediction on ETTh datasct
Prediction on ILI datasct
15
GroundTruth
GroundTruth Prodiction
Prediction
10
Predictio Output
Prediction
200 400
600
800
1000
1200
2000
4030
6000
8000
10000
Time step
Time step
(c) Prediction Exchange dataset
(d) Prediction THOR dataset
Figure 7: Prediction output and uncertainty estimation of the proposed feedforward adaptation on (a) ETTh1 dataset, (b) III dataset (Ist dimension of the output), (c) Exchange dataset (6th dimension of the output), and (d) THOR dataset (Ist dimension of the output). The blue dashed region represents the estimated uncertainty of the prediction.
One of the advantages of the proposed feedforward adaptation is it could provide uncertainty esti- mation as shown in algorithm 2 Figure 7 shows the prediction output (blue curve), ground truth label (red curve), and uncertainty estimation (blue dashed region) on four different datasets. As can be seen, in most cases, the estimated uncertainty covers the real ground truth value. Which validates the effectiveness of the proposed uncertainty estimation.
24


Reviews:
[{"review_id": "spJ_mi4Tpzz", "reviewer": "Reviewer_Qtjr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this study an online feed-forward adaptation approach is proposed to address distribution shift in  upcoming test data with respect to the training data. The core idea in feed-forward adaptation is finding a critical example from the memory, that is the most related to the current example, and adjust the model to predict the desired output of that critical example (in the past). This approach has been compared with the feedback adaptation approach in which the prediction model is updated according to the loss measured for the latest received ground truth. The authors also provided an analysis of the uncertainty of the prediction.\n", "summary_of_the_review": "In my opinion, this manuscript is not ready yet to be published at ICLR. However, it depends on how the authors address the raised issues.  ", "strength_and_weaknesses": "The studied topic is interesting and it indeed seems to be the most viable learning scenario in practice in many applications. \nThe contributions are clearly mentioned and the introduction motivates the reader to the topic. \nThe proposed approach is interesting and specially introducing an approach uncertainty quantification can be very useful. \n\nHowever, when we get to the experimental results, the experiments lack the claims made in the paper. \nI would expect to see some experiments related to the uncertainty measurement for the real datasets. \nAlso, for the real datasets, there is no experiment to study for each dataset, how much on average (or one single run), the critical selected examples, differ from the latest examples, and illustrate how this deviation relates to gaining in feed-forward in compare with feedback.  Also you could check for each dataset, how many times the critical examples, where actually the latest ones, or analyze the time invariance element empirically. \n\nIn Page 3, “Let f_t  denotes the ground-truth predictor … that generates the ground-truth input-output tuples ”, please make it clear that it is only predicting the output not the tuple (not a generative approach).\n \nIn table 1, you could use bold face and underlined for the best and second best results, to make it easier to read.\n\nThere are typos and ambiguous sentences in the manuscript, such as:\n - Page 3: “Online adaptation continually learning from streaming data, which is belong…”  wrong sentence, please rewrite it.\n- Page 4, the last sentence of the page is incomplete. “check the details in.”\n- Page 5, eq(5) $\\hat {K}$, is not defined.\n- Several times in the main manuscript you referred to an equation that was in in appendix, whereas you had that same equation nearby too. Examples: Page 5, your referred to Eq. 29, instead of Eq. 8.\nPage 6, Eq. 31 instead of Eq. 15. Page 7, Eq. 60, instead of Eq. 19 and Eq. 61 instead of Eq. 20. \n- In algorithm 2, line4, it should be s* in the subscript not s.\n- In Lemma 2, B and $\\hat {B}$ should be K and $\\hat {K}$.\n   \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BO0h1abov5v", "reviewer": "Reviewer_kCCH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the problem of online continual learning. The paper proposes a new algorithm which takes into account previously observed data samples instead of only the newly observed data sample.", "summary_of_the_review": "In summary, the paper studies an important and interesting problem in online continual learning and proposes a new method for adaptivity of models in dynamic environments. However, the paper does motivate the importance of the proposed method well and to show the effectiveness of the proposed algorithm more analytical discussions should be added to the paper.", "strength_and_weaknesses": "Strength: The idea of feedforward adaptation is interesting in online continual learning.\n\nWeaknesses:\n\n- The paper does not motivate the benefit of selecting the most similar data samples well. One simple solution is the learner adapt the model using all data samples in the buffer. Specifically, the paper does not consider the problem of online decision making and the learner only updates the model upon receiving new data sample without making decision for the newly observed data sample. In this case, the importance of adaptation is unclear.\n\n- The paper does not present any theoretical results in form of a theorem or lemma to present the error bound. It would be useful if the paper presents the error bound in the paper and compares the error bound of the proposed algorithm with that of the case where all samples in the buffer are employed to update the model.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4g8RAIfcjCw", "reviewer": "Reviewer_CSL7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of using online adaptation method in time-series prediction tasks. Existing approaches that feedback based on the latest prediction errors have the risk of forgetting past information. To address this challenge, the authors of this paper propose an approach that uses the critical data samples from the memory buffer to optimize the prediction model. The authors derive a bound of the prediction error, which gives the insight that the proposed approach will perform well if the ground-truth prediction functions are time-varying at a very slow rate. They also use experiments to show the proposed method performs better than existing approaches with and without adaptation.", "summary_of_the_review": "In summary, I feel the contributions made in this work is not significant enough for a conference publication. The theoretical contributions are not very meaningful, and the novelty of algorithm design is limited. I'm uncertain about how significant the empirical improvements on the real-world datasets are. Therefore, I would vote for reject with a low confidence score.", "strength_and_weaknesses": "Strength:\n\nThis paper studies an interesting question that is impactful for the fields of online learning and online control.  The intuition behind the proposed change to existing online adaptation methods is convincing, and some intuitions have been verified in the first synthetic experiment. Specifically, I like the plot in Figure 1 (a) that shows how the performance of the proposed approach depends on the time-varying shift of the function.\n\nWeakness:\n\nI feel the major weakness of this work is on the theoretical part. If I understand correctly, the error bound in equation (8) is just an upper bound of the prediction error $e_{t+1}$. I don’t think one can claim feedforward is better than feedback because the $E[B_e^{ff}] \\leq E[B_e^{fb}]$, because such claim can only be made when an upper bound is smaller than a lower bound. And it is also worth noticing that the proof only works for a specific special case. These two factors make me feel the theoretical guarantee is very weak.\n\nAnother weakness of this work is on the algorithm design. It has been shown both from the theory and the simulation that the proposed feedforward approach is prone to $\\delta$, which characterize how fast the ground-truth prediction function changes. In other words, one needs to decide whether to use the feedforward approach based on $\\delta$. However, the value of $\\delta$ is not known before the online prediction process starts. Even if one can switch algorithms in the middle based on the historical experience, it is unclear how to estimate $\\delta$ because the ground-truth prediction functions are unknown.\n\nFor the experiments, I’m uncertain about whether the improvement made on the real-world datasets are significant.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject