Online Model Adaptation with Feedforward Compensation
Authors: ABULIKEMU ABUDUWEILI
Changliu Liu
Venue: CoRL 2023 Poster (2023)
Keywords: Online Adaptation
Optimization
Behavior prediction

Abstract: To cope with distribution shifts or non-stationarity in system dynamics, online adaptation algorithms have been introduced to update offline-learned prediction models in real-time.  Existing online adaptation methods focus on optimizing the prediction model by utilizing feedback from the latest prediction error. Unfortunately, this feedback-based approach is susceptible to forgetting past information. This work proposes an online adaptation method with feedforward compensation, which uses critical data samples from a memory buffer, instead of the latest samples, to optimize the prediction model. We prove that the proposed approach achieves a smaller error bound compared to previously utilized methods in slow time-varying systems.  We conducted experiments on several prediction tasks, which clearly illustrate the superiority of the proposed feedforward adaptation method. Furthermore, our feedforward adaptation technique is capable of estimating an uncertainty bound for predictions.

Online Model Adaptation with Feedforward Compensation
Abulikemu Abuduweili, Changliu Liu Robotics Institute, Carnegie Mellon University {abulikea, cliu6 Gandrrew cmu, edu
Abstract: To cope with distribution shifts or non-stationarity in system dynam- ics, online adaptation algorithms have been introduced to update offline-learned prediction models in real-time. Existing online adaptation methods focus on opti- mizing the prediction model by utilizing feedback from the latest prediction error. However, this feedback- tion. This work proposes an online adaptation method with feedforward com- pensation, which uses critical data samples from memory buffer, instead of the latest samples, to optimize the prediction model. We prove that the proposed ap- proach achieves a smaller error bound compared to previously utilized methods in slow time-varying systems. Furthermore, our feedforward adaptation technique is capable of estimating an uncertainty bound for predictions.
approach is susceptible to forgetting past informa-
Keywords: Online Adaptation, Optimization, Behavior Prediction
Introduction
Robots rely on prediction models of human behavior other agents plan safe and efficient mo- tions within their environment. The development learning of accurate and reliable behavior pre- diction models is crucial for the successful deployment of autonomous systems in environments involving humans or diverse agents [1, 2, 3]. However, creating a universal prediction model or algorithm capable of accurately predicting different agents in various scenarios is extremely chal- lenging. Recognizing this limitation, it becomes necessary and advantageous for robots to have the capability to adapt their prediction models online when a high-fidelity behavior prediction model is available [4, 5, 6, 7]. For instance, consider the prediction of human motion in human-robot col- laboration, due to variations in individual preferences and behavioral styles, prediction models that effectively capture one person's 's behavior may not be applicable to another individual. In such cases, online adaptation of the human motion prediction model becomes essential, allowing the robot to adapt its predictions by observing the specific behaviors of the individual [8, 9]. Similarly, in the context of autonomous driving, the prediction of surrounding road participants' behavior necessi- tates adaptation [10, 11]. Driving behavior is subject to change based on road conditions, weather, and various environmental factors. During the training phase, is impractical collect data encom- passing all potential scenarios. Consequently, the prediction model of the agent must adapt online while observing real-time driving behaviors.
In online adaptation, a prediction model receives sequential observations, and an online optimization algorithm (such as stochastic gradient descent, SGD) employed to update the model based on the prediction loss computed from the observed data [12]. The goal of online adaptation is improve prediction accuracy in subsequent rounds. Most existing online adaptation approaches are based on feedback compensation [13, 14, 15], analogous to feedback control. In feedback adaptation, the prediction model exclusively incorporates the most recent data received. Following the observation of a new sample, the online optimization algorithm updates the prediction model by calculating the prediction loss between the previous prediction and the latest ground truth. However, this feedback compensation approach is prone to disregarding past information, thus limiting its effectiveness.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

In this work, we propose feedforward compensation in online adaptation to maximize information extraction from existing data, especially those that are more important. In the proposed feedfor- ward adaptation, we enable recalling to compensate for potential shortsighted behaviors due to forgetting in general online adaptation [16]. To achieve balance between forgetting and recalling, we maintain a memory buffer by storing recent L-steps observations. When similar trajectories are observed, the feedforward adaptation method will pull similar samples from the memory buffer to enhance learning. For example, in human behavior prediction tasks, human subject may exhibit similar behavior patterns different days. Discovering such patterns becomes challenging using conventional online adaptation that solely relies on the most recent data, whereas our feedforward adaptation method with a memory buffer facilitates their identification. In addition, the proposed feedforward adaptation is capable of estimating uncertainty bound by comparing the current ob- servation and historical observations. The proposed approach shares similarity with replay-based continual learning [17], as both methods involve retaining critical past samples in buffer and sub- sequently replaying them enhance the learning process. However, many replay-basee continual learning techniques employ random replay mechanisms [18, 19] or focus on task class incremental problems [17, 20J. Furthermore, online adaptation emphasizes local performance, while continual learning emphasizes generality. Further discussion on this topic is available in Appendix G. Our main contributions are summarized below'.
We introduce a general online adaptation framework.
We propose a feedforward compensation method in online adaptation. The proposed feedforward method has a smaller error bound than feedback methods in slow time-varying systems.
We propose method for uncertainty- bound estimation, which is agnostic from optimizers. We conduct extensive experiments to show that the proposed feedforward adaptation is superior to conventional feedback adaptation.
2 Problem Overview
Behavior prediction Behavior prediction is sub-topic of time series prediction, which mainly fo- cuses on predicting future motions (e.g. trajectory) of human or other agents given the past and cur- rent observations [21, 22. 23]. Assume the transition function of agent's behavior is denoted as f. At time step f, the input to the function is X, = [ It-1+2; ... ,Iz), which denotes the stack of /-step recent observations (or states). The output of the function Y = [We+ 3/44, , which denotes the stack of O-step future predictions. The observations x, and y/: are vectors that may contain trajectories features, and 2₁ Yr for some cases (e.g. univariate prediction).
Y = (t,, X²).
For prediction, parameters 9r to estimate the ground-truth transition function f(t, X2). The analysis of the online adaptation in the paper is based the following two conditions about the local smoothness of transi- tion function f. Assume we maintain a L-size buffer to store recent L-steps historical observations.
parameterized model (e.g. Neural Networks) f(0r, X₂) with learnable
Assumption 1. K-Lipschitz continuity condition. At time step the function is locally K Lips- chitz continuious for the recent L steps input data, if the following holds Vs [t L, t - 1]:
Xt) f(t, K]JX, Xell.
where K is the bound (real number) for the change of the value of the function over input (observa- tion) space. Intuitively, a Lipschitz continuous function is limited in how dramatically the function value can change over input space. Similar to (2), we assume our parameterized function fe(0,,)) :) (e.g. Neural Networks) is locally Lipschitz continuous Vs E [t - L, 1]. with constant value K:
[|, X.) A(0,, X.)II < K| X.Il-
(3)
'The code of the paper is available at https://github. http://www..oomtitteleeettttteee wwm///neelliii--oooooo------ee contootllootttolll Feedforward_ Adapptttii..

---

Assumption 2. 8 time-varying condition. At time step t, a transition function f is 5 time-varying for the recent I. steps under the input Xs, if the following holds Va E [t I,, 1]:
Xx) f (x)s 8 s|.
(4)
where 8 E IR+ is the bound for the change rate on fixed input observation. It is equivalent the local K-Lipschitz continuity condition over time space i, instead of input observation space X. d time-varying condition is common for time-series tasks because they usually do not change abruptly.
Online adaptation Online adaptation also can be called adaptable prediction, since it makes an inference concurring with updating model parameters [7, 9]. In online adaptation, the estimate of the model parameter is updated iteratively when new data is received. Online adaptation explores local overfitting to minimize the expected prediction error e++1, at time step t:
Cerr(t)) min B[et+1] = min B|I|Y f(0, (,)
vb+ is the ground truth observation (to be received in the future) and Y++1 = f(0r, X,)) [ -,+ Di+o) is the predicted outcome from the learned model parameter 0,. The adaptation objective can be in any l norm.
where Ye+1 - X.) = 19++1
22, ,
Feedback adaptation As discussed above, the goal of the online adaptation is to minimize the expected prediction error in the future (5). Due to the lack of ground-truth value in the current steps, it is not feasible to directly minimize the prediction error. In feedback adaptation, the objective of minimizing the prediction error in the future is approximated by minimizing the fitting error in the past (t) ming, I f(e,, X1-1)ll + yo - 1) = ming, A f(,, ((,,,,111)) X,-1)ll, where 0 A < 1 is forgetting factor. Then the optimization is done recursively. At time step after receiving the current observations (Zz, Be), the current samples were used adjust the pa- rameters of the prediction model by an online optimizer (e.g. SGD). Then new prediction is made using the new optimized parameters, In the next time step, the estimate will be updated again given the new observation and the process repeats. In our main experiment and analysis, we utilize Stochastic Gradient Descent (SGD) as the optimizer. This selection is equivalent to setting À = in (t),,, resulting in the objective of optimizing the model using the latest observation. Additionally, we present results for the case where x I, utilizing the Extended Kalman Filter (EKF) as the optimizer, detailed in Appendix F.1.
Several prior works have aimed to enhance the robustness of feedback adaptation, with examples such as Abuduweili et al. [5], which employs multi-epoch training strategies to reuse hard samples, resulting in improved overall performance. However, it's important to note that feedback adaptation, in its current form, cannot fundamentally address the problem of forgetting. This limitation arises from the recent-based discounting in the objective function (1). To tackle this challenge, propose a Feedforward Adaptation strategy with different objective functions, enabling the recall of important or similar past experiences. The next section will delve into the methodology in detail.
3 Methodology
3.1
General Online Adaptation Framework
Feedback adaptation recursively utilizes the latest observations to adapt its models, which may re- sult in forgetting important past information. In order to enable recalling, we generalize the data compensation strategy of feedback adaptation (i.e. utilizing the latest observation) to samples from recent L-steps historical observations. The general online adaptation framework is shown in Al- gorithm 1. time step f, after receiving the current observations (æt, we sample the critical onput-output pairs (X,, 1s++) from recent L-steps historical observations. The critical pair will then be used to adjust the parameters of the model by online optimizer (e.g. SGD). As special case, the critical pairs are composed by the latest observations X, X,-1 in feedback adaptation.
3

---

Algorithm General Online Adaptation Framework (Adaptable Prediction)
Require: Initial predictor ;)) with parameters Do. Optimizer ;;);;;); buffer size L
Ensure: Sequence of predictions (f2+1)==1
1: for t 1. , do
2: Receive the ground truth observations Fz. Vt
3: Find the critical input-output pairs for adaptation (Xs Vo+1), with 1 LS <1
4: Adaptation: 0: = ( 1s+1, Us+1), where Ys+1 [ist1, Ub+0] f(01-1,
5: Prediction: Y,41 [91-1. :.. Bioo] = Xi)
6: end for
3.2 Feedforward Adaptation
We propose feedforward adaptation algorithms based on the feedforward compensation, which shares similarities with feedforward control strategies [24]. The proposed feedforward adapta- tion directly minimizes prediction eтror (bound) by compensating for historical data without relying on feedback mechanisms. The feedforward adaptation strategy works by estimating the error bound of the prediction model and selecting historical data that can compensate for the model to minimize this error bound. Specifically, when prediction model encounters sample resembling very early data, the feedforward adaptation method retrieves historically similar samples enhance the learn- ing process. By incorporating this mechanism, the model benefits from improved ability make predictions without disregarding past information, thereby achieving minimum error bound.
Feedforward adaptation aims to minimize the upper bound of the prediction error:
(t)(t) min Bound[er+1] min Bound[]|Y (.).,x,)]l):
That is equivalent to optimizing the worst-case scenarios [25]. Minimizing the error bound of the prediction model is critical to providing robust prediction and safe planning [26]. To establish sample selection strategy in feedforward compensation for minimizing the error bound, we estimate the error bound in Lemma 1.
Lemma 1. Error Bound of Online Adaptation. Under the K K-Lipschitz continuity condition and
5 tary-na condition (2) to (4), the (prior) prediction error e++1 of general online adaptation (Algorithm 1) has the following upper bound:
c4+1 SKI|X, Xxll 8/6 s) |Y f((0, X.)II (1 X.) f00,,,,)1 $ SKK + (K++R) X, X,I + 5|t - -|+ +|V++++++1 - ((0,,x.)]
(8)
The error bound estimation comprises three components. The first term accounts for the difference between the current sample X₁ and the selected critical sample X,; the second term considers the time difference between current step and the step associated with the selected critical samples; the third term represents the (posterior) fitting error of input-output tuple (X., Y++1). which tends to be small through appropriate fitting techniques. The proof can be found in Appendix A.
Directly minimizing the error bound (8) is impractical since do not exactly know the Lipschitz constant K and time-varying factor 8. However, in slow time-varying system, it is possible to simplify the aforementioned error bound.
Slow time-varying system is one whose transition function (or behavior) changes slowly over time, i... O. The slow time-varying system only assumes that the transition function f varys slowly (locally) within recent L-steps, not (globally) for every step. Slow time-varying systems in real-world applications because we can choose the adaptation step to be small or the adaptation rate to be fast compared to the changing rate of the dynamics. 2 We describe the detailed condition
common
In some cases, we can convert the non-slow-varyinng system into slow-varying system by differencing. more details are shown in Section 4.2.

---

for feedforward udaptation in Section 4.1. In the slow time-varye vimy-igrinng system, let = 0 in the error bound (8), the optimization objective for the error bound can he simplified as below:
min Bound/e441 = min K X) - X,ll || f(or.
(9)
The unknown parameter still makes it difficult to directly optimize the above objective. address the problem, we change the joint minimization over s and 6, to a bi-level optimization which first minimizes the first term of objective (9) over the sampling time step then minimizes the second term of the objective over parameter 8t. Thus, the simplified objective function becomes:
final min (Y++1 A(0,, x
(10)
8t
8.1. 8 = arg
min KE Lut-1]
||x
Xill
(11)
In summary, the proposed feedforward adaptation method operates as follows: selects the most similar samples the current observation create the critical pair -1). as determined by (11). Subsequently, it employs this critical pair to optimize the prediction model following the prin- ciples outlined in (I0). When connected with Algorithm I, this approach is essentially replacement for the operations in line 4 and line 5 within Algorithm 1, substituting (1 I) and (10) in their place. For comprehensive view of the feedforward adaptation algorithm, please refer to Appendix C.4.
3.3 Uncertainty Estimation
The error bound (7) provides uncertainty estimation of the prediction results. Here we use estimation of K. and ò approximate real K and in (7). We use confidence factor (0, 1] to decay the error bound. The uncertainty estimation Ü'41 for prediction Y++1 be computed as:
Ú.+1 =o (K, X.Il 5(t s) + ||Y f(0,. XI) (l X.)
(12)
The confidence factor a is a predefined hyperparameter, e-g. = 0.9 for 90% confidence of un- certainty estimation. 8 and K, are also predefined hyperparameters, in our experiments we set 8 10-4 K 1. In order to improve the accuracy of the uncertainty estimation, option to iteratively update K, based on the estimated uncertainty U, and the actual error et, such that if the previous uncertainty estimation is much larger than the real error then we shrink the K, value; and vice versa. The K, estimation rule are chosen to be:
have the
Ún 1X X.Il
K, = Rr-1 +
B 1010.11.
(13)
Analysis
4.1
Comparison between Feedforward and Feedback Adaptation
The main difference between feedforward adaptation and feedback adaptation lies in the critical sample selection strategy that used in the optimization. In feedback adaptation, the critical sample is composed by the latest observation X, = X,-1- In the proposed feedforward adaptation, the critical sample is the most similar sample to the current observation within recent L-steps X, arg minx, ||X, - X&|l- In the following amalysis, we assume there is no fitting error and mainly focus on the data selection strategy. The important findings regarding the expected error bounds for feedforward and feedback adaptations presented in Lemma 2. Proof of these results can be found in Appendix B.
Lemma 2. Expected Error Bound. Considering a transition function X), and parameterized model f(0, X) subject to the K K-Lipschitz continuity condition and time-varying condition (2) (4). Let E|IXX X--1]]== D represent the expected distance between consecutive samples, and Elminx, |X, X2I] = D* denote the expected minimum distance. We have the following results for error bound for feedback adaptation B.fb and error bound for feedforward adaptation B!!
5

---

Feetark
[b.1]
(b.3)
Feethoward 3-2)
1.5
64
Feettwwrt 3-5
Feeftowert 3-3)
Ta Target
1.2
Fedhowerd (:11)
Fauthwerd (-100)
Fauthowert 1200000)
(b.2)
(b.4)
40 600
40
Average error
(b) Prediction output
Figure 1: (a) Average error of feedback and feedforward adaptation with different buffer sizes L and vary-ng factors 5. (b) Prediction output of feedforward (blue curve) and feedback adaptation (green curve) with buffer size 100 and time- imr-ingy factor 5 10 and d = 10-
(a) Expectated error bound for feedback adaptation: E[B:A] = (K K)D+ +
(b) Expected error bound for feedforward adaptation: E| S (K R)D* + &L.
(c) If time-varying factor 5 is smaller, specifically K* < smaller expected error bound than feedback adaptation.
D-D
L-1 feedforward adaptation has
(d) If the inputs are random variables sampled from the uniform distribution, X₃ ~ U(0, 1), we have more concrete conclusions. (1) E E[B[[] K+K 6; (2) B[B!'] L+2
K+K ST. (3) If
8
< Tz, feedforward adaptation with buffer size I 2 oulperforms feedbuck adaptation.
K+K
The optimal buffer size for minimum error bound is L = KKR- -2
Numerical Example. We design a toy example to evaluate Lemma where we use a neural network learn the linear time-narying system yc+1 f(x)) sin 2, at where I; u( 1).
The results in Fig. la validate the theoretical results from Lemma 2(d). For smaller time-varying factor 6, feedforward adaptation with a larger buffer size achieves a smaller prediction error. How- ever, if is larger, the performance the feedforward adaptation deteriorates. If 5 < 0.1, feed- forward adaptation (buffer size L 2 2) outperforms feedback adaptation. Conversely, if > 0.1, feedback adaptation is better. The threshold 8* 1 (K K)= 0.1 aligns with Lemma 2(d).
Figure lb illustrate the comparison of prediction results between feedforward and feedback adap- tation. The first row shows the prediction results for 10-3 using feedforward adaptation with = 100 (b. 1) and feedback adaptation (b.3). As shown in the figure, feedforward adaptation results in better prediction. The second row shows the prediction results for 6 = 10 -2 The relatively poor performance observed in the feedforward approach depicted in Fig. lb.2 aligns with our theoretical analysis. Compared to feedback adaptation, the feedforward adaptation method excels learning the prediction model using critical samples, enabling it to effectively capture the input-dependent component, such as sin in the numerical examples. However, it struggles capture the time dependency, like at in the numerical examples. Fortunately, the linear time dependency can be effectively mitigated through differencing (as discussed in the next section).
4.2 Applications of Feedforward Adaptation
According to Lemma 2(c), feedforward adaptation outperforms feedback adaptation when 8 z 0, regardless of the values of 8, K, D D*. To aid in making this determination, we propose a straight- forward criterion based on time series stationarity analysis.
stationary time series is one whose properties do not depend on the time at which the series is observed [27]. For time series (X,, Y++1), stationarity implies that the transition function f Xx + Y-+1 does not have significant varying factor 8. To determine the stationarity of given time series signal, we employ the ADF test [28]. If the time series is found to be stationary, indicating

---

5 = O, feedforward adaptation can be used to achieve improved results. If the original time series is non-stationary, we can employ differencing to transform the series and assess stationarity again based on the differenced signal. Differencing calculates the difference between two consecutive observations [29], i.e. predicting d = Y - Y, instead of Y++1- It stabilizes the mean of a time series and thus reduces the trend. Differencing has the ability to convert many non-stationary series into stationary ones. Readers are referred to Appendix C for more explanations.
Experiments
Dataset. In this section, we conduct a comprehensive evaluation of the proposed feedforward adap- tation on both rohotics-related and real-world time-series benchmarks. The robotics-related scenar- ios include the following tasks. 1) Human Motion Prediction in Human- Rooot Collaboration: For this task, utilize the THOR dataset [30] to predict human motion and the Assembly dataset [31] to predict arm motion. 2) Vehicle Trajectory Prediction in Autonomous Driving: We employ the NGSIM dataset [32] to assess vehicle trajectory prediction performance. 3) Robotic Arm Trajectory Prediction for Quality Control and Monitoring: The task involving the prediction of robotic arm trajectories in bock--ad--laas scenarios. Moreover, we evaluate the feedforward adaptation on three well-established real-world time-series benchmarks: 1) Electricity Transformer Temperature (ETT) dataset [33]; 2) Exchange-Ree Rate dataset [34]; 3) Inftuenza-like -liee Illness (ILI) dataset.
Experimental Design. We employ a Multi-layer Perceptron (MLP) with direct multistep (DMS) prediction strategy [35] as our parameterized prediction model f(0,:). We first train prediction models on the train set. In the evaluation phase, we simulate real-world applications by incremen- tally receiving observations from the test set. At each time step, we perform online adaptation by optimizing the model using stochastic gradient descent (SGD) with selected previous observations. Subsequently, we obtain the prediction output from the updated model. The prediction results are evaluated using the root-mean-squee -quaand error (RMSE) metric. More details of the experimental de- sign can be found in Appendix D.
Baselines. We compare the proposed feedforward adaptation method with five baseline strategies. 1) w/o adapt performs prediction without any adaptation. 2) Feedback adaptation utilizes the latest observations to adapt its models [5]. 3) Experience Replay (ER) introduces random sample replay from this buffer to enhance learning [18] 4) Average Gradient Episodic Memory (A-GEM) aims to minimize loss on current data under the constraint of avoiding loss increase replayed data [19]. 5) Sequential Monte Carlo Dropout (SMCD) employs particle filter to sustain distribution over dropout masks, thereby dynamically adapting the neural network to changing environment [36].
Results. We conducted experiments across 10 distinct random seeds and present the mean results alongside their corresponding standard deviations. Results of the robotics-related datasets are shown in Table 1. Additionally, results for the general time-series prediction benchmarks are presented in Table 2. Evidently, the feedforward adaptation method consistently surpasses other baseline approaches across all datasets. This shows the effectiveness of the similarity- sample selection strategy in online adaptation.
Performance (RMSE) comparison between the proposed feedforward adaptation method and other basclines on Robotic-related datasets. We use boldface and underline for the best and second-best results.
Table 1:
Method Dataset
THOR (m)
Assembly (cm)
NGSIM (m)
Robot arm (rad)
w/o adapt
1.208 = 0.005
1.324 + 0.001
1.203 # 0.011
0.257 + 0.001
ER
0.914 = 0.002 0.002
1.188 + 0.001
0.985 + 0.015
0.210 + 0.001
A-GEM
0.873 + 0.002
1.194 : 0.001
0.935 + 0.012
0.216 + 0.001
SMCD
0.937 0.006
1,201 0.001
1.002 + 0.015
0.210 0.002
Feedback
0.891 = 0.002
1.191 + 0.001
0.963 4 0.012
0.214 + 0.001
Fecdforward
0.839 * 0.002
1.180 0.001
0.901 + 0.011
0.193 + 0.001

---

Table 2: Performance (RMSE) comparison between the proposed feedforward adaptation method and other baselines on general time-series benchmarks.
Method Dataset
ETTh1
Exchange
w/o adapt
0.485 + 0.011
0.783 = 0.006
2.195 = 0.009
ER
0.391 + 0.013
0.601 + 0.013
1.943 0.016
A-GEM
0.373 # 0.013
0.619 = 0.013
1.906 = 0.012
SMCD
0.413 0.015
0.673 0.008
2.051 0.010
Feedback
0.383 = 0.013
0.619 0.012
1,953 0.013
Feedforward
0.357 + 0.012
0.589 0.012
1.843 0.011
GroundTruth
2.00
wioo Adapt
Foodforwad Prodiction
Random
0.30
Feedback
Prediction Output
0.25
1.50
Feedforward
Prodiction Error
0.20
3.00
0.25
0.75
0.10
0.50
0.25
0.05
500
1000
1500
2000
2500
500
1000
1500
2000
2800
3000
Time
Time step
(a) Prediction and uncertainty
(b) Online prediction error
Figure 2: Experimental results Assembly dataset. (a) segment of the prediction output and uncertainty estimation of the proposed feedforward adaptation. For simplicity, only the prediction for the future 10th step in the first dimension is plotted. The blue dashed region represents the uncertainty. (b) Comparison of prediction errors between the feedforward adaptation and other baselines.
One advantage of the proposed feedforward adaptation is its capability provide uncertainty esti- mation, shown in (12) and (13). Figure 2a shows the prediction output (blue curve), ground truth label (red curve), and uncertainty estimation (blue dashed region) the Assembly dataset. It is evi- dent that the estimated uncertainty generally encompasses the actual ground truth value, confirming the effectiveness the proposed uncertainty estimation. Figure 2b showcases the real prediction error for different adaptation methods over time, with feedforward adaptation exhibiting the small- est error. Moreover, in the context of periodic data, feedforward compensation inherently possesses the capacity capture the underlying periodic patterns within the time series, as demonstrated in Appendix E.2. Furthermore, the proposed feedforward compensation strategy demonstrates the po- tential to generate additional improvements beyond the state-of-the-at optimization algorithms for online adaptation. This approach can be seamlessly integrated with any leading-edge prediction model, offering an avenue for further enhancements. More discussions are shown in Appendix F.
6 Conclusion, Limitation, and Future work
This paper investigates effective feedforward adaptation algorithm for behavior prediction tasks. We provide evidence that feedforward adaptation exhibits smaller error bound compared to con- ventional feedback adaptation in slow time-varying systems.
One of the limitations of the work is that the proposed feedforward adaptation method only works well on slow-varying systems. In the future, we plan to explore the integration of feedforward and feedback adaptation in more general systems, aiming for comprehensive approach and applications.

---

Acknowledgments
This research supported by the National Science Foundation (NSF) under Grant No. 2144489.
References
[1] Y. Cheng, L. Sun, C. Liu, and M. Tomizuka. Towards efficient human-robot collaboration with robust plan recognition and trajectory prediction. IEEE Robotics and Aulomation Letters, (2):2602-2609, 2020.
[2] Li, H. Ma, and M. Tomizuka. Conditional generative neural system for probabilistic trajec- tory prediction. In 2019 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 6150-6156. IEEE, 2019.
[3] R. Liu, Chen, A. Abuduweili, and C. Liu. Proactive human-robot co-assembly: Leveraging human intention prediction and robust safe control. In 2023 IEEE Conference on Control Technology and Applications (CCTA), pages 339-345, 2023.
[4] Pastor, Righetti, M. Kalakrishnan, and S. Schaal. Online movement adaptation based on previous sensor experiences. In 20/ IEEF/RSJ International Conference Intelligent Robots and Systems, pages 365-371. IEEE, 2011.
[5] A. Abuduweili and C. Liu. Robust nonlinear adaptation algorithms for multitask prediction networks. International Journal of Adaptive Control and Signal Processing, 35-3):314-341, 2021.
[6] L. Wang, Y. Hu, L. Sun, W. Zhan, M. Tomizuka, and Liu. Transferable and adaptable driving behavior prediction. arXiv preprint arXiv:2202.05140, 2022.
[7] A. Abuduweili, S. Li, and C. Liu. Adaptable human intention and trajectory prediction for human-robot collaboration. arXiv preprint arXiv: (o0::1999.00099 00099, 2019.
[8] Y. Cheng, W. Zhao, C. Liu, and M. Tomizuka. Human motion prediction using semi-adaptable neural networks. 2019 American Control Conference (ACC), pages 4884-4890, 2019. doi: 10.23919/ACC.2019 4980..
[9] A. Bajesy, A. Siththaranjan, J. Tomlin, and A. D. Dragan. Analyzing human models that adapt online. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 2754-2760. IEEE, 2021.
[10] W. Si, T. Wei, and C. Liu. Agen: Adaptable generative prediction networks for autonomous driving. In 2019 IEEE Intelligent Vehicles Symposium pages 281-286. IEEE, 2019.
[11] P. Kothari, D. Li, Y. Liu, and A. Alahi. Motion style transfer: Modular low-rank adaptation for deep motion forecasting. In Conference on Robot Learning, pages 774-784. PMLR, 2023.
[12] A. Blum. On-line algorithms in machine learning. Online algorithms: the state of the art, pages 306-325, 2005.
[13] A. Tonioni, F. Tosi, M. Poggi. S. Mattoccia, and L. D. Stefano. Real-time self-adaptive deep stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 195-204 2019.
[14] A. Abuduweili and C. Liu. Robust online model adaptation by extended kalman filter with exponential moving average and dynamic multi-epoch strategy. In Learning for Dynamics and Control, pages 65-74. PMLR, 2020.
[15] S. Bhasin, R. Kamalapurkar, H. T. Dinh, and W. E. Dixon. Robust identification-based state derivative estimation for nonlinear systems. IEEE Transactions Automatic Control, 58(1): 187-192, 2012.

---

[16] C. Paleologu, J. Benesty, and S. Ciochina. A robust variable forgetting factor recursive least- squares algorithm for system identification. IEEE Signal Processing Letters, 15:597-600, 2008.
[17] B. Bagus and A. Gepperth. An investigation of replaybbased approaches for continual learning. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1-9. IEEE, 2021.
[18] A. Chaudhry, M. Rohrbach, M. Elhoseiny, 'T. Ajanthan, P. K. Dokania, H. Torr, and M. Ran- zato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 10486, 2019.
[19] A. Chaudhry, M. Ranzato, M. Robrbach, and M. Elhoseiny. Efficient lifelong learning with a-gem. In International Conference on Learning Representations, 2018.
[20] Yin, A. Abuduweili, S. Zhao, L. Xu, C. Liu, and S. Scherer. Bioslam: A bioinspired lifelong memory system for general place recognition. IEEE Transactions on Robotics, pages 1-20, 2023.
[21] Butepage, M. J. Black, D. Kragic, and H. Kjellstrom. Deep representation learning for hu- man motion prediction and classification. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 6158-6166, 2017.
[22] S. Lefèvre, D. Vasquez, and C. Laugier. A survey on motion prediction and risk assessment for intelligent vehicles. ROBOMECH journal, 1(1):1-14, 2014.
[23] A. S. Weigend. Time series prediction: forecasting the future and understanding the past. Routledge, 2018.
[24] R. C. Dorf. Modern control system. 2008
[25] Roughgarden. Beyond the worst-case analysis of algorithms. Cambridge University Press,
2021.
[26] N. E. Du Toit and J. W. Burdick. Robot motion planning in dynamic, uncertain environments. IEEE Transactions on Robotics, 28(1):101-115, 2011.
[27] D. Kwiatkowski, P. C. Phillips, P. Schmidt, and Y. Shin. Testing the null hypothesis of station- arity against the alternative of a unit root: How sure are we that economic time series have a unit root? Journal of econometrics, 54(1-3):159-175 1992.
[28] D. A. Dickey and A. Fuller. Distribution of the estimators for autoregressive time series with unit root. Journal of the American statistical association, 74(366a):427 431, 1979.
[29] R. J. Hyndman and G. Athanasopoulos. Forecasting; principles and practice. OTexts, 2018.
[30] A. Rudenko, T. P. Kucner, C. S. Swaminathan, R. T. Chadalavada, K. O. Arras, and A. J. Lilienthal. Thör: Human-robot navigation data collection and accurate motion trajectories dataset. IEEE Robotics and Automation Letters, 5(2):676-682) 2020.
[31] R. Liu and C. Liu. Human motion prediction using adaptable recurrent neural networks and inverse kinematics. IEEE Control Systems Letters, 5(5):1651 2020.
[32] J. Colyar and Halkias. Us highway 101 dataset. Federal Highway Administration (FHWA), Tech. Rep. PHWA--RT-07-0000 2007.
[33] H. Zhou, S. Zhang, Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 11106-11115, 2021.
10

---

[34] G. Lai, W.-C. Chang, Y. Yang, and H. Liu. Modeling long-and and short-term temporal patterns with deep neural networks. In The st international ACM SIGIR conference on research & development in information retrieval, pages 95-104, 2018.
[35] A. Zeng, M. Chen, L. Zhang, and Q. Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205. 13504, 2022.
[36] Carreno, D. Kulic, and M. Burke. Adapting neural models with sequential monte carlo dropout. In Conference on Robot Learning, pages 1542-1552. PMLR, 2023.
[37] Z. Allen-Zhu, Y. Li, and Z. Song. A convergence theory for deep learning via over- parameterization. International Conference on Machine Learning, pages 242-252. PMLR, 2019.
[38] R. Pyke. Spacings. Journal of the Royal Statistical Society: Series B (Methodological), 27(3): 1965.
[39] G. A. Noghre, V. Katariya, A. D. Pazho, C. Neff, and H. Tabkhi. Pishgu: Universal path prediction architecture through graph isomorphism and attentive convolution. arXiv preprint arXiv:2210.08057, 2022.
[40] L. Wang. X. Zhang, H. Su, and Zhu. A comprehensive survey of continual learning: Theory, method and upplication. arXiv preprint arXiv:2302.00443 2023.
[41] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and representation learning. Proceedings of the IEEE conference on Computer Vision and Pat- tern Recognition, pages 2001-2010, 2017.
[42] C. F. Perez, F. P. Such, and Karaletsos. Efficient transfer learning and online adaptation with latent variable models for continuous control. arXiv preprint arXiv: 182.:398,, 2018.
[43] A. Nagabandi, C. Finn, and S. Levine. Deep online learning via meta-learning: Continual adaptation for model-based rl. arXiv preprint 2018.

---

A Lemma 1: Error Bound of Online Adaptation
In this section, we establish the error bound for prediction errors in the general online adaptation. At time step t, we select the critical input-output pairs (X,, M+++) from recent L-steps observa- These critical pairs are utilized to update the parameters the prediction model, resulting in
refined model. Subsequently, predictions are made using the newly optimized parameters.
Assuming that the transition function f X + Y4+1 satisfies the K-Lipschitz continuity condition and the vime-varying condition.
Bound of ground-truth difference. Given transition function f(t, X), if the K KLLisschitz conti- nuity and time-vare varying conditions holds within recent L steps, then the ground-truth value Y441 and 1 has the following property:
Y 1+1l = (t) - f(s, x.)Il s KI|X, x.1l + s|
(14)
The proof is shown below:
|Y41
Ys+1
[[f(t, - fss
- [t, X1) - f(t, + f(t,,,)) f(,
[t, - -((1,,,) (1](,
(triangle inequality) K||,, - -X.ll + (1((,,) - f(s, (K Lipschitzness)
$ X1 - x,ll + o s| (5 time varying)
(15)
Error Bound of Online Adaptation. For time step t, the (prior) prediction error et+1 has the following inequality:
e++1 ||Y++1 Yzt1ll = ||Y++1 f(00, X.)1
= ||Yr+1 - Yo+1 + Ys+1 - f(0,, X.) + X.) f(0,
< ||Y++1 - Ys+1|l |1 + ||Y f(0,, X.)Il + (1, X.) f(0,, X.)I| (triangle inequality)
S K - X1,1 + 6 + ||| f(0,, XI, IIf(0., X.)l
(16)
The first two terms come from the difference between ground-truth Y++1 Ys+1, the third term (posterior) fitting error for input-output tuple (X,, 1)+1), and the last term is the difference between two predictions. Combining the above inequality with the Lipschitz continuity condition for f(0,, Xt), we obtain the error bound for general online adaptation is shown below:
e111 S (K + K)]IX, X.Il + 8/6 s| + ||Y+1 f(0,, X.)I
(17)
Then lemma is derived.
B Comparison of Error Bound between Feedforward Adaptation and Feedback Adaptation
B.1 Lemma (a,b,e): Expected Error Bound
Considering the error bound (17), the first two terms are associated with the specific data compensa- tion strategy, while the last represents the posterior fitting error on the selected samples. In this study, our main focus is on the data compensation strategy, and we do not prioritize the data fitting aspect. A Additionally, with powerful neural network prediction model, achieving very small fitting error (almost zero) is relatively straightforward [37]. Therefore, we can disregard the fitting error when comparing feedforward and feedback adaptation. By neglecting the fitting error, we obtain an approximate upper bound Be for general online adaptation, as shown below:
B. = (K + K X, + 8|t s
(18)
Error Bound for Feedforward Adaptation. In feedforward adaptation, the selected input-output pairs are the most similar samples to the current observation X, arg minx, ||Xe -X,ll from
12

---

L-size buffer, and = arg (t [1,-1] |X, - XiIl- feedforward adaptation:
Then we have an enror bound / for
B (K K) - x, 8|t s (K R) X,1l +8
(19)
X, arg
min Xx | XiIl
(20)
Error Bound for Feedback Adaptation. In feedback adaptation, the selected input-output pairs the latest observations X, -, and -- 1. Then have error bound _ for feedforward adaptation:
B (K ,, x,- +
(21)
Comparison of the expected error bound between Feedforward and Feedback Adaptation. Let the expected distance between consecutive samples is D):
D B|||X; X1-1
(22)
Let the expected minimum sample distance is D*:
D := E X,lll Elmin ||x, Xill]-
(23)
Then the expected enror bound for feedforward adaptation is:
B[Be'] S (K EEIIXX XxIll |ll 8L (K K)D* + 8L.
(24)
The expected error bound for feedback adaptation is:
B[B (K K)][||X; X,- (K RDD
(25)
Consider the conditions that feedforward adaptation has a smaller error bound than feedback adap- tation in expectation. In order to make: E[B!'| < E[B[', we have:
(K + * + 8L (K +K)D+ +
(26)
D*
(27)
K+R T-1
Equation (27) represents the condition under which feedforward adaptation surpasses feedback adaptation in terms of the expected error bound. Here, the hyperparameter I. denotes the prede- fined buffer size. It is important to note that when L = 1, feedforward adaptation equivalent to feedback adaptation. Therefore, our focus is primarily on the case when L 1. From the equation, observe that if the system exhibits smaller time-varying property 8 compared the Lipschitz constant K, and smaller minimum sample distance D* feedforward adaptation is more likely to achieve greater improvement over feedback adaptation. For instance, when 8 0, we have E[B!"] < BiB for any K, K, D, D", and L.
By combining (24), (25) and (27), we can conclude Lemma 2 (a,b,c).
B.2 Lemma (d): Expected Error Bound on Random-input System
Consider transition function f with randomly sampled input observations. Specifically, input X, is random variable sampled from the uniform distribution: X, ~ L/(0. 1). In this case, the current sample X, and last sample X,-1 are independent random variables from ((0, 1). According to [38], the expectation of the distance between these two independent and miitorn-distributee variables is 3. Then for feedback adaptation
E| X, - X.-IIll 3' for X1, X,-1 ~ U(00
(28)
The term Elminx; [] X;ll] represents the expected minimum distance between the current sample X, and previous L samples in the buffer, which is Liz [38], according to [38]. Then for feedforward adaptation:
E X.Il = El
min
|x - X,1l
for Xt, X; ~ 4(0,1)
(29)
-
L+2
13

---

Let D 3. - bound for feedforward and feedback adaptation on the system with random input:
E+2 on the expected error bound (24) and (25), we obtain the expected error
K + K L+2
| - (K+ + K))D +
+ 8L
(30)
K K
E[B:"| - (K+ R)D +
(31)
3
Consider the conditions that feedforward adaptation has a smaller error bound than feedback adap- lation in expectation. In order to make: EIB!' E[B%', we have:
(K + K)D* + 5L < (K R)D +
(32)
K K+ K
+ +8L K K+K
+
(33)
T++
3
8
(34)
K+K 6
L = 1, the feedforward adaptation is equal to the feedback adaptation. For feedforward adapta- tion, we have L 1. Then we consider the buffer size as general settings, then conclude the conditions for applying feedforward adaptation:
0.083
(35)
K+ K
K
31
12
In this case, with the optimal buffer size L L = - K+K 2, feedforward adaptation achieves the smallest expected error bound:
E B = 2 5(K + R) 26
(36)
As can be seen, if 8 as O, feedforward adaptation could achieve the zero expected error bound with optimal buffer size L*, while feedback adaptation cannot converge to zero expected error bound.
Thus, given a prediction system f with a random input state, if K+K + with buffer size = 2, feedforward adaptation achieves the smaller expected error bound than feedback adaptation. this
case, the optimal buffer size for minimum error bound is L*
K+R 2.
By combining (30), (31), (35) and (36), one can conclude Lemma 2 (d).
B.3 Synthetic Experiments: Linear Time-varying System
We design a toy experiment to evaluate Lemma 2. We consider the following linear time-varying system
Vt+1 f(x)) sin + 5t, 24 (0,
Our parameterized prediction model is a one-layer perception with Sigmoid activation function.
Út = f(V., by; x.) S(Vaat)) + b, 1++eVes e-Ve + b.
(37)
Where S(.) denotes a Sigmoid activation function. We have The Lipschitz constant K and i for the ground-truth function f and the one-layer perception f:
0f
K sup
sup cos(x:)) =
(38)
0x,
0f sup 1/ S(V(z)) (1 (1- S(Vx2)) 0.25 sup [V]
K sup(1 Oz.
(39)
14

---

We use SGD as an optimizer in feedback and feedforward adaptation. During training, we keep the ||Vill ll bounded, i.e. |V[|| I 1, then K = 0.25. We use Lemma 3 (30) and (31) to calculate the error
bound for feedback and feedforward adaptation:
B[B!" 12 +
(40)
E[B/1] = 4L + 8
+ 8L
(41)
Then we calculate the threshold 6* (35). v 8 feedforward adaptation has a smaller error bound.
8* 1
(42)
K + K
12
+ 8
1 (K +
1
00 0.1
(43)
12
Thus, in the toy experiment, 8 < 0.1, feedforward adaptation has a smaller error bound. The experimental results are shown in Figure of the main paper.
C Applications of Feedforward Adaptation
When determining whether to apply feedforward adaptation to system or time-series function, Lemma 2(c) can serve as criterion. However, estimating the values of 6, K, D, D for the system is required. As a straightforward and conservative approach, if & 0, feedforward adaptation outperforms feedback adaptation for any 6, K, D, D To simplify this decision decision-making process, we propose a simple criterion based on the widely used stationarity test in time-series analysis.
C.1 Stationary time series and ADF test
A stationary time series is one that exhibits properties that do not depend on time. Therefore, a sta- tionary time series does not possess trends or seasonality, In the context of time series (Xe, Y++1), stationarity implies that the transition function f X, - Y4+1 is not explicitly linked to the time step In accordance with the ó time-varying condition, which equivalent to 0.
The Augmented Dickey-Fuller (ADF) test is widely used method for detecting the stationarity of a time series [28]. It tests the null hypothesis that a time series is non-stationary or time-dependent (i.e.. it has a unit root), while the alternative hypothesis suggests stationarity, indicating that it cannot be represented by a unit root. The ADF test yields a p-value that is used to assess the test. As the proposed feedforward algorithm does not necessitate an exceedingly strict stationary condition, we have set the threshold between stationary and non-stationary in our task to be 0.1. If the p-value is less than 0.1, we reject the null hypothesis and conclude that the series is stationary. Conversely, the p-value is greater than equal to 0.1, fail to reject the null hypothesis and conclude that the series is non-stationary.
C.2 Differencing
In many real-world scenarios, time series signals exhibit non-stationarity. Therefore, is crucial to transform these non-stationary signals into stationary ones in order apply feedforward adaptation effectively. One approach achieve this is by computing the differences between consecutive ob- servations, denoted as dr+1 Y Y,. This process is commonly referred to differencing [29]. Differencing helps stabilize the mean of a time series by eliminating changes in its level and remov- ing trends. By applying differencing, it becomes possible to convert many non non-stationary series into stationary ones, thereby facilitating the use of feedforward adaptation.
C.3 Criterion for applying feedforward adaptation
The criterion and procedure for applying feedforward adaptation presented in Figure 2 of the main paper. In this approach, given time series (Xe, Y--1), such as the training set of the prediction
15

---

task f: XY + Y(+1, we follow a specific process based on the stationarity of the series. If the ADF test indicates that the series is stationary, we directly apply feedforward adaptation to the original series. This involves prediction and adaptation on Y1+1 = ft, X2). If the series is found to be non- stationary, we employ differencing by calculating the difference between consecutive observations, denoted as d++1 Y Y.. We then assess the stationarity of the differenced signal d If it is determined to be stationary,
proceed with feedforward adaptation the difference series. This entails prediction and adaptation on d,+1, followed by converting it back to Y1+1 Y, + de+1 based on the value of Y. If the differenced signal remains non-stationary even after differencing, we resort to feedback adaptation for handling the non-stationary signal. The criterion and procedure for applying feedforward adaptation are shown in Fig. 3. Please note that the differencing operator can be applied up to K 2 times. In our experiments, we utilize at most one differencing operation, which is equivalent predicting velocity trajectory datasets.
Feodforward Adapt Yeus?
Feedforward Adape (of de12)
Yes
Vee = Xtt
(6 > D
staticoary?
No
Differencing: dia, = Y:- 7
Is starionary
(8 = 0
No
Foodluuck Adiapt
Test
ADF Tex
most times
Y/ - d, Figure 3: The criterion and procedure for applying feedforward adaptation.
C.4 Feedforward Adaptation Algorithms
Algorithm 2 Online Adaptation with Feedforward Compensation
Require: Initial predictor :0),;) with parameters 0g. Optimizer :,;, L-size buffer B
Ensure: Sequence of predictions (fe+1) and estimated uncertainty {U++1)tm
1: for t 1,2, T do
2: Receive the ground truth observation values 22, yr: Construct input X, = [x, [ "1 2.1
3: Find the critical (similar) input-output pairs
from buffer B by (11)
4: Adaptation by (10): 0, = 10s,-1,0s++1,11 41)
5: Prediction: Y1+1 = [bt+1, ... 0t+o| = f0,, Xt)
6: Uncertainty Út+1 Estimation by (12) and (13)
7: Add current data to buffer: B B.append(X) X yt)
8: if size (B) > L then
9: B keep more recent samples (B, L)
10: end if
11: end for
D Additional Details of Experimental Design
D.1 Dataset
We evaluate the effectiveness of the proposed feedforward adaptation method in three robotic-related scenarios: (1) Human motion prediction in human-robot collaboration, using the THOR dataset and Assembly dataset; (2) Vehicle trajectory prediction in autonomous driving, using the NGSIM dataset; and (3) Robotic arm trajectory prediction for quality control and monitoring purposes, using the Robot arm trajectory dataset. The specific tasks for each dataset are illustrated in Fig. 4.
The description of the robotic-related datasets is shown below.
THOR' is a public dataset of human motion trajectories, recorded in controlled indoor experi- ment [30]. Which includes the motion trajectories with diverse and accurate social human motion
htttp://thor.. oru.
16

---

Obstacle
Hommet 1
Helmet 2
Helmet
Homret4
Hefmet
Helmet Velodyne
Frame-15341
(a) THOR human motion prediction dataset
(b) Arm motion prediction in assembly tasks
KIHOVA
NGSIM vehicle trajectory prediction (d) Robot trajectory prediction in pick- dataset
and-place tasks
Figure 4: Illustration of tasks in different datasets. Figure is copied from the public website of the THOR dataset http::/thhtt oru. oru.se/: Figure (b) is copied from the website of the Assembly dataset https: htt///githbb thub. com/intelligent control-lab 1erll-llaaHumannAeeee Arsman_Assembly_eee (c) is copied from the public website of the NGSIM dataset https://datal transportation. gov/Automobiles wwt./eeeeeeeeeaaiiieee Simulation MOoo-NNGSM---- W-W-NNGIIII------ ProgramII880- I-80-aim-- 2577-gpny; Figure (d) shows KINOVA robot arm performing pick-and-ple -lacee tasks in our dataset.
Figure
data in a shared indoor environment. In our experiments, we use No. 2 ~ 4 agent's trajectory as train set and No. 5 ~ 10 agent's trajectory as test set.
Assembly dataset records arm motions assembly tasks. This dataset includes different assembly tasks. Each task requires the human to LEGO pieces to assemble object. In our experiments, we use task 2 as a train set and task 3 ~ 5 as test set.
NGSIM dataset: US 101 human driving data from Next Generation SIMulation dataset 5. The dataset contains highway driving trajectories captured by cameras mounted on top of surrounding buildings [32]. In experiment,
subset of the dataset which contains 100 trials of different agents. We use No. ~ 50 trial's trajectory as a train set and No. 50 100 trial's trajectory as a test set.
We collect the Robot arm trajectory dataset, which records the joint position (De- Mnvit-Hartenbergg parameters) of the KINOVA Gen 3 (7 DoF) robotic arm in pick-and-place tasks. This dataset includes 4 pick-and-place tasks for picking objects from different positions on workbench. In our experiments, we use task ~ 2 as a train set and task 2 ~ 4 as test set.
Data
17

---

In addition to the evaluation of four robotics-related datasets, we further assess the proposed feed- forward adaptation method using three real-world time-series benchmarks: ETT (Electricity Trans- former Temperature), Exchange- and ILI (Influenza-like Illness) dataset. These three datasets are extensively employed within the domain of time-series prediction research. The description of the time-series benchmarks datasets shown below.
ETT [33] dataset contains the data collected from electricity transformers, including load and oil temperature that recorded every 15 minutes between July 2016 and July 2018. Which consists of two hourly-level datasets (ETTh) and two 15-minute-level datasets (ETTm). In our experiments, we used the first hourly-level dataset ETTh1 as a univariance prediction task.
Exchanne-Rate [34] records the daily exchange rates of eight countries from 1990 2016.
ILI describes the ratio of patients with ILI and the total number of patients. Which includes the weekly recorded influenza-like illness (ILI) patients data from the Centers for Disease Control and Prevention of the United States between 2002 and 2021.
D.2 Stationarity test of datasets
As discussed in Appendix C, we use the ADF method to test the stationarity of the time-series data and check the slow-varying property of its transition function. If the p-value of the ADF test is less than 0.1, we can reject the null hypothesis and conclude that the time series is stationary. If the p-value of the ADF test is greater than 0.1, we cannot reject the null hypothesis and conclude that the time series is non-stationary.
Table 3: ADF test results for raw time-series and the difference signal on Thor, Assembly, NGSIM, Robot arm datasets.
Dataset
THOR
Assembly
NGSIM
Robot arm 0.09 (stationary)
P value on Raw Series
0.005 (stationary)
4e-3 (stationary)
0.34 (nonstationary)
P value on Difference
< 0.001
< 0.001
0 (stationary)
0.008
Table 4: ADF test results for ETTh1, Exchange-rate, and ILI datasets.
Dataset
ETThl
Exchange-rate
II.I
P value on Raw Scries
0.008 (stationary) ()
0.533 (non-stationary) 0.049 (stationary)
value on Difference
0.001
The outcomes of the Augmented Dickey-Fuller (ADF) test on the robotics-related datasets pre- sented in Table 3. It is evident that the original trajectories (raw signals) for the THOR, Assembly, and Robot Arm datasets exhibit stationarity. This suggests that the transition function for these datasets changes slowly over time, On the other hand, for the NGSIM dataset, the original trajectory non-stationary, but the difference signal (velocity) is stationary. In our experiments, we apply feedforward adaptation to the raw trajectory for the THOR, Assembly, and Robot Arm datasets, and we apply feedforward adaptation to velocity (difference signal) for the NGSIM dataset.
In the case of general time-series datasets, to maintain consistency with previous works [33] that directly predict raw time-series signals (without differencing), we employ feedforward adaptation on the original raw signals. For reference, the ADF test results of the general time-series datasets are depicted in Table 4.
D.3 Experimental design
Parameterized Prediction models. We utilize a Multi-layer Perceptron (MLP) with a direct mul- tistep (DMS) prediction strategy [35]. The choice of MLP with DMS is motivated by the superior performance of simple MLP over many larger -rasdformer-based models, as reported in [35]. Our
odc. http://hwaggggeeoeo//////////ttt
18

---

MLP architecture consists of two layers. The first layer can be considered as an Encoder, denoted as X, = W X,, Following the encoder, the MLP incorporates layer normalization, an activation The layer
function, and a final linear projection represented as Y+1 = V normalization and the final projection can be viewed as a decoder. It worth noting that we do not flatten the input for the MLP. The expression X, W X, represents linear layer applied along the temporal axis.
Baselines. We compare the proposed feedforward adaptation method with five baseline strategies.
w/o adapt performs prediction without any adaptation, serving as lower bound for all adaptation methods.
Feedback adaptation is widely used online adaptation strategy, which utilizes the latest obser- vations to adapt its models and minimize the fitting error for the last observations [5].
Experience Replay (ER) is a widely used replay-based continual learning approach [18]. ER maintains a compact memory buffer containing select old training samples. During each iteration, ER introduces random sample replay from this buffer to enhance learning.
Average Gradient Episodic Memory (A-GEM) integrates memory-relay and constrained optimiza- tion [19]. Similar to ER, A-GEM employs random data replay from the memory buffer. However, A-GEM's unique aspect is that its objective doesn't 't directly minimize loss on replayed samples. Instead, aims to minimize loss on current data under the constraint of avoiding loss increase on replayed data.
Sequential Monte Carlo Dropout (SMCD) is simple and effective approach to adapting neural models in response to changing settings [36]. SMCD treats learning network with dropout layer akin to learning an ensemble of prediction distribution. At the adaptation phase, SMCD em- ploys a particle filter to sustain a distribution over dropout masks, thereby dynamically adapting the neural model to evolving settings.
Hyperparameters. For offline training, we follow the strategy in [35]. In adaptation, we set the learning rate of SGD as 1/ 0.001. Buffer size for feedforward adaptation L 1000. For uncertainty estimation, we set K
Input and output horizon of prediction task. For the robotics-related datasets, the prediction model utilizes the most recent 1 second of observations to predict the trajectory for the next sec- onds. To ensure consistent sampling frequencies, we subsampled the THOR and Assembly datasets to 20Hz. For these datasets, we set the input horizon to 20 and the prediction horizon to 40. The NGSIM dataset has sampling frequency of 15Hz, so we adjusted the input horizon 15 and the prediction horizon to 30 accordingly. As for the Robot trajectory dataset, subsampled it to sampling frequency of 25Hz and set the input horizon to 25 and the prediction horizon to 50.
Adaptation and Evaluation. We first train prediction models on the train set. In the evaluation phase, we simulate real- world applications by incrementally receiving observations from the test set. At each time step, online adaptation used to optimize the normalization layers of the model using stochastic gradient descent (SGD) with selected previous observations. To leverage the pretrained feature extraction part of the model without changing the parameters of the entire model, we only update the parameters of the normalization layer. Subsequently, obtain the prediction output from the updated model. The prediction results are evaluated using the pot-mean-sqquaee error (RMSE) metric.
Concerning the real-world time-series benchmarks, we align our settings with those presented the LTSF-benchmark [35]. The ETThl and Exchange-rate datasets share an input horizon of I 96 and output horizon of O 192. Conversely, the ILI dataset entails input horizon of I 36 and output horizon of O 36.
19

---

10.0
7.3
Feodtorwad Prediction
Random
Feedback
Prediction Output
5.0
Feedforward
2.0
Prediction Error
0.0
1.5
1.0
-5.0
0.5
10.0g
200
400
600
800
1000
0.0
200
4000
bit0
800
1000
Time step
Time step
(a) Prediction and uncertainty
(b) Online prediction error
Figure 5: Experimental results THOR dataset.
E
Additional Experimental Results
E.1
Prediction output and Prediction Error
Figure 5a shows the prediction output (blue curve), ground truth label (red curve), and uncertainty estimation (blue dashed region) the THOR dataset. Figure 5b shows the real prediction error for different adaptation methods over time. Notably, feedforward adaptation exhibits the lowest prediction error among them.
650
GraundTrutt
w/o
Feedforwad Prediction
30
Random
600
Feedback
Prediction Output
Prediction Error th+
25
Feedforward
20
35
450
10
400
5
300
200
400
600
800
1000
200
400
800
1000
Time step
Time step
(a) Prediction and uncertainty
(b) Online prediction error
Figure 6: Experimental results on NGSIM dataset.
Figure 6a shows the prediction output (blue curve), ground truth label (red curve), and uncertainty estimation (blue dashed region) on the NGSIM dataset. Figure 6b shows the real prediction error for different adaptation methods over time. Notably, feedforward adaptation exhibits the lowest prediction error among them.
Figure 7a shows the prediction output (blue curve), ground truth label (red curve), and uncertainty estimation (blue dashed region) on the Robot arm dataset. Figure 7b shows the real prediction error different adaptation methods over time. Notably, feedforward adaptation exhibits the lowest prediction error among them.
Figure 8a shows the prediction output (blue curve), ground truth label (red curve), and uncertainty estimation (blue dashed region) on the Etth1 dataset. Figure 8b shows the real prediction error for different adaptation methods over time. Notably, feedforward adaptation exhibits the lowest prediction error among them.
E.2 Study of the sample selection strategy of different adaptation methods
Feedforward adaptation selects samples with the smallest sample difference minx, XX, X.). This selection strategy allows feedforward adaptation to inherently capture the periodicity in time-series
20

---

-0.4
0.35
wlo Adapt
-0.6
M
Prodiction
0.30
Feedback
Prediction Oudput
-0.8
Prediction Erar 6r
025
Feedforward
-12
0.15
0.10
-1.6
0.05
0.00
-2.00
400
800
1000
1200
1400
200
460
600
860
1000
1200
1400
Time step
Time step
(a) Prediction and uncertainty
(b) Online prediction error
Figure 7: Experimental results on Robot arm dataset.
0.0
GroundTruth
0.8
Predidion
-0.
0.5
Random
1.0
Prediction Error
0.4
Feadback
Foodforward
Prodictin
0.3
1.5
0.2
-20
0.1
0.0
-2.5
0
500
1000
1500
2003
2520
500
1000
Time step
1600
2000
2500
Time step
(a) Prediction and uncertainty
(b) Online prediction
Figure 8: Experimental results on ETThl dataset.
data when faced with periodic patterns. the case the robot arm dataset, as depicted in Figure 7a, we observe an approximate periodicity of T 420. This is evident from the FFT (Fast Fourier Transform) period analysis depicted in Figure 9a. Figure 9b, demonstrate how many samples were chosen from (t (- 420 steps earlier during the feedforward compensation process, aligning with the repetition period of T s 420. Feedforward adaptation's 's selection of the most similar samples to the current sample facilitates the extraction of hidden periodic patterns within the input signal over time. Consequently, the distribution of (f s) exhibits similarity to the FFT period analysis.
15.000 00.20
FFT intensity
Count (percent)
0.15
0.10
0.05-
0
200
400
600
800
1000
0
200
400
600
800
1060
Period T
time shift of selected sample c-s
(b) Time shift selected samples Figure 9: Experimental results Robot arm dataset. (a) FFT period analysis. (b) Timeshift between current sample X, and selected sample X, in feedforward adaptation.
(a) FFT period analysis
21

---

F Discussion
E1
Incorporating Feedforward Adaptation with SOTA optimization algorithms
The proposed feedforward compensation strategy exhibits the capability to yield further enhance- ments top of the SOTA optimization algorithms for online adaptation. Many prior works consider the information gain in feedback fashion. Here we specifically compare with MEKF [14]. The MEKF introduces an Extended Kalman Filter (EKF))based methodology for effectively leveraging samples to achieve reduced fitting errors. Nevertheless, this paper [14] still incorporates the feed- back compensation strategy. To maximize these advancements, we can incorporate the feedforward compensation approach with MEKF. Tables 5 and offers comparison between the original MEKF (feedback-baase and our refined version of MEEKF+Feddooows of feedforward adaptation distinctly enhances the performance of MEKF.
As be seen, the incorporation
Table 5: Performance (RMSE) comparison between the original MEKF approach and MEKF
feedforward adaptation approach on Robotic-related datasets.
Method Dataset
THOR (m)
Assembly (cm)
NGSIM (m)
Robot arm (rad)
MEKF 0.865 = 0,002
1.184 = 0.001
0.895 + 0.009
0.191 = 0.002
MEKF + Feedforward
0.831 0.002
1.179 0.001
0.866 + 0.009
0.179 0.002
Table 6: Performance (RMSE) comparison between the original MEKF approach and MEKF feedforward adaptation approach on general time-series benchmarks. Exchange
Method Dataset
Dataset
ETTh1
ILI
MEKF
0.359 = 0,008
0.595 = 0.011
1.848 = 0.018
MEKF + Feedforward
0.346 + 0.007
0.583 0.011
1.697 0.018
F.2 Incorporating Feedforward Adaptation with SOTA prediction models
The proposed feedforward adaptation approach can be readily applied any SOTA prediction model, providing further enhancements.As a demonstration, incorporated the feedforward adap- tation strategy into the Pishgu [39], which has shown promising results the NGSIM dataset. The results of this application are documented in Table 7.
Table 7: Performance comparison between the original Pishgu model and Pishgu feedforward adaptation approach on NGSIM dataset.
RMSE (m)
Is
2s
38
4s
5s
Pishgu
0.15
0.46
0.82
1.25
1.74
Pishgu Feedforward Adapt
0.13
0.39
0.69
1.03
1.41
F.3 Computational complexity
The computational complexity of the proposed approach involves two primary steps per iteration: sample selection and optimization using the selected samples. 1) Sample Selection: This step volves selecting samples based on similarity calculations between the current observation and all samples in the buffer. Assume the input sample X, R has the input horizon and the dimension of each horizon (or coordinates) is m. I, is the size of the buffer containing previous samples. The complexity of sample selection can be estimated as O mm² L). 2) Optimization: The computational complexity of the optimization step is influenced by factors such as the model size and the optimizer used. In our experiments, we utilized a relatively small MLP model with approximately 7.8K parameters. Given that the dimensionality of the trajectory dataset not as extensive that of images and considering our use of reasonably sized buffer (e.g., L 1000), the computational load remains manageable. To offer practical perspective, we present time usage data for various methods in Table 8, It's worth noting that the time required for sample selection
22

---

is generally less than that for optimization, further demonstrating the feasibility of our approach in terms of computational efficiency.
Table 8: Time usage (sample selection and optimization) per iteration of different methods (unit: ms) on THOR dataset.
Time (ms)
ER
A-GEM
SMCD
Feedback
Feedforward
Sampling
0.60
0.59
0.35
0.35
0.99
Optimization
2.42
4.16
2.92
2.36
2.38
Total
3.02
4.75
3.27
2.71
3.37
G Further Related Works
G.1 Related Works in Continual Learning
The proposed feedforward adaptation strategy maintains memory buffer by storing recent L-step observations. It then selects important samples (those with higher similarity to the current sam- ple in our implementation) from the buffer to enhance learning. This method shares similarities with replay-based continual learning, as both approaches involve retaining crucial past samples in a buffer and replaying them to enhance the learning process. However, numerous replay-based con- tinual learning methods, such as Experience Replay [18] and Average Gradient Episodic Memory (A-GEM) [19], rely random replays. On the contrary, replay-based continual learning meth- ods that employ importance sampling (as opposed to random replay) specifically designed for scenarios involving maskiincrementa or maas-innrremenal learning [17, 40, 20]. These methods are often developed within the context of classification tasks, which might not directly align with the task structure of our online adaptation and regression tasks. For instance, methods like iCaRI. [41] primarily operate on class-wise samples, and approaches such as Ring Buffer and k-Means based sampling [18] rely on class information for aggregating samples. Our proposed methods do not rely on class task information to sample important samples based on similarity. Furthermore, on- line adaptation places its emphasis the localized performance of multiple upcoming predictions, which underscores the significance of both recalling and forgetting. Conversely, many continual learning methods prioritize a more global and generalized prediction capability, which tends to em- phasize the importance of effective recalling.
G.2 Related Works in Adaptation of Control tasks
Many researchers investigate the online adaptation property of prediction models or policy models in control tasks, aiming generalize subtle variations of the environment. As an example. SMCT is simple and effective approach adapting neural models in response to changing environments [36]. While both our method and SMCT aim at adapting neural models to changing data distri- butions, there are notable differences in their approaches. SMCT primarily utilizes feedback data compensation strategies to adapt the dropout layer using particle filters, while our proposed method employs feedforward compensation strategies to extract more crucial samples. Furthermore, SMCT focuses on adapting to dropout layers using gradient-free methods, whereas our approach adapts arbitrary layers through gradient-based optimization. Latent variable models [42] are oriented towards control problems where predicted outputs impact the environment and subsequent obser- vations. Which is designed to collect many data points trajectories while interacting with the environment under specific latent variables or dynamics (ct), which is setup not applicable to our online adaptation of time-series prediction tasks. Meta-learning approach [43] in an online learning context employs strategies like SGD for parameter updates and feedback compensation for sample selection. In contrast, our approach employs feedforward compensation for sample selection and subsequently utilizes these selected samples for model updates.
23


Reviews:
[{"review_id": "isDSHKGvd2", "reviewer": "Reviewer_sMeX", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this paper, the authors present an online model adaptation approach. In contrast to many existing approaches where simply the “oldest” data will be discarded, the proposed approach works more in an feedforward fashion, i.e., it uses critical data samples from a memory buffer instead the latest sample. Furthermore, under some mild assumptions, error bound for the prediction are presented. A toy example and experiments with real world data sets show the superior of the proposed approach.", "summary_of_recommendation": "I really enjoyed reading the paper and the feedback/feedforward perspective on online model adaptation is interesting and worth to publish.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- What is the computational complexity of the proposed approach?\n- How does it related to the idea of having local models, for instance, local GP models where, depending on the current state, the closest model is activated and used for predictions. Instead of having a memory, can it be seen as some “local” models?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "quality: It is a high quality paper and I enjoyed reading it\n\nclarity: The paper is well written and the presented method is clear\n\noriginality: The idea seems to be original and, as far as I know, a new and interesting way to see the problem of online model adaptation\n\nsignificance: As the general pipeline is not new but just with other ML models, the significants of this work might be limited\n\nStrengths:\n- The paper presents an interesting way to think about online model adaptation, i.e., from the perspective of feedback and feedforward control\n- The proposed approach seems to be fairly simple but outperforms other standard methods\n\nWeaknesses:\n\n- I think that the paper could be strengthen by connecting the presented idea with other related work. For instance, there is a lot of work about information gain of data, i.e., which data is most representative or other works, e.g., “Robust Online Model Adaptation by Extended Kalman Filter”\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "dfraPveWvd", "reviewer": "Reviewer_GpzZ", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a continuous online learning method for trajectory prediction. For every input, before preforming inference, proposed method will look for the most similar data sample in a buffer and finetune the model with the sample. ", "summary_of_recommendation": "Given that paper:\n- Lacks results against existing baseline methods, \n- Relies on not valid assumptions\n- Has questionable evaluation results\n\na weak rejection is given.", "technical_quality": "Poor: Lack of required evidence to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- Why the proposed method is called \"feedforward\"? The proposed method is actually still using error back-propagation like \"feedbackward\".\n\n- What's the difference between the proposed method to existing works in the continuous learning area?\n\n- What's the performance on more complex tasks?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Strength\n  - Paper is well written and easy to follow.\n  - The topic, online adaptation for neural network, is crucial\n- Weakness\n  - Need more comparison with existing work. In my understanding, proposed method can be categorized as continuous learning with replay buffer, which has abundant existing works to be compared with.\n  - As stated in the limitation, the method relies on an assumption that there is not abrupt change in data distribution, which is not true. Imagine going pass a cross: trajectory distribution before the cross and after the cross is significantly different. Thus the performance of the proposed method is questionable. The benefit may solely come from the assumption that there is no abrupt change in the data distribution (which is not true as stated above), instead of really making the model learn.\n  - Questionable dataset/benchmark selection. Selected dataset is **not** widely used for benchmarking trajectory prediction. For example, THOR is only cited by 43 other work, Assembly is cited by 21 other works, NGSIM is only cited by 4, robotic arm is collected by authors. Given that there are abundant datasets for trajectory prediction, this makes the evaluation results less reliable.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "aKt-bJsYp6", "reviewer": "Reviewer_yQR8", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The ability to adapt prediction models online is key for a successful robot deployment in dynamic environments. Existing approaches rely on adaptation schemes where only the latest observation and prediction error are considered, thus making them susceptible to forgetting past, and likely, useful information. In this work, the authors tackle the \"forgetting\" problem by proposing a feedforward adaptation framework in which past, critical samples stored in a memory buffer are used to adapt the model and generate predictions about the future. The authors also show how the proposed framework provides uncertainty estimates and guarantees on performance when applied to slow time-varying systems. Evaluations and analyses on a toy example and 4 publicly available datasets are also included, showing that the proposed feedforward adaptation framework outperforms the chosen baselines.", "summary_of_recommendation": "In the current state, I recommend a “Weak Reject” for this paper. The paper tackles an interesting problem and proposes a novel framework for online model adaption. However, additional clarity and comparisons are required to fully understand the applicability and limitations of the proposed approach.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "In terms of the related work, I believe the paper requires a more thorough discussion and comparison with other online adaptation methods (e.g. [1]) and approaches in which past observations are taken into account during adaption via latent representations (e.g. [2]). \n\nIn terms of the experimental results, a more detailed description of the experimental setup and observed results is needed. For instance, it will be beneficial to have more details about: \n\n- the buffer size and update process, the number of critical samples required, the dimensionality of datasets/tasks, the meaning of the prediction output, observation and prediction lengths, and critical sample selection. Ablations that show how performance changes based on these choices should also be included\n- do metrics reported in Table 1 and the subsequent figures include multiple runs with different seeds or data splits? Although Table 1 shows that the feedforward framework performs better, including results from multiple results (mean + standard deviation) will help the reader to better gauge the overall performance of the proposed method.\n- in Figure 1(b.2), why does the feedforward approach result in what looks like a poor prediction output\n\nI also believe that additional comparisons with approaches that capture or use past observations for adaption ([2]) or meta-learning-based models ([3]) would provide insights into the applicability and benefits of the proposed framework.\n\n[1] Carreno, Pamela, Dana Kulic, and Michael Burke. \"Adapting Neural Models with Sequential Monte Carlo Dropout.\" Conference on Robot Learning. PMLR, 2023.\n\n[2] Perez, Christian F., Felipe Petroski Such, and Theofanis Karaletsos. \"Efficient transfer learning and online adaptation with latent variable models for continuous control.\" arXiv preprint arXiv:1812.03399 (2018).\n\n[3] Nagabandi, Anusha, Chelsea Finn, and Sergey Levine. \"Deep online learning via meta-learning: Continual adaptation for model-based rl.\" arXiv preprint arXiv:1812.07671 (2018).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses an important question in the robotics community: how to efficiently adapt predictive models online based on observed data? The different experiments show that the proposed method has the potential to scale and be applied to different, complex domains. I also appreciate the inclusion of a clear criterion and procedure for applying the proposed framework. Overall, the paper is well-written and the chosen experiments do a good job of showcasing the performance and applicability of the method.\n\nThe main weaknesses of the paper relate to the positioning w.r.t the state-of-the-art (i.e., related work) and experimental results.\n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "Q3wKNIddLBs", "reviewer": "Reviewer_yPZ4", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "To address time-varying-drift for time series (or sequential) forecasting problems, the authors of this work present a method called “Online Adaptation with Feedforward Compensation.” This approach continuously updates a pre-trained model by preferentially sampling the states closest to the current state from an experience replay buffer. The authors provide theoretical analysis of this approach’s benefits and trade-offs relative to a naive sampling of the latest state’s experience buffer, which they term Feedback Adaptation. They further present a method to bound prediction uncertainty.\n\nEmpirical results are provided for a toy example, which is used to explore the benefits and trade-offs of the method based on the degree of non-stationarity and the experience replay buffer length. The paper presents results comparing variations in sampling of their approach to several robotic specific time series forecasting data sets along with a real world pick and place manipulation task.", "summary_of_recommendation": "The proposed method is clear and relevant to the field of robotics. While the paper provides theoretical analysis to ground the proposed approach compared to trivial alternatives, it lacks a thorough comparison to state of the art time series forecasting techniques and other experience replay sampling methods for continual learning. The reviewer is unable to fully assess the impact of the proposed method because of this lack of evidence. As it stands, the reviewer recommends the paper be rejected unless these shortcomings are addressed.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "The authors situate their work in the field “ online adaptation methods” (e.g., lines 3 and 93) but do not provide any explicit references to prior work in this area. Could you add further citations for this area of research?\n\nThe analysis of when to use feedback vs feedforward both along with the error bound prediction \\hat{K} is important for estimating the error bound.The analysis in the supplementary system is for a trivial network representing a simple problem. It would be instructive to include how determine this parameter (along with K and \\delta) to make the proposed method more applicable. Would it be possible to provide such analysis?\n\nCould the authors provide additional experiments for ETT and/or compare SOTA results for the datasets they tested including NGSIM, Thor, and Assembly?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a method to address prediction/forecasting of (slow) time-varying, dynamics systems. The problem domain is one that is germane to robotics–especially practical applications thereof. The presentation is clear and the theoretical analysis appears sound. The derived uncertainty bounds are perhaps the most interesting and most verifiably novel aspect of the paper.  However, the core novelty and impact of the nearest state experience replay sampling and impact is unclear to the reviewer as the authors do not situate their results in the rich context of other continual learning methods or recent state-of-the-art (SOTA) for time series forecasting.\n\nThere is a rich literature focused on continual learning (e.g., refer to the recent survey in [1], especially section 4.2, which enumerates numerous experience replay approaches or [2], which is focused on experience replay). The impact of the paper would be greatly solidified if a theoretical or empirical comparison to other experience replay approaches were included.  \n\nThe value would be even further cemented if the proposed method was compared against SOTAfor times series forecasting (especially for the target of weakly non-stationary dynamic systems).  This includes a comparison to additional datasets that are more prevalent for time series prediction such as ETT [3] or at least a comparison of SOTA for the datasets considered in the papers.  As an example, the recent work in [4] achieved a RMSE of 0.46m @ 2s for NGSIM but under different settings: a 70% train and 10% validation and 20% test split and with a 3-second input producing a 5-second forecasting prediction. A direct comparison of this approach, or whichever approach has achieved SOTA for NGSIM, Thor, and the Assembly tasks would help the reviewer and reader assess the contribution of the proposed method.\n \nThe authors situate their work in the field “ online adaptation methods” (e.g., lines 3 and 93) but do not provide any explicit references to prior work in this area.\n\nThe analysis of when to use feedback vs feedforward both along with the error bound prediction \\hat{K} is important for estimating the error bound.The analysis in the supplementary system is for a trivial network representing a simple problem. It would be instructive to include how determine this parameter (along with K and \\delta) to make the proposed method more applicable.\n\n[1] Wang, Liyuan, Xingxing Zhang, Hang Su, and Jun Zhu. \"A comprehensive survey of continual learning: Theory, method and application.\" arXiv preprint arXiv:2302.00487 (2023).\n\n[2] Bagus, Benedikt, and Alexander Gepperth. \"An investigation of replay-based approaches for continual learning.\" In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-9. IEEE, 2021.\n\n[4] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference, volume 35, pages 11106– 11115. AAAI Press, 2021\n  \n[3] Alinezhad Noghre, Ghazal, Vinit Katariya, Armin Danesh Pazho, Christopher Neff, and Hamed Tabkhi. \"Pishgu: Universal Path Prediction Network Architecture for Real-time Cyber-physical Edge Systems.\" In Proceedings of the ACM/IEEE 14th International Conference on Cyber-Physical Systems (with CPS-IoT Week 2023), pp. 88-97. 2023.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for online model adaptation using past experiences, adapting the model prior to predicting the next data point using this historic data, hence, the term of feedforward adaptation is introduced.
Overall, the reviewers agreed on the importance of the presented problem, the interesting approach, and the very good presentation. But they also agreed on the main weakness of the paper in its current form, which is the comparison to state-of-the-art methods – both in terms of theoretical embeddings as well as in the conducted experiments. During the rebuttal, the authors provided extensive new and requested evaluations and clarifications. The updated version is a great improvement, grounds the interesting and promising idea onto a more solid foundation, and is a valuable contribution to CoRL.

**Importantly though, the updated paper is too long and the authors need to shorten the paper to the required 8 pages without removing important and significant content!**
Decision: Accept (Poster)