Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: In this paper, the authors present an online model adaptation approach. In contrast to many existing approaches where simply the “oldest” data will be discarded, the proposed approach works more in an feedforward fashion, i.e., it uses critical data samples from a memory buffer instead the latest sample. Furthermore, under some mild assumptions, error bound for the prediction are presented. A toy example and experiments with real world data sets show the superior of the proposed approach.
review: quality: It is a high quality paper and I enjoyed reading it

clarity: The paper is well written and the presented method is clear

originality: The idea seems to be original and, as far as I know, a new and interesting way to see the problem of online model adaptation

significance: As the general pipeline is not new but just with other ML models, the significants of this work might be limited

Strengths:
- The paper presents an interesting way to think about online model adaptation, i.e., from the perspective of feedback and feedforward control
- The proposed approach seems to be fairly simple but outperforms other standard methods

Weaknesses:

- I think that the paper could be strengthen by connecting the presented idea with other related work. For instance, there is a lot of work about information gain of data, i.e., which data is most representative or other works, e.g., “Robust Online Model Adaptation by Extended Kalman Filter”

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I really enjoyed reading the paper and the feedback/feedforward perspective on online model adaptation is interesting and worth to publish.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a continuous online learning method for trajectory prediction. For every input, before preforming inference, proposed method will look for the most similar data sample in a buffer and finetune the model with the sample. 
review: - Strength
  - Paper is well written and easy to follow.
  - The topic, online adaptation for neural network, is crucial
- Weakness
  - Need more comparison with existing work. In my understanding, proposed method can be categorized as continuous learning with replay buffer, which has abundant existing works to be compared with.
  - As stated in the limitation, the method relies on an assumption that there is not abrupt change in data distribution, which is not true. Imagine going pass a cross: trajectory distribution before the cross and after the cross is significantly different. Thus the performance of the proposed method is questionable. The benefit may solely come from the assumption that there is no abrupt change in the data distribution (which is not true as stated above), instead of really making the model learn.
  - Questionable dataset/benchmark selection. Selected dataset is **not** widely used for benchmarking trajectory prediction. For example, THOR is only cited by 43 other work, Assembly is cited by 21 other works, NGSIM is only cited by 4, robotic arm is collected by authors. Given that there are abundant datasets for trajectory prediction, this makes the evaluation results less reliable.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Given that paper:
- Lacks results against existing baseline methods, 
- Relies on not valid assumptions
- Has questionable evaluation results

a weak rejection is given.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: To address time-varying-drift for time series (or sequential) forecasting problems, the authors of this work present a method called “Online Adaptation with Feedforward Compensation.” This approach continuously updates a pre-trained model by preferentially sampling the states closest to the current state from an experience replay buffer. The authors provide theoretical analysis of this approach’s benefits and trade-offs relative to a naive sampling of the latest state’s experience buffer, which they term Feedback Adaptation. They further present a method to bound prediction uncertainty.

Empirical results are provided for a toy example, which is used to explore the benefits and trade-offs of the method based on the degree of non-stationarity and the experience replay buffer length. The paper presents results comparing variations in sampling of their approach to several robotic specific time series forecasting data sets along with a real world pick and place manipulation task.
review: The authors propose a method to address prediction/forecasting of (slow) time-varying, dynamics systems. The problem domain is one that is germane to robotics–especially practical applications thereof. The presentation is clear and the theoretical analysis appears sound. The derived uncertainty bounds are perhaps the most interesting and most verifiably novel aspect of the paper.  However, the core novelty and impact of the nearest state experience replay sampling and impact is unclear to the reviewer as the authors do not situate their results in the rich context of other continual learning methods or recent state-of-the-art (SOTA) for time series forecasting.

There is a rich literature focused on continual learning (e.g., refer to the recent survey in [1], especially section 4.2, which enumerates numerous experience replay approaches or [2], which is focused on experience replay). The impact of the paper would be greatly solidified if a theoretical or empirical comparison to other experience replay approaches were included.  

The value would be even further cemented if the proposed method was compared against SOTAfor times series forecasting (especially for the target of weakly non-stationary dynamic systems).  This includes a comparison to additional datasets that are more prevalent for time series prediction such as ETT [3] or at least a comparison of SOTA for the datasets considered in the papers.  As an example, the recent work in [4] achieved a RMSE of 0.46m @ 2s for NGSIM but under different settings: a 70% train and 10% validation and 20% test split and with a 3-second input producing a 5-second forecasting prediction. A direct comparison of this approach, or whichever approach has achieved SOTA for NGSIM, Thor, and the Assembly tasks would help the reviewer and reader assess the contribution of the proposed method.
 
The authors situate their work in the field “ online adaptation methods” (e.g., lines 3 and 93) but do not provide any explicit references to prior work in this area.

The analysis of when to use feedback vs feedforward both along with the error bound prediction \hat{K} is important for estimating the error bound.The analysis in the supplementary system is for a trivial network representing a simple problem. It would be instructive to include how determine this parameter (along with K and \delta) to make the proposed method more applicable.

[1] Wang, Liyuan, Xingxing Zhang, Hang Su, and Jun Zhu. "A comprehensive survey of continual learning: Theory, method and application." arXiv preprint arXiv:2302.00487 (2023).

[2] Bagus, Benedikt, and Alexander Gepperth. "An investigation of replay-based approaches for continual learning." In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-9. IEEE, 2021.

[4] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference, volume 35, pages 11106– 11115. AAAI Press, 2021
  
[3] Alinezhad Noghre, Ghazal, Vinit Katariya, Armin Danesh Pazho, Christopher Neff, and Hamed Tabkhi. "Pishgu: Universal Path Prediction Network Architecture for Real-time Cyber-physical Edge Systems." In Proceedings of the ACM/IEEE 14th International Conference on Cyber-Physical Systems (with CPS-IoT Week 2023), pp. 88-97. 2023.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The proposed method is clear and relevant to the field of robotics. While the paper provides theoretical analysis to ground the proposed approach compared to trivial alternatives, it lacks a thorough comparison to state of the art time series forecasting techniques and other experience replay sampling methods for continual learning. The reviewer is unable to fully assess the impact of the proposed method because of this lack of evidence. As it stands, the reviewer recommends the paper be rejected unless these shortcomings are addressed.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The ability to adapt prediction models online is key for a successful robot deployment in dynamic environments. Existing approaches rely on adaptation schemes where only the latest observation and prediction error are considered, thus making them susceptible to forgetting past, and likely, useful information. In this work, the authors tackle the "forgetting" problem by proposing a feedforward adaptation framework in which past, critical samples stored in a memory buffer are used to adapt the model and generate predictions about the future. The authors also show how the proposed framework provides uncertainty estimates and guarantees on performance when applied to slow time-varying systems. Evaluations and analyses on a toy example and 4 publicly available datasets are also included, showing that the proposed feedforward adaptation framework outperforms the chosen baselines.
review: The paper addresses an important question in the robotics community: how to efficiently adapt predictive models online based on observed data? The different experiments show that the proposed method has the potential to scale and be applied to different, complex domains. I also appreciate the inclusion of a clear criterion and procedure for applying the proposed framework. Overall, the paper is well-written and the chosen experiments do a good job of showcasing the performance and applicability of the method.

The main weaknesses of the paper relate to the positioning w.r.t the state-of-the-art (i.e., related work) and experimental results.


robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: In the current state, I recommend a “Weak Reject” for this paper. The paper tackles an interesting problem and proposes a novel framework for online model adaption. However, additional clarity and comparisons are required to fully understand the applicability and limitations of the proposed approach.
review_title: null

Meta-review: The paper proposes a method for online model adaptation using past experiences, adapting the model prior to predicting the next data point using this historic data, hence, the term of feedforward adaptation is introduced.
Overall, the reviewers agreed on the importance of the presented problem, the interesting approach, and the very good presentation. But they also agreed on the main weakness of the paper in its current form, which is the comparison to state-of-the-art methods – both in terms of theoretical embeddings as well as in the conducted experiments. During the rebuttal, the authors provided extensive new and requested evaluations and clarifications. The updated version is a great improvement, grounds the interesting and promising idea onto a more solid foundation, and is a valuable contribution to CoRL.

**Importantly though, the updated paper is too long and the authors need to shorten the paper to the required 8 pages without removing important and significant content!**
Decision: Accept (Poster)