Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a deep learning model to predict the motion trajectory of agents from the pre-trained knowledge on agents of different type. The model design is based on the premise that motion style across different type of agents include common features which can be transferred to target domain. The proposed motion style adapter uses two trainable low-rank weight matrices: one inferring the style factors that need to be matched with the target domain and the second one performs the updates. During update, the pre-trained model is frozen and the low-rank matrices are trained with a low volume samples of the target domain. The second contribution include in decoupling the style adapter into scene-specific style and agent-specific style. The experiments are conducted on 3 different data-set. The results show that the proposed model out-performs with respect to error when compared to fine-tuning, partial model fine tuning and parallel adapter design.
strengths_and_weaknesses: The paper is clearly motivated in adapting pre-trained models to different domains. The language is easy to follow and the experimental results corroborates the claim. The experimental analysis also compares their method with other related work. 
Many of the details are provided in the supplementary for the paper. Without the supplementary the architecture is not detailed and needs clarity in places. The architecture is similar to one published paper which is not cited.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The updated recommendation is based on the following: The issues raised in the first review were addressed in the rebuttal. However, the additional explanations as suggested need to be incorporated for publication.
issues: - Please clearly state the differences from the architecture in reference 7 in the introduction.

- The authors should consider and cite the following works, especially the body of work on meta-learning:
[2] Finn, Chelsea, Pieter Abbeel, and Sergey Levine. "Model-agnostic meta-learning for fast adaptation of deep networks." International conference on machine learning. PMLR, 2017. : This work uses meta-learning to adapt to new task using small dataset.

[3] Gui, Liang-Yan, et al. "Few-shot human motion prediction via meta-learning." Proceedings of the European Conference on Computer Vision (ECCV). 2018.

- The function L($x_i$, $y_i$; $W_enc$, $W_dec$) in Equation (1) needs to be defined. Although it is intuitive in the community as Loss function, it needs to be stated for completeness of narrative as a published work. Also the authors need to state what mathematical functions it represents.

-The difference in performance between S and S+F and A and A+F in the experiments in Sec 4.4 needs to be stated in the discussion. This completes the explanation of the behavior of fusion encoder in presence and absence of two encoders.

- The size of the pre-trained data-set used in each experiment is required. Some of these detail is provided in the supplementary material. This is required to evaluate how low the volume is of target domain dataset. Also, how the performance is affected if the size of the pre-trained data-set increases or decreases and what the size of the target domain data set need to be to obtain a comparable or tolerable result. Additional experimental analysis showing the impact of the pre-trained data set size compared to the size of the target domain dataset will be insightful.

- In the experiments in Sec 4.4, it would generalize the results more if other combination of scene ids are used i.e., {1,2,3} for pre-training and 4 for adapting and others. 

- Some of the details which are provided in the supplementary materials need to be moved in the main paper. This include the system architecture and how the proposed encoders are added to the existing deep learning models- the division of layers, where the adapters are applied and is there any additional hyperparameters? These details are required to implement the work without the support of supplementary.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper explores the problem of domain adaptation in motion forecasting. In contrast to previous methods that directly fine-tune the whole encoder, it proposes a low-rank motion style adapter that only accounts for part of the model parameters. A modular adapter strategy that disentangles the features of scene context and motion history to facilitate the fine-grained choice of adaptation layers. Experiments are conducted on Stanford Done and Intersection Drone dataset, as well as transferred to Lyft Level 5 dataset. 
strengths_and_weaknesses: __Strengths__
- The problem and motivation make sense to me as the data-driven motion forecasting methods are not generalizable, and the adaptation component is useful during inference. 
- The technical solutions are reasonable.
- Experiments verify the effectiveness of the method. 
- The paper is easy to follow. 

__Weaknesses__
- Can the authors explain the intuition that “the style shifts across forecasting domains often reside in a low-dimensional space”? I understand the statements in Line 133 that the style shifts across forecasting domains usually only exist in a small portion of latent factors. Is this consistent with “low-dimensional space”?
- Any variations or analyses on the choices of MoSA? 
- Any analyses/visualization for the learned parameters/maps in MoSA? and the decoupled ones? 
- The ablation studies on the Modular Motion Style Adapters are a little bit confusing. I didn’t observe conclusions in the section. It seems that for different cases, the optimal choices are different. How did the authors implement the network while inference? What’s the performance without the modular design? 
- The experiments in this paper seem all transferred within the same dataset, such as pedestrian-cyclist/truck-car. However, I think this is only a small part of the applications. More comprehensive experiments shall include cross-dataset/cross-category/cross-scenes. 

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall I think this is a paper with reasonable motivations and solutions and deserve to be published. However, I have some concerns regarding its experiments. Hope the authors can answer these questions. 
issues: Please answer my questions on weaknesses.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Core idea of the paper: Introduce domain adaptation techniques to the field of motion forecasting. This is demonstrated in two ways. Agent adaptation or scene adaptation. In Agent adaptation experiments, authors trained the initial model on a set of scenes on a particular agent and adapted to a different agent type from the same set of scenes. In Scene adaptation, authors trained a particular agent type on a set of scenes and demonstrated adaptation of the same agent performance on a different set of scenes. Adaptation methods is a very practical and important area of study which is proliferating in the field of NLP and vision and not so much in the field of motion forecasting. So, the core idea of the paper is quite relevant and very impactful.
strengths_and_weaknesses: Weaknesses

1. From figure 1, it seems like MoSA block is nothing but two linear layers without any activation function?
2. Experiments Figure 5 suggests that MoSA block outperforms fine-tuning of the full model. It is very surprising to see this conclusion as I would expect that the MoSA block tuning should either be lower or at par in performance with the fine-tuning the full model. Any intuition as to why this is the case?
3. Similarly in Figure 7 and 8: It is again very surprising to see that as we fine-tune more parts of the architecture, the overall performance is not doing better but on the contrary it is performing worse. 


robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: Though the work on introducing adaptation to the field of motion forecasting is new, I find the experiments to be quite in adequate and conclusions to be in effective or even counter intuitive. This suggests that he authors should re do the expeirments more carefully or provide strong supporting material or evidence along with explanation as to why full training provides overall worse performance compared to fine-tuning just the MOSA layers.
issues: -
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper is interested in motion forecasting, and specifically the problem of adapting a motion forecasting model to some change in "style", which is either a scene change or change in agent behavior. The key insight is that in the low-data regime, finetuning the whole model or some parts may be ineffective, and its better to train a new, low rank matrix to predict a residual to the frozen models prediction. Experiments on some standard datasets suggest this is better than full or partial finetuning. Additionally the paper proposes to explores decomposing the encoder into a separate "scene" and "agent" part.
strengths_and_weaknesses: *Strengths*

- Overall it makes sense that motion forecasting models may need to be adapted to new scenarios, and low-data methods for adaptation are important. 
- The approach of training a residual model that is constrained to be low rank makes sense as a way to adapt the model while optimizing a small number of parameters (which is important in the low-data regime).
- The method overall is pretty straightforward and explained in a clear way.

*Weaknesses/Questions*

Method:
- Overall I think the method makes sense for the low-data regime, though I'm not sure how novel the idea of adaptation by training a residual over a frozen model is. For example in robot learning many papers learn residual policies over an expert for adaptation [1], and in general in the idea of training a new layer on top of a frozen model is commonplace in machine learning. At the same time, I think applying these ideas to a new problem like motion forecasting is worthwhile.
- The low rank constraint on the residual layer is an interesting addition. Are the authors aware of other works in the meta-learning/finetuning literature that apply a similar constraint? 
- I'm not entirely convinced by the claim of the decomposed encoder which separates style and movement components of motion prediction. As far I understand, nothing explicitly forces each head to focus on style/motion respectively correct? How can the model be guaranteed to decompose the input into those respective components?

Experiments:
- It would be worthwhile to include more ablations which highlight the benefits of the proposed method. For example, if it is indeed the case that the low-rank constraint on the residual weights is important, the paper should include ablations which remove and also vary the size of r, and analyze how it impacts performance. 
- Additionally, for the experiment which looks at the decomposed Y-net into the scene and motion components, I don't see a head-to-head comparison between including vs. not including the decomposition. The experiment instead shows the impact of applying MoSA to the different parts of the decomposed module, but I don't see how the conclusion "It is clear that modularization helps to boost the performance of MoSA." can be drawn from Figure 7 and 8. 

Fit:
- Overall, I think the focus of this paper is on effective adaptation of a pre-trained model using a residual layer. While it's applied to motion forecasting, the same MoSA method seems like it could be beneficial in general. So I wonder if this paper might be better evaluated at a learning/vision conference. This paper also does not apply the method on any real or simulated robotic control problem. 

[1] Johannik et al. Residual Reinforcement Learning for Robot Control. 2018. 
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall proposes an interesting approach for adapting motion forecasting models that seems to improve prediction quality, but more thorough experiments are needed to support all of the claims about the method, and ultimately the work is not applied on any real or simulated robotic control problem. 
issues: See Strengths and Weaknesses.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper introduces a few-shot adaptation architecture for motion style transfer. The architecture simple uses two learnable matrices  (A and B) that are inserted at each layer and the output (B^l * A^l * h^l) is simply added, coordinate wise, to the original output of each hidden layer of the original network (whose weights are held frozen). The proposed technique has far-fewer trainable parameters than fine tuning the original network, for example. The paper claims that these parameters capture the "style" of motion or the essence of the new "scene". An additional decomposition is explored in which the scene and agent information are modularized and separate weights are depending depending on the labeled data (agent, scene, or both have changed). Comparisons to competing methods on three datasets show that the proposed method is effective at obtaining superior performance on standard error metrics. 
strengths_and_weaknesses: Strengths:
-The proposed method is conceptually simple
-The proposed method appears to achieve superior results under apparently fair comparisons 

Weaknesses: 
-There are several unsubstantiated, intuitive claims in the paper. 
-The method apparently requires "human intervention".
-Missing comparison to methods that use "adaptive normalization layers"
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I recommend acceptance for two main reasons: 1-the proposed method seems to advance the state of the art on three datasets, 2-the proposed technique appears to be simple and fairly reproducible (*provided the mention of "human intervention" in the limitations section is is clarified). 
issues: -There are several unsubstantiated, intuitive claims in the paper. For example: lines 142-143 and 165-168. 
-Although I do not fully understand why, the method apparently requires "human intervention" (in the form of labels?) as outlined in 255-257. 
-The authors mention that a comparison to methods that use "adaptive normalization layers" (e.g. FiLM, or [29]) would be presented, but I did not find this comparison in the experiments.
review_title: null

Meta-review: metareview: Summary: This paper addresses domain adaptation for motion forecasting. Unlike earlier methods, it provides a low-rank motion style adaptor that only accounts for a portion of the model parameters. They introduce domain adaptability to motion prediction. There are two ways to demonstrate this. Agent/scene adaptation. In Agent adaptation experiments, authors trained a model on a series of scenes involving one agent type and then adapted it to a different agent type. In  scene adaptation  an agent type  is trained on a set of scenes and demonstrated adaptation to a different set. A further decomposition is investigated in which scene and agent information are modularized and distinct weights are based on labeled data (agent, scene, or both have changed). Comparisons to competing approaches on three datasets demonstrate that the suggested strategy achieves improved standard error metric performance. 

Strengths 
1.	Data-driven motion forecasting approaches cannot be easily generalized and the adaption component presented is useful for inference. 
2.	The experiments demonstrate the method's efficacy. 
3.	 Motion forecasting models as presented in this paper show that they need to be changed to new conditions, and that low-data adaption techniques are crucial. 
4.	It makes sense to train a residual model that is limited to have a low rank in order to modify the model while maximizing a small number of parameters which is important in the low-data regime). 
5.	The method presented are well-explained.

Weaknesses
1.	More explanation is needed to explain why style shifts across forecasting domains frequently reside in a low-dimensional space and if the definition presented is consistent with the concept of "low-dimensional space"? 
2.	More clarity is needed about the Modular Motion Style Adapters and what conclusions were made. 
3. The paper does not show the implementation of the network with inference and the performance result without the modular design. 
4.	The experiments in this paper seem all transferred within the same dataset, such as pedestrian-cyclist/truck-car.  More comprehensive experiments shall include cross-dataset/cross-category/cross-scenes.
5.	The MOSA block as presented reads like the two linear layers without any activation function. The contribution of this MOSA needs to be better explained. 
6.	the method makes sense for the low-data regime, It is not clear how novel the idea of adaptation by training a residual over a frozen model is. 
7.	The low rank constraint on the residual layer is an interesting addition. The authors should investigate similar work on meta-learning/finetuning that apply a similar constraint. 

Update: 
We thank the authors for making relevant revisions to the paper and responding to the concerns of the reviewers. 

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)