Motion Style Transfer: Modular Low-Rank Adaptation for Deep Motion Forecasting
Authors: Parth Kothari
Danya Li
Yuejiang Liu
Alexandre Alahi
Venue: CoRL 2022 Poster (2022)
Keywords: Motion Forecasting
Trajectory Prediction
Distribution Shifts
Transfer Learning

Abstract: Deep motion forecasting models have achieved great success when trained on a massive amount of data. Yet, they often perform poorly when training data is limited. To address this challenge, we propose a transfer learning approach for efficiently adapting pre-trained forecasting models to new domains, such as unseen agent types and scene contexts. Unlike the conventional fine-tuning approach that updates the whole encoder, our main idea is to reduce the amount of tunable parameters that can precisely account for the target domain-specific motion style. To this end, we introduce two components that exploit our prior knowledge of motion style shifts: (i) a low-rank motion style adapter that projects and adjusts the style features at a low-dimensional bottleneck; and (ii) a modular adapter strategy that disentangles the features of scene context and motion history to facilitate a fine-grained choice of adaptation layers. Through extensive experimentation, we show that our proposed adapter design, coined MoSA, outperforms prior methods on several forecasting benchmarks.

Motion Style Transfer: Modular Low-Rank Adaptation for Deep Motion Forecasting
Parth Kothari
Danya Li
Yuejiang Liu
Alexandre Alahi
École Polytechnique Fédérale de Lausanne (EPFL)
Deep motion forecasting models have achieved great success when trained on massive amount of data. Yet, they often perform poorly when training data is limited. To address this challenge, propose transfer learning approach for efficiently adapting pre-trained forecasting models to new domains, such as unseen agent types and scene contexts. Unlike the conventional fine-tuning ap- proach that updates the whole encoder, our main idea is reduce the amount of tunable parameters that can precisely account for the target domain-specifie motion style. To this end, we introduce two components that exploit our prior knowledge of motion style shifts: (i) low-rank motion style adapter that projects and adjusts the style features at a fow-dimensional bottleneck; and (ii) modular adapter strategy that disentangles the features of scene context and motion his- tory to facilitate a fine-grained choice of adaptation layers. Through extensive experimentation, we show that our proposed adapter design, coined MoSA, out- performs prior methods on several forecasting benchmarks. Code available at https://gitte /iiggtht.b...... www.coot/iaeeeeeeeee rrritteeel/oooiooooooooorrroooee style- ttansfer
Abstract:
Keywords: Motion Forecasting, Distribution Shifts, Transfer Learning
Introduction
Motion forecasting is an essential pillar for the successful deployment of autonomous systems in environments comprising various heterogeneous agents. It presents the challenges of modeling (i) universal etiquette (e.g., goal goal-directed behaviors, avoiding collisions) that govern general motion dynamics of all agents; and (ii) social norms (e.g., the minimum separation distance, preferred speed) that influence the navigation styles of different agents across different locations. Owing to the success of deep neural networks large-scale datasets, learning prediction models in data-driven manner has become a de-facto approach for motion forecasting and has shown impressive results [1, 2, 3, 4].
However, existing deep forecasting models suffer from inferior performance when they encounter novel scenarios [5, 6, 7, 8]. For instance, a network trained with large-scale data for pedestrian forecasting struggles to directly generalize to cyclists. Some recent methods propose to incorporate strong priors robust to the underlying distribution shifts [9, 10, 11]. Yet. these priors often make strong assumptions the distribution shifts, which may not hold in practice. This shortcoming motivates the following transfer learning paradigm: adapting a forecasting model pretrained on one domain with sufficient data to domains such unseen agent types and scene contexts as efficiently as possible.
One common transfer learning approach is fine-tuning a pretrained model on the data collected from target domain. However, directly updating the model is often sample inefficient, as it fails to exploit the inherent structure of the distributional shifts in the motion context. In the forecasting setup, the physical laws behind motion dynamics are generally invariant across geographical locations and agent types: all agents move towards their goal and avoid collisions. As result, the distribution shift can be largely attributed to the changes in the motion style, defined as the way an agent interacts with its surroundings. Given this decoupling of motion dynamics, it can be efficient for an adaptation algorithm to only account for the updates in the target motion style.
In this work, we efficiently adapt deep forecasting model from one motion style to another. We refer this task as motion style transfer. We retain the domain invaariat dynamics by freezing the
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

pre-trained network weights. To learn the underlying shifts in style during adaptation, we introduce motion style adapters (MoSA), which are new modules inserted in parallel to the encoder layers. The style shift learned by MoSA is injected into the frozen pre-trained model. We hypothesize that the style shifts across forecasting domains often reside in a low-dimensional space. To formulate this intuition, we design MoSA as a low-dimensional bottleneck, inspired by recent works in language [12, 13]. Specifically, MoSA comprises two trainable matrices with a low rank. The first matrix is responsible for extracting the style factors to be updated, while the second enforces the updates. MoSA learns the style updates by adding and updating less than 2% of the parameters in each layer.
In low-resource settings. it can be difficult for MoSA to distinguish the relevant encoder layers updates from the irrelevant ones, resulting in sub-optimal performance. To facilitate an informed choice of adaptation layers, we propose modularized adaptation strategy. Specifically, we consider forecasting architectures that disentangle the fine- mne-grained scene context and past agent motion using two independent low-level encoders. This design allows the flexible injection of MoSA to one encoder while leaving the other unchanged. Given the style transfer setup, our modular adaptation strategy yields substantial performance gains in the low-data regime.
We empirically demonstrate the efficiency of MoSA on the state-of-the-an model Y-Net [2] on the heterogenous SDD [14] and inD [15] datasets in various style transfer setups. Next, we highlight the potential of our modularized adaptation strategy on two setups: Agent Motion Style Transfer and Scene Style Transfer. Finally, to showcase the generalizability of MoSA in self-driving applications, we adapt a large-scale model trained on one part of the city to an unseen part, on the Level 5 Dataset [16]. Through extensive experimentation, we quantitatively and qualitatively show that given just 10-30 samples the new domain, MoSA improves the generalization error by 25% on SDD and inD. Moreover, our design outperforms standard fine-tuning techniques by 20% on the Level 5 dataset.
2 Related Work
Motion forecasting. Classical models described the interactions between various agents based on domain knowledge but often failed to model complex social interactions in crowds [17, 18, 19]. Following the success of Social LSTM [1], various data-driven forecasting models have been proposed to capture social interactions directly from observed data [3, 20, 21, 22, 233. These methods heavily rely on a large and diverse set of training data, which may not be readily available for novel agents and locations. In this work, we efficiently adapt a pretrained model to unseen target domains.
Distribution shifts. The primary challenge in adapting to new domains lies in tackling the underlying distributional shifts. In the motion context, negative data augmentation techniques have been applied in limited scope to reduce collisions [10] and off-road predictions [24] on new domains. Closely related to our work, Liu et al. [25] proposed to reuse the majority of pre-trained parameters for efficient adaptation. However, there exist key differences in the methodology:
(1) we does not require access to multiple training domains with varying styles in order to perform adaptation (2) we introduce low-rank adapters instead of finetuning existing parameters, to model domain shifts. Domain adaptation is another paradigm that allows a learning algorithm to observe unlabelled test samples. While this approach is effective for supervised visual tasks [26, 27, 28], it is not ideal for motion forecasting where the crucial challenge is sample efficiency as labels (future trajectories) are fairly easy to acquire. Therefore, propose perform transfer learning using limited data.
Transfer learning. The standard approach of fine-tuning the entire or part of the network [29, 30] has been shown to outperform feature-based transfer strategy [31, 32]. In the motion context, transfer learning given limited data often requires special architecture designs like external memory [33] and meta-learning objectives [34, 35] that require access to multiple training environments. Wang et al. [36] performed online adaptation across different scenarios for vehicle prediction domains. Recently, there has been a growing interest in developing parameter-efficient fine-tuning (PET) methods in both language and vision, as they yield a compact model [37, 12, 13] and show promising results in outperforming fine-tuning in low-resource settings [13, 38). Similar in spirit to PET methods, introduce additional parameters in our network that account for the updates in target style.
Motion Style. the popular work of Robicquet et al. [14] defined navigation style as the way different agents interact with their surroundings. introduced social sensitivity as two handcrafted descriptions of agent style and provided them as input to the social force model [17]. In this work, we model style

---

Training
1-1
Went
Wace
Pretrained
Wese geled
Adaptation
h
Mlse
Wise
Motion Style Adapter
A' infer varying style factoors update varying style factors
Warssa
Figure I: We present an efficient transfer learning technique that adapts a forecasting model trained with sufficient labeled data (e.g., pedestrians), novel domains exhibiting different motion styles (e.g., cyclists). We freeze the pretrained model and only tunc few additional parameters that aim to learn the underlying style shifts (left). We hypothesize that the style updates low-dimensional space. Therefore, we propose motion style adapters with a low-rank decomposition (r d), designed to infer and update the few style factors that vary in the target domain (right).
domains lie in
latent variable that is learned in data-driven manner. Furthermore, decouple motion style into scene-style components and agent-style components to favour efficient adaptation.
Method
Deep neural networks have shown remarkable performance in motion forecasting thanks to the availability of scrge-scale datasets. However, these models often struggle to generalize when unseen scenarios encountered in the real world due to underlying differences in motion style. In this section, we first formally introduce the problem setting motion style transfer. Subsequently, we introduce the design of our style adapter modules that help effectively tackle motion style transfer. Finally, we describe the application of style adapters to a modularized architecture order to perform style transfer in more efficient manner.
3.1 Motion Style Transfer
Motion style. Modelling agent motion behavior involves learning the social norms (e.g., minimum separation distance to others, preferred speed, valid areas of traversal) that dictate the motion of the agent its surroundings. These norms differ across agents well as locations. For instance, the preferred speed of pedestrians differs from that of cyclists; the separation distance between pedestrians in parks differs from that train stations. To describe these agent-specific (or scene- specific) elements that govern underlying motion behavior, we define the notion of "motion style". Motion style is the collective umbrella that models the social norms of agent given surroundings.
Problem statement. Consider the inductive transfer learning setting for deep motion forecasting across different motion styles. Specifically, are provided forecasting model trained on large quantities of data comprising particular set of style(s) and our goal is to adapt the model to the idiosyncrasies of target style as efficiently as possible. We denote the model input and ground-trulh future trajectory of an agent using and respectively. The input x; comprises the past trajectory of the agent, surrounding neighbors, and the surrounding context map. The context can take various forms like RGB images or rasterized maps. We assume that the data corresponding to an agent type generated by an underlying distribution (x; (::s) parameterized by s. the style of the agent. As mentioned earlier, the style is dictated by both the agent type and its surroundings.
Training. The forecasting model has encoder-decoder architecture (see Fig. 1) with weights W.... and Waee respectively. The training dataset, Ds of size N is given by U.es (2), Mt) (J.....N' where S is a collection of motion styles observed within the dataset. The model is trained to minimize a loss objective such as the negative log likelihood (NLL) [1, 3] loss or variety loss [39]:
N
Lirain Ds; W Wene; Waee)
C(x,, Mii W Waney Waec).
(I)

---

Adaptation. When a novel scenario with style s' (s' & S) is encountered, it leads to a distribution shift and the learned model often struggles directly generalize the corresponding dataset of size Ntarget. The common approach to tackling such shifts is to
D2 = (1', D.)eE(t................. fine-tune the entire or part of the pretrained model. Fine-tuning optimizes an objective similar to training, but on the new dataset:
Nurreet
Codapt (Der; : W Weney Wdec) Ntorget
Clal, y Wene: d.ec).
In this work, we aim to develop an adaptation strategy for efficient motion style transfer, i.e., cases where Ntrrget is small (Ntarget < N). Often, motion behaviors do change drastically across domains. Instead, most of the behavioral dynamics are governed by universal physical laws (e.g.. influence of inertia, forecasting model and introduce motion style adapters, termed MoSA, to capture the target motion style. As shown Fig. I, we adapt a pre-trained forecasting model by fine-tuning W M with following objective: Ladope (Dor WMOSA)
We therefore propose to freeze the weights of the pretrained
Naarget
Clayy v/s WMOSA)-
(3)
Nearget
3.2 Motion Style Adapters
Our main intuition is that the style shifts across forecasting domains are usually localized - they are often due to the changes in only few variables of the underlying motion generation process. Therefore. during style transfer, we only need to adapt the distribution of this small portion of latent factors. while keeping the rest of the factors constant. These updates would correspond the changes in motion style (s s') in the target domain, as the general principles of motion dynamics (e.g., avoid collisions, move towards goal) remain the same across domains for all agents. We design motion style adapters, referred to as MoSA, to carry out these updates.
Our proposed MoSA design comprises a small number of extra parameters added to the model during adaptation (see Fig. 1). Each module comprises two trainable weight matrices of low rank, denoted by A and B. The first matrix A is responsible for inferring the style factors that required to be updated to match the target style, while the second matrix B performs the desired update. The low rank realizes our intuition that style updates reside in low-dimensional space, by restricting the number of style factors that get updated (r < d where d is the dimension size of an encoder layer). Therefore, during adaptation, the weight updates of the encoder are constrained with our low-rank decomposition W MoSAA = BAA The pretrained model is frozen and only A and B are trained.
For brevity, let consider the adaptation of encoder layer with input h' and output + As shown in Fig. 1, W ene and WhrSS are multiplied with the same input h², and their respective output vectors summed coordinate-wist shown below:
(Train) 1+1 = W'nch'
(4)
(Adapt) 11+1 w wi'ch' + W Wh Wi Wh + A'A.
(5)
has been shown that initialization plays a crucial role in parameter-efficient transfer learning [37, 12]. Therefore, following common practices, matrices A and B are initialized with near-zero function [12], so that the original network unaffected when training starts. Furthermore, the initialization provides flexibility to these modules to ignore certain layers during motion style updates. Despite this flcxibility, the total number of extra parameters is significant and can be inconducive to efficient style transfer. Therefore, to further boost sample efficiency, we present modular adaptation strategy which we describe next.
3.3 Decoupling Motion Style Adapters
Motion style can be decoupled into scene-specific style and agent-specific style. Scene-specific style dictates the changes motion due to physical scene structures. For instance, cyclists prefer to stay on the lanes while pedestrians move along sidewalks. The agent-specific style captures the underlying

---

Scene Adaptation
Scene
Fusion
Agent Adaptation
Modularized encoder
Figure 2: Our modular style transfer strategy updates only subset of the encoder to account for underlying style shifts. For instance, we adapt the scene encoder only to model scene style shifts (top right). While for the underlying agent motion shift, we only update the agent motion encoder (bottom right). This strategy boosts performance in low-resource settings.
navigation preferences of different agents like distance to others and preferred speed. Modularizing the encoder into two parts that account for the physical scene and agent's s past motion independently help decouple the functionality of our motion style adapters and further improve performance.
Consider the modularized motion encoder designs shown in Fig. 2. The modularized encoder models the input scene and agent's past history independently. The fusion encoder then fuses the two representations together. This design has the advantage to decouple the task of the style adapters into scene-specifie updates and agent-specific updates. Given the modularized setup, the nature of the underlying distribution shifts can help guide which modules within the model are required to be updated to the target style. As demonstrated in Section. 4.4, given different categories of style transfer setups, decoupling style adapters can improve the adaptation performance while significantly reducing the number of updated parameters.
4 Experiments
evaluate our method on a total three datasets to study the performance of motion style adapters: Stanford Drone Dataset (SDD) [14], the Intersection Drone Dataset (InD) [15], and Level 5 Dataset (L5) [16]. We evaluate each method over five experiments with different random seeds. More implementation details and ablations summarized in the Appendix.
Baselines. We use the otate-of-the-arr Y-Net model [40] on SDD and inD) and the Vision Transformer (ViT) [41] on L,5 across all methods. We compare the following:
Full Model Finetuning (FT) [29]: we update the weights of the entire model.
Partial Model Finetuning (ET) [25]: we update the weights of the Y-Net encoder SDD and inD and the last two layers of VIT for Level 5.
Parallel Adapters (PA) [42]: we insert convolutional layer with fixed filter size in parallel to each encoder layer and update the weights these added layers. This baseline does not incorporate the low-rank constraint.
Adaptive Layer Normalization [43, 44]: we update the weights and biases of the layer normalization. Motion Style Adapters (MoSA) [Ours]: we insert our motion style adapters in parallel each encoder layer in SDD and inD, and in parallel to query and value matrices of multi-headed attention in Level During modularized adaptation, we add our modules only across the specified encoders.
Metrics. We use the established Average Displacement Error (ADE) and Final Displacement Error (FDE) metries for measuring the performance model predictions. ADE calculated as the L error between the predicted future and the ground truth averaged over the entire trajectory while FDE is the error between the predicted future and ground truth for the final predicted point [1]. For multiple predictions, the final error is reported as the min error over all predictions [39]. Additionally, we define the generalization error as the error of pretrained model on the target domain. The more the dissimilarity between the source domain and target domain, the higher the generalization error.
5

---

Table I: Evaluation of adaptation methods for motion style transfer (pedestrians eyclists) SDD and scene style transfer on InD using few samples N Ntarget = (10, 20, 30). Error reported is Top-20 FDE in pixels. The generalization error on SDD 58 pixels and on inD is 33 pixels. Our proposed motion style adapters (MoSA) outperform competitive baselines and improve upon the generalization error by 25% in both setups. Mean and standard deviation were calculated over runs.
Stanford Drone Dataset
Intersection Drone Dataset
Nuaget
10
20
30
10
20
30
FT
57.28 + 1.21
52.61
+: 0.87
46.31 +: 1.79
27.92 + 1.99
25.15 + 1.08
23.18 +: 0.64
ET [25]
51.88 + 1.32
46.78
+ 1.78
43.13 + 1.03
28.06 1 0.68
23.19 1 1.39
21.13 +: 1.00
PA [42]
52.77 + 0.85
47.75
+ 1.83
44.70
+ 1.28
28.71
+ 1.50
26.10
+ 0.74
25.00
1.08
MoSA (ours)
49.98 + 1.05
45.55
+ 0.77
41.69
1 0.88
25.18 + 0.72
21.70
+ 0.84
20.35 +: 1.18
observation grounctruth prediction
observation
observation
grounderuth prodiction
groundrrun prodiction
Figure 3: Heatmap of pedestrians motion (in blue) and cyclists mo- tion (in red) on SDD deathCircle location.
Figure Illustration of the difference in goal decoder output of Y-Net, pre-trained on pedestrians, after model adaptation cyclist data using our motion style adapters. Using only 20 samples, Y-Net learns to focus on the road lanes (red) instead of sidewalks (blue) for cyclist forecasting.
4.1 Motion Style Transfer across Agents on Stanford Drone Dataset
We perform short-term prediction, where the trajectory is predicted for the next 4.8 seconds, given 3.2 seconds of observation. The Y-Net model trained on pedestrian data across all scenes and adapted to the cyclists' data in death O, as there exists a clear distinction between the motion style of pedestrians and cyclists (see Fig. 3). We adapt the model using Nearget {10, 20, 30) samples.
Tab. quantifies the performance of various adaptation techniques. The model trained on pedestrians does not generalize to cyclists evidenced by the high generalization error of 58 pixels. Our MoSA design reduces this error by ~ 30% using only 30 samples. Moreover, MoSA outperforms the baselines while keeping the pretrained model frozen and updating only 2% additional parameters. Fig. 4 illustrates the updates in the Y-Net goal decoder output (red means increase in focus and blue means decrease in focus) on model adaptation using MoSA. Adapted Y-Net successfully learns the style differences between the behavior of pedestrians and cyclists: 1) it correctly infers valid areas of traversal, 2) effectively captures the multimodality of cyclists, and 3) updates the motion style parameters as the new cyclist goal positions (red) are farther from the end of the observation position, compared to the un-adapted goal positions (blue).
4.2 Motion Style Transfer across Scenes on Intersection Drone Dataset
We perform long-term prediction, where trajectory in the next 30 seconds is predicted, given seconds of observation. The Y-Net model is trained on pedestrians in {scene2, scene3, scene4] and tested on unseen scene scenel. We adapt the model using Ntarget = {10, 20, 30) samples.
Despite the long-term prediction setup, the generalization eπror is 33 pixels which is lower compared to SDD, as the target domain is more similar to the source domain. Tab. 1 quantifies the performance of scene style transfer across all methods. Using just 30 samples. MoSA improves the generalization error by ~ 40% and outperforms its counterparts. The superior performance in comparison PA justifies the importance of the low-rank constraint.

---

(a) Input sample
(b) Before adaptation
(c) After adaptation
Figure 5: Attention heatmaps of the last layer of ViT before and after model adaptation unseen route in Level 5. After adaptation, the attention maps are more refined. The ego (in green box) better focuses the different possible future routes and the vehicles in front.
Adaptive LayerNorm
70
Final Layers Fine-tune
Full Model Fine-tune
60-
MoSA
FDE
FDE
50
39
30
24
100
200
300
400
500
600
Batches
target
Figure 6: Evaluation of adaptation techniques for long-term motion prediction (25 secs) on Level 5. Error meters for 5 seeds.
Figure 7: Impact of the pre-trained dataset size (relative size shown on x-axis) motion adapta- tion on L5. Target dataset size fixed 15 batches.
4.3 Motion Style Transfer across Scenes on Lyft Level 5 Dataset
We divide the L.5 dataset into two splits based on the data collection locations thereby, constructing scene style shift scenario. We train the ViT-Tiny model on the majority route and adapt it to the smaller route not seen during training. To simulate low-resource settings, we provide the frames, sampled at different rates, that cover the unseen route only once.
Fig. quantitatively evaluates the performance of various adaptation strategies. MoSA performs superior in comparison to different baselines while adding and updating only 5% of the full model parameters. It is apparent that our low-rank design is smarter way of adapting models in the motion context as compared to fine-tuning the model. Fig. 7 empirically validates that bigger pre-training dataset size results in better adaptation performance. Finally, Fig. 5 qualitatively illustrates improvement of the attention heatmaps of the last layer of ViT post model adaptation using MoSA. MoSA helps to better focus on the different possible future routes and the vehicles in front. Additional visualizations are provided in the Appendix.
7

---

Generalizetion Fine-turing
28
Ceneralication Fine tuning
10.8
26-
10.4
10.0
22
9.6
A+F
S+F
A+F
S+A4F
Figure 8: Agent Motion Transfer from cars to trucks on Inl with Narget = 20 samples using different MoSA configura- tions. [A] performs best while [S] worsens performance. Error Top-20 FDE pix.
Figure Scene Transfer on Inl pedes- trians with Ntrrgt 20 samples using different MoSA innfiuurations(SStT per- forms best while [A] worsens performance.
Error is Top-20 FDE in pix.
4.4 Modular Motion Style Adapters
Now, we demonstrate the effectiveness of applying our adapters on top modularized architecture on two setups: Scene generalization and Agent Motion generalization. As shown in Fig. 2, we modularize the Y-Net architecture. We treat scene and agent motion independently for the first two layers of the encoder before fusing the learned representations. We refer to this design Y-Net-Mod. Given Y-Net-Mod, we consider five cases based on modules on which MoSA is applied: (1) scene only [S], (2) agent motion encoder only [A], (3) scene and fusion encoder [S+F], (4) agent motion and fusion encoder [A+F], and (5) scene, agent motion and fusion encoders together [S+A+F].
Agent motion generalization: In scenel of inD, cars and trucks share the same scene context differing only in their velocity distribution. Fig. represents the performance style transfer from cars to trucks on 20 samples under five different adaptation cases. It is interesting to note that adapting the agent motion encoder alone [A] performs the best while including the scene encoder for adaptation deteriorates performance ([S] worse than [A], [S+A+F] worse than [A+F]).
Scene generalization: We train the Y-Net-Mod model on pedestrian data scene ids 32, 4) and adapt it on scenel of inD. Fig. represents the performance of scene style transfer on 20 samples in the five cases. Contrary to the previous setup, adapting the scene encoder [S] is clearly more important than the agent motion encoder [A]. Further, adapting the agent encoder deteriorates performance ([S+A+F] worse than [S+F]). It is clear that modularization helps to boost the performance of MoSA.
5 Limitations
We demonstrated the effectiveness of decoupling our proposed motion style adapters using Y-Net- Mod. However, during training, we do not enforce any constraints on the learning objective favor effective modularization that can result in more efficient adaptation. Developing training strategies that allow quick adaptation using MoSA a potential line of future work. Another limitation the requirement of human intervention in determining the modules to be adapted given the nature of the style shift. A future direction is automating the selection of layers to be adapted for target domains.
6 Conclusion
We tackle the task of efficient motion style transfer wherein we adapt pre-trained forecasting model using limited samples from an unseen target domain. We hypothesize that the underlying shift across domains often resides in a low-dimensions space. We formulated this intuition into our motion style adapter (MoSA) design, which is trained to infer and update the style factors variation the target domain while keeping the pre-trained parameters frozen. Additionally, adaptation strategy that updates only subset of the model given the style transfer setup. Extensive experimentation three real-word weal-world datasets demonstrates the effectiveness of our approach.
present modularized

---

Acknowledgments
This work was supported by Honda R&D Co. Ltd, the Swiss National Science Foundation under the Grant 200021-L92326, and EPFL. We also thank VITA members and reviewers for their valuable comments,
References
[1] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese. Social Istm: Human trajectory prediction crowded spaces. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 961-971, 2016.
[2] K. Mangalam, Y. An, H. Girase, and J. Malik. From goals, waypoints & paths long term human trajectory forecasting. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 15213-15222, 2021.
[3] Kothari, S. Kreiss, and Alahi. Human trajectory forecasting in crowds: A deep learning perspective. IEEE Transactions on Intelligent Transportation Systems, pages 1-15, 2021.
[4] т. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++: Dynamically-ecssbbe trajectory forecasting with heterogeneous data. In ECCV, 2020.
[5] L. Wang, Y. Hu, L. Sun. W. Zhan, M. Tomizuka, and C. Liu. Transferable and adaptable driving behavior prediction. arXiv preprint arXiv:2202.05144 2022.
[6] Y. Xu, L. Wang, Y. Wang, and Y. Fu. Adaptive trajectory prediction via transferable gnn. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1520-6531, 2022.
[7] S. Saadatnejad, M. Bahari, P. Khorsandi, M. Saneian, S.-M. Moosavi-Dezfooli, and A. Alahi. Are socially-aware trajectory prediction models really socially-aware? Transportation research part C: emerging technologies, 141:103705, 2022.
[8] M. Bahari, S. Saadatnejad, Rahimi, M. Shaverdikondori, A. H. Shahidzadeh, S.-M. Moosavi- Dezfooli, and A. Alahi. Vehicle trajectory prediction works, but not everywhere. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 17123-17133, 2022.
[9] J. Liang, L. Jiang, and A. Hauptmann. Simaug: Learning robust representations from simulation for trajectory prediction. In European Conference on Computer Vision, pages 275-292, Springer, 2020.
[10] Y. Liu, Q. Yan, and A. Alahi. Social nee: Contrastive learning of socially-aware motion representations. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15118915129, 2021.
[11] Bhattacharyya, C. Huang, and K. Czarneck.. Ssl-lanes: Self-supervised learning for motion forecasting in autonomous driving. arXiv preprint arXiv:2206. 14116, 2022.
[12] E. J. Hu, P. Wallis, Z. Allen-Zhu, Li, S. Wang, L. Wang, W. Chen, al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
[13] R. K. Mahabadi, J. Henderson, and S. Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. In NeurIPS, 2021.
[14] A. Robiequet, A. Sadeghian, A. Alahi, and S. Savarese. Learning social etiquette: Human trajectory understanding in crowded scenes. In ECCV, 2016.
[15] J. Bock, R. Krajewski, T. Moers, S. Runde, L. Vater, and L. Eckstein. The ind dataset: A drone dataset of naturalistic road user trajectories at german intersections. 2020 IEEE Intelligent Vehicles Symposium (IV), pages 1929-1934, 2020.

---

[16] J. L. Houston, G. C. A. Zuidhof, L. Bergamini, Y. Ye, A. Jain, S. Omari, I. Iglovikov. and Ondruska. One thousand and one hours: Self-driving motion prediction dataset. In CoRL, 2020.
[17] Helbing and Molnár. Social force model for pedestrian dynamics. Physical review. Statistical physies, plasmas, fluids, and related interdisciplinary topics, 51 5:4282-4288 4286, 1995.
[18] Coscia, F. Castaldo, F. A. N. Palmieri, A. Alahi, S. Savarese, and L. Ballan. Long-term path prediction in urban scenarios using circular distributions. Image Vis. Comput., 69:81-91, 2018.
[19] G. Antonini and M. Bierlaire. Discrete choice models for pedestrian walking behavior. 2006.
[20] A. Vemula, K. Muelling, and J. Oh. Social attention: Modeling attention in human crowds. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1-7, 2018.
[21] A. A. Mohamed, K. Qian, M. Elhoseiny, and C. G. Claudel. Social-stgenn: A social spatio- temporal graph convolutional neural network for human trajectory prediction. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14412-14420, 2020.
[22] P. Kothari, B. Sifringer, and A. Alahi. Interpretable social anchors for human trajectory forecasting in crowds. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15551-15561 2021.
[23] Kothari and A. Alahi. complaacompliant generative adversarial networks for human trajectory forecasting, ArXiv, abs/2209. 122443,,,,.. 2022.
[24] D. Zhu, M. Zahran, L. E. Li, and M. Elhoseiny. Motion forecasting with unlikelihood training in continuous space. In Conference on Robot Learning, pages 1003-1012. PMLR, 2021.
[25] Y. Liu, R. Cadei, J. Schweizer, Bahmani, and A. Alahi. Towards robust and adaptive motion forecasting: A causal representation perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17081-17092, 2022.
[26] G. Csurka. Deep visual domain adaptation. 2020 22nd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), pages 1-8, 2020.
[27] M. Wang and W. Deng. Deep visual domain adaptation: A survey, Neurocomputing, 312: 135-153, 2018.
[28] S. Zhao, B. Li, C. Reed, P. Xu, and K. Keutzer. Multi-source domain adaptation in the deep learning era: A systematic survey, ArXiv, abs/2002. 2020.
[29] Howard and S. Ruder. Universal language model fine-tuning for text classification, In ACL, 2018.
[30] A. Radford and K. Narasimhan. Improving language understanding by generative pre-training. 2018.
[31] D. M. Cer, Y. Yang, yi Kong. N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo- Cespedes, S. Yuan, C. Tar, B. Strope, and R. Kurzweil. Universal sentence encoder for english. In EMNLP, 2018.
[32] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and Dean. Distributed representations of words and phrases and their compositionality. In NIPS, 2013.
[33] C. 7ang, M. Pei, and Y. Kong. Few-shot human motion prediction via learning novel motion dynamics. In IJCAI, 2020.
[34] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference machine learning, pages 1126-1135. PMLR, 2017.
[35] I.. Gui, Y.-.. Wang, D. Ramanan, and M. F. Moura. Few-shot human motion prediction via meta-learning. In ECCV, 2018.
10

---

[36] L. Wang, Y. Hu, L. Sun. W. Zhan, M. Tomizuka, and C. Liu. Transferable and adaptable driving behavior prediction. ArXiv, abs/2202.051400 2022.
[37] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. At- tariyan, and S. Gelly, Parameter-efffiient transfer learning for nip. In ICML, 2019.
[38] H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. Raffel. Few-shot parameter- efficient fine-tuning is better and cheaper than in-context learning. ArXiv, abs/2205.05638, 2022.
[39] A. Gupta, J. Johnson, L. Fei-Fei, Savarese, and A. Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2255-2264, 2018.
[40] K. Mangalam, An, H. Girase, and Malik. From goals, waypoints & paths long term human trajectory forecasting. In Proceedings of the IEEEJCVE International Conference on Computer Vision, pages 15233-15242, 2021.
[41] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2020.
[42] S.-A. Rebuffi, H. Bilen, and A. Vedaldi. Efficient parametrization of multi-domain deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Putlern Recognition, pages 8119-8127, 2018.
[43] Y. Li, Wang, J. Shi, J. Liu, and X. Hou. Revisiting batch normalization for practical domain adaptation. 2017.
[44] H. De Vries, F. Strub, J. Mary, H. Larochelle, O. Pietquin, and A. C. Courville. Modulating early visual processing by language. Advances in Neural Information Processing Systems, 30, 2017.


Reviews:
[{"review_id": "vSdTnElUp9v", "reviewer": "Reviewer_PiYP", "summary": "The paper proposes a deep learning model to predict the motion trajectory of agents from the pre-trained knowledge on agents of different type. The model design is based on the premise that motion style across different type of agents include common features which can be transferred to target domain. The proposed motion style adapter uses two trainable low-rank weight matrices: one inferring the style factors that need to be matched with the target domain and the second one performs the updates. During update, the pre-trained model is frozen and the low-rank matrices are trained with a low volume samples of the target domain. The second contribution include in decoupling the style adapter into scene-specific style and agent-specific style. The experiments are conducted on 3 different data-set. The results show that the proposed model out-performs with respect to error when compared to fine-tuning, partial model fine tuning and parallel adapter design.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clearly motivated in adapting pre-trained models to different domains. The language is easy to follow and the experimental results corroborates the claim. The experimental analysis also compares their method with other related work. \nMany of the details are provided in the supplementary for the paper. Without the supplementary the architecture is not detailed and needs clarity in places. The architecture is similar to one published paper which is not cited.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The updated recommendation is based on the following: The issues raised in the first review were addressed in the rebuttal. However, the additional explanations as suggested need to be incorporated for publication.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Please clearly state the differences from the architecture in reference 7 in the introduction.\n\n- The authors should consider and cite the following works, especially the body of work on meta-learning:\n[2] Finn, Chelsea, Pieter Abbeel, and Sergey Levine. \"Model-agnostic meta-learning for fast adaptation of deep networks.\" International conference on machine learning. PMLR, 2017. : This work uses meta-learning to adapt to new task using small dataset.\n\n[3] Gui, Liang-Yan, et al. \"Few-shot human motion prediction via meta-learning.\" Proceedings of the European Conference on Computer Vision (ECCV). 2018.\n\n- The function L($x_i$, $y_i$; $W_enc$, $W_dec$) in Equation (1) needs to be defined. Although it is intuitive in the community as Loss function, it needs to be stated for completeness of narrative as a published work. Also the authors need to state what mathematical functions it represents.\n\n-The difference in performance between S and S+F and A and A+F in the experiments in Sec 4.4 needs to be stated in the discussion. This completes the explanation of the behavior of fusion encoder in presence and absence of two encoders.\n\n- The size of the pre-trained data-set used in each experiment is required. Some of these detail is provided in the supplementary material. This is required to evaluate how low the volume is of target domain dataset. Also, how the performance is affected if the size of the pre-trained data-set increases or decreases and what the size of the target domain data set need to be to obtain a comparable or tolerable result. Additional experimental analysis showing the impact of the pre-trained data set size compared to the size of the target domain dataset will be insightful.\n\n- In the experiments in Sec 4.4, it would generalize the results more if other combination of scene ids are used i.e., {1,2,3} for pre-training and 4 for adapting and others. \n\n- Some of the details which are provided in the supplementary materials need to be moved in the main paper. This include the system architecture and how the proposed encoders are added to the existing deep learning models- the division of layers, where the adapters are applied and is there any additional hyperparameters? These details are required to implement the work without the support of supplementary.\n"}}, {"review_id": "jhHSw8mAulR", "reviewer": "Reviewer_Qhox", "summary": "The paper is interested in motion forecasting, and specifically the problem of adapting a motion forecasting model to some change in \"style\", which is either a scene change or change in agent behavior. The key insight is that in the low-data regime, finetuning the whole model or some parts may be ineffective, and its better to train a new, low rank matrix to predict a residual to the frozen models prediction. Experiments on some standard datasets suggest this is better than full or partial finetuning. Additionally the paper proposes to explores decomposing the encoder into a separate \"scene\" and \"agent\" part.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\n- Overall it makes sense that motion forecasting models may need to be adapted to new scenarios, and low-data methods for adaptation are important. \n- The approach of training a residual model that is constrained to be low rank makes sense as a way to adapt the model while optimizing a small number of parameters (which is important in the low-data regime).\n- The method overall is pretty straightforward and explained in a clear way.\n\n*Weaknesses/Questions*\n\nMethod:\n- Overall I think the method makes sense for the low-data regime, though I'm not sure how novel the idea of adaptation by training a residual over a frozen model is. For example in robot learning many papers learn residual policies over an expert for adaptation [1], and in general in the idea of training a new layer on top of a frozen model is commonplace in machine learning. At the same time, I think applying these ideas to a new problem like motion forecasting is worthwhile.\n- The low rank constraint on the residual layer is an interesting addition. Are the authors aware of other works in the meta-learning/finetuning literature that apply a similar constraint? \n- I'm not entirely convinced by the claim of the decomposed encoder which separates style and movement components of motion prediction. As far I understand, nothing explicitly forces each head to focus on style/motion respectively correct? How can the model be guaranteed to decompose the input into those respective components?\n\nExperiments:\n- It would be worthwhile to include more ablations which highlight the benefits of the proposed method. For example, if it is indeed the case that the low-rank constraint on the residual weights is important, the paper should include ablations which remove and also vary the size of r, and analyze how it impacts performance. \n- Additionally, for the experiment which looks at the decomposed Y-net into the scene and motion components, I don't see a head-to-head comparison between including vs. not including the decomposition. The experiment instead shows the impact of applying MoSA to the different parts of the decomposed module, but I don't see how the conclusion \"It is clear that modularization helps to boost the performance of MoSA.\" can be drawn from Figure 7 and 8. \n\nFit:\n- Overall, I think the focus of this paper is on effective adaptation of a pre-trained model using a residual layer. While it's applied to motion forecasting, the same MoSA method seems like it could be beneficial in general. So I wonder if this paper might be better evaluated at a learning/vision conference. This paper also does not apply the method on any real or simulated robotic control problem. \n\n[1] Johannik et al. Residual Reinforcement Learning for Robot Control. 2018. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall proposes an interesting approach for adapting motion forecasting models that seems to improve prediction quality, but more thorough experiments are needed to support all of the claims about the method, and ultimately the work is not applied on any real or simulated robotic control problem. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "See Strengths and Weaknesses.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}, {"review_id": "dA_HqMQzSM", "reviewer": "Reviewer_SST5", "summary": "This paper explores the problem of domain adaptation in motion forecasting. In contrast to previous methods that directly fine-tune the whole encoder, it proposes a low-rank motion style adapter that only accounts for part of the model parameters. A modular adapter strategy that disentangles the features of scene context and motion history to facilitate the fine-grained choice of adaptation layers. Experiments are conducted on Stanford Done and Intersection Drone dataset, as well as transferred to Lyft Level 5 dataset. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths__\n- The problem and motivation make sense to me as the data-driven motion forecasting methods are not generalizable, and the adaptation component is useful during inference. \n- The technical solutions are reasonable.\n- Experiments verify the effectiveness of the method. \n- The paper is easy to follow. \n\n__Weaknesses__\n- Can the authors explain the intuition that “the style shifts across forecasting domains often reside in a low-dimensional space”? I understand the statements in Line 133 that the style shifts across forecasting domains usually only exist in a small portion of latent factors. Is this consistent with “low-dimensional space”?\n- Any variations or analyses on the choices of MoSA? \n- Any analyses/visualization for the learned parameters/maps in MoSA? and the decoupled ones? \n- The ablation studies on the Modular Motion Style Adapters are a little bit confusing. I didn’t observe conclusions in the section. It seems that for different cases, the optimal choices are different. How did the authors implement the network while inference? What’s the performance without the modular design? \n- The experiments in this paper seem all transferred within the same dataset, such as pedestrian-cyclist/truck-car. However, I think this is only a small part of the applications. More comprehensive experiments shall include cross-dataset/cross-category/cross-scenes. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall I think this is a paper with reasonable motivations and solutions and deserve to be published. However, I have some concerns regarding its experiments. Hope the authors can answer these questions. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please answer my questions on weaknesses. ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "NJdVJAs7NcNH", "reviewer": "Reviewer_krte", "summary": "The paper introduces a few-shot adaptation architecture for motion style transfer. The architecture simple uses two learnable matrices  (A and B) that are inserted at each layer and the output (B^l * A^l * h^l) is simply added, coordinate wise, to the original output of each hidden layer of the original network (whose weights are held frozen). The proposed technique has far-fewer trainable parameters than fine tuning the original network, for example. The paper claims that these parameters capture the \"style\" of motion or the essence of the new \"scene\". An additional decomposition is explored in which the scene and agent information are modularized and separate weights are depending depending on the labeled data (agent, scene, or both have changed). Comparisons to competing methods on three datasets show that the proposed method is effective at obtaining superior performance on standard error metrics. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-The proposed method is conceptually simple\n-The proposed method appears to achieve superior results under apparently fair comparisons \n\nWeaknesses: \n-There are several unsubstantiated, intuitive claims in the paper. \n-The method apparently requires \"human intervention\".\n-Missing comparison to methods that use \"adaptive normalization layers\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend acceptance for two main reasons: 1-the proposed method seems to advance the state of the art on three datasets, 2-the proposed technique appears to be simple and fairly reproducible (*provided the mention of \"human intervention\" in the limitations section is is clarified). ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "-There are several unsubstantiated, intuitive claims in the paper. For example: lines 142-143 and 165-168. \n-Although I do not fully understand why, the method apparently requires \"human intervention\" (in the form of labels?) as outlined in 255-257. \n-The authors mention that a comparison to methods that use \"adaptive normalization layers\" (e.g. FiLM, or [29]) would be presented, but I did not find this comparison in the experiments. "}}, {"review_id": "4dB_IOKSyxB1", "reviewer": "Reviewer_g1Er", "summary": "Core idea of the paper: Introduce domain adaptation techniques to the field of motion forecasting. This is demonstrated in two ways. Agent adaptation or scene adaptation. In Agent adaptation experiments, authors trained the initial model on a set of scenes on a particular agent and adapted to a different agent type from the same set of scenes. In Scene adaptation, authors trained a particular agent type on a set of scenes and demonstrated adaptation of the same agent performance on a different set of scenes. Adaptation methods is a very practical and important area of study which is proliferating in the field of NLP and vision and not so much in the field of motion forecasting. So, the core idea of the paper is quite relevant and very impactful.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Weaknesses\n\n1. From figure 1, it seems like MoSA block is nothing but two linear layers without any activation function?\n2. Experiments Figure 5 suggests that MoSA block outperforms fine-tuning of the full model. It is very surprising to see this conclusion as I would expect that the MoSA block tuning should either be lower or at par in performance with the fine-tuning the full model. Any intuition as to why this is the case?\n3. Similarly in Figure 7 and 8: It is again very surprising to see that as we fine-tune more parts of the architecture, the overall performance is not doing better but on the contrary it is performing worse. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Though the work on introducing adaptation to the field of motion forecasting is new, I find the experiments to be quite in adequate and conclusions to be in effective or even counter intuitive. This suggests that he authors should re do the expeirments more carefully or provide strong supporting material or evidence along with explanation as to why full training provides overall worse performance compared to fine-tuning just the MOSA layers.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "-", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)