ManiCast: Collaborative Manipulation with Cost-Aware Human Forecasting
Authors: Kushal Kedia
Prithwish Dan
Atiksh Bhardwaj
Sanjiban Choudhury
Venue: CoRL 2023 Poster (2023)
Keywords: Collaborative Manipulation
Forecasting
Model Predictive Control

Abstract: Seamless human-robot manipulation in close proximity relies on accurate forecasts of human motion. While there has been significant progress in learning forecast models at scale, when applied to manipulation tasks, these models accrue high errors at critical transition points leading to degradation in downstream planning performance. Our key insight is that instead of predicting the most likely human motion, it is sufficient to produce forecasts that capture how future human motion would affect the cost of a robot's plan. We present ManiCast, a novel framework that learns cost-aware human forecasts and feeds them to a model predictive control planner to execute collaborative manipulation tasks. Our framework enables fluid, real-time interactions between a human and a 7-DoF robot arm across a number of real-world tasks such as reactive stirring, object handovers, and collaborative table setting. We evaluate both the motion forecasts and the end-to-end forecaster-planner system against a range of learned and heuristic baselines while additionally contributing new datasets. We release our code and datasets at https://portal-cornell.github.io/manicast/.

ManiCast: Collaborative Manipulation with Cost-Aware Human Forecasting
Kushal Kedia Cornell University
Prithwish Dan Cornell University
Atiksh Bhardwaj Cornell University
Sanjiban Choudhury Cornell University
Seamless human-robot manipulation in close proximity relies on ac- curate forecasts of human motion. While there has been significant progress learning forecast models at scale, when applied to manipulation tasks, these mod- els accrue high errors at critical transition points leading to degradation down- stream planning performance. Our key insight is that instead of predicting the most likely human motion, sufficient to produce forecasts that capture how future human motion would affect the cost of a robot's plan. We present MANI- CAST, a novel framework that learns cost-aware human forecasts and feeds them model predictive control planner to execute collaborative manipulation tasks. Our framework enables fluid, real-time interactions between a human and a 7-DoF robot arm across number of real- world tasks such as reactive stirring, object han- dovers, and collaborative table setting. We evaluate both the motion forecasts and the end-to-end peecaster-plannee system against range of learned and heuristic baselines while additionally contributing new datasets. We release our code and datasets at ht http:/porallccooceee github. wi/iii/aniii //m/a/iias///////
Abstract:
Keywords: Collaborative Manipulation, Forecasting, Model Predictive Control
1 Introduction
Seamless collaboration between humans and robots requires the ability to accurately anticipate hu- man actions. Consider shared manipulation task where human and a robot collaborate to cook a soup as the robot stirs the pot, the human adds in vegetables. Such close proximity interactions require fluid adaptions to the human partner while staying safe. To do this, the robot must predict or forecast the human's 's arm movements, and plan with such forecasts. This paper addresses the problem of generating human motion forecasts that enable seamless collaborative manipulation.
Recent works have made considerable progress training human motion forecast models by lever- aging large-scale human activity datasets such as AMASS [1] and Human 3.6M [2]. However, directly applying such models for human-robot collaboration presents several challenges. First, the space of all possible human motions is very large and pre-trained forecast models typically average out their performance over the distribution of activities seen train time. Second, although fine- tuning with task-specific data helps improve overall forecasting accuracy, it does not necessarily lead to better performance for downstream planning. This because these models are usually very accurate predicting frequent and predictable events in the data, e.g. continuing to stir ladle. However, they struggle with rare and more unpredictable transitions, c.g. pulling the ladle back as a human hand enters the workspace. Such transitions critical for seamless collaboration and errors at such data points have a substantial impact on the overall performance of the robot.
Our key insight is that instead of predicting the most tikely human motion, it is sufficient to produce forecasts that capture how future human motion would affect the cost of a robot's plan. For instance, in the cooking task, the robot's planned trajectory has high cost if it comes close to the human arm, and low cost otherwise. While trying to accurately predict how the human arm may move at any given moment is difficult, it is much easier to predict whether that movement results in a high cost. We achieve this by modifying the forecast training objective to match the cost of
[kkk837, pd337, ab2635, sc2582)8corncll.com
7th Conference on Robot Learning (CoRI. 2023), Atlanta, USA.

---

Reactive Stirring
Object Handover
Collaborative Table Setting
Figure Closed-loop, real-time collaborative human-robot manipulation across three different kitchen tasks by combining learned human pose forecasts with model predictive control.
ground truth future motions rather than the exact motion itself. We propose novel framework, MANICAST (Manipulation Forecast), that learns cost-aware human motion forecasts and plans with such forecasts for collaborative manipulation tasks. At train time, fine-tune pre-trained human motion forecasting models on task specific datasets by upsampling transition points and upweighting joint dimensions that dominate the cost of the robot's planned trajectory. At inference time, feed these forecasts into a model predictive control (MPC) planner to compute robot plans that are reactive and keep a safe distance from the human. To the best of our knowledge, this is the first paper to leverage large-scale human motion data and state-of -frt pre-trained human forecast models to integrate with real-time MPC planner for collaborative human-robot manipulation tasks. Our key contributions are:
1. A method to train human motion forecast models to be used with downstream planner for collaborative human-robot manipulation.
2. A new dataset of human -umman collaborative manipulation on 3 kitchen tasks. Real-world evaluation of a human-robot team on each of the 3 tasks and comparison against a range of learned and heuristic baselines.
2 Related Work
Navigation with Human Forecasts. Navigating around humans has been long-standing challenge in areas such as self-driving [3-8] and social robotics [9-15]. In the context of self-driving, there's rich history in trajectory forecasting of agents observed by the autonomous vehicle (AV) [16] ranging from physics-based methods (c.g. Constant Velocity/Acceleraaion [17-21]. Kalman Fil- ter [22-24]) to machine learning- based methods (c.g. Gaussian Processes [25, 26], Hidden Markov Models [27, 28]) to inverse reinforcement learning [29, 30J. In recent years, the rise of sequence models in deep learning has led to great advances in accurately predicting future states of traffic participants. Parallel to the developments in autonomous driving, social navigation in crowd en- vironments has similarly converged on sequence models [31-35] forecast trajectories within crowd. In addition to making predictions about trajectories, many recent works have used forecasts as inputs to motion planners [36-38] or jointly forecasted and planned [39, 40). The locations swept out by forecasts are considered to be "unsafe" collision regions that the planner avoids. Conse- quently, evaluation of trajectory forecasting has shifted from accuracy-based metrics [41] to more task-aware metrics that focus on the performance of downstream planner. However, all of these works approach forecasting as 2D problem, whereas we look to plan 7D robot manipulation tra- jectories by forecasting 2ID human pose, which is higher dimensional and much more complex.
Human Pose Forecasting, Human pose forecasting involves predicting how a human pose evolves in both space and time. Several works have leveraged Recurrent Neural Networks (RNN) [42, 43] and Transformer Networks [44] to model time, while graph-based models such as Graph Convolu- tional Networks (GCN) [45, 46] have been used to model the interaction of joints in space. Mao et al. [45] note that human motion over a time horizon smooth and periodic. They exploit this obser- vation by encoding the context of an agent with a Discrete Cosine Transform (DCT) and then apply

---

an attention module to capture the similarities between current and historical motion sub- sub-sequences. STS-GCN [46] learns the adjacency matrix between different joints and across different timesteps separately to bottleneck the cross-talk interactions between joints across time and use Temporal Convolutional Network (TCN) for decoding the predictions over fixed time horizon. Recent works have extended forecasting from Lingle-peroon to multi-person forecasting [47, 48). Although there have been many advances in pose forecasting, to the best of our knowledge, our work is the first to integrate an entire upper body forecast into robot manipulation planner.
Plose-Prxxmmity Human Robot Collaboration For close proximity human-robot tasks where flu- idity is important, forecasting human pose is essential. While there are various works that forecast human pose in the context of robotics [49-53], they don't integrate such predictions into robot planners. Research focused on robot manipulation planning [54-56] around humans has modeled humans some form at planning time, whether that be assuming static poses [56, 57], tracking the current pose [58, 59], or making predictions only about the motion of specific joints such as the wrist or head (3). Mainprice et al. [64] predicts single-arm reaching motion using motion capture of two humans performing& a collaborative task in a shared workspace and learns a collision avoidance cost function using Inverse Optimal Control (IOC). He et al. [65] introduce a hierarchical approach with high and low-level controller to generate feasible trajectories and ensure safety. Scheele et al. [60] use an unsupervised online learning algorithm to build a model that predicts the remainder of human reaching motion given the start of it. Ling al. [61] use the human's head pose, wrist position, and wrist speed as inputs to their forecasting Long Short- Memory (LSTM) model to predict future wrist positions which are used as input to their robot motion planner. Oguz et al. [66] propose a framework to detect and classify interactions during dlose-prxximing human-robot interactions. Prasad et al. [67] propose framework to plan humanoid motion plans representing human motion as a latent variable. In contrast to these methods, we utilized orecasting models pre-trained on larger, more diverse data to solve human-robot collaboration tasks. We focus on the efficacy of the forecaster, especially in extended interaction settings where performance in transition windows is critical but forms small part of our collected dataset.
Problem Formulation
The human's state timestep 1, s/ E R are the 3-D coordinates of J upper-body joints. Let the context", = {s" " Jf), }. be the history of human states over the past k timesteps. Let future human and robot trajectories over horizon T be En = ( ...... } and SR = (5..5....... s7 respectively. Let C( 5x) denote the cost of the robot trajectory given the human trajectory. Let (5n Φ) exp(-
Notation.
pe (o)l)) be the learnt forecast model, modelled as a .f.
Gaussian, where uo (0) )))()))))) the predicted mean trajectory, and 0 are the learned parameters.
Planning. The goal of the planner is to compute the optimal robot trajectory ČR given human trajectory En. At inference time, En is unknown, and the planner must rely on a forecast &n ~ (-(0) of the human trajectory generated by the model. The planner then solves for the lowest cost robot trajectory given the forecast - arg min &e $x-Po(.1)
E C(ÉRlEH).
Cost-Aware Human-Pose Forecasting. Typically, the forecast model is trained by maximizing log-liielihood (MLE) of the ground truth human trajectory ŠH, i.e., maxe log Po (5xlo), (). which for a Gaussian model corresponds to L2 loss. However, low log-loss can still result in a high mismatch between the costs C( (5)) and C.1g2). which can result in poor downstream planner performance. For instance, in Fig. 4, the "base model" minimizing log-loss has low errors everywhere except at critical transition points, which has significant impact on the cost function. Instead, we choose loss function {()) such that the costs calculated with the forecasts match the cost calculated with ground truth for any robot plan ER:
((0)- = E. PLEn) -Eg C(ERI
This loss is equivalent to matching moments of a cost function [68, 69]. However, directly imple- menting this as a loss function is challenging in practice for a number of reasons. First, the cost
The context can have other factors like the history of the robot's past states, which we exclude for simplicity
3

---

Training Time
Inference Time
Activity Dhsoseet
Ganerati
Sieeed
Pre-train
(AMASS)
Large- Scaae
MANICAST
Forecaats
(Coos Awerr--wwaaaa Losa)
Collbbrstive Maripufalion (CoMin)
Sumple
Fineture
Plans
STORM
Upsample transitons and
upwelght joints
MANICAST
Figure 2: Overview of training and inference pipeline for MANICAST. During training, pre-train MAN- ICAST a large human activity database and finetune on CoMaD using cost-aware loss by upsampling transitions and upweighting wrist joints. During inference, we use sampling-baset MPC, STORM, to rollout multiple robot trajectories. evaluate their costs with MANICAST forecasts and select the best plan.
function has non-iifferentiable components like collision check modules. Second, the cost function may not be convex and have poor convergence rates. Finally, the cost function may not be known upfront at train time, and likely to be tuned frequently. We address these challenges next.
Approach
We present MANICAST, a framework that learns cost-aware human forecasts and plans with such forecasts for collaborative manipulation tasks. Fig. shows both the training time and inference time process. At train time, we pre-train a state-of-the-art forecast model (STS-GCN [46]) on large-scale human activity data (AMASS [I]) and fine-tune the forecasts on our Collaborative Manipulation Dataset (CoMaD), with a focus optimizing downstream planning performance. At inference time, we detect the human pose at 120Hz using an Optitrack motion capture system, generate forecasts at 120Hz, feed the forecasts into MPC planner (STORM [70]) to compute robot plans at 50 Hz.
4.1 Train Time: Fine-tune Forecasts with Cost-Aware Loss
We first pre-train the model using standard MLE loss on a large scale human activity dataset to get a baseline forecast model. We then fine-tune this model on our own Collaborative Manipula- tion Dataset (CoMaD), that has data of two humans collaboratively performing manipulation tasks. Since directly optimizing the cost-aware loss in (1) is challenging, we propose two strategies to approximately optimize the loss without significant changes to the model training pipeline.
Strategy 1: Importance Sampling. While the MLE loss measures errors uniformly on the distri- bution P()), the cost-aware loss ((0) (1) is sensitive errors at contexts where costs generally higher. Notably, for many tasks, transition points where the human comes into the robot's 's workspace are typically high cost and hence dominate the loss. Since transition points are infrequent, the fine- tuned model has higher errors on them. Our strategy is to assign greater importance to transition points by importance sampling and then finetuning on new distribution, which we formalize below.
Let (m) be the maximum cost of a robot trajectory that a given context can induce, which we compute from collected data. We defined a transition distribution as (Ø))) P( ((Ca)I((axx(())
(Ø) 8). i.e, the distribution over contexts that induce a cost higher than a percentile threshold 8. Instead of minimizing the loss P()), we choose a new distribution @(Ø) 0.5P( 0.5P, (Φ) that mixes the original distribution with the transition distribution, effectively up-sampling the transitions. We prove the following performance bounds.
Lemma 1. For model O with bounded loss of E on P()), the final loss is bounded as E(0) S Cmax,, where Cmax = max Cmux ).). In contrast, for model with bounded loss of on the new distribution @ the final loss is bounded by E(@) 2 max 6 Cmax mma Ep(s) [[(Cmaax ()) 2 8)])
We refer the reader to the Appendix for the proof and interpretation of the bound. For our tasks, we choose a small 8 (10%) that works well in practice.

---

All Joonnt FOR (mm))
HANDOVERS
All Joints FGE (mm)
COMAD AVERAGE
20.
104
All Jeints FDe (ive)) (Tramsilion)
All Joints FDE (mmn) [Trarantora]
CoMaD ÁVERAUE
194
223
213
REACTIVE STIRKING
CoMAD AVERAGE
White FOE (mm)) (Transitons)
Westle FDE (mm)
Traniitonns)
100
323
BASE
SCRATCH
FNETUNED
MANICAST
MANICAST-W
Figure 3: Forecasting Metries (All Joints FDE, T-All Joints FDE, T-Wrist FDE) all tasks in CoMaD.
Strategy 2: Dimension Weighting. While the MLE or L2 loss sets equal weights to all joint dimen- sions J, the cost-aware loss (()) (1) is sensitive errors along certain dimensions. For example for a handover task, small error predicting the wrist position can have large impact on the cost. We empirically tune weights R the MLE loss upweight joints that are explicitly used as terms in the cost functions we define.
We combine these two strategies optimize the following proxy loss:
0()) = +0.s.
E
log Poggo
(2)
5m-PP(10)
4.2 Inference Time: Sampling-Ban Model Predictive Control with Learned Forecasts
At inference time, we invoke sampling based model predictive control (STORM [70]) to compute plans for 7-DoF robot arm. We designed set of three cost functions, one for each collaborative manipulation task, that take as input candidate robot plan and the forecasts generated by MAN- ICAST and computes cost. At every timestep, given context Φ, the model predictive control samples plans, evaluates the cost of each plan and updates the sampling distribution till conver- gence, returning the minimum cost plan. The robot takes step along the plan and replans. We provide details on the cost function and the MPC planner in the Appendix.
Experiments
5.1 Experimental Setup
Collaborative Manipulation Dataset (CoMaD). We design collaborative manipulation tasks shown in Fig (1) Reactive Stirring: Robot stirs pot while making way for human pouring in vegetables (2) Object Handovers: Robot moves to receive an object behind handed over by the hu- man (3) Collaborative Table Setting: robot and the human manipulate objects on table while not getting in each other's way. To train our forecasting model, we collect a dataset of two humans executing these tasks. contains 19 episodes of reactive stirring, 27 episodes of handovers. and 15 episodes of collaborative table setting. We refer to Appendix for more details.
Large Human-Activity Databases. The AMASS (Archive of Motion Capture As Surface Shapes) [1] dataset a large and diverse collection of human motion. consists over 40 hours of single-human motion spanning over 300 subjects. Hence pre-train models AMASS.
Forecast Models. We run the planner with different forecasts. (1) Ground Truth: CUR assumes the current human pose remains constant over the planning horizon, and FUT uses the real fu-
5

---

Model
GROUNDTRUTH
Metrie
CUR
CVM
FINETUNED
MANICASI-T
MANICAST
MANICAST-W
HAND. REACT
Stop (ms)
(=:))
(=50.9)
303.2
2711 (-256) 454.4(336)
(425.8))
(00.0(2229))
Restart Time (ms)
r++0;
288.6(+18.2)
441. (+388.)
(+33.5)
496. (+27.9)
FDR
0%
67%
0%
11%
0%
11%
Goal Detoction (ms))
0 (==))
(#11660
189.3 (+:7.88
(+:1337)
450.0
(=124.8)
488.4 13:3344)
Correct Goal Rate
100%
20%
100%
100%
100%
100%
Length (mm)
485.0
(1.99.0
(1:42.0)
(11450)
436.0 (149.44
Time to Goal (6)
4,59 (+0.37) 37)
3.77 (#0.86)
3.87 (+0.28)
3,91 (+0.32)
3.96(=0:54) 54)
4.17 (+0045)
Table 1: We integrate forecasts of different models into STORM for the reactive stirring and object handover tasks. The standard error for each planning metric is shown inside parentheses.
tures obtained from playing back CoMaD's episodes. While the latter can not be used for real-world closed-loop planning, it's the gold standard for comparing other models. (2) Baselines: The Constant Velocity Model (CVM) calculates future human pose by integrating constant velocity, calculated from the immediate 0.4s history of each joint. The WORST case model is a conservative model that constructs two large spheres (arm length radius) centered at each shoulder joint. (3) LEARNING- BASED: We train our forecasting models using the STS-GCN [46] architecture. The BASE model is trained only AMASS data, whereas the SCRATCH model is trained only on CoMaD. The FINE- TUNED model pre-trains on AMASS, but is finetuned on CoMaD only using the MLE objective. MANICAST is finetuned using the loss objective in Eq 2 with the weights wj 1. MANICAST-T is finetuned only on transition data in CoMaD. MANICAST- W upweights loss for wrist joints by 5x.
0.6
(m)
0.4
0.2
Future
ManiCast
0.0
Current
-0.2
150
All Joints ADE (mm)
ManiCast
100
50
10s
15s
20s .255 30s
Figure 4: (Top) The x- x-position of the reaching human's 's wrist in a Reactive Stirring test set episode. x 2 0.4 indicates the wrist is near the pot. (Bottom) BASE model's 's forecasts have high errors dur- ing transitions and lag behind the current pose. MANICAST, trained on CoMaD by upsampling transitions, predicts the reaching human's 's pose faster than tracking the current pose.
Forecasting Metrics. We measure the Average Displacement Error (ADE) across all 25 timesteps of prediction and the Final Displacement Error (FDE) at the final timestep of prediction. Metrics are measured separately for 7 upper body joints and wrist joints. We also report the wrist errors inside the transition windows for the stirring and handover tasks.
Planning Metrics. For quantitative results, we ran 7-DoF Franka Emika Research in the real- world and played back a recording of a human partner from CoMaD. Reactive Stirring: Test set had 9 human movements into the pot. We report average time required by the robot to stop and restart motion compared to using the current position for planning. We also report the False Detection Rate (FDR) for the human coming into the robot's workspace. Object Handover: Test set had 10 han- dovers. We report the average gain in Goal Detection time compared to using the current pose and the percentage of times when the correct goal is detected by the forecaster. Among instances where the correct goal is detected, we report the average time the robot arm took to reach the handover location and the total path distance moved by end-effector. For qualitative closed-loop evaluation entire pipeline with two new human subjects not part of CoMaD.
(Fig 1, Fig 5, Fig 6),

---

RCB VIEW
MANICAST
CURRENT POSE
CONSTANT VIL
Current Pose
Forecasted Pose
Planned Path
Figure 5: Integrating forecasting models with MPC for real-world human-robot handovers. MAN-
ICAST forecasts allow the robot to track shorter path to the handover location in less time than following the current pose or the unrealistic constant velocity model's predictions.
Planning with current pose tracking
Planning with MANICAST forecasts
Collision avoidance in Collaborative Table Setting. Following the current pose, the robot arm nearly collides with the human due to delayed prediction. MANICAST avoids this by slowing down when the human reaches in and speeding up when the human retracts their arm.
Figure 6:
5.2 Results and Analysis
01. MANICAST forecasts outperform current pose tracking. Figure 2 shows that MANICAST models have low FDE in forecasting human pose with improved planning performance (Table 1). In the reactive stirring task, MANICAST models predict both the arrival and departure of humans before the current pose. Fig 4 showcases MANICAST predictions over an entire episode from the CoMaD's test set. In the object handover task, current pose tracking predicts the goal location slower than the MANICAST models. As Fig 5 demonstrates, the robot's ss end effector chases the current wrist pose towards its eventual final location, leading longer trajectory that takes more time to execute. Further, planning with forecasts leads to safe motion as collisions can prevented with future human positions (Fig 6).
02. BASELINE models predict dynamically infeasible forecasts leading to suboptimal planning performance. As seen in Fig 5, since the CVM's predictions are not dynamically constrained, its forecasts overpredict future human positions making the robot arm deviate from the optimal path during handovers. On the other hand, seen in Table in the Appendix, the robot arm remains retracted during the reactive stirring task and does not approach the human's 's wrist for handovers following the extremely conservative WORST case model.
03. Training on AMASS or CoMaD alone is sufficient achieve optimal performance. The BASE and SCRATCH models produce erroneous forecasts on CoMaD (Fig 3). In most cases, they have larger prediction errors than simply tracking the current pose. While produces the most accurate forecasts on AMASS (Tab 2 in Appendix), the activities in CoMaD contain reaching arm motions and abrupt transitions that are not captured by the model. Fig 4 shows that its predictions lag behind current pose, when predicting the arm to retract. Whereas, SCRATCH
BASE model
7

---

is only trained on CoMaD. While this model is exposed to task-specific data, its training does not converge as forecasting 21D human pose data-intensive complex task. Consequently, planning with both of these forecasters leads to worse performance than just tracking the current pose (Table 1).
04. By upsampling transition points, MANICAST improves the accuracy and efficiency of task completion. Observing Fig 3, we note that MANICAST models have higher displacement errors on CoMaD's test set than the FINETUNED model trained on just the MLE objective. However, since the MANICAST models upsample transitions, they have lower prediction errors in transition windows relevant to planning performance. In the reactive stirring task, they detect the arrival and departure of the human in the robot's workspace quicker than FINETUNED. In the object handover task, this difference is pronounced as the MANICAST models predict the final handover location more than 2 times faster than FINETUNED (Table 1). MANICAST-T is trained on only transitions and planning with it can lead to erratic performance in some cases. For example, in the reactive stirring task, it falsely predicts the human's ' arrival into the robot's workspace in a test set episode.
O5. Upweighting wrist joints in the loss function can improve performance in planning tasks. For the collaborative manipulation tasks considered in this work, we note that predicting the location of the wrist joints has higher priority than the other upper body joints. MANICAST-W assigns 5 times more weight to the wrist joints in its loss function. This leads to lower forecasting errors for the wrist joint in CoMaD compared to all other models (Fig 3). In most cases (Table 1), this model has the best planning performance with faster detection of human pose in the robot's workspace during both the reactive stirring and handover tasks. In some cases, we note that its predictions can be unstable leading to one false detection in the reactive stirring task and larger end-effector path movement in the handover task.
6 Discussion
This work considers the problem of collaborative robot manipulation in the presence of humans in the workspace. We present MANICAST, novel framework for seamless human-robot collaboration that generates cost-aware forecasts and plans with them in real-time. Our approach was thoroughly tested three collaborative tasks with human partner in real-world setting. MANICAST lever- ages large databases of human activity. Producing dynamically consistent 21D human pose is a complex and high-dimensional problem. However, by training on a large dataset, ManiCast is able to learn the statistical regularities of human motion. MANICAST optimizes planning performance by using an approximate cost-aware loss function. We finetune the forecasting model on novel Collaborative Manipulation Dataset (CoMaD) by upsampling transition points and upweighting rel- evant joint dimensions. Our system can track human user's pose, forecast their movements, and control robotic arm in real-time and at high speed. Our system's 's forecasting module runs at 120Hz, used by an MPC at 50Hz. Fast replanning enables robots to collaborate safely with humans by allowing them to adjust their motion in response to abrupt changes in their environment. In fu- ture directions of work, we plan to extend our framework to produce conditional forecasts of human motion given robot trajectories. Further, we will attempt to directly optimize the cost function for task instead of approximating the forecasting loss function.
7 Limitations
Several reasons limit our approach's deployment to real-world human users. Firstly, method relies on a motion capture system consisting of 10 cameras to track the history of a humaa's 's pose. Further, the user is required wear motion capture suit which can be cumbersome. Future work will attempt to detect human skeletons using an ego-centric camera view. Secondly, CoMal cap- tures just two human subjects across its entire dataset. While we qualitatively show generalization to a new human subject, we do not extensively test it on a range of users. Additionally, the fore- caster may not generalize to users with movement styles not represented in the data. Future datasets should encompass multiple subjects with diverse movement styles for widespread usage of human- pose forecasting in personal robotics. This must be accompanied by a comprehensive user study to test the overall pipeline. Finally, we only considered the upper body skeleton for forecasting motion as there was significant occlusion in detecting the lower body. Our forecaster can not be directly applied to robotics applications in which the human's 's lower body movement is of interest.
8

---

Acknowledgments
This work was partially funded by NSF RI (#2312956).
References
[1] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. Black. AMASS: Archive of motion capture surface shapes. In International Conference on Computer Vision, pages 5442-5451, Oct. 2019.
[2] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7):1325-1339 2013.
[3] Z. Huang, H. Liu, J. Wu, and C. Lv. Differentiable integrated motion prediction and planning with learnable cost function for autonomous driving. arXiv preprint arXii:2207.1042) 2022.
[4] A. Sadat, M. Ren, A. Pokrovsky, Y.-.. Lin, E. Yumer, and R. Urtasun. Jointly learnable behavior and trajectory planning for self-driving vehicles. In 2019 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages
IEEE, 2019.
[5] Z. Huang, H. Liu, J. Wu, and C. Lv. Conditional predictive behavior planning with inverse reinforcement learning for human-like autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 2023.
[6] K. Mangalam, E. Adeli, K.-H. Lee, A. Gaidon, and J. C. Niebles. Disentangling human dy- namics for pedestrian locomotion forecasting with noisy supervision. In Proceedings of the IEEE/CVF Winter Conference Applications Computer Vision, pages 2784-2793, 2020.
[7] A. Monti, A. Bertugli, S. Calderara, and R. Cucchiara. Dag-net: Double attentive graph neural network for trajectory forecasting. In 2020 25th International Conference on Pattern Recog- nition (ICPR), pages 2551-2558 IEEE, 2021.
[8] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data. Computer Vision-ECCV 2020: 16th Eu- ropean Conference, Glasgow, UK, August 23-28 2020, Proceedings, Part XVIII 16. pages 683-700. Springer, 2020.
[9] Kothari and A. Alahi. Safety-compliant generative adversarial networks for human trajectory forecasting, IEEE Transactions Intelligent Transportation Systems, 2023.
[10] H. Nishimura, B. Ivanovic, A. Gaidon, M. Pavone, and M. Schwager. Risk-sensitive sequential action control with multi-modal human trajectory forecasting for safe crowd-robot interaction. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11205-11221 11122. IEEE, 2020.
[11] C. Chen, Y. Liu, S. Kreiss, and A. Alahi. Crowd-robot interaction: Crowd-aware robot navi- gation with attention-based deep reinforeement learning. In 2019 international conference robotics and automation (ICRA), pages 6015-6022, IEEE, 2019.
[12] Poddar, C. Mavrogiannis, and S. S. Srinivasa. From crowd motion prediction to robot navigation in crowds. arXiv preprini arXiv:2303.01422 2023.
[13] S. Liu, P. Chang, Z. Huang, N. Chakraborty, W. Liang, J. Geng, and K. Driggs-Campbell. Intention aware robot crowd navigation with attention-based interaction graph. arXiv preprint arXiv:2203. 0220, 2022.
[14) K. Li, M. Shan, K. Narula, S. Worrall, and E. Nebot. Socially aware crowd navigation with multimodal pedestrian trajectory prediction for autonomous vehicles. In 2020 IEEE 23rd In- ternational Conference Intelligent Transportation Systems (ITSC), pages 1-8. IEEE, 2020.
[15] C. Mavrogiannis, K. Balasubramanian, S. Poddar, A. Gandra, and S. S. Srinivasa. Winding through: Crowd navigation via topological invariance. IEEE Robotics and Automation Letters, 8(1):121-128, 2022.

---

[16] Y. Huang, J. Du, Z. Yang, Z. Zhou, L. Zhang, and H. Chen. A survey on trajectory prredcctot methods for autonomous driving. IEEE Transactions on Intelligent Vehicles, 7(3):652-66) 674, 2022.
[17] A. Polychronopoulos, M. Tsogas, A. J. Amditis, and L. Andreone. Sensor fusion for predicting vehicles' path for collision avoidance systems. IEEE Transactions on Intelligent Transporta- tion Systems, 8(3):549- 562,9956622200 2007.
[18] R. Schubert, E. Richter, and G. Wanielik. Comparison and evaluation of advanced motion models for vehicle tracking. In 2008 1/th international conference information fusion, pages 1-6. IEEE, 2008.
[19] N. Kaempehen, B. Schiele, and K. Dietmayer. Situation assessment of an autonomous emer- geney brake for arbitrary vehicle-to-vchice collision scenarios. IEEE Transactions on Intelli- gent Transportation Systems, 10(4):678-60 (877 2009.
[20] R. Pepy. A. Lambert, and H. Mounier. Reducing navigation errors by planning with realistic vehicle model. In 2006 IEEE Intelligent Vehicles Symposium, pages 300-307. IEEE, 2006.
[21] C.-F. Lin, A. G. Ulsoy, and D. J. LeBlanc. Vehicle dynamics and external disturbance esti- mation for vehicle path prediction. IEEE Transactions on Control Systems Technology, 8(3): 508-518, 2000.
[22] V. Lefkopoulos, M. Menner, A. Domahidi, and M. N. Zeilinger. Interaction-awarr motion prediction for autonomous driving: A multiple model kalman filtering scheme. IEEE Robotics and Automation Letters, 6(1):80-87, 2020.
[23] B. Jin, B. Jiu, T. Su, H. Liu, and G. Liu. Switched kalman filter-interacting multiple model algorithm based optimal autoregressive model for manoeuvring target tracking. IET Radar, Sonar & Navigation, 9(2): 199-209, 2015.
[24] N. Kaempchen, Weiss, M. Schaefer, and Dietmayer. Imm object tracking for high dynamic driving maneuvers. In IEEE Intelligent Vehicles Symposium, 2004, pages 325-830. IEEE, 2004.
[25] Joseph, F. Doshi Velez, A. Huang, and N. Roy. A bayesian nonparametric approach to modeling motion patterns. Autonomous Robots, 31:383-400, 2011.
[26] Q. Tran and Firl. Online maneuver recognition and multimodal trajectory prediction for intersection assistance using non-parametric regression. In 2014 ieee intelligent vehicles sym- posium proceedings, pages 918-923. IEEF, 2014.
[27] Q. Deng and D. Söffker. Improved driving behaviors prediction based on fuzzy logic-hidden markov model (fl-hmm). In 2018 IEEE Intelligent Vehicles Symposium (TV), pages 2003-2008. IEEE, 2018
[28] H. Berndt, Emmert, and Dietmayer. Continuous driver intention recognition with hidden markov models. In 2008 11th International IEEE Conference on Intelligent Transportation Systems, pages 1189-11 1 194. IEEE, 2008
[29] D. S. González, O. Erkent, V. Romero-Cano, J. Dibangoye, and C. Laugier. Modeling driver behavior from demonstrations in dynamic environments using spatiotemporal lattices. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3384-3390, 2018. 109:100.119/// 2....0111....8.... 8460208.
[30] N. Deo and M. M. Trivedi. Trajectory forecasts in unknown environments conditioned on grid-based plans, 2021.
[31] P. Kothari, S. Kreiss, and A. Alahi. Human trajectory forecasting in crowds: A deep learn- ing perspective. IEEE Transactions on Intelligent Transportation Systems, 23(7):7386-7400, 2021.
[32] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 961-971, 2016.
10

---

[33] D. Varshneya and G. Srinivasaraghavan. Human trajectory prediction using spatially aware deep attention models. arXiv preprint arXiv:1705.088 2017.
[34] Bartoli, G. Lisanti, L. Ballan, and A. Del Bimbo. Context-aware trajectory prediction. In 2018 24th International Conference on Patlern Recognition (ICPR), pages 1941-1946. IEEE, 2018.
[35] T. Fernando, S. Denman, S. Sridharan, and C. Fookes. Soft+ hardwired attention: An 1stm framework for human trajectory prediction and abnormal event detection. Neural networks, 108:466-444 478666478.220 2018.
[36] L. Chen, L. Platinsky, S. Speichert, B. Osinski, O. Scheel, Y. Ye, Grimmett, L. D. Pero, and P. Ondruska. What data do we need for training an av motion planner? 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1066-1072 2021.
[37] A. Kendall, Hawke, D. Janz, Mazur, D. Reda, M. Allen, V.-D. Lam, A. Bewley, and A. Shah. Learning drive in a day. 2019 International Conference on Robotics and Automa- tion (ICRA), pages 8248-8254, 2018.
[38] D. S. González, J. Pérez, V. M. Montero, and F. Nashashibi. A review of motion planning techniques for automated vehicles. IEEE Transactions on Intelligent Trunsportation Systems, 17:1135-11455 145, 2016.
[39] J. L. Vázquez, A. Liniger, W. Schwarting, D. Rus, and L. V. Gool. Deep interactive motion pre- diction and planning: Playing games with motion prediction models. ArXiv, abs/2204.02392, 2022.
[40] Ngiam, V. Vasudevan, B. Caine, Z. Zhang, H.-T. L. Chiang, J. Ling, R. Roelofs, A. Bewley, C. Liu, A. Venugopal, D. J. Weiss, B. Sapp. Z. Chen, and J. Shlens. Scene transformer: unified architecture for predicting future trajectories of multiple agents. In International Conference Learning Representations, 2022.
[41] B. Ivanovic and M. Pavone. Rethinking trajectory forecasting evaluation, 2021.
[42] K. Fragkiadaki, S. Levine, P. Felsen, and J. Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE international conference on computer vision, pages 4346-4354, 2015.
[43] A. Jain, A. Zamir, S. Savarese, and A. Saxena. Deep learning spatio-temporal graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV. USA, pages 27-30, 2016.
[44] Y. Cai, L. Huang, Y. Wang, T.-J. Cham, J. Cai, J. Yuan, J. Liu, X. Yang, Zhu, Shen, et al. Learning progressive joint propagation for human motion prediction. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16, pages 226-242. Springer, 2020.
[45] W. Mao, M. Liu, and M. Salzmann. History repeats itself: Human motion prediction via motion attention. In European Conference on Computer Vision, 2020.
[46] T. Sofianos, A. Sampieri, L. Franco, and F. Galasso. Space-time-separable graph convolutional network for pose forecasting. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11189-11198 2021.
[47] E. Vendrow, Kumar, E. Adeli, and H. Rezatofighi. Somoformer: Multi-person pose fore- casting with transformers. arXiv preprint arXiv:2208. 14023, 2022.
[48] Wang, H. Xu, M. G. Narasimhan, and X. Wang. Multi-person 3d motion prediction with multi-range transformers. In Newral Information Processing Systems, 2021.
[49] I. Gui, K. Zhang, Y.X. Wang, X. Liang, J. M. F. Moura, and M. M. Veloso. Teaching robots to predict human motion. 2018 IEFF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 562-567, 2018

---

IEEE Transactions on [50] J. Laplaza, A. Pumarola, F. Mloreno-Noguee and A. Sanfeliu. Attention deep learning based model for predicting the 3d human body pose using the robot human handover phases. In 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN), pages 161-166. IEEE, 2021.
[51] J. Zhang, H. Liu, Q. Chang, I.. Wang, and R. X. Gao. Recurrent neural network for motion trajectory prediction in human-robot collahorative assembly. CIRP annals, 69(1 (1):99212, 2020.
[52] J. Laplaza, F. Moreno-Noguer, and A. Sanfeliu. Context attention: Human motion prediction using context information and deep learning attention models. In ROBOT2022: Fifth Iberian Robotics Conference: Advances in Robotics, Volume I, pages 102-112. Springer, 2022.
[53] Q. Li, G. Chalvatzaki, J. Peters, and Y. Wang. Directed acyclic graph neural network for human 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 3197-3204, 2021. URL https:: /aaps:///p.. /app. semant cesccolar. Corg/sorp 239039533.
motion prediction.
[54] M. Faroni, M. Beschi, and N. Pedrocchi. An mpe framework for online motion planning in human-robot collaborative tasks. In 20/9 24th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), pages 1555-1558, IEEE, 2019.
[55] M. V. Minniti, R. Grandia, K. Fäh, F. Farshidian, and M. Hutter. Model predictive robot- environment interaction control for mobile manipulation tasks. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 1651-1657. IEEE, 2021.
[56] W. Yang, B. Sundaralingam, C. Paxton, I. Akinola, Y.W. Chao, M. Cakmak, and D. Fox. Model predictive control for fluid human-to-robot handovers. In 2022 International Confer- ence on Robotics and Automation (ICRA), pages 6956-6962. IEEE, 2022.
[57] E. A. Sisbot and R. Alami. A human-aware manipulation planner. Robotics, 28(5): 1045-107,7 2012.
[58] P. A. Lasota, G. F. Rossano, and J. A. Shah. Toward safe close-proximity human-robot inter- action with standard industrial robots. In 2014 IEEE International Conference on Automation Science and Engineering (CASE), pages 339-344. IEEE, 2014.
[59] H. Liu and L. Wang. Collision- free human-robot collaboration based on context awareness. Robotics and Conputer-lntegratted Manufacturing, 67:101997, 2021.
[60] S. Scheele, P. Howell, and H. Ravichandar. Fast anticipatory motion planning for close- proximity human-robot interaction. arXiv preprint arXiv:2305. 11978 2023.
[61] H. Ling, G. Liu, L. Zhu, B. Huang, Lu, H. Wu, G. Tian, and Z. Ji. Motion planning combines human motion prediction for human-robot cooperation. In 2022 12th International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), pages 672- 677. IEEE, 2022.
[62] V. Unhelkar, P. A. Lasota, Q. Tyroller, R.-D. Buhai, L. Marceau, B. Deml, and J. A. Shah. Human-aware robotic assistant for collaborative assembly: Integrating human motion predic- tion with planning in time. IEEE Robotics and Automation Letters, 3:2394 24014 2018.
[63] J. Mainprice, R. Hayne, and D. Berenson. Predicting human reaching motion in collabora- tive tasks using inverse optimal control and iterative re-planning. 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 885-892, 2015.
[64] Mainprice, R. Hayne, and D. Berenson. Goal set inverse optimal control and iterative re- planning for predicting human reaching motions in shared workspaces. IEEE Transuctions on Robotics, (2(4))897-99))) 2016.
[65] S. He, W. Zhao, C. Hu, Y. Zhu, and C. Liu. A hierarchical long short term safety frame- work for efficient robot manipulation under uncertainty. Robotics and computer-Integruute Manufacturing, 82:102522 2023, ISSN 0736-5845.
12

---

[66] O. S. Oguz, W. Rampeltshammer. S. Paillan, and D. Wollherr. An ontology for human-humai human interactions and learning interaction behavior policies. ACM Transactions on Humann-Rbbn Interaction (THRI), 8(3):1-26, 2019.
[67] V. Prasad, D. Koert, R. M. Stock-Homburg. J. Peters, and G. Chalvatzaki. Mild: Multi- modal interactive latent dynamics for learning human-robot interaction. 2022 IEEE-RAS 2 International Conference Humanoid Robots (Humanoids), pages 472-479, 2022. URL https: / /api. semant wawatticccchaa... org/ :
[68] B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, et al. Maximum entropy inverse reinforce- ment learning. Aaai, volume 8, pages 1433-1438. Chicago, II, USA, 2008.
[69] G. Swamy, S. Choudhury, J. A. Bagnell, and S. Wu. Of moments and matching: A game- theoretic framework for closing the imitation gap. In International Conference on Machine Learning, pages 10022-10332, PMLR, 2021.
[70] M. Bhardwaj, B. Sundaralingam, A. Mousavian, N. D. Ratliff, D. Fox, F. Ramos, and B. Boots. Fast joint space moeecipredictive control for reactive manipulation. In Conference on Robot Learning, 2021.
[71] G. Williams, N. Wagener, B. Goldfain, Drews, M. Rebg. B. Boots, and E. A. Theodorou. Information theoretic mpe for model-based reinforcement learning. 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 1714-1721, 2017.
13

---

A
Appendix
We run additional experiments, especially focusing on the importance wrist prediction. We also provide the proof for lemma and include additional details about the MPC planner, the collabora- tive manipulation tasks, our dataset, and model implementation.
A.1
Additional Baselines
We report forecasting metrics across AMASS and CoMaD datasets in Table 2 and planning metrics, including more baselines Table 3. Section 5.1 explains the different model implementations.
A.2 Focusing Wrist Predictions
In this section, we experiment with different wrist weights for MANICAST-V W models. Further, we compare with variant of the FINETUNED that upweights wrist dimensions, FINETUNED-W (MLE + Wrist Weighting).
Figure 7: (Reactive Stirring) Forecasting Errors (with error bars) for different wrist weights.
Observing the graphs in Fig 7, we note that all MANICAST- W models that upsample transition data have lower all joints and wrists ADE in the transition periods than the FINETUNED- W model that upweights wrist errors on the MLE loss. We observe a trade-off exists between all joints' and wrists' forecasting errors. An increase in the wrist weight in MANICAST-W reduces prediction errors the wrist joints, but at the same time, the prediction errors on all joints increase. Furthermore, the decrease in wrist errors eventually plateaus around wrist weight of 5, which justifies this hyperparameter choice for the MANICAST-W model presented in the main paper.
A.3 Fitting a Typical Pose to a Wrist Only Forecast
We consider the simpler problem of only predicting the wrist joint position in the Reactive Stirring Task. We still utilize the entire upper body history as input to the forecasting model which we name WRISTONLY. We construct the rest of the upper body by assuming the upper body at the last observable timestep remains still in the future. We consider a variant of the model that upsamples transition points, naming it WRISTONLY-T.
Figure 8: (Reactive Stirring) Comparing Forecasting Errors with TYSICAL model
In Fig 8, we observe that wrist forecasting enrors for the WRISTONLY and WRISTONLY-T are sim- ilar to the FINETUNED- W and MANICAST-T models that also predict the rest of the human upper body. We will still find that upsampling transition data points helps reduce the forecasting emror.
14

---

Metrics (mm)
BASE
SCRATCH
FINETUNED
MANICAST-T
MANICAST
MANICAST-W W
RE REACTIVESTIR AMASS
All Joints ADE
63.0 (20.11
(9.0 (2002)
(00.5 (20.)
(=.1
All Joints FDE
92.1 (=0.2)
196.7 (2:02)
(=0.2)
[415 (+0.2)
(+00.2)
Wrists ADE
103.9 (+002)
263.0 (+:03)
(=0.3)
(+0.3)
(57.2
156.0 (+:03)
Wrists FDE
154.8 (+0044
270.4))
218.6(+0.4)
(+0.4
212.8
2+0.4))
All Joints ADE
67.7 (=1.0)
(=0.9)
(+007)
(0.66)
45.1 (+006)
(=006))
All Joints FDE
110.8 (+-2.0)
102.1
(+10))
(1.3(+
(+15))
(+
Wrists ADE
94.8 (1:1.4)
(3.1))).3)
(11.1)
(5.9
(E1.0)
(1.90)
Wrists FDE
(4.28)))..)
(12.3))
(12.3
(31.7
(12.0)
112.2
T-AIl Joints ADE
92.2(=2211)
(4.77)
(+1.5)
(0.7(=13))
(8.43)
60.7 (=1.3)
T-All Joints FDE
(4:4.)
(43.5)
(24.9
(23.0)
113.4
120.1 (+3.0)
T-Wrists ADE
(3:33.33333)
(+2.7
(4294
(-.1
89.5 (+:222
85.3 (=2.1)
T-Wrists FDE
233.8 (4:6.0)
(44.9)
(=4.9)
168.9 (24.4)
163.9 (1:455)
All Joints ADE
(=00.))
(-0.3)
(+0.3)
(=0.3)
(2003)
32.4 (=0.3)
All Joints FDE
70.4 (+0.7)
(+0.6)
(4.6))))))
(-.66)
54.7 (4..6)
54.8 (+006)
TABLESET HANDOVER
Wrists ADE
(1.0.5)
(1..5)
(110.)
50.7
(1.04)
(4:0.4
Wrists FDE
100.1
(20.8
(9.0
84.5
79.4
77.2
T-All Joints ADE
(2.077)
(1.06)
(2.1
(1.5(2
41.05
40.7 (=0.5)
T-All Joints FDE
(=1.3)
87.2 (=10)
(6.91
(4.8(=1.1)
(3.7
73.0 (=100)
T-Wrists ADE
84.8 (+112)
91.2
(7.6
(-.89
65.8 (+0.9)
62.6 (=0.9)
T-Wrists FDE
146.7 (4:.11)
(21.6)
(21.)
(
(18.7
113.4 (11.7)
All Joints ADE
(01.3)
(=1.)
(8.6
(-.6))
(30.7)
56.1 (=1.0)
All Joints FDE
137.7 (+2.11
134.3 (+1.9)
(01.5
106.7 (+2.0)
101.7 (+1.)
105.1 (+1.6)
Wrists ADE
107.3 (+:.5)
114.8 (+1.3)
(11.2)
79.6 (:44)
76.7 CELO)
74.8 C 74.8
Wrists FDE
185.5 (1.2.6)
165.0 (12.2)
151.1 (22.2)
143.1 (12.3)
140.9 (12.1)
138.8 (12.0)
Table 2: Average forecast metrics (in mm) for all models across all datasets.
Midel
Basae
EJVENINO RAyND
Stop Tiacc lae)
(120))
1000
1359 (2188)
257.5 (2584)
200.3 19223
3463 (215.8)
2900 (239g)
FUR
67%
100%
11%
Dleecare (ma)
01:00)
.127 01)16
21A1
009
Path Length Sean
4.9 Di
3811 (::30
435 (2:55))
488 0 2)
466.2 (4:3.2)
435.0
428 1-
4040 (4450)
435.0 (249
Time Goalco
4.99 LUC.P)
2 (60.44)
(2108)
(10-4)
4 L-00
Table We integrate forecasts different models into STORM for reactive stirring and object handover tasks. The standard error for each planning metric is shown inside parentheses.
WRISTÖNLY-T has lower wrist ADE than WRISTONLY during transition windows. As expected, All Joints ADE for these models are significantly higher than FINETUNED and MANICAST. While this might be fine for the reactive stirring and handover task, we cannot use this model for the col- laborative table-setting task. Such an approach would generally be limited to tasks solely relying on wrist forecasting.
A.4 Proof for Lemma
Proof. We first analyze the performance of a model trained on P(ő).
Let's assume that model trained with MLE loss on P(O) bounds the average L.1 distance between the ground truth distribution P(šul0) and the learned distribution Pa(5n ()) on p()) by E, i.e.
Σ P(O) Σ P(Enlo)- - Po(E)))) <e
5M
Then, for any given ČR the robot trajectory ER. the final loss E(0), i.e. the expected cost difference of ER due to the ground truth distribution and the forecast model can be expressed as:
15

---

P(o) ((0) - P(())
((( - C(S 5H) Po Po
SN
Σ P(O)
Σ CC(ER15H) (P(5511)))))) -oo Po(55) (8))
5H
Σ P(0)
[CC(5x15m) @ 6/ (ill=EiPe [I((5H))) - Po(Ems 10)
(Holder's ineq.)
EN
(Cmax( (0)
Σ P(o)
[I Po(5m10)) [P((rro))
SH
max Caax (Ø) E)EP())
Σ
(((5))))) P Po(5x100)
Sa
where Cmax (Ø) (C(5Rl5 || is the maximum cost a robot trajectory given a context, Cmax = max, Crax (Ø) is the maximum cost across all context. Cmax inflated bound for the model above.
be high in general, resulting in an
Now let's assume we train model to minimize loss on the new distribution @(0) 0.5P()) + 0.5Pr(0) and get the following bound
Ee(o) (((Erl)))Poo(Mn se
Then the loss can be expressed as:
{(0) [p(O))(())
-CC))))) (o
5o11
Sn
Σ P())
Cmax (Ø)
Σ P((5l))) Po(5u10))
E11
P(a) (0)
Σ x()))
Σ ((((H))) - Po(Ex19))
2(0)
P(O) Pmm)
< max
Σe(
Σ P(EHl0)) (Exlo))
((0)
5y
For @(0) = 0.5P(() + ( we need to bound the ratio
max P( Cmaa
Pmm) (9)
2(Φ) = -ax
(.5P(0)+00552)))
There are consider:
Case 1: Cmax (0) < 8. Then (0) = 0, and the ratio is bounded by
((x) S max 0.5P()) 0.5P(g) < 28
P())8
Case 2: Cmux (0) > 8. Then ratio is maximized when Cmax (0) is maximized for @ 0"
Pma)) (0) P()') mmxx < P(ő Cmax Eg P()) (0) 8)
max 0,5P()) + +r.5p(
(x*)
0.5P(x*)
2Cman Ep(o) [l((Cau (0) 1 ()]
16

---

P(O)Cmax(0) ((0)
Combining these cases, bound the ratio max
P(a) (0) max @(Φ) I 22ax ( Cmx Bp(o) (Tmxxa 2 8)1)
The ratio above can be no worse than Cmax by factor of 2, and can be much smaller based on choice of Intuitively setting ò to be very high makes the transition probability Pr(0) peaky driving down the second term, while making ô to be small makes the transition probability close to the original distribution, driving down the first term.
A.5 MPC Planner Details
We the open-sourced STORM codebase to implement sampling- based model-predictee control on a 7-DOF Franka Research 3 robot arm. At every timestep. the planner samples robot trajectories and evaluates the cost function with MANICAST forecasts. The robot executes the first action from the lowest-cost plan and updates its sampling distribution for the next timestep using the MPPI [71] algorithm. The manipulation components of the cost function independent of the human remain unchanged. We additionally introduce a collaborative task task-specific cost component (T (T(5x15H) that depends on the future human trajectory. The cost function optimized by the planner is laid out in Eq.3. Self-collisions are checked by training the jointNERE model introduced by Bhardwaj et al. [70].
C(ÉR (H)) C,CCttop ac, (5x) + Cmanip O (5x) + C.Con (T,((
(3)
A.6 Tasks for Collaborative Manipulation
We describe three collaborative manipulation tasks that focus on house-hold cooking activities.
Reactive Stirring: In this cooking task, the human and robot share common workspace. While the robot arm performing stirring motion, the human may add vegetables into the pot. The robot arm preemptively predicts the arrival of the human arm and retracts back to give the human arm sufficient space reach into the pot. The task-specific component of the cost function is:
T(5RISH)= Σ [D(S sn SE) 1|s nTM || [ spot)
El
11
E
(4)
The cost function checks whether the human's 's position (8 (") is close to the pot's position (s²*t) and decides whether move to pre-defined resting position (stest or to continue stirring in a circular trajectory (5stir) starting from the current state of the robot (sf)) A cost-aware forecasting model for this task should be able predict the arrival and departure of the human ahead of time.
Human Robott Handovers: Handovers of objects are an important task in the kitchen. When human is handing over an object, robot arm should move towards the intended handover location. The task-specific component can be described as:
T(ER En)
Σ
Cpese (x.", GraspPosee Xee x
(5)
Similar to prior work [56]. the robot motion is initiated when the human arm has picked up the handover object. The robot's end-effector (X,r) moves towards grasp location that computed using the final wrist position of the human (&H The orientation of the grasp pose is calculated by drawing straight line from the current end-effector position (X.*) to the grasp location.
Collaborative Table Setting: Movements on top of a table in the presence of human in the workspace are a common collaborative manipulation task. Motion planners should not only avoid
hhnps//githib..om/////abaaaa
17

---

collision in the current timestep but also be able to forecast future motion and preemptively avoid collisions with the human body. The cost function is simply given by:
T T E pase (x, xx) BCcull (5),)))))
Here, B is the relative weight given to the collision avoidance component compared to the goal- reaching component. Collisions are checked between the human body and robot arm by representing them as a pack of sphere and cuboid rigid bodies.
A.7 Collaborative Manipulation Dataset (CoMaD)
Similar real-world collaborative activity, in much of the episode, both humans perform their respective cooking tasks in isolation. Episodes of reactive stirring and handovers contain 3-5 close- proximity interactions, each of which short (4-5 seconds) compared the length of the overall episode (30-60 seconds). Often, these interactions are initiated by verbal requests or subtle facial gestures. Collaborative table setting consists almost entirely of close-proximity fast human arm movements. We collect an RGB visual view of the scene containing audio along with motion capture data of both humans' upper bodies. We also annotate transition windows for interactions in each episode.
A.8 Model Implementational Details
We train forecasting models using the STS-GCN [46] architecture ton consisting of 7 joints (Wrists, Elbows, Shoulders, and Upper Back). The last 0.4 seconds (10 timesteps) of motion input the models and the next second (25 timesteps) of motion is pre- dicted. We pretrain for 50 epochs on AMASS (I hour) and finetune on CoMaD for 50 epochs (5 minutes). We divided the episodes in CoMaD into train, validation, and test sets (8:1:1).
upper body skele-
18


Reviews:
[{"review_id": "mnq5uyhznr", "reviewer": "Reviewer_1c4Z", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper \"ManiCast: Collaborative Manipulation with Cost-Aware Human Forecasting\" proposes a framework that integrates cost-aware human forecasts into a model predictive control planner for collaborative manipulation tasks. This review aims to provide constructive feedback to improve the scientific writing and address specific aspects of the paper.", "summary_of_recommendation": "In conclusion, the paper \"ManiCast: Collaborative Manipulation with Cost-Aware Human Forecasting\" presents a valuable framework for incorporating cost-aware human forecasts into collaborative manipulation tasks. The strengths of the paper lie in its well-written structure, the clear advantages of the cost-based strategy, and the justification for task-oriented forecasting loss. Addressing the weaknesses by providing more in-depth analysis and comparisons, as suggested, would significantly enhance the overall scientific contribution of the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Can the authors explain whether the improved accuracy for wrists is a result of assigning more weight to wrists during training or due to a specific focus on predicting wrists with a typical pose?\nThe review mentions that the overall error of ManiCast on all joints appears higher than that of the Finetuned baseline. Can the authors provide a detailed analysis and discuss the reasons behind this difference in performance?\nHow does the proposed approach compare to a basic planning scheme for the stirring task, which involves detecting the stirring place, moving, and stirring while avoiding human wrists? Can the authors evaluate whether the proposed approach outperforms this simple scheme and provide insights into the relative strengths and limitations?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well-structured, effectively introducing the problem domain and motivating the need for cost-aware human forecasting in collaborative manipulation tasks. The experimental results presented clearly demonstrate the advantages of the cost-based strategy employed in the proposed framework. The use of task-oriented forecasting loss is justified and delivers a clear message to readers.\n\nHowever, there are certain weaknesses that need to be addressed. In Table I, the \"Average forecast metrics\" show that the Base model produces the lowest error on all joints, while the proposed model exhibits better accuracy for wrists. It would be beneficial to elaborate on whether the proposed model assigns more weight to the wrists during training or solely focuses on predicting wrists while fitting a typical pose to their positions. Providing further analysis and discussion on this aspect would strengthen the paper.\n\nAdditionally, more detailed analyses and discussions are required to compare the performance of the Finetuned baseline and ManiCast. The overall error of ManiCast on all joints appears higher than that of the Finetuned baseline. Expanding on this comparison would provide valuable insights and help readers understand the relative performance of the two approaches.\n\nAnother important consideration is the comparison between the proposed approach and a simple planning scheme for the stirring task. It is essential to evaluate whether the proposed approach can outperform a basic planning scheme that involves detecting the stirring place, moving to that location, and stirring while avoiding human wrists. Since all the actions in this task are deterministic, it is crucial to assess whether the proposed approach offers advantages over such a basic planning scheme.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "Zce41EAvjT", "reviewer": "Reviewer_Z1jy", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces ManiCast, a framework for collaborative manipulation in robotics that focuses on generating cost-aware human forecasts. The main idea is to produce forecasts that capture how future human motion would affect the cost of a robot's plan, rather than predicting the most likely human motion. It combines pre-trained human forecast models with a real-time model predictive control planner. \nAs the main contribution, the paper presents a novel approach to collaborative manipulation in robotics, emphasizing cost-aware human forecasts as a means to enable seamless interactions and improve planning performance in close proximity tasks.\nThe framework is evaluated through real-world tasks (reactive stirring, object handovers, and collaborative table setting), and compared against a range of learned and heuristic baselines. A new dataset comprising human-human collaborative manipulation on three kitchen tasks is also introduced.", "summary_of_recommendation": "While the work addresses an important problem in robotics, and proposes a novel adaptation of the loss function for motion prediction, the review identifies several weaknesses in the paper, including the lack of convincing evidence to support the claims, the need for major revisions to improve clarity and organization, and uncertainty regarding the impact of the work. The reviewer suggests that significant additional evidence, thorough revisions, and more compelling evaluations are necessary to strengthen the paper. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "As highlighted in the review section:\n\n- Clarify and rephrase the contribution claims to better align with the presented work and its outcomes.\n\n- Provide a more thorough explanation and justification for the emphasis on upper body movement prediction.\n\n- Consider selecting evaluation tasks that involve more representative close-proximity interactions to better demonstrate the effectiveness of the proposed framework.\n\n- Improve the organization and contextualization of the related work section, ensuring clear connections to the authors' work and addressing any missing relevant references.\n\n- Conduct a more extensive evaluation with a variety of tasks and provide a deeper analysis and interpretation of the results.\n\n- Consider including a simple learning-based baseline for comparison to better evaluate the proposed approach.\n\n- Streamline and clarify the presentation of evaluation results, including Tables 1 and 2, and provide detailed explanations for the data and analysis.\n\n- Analyze and discuss the table setting task, as it represents a more critical scenario for motion prediction and reactive robot motion planning.\n\n- Revisit the justification for joint position prediction and consider exploring joint angle predictions given a kinematic model, which could reduce output dimensionality.\n\n- Consider incorporating more explicit analysis of the impact and significance of wrist predictions, which appear to be the most influential in the evaluation tasks.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality:\n\nThe quality of the work is generally fine. The paper addresses an important problem in robotics, namely seamless human-robot collaboration in close proximity tasks. The authors present a well-defined framework, ManiCast, that combines cost-aware human forecasts with a model predictive control planner. The experiments are conducted on real-world tasks, and the proposed framework is compared against various baselines, providing an evaluation of its performance. \n\nClarity:\n\nThe paper is generally well-written. The introduction sets the context and motivation for the research, and the problem formulation is clearly presented. The authors provide a fair explanation of the proposed framework, including the training and inference processes. However, the wording of some of the premises might be restructured to align the contributions with the achieved outcomes. Also, some sections, particularly in the related work, could benefit from clearer organization and more explicit connections to the authors' work. Additionally, the presentation of the evaluation results could be further improved by providing more specific details and analysis.\n\nOriginality:\n\nThe paper introduces a novel approach to collaborative manipulation by focusing on cost-aware human forecasts. While there is existing research on human-robot collaboration and human motion forecasting, the combination of these two aspects in the proposed framework is original. The authors integrate state-of-the-art pre-trained human forecast models with an existing model predictive control planner, showcasing the effectiveness of their approach. The upsampling of transition points and the dimension weighting strategies contribute to the originality of the work.\n\nSignificance:\n\nThe work presented in the paper has some implications for the field of robotics, specifically in the area of collaborative manipulation. The ability to accurately forecast human motion while considering the cost of robot plans is crucial for safe and efficient human-robot interactions. The proposed framework, ManiCast, addresses this challenge but falls short on providing convincing evaluations in some aspects. The introduction of the CoMaD dataset is valuable but its coverage does not seem to be large enough to be effectively used by the community. \n\nStrengths:\n\n- The integration of cost-aware human forecasts with a model predictive control planner is a novel and valuable contribution to the field.\n\n- The use of real-world tasks and the creation of the CoMaD dataset enhance the practical relevance of the research.\n\nWeaknesses:\n\n- The contribution claims are not well-supported by what is presented in the paper. I agree with the first contribution, i.e., the introduction of the combined forecasting and MPC-based planner. However, the introduced dataset only includes two persons (unclear backgrounds, biases, etc.) and three tasks which makes it hardly a full-scale dataset that can be used by the community for diverse purposes. Similarly, the other contribution on ‘extensive real-world evaluation’ also falls short considering the coverage of tasks and evaluations reported.\n\n- The paper emphasizes, as one aspect of its novelty, the forecasting of the full upper body. However, the tasks considered in the experiments are not convincing enough to understand the need for such upper body movement prediction. \n\n- Close-proximity interactions are emphasized, however, the two tasks (handover, stirring), due to setups being investigated, do not seem to include such a close interaction that is expected to impose harder motion planning constraints on the robot partner. The third task, table setting, has not been investigated thoroughly.\n\n- Some sections, particularly in the related work, could benefit from clearer organization and better contextualization within the authors' work. \n  - There are some works in the literature that combines human hand/arm motion prediction together with robot motion planning and/or action selection (some examples {1,2}, but there are definitely more) which have not been covered.\n  - My concern is less on the coverage of additional papers but more on how some premises are phrased: e.g., “To the best of our knowledge, this is the first paper to combine state-of-the-art pre-trained human forecast models like STS-GCN [3] with a real-time MPC planner [4] for collaborative human-robot manipulation tasks.” Yes this work might be the first to combine STS-GCN and an MPC planner, but this sentence sounds also a little bit like this is the first work to combine human motion prediction and a robot planner. I’d suggest rephrasing these sentences to emphasize that the new parts are the used methods, not the formulation of a predictor and a reactive planner. Similar arguments provided in the Related Work should be reconsidered, and if necessary, rephrased. \n  - “...to the best of our knowledge, our work is the first to integrate an entire upper body forecast into a robot manipulation planner.” But what is necessary/good about upper body movement prediction?\n  - “In contrast to these methods which focus on specific motions such as reaching and consider certain parts of the body, our work uses a history of human poses to predict future poses of the entire upper body in real-time for safe planning around the human body.” This movements in this work are also reaching motions, and other works also consider the history, so these premises are not well-supported in my opinion.\n \n\n- The experiments are not extensive.\n\n- The presentation of the evaluation results could be more detailed and include deeper analysis and interpretation.\n  \n  - The BASE and SCRATCH baselines are meaningful, but to better evaluate the proposed approach a simple learning-based baseline may be implemented (e.g., just a plain recurrent neural network policy)\n  \n  -  Table 1 could be placed in the appendix. The main text would be better off with a summarized version of this data.\n  \n  - Similarly, Table 2 could include only the important bits. And what is presented in this table should be explained clearly. As far as I understood what is being considered is the difference between the restart time and stop time for the stirring task but it’s not clearly explained. Why should we not care about the response time for stopping, but just the difference? And also is it well justified to have a higher FDR when the reaction time is faster for MANICAST-W?\n  \n  - Overall, a good chunk of data in those tables have not been discussed, nor explained.\n\n  - Figure 3: Is the unit of time in the x-axis in seconds? A reaching motion takes ~10seconds?\n\n  - why isn’t any detailed analysis of the table setting task? It seems to be a more representative close-proximity task where motion prediction and reactive robot motion planning would be more critical.  \n\n  - The upper body emphasis is not well analyzed and supported by the evaluations. Wrist seems to have the most influence, that is to be expected due to the nature of the task setups which do not have much close-proximity collaborative motions.\n    - How about just using the wrist prediction? \n\n  - It’s not clear why joint positions are predicted. \n    - How do you enforce kinematic consistency?\n    - Have you considered joint angle predictions given a kinematic model, which you can get from the motion tracker? Then the output dimensionality would be much smaller as well.\n\n\n_1. “Goal Set Inverse Optimal Control and Iterative Re-planning for Predicting Human Reaching Motions in Shared Workspaces”, Jim Mainprice, Rafi Hayne, Dmitry Berenson, 2016, TRO._\n\n_2. “An Ontology for Human-Human Interactions and Learning Interaction Behavior Policies”, O S Oguz, W Rampeltshammer, S Paillan, D Wollherr, 2019, T-HRI._", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "ZMHxoKe5oK", "reviewer": "Reviewer_M4Pj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors in this paper first collected a dataset called CoMaD which contains motion capture data of two people doing 3 collaborative cooking tasks. Then the authors introduce ManiCast which is an algorithm for forecasting future human motion and then adjusting the planning cost for the robot in regions that are close to the human in case it leads to a collision. ManiCast is able to run in realtime at 120 Hz to update the robot's MPC planner at 50 Hz.  ", "summary_of_recommendation": "I decided on a weak accept because there is no video. I need more proof that the method is able to plan faster around humans while ensuring safety compared to other previous human robot safety papers from researchers like Professor Changliu Liu which you have cited none of. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Needs a supporting video to show the real time operation of the human prediction and planner all working together on the Franka Research 3. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I think the paper is well written and is clear. It is a combination of human prediction works and human robot safety works. I think that there is valuable insights on how to train the ManiCast with importance sampling and dimension weighting to make this work because the human's wrist's direction is important for handling handover tasks. I think it would have been nice to have a video, but I guess it is difficult to blur everyone out for anonymity. That is the only weakness so far as I would like to see the system work in real time on video.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "YKv_D7iXGR", "reviewer": "Reviewer_EFoV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents an alternative learning objective for human pose forecasting for human-robot manipulation. Instead of only using pose prediction loss, it uses cost prediction loss to finetune the prediction model. The model was evaluated on three tasks -- Reactive Stirring Object Handover Collaborative Table Setting. Experimental results show that the prediction results, though not more accurate in terms of errors for all joints, are more accurate for planning.", "summary_of_recommendation": "My main concern is that if a simpler approach is essentially enough to achieve a similar result. If this is adequately addressed, I am willing to increase my rating.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please address my concerns above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n1. Very interesting idea and the task-oriented prediction loss is technically sound and novel. \n\n2. The results support the hypothesis that cost-based loss can improve prediction for planning.\n\n3. Writing is clear.\n\nWeaknesses:\n\n1. My main concern is that it is unclear if there could be a simpler approach that can achieve the same result. Looking at the pose prediction errors, it is clear that the *finetuned* model using MLE loss produces the lowest error on all joints, but the proposed model has better accuracy for wrists, which I assume are more important for planning. However, would adding more weights to the MLE loss on wrists give you a similar benefit? Or could you simply only train to predict wrists and fit a typical pose to the predicted wrists' positions?\n\n2. On a related note, there should be more analyses/discussions on the performance comparison between the *finetuned* baseline and ManiCast. E.g., I do not think it is accurate to say ManiCast has a more accurate prediction than *finetuned*. Its overall error on all joints is higher than *finetuned*.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel framework for robotics that emphasizes the integration of cost-aware human forecasts into a model predictive control (MPC) planner. 

Instead of predicting the most likely human motion, ManiCast aims to understand how future human movements might impact the robot's planning costs. This approach enhances collaborative tasks by enabling smoother interactions, especially in close proximity. The authors evaluated the framework's efficiency through real-world tasks and introduces a new dataset on human-human collaborative manipulation in kitchen scenarios. This review seeks to provide feedback to enhance the paper's scientific presentation and address its specific elements.

The majority of the reviewers agreed on the merits and technical soundness of the paper, and they found the author response helpful and convincing. The reviewers unanimously recommend accepting the paper (though `Z1jy` forgot to change the rating in the system). I recommend accepting this paper. The authors are encouraged to address the remaining issues, such as "1) effectiveness of predicting the full upper-body marginal,  and 2) tasks being considered are not really close-proximity tasks".
Decision: Accept (Poster)