Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper "ManiCast: Collaborative Manipulation with Cost-Aware Human Forecasting" proposes a framework that integrates cost-aware human forecasts into a model predictive control planner for collaborative manipulation tasks. This review aims to provide constructive feedback to improve the scientific writing and address specific aspects of the paper.
review: The paper is well-structured, effectively introducing the problem domain and motivating the need for cost-aware human forecasting in collaborative manipulation tasks. The experimental results presented clearly demonstrate the advantages of the cost-based strategy employed in the proposed framework. The use of task-oriented forecasting loss is justified and delivers a clear message to readers.

However, there are certain weaknesses that need to be addressed. In Table I, the "Average forecast metrics" show that the Base model produces the lowest error on all joints, while the proposed model exhibits better accuracy for wrists. It would be beneficial to elaborate on whether the proposed model assigns more weight to the wrists during training or solely focuses on predicting wrists while fitting a typical pose to their positions. Providing further analysis and discussion on this aspect would strengthen the paper.

Additionally, more detailed analyses and discussions are required to compare the performance of the Finetuned baseline and ManiCast. The overall error of ManiCast on all joints appears higher than that of the Finetuned baseline. Expanding on this comparison would provide valuable insights and help readers understand the relative performance of the two approaches.

Another important consideration is the comparison between the proposed approach and a simple planning scheme for the stirring task. It is essential to evaluate whether the proposed approach can outperform a basic planning scheme that involves detecting the stirring place, moving to that location, and stirring while avoiding human wrists. Since all the actions in this task are deterministic, it is crucial to assess whether the proposed approach offers advantages over such a basic planning scheme.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: In conclusion, the paper "ManiCast: Collaborative Manipulation with Cost-Aware Human Forecasting" presents a valuable framework for incorporating cost-aware human forecasts into collaborative manipulation tasks. The strengths of the paper lie in its well-written structure, the clear advantages of the cost-based strategy, and the justification for task-oriented forecasting loss. Addressing the weaknesses by providing more in-depth analysis and comparisons, as suggested, would significantly enhance the overall scientific contribution of the paper.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The authors in this paper first collected a dataset called CoMaD which contains motion capture data of two people doing 3 collaborative cooking tasks. Then the authors introduce ManiCast which is an algorithm for forecasting future human motion and then adjusting the planning cost for the robot in regions that are close to the human in case it leads to a collision. ManiCast is able to run in realtime at 120 Hz to update the robot's MPC planner at 50 Hz.  
review: I think the paper is well written and is clear. It is a combination of human prediction works and human robot safety works. I think that there is valuable insights on how to train the ManiCast with importance sampling and dimension weighting to make this work because the human's wrist's direction is important for handling handover tasks. I think it would have been nice to have a video, but I guess it is difficult to blur everyone out for anonymity. That is the only weakness so far as I would like to see the system work in real time on video.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I decided on a weak accept because there is no video. I need more proof that the method is able to plan faster around humans while ensuring safety compared to other previous human robot safety papers from researchers like Professor Changliu Liu which you have cited none of.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper introduces ManiCast, a framework for collaborative manipulation in robotics that focuses on generating cost-aware human forecasts. The main idea is to produce forecasts that capture how future human motion would affect the cost of a robot's plan, rather than predicting the most likely human motion. It combines pre-trained human forecast models with a real-time model predictive control planner. 
As the main contribution, the paper presents a novel approach to collaborative manipulation in robotics, emphasizing cost-aware human forecasts as a means to enable seamless interactions and improve planning performance in close proximity tasks.
The framework is evaluated through real-world tasks (reactive stirring, object handovers, and collaborative table setting), and compared against a range of learned and heuristic baselines. A new dataset comprising human-human collaborative manipulation on three kitchen tasks is also introduced.
review: Quality:

The quality of the work is generally fine. The paper addresses an important problem in robotics, namely seamless human-robot collaboration in close proximity tasks. The authors present a well-defined framework, ManiCast, that combines cost-aware human forecasts with a model predictive control planner. The experiments are conducted on real-world tasks, and the proposed framework is compared against various baselines, providing an evaluation of its performance. 

Clarity:

The paper is generally well-written. The introduction sets the context and motivation for the research, and the problem formulation is clearly presented. The authors provide a fair explanation of the proposed framework, including the training and inference processes. However, the wording of some of the premises might be restructured to align the contributions with the achieved outcomes. Also, some sections, particularly in the related work, could benefit from clearer organization and more explicit connections to the authors' work. Additionally, the presentation of the evaluation results could be further improved by providing more specific details and analysis.

Originality:

The paper introduces a novel approach to collaborative manipulation by focusing on cost-aware human forecasts. While there is existing research on human-robot collaboration and human motion forecasting, the combination of these two aspects in the proposed framework is original. The authors integrate state-of-the-art pre-trained human forecast models with an existing model predictive control planner, showcasing the effectiveness of their approach. The upsampling of transition points and the dimension weighting strategies contribute to the originality of the work.

Significance:

The work presented in the paper has some implications for the field of robotics, specifically in the area of collaborative manipulation. The ability to accurately forecast human motion while considering the cost of robot plans is crucial for safe and efficient human-robot interactions. The proposed framework, ManiCast, addresses this challenge but falls short on providing convincing evaluations in some aspects. The introduction of the CoMaD dataset is valuable but its coverage does not seem to be large enough to be effectively used by the community. 

Strengths:

- The integration of cost-aware human forecasts with a model predictive control planner is a novel and valuable contribution to the field.

- The use of real-world tasks and the creation of the CoMaD dataset enhance the practical relevance of the research.

Weaknesses:

- The contribution claims are not well-supported by what is presented in the paper. I agree with the first contribution, i.e., the introduction of the combined forecasting and MPC-based planner. However, the introduced dataset only includes two persons (unclear backgrounds, biases, etc.) and three tasks which makes it hardly a full-scale dataset that can be used by the community for diverse purposes. Similarly, the other contribution on ‘extensive real-world evaluation’ also falls short considering the coverage of tasks and evaluations reported.

- The paper emphasizes, as one aspect of its novelty, the forecasting of the full upper body. However, the tasks considered in the experiments are not convincing enough to understand the need for such upper body movement prediction. 

- Close-proximity interactions are emphasized, however, the two tasks (handover, stirring), due to setups being investigated, do not seem to include such a close interaction that is expected to impose harder motion planning constraints on the robot partner. The third task, table setting, has not been investigated thoroughly.

- Some sections, particularly in the related work, could benefit from clearer organization and better contextualization within the authors' work. 
  - There are some works in the literature that combines human hand/arm motion prediction together with robot motion planning and/or action selection (some examples {1,2}, but there are definitely more) which have not been covered.
  - My concern is less on the coverage of additional papers but more on how some premises are phrased: e.g., “To the best of our knowledge, this is the first paper to combine state-of-the-art pre-trained human forecast models like STS-GCN [3] with a real-time MPC planner [4] for collaborative human-robot manipulation tasks.” Yes this work might be the first to combine STS-GCN and an MPC planner, but this sentence sounds also a little bit like this is the first work to combine human motion prediction and a robot planner. I’d suggest rephrasing these sentences to emphasize that the new parts are the used methods, not the formulation of a predictor and a reactive planner. Similar arguments provided in the Related Work should be reconsidered, and if necessary, rephrased. 
  - “...to the best of our knowledge, our work is the first to integrate an entire upper body forecast into a robot manipulation planner.” But what is necessary/good about upper body movement prediction?
  - “In contrast to these methods which focus on specific motions such as reaching and consider certain parts of the body, our work uses a history of human poses to predict future poses of the entire upper body in real-time for safe planning around the human body.” This movements in this work are also reaching motions, and other works also consider the history, so these premises are not well-supported in my opinion.
 

- The experiments are not extensive.

- The presentation of the evaluation results could be more detailed and include deeper analysis and interpretation.
  
  - The BASE and SCRATCH baselines are meaningful, but to better evaluate the proposed approach a simple learning-based baseline may be implemented (e.g., just a plain recurrent neural network policy)
  
  -  Table 1 could be placed in the appendix. The main text would be better off with a summarized version of this data.
  
  - Similarly, Table 2 could include only the important bits. And what is presented in this table should be explained clearly. As far as I understood what is being considered is the difference between the restart time and stop time for the stirring task but it’s not clearly explained. Why should we not care about the response time for stopping, but just the difference? And also is it well justified to have a higher FDR when the reaction time is faster for MANICAST-W?
  
  - Overall, a good chunk of data in those tables have not been discussed, nor explained.

  - Figure 3: Is the unit of time in the x-axis in seconds? A reaching motion takes ~10seconds?

  - why isn’t any detailed analysis of the table setting task? It seems to be a more representative close-proximity task where motion prediction and reactive robot motion planning would be more critical.  

  - The upper body emphasis is not well analyzed and supported by the evaluations. Wrist seems to have the most influence, that is to be expected due to the nature of the task setups which do not have much close-proximity collaborative motions.
    - How about just using the wrist prediction? 

  - It’s not clear why joint positions are predicted. 
    - How do you enforce kinematic consistency?
    - Have you considered joint angle predictions given a kinematic model, which you can get from the motion tracker? Then the output dimensionality would be much smaller as well.


_1. “Goal Set Inverse Optimal Control and Iterative Re-planning for Predicting Human Reaching Motions in Shared Workspaces”, Jim Mainprice, Rafi Hayne, Dmitry Berenson, 2016, TRO._

_2. “An Ontology for Human-Human Interactions and Learning Interaction Behavior Policies”, O S Oguz, W Rampeltshammer, S Paillan, D Wollherr, 2019, T-HRI._
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: While the work addresses an important problem in robotics, and proposes a novel adaptation of the loss function for motion prediction, the review identifies several weaknesses in the paper, including the lack of convincing evidence to support the claims, the need for major revisions to improve clarity and organization, and uncertainty regarding the impact of the work. The reviewer suggests that significant additional evidence, thorough revisions, and more compelling evaluations are necessary to strengthen the paper.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper presents an alternative learning objective for human pose forecasting for human-robot manipulation. Instead of only using pose prediction loss, it uses cost prediction loss to finetune the prediction model. The model was evaluated on three tasks -- Reactive Stirring Object Handover Collaborative Table Setting. Experimental results show that the prediction results, though not more accurate in terms of errors for all joints, are more accurate for planning.
review: Strengths:

1. Very interesting idea and the task-oriented prediction loss is technically sound and novel. 

2. The results support the hypothesis that cost-based loss can improve prediction for planning.

3. Writing is clear.

Weaknesses:

1. My main concern is that it is unclear if there could be a simpler approach that can achieve the same result. Looking at the pose prediction errors, it is clear that the *finetuned* model using MLE loss produces the lowest error on all joints, but the proposed model has better accuracy for wrists, which I assume are more important for planning. However, would adding more weights to the MLE loss on wrists give you a similar benefit? Or could you simply only train to predict wrists and fit a typical pose to the predicted wrists' positions?

2. On a related note, there should be more analyses/discussions on the performance comparison between the *finetuned* baseline and ManiCast. E.g., I do not think it is accurate to say ManiCast has a more accurate prediction than *finetuned*. Its overall error on all joints is higher than *finetuned*.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: My main concern is that if a simpler approach is essentially enough to achieve a similar result. If this is adequately addressed, I am willing to increase my rating.
review_title: null

Meta-review: The paper introduces a novel framework for robotics that emphasizes the integration of cost-aware human forecasts into a model predictive control (MPC) planner. 

Instead of predicting the most likely human motion, ManiCast aims to understand how future human movements might impact the robot's planning costs. This approach enhances collaborative tasks by enabling smoother interactions, especially in close proximity. The authors evaluated the framework's efficiency through real-world tasks and introduces a new dataset on human-human collaborative manipulation in kitchen scenarios. This review seeks to provide feedback to enhance the paper's scientific presentation and address its specific elements.

The majority of the reviewers agreed on the merits and technical soundness of the paper, and they found the author response helpful and convincing. The reviewers unanimously recommend accepting the paper (though `Z1jy` forgot to change the rating in the system). I recommend accepting this paper. The authors are encouraged to address the remaining issues, such as "1) effectiveness of predicting the full upper-body marginal,  and 2) tasks being considered are not really close-proximity tasks".
Decision: Accept (Poster)