Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: - The authors formulate a computational model for a human's classification of the behaviour typified by a robot's trajectory
- In doing so, they are also able to formulate the inverse problem: a model to generate robot trajectories to best typify a desired behaviour
- In a simulation domain, the authors had AMT workers classify sets of robot trajectories. An exploratory factor analysis revealed three factors that the humans' evaluated the robot trajectories upon. These were the target behaviour types in a subsequent experiment
- The authors generated a model that was able to classify the likelihood of ratings of multiple people along the three factors (behaviours) for any given trajectory [as mentioned later, I am slightly confused on this point though]
- The authors use this model, and a hill-climbing search algorithm, to then generate trajectories in a new environment that try to satisfy a desired rating from humans.
- The authors find that they are able to predict participants' scores on the new trajectories at better than chance

strengths_and_weaknesses: **Strengths**

- I like the computational formulation of the problem. However, I commend the authors even more on their data-driven approach to the issue of getting the behaviour description objectives (the exploratory factor analysis)
- Qualitatively, many of the results presented by the authors make sense:
  - The images of the trajectories (probably cherry-picked to an extent?) would conform to the adjectives that the authors ascribe to them
  - The lower predicted KL-divergence of generated broken trajectories, when the suboptimality bound is high, also makes sense
  - The distributions of 4b are a good sign of the authors having accomplished part of what they set out to do
- The results from the user studies are compelling evidence of the efficacy of this work.

**Scope for Improvement**

- The figures are hard to parse and it takes a while to know what the main result depicted in them are. 4c is particularly difficult. Focusing on that one, I think the authors are trying to showcase three results here:
  - The ability of the model to provide a prediction of how humans might rate trajectories
  - The ability of their framework to elicit human ratings that conform to a desired set of behaviour ratings
  - The agreement of their models' predictions of human ratings to the actual ratings

As such, the plots for experiment 1 then serve a different purpose than those for experiment 2. Therefore I recommend the authors separate those and particularly focus on purpose 2 (above) for the plots from experiment 2

- Related to the above point, for experiment 2, I think it's more important to showcase the divergences of participant ratings from the _target_ distribution of ratings instead of with the models' predictions. I suspect, given the current plots, that this might even lead to a more interesting story.
- The authors couch their results in the context of hypotheses: however, they do not provide the details of any corresponding hypothesis tests (either Frequentist with p-values or Bayesian with ROPE / pd). As such, it's hard to say whether the hypotheses are supported (or the corresponding null hypotheses are rejected). I recommend the authors remove such language unless they perform and show the results of _quantitative_ hypothesis tests.
- The text mentions a suboptimality condition of 12x; yet in the appendix, it is plotted as 2^3. Which is correct?

summary_of_recommendation: Although the paper has some weaknesses, they can be addressed by improving the presentation of the evaluations, as mentioned above. If this is done, I'm in favour of including this paper in the technical program.

Post-Rebuttal Update:
I thank the authors for their clarifications and find that the appendices, particularly Appendix D, is very helpful. I am much more in favour of accepting this paper into the technical program, but I am curious to see what my fellow reviewers say.
issues: (See scope for improvement; above)
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper is studies the problem of inferring high-level behavioral attributes for robot motion plans. The work builds on a user study and psychology literature to to postulate a space of behaviors and features that are expected to inform behavioral attribution. Central to the approach is learning a multi-model distribution over attributes given a trajectory. The model is trained from crowd-sourced data and is used to synthesize trajectories that can balance motion cost and the expression of specified attributes. The model is validated using a user study. 
strengths_and_weaknesses: Strength 
- The problem addressed in this work is well motivated and clearly articulated. Authors present a simple but elegant solution for the problem of incorporating behavioral attributions to robot motion planning. 
- The experiments and user study is well thought out and rigorously conducted. 

Weaknesses
- The problem is confined to metric characteristics of the robot’s trajectory and the overall coverage attained by the planner in the environment. Intuitively, it appears that symbolic properties of the goal and map context would also play a role in behavioral attribution. For example, a robot moving around a glass vase quickly may be considered rushed/careless but the same may not be interpreted if the robot moves around other furniture. Richer symbolic attributes of the goal and the environment lie beyond the present approach. 
- The experimental evaluation does not explicitly compare against a baseline. Would recommend either comparing with an approach for related tasks such as legibility or state that the suitable baseline does not exist. 
summary_of_recommendation: The paper clearly articulates the technical gap and presents an interesting approach to learning a model for behavior attribution and then using the model to synthesize plans that balance cost-to-goal with its behavioral attribution. 
issues: - While reading the paper, I was less clear on how the trajectory data set was created. I understand that the trajectories were generated by varying the parameters of a conventional motion planner and were later annotated via crowd sourcing.  I would request more technical details. Alternatively, human turkers could have drawn trajectories on the map for specific behavioral attributes to more directly elucidate their mental model.  
- How extensible is the trajectory generation process to richer behavioral specification? For example, can the framework handle the case where a human specifies that a particular attribute “should" be present and another distinct attribute "should not” be present in the synthesized trajectory.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a method to not only build a predictive model of human attributions based on robot motions as has been done in prior work, but to use this model to generate trajectories with those attributes. The generated trajectories are constrained to a good enough task performance. To this end, human attributions from a Mechanical Turk study are used to probabilistically map attributions to robot trajectories in a coverage task. Trajectories are then generated to optimize the Kullback-Leibler divergence of the behavior attribution model relative to a desired behavioral score factor distribution. This optimization is subject to an inequality constraint on the task cost. The value of the inequality is a design parameter that dictates how suboptimally the robot can behave in service of optimizing the behavioral attributes. The generated trajectories were then entered into a separate Mechanical turk experiment assessing how well the generated trajectories were perceived by humans. The approach was somewhat successful in eliciting the intended human perceptions by the participants.
strengths_and_weaknesses: Strengths
This work goes beyond prior work for extracting human perception and attributions for robot motions and tries to integrate these models into the motion generation pipeline similar to generation of legible robot motion.

The experiments were well-designed. The authors were able to build a reasonable and understandable model of human attributions mapped to trajectories. The formulation of a cost for matching the probabilistic model could be valuable in other scenarios where one would generate trajectories with higher level behavioral attributes.

Weaknesses
A major weakness of the paper was the lack of detail in the optimization. The paper describes a hill-descending search in the space of trajectories, but why is this in the space of trajectories when section 4.1.2 says that the space of trajectories is `too large to explore directly’? It appears that the configuration space is already discretized into states. (This is implied by the comments on the state transition costs.) I think that what the paper means is that the space of trajectories is too large to map attributions to directly, since the described optimization is clearly exploring over \xi not \phi. I understand the naïve action level choices would be included, but more detail is needed on how to choose motion templates and encode them in the optimizations. In its current form, an average graduate student would be daunted by the task of replicating this or applying it to a robot arm without the relevant details.

I would also like to know why the factors are between -3 and 3. It seems like an arbitrary choice, so why this one in particular what are the consequences of making this domain larger or smaller?

Figure/Table Issues:
There are tables listed under Figure 2, 3, and 4. It seems bad practice to include tables as subfigures. They should just be tables using the \tabular environment. The paper also refers to fig. 2a as table 2a, which makes it inconsistent with itself. This practice also has made the text in fig./table 2a very small and hard to read. Fig/table 3a seems to have all the same data as what is contained in the plot in fig. 3b, so why not just drop the table altogether and leave the plot. Finally, figure 4 is hard to follow. I did not initially realize that it was two column all the way down until it hits the table which breaks the two column format above it.

summary_of_recommendation: The paper demonstrates a comprehensive approach to generating robot trajectories that exhibit behavioral attributes from collecting data, building a model, and exploiting that model to generate trajectories. The generation of trajectories is a significant advance over past work that only classified trajectories. However, some significant details about the optimization process is missing and the presentation of graphical and tabular data makes the paper somewhat difficult to digest. 

Post Rebuttal Update: The author's have added significant clarifications and addressed most of the concerns of myself and the other reviewers. For this reason, I have updated my score to weak accept.
issues: A revision should make the tables into Tables separate from any plots or graphs. The revisions should also clarify the reasoning for the low dimensional trajectory representation and provide more details on the optimal search for trajectories (at least in an appendix).
review_title: null

Meta-review: This paper addresses the issue of inferring high-level behavioral attributes for robot motion planning, and presents a framework to involve human attributions from a Mechanical Turk study and then used to evalute and generate robot motion trajectories in a coverage task. 

I agree with reviewers that the problem this paper addressed is well motivated and formulated, and the experiments is also well-designed, it's good to see the user study takes supports from psychology literatures. 

While the presentation on both the details of the methods and analysis of expreimental results seems a bit weak, as reviewers pointed out. I agree with these points which might be valuable in improving the quality of the paper, mainly including:

1. The lack of details in the optimization process. 
2. The experimental evaluation and analysis needs to be greatly improved, like explicitly compare against a baseline, carefully contrast the model with human, the results of quantitative hypothesis tests, etc.
3. Many more minor points as reviewers mentioned.

More added: 
I would like to thank authors for your carefully and detailed responses to comments raised by reviewers.
Although there are still rooms for this research to be improved, the updated version seems better, and the scores of this paper was improved after rebuttle. 
I thus recommend this paper to be accepted.
Decision: Accept (Poster)