Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an approach for learning *backchannelling behaviors* for companion robots, an important aspect for cueing and signaling in multi-party interaction in general. Concretely, this paper presents a learned approach for deciding when to *nod* based on visual and speech/acoustic context from a human in conversation.

Critical to the proposed approach is intelligent application of data augmentation; obtaining backchanneling (nodding paired with videos of partner’s head pose, speech) data at scale is hard, and as such, naively training models results in robots with behaviors that are either overfit to speakers from the training set (and thus fail to generalize) or total collapse. By selectively masking and warping the input features (visual head poses, audio features) not only can we enrich the dataset, but we can learn better, more generalizable backchanneling behaviors.

The proposed model is simple, but effective (quite elegant!). Trained on data collected from 12 speakers primed with a diverse array of prompts, the resulting “virtual robot” experiment shows that not only is data augmentation critical to strong backchanneling performance (both qualitatively and quantitatively), but also out performs a strong, rule-based baseline. 

A possible flaw in this work is the lack of deployment on a real robot, instead doing all experimentation with a virtual robot in a web interface; due to the COVID-19 pandemic, I think this is fair, and makes a lot of sense. I appreciate that the paper presents a clear path for deploying and evaluating with actual physical robots, and I am convinced the results will hold.

strengths_and_weaknesses: In terms of strengths, the problem is well-motivated, the proposed solution is elegant, the “key insight” (data augmentation) is also motivated, and proven to be necessary, and in general, the proposed system works convincingly from both a qualitative and quantitative perspective.

As a weakness; it does seem that backchanneling is just a smaller component of a larger HRI system (e.g., for human-robot dialogue, or assistive robotics). I understand the need to evaluate this specific behavior alone (and from the prior work cited, it seems that this is an established problem), but I do wonder if things change in the context of a broader system (e.g., is nodding really that important when a robot is equipped with the ability to provide dense feedback in terms of actual physical motion, or speech).
summary_of_recommendation: The proposed method is well-motivated, well-constructed, and the evaluation is strong. It’s clear that data augmentation is important for getting generalizable behavior from small backchanneling datasets, and that the proposed approach is a good step in solving this task.
issues: [90 - 92] “Although data augmentation… it has not yet been applied to human-robot interaction” isn’t quite correct; several papers on human-robot interaction (e.g. learning from corrections, language instructions, etc.) all use data augmentation. Would temper this claim.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: - This work attempts to learn a policy of when a robot should nod, given the prosody and head orientation of a speaker
- The problem is of relevance because backchanneling is a crucial mechanism for maintaining the flow of conversation, and robots that are deployed as social agents will need such a capability
- The authors collect a dataset of conversations, and given annotations of when listeners nod, they train an LSTM on speech and head orientation data
- In order to combat the low generalizability of prior models to new speakers and provide sufficient data to train the model, the authors use data augmentation techniques such as random shifts and masks to prevent overfitting
- The authors's results show that the LSTM trained with data augmentation outperforms other baselines

strengths_and_weaknesses: **Strengths**

- The authors's motivation, formulation of the problem, and approach are straightforward and easy to follow
- The results in the paper are promising

**Scope for Improvement**

The biggest scope for improvement in this work lies in the presentation of the results:

- The Likert data could be better presented (show significance results in the plots!) and better analyzed (one should aggregate multiple items into a combined "scale"). For an excellent primer on how to approach Likert metrics, please see Schrum et al., HRI'20
- What was the loss function that was used for training the models? Additionally, was the loss used for early stopping (recommended) or accuracy (not recommended)? Clarification on this matter would be great
- (this is speculation) The F1 and accuracy scores are not that high (despite the promising results). Given the qualitative evaluation presented by the authors, might it be worthwhile to report an Intersection over Union-like metric instead (as is commonly used in computer vision)? If the authors wish to be truly rigorous, they can provide a correlation of the F1 metric with the IoU metric

summary_of_recommendation: Although the paper has weaknesses in its presentation of the results, they can be addressed by following some of the suggestions presented above. I am therefore, weakly in favour of accepting the paper into the technical program.

Post-Rebuttal Update:
I thank the authors for updating their paper and I find that it is more ready than before for publication. I am in favour of including this in the technical program.
issues: Barring the last point in the scope for improvement, I would like to see a better Likert analysis and some more clarification on the LSTM training.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper is concerned with ways of generating backchanneling behaviors for robots (social cues like nodding appropriately placed in the midst of a conversation with a human). Data-driven methods tend to overfit to the (scarce) training set and cannot adapt appropriately to new speakers, so the paper presents a data augmentation method for the input speech and visual features that improves backchanneling behaviors without collecting additional data. The data augmentation schema warps and masks portions of each human speaker’s input features.

The work diligently collected their own dataset of unscripted but structured 2-person conversations, and used both audio and visual features for training a model that predicts when the agent should or shouldn’t nod. The paper evaluates the data augmentation strategy on this dataset with positive results, and later conducts a user study with a virtual robot that nods whenever the trained model specified it should do so. 
strengths_and_weaknesses: I thought the data collection and experimental analysis were really well done, with thorough evaluation and discussion. Additionally, the writing was clear and enjoyable, and the work was well placed within existing literature.

Overall, I liked the paper but there are a few sticking points that concern me/could improve this work. First, the paper discusses backchanneling behaviors as a whole but in reality only nodding is explored. I think the results would be stronger if they showcased a range of backchanneling behaviors (smiling or “hmm”s as well); otherwise, the work essentially boils down to a nodding predictor, which is not novel enough, in my opinion.

Secondly, while Section 4 compares LSTM with and without data augmentation, I was surprised to see that the same comparison didn’t exist in the user study. Since the main contribution of the paper seems to be using data augmentation to improve backchanneling, it seems natural that the study should focus on the effect of data augmentation, and not the effect of different types of predictors.

Lastly, unless I am interpreting Fig. 6 wrong, the study results don’t show a significant difference between the learned and the rule based model. I would have liked to see more discussion for why this is the case, especially since the forced choice questions in Fig. 7 indicate a significant difference.

summary_of_recommendation: Although (mostly) diligently done, the experiments in the paper are limited and I am reluctant to accept the paper without experiments on more backchanneling behaviors. Moreover, I am not convinced that the data augmentation method presented has a significant impact on people’s perception of the robot in conversation.

UPDATE AFTER-REBUTTAL: I thank the authors for their response. I have updated my score to a weak accept.
issues: To improve the submission, I would suggest adding another couple backchanneling behaviors, reframing the paper to only focus on nodding, or making a strong argument for why the experiments presented are expected to generalize to other backchanneling behaviors. I also think rerunning the study to compare LSTMs with and without data augmentation would more directly study whether the proposed data augmentation is effective.
review_title: null

Meta-review: This paper introduces a technique for automatically learning social backchanneling, in this case specifically the timing of nodding behaviors, from observation of human-human interactions.  

Strengths:

The paper introduces a compelling learning approach which uses data augmentation techniques, such as random shifts and masks, to generalize the learned behaviors to new scenarios and speakers even from relatively little data.

The work is clearly motivated and well presented.

The experimental analysis is thorough and convincing.

Weaknesses:

Several technical details relating to the presentation of the algorithm and figures would improve the technical aspects of the paper.  These include treatment of Likert data and clarification of loss function.
Decision: Accept (Poster)