Learning Backchanneling Behaviors for a Social Robot via Data Augmentation from Human-Human Conversations
Authors: Michael Murray
Nick Walker
Amal Nanavati
Patricia Alves-Oliveira
Nikita Filippov
Allison Sauppe
Bilge Mutlu
Maya Cakmak
Venue: CoRL2021 Poster (2021)
Keywords: social robots
backchanneling
data augmentation

Abstract: Backchanneling behaviors on a robot, such as nodding, can make talking to a robot feel more natural and engaging by giving a sense that the robot is actively listening. For backchanneling to be effective, it is important that the timing of such cues is appropriate given the humans' conversational behaviors. Recent progress has shown that these behaviors can be learned from datasets of human-human conversations. However, recent data-driven methods tend to overfit to the human speakers that are seen in training data and fail to generalize well to previously unseen speakers. In this paper, we explore the use of data augmentation for effective nodding behavior in a robot. We show that, by augmenting the input speech and visual features, we can produce data-driven models that are more robust to unseen features without collecting additional data. We analyze the efficacy of data-driven backchanneling in a realistic human-robot conversational setting with a user study, showing that users perceived the data-driven model to be better at listening as compared to rule-based and random baselines.

Learning Backchanneling Behaviors for Social Robot via Data Augmentation from Human-Hunan Conversations
Michael Murray', Nick Walker', Amal Nanavati', Patrícia Alves-Oliveira', Nikita Filippov', Allison Sauppé", Bilge Mutlu?, and Maya Cakmak'
'Paul G. Allen School of Computer Science and Engineering, University of Washington
DDeparmeent of Computer Sciences, University of Wisconsin-Maddion (mmurr, nswalker, amain, patri, nikitaf, meakmak] @cs. washington.edu fasauppe, bilge} @cs. @ed.sc.es.
A Abstract: Backchanneling behaviors robot, such nodding, can make talk- ing to a robot feel more natural and engaging by giving a sense that the robot is actively listening. For backchanneling to be effective, it is important that the timing of such cues is appropriate given the humans' conversational behaviors. Recent progress has shown that these behaviors can be learned from datasets of human-humaa conversations. However, recent data-driven methods tend to overfit to the human speakers that seen in training data and fail generalize well to previously unseen speakers. In this paper, we explore the use of data augmen- tation for effective nodding behavior in robot. We show that, by augmenting the input speech and visual features, we can produce data-driven models that are more robust unseen features without collecting additional data. We analyze the efficacy of data-driven backchanneling in a realistic human- conversational setting with user study, showing that users perceived the data-driven model to be better at listening as compared to rule-based and random baselines.
Keywords: social robots, backchanneling, data augmentation
Introduction
Social robots that can communicate with people naturally and effectively serve wide variety of important daily life contexts, from companion robots for older adults [1], robots that help children with social-emotional development [2], or for robots that support teens to cope with stress and mental health challenges [3]. Verbal communication is the heart of many such applications of social robots. If social robots are to become part of daily life, it is critical that talking to robot feels natural and engaging. One way robots can remain active a conversation while the human is speaking is through backchanneling. Backchannels, such as smiles, nods, short utterances like "hmm", cues that humans provide to each other during a conversation to indicate to the speaker that they are still actively listening and understanding [4]. Backchannels make conversation feel smooth and natural [5]. When performed by a robot or virtual agent, backchannel responses have been shown to give users increased perception of attentive listening, an increased feeling of rapport, and desire to continue interacting [6. 7].
Research into the use of backchannels in human communication has found speaker cues that can elicit backchannel responses, such as periods of low pitch [8], rapidly changing intonations [9], and certain patterns of speaker gaze [10]. Many early approaches to programming backchannel behavior have been based on rules involving these cues [8. 11, 12]. Others have taken data-driven approaches, learning backchanneling behavior from datasets of numan-human conversations. Over time, several data-driven approaches have been explored including decision trees [13, 14], sequential probabilistic models [15, 16, 17]. and more recently, deep neural networks [18, 19, 20, 21]. Deep neural networks have shown promising performance in predicting when to backchannel, however, these approaches
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

typically suffer from the generalization problem in which the learned backchanneling policies tend overfit human speakers that are seen in training data and fail generalize well to previously unseen speakers. One of the major contriburing factors to this problem is data scarcity. Learning backchanneling behavior requires annotated video datasets of human-human conversations which are expensive and time-consuming to collect. As a result, the datasets used to train backchanneling algorithms typically have a small number of human speakers. For example, the Interactive Emotional Dyadic Motion Capture Database (IEMOCAP) [22]. which is the most commonly used dataset for learning backchanneling behaviors, consists of conversations from just 10 unique speakers. Due to the challenges of collecting sufficiently large datasets, it is crucial problem to efficiently learn more generalized policy for backchanneling without collecting additional data.
Data augmentation is a technique that has contributed to significant improvements in performance under conditions of data scarcity in domains such as image classification [23, 24, 25, 26] and speech recognition [27, 28, 29, 30]. With data augmentation, we produce new datapoints by augmenting existing datapoints, for example by warping the original input. This makes the training dataset effectively larger without collecting additional data, as multiple augmented versions of single example can be used to train a model.
In this paper, we present a data augmentation scheme to learn more robust backchanneling behaviors for a social robot from human -umman conversation data. We introduce method that warps and masks portions of each human speaker's ' input features. Our experiments show that the technique results in improved performance on the metrics of accuracy and Fl-score. Finally, we conducted a user study to examine the efficacy of a data-driven model for online human-robot interaction. As recent works have evaluated their models using pre-collected datasets, we contribute- to our knowledge- the first user study to examine a neural network based approach to backchanneling. Our results showed that, compared to rule-based and random baselines, the neural network approach elicits more natural and preferable interactions with listening robot.
Related Work
Modeling Human Backchannel Behavior The description, modeling, and understanding of hu- man backchanneling behavior has been a subject of research since early comprehensive studies of non-verbal behavior in the 1970s [31, 4]. Early models sought to describe relationships between backchannel behaviors and speaker cues spanning gesticulation, gaze, syntactic markers and par- alanguage [5, 32]. The ability to automatically detect speaker cues, like periods of low pitch or pauses, led to the first applications of rule-based backchanneling systems [8, 33, 11, 12]. The use of data driven methods such as decision trees [13, 14], sequential probabilistic models [15, 16, 17], and neural networks [18, 34, 19, 20, 21], has reduced the dependence on hand-designed rules.
Embodied Backchanneling Agents The impact of backchanneling behaviors on interactions with virtual animated agents has been extensively studied [35, 36, 37]. Despite promising applications in educational or eldercare robotics, there are relatively few systems that have been designed to work with a physical robot. Lala et al. [38] evaluated an integrated backchanneling dialogue agent on ERICA, a realistic humanoid robot, finding that their approach was perceived as more natural and empathetic than rule-based alternatives. Inoue et al. [39] compared user responses the system with those generated by a human operator and saw that their system matched the human control in basic listening skills but fell short in ratings for complex attributes like empathy. Park et al. [7] showed that children perceived toy robot listener using rules-basee backchannel generation approach as more attentive compared a random baseline.
Data Augmentation Data augmentation has been applied a wide variety of problems to reduce overfitting in neural networks. Image augmentations, such as horizontal flips and random cropping or translations, are commonly used in image classification and detection models [23, 24, 25, 26]. Krizhevsky et al. [24] notably achieved state-of oftee-of-the-att results on image classification and showed that their augmentations reduced the error rate of their model by over 1%. Additionally, speech augmentations like speed perturbation, pitch adjustments, and additional noise are commonly used for automatic speech recognition [27, 28, 29, 30]. Recently, Park et al. [29] introduced SpecAug- ment, which achieves state-of-the-art performance on several speech recognition tasks by masking

---

FO
Nod
MFCC
LSTM
No
Nod
Head
(From top to bottom) Sequences of FO, MFCC, and head pose features extracted from the raw audio and video inputs. These features are concatenated and fed into an LSTM network which outputs a nod/no-nod decision for the robot.
Figure I:
the mel-spectogram representation of speech. Although data augmentation shows promise in many domains, it has not yet been applied to robot backchanneling, which we explore in this work.
3 Method
The goal of this work is to improve listening behaviors in robot during a conversation with human. Following the recent work of Ruede et al. [19], we model our policy as a recurrent neural network that maps sequences of input from the speaker output actions on the robot. In this paper, we focus on nodding as a backchannelling modality because of it's prevalence and utility in human-human conversations [40, 41, 42], but we expect that the techniques we present will generalize other backchanneling modalities. We aim to enable accurate prediction while maintaining the feasibility of deploying the model on computationally constrained robotics platforms.
Acoustic Features To summarize the speaker's acoustic features we extract the fundamental fre- quency (FO) and 14th order mel-frequency cepstrum coefficients (MFCCs). FO is an estimation of the speaker's pitch and the MFCCs represent the short-term power spectrum of the sound. Both fea- tures are commonly used in speech processing and can be calculated efficiently without introducing significant latency. We extract the features over 20ms windows and concatenate them to produce single acoustic feature representation. This configuration significantly compresses the audio signal while maintaining the ability distinguish prosodic cues and phonemes.
Visual Features As a summary visual feature we extract the speaker's head pose, as an euler angle, using a fast facial landmark detector [43] and estimating the correspondence of points to a 3D head model by solving the perspective-n-pont problem. This was informed by work showing gaze as cue for backchannels [10]. To moderate noise inherent in this pipeline, we smooth the angle using Kalman filter.
Model We model the backchanneling policy as a long short term memory (LSTM) recurrent neural network [44], which is commonly used for modeling sequential data. At each step, the network receives the last two seconds of features as input and produces a binary output (Figure I).
we mentioned above, data scarcity is an issue when training neural net- works for backchanneling tasks. Therefore, given training dataset of the aforementioned input features and binary output, we propose using data augmentation add one new example for every example in the dataset.
Data Augmentation
Following Park al. [29], we augment the MFCC features by warping them across time, masking blocks of utterances over time, and masking blocks of consecutive frequency channels. An example of these augmentations can be seen in Figure We augment the FO features by warping them across time and masking blocks of utterances over time. Finally, we augment the visual summary head pose feature by randomly warping the head pose in space and time, and by masking random blocks of head poses over time. These augmentations are intended to improve the model's robustness to partial loss of information (frequency, segments of speech, or head pose information) and deformations across time.
3

---

Figure 2: Spectogram visualization of an MFCC feature (left). And the same feature with warping and masking augmentation applied (right).
Data Collection and Systematic Evaluation
To evaluate our data-driven backchanneling models, gathered dataset of human non-verbal backchanneling behaviors. While we considered existing datasets such as IEMOCAP [22] and CCDb [45], no existing dataset met our criteria of being both unscripted and structured, with distinet speaker and listener roles. We believe these criteria will promote natural head nods that are more appropriate for listening robot. This lead
collect our own dataset meet these criteria.
Using a custom peer-to-pent video chat tool, we recorded audio and video of conversational interac- tions between two people. Each interaction was structured around prompt, with one person tasked to respond to the prompt while the other person was listening and performing natural backchannels. Every session was composed of researcher and participant selected from that researcher's social network. After obtaining consent, participants were given instructions on how to position themselves for optimal data collection-in well-lit room, with bright lights, and with their head in the middle of the frame. The researcher and participant then engaged in 10 interactions where they took turns prompting each other and listening to the other talk for 1-2 minutes. The prompts consisted of two random prompts from each of the following categories:
Happy (e.g. "Tell me about your most fun memory from last summer")
Neutral (c.g. "Tell me about your day yesterday")
Teaching (e.g. "Teach me how to cook your favorite dish")
Frustrated (e.g. "Tell me about pet peeve of yours")
Practice Talk (e.g. "You are giving a speech at your best friend, Sam's, wedding tomorrow. Practice that speech front of me").
These categories were chosen reflect the general scenarios in which envision users speaking to an actively listening social robot. Accounting for interactions missing due to recording issues, collected total of 84 interactions from 12 participants (9 male, female), 3 of whom were researchers. Interactions lasted 97.2 seconds on average (SD=31.2).
4.1 Annotation
We annotated listener nod behavior using a semi-automatic eye-position based scheme. The left eye keypoint position was extracted across the duration the video using [43]. The range of this raw pixel position then normalized to be between 0 and 1, and smoothed. The resulting signal was then processed to detect peaks and valleys. Portions of the signal with or more peaks or valleys than 50ms between them were then checked and adjusted manually using ANVIL [46]. After annotation was complete, an average 21.5% of per-interaction listener time was labeled as nodding (SD=10.8).
with
marked candidate nod windows. These candidates
4.2 Training
To train the RNNs we first split the dataset up into examples and labels where each example is a sequence of features extracted over the past 2 seconds and each label indicates whether the human subject nodded at that point in time. Because our dataset contains many more negative labels than positive labels, we account for the imbalance by assigning each class weight that inversely proportional its frequency: number of negative examples and n. the number of positive examples). We then split the dataset
_ (where n. is the aーt=_ w_ т=_ a and w4 R=
by individual speakers, and use 6-fold cross-validation to train the model. In each fold, the examples

---

Table 1:
Comparison of LSTM, LSTM with data augmentation, and rule-based baseline. Reported
numbers are averages across all validation folds.
Training (Seen Speakers)
Validation (Unseen Speakers)
Method
Acc.
Prec.
Rec.
F1
Acc.
Prec.
Rec.
F1
Rule-based (Ward) [11]
0.60
0.20
0.14
0.23
LSTM [20]
0.80
0.31
0.29
0.60
0.62
0.18
0.16
0.24
LSTM Augmentation
0.80
0.29
0.25
0.57
0.64
0.23
0.24
0.29
from two speakers are held out for validation and the remaining examples are used for training. We train the model by optimizing cross-entropy loss and perform early-stopping according to the loss.
4.3 Results
We compare the LSTM-based approach, with and without data augmentation, with a rule-based baseline. We chose Ward [11] as our baseline because was the highest performing rule-based approach on our dataset. Table shows the performance of the three models in terms of accuracy, precision, recall, and Fcorc. Precision the probability that predicted head nods match the actual annotated human behavior, while recall is the probability that an annotated human head nod is predicted by our model. The Fi-score, which is the harmonic mean of precision and recall, is typically used as the key evaluation metric for imbalanaced datasets like ours [47].
We observe that both LSTM models outperform the rule-based model and the LSTM model with data augmentation achieves the highest accuracy, precision, recall, and F1 Fi-score among all of the approaches. Figure shows a plot of the training and validation -scccesss across training epochs to further illustrate this reduction in the performance gap between training and validation. Additionally, Figure we present examples of qualitative improvements from two unseen speakers.
0.55
Thining
1.TT Validaiion Fl
0.50
LSTM-Asg Traising Fl
0.45
ISM--Aa
sore
0.15
15
Epochs
Figure 3: Fl-score over 50 epochs for selected training fold. Data augmentation achieves higher performance on the validation due to less overfitting to the training set.
5 User Evaluation
While the systematic evaluation demonstrated the capacity for data-driven models to accurately predict backchanneling behaviors in human-human interactions, it does not capture the qualitative
LSTM
LSTM
Aug
LSTM-Aug
Figure 4: Visual timelines indicating nods over the duration of two example sessions that were both unseen during training. On the left, we can see that the LSTM nods too infrequently and the more robust data augmentation method is closer the ground truth. the right, the LSTM nods too frequently and again the data augmentation method looks closer to the ground truth.
5

---

Talle pour Tecott
Figure 5: As part of the user study, participants talked to a virtual robot through an online in- terface, responding to a specific prompt (top). The robot nods when the model specifies.
Figure 6: While the user study employed a vir- tual robot to adapt to the COVID-19 pandemic, we also demonstrate that our method can be de- ployed on a physical Mayfield Kuri robot.
experience of interacting with robot that uses the models produce nodding behaviors. We investigated the effectiveness of these models in generating backchanneling behaviors on robot through a user study in which participants talked to a virtual robot. We used a virtual robot to adapt to the COVID-19 pandemic.
5.1 Study Design and Procedure
Our study involved participants talking to a virtual Mayfield Kuri robot that uses either a learned model. rule-based model, or random baseline to determine when to nod. Specifically, the three conditions in our study were:
Learned uses the best performing learned model, the LSTM model with data augmentation (Sec. 3), trained the whole dataset. The model takes in audio and visual features, and the robot produces a nod whenever the model assigns a probability above 50% to the nod class.
Ward [11] the aforementioned rule-based baseline. The model uses a rolling window to approximate the distribution of the speaker's pitch and nods upon detection of: (1) region of pitch less than the -ercepeeee pitch level (2) continuing for least 110 mil- liseconds (3) coming after at least 700 milliseconds of speech (4) no backchannels in the preceding 800 milliseconds (5) after 700 millisccond wait.
Random produces nod with nod duration sampled from normal distribution fit the du- rations in our collected dataset. Wait times between nods are sampled from an exponential distribution fit the wait times our collected dataset.
To allow for direct comparisons of the three conditions, we performed within suthin-participants study in which participants interacted with all three conditions. Each condition was presented as different robot, named Robot 1-3. The order of conditions was counter-balanced lection procedure, this evaluation took place through the same custom online video communication interface. Participants were shown a simulated robot in a living space, intended to mimic a webcam display of physical robot (see Figure 5). The robot exhibited naturalistic blinking and fidgeting idle bchavior, and nodded when the model specified.
Mirroring our data col-
After reading instructions, participants went through a familiarization phase where they talked to robot that had the idle behavior but did not nod. The interaction started with question prompt from the robot. Participants were instructed to talk for 1-2 minutes responding to the prompt. The time elapsed was displayed on the screen for guidance and participants could move on at any point by pressing the "Complete session" button. After familiarization, they were taken to the study, where they interacted with each Frustrated category of the data collection (Sec. 4) and the order of the prompts was kept constant. After the interaction with each model, participants were taken questionnaire that asked them about their perceptions of the robot and its listening and nodding behavior (Table 2).
three models. The prompts for the study were taken from the

---

Table 2: Survey questions from the evaluation questionnaire. The Likert statements required par- ticipants to choose their level of agreement with options ranging from 1-5 (strongly disagree to strongly agree). Frequency questions had continuous slider between 0% of the time (never) and 100% of the time (always). Open ended questions were free text entry. Participants responded to this questionnaire after each condition.
Likert Statements
Frequency Questions
The robot listening carefully.
How often did the robot nod its head at an inap- propriate time?
The robot's behavior was natural.
The robot cared about what was saying.
How often did the robot miss head nod opportuni- ties?
The robot did not care about what was saying.
The robot was not listening me.
The robot understood what was saying.
Open Ended Questions
a close connection with the robot.
Talking with the robot was similar to talking with a friend.
What was it like to speak with the robot? Describe positive and negative aspects of the robot's behavior Describe any similaritics or differences between talking to the robot and talking to friend
Talking with the robot similar talking with stranger.
Talking with the robot was comforting.
After interacting with all three models and completing all three questionnaires, participants were asked to select the robot that seemed the most natural, that they enjoyed speaking the most, that nodded the most, and that nodded the least. Finally, they were asked demographic questions. Participants for this study a greater than 95% approval rate and who were in the U.S., and were compensated $3 for their participation in this approximately 25 minute user study.
recruited on Amazon Mechanical Turk, limited to users who had
5.1.1 Results
Our study was completed by 102 participants (53 male, 46 female, 3 other; ages 18-75). To group the Likert items performed exploratory factor analysis with promax rotation, yielding two factors shown in the appendix Table 7.1. We interpret the first factor as measuring the perceived nodding and listening skill of the robot. We interpret the second factor measuring what extent talking with the robot elicited feelings of comfort or closeness. The factors show good reliability with 0.88 and 0.92 respectively. We group the continuous frequency questions into single scale measuring the perceived frequency of mistakes. Figure summarizes participant responses on these scales. Figure shows the distribution of participant preferences in forced-choice questions.
We used repeated measures ANOVA to measure the statistical significance of responses on the Likert and frequency scales. Using a significance threshold of < 0.05 for all statistical tests, there was a significant effect on the perceived nodding skill (2 (2,20 = 9.86, p < 0.001] and the perceived frequency of mistakes [2,,, 204) 4.56, p < 0.01]. Post hoc comparisons using the Tukey HSD test indicated that the mean score of the learned model significantly higher than the random model
Percaived Nodking and Listaning Skill
Comfort Level
Perceived Frequency of Mistakes
140
13
Random Model
20
Learned Model
Rule Based Model
Figure 7: Box plots of participant responses to Likert scale and frequency questions in the user study (see Table 2 for exact questions). Whiskers represent 1.5 times inter-quartile range and the red line indicates the median value.

---

Felt Most Heard
Enjoyed Most
Enjoyed Least
Best Listener
40-
Random Model
30-
30
30
Learned Model
20
20-
Rule Based Modell
10-
10
I'm Not Sure
Figure 8: Histograms of participant responses to forced-choice questions asked after interacting with all conditions.
on both scales. The learned model also outperforms the rule-based model, although that difference is not significant.
User responses to the forced choice questions indicate a strong preference for the learned and rule- based models over the random model. The learned model was the most common choice for "Which robot made me felt most heard". "Which robot did you enjoy speaking to the most?", and "Which robot was the best listener?". Additionally, it was the least common choice for "Which robot did you enjoy speaking to the least?". While there was no significant difference between the learned and rule- based models on our Likert and frequency scales, there is a clear preference for the learned model over the rule-based model in the forced choice responses. This could indicate that the differences are too small to appear on the scales but still large enough to result in a clear preference or could indicate that it difficult for participants express a scale the differences they noticed when interacting with the virtual robot.
Some participants noticed the timing of nods, e.g., one said the learned model appeared "better programmed to nod at more apprepriate intervals"; another said the rule-based model would "nod at more appropriate times and didn't I miss as many opportunities to nod". While some participants were specific about the reasoning for their choice, as in the above quotes, others could not articulate behavioral reasons but rather answered based on how they felt after the interaction; e.g., comments about the learned model included that it "seemed natural and normal", "felt like he was really listening", and "seemed more life like.". Some users stated that they could not discern a difference; e.g. one user stated that there was "no detectable difference" and another stated "it seemed like this robot had better timing when it came nodding than before, but I'm not sure if that is actually the case or not". Some users expressed optimism about the utility of a backchanneling robot; e.g. one user said "I would use this to practice job interviews" and another suggested to "make these robots some kind of therapy app". Other participants felt that the experience of talking to the robot was negative, highlighting important difficulties of social robot design; e.g. one person stated that "Speaking to the robot fell dehumanizing.".
6 Discussion
In this paper, we apply data augmentation to the task of learning non-verbal backchanneling behav- iors from human-human interactions. Our experiments show that our data augmentation technique reduces overfitting and improves backchanneling prediction performance for unseen speakers. Data augmentation is not the only technique that has been developed to reduce overfitting, and an excit- ing direction for future work to explore additional techniques such as pre-training and multi-task learning. Combining these techniques with reinforcement learning as in Hussain et al. [21] is another exciting future direction.
The results of our user study indicate that user perception of data-driven backchanneling behaviors positive, with the data-driven approach being preferred in foreed choice questions and eliciting nodding behavior that appears significantly more natural than a random baseline. Due to COVID- 19 limitations our user study was conducted online with a virtual robot so we aim to repeat our evaluation with a physical robot in future work. Although our evaluation involved a virtual robot, the learned model was developed to work the same way on physical Kuri robots using on-board sensors and head nods. Future work could extend our approach to provide richer backchanneling with additional outputs like robot facial expressions and vocalizations. Because our work is focused on augmenting the input features, we expect that the techniques we have presented will generalize well to other backchanneling modalities. Further investigation of backchanneling within the context of larger human-roha robot interaction systems is another important direction for future work.
8

---

Acknowledgments
This work supported by the National Science Foundation, collaborative awards IIS-1924435 and IIS- NSS-1925043 "Program Verification and Synthesis for Collaborative Robots."
References
[1] Fasola and M. J. Mataric. Using socially assistive human- robot interaction to motivate physical exercise for older adults. Proceedings of the IEEE, 100(8):2512 01-21222226.. 2012.
[2] H. W. Park, R. Rosenberg- Kima, M. Rosenberg, G. Gordon, and C. Breazeal. Growing growth mindset with social robot peer. In Proceedings the 2017 ACM/IEEE International Confer- ence on Human-Robot Interaction, pages 137-145, 2017.
[3] E. A. Björling, W. M. Xu, M. E. Cabrera, and M. Cakmak. The effect of interaction and design participation on teenagers' attitudes towards social robots. In 2019 28th IEEE International Conference Robot and Human Interactive Communication (RO-MAN), pages 1-7. IEEE, 2019.
[4] V. H. Yngve. On getting a word in edgewise. In CLS-70, pages 567-577. University of Chicago, 1970.
[5] S. Duncan. Some signals and rules for taking speaking turns in conversations. Journal of personality and social psychology, 23(2):283, 1972.
[6] J. Gratch, A. Okhmatovskaia, F. Lamothe, S. Marsella, M. Morales, R. Werf, and L.- Morency. Virtual rapport. pages 14 14-27, 08 2006. ISBN 978-3-540-37593-7, doi: 1007/ 1821830
[7] H. W. Park, M. Gelsomini, J. Lee, and C. Breazeal. Telling stories robots: The effect of backchanneling a child's storytelling. pages 100-108, 03 2017. do::10.1145/299982. 3020245.
[8] N. Ward. Using prosodic clues to decide when to produce back-channel utterances. In Proceed- ing of Fourth International Conference on Spoken Language Processing. ICSLP'96, '96, volume 3, pages 1728-1731. IEEE, 1996.
[9] A. Gravano and J. Hirschberg. Backchannel-inviting cues in task-oriented dialogue. In Tenth Annual Conference of the International Speech Communication Association, 2009.
[10] A. Hjalmarsson and C. Oertel. Gaze direction back-channel inviting in dialogue. In IVA 2012 workshop on realtime conversational virtual agents, volume 9, Citeseer, 2012.
[11] N. Ward and W. Tsukahara. Prosodic features which back-channé responses in english and japanese. Journal of pragmatics, 32(8): 1177-117771 2000.
[12] K. P. Truong, R. Poppe, and D. Heylen. A rule-based backchannel prediction model using pitch and pause information. In Eleventh Annual Conference of the International Speech Com- munication Association, 2010.
[13] M. Takeuchi, Kitaoka, and Nakagawa. Timing detection for realtime dialog systems using prosodic and linguistic information. 01 2004.
[14] R. Nishimura, Kitaoka, and S. Nakagawa. A spoken dialog system for chat-like conver- sations considering response timing. In V. Matousek and P. Mautner, editors, Text, Speech and Dialogue, 10th International Conference, TSD 2007, Pilsen, Czech Republic, September 3-7, 2007, Proceedings, volume 4629 of Lecture Notes in Computer Science, pages 599-606. Springer, 2007. 1001:0111007978---5400000 7_77. URL /ttps://ddi..rgg/ 118-3-540-74628-7.12
[15] I..P. Morency, de Kok, and Gratch. A probabilistic multimodal approach predicting listener backchannels. Autonomous Agents and Multi-Agen Systems, 20(1):70-84, 2010.

---

[16] I. de Kok, D. Ozkan, D. Heylen, and L.-P. Morency. Learning and evaluating response predic- tion models using parallel listener consensus. In International Conference on Multimodal In- terfaces and the Workshop on Machine Learning for Multimodal Interaction, ICMI-MLMI '10, New York, NY, USA, 2010. Association for Computing Machinery. ISBN 3781450304146 doi:10.1 41/4////199,,./ 1999908. URL https: .arg 10. 1145/1891903 1891908.
[17] L. Huang, L.-P. Morency, and J. Gratch. Parasocial consensus sampling: Combining multiple perspectives to learn virtual human behavior. In Proceedings of the 9th International Con- ference on Autonomous Agents and Multiagent Systems: Volume 1- Volume 1, AAMAS '10, page 1265-1272, Richland, SC, 2010. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 97809826571199
[18] M. Mueller, D. Leuschner, L. Briem, M. Schmidt, K. Kilgour, S. Stueker, and A. Waibel. Using neural networks for data-driven backchannel prediction: A survey on input features and train- ing techniques. In M. Kurosu, editor, Human- Computtr Interaction: Interaction Technologies, pages 329-340, Cham, 2015. Springer International Publishing. ISBN 978-3-319-208 10966666...666...
[19] R. Ruede, M. Müller, S. Stüker, and A. Waibel. Yeah, Right, Uh- Uuh: A Deep Learn- ing Backchannel Predictor, pages 247-258. Springer International Publishing, Cham, 2019. ISBN 978-3-319-92108-22
25. URL btttss:/ddo..om
10.
[20] B. B. Türker, E. Erzin, Y. Yemez, and T. M. Sezgin. Audio-visud prediction of head-nod and turn-taking events in dyadic interactions. In B. Yegnanarayana, editor, Interspeech 2018, 19th Annual Conference of the International Speech Communication Association, Hyder- abad, India, 2-6 September 2018, pages 1741-1745. ISCA, 2018. Noi:10.21437/nnerssppp 2018-2215. URL htt.st:::: htttps:///o..... org/o0./////// 10. 21437 / IInteeeeeeeeeeheeeeeeeeeeeeeeeee 2018-2215.
[21] N. Hussain, E. Erzin, T. M. Sezgin, and Y. Yemez. Speech driven backchannel generation using deep nentwwork for enhancing engagement in human-robot interaction. In G. Kubin and Z. Kacic, editors, Interspeech 2019, 20th Annual Conference of the International Speech Com- munication Association, Graz, Austria, 15-19 September 2019, pages 4449-4449. ISCA, 2019. 200..2144///nnessecch.2229299 2019-2521.
URL atttps://dol /ttp://///ooooo//////////// 21437/Inneetee IIterrpeerrr...
[22] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower Provost, S. Kim, J. Chang, Lee, and S. Narayanan. lemocap: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42:335-359, 12 2008. 10.:10..000/// 10.1000///057900000 9076-6.
[23] Y. Simard, Steinkraus, J. C. Platt, et al. Best practices for convolutional neural networks applied to visual document analysis. In Icdar, volume 3. Citeseer, 2003.
[24] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
[25] S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint 165.::160..07146 2016.
[26] R. Girshick, I. Radosavovic, G. Gkioxari, P. Dollár, and K. He. Detectron. http///acbbokkeeeaaareaattttteet 2018.
[27] T. Ko, V. Peddinti, D. Povey, and S. Khudanpur. Audio augmentation for speech recognition. In INTERSPEECH, 2015.
[28] L. Tóth, G. Kováes, and D. V. Compernolle. A perceptually inspired data augmentation method for noise robust enn acoustic models. In SPECOM, 2018.
[29] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q V. Le. Specaug- ment: A simple data augmentation method for automatic speech recognition. arXiv preprint 1XXvv1904.008799 2019.
[30] J. Xu, X. Tan, Y. Ren, T. Qin, Li, S. Zhao, and T.-Y. Liu. Lrspeech: Extremely low-resource speech synthesis and recognition. Proceedings of the 26th ACM SIGKDD International Con- ference on Knowledge Discovery Data Mining, 2020.
10

---

[31] P. Ekman and W. V. Friesen. The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica, 1(1), 1969. 150.155///mmm.99.....
[32] S. Duncan Jr. On the structure of speaker-auditon interaction during speaking turns. Language in society, pages 161-180, 1974.
[33] Y. Okato, K. Kato, M. Kamamoto, and S. Itahashi. Insertion of interjectory response based on prosodic information. In Proceedings of /VTTA '96. Workshop on Interactive Voice Technology for Telecommunications Applications, pages 85-88, 1996.
[34] R. Ruede, M. Müller, S. Stüker, and A. Waibel. Enhancing backchannel prediction using word embeddings. F. Lacerda, editor, Interspeech 2017, 18th Annual Conference of the Interna- tional Speech Communication Association, Stockholm, Sweden, August 20-24, 2017, pages 879-883, ISCA, 2017. URL http:://ww. isca ipw.se. org/ aaaaarcciiveeeeeeeeeee org./rrchrre//mtrrpee.... 2017/abstracts11666 html.
[35] M. Goswami, M. Manuja, and M. Leekha. Towards social & engaging peer learning: Predict- ing backchanneling and disengagement in children, 2020.
[36] A. Andriella, R. Huertas-Gareía, S. Forgas-Coll, C. Torras, and G. Alenyà. Discovering so- ciable: Using conceptual model to evaluate the legibility and effectiveness of backehan- nel cues in an entertainment scenario. In 2020 29th IEEE International Conference Robot and Human Interactive Communication (RO-MAN), pages 752-759, Aug 2020. doi: 10.1109/RO- SAN.OOAMAA77096222000222222
[37] G. Aist and J. Mostow. When speech input is not an afterthought: A reading tutor that listens. In Workshop on Perceptual User Interfaces, 1997.
[38] D. Lala, P. Milhorat, K. Inoue, M. Ishida, K. Takanashi, and T. Kawahara. Attentive listening system with backchanneling, response generation and flexible turn-taking. In Proceedings of the 18th Anmual SIGdial Meeting on Discourse and Dialogue, pages 127-136, Saarbrücken, Germany, Aug. 2017. Association for Computational Linguistics. doi:10. 1663/v11117-5516. URL https: / /www aclweb. org/ anthology/W17--5116
Linguittsii
[39] K. Inoue, D. Lala, K. Yamamoto, S. Nakamura, K. Takanashi, and T. Kawahara. An atten- tive listening system with android ERICA: Comparison of autonomous and WOZ interactions. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 118-127, 1st virtual meeting, July 2020. Association for Computational Lin-
guistics. URL httph:tts///t wwtpw/wwwwww......///./w//w. acwweb.org/ htt.aaaooololo20...oo sigdial-1..iii
[40] U. Hadar, T. J. Steiner, and F. C. Rose. Head movement during listening turns in conversation. Journal of Nonverbal Behavior, 9(4)2114 228, 1985.
[41] D. Heylen. Challenges ahead: Head movements and other social acts in conversations. Virtual Social Agents, pages 45-52, 2005.
[42] A. Kendon. Some uses of the head shake. Gesture, 2(2):147-182, 2002.
[43] V. Kazemi and J. Sullivan. One millisecond face alignment with an ensemble of regres- sion trees. In Proceedings of the 2014 IEEE Conference Computer Vision and Pat- tern Recognition, CVPR 144 page 1867-1874, USA, 2014. IEEE Computer Society. ISBN 9781479951188 doi: 111:0..119/CVPPP2000 2014.241. URL https: //doi. httt:::://oo..or/11011/00// 2014.244
[44] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 98): 1780, 1997.
[45] A. J. Aubrey, D. Marshall, L. Rosin, J. Vandeventer, D. Cunningham, and C. Wallraven. Cardiff conversation database (ccdb): A database of natural dyadic conversations. In 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 277-282, 2013. doi: 10. 1w.11///VVR..2013..85

---

[46] M. Kipp. ANVIL a generic annotation tool for multimodal dialogue. In P. Dalsgaard, B. Lindberg. H. Benner, and Z. Tan, editors, EUROSPEECH 2001 Scandinavia, 7th Eu- ropean Conference on Speech Communication and Technology, 2nd INTERSPEECH Event, Aalborg, Denmark, September 3-7, 2001, pages 1367-1370. ISCA, 2001. URL. http: www.wisca-speet.
org/ / htwpaiiieeeeoooeeee.e.ccmmo 2atpaa/00001011a11 html..
[47] L. de Kok and D. Heylen. A survey evaluation metrics for backchannel prediction models. In Proceedings of the Interdisciplinary Workshop on Feedback Behaviors in Dialog, 2012.
12

---

7 Appendix
7.1
Likert item factor analysis loading matrix
Variable
F1
F2
The robot was listening carefully. The robot was not listening to me. The robot's behavior was natural. The robot understood what was saying. The robot cared about what was saying. The robot did not care about what was saying. I felt a close connection with the robot. Talking with the robot was similar to talking with friend.
0.059
0.881
0.241
0.939
0.404
0.471
0.403
0.489
0.643
0.328
-0.558
0.305
0.713
0.238
0.955
-0.026
Talking with the robot was similiar talking with stranger.
-0.839
0,251
Talking with the robot was comforting.
0.595
0.300
13


Reviews:
[{"review_id": "ww34pFi7NIw", "reviewer": "Reviewer_uPNX", "summary": "This paper is concerned with ways of generating backchanneling behaviors for robots (social cues like nodding appropriately placed in the midst of a conversation with a human). Data-driven methods tend to overfit to the (scarce) training set and cannot adapt appropriately to new speakers, so the paper presents a data augmentation method for the input speech and visual features that improves backchanneling behaviors without collecting additional data. The data augmentation schema warps and masks portions of each human speaker’s input features.\n\nThe work diligently collected their own dataset of unscripted but structured 2-person conversations, and used both audio and visual features for training a model that predicts when the agent should or shouldn’t nod. The paper evaluates the data augmentation strategy on this dataset with positive results, and later conducts a user study with a virtual robot that nods whenever the trained model specified it should do so. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I thought the data collection and experimental analysis were really well done, with thorough evaluation and discussion. Additionally, the writing was clear and enjoyable, and the work was well placed within existing literature.\n\nOverall, I liked the paper but there are a few sticking points that concern me/could improve this work. First, the paper discusses backchanneling behaviors as a whole but in reality only nodding is explored. I think the results would be stronger if they showcased a range of backchanneling behaviors (smiling or “hmm”s as well); otherwise, the work essentially boils down to a nodding predictor, which is not novel enough, in my opinion.\n\nSecondly, while Section 4 compares LSTM with and without data augmentation, I was surprised to see that the same comparison didn’t exist in the user study. Since the main contribution of the paper seems to be using data augmentation to improve backchanneling, it seems natural that the study should focus on the effect of data augmentation, and not the effect of different types of predictors.\n\nLastly, unless I am interpreting Fig. 6 wrong, the study results don’t show a significant difference between the learned and the rule based model. I would have liked to see more discussion for why this is the case, especially since the forced choice questions in Fig. 7 indicate a significant difference.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although (mostly) diligently done, the experiments in the paper are limited and I am reluctant to accept the paper without experiments on more backchanneling behaviors. Moreover, I am not convinced that the data augmentation method presented has a significant impact on people’s perception of the robot in conversation.\n\nUPDATE AFTER-REBUTTAL: I thank the authors for their response. I have updated my score to a weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "To improve the submission, I would suggest adding another couple backchanneling behaviors, reframing the paper to only focus on nodding, or making a strong argument for why the experiments presented are expected to generalize to other backchanneling behaviors. I also think rerunning the study to compare LSTMs with and without data augmentation would more directly study whether the proposed data augmentation is effective."}}, {"review_id": "PO9gu1PnASo", "reviewer": "Reviewer_kruZ", "summary": "- This work attempts to learn a policy of when a robot should nod, given the prosody and head orientation of a speaker\n- The problem is of relevance because backchanneling is a crucial mechanism for maintaining the flow of conversation, and robots that are deployed as social agents will need such a capability\n- The authors collect a dataset of conversations, and given annotations of when listeners nod, they train an LSTM on speech and head orientation data\n- In order to combat the low generalizability of prior models to new speakers and provide sufficient data to train the model, the authors use data augmentation techniques such as random shifts and masks to prevent overfitting\n- The authors's results show that the LSTM trained with data augmentation outperforms other baselines\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The authors's motivation, formulation of the problem, and approach are straightforward and easy to follow\n- The results in the paper are promising\n\n**Scope for Improvement**\n\nThe biggest scope for improvement in this work lies in the presentation of the results:\n\n- The Likert data could be better presented (show significance results in the plots!) and better analyzed (one should aggregate multiple items into a combined \"scale\"). For an excellent primer on how to approach Likert metrics, please see Schrum et al., HRI'20\n- What was the loss function that was used for training the models? Additionally, was the loss used for early stopping (recommended) or accuracy (not recommended)? Clarification on this matter would be great\n- (this is speculation) The F1 and accuracy scores are not that high (despite the promising results). Given the qualitative evaluation presented by the authors, might it be worthwhile to report an Intersection over Union-like metric instead (as is commonly used in computer vision)? If the authors wish to be truly rigorous, they can provide a correlation of the F1 metric with the IoU metric\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the paper has weaknesses in its presentation of the results, they can be addressed by following some of the suggestions presented above. I am therefore, weakly in favour of accepting the paper into the technical program.\n\nPost-Rebuttal Update:\nI thank the authors for updating their paper and I find that it is more ready than before for publication. I am in favour of including this in the technical program.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Barring the last point in the scope for improvement, I would like to see a better Likert analysis and some more clarification on the LSTM training.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "5grWRpXXn_I", "reviewer": "Reviewer_Uy2q", "summary": "This paper presents an approach for learning *backchannelling behaviors* for companion robots, an important aspect for cueing and signaling in multi-party interaction in general. Concretely, this paper presents a learned approach for deciding when to *nod* based on visual and speech/acoustic context from a human in conversation.\n\nCritical to the proposed approach is intelligent application of data augmentation; obtaining backchanneling (nodding paired with videos of partner’s head pose, speech) data at scale is hard, and as such, naively training models results in robots with behaviors that are either overfit to speakers from the training set (and thus fail to generalize) or total collapse. By selectively masking and warping the input features (visual head poses, audio features) not only can we enrich the dataset, but we can learn better, more generalizable backchanneling behaviors.\n\nThe proposed model is simple, but effective (quite elegant!). Trained on data collected from 12 speakers primed with a diverse array of prompts, the resulting “virtual robot” experiment shows that not only is data augmentation critical to strong backchanneling performance (both qualitatively and quantitatively), but also out performs a strong, rule-based baseline. \n\nA possible flaw in this work is the lack of deployment on a real robot, instead doing all experimentation with a virtual robot in a web interface; due to the COVID-19 pandemic, I think this is fair, and makes a lot of sense. I appreciate that the paper presents a clear path for deploying and evaluating with actual physical robots, and I am convinced the results will hold.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In terms of strengths, the problem is well-motivated, the proposed solution is elegant, the “key insight” (data augmentation) is also motivated, and proven to be necessary, and in general, the proposed system works convincingly from both a qualitative and quantitative perspective.\n\nAs a weakness; it does seem that backchanneling is just a smaller component of a larger HRI system (e.g., for human-robot dialogue, or assistive robotics). I understand the need to evaluate this specific behavior alone (and from the prior work cited, it seems that this is an established problem), but I do wonder if things change in the context of a broader system (e.g., is nodding really that important when a robot is equipped with the ability to provide dense feedback in terms of actual physical motion, or speech).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The proposed method is well-motivated, well-constructed, and the evaluation is strong. It’s clear that data augmentation is important for getting generalizable behavior from small backchanneling datasets, and that the proposed approach is a good step in solving this task.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "[90 - 92] “Although data augmentation… it has not yet been applied to human-robot interaction” isn’t quite correct; several papers on human-robot interaction (e.g. learning from corrections, language instructions, etc.) all use data augmentation. Would temper this claim."}}]

Author rebuttal:

Meta-review: This paper introduces a technique for automatically learning social backchanneling, in this case specifically the timing of nodding behaviors, from observation of human-human interactions.  

Strengths:

The paper introduces a compelling learning approach which uses data augmentation techniques, such as random shifts and masks, to generalize the learned behaviors to new scenarios and speakers even from relatively little data.

The work is clearly motivated and well presented.

The experimental analysis is thorough and convincing.

Weaknesses:

Several technical details relating to the presentation of the algorithm and figures would improve the technical aspects of the paper.  These include treatment of Likert data and clarification of loss function.
Decision: Accept (Poster)