# Learning Social Robot Behaviors for Interacting with Staff and Customers

Malcolm Doering, Dražen Brščić, Takayuki Kanda

► To cite this version:

Malcolm Doering, Dražen Brščić, Takayuki Kanda. Learning Social Robot Behaviors for Interacting with Staff and Customers. Machine Learning in Human-Robot Collaboration: Bridging the Gap (workshop). 17th Annual ACM/IEEE International Conference on Human-Robot Interaction (HRI 2022), Mar 2022, Sapporo, Japan. hal-04339455

HAL Id: hal-04339455

https://hal.science/hal-04339455v1

Submitted on 13 Dec 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
Learning Social Robot Behaviors for Interacting
with Staff and Customers

1st Malcolm Doering

2nd Dražen Brščić

3rd Takayuki Kanda

Dept. of Social Informatics, Graduate School of Informatics

Kyoto University

Kyoto, Japan

{doering, drazen, kanda}@i.kyoto-u.ac.jp

***Abstract***—Data-driven imitation learning has been used to train social robots to interact with human customers in service domains. However, up till now previous works on imitation learning of social interaction behaviors for human-robot interaction have focused only on customer-shopkeeper interactions, but not on how the robot should interact with fellow staff members. This work presents work in progress on training social service robots to both perform customer service and work together with human staff. Interviews of people with retail work experience were conducted to discover what types of tasks staff members work together on and how they work together. From this, representative interaction scenarios for role-play were created for the purpose of data collection. Finally, a preliminary interaction behavior learning system is proposed.

***Index Terms***—Human-robot interaction, imitation learning, machine learning, human-robot teaming

# I. INTRODUCTION

Recently social robots have been applied in a variety of domains, such as elder care [1], personal companions [2], hotel concierges [3], workout partners [4] and in day-to-day interaction [5], which require various social interaction behaviors. One approach to developing interaction behaviors is to manually code them or use integrated development environments such as choreograph [6] or interaction composer [7], but this is tedious, time-consuming, and requires the developer to anticipate a myriad social scenarios. Another approach, which overcomes these limitations, is to use data-driven imitation learning to learn social interaction behaviors automatically from example interaction data without manual data annotation. With this approach, interaction data can be collected via passive sensor networks in places that people frequently interact and the repeatable, formulaic behaviors, which characterize many domains where social service robots might be useful (e.g. retail, restaurants, and museum tours), can be learned via machine learning with reduced effort by developers.

Previous work on data-driven imitation learning has focused on one-to-one interactions [8], one-to-many interactions [9], learning proactive behaviors [10], curiosity-driven learning

[11], resolving ambiguity [12], remembering customer preferences [13] and adapting to changing product inventory [14]. Furthermore, these works focused on retail scenarios where one shopkeeper robot interacts with customers, but the case where a shopkeeper robot must also work together with other staff members has been less explored. This work reports on work-in-progress on a data-driven imitation learning approach to train a robot to both interact appropriately with other staff and perform customer service.

# II. RELATED WORKS

## A. Data-driven Imitation Learning of Social Interaction Behaviors

Data-driven systems for designing behaviors for virtual humans were explored early on: The ‘Restaurant Game’ was a virtual restaurant in which two human players, playing the roles of customer and waiter, could navigate around the restaurant and interact via text and predefined actions [15], [16]. ‘Plan networks’ were automatically extracted from the interaction data, enabling the automation of virtual characters. The ‘Mars Escape’ game used a similar approach, and went a step further by embodying the learned behaviors in a real robot [17]–[19]. In contrast, data collected in the real, physical world, which we propose to collect and use for training, has the additional challenge of sensor noise, which is absent in virtual worlds.

[8], [20] introduced the concept of training a camera shopkeeper robot from examples of natural human-human interaction without any manual data annotation, on which this work is based. The approach consists of clustering the raw interaction data to find discrete common actions, and then training a neural network classifier using the interaction state at each point in time as inputs and the action cluster IDs as target outputs.

Much research has been done in extending the data-driven imitation learning approach to HRI. [9] focused on applying the data-driven imitation learning framework to one-to-many interaction (i.e. one shopkeeper, many customers). [12] focused on the problem of modeling the hidden structure of interaction, which enabled resolution of ambiguous customer speech. [13] showed how gated recurrent neural networks could be used to learn a memory model of customer behavior.

M. Doering, D. Brščić, and T. Kanda are also with Advanced Telecommunications Research Institute International (ATR). This work was supported in part by JST, AIP Trilateral AI Research, Grant Number JPMJCR20G2, Japan (problem formulation) and in part by JST CREST under Grant Number JPMJCR17A2, Japan (data collection).
[11] presented a system that could explore different robot behaviors and learn online, which resulted in more varied, interesting robot behaviors than previous approaches and enabled some customization to customers' individual differences. [14] introduces an approach that automatically adapts the robot's behavior to account for changing products in the store. Those works focused on customer-staff interactions. In contrast, the current work focuses on staff-staff interactions and customer-staff-staff interactions.

## B. Human-robot Teaming

Human-robot teaming focuses on how humans and robots can work together to complete tasks. One approach to human-robot teaming is to create a top down model of how the robot should interact with other team members. Robots can act as team members by merely reacting to the state of the world, planning its actions to reach a goal, with or without a model of the humans in the space, and with or without a model of the human's mental model of the robot (trust and expectations of the robot) [21], [22]. Examples of the model of the human include its cognitive and physical state (abilities, preferences, intentions, goals, model of reality, etc.) [23], [24].

Another approach to human-robot teaming is apprenticeship learning, i.e. learning from an expert's demonstrations. Inverse reinforcement learning (IRL) is one technique for apprenticeship learning, which tries to learn a reward function from demonstration data and a state-action policy from that [25]. [26] uses IRL to learn different policies for teaming with different types of users and demonstrates their technique in a task where a robot arm holds an object for a human to work on. Our proposed method, based on [8], [13], is another type of apprenticeship learning, called imitation learning which tries to directly learn a state-action policy without learning a reward function first.

# III. TASKS IN A RETAIL STORE

## A. *Interviews of Retail Store Staff*

It is currently difficult to collect data of customer-staff and staff-staff interactions in a real retail store, so we propose to collect data from role-played interactions in the lab instead. To create representative role-play scenarios, we interviewed people with retail work experience to learn more about the types of tasks and interaction that occurs there.

12 native level Japanese speakers with retail experience were recruited to participate in the interviews (7 female, 5 male, average age 39.3, s.d. 9.8). The interview format was semi-structured and consisted of first informing the participant of the interview's purpose and then asking 13 questions about what tasks they worked on in the retail store, how they worked and interacted with other staff, how they interacted with customers, etc. Participants were encouraged to respond freely and an interpreter helped translate between English and Japanese. Each interview took between one and two hours and each participant was payed 6000 JPY (about 54 USD).

The participants had experience in a variety of different retail stores, including large department stores, grocery stores,

specialty shops (ballet, pets, alcohol, suits), apparel shops, shopping malls, and convenience stores. The roles of participants in these stores ranged from store manager, section manager, owner, part-time staff, customer service manager, regional manager, direction leader of marketing, department store direction leader, and general full-time staff. We focused the interviews on their experience on the floor, where we envision robots to be most useful.

## B. Summary of Tasks

1) *Tasks:* The tasks collected fit into 11 broad categories: managing the cash register, supervising and managing junior workers, managing stock, customer service, shoplifting prevention, communication among staff, cleaning the store, spying on rival stores, store-specific tasks (e.g. various tasks dealing with animals in the pet shop), and other tasks (advertising, organizing shopping carts, etc.).

2) *Customer-staff Interactions:* We found that many customer-staff interactions occur in all types of retail stores, for example greeting customers, answering customer questions, asking customer preferences, recommending products, answering phones, walking around the store to ask if any customer needs help, guiding customers who ask for a specific product, etc. Some interactions only occur in certain types of stores, for example helping customers try on makeup, measuring customers for suit fitting, helping customers try on shoes, and gift wrapping. Sometimes certain types of customers need to be classified and dealt with in different ways, such as customers who need help vs. just browsing, customers who are likely to complain, and regular customers. Some types of interaction occur only with regular customers, such as making small talk or taking actions based on their preferences.

3) *Staff-staff Interactions and Communication:* Many types of staff-staff interaction occur where the staff are co-present, such as when one staff member trains another to do some task, asks another to do some task, asks another a question, needs access to a restricted resource, or when multiple staff collaborate on a task (e.g. stocking shelves or taking inventory).

More generally, communication among staff is important for running a retail store. In many stores the day opens or closes with a staff meeting in which important information is shared (e.g. about events in the area expected to draw many customers, recent shoplifting events in the area, which displays to put out, and the status of ongoing tasks such as product orders). During busy times, staff communicate where they are going and what they are doing. Non-verbal communication is also important, for example eye gaze can be used to signal they need assistance with a customer. At the end of a shift, staff will tell the person taking over the state of on-going tasks, what must be done, etc. Communication is often face-to-face, but in some large stores staff communicate via microphones and ear pieces or some idiosyncratic system (e.g. in one store staff communicated which department needs help by sending by cell phone a number representing the department). When staff are not co-present but must communicate, written notes
<figure><img src="image_2.png" /><figcaption>Fig. 1. Example of staff executing multiple tasks in parallel. They coordinate among themselves based on skills, preferences, etc. and the state of the world (e.g. presence of customers) to decide who will do which tasks.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 2. Example of staff-staff-customer 3-way interaction. When the customer asks for information unknown to the staff member, she goes to fetch a second, more knowledgeable, staff member and introduces her to the customer.</figcaption></figure>

are used. Sometimes staff must also communicate with other branches, for example to coordinate their displays. During non-busy times, small talk is important to let staff learn about each other (including their skills and preferences) and build good relationships (which makes it easier to later ask for help). Staff also communicate among themselves to share the most efficient ways of performing tasks.

4) 3-way Customer-staff-staff Interactions: Some interactions involve a customer and multiple staff, such as when one staff member brings a trainee along to demonstrate customer service or when a customer asks a question requiring the staff member to bring someone more knowledgeable with information about a product or how to perform some process (like using a printer at a convenience store).

5) Planning and Coordination: We found that most frequently staff split up the tasks and worked on them individually, in parallel (e.g. managing the cash register, announcing sales, and stocking shelves). Sometimes multiple staff worked together on the same task at the same time, for example in stock management one person would count product while another checks a list. Sometimes multiple staff worked together on the same task at different times for example when a product arrives and must later be given to a certain customer. Sometimes lower level workers must coordinate with higher level workers because they do not have access to certain resources, such as using a computer to make orders.

6) Task Allocation: When coordinating multiple tasks to decide who will do which tasks, sometimes staff decide among themselves and other times a manager decides. In either case, several attributes of each staff member are considered when making the decision, including their preferences, skills, job title or position, and gender (e.g. in some stores it is better to let female staff help female customers). Furthermore, attributes of the task are also considered, including task priority, department or section of the task, and time of day or required sequence of tasks.

## IV. PROPOSED DATA COLLECTION

We set up a (7x8m) lab space to look like a camera shop. We plan to invite participants to role-play staff-staff, customer-staff, and customer-staff-staff interactions, and record their audio and motions. Each data collection session will have two staff members (two-recurring participants with service experience) and one customer, who will role-play multiple scenarios. Kinect Azure sensors will be used to track participants motions (32 joint positions) over time, wireless lapel-mounted microphones will record audio data, inter-channel suppression [27] will be used to isolate individual participants' audio, and automatic speech recognition will transcribe their speech. To make the social interaction behaviors feasible to learn within time and resource constraints, we specified the camera attributes customers should mainly talk about (price, weight, color, resolution, and camera-specific features). To elicit various behaviors, we will assign staff roles (manager, junior who likes customer service, junior who dislikes customer service, etc.) and customer roles (portrait photographer, nature photographer, novice photographer, curious customer, or window shopper) per scenario.

We found that staff tended to work together in two different ways: First, splitting up tasks and working on them independently, in parallel and second, working together on tasks that required staff to interact.

**Independent tasks done in parallel** include things like managing the cash register, stocking shelves, and helping customers (Fig. 1). From the staff interviews, we found that staff would communicate among themselves to split up the tasks based on their skills, preferences, etc. and then perform them independently. Interesting dynamics occur when the state of the world changes while they are performing these tasks, such as a customer asking for help, requiring one staff member to call over another who is more knowledgeable but prefers not to interact with customers (Fig. 2). To record training data containing examples of such interactions, we assigned staff participants preferences such as 'liking customer service,' 'disliking customer service,' 'in-depth knowledge of products,' and 'sparse knowledge of products.'
<figure><img src="image_4.png" /><figcaption>Fig. 3. System Overview. Before runtime, speech features, typical stopping locations, and shopkeeper speech clusters are learned from the training data and the neural network is trained. During runtime, speech and tracking data are collected from a sensor network in the real world, the raw data is abstracted using the models of typical behavior patterns, vectorized, and input to the trained neural network. The neural network outputs the robot's next action.</figcaption></figure>

**Tasks requiring staff to work together** include situations where one staff member has access to a restricted resource, such as access to a stock room, a computer, or a license to handle prescription medications, requiring staff without access to interact with that staff member to complete the task, for example when fetching a product from behind the counter for a customer. To record such interactions, we gave one staff participant permission to access a 'stock room' and a 'behind the counter' area containing products which needed to be brought to the customer.

The data collection procedures were approved by the Kyoto University and ATR internal ethics review boards

V. PROPOSED LEARNING SYSTEM

The goal of the system is to learn the social interaction behaviors of a shopkeeper from examples of human-human interaction, such that a robot could perform as a shopkeeper in interaction with real humans. Fig. 3 shows an overview of the complete system that would enable a robot shopkeeper to function at runtime.

The proposed system is based on those in [8], [13]. Data collected from the human-human interactions (Sec. IV) is used for training a neural network to predict the next shopkeeper action given the current interaction state. To reduce the dimensionality and make the learning problem more tractable, the raw data is abstracted to a higher level of meaningful information, such as typical stopping locations (via stopping location clustering), proxemics formations (via existing HRI models of proxemics formations, such as *waiting*, *face-to-face*,

and *present object* [28]), and common shopkeeper actions (via clustering of speech and locomotion actions). Furthermore, the interaction data is discretized into sequences of discrete actions by segmenting the data whenever a customer action is detected (moving to a new location or saying an utterance) or a timeout occurs. In this way, interaction state input - action output training pairs can be extracted from the data without manual data annotation.

After training the network, it can be used to automate a robot to interact with a human customer (Fig. 3). The same sensor network used for human-human data collection is used at run time. To decide the robot's actions in real time, the raw data coming from the sensor network is discretized and abstracted in the same way as was done to prepare the training data. It is then fed into the trained neural network. The neural network then outputs the robot's next action - a shopkeeper action cluster ID corresponding to a spatial state, state target, and speech cluster. The robot moves to the location indicated by the output spatial state and state target and synthesizes the medoid utterance of the speech cluster with text-to-speech. In this way, the robot can respond to the constantly changing state of the interaction with the various interaction behaviors learned from the human-human training data.

The focus of this work is for the robot to interact with fellow staff, not only customers (as the previous work focused on [8]). Therefore, it remains to be discovered how well this proposed approach, which is based on the previous work, generalizes to situations with both staff-staff, staff-customer, and 3-way interactions among multiple staff and customers. Specifically,
it is anticipated that it may be necessary to integrate into the
imitation learning approach a method of modeling the human
staff member's mental state (e.g. preferences and intentions)
so that the robot can smoothly coordinate its behaviors with
fellow staff [21], [22], [24].

VI. FUTURE WORK

This paper reports work in progress towards the goal of
training a shopkeeper robot to interact socially with both other
staff and customers using a data-driven imitation learning ap-
proach. Thus far, we have conducted interviews of people with
retail work experience to identify the types of tasks performed
in retail stores and the ways in which staff interact with each
other and with customers. Based on the interviews, we crafted
representative retail interaction scenarios that participants can
role-play in an in-lab data collection area to be recorded
with a passive sensor network. Data collection, training, and
evaluation of the proposed learning approach remains to be
done.

If successful, the work will demonstrate a method for
training a human-robot interaction system to interact with
retail staff and customers, without requiring expensive manual
data annotation or hand-coding of interaction behaviors. In
the future, such an approach could be applied to a variety of
domains, wherever interaction training data can be collected,
towards the goal of making social service robots more ubiq-
uitous and helpful to society.

REFERENCES

[1] K. Kuwamura, S. Nishio, and H. Ishiguro, *Designing Robots for Positive Communication with Senior Citizens*. Springer, 2016, pp. 955–964.

[2] H. A. Samani, A. D. Cheok, F. W. Ngiap, A. Nagpal, and M. Qiu, "Towards a formulation of love in human-robot interaction," in *RO-MAN*. IEEE, 2010, Conference Proceedings, pp. 94–99.

[3] S. Guo, J. Lenchner, J. H. Connell, M. Dholakia, and H. Muta, "Conversational bootstrapping and other tricks of a concierge robot," in *HRI*, 2017, Conference Proceedings, pp. 73–81.

[4] D. J. Rea, S. Schneider, and T. Kanda, “’’ is this all you can do? harder!” the effects of (im) polite robot encouragement on exercise effort,” in *Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction*, 2021, pp. 225–233.

[5] S. Rosenthal, J. Biswas, and M. Veloso, “An effective personal mobile robot agent through symbiotic human-robot interaction,” in *AAMAS*. IFAAMAS, 2010, Conference Proceedings, pp. 915–922.

[6] E. Pot, J. Monceaux, R. Gelin, and B. Maisonnier, "Choregraphe: a graphical tool for humanoid robot programming," in *RO-MAN*. IEEE, 2009, Conference Proceedings, pp. 46–51.

[7] D. F. Glas, T. Kanda, and H. Ishiguro, “Human-robot interaction design using interaction composer: Eight years of lessons learned,” in *HRI*. IEEE Press, 2016, Conference Proceedings, pp. 303–310.

[8] P. Liu, D. F. Glas, T. Kanda, and H. Ishiguro, “Data-driven hri: Learning social behaviors by example from human-human interaction,” *IEEE T-RO*, vol. 32, no. 4, pp. 988–1008, 2016.

[9] A. Nanavati, M. Doering, D. Brščić, and T. Kanda, “Autonomously learning one-to-many social interaction logic from human-human interaction data,” in *Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction*, ser. HRI ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 419–427. [Online]. Available: https://doi.org/10.1145/3319502.3374798

[10] P. Liu, D. F. Glas, T. Kanda, and H. Ishiguro, “Learning proactive behavior for interactive social robots,” *Autonomous Robots*, vol. 42, no. 5, pp. 1067–1085, 2017.

[11] M. Doering, P. Liu, D. F. Glas, T. Kanda, D. Kulić, and H. Ishiguro, “Curiosity did not kill the robot: A curiosity-based learning system for a shopkeeper robot,” *ACM Transactions on Human-Robot Interaction (THRI)*, vol. 8, no. 3, p. 15, 2019.

[12] M. Doering, D. F. Glas, and H. Ishiguro, “Modeling interaction structure for robot imitation learning of human social behavior,” *IEEE Transactions on Human-Machine Systems*, 2019.

[13] M. Doering, T. Kanda, and H. Ishiguro, “Neural-network-based memory for a social robot: Learning a memory model of human behavior from data,” *J. Hum.-Robot Interact.*, vol. 8, no. 4, Nov. 2019. [Online]. Available: https://doi.org/10.1145/3338810

[14] M. Doering, D. Brščić, and T. Kanda, "Data-driven imitation learning for a shopkeeper robot with periodically changing product information," *J. Hum.-Robot Interact.*, vol. 10, no. 4, Jul. 2021. [Online]. Available: https://doi.org/10.1145/3451883

[15] J. Orkin and D. Roy, “The restaurant game: Learning social behavior and language from thousands of players online,” *Journal of Game Development*, vol. 3, no. 1, pp. 39–60, 2007.

[16] —, “Automatic learning and generation of social behavior from collective human gameplay,” in *AAMAS*. IFAAMAS, 2009, Conference Proceedings, pp. 385–392.

[17] S. Chernova, J. Orkin, and C. Breazeal, “Crowdsourcing hri through online multiplayer games,” in *AAAI Fall Symposium: Dialog with Robots*, 2010, Conference Proceedings, pp. 14–19.

[18] S. Chernova, N. DePalma, E. Morant, and C. Breazeal, “Crowdsourcing human-robot interaction: Application from virtual to physical worlds,” in *RO-MAN, 2011 IEEE*. IEEE, 2011, Conference Proceedings, pp. 21–26.

[19] C. Breazeal, N. DePalma, J. Orkin, S. Chernova, and M. Jung, “Crowdsourcing human-robot interaction: New methods and system evaluation in a public environment,” *JHRI*, vol. 2, no. 1, pp. 82–111, 2013.

[20] P. Liu, D. F. Glas, T. Kanda, H. Ishiguro, and N. Hagita, “How to train your robot-teaching service robots to reproduce human social behavior,” in *RO-MAN*. IEEE, 2014, Conference Proceedings, pp. 961–968.

[21] T. Chakraborti, S. Kambhampati, M. Scheutz, and Y. Zhang, “Ai challenges in human-robot cognitive teaming,” *arXiv preprint arXiv:1707.04775*, 2017.

[22] M. Wooldridge, *An introduction to multiagent systems*. John wiley & sons, 2009.

[23] M. C. Gombolay, C. Huang, and J. Shah, “Coordination of human-robot teaming with human task preferences,” in *2015 AAAI Fall Symposium Series*, 2015.

[24] A. Clodic, E. Pacherie, R. Alami, and R. Chatila, “Key elements for human-robot joint action,” in *Sociality and normativity for robots*. Springer, 2017, pp. 159–177.

[25] P. Abbeel and A. Y. Ng, “Apprenticeship learning via inverse reinforcement learning,” in *Proceedings of the twenty-first international conference on Machine learning*, 2004, p. 1.

[26] S. Nikolaidis, R. Ramakrishnan, K. Gu, and J. Shah, “Efficient model learning from joint-action demonstrations for human-robot collaborative tasks,” in *2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI)*. IEEE, 2015, pp. 189–196.

[27] C. T. Ishi, C. Liu, J. Even, and N. Hagita, “Hearing support system using environment sensor network,” in *2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*. IEEE Press, 2016, p. 1275–1280. [Online]. Available: https://doi.org/10.1109/IROS.2016.7759211

[28] P. Liu, D. F. Glas, T. Kanda, and H. Ishiguro, “Two demonstrators are better than one-a social robot that learns to imitate people with different interaction styles,” *IEEE Transactions on Cognitive and Developmental Systems*, 2017.