Reviewerly/exHarmony huggingface dataset:
Recognizing Social Touch Gestures using Optimized Class-weighted CNN-LSTM Networks

Socially aware robotic applications such as companion and therapeutic robots usually require human emotions or intent to be conveyed. As the scope of these applications increases, the need for recognizing affective touch gestures which are often used to convey these emotions or intent becomes eminent. However, existing touch gesture recognition modalities either have low recognition accuracy or depend heavily on carefully hand-crafted features, therefore limiting their deployment in real-life applications. Motivated by the need for learning models with superior accuracy which do not rely on manually selected hand-crafted features, this paper proposes an optimized class-weighted CNN-LSTM for social touch gesture recognition evaluated on the CoST and HAART datasets. Specifically, contrary to vanilla training schemes where equal importance is given to each class in the dataset, different class weights are introduced to give priority to classes that are difficult for the network to distinguish during training. Furthermore, the weights associated with each of the classes are obtained through optimization using Genetic Algorithm. The proposed model demonstrates superior performance compared with other existing models in the literature.
Yousu: A mythical character robot design for public scene interaction

With the advancement of interactive technology in the information age, the problem of “visual blindness” in the field of display design has become increasingly prevalent in public scene interaction design. Therefore, it has become crucial to address how new forms of interaction and interaction scenes can be adopted to attract the public. In the context of robotics’ continuous development, robots are playing an increasingly prominent role as interactive subjects in public scene interaction experiences. In new fields such as digital entertainment, spatial experience, and new media art, various typical scenes of robot interaction have emerged. In this study, a window robot named “Yousu” was developed based on an ancient Chinese mythological character and deployed in the window of a bookstore in Beijing. A user experiment was conducted to investigate how to design a reasonable and effective character robot interaction in public scenes to enhance the interaction scenes’ attractiveness.
A Framework for Realistic Simulation of Daily Human Activity

For social robots like Astro which interact with and adapt to the daily movements of users within the home, realistic simulation of human activity is needed for feature development and testing. This paper presents a framework for simulating daily human activity patterns in home environments at scale, supporting manual configurability of different personas or activity patterns, variation of activity timings, and testing on multiple home layouts. We introduce a method for specifying day-to-day variation in schedules and present a bidirectional constraint propagation algorithm for generating schedules from templates. We validate the expressive power of our framework through a use case scenario analysis and demonstrate that our method can be used to generate data closely resembling human behavior from three public datasets and a self-collected dataset. Our contribution supports systematic testing of social robot behaviors at scale, enables procedural generation of synthetic datasets of human movement in different households, and can help minimize bias in training data, leading to more robust and effective robots for home environments.
Exploring the Influence of Self-Avatar Similarity on Human-Robot Trust

Avatars as digital portrayals of humans play a pivotal role in fostering embodiment and immersion in virtual reality (VR) environments by providing users with a visual representation of their virtual presence. In the context of human-robot interaction (HRI), understanding the dynamics of trust formation between humans and avatars are essential for successful collaboration and communication. This study methodically investigates the effect of the similarity between players and their avatars on the establishment of trust between human users and robots in a VR environment. Our research implies that enhancing the resemblance of self-avatars boosts trust dynamics in human-robot interactions, yet it doesn’t necessarily foster cooperation. Moreover, this research highlights the importance of designing virtual avatars and robots that can effectively communicate and adapt to user preferences, thereby fostering a more engaging and productive HRI experience in VR environments.
Personality-Adapted Language Generation for Social Robots

Previous works in Human-Robot Interaction have demonstrated the positive potential benefit of designing social robots which express specific personalities. In this work, we focus specifically on the adaptation of language (as the choice of words, their order, etc.) following the extraversion trait. We look to investigate whether current language models could support more autonomous generations of such personality-expressive robot output. We examine the performance of two models with user studies evaluating (i) raw text output and (ii) text output when used within multi-modal speech from the Furhat robot. We find that the ability to successfully manipulate perceived extraversion sometimes varies across different dialogue topics. We were able to achieve correct manipulation of robot personality via our language adaptation, but our results suggest further work is necessary to improve the automation and generalisation abilities of these models.
Human-Aware Goal-Oriented Autonomy through ROS-Integrated Timeline-based Planning and Execution

Robots acting in real-world environments may interact with humans at different levels of abstraction (e.g., process, task, physical), entailing different control and coordination challenges. When acting in social situations, robots should be able to pursue (joint) goals by behaving according to the context as well as the skills/features of involved humans. Although reliable and effective, standard control techniques may limit the adaptability of robots. Novel control technologies based on Artificial Intelligence can endow robots with the cognitive capabilities needed to achieve a higher level of autonomy in terms of flexibility, reliability, and awareness. In this context, this paper introduces a goal-oriented acting framework based on timeline-based planning and execution. The framework is evaluated on a realistic Human-Robot Collaboration manufacturing scenario. Results show the capability of dealing with the uncontrollable dynamics of humans achieving effective and reliable collaborations.
S&amp;Reg: End-to-End Learning-Based Model for Multi-Goal Path Planning Problem

In this paper, we propose a novel end-to-end approach for solving the multi-goal path planning problem in obstacle environments. Our proposed model, called S&Reg, integrates multi-task learning networks with a TSP solver and a path planner to quickly compute a closed and feasible path visiting all goals. Specifically, the model first predicts promising regions that potentially contain the optimal paths connecting two goals as a segmentation task. Simultaneously, estimations for pairwise distances between goals are conducted as a regression task by the neural networks, while the results construct a symmetric weight matrix for the TSP solver. Leveraging the TSP result, the path planner efficiently explores feasible paths guided by promising regions. We extensively evaluate the S&Reg model through simulations and compare it with the other sampling-based algorithms. The results demonstrate that our proposed model achieves superior performance in respect of computation time and solution cost, making it an effective solution for multi-goal path planning in obstacle environments. The proposed approach has the potential to be extended to other sampling-based algorithms for multi-goal path planning.
A behavioural transformer for effective collaboration between a robot and a non-stationary human

A key challenge in human-robot collaboration is the non-stationarity created by humans due to changes in their behaviour. This alters environmental transitions and hinders human-robot collaboration. We propose a principled meta-learning framework to explore how robots could better predict human behaviour, and thereby deal with issues of non-stationarity. On the basis of this framework, we developed Behaviour-Transform (BeTrans). BeTrans is a conditional transformer that enables a robot agent to adapt quickly to new human agents with non-stationary behaviours, due to its notable performance with sequential data. We trained BeTrans on simulated human agents with different systematic biases in collaborative settings. We used an original customisable environment to show that BeTrans effectively collaborates with simulated human agents and adapts faster to non-stationary simulated human agents than SOTA techniques.
Tell Me More, Tell Me More: AI-Generated Question Suggestions for the Creation of Interactive Video Recordings

Time-Offset Interaction Applications (TOIAs) are narrative-sharing systems that use databases of previously recorded videos of real people to mimic conversations with them. These video databases comprise large (the larger, the better) collections of videos of answers paired with specific questions. This paper focuses on a solution to the challenge of creating such databases without exhausting their creators’ creativity, energy, and interest. We describe the design and development process of Question Suggester (QS)-an intelligent GPT-3-based service that generates suggested questions following up a conversation based on the history of recorded questions and answers. We conduct a user study to empirically evaluate the value of QS for reducing the effort to create a video database while creating an interaction that is enjoyable. The users’ average experience rating for QS is 4.6 compared to 4.0 when QS is not used (on a 1-5 scale, p-value<0.05). The experience with interactions so created is more enjoyable, too (3.7vs. 3.3, p-value<0.05). The usage metrics and qualitative feedback confirm that QS is essential for interactive video-recording systems and for increasing their adoption.
Asura hands: Own and control two left hands in immersive virtual reality environment

Body ownership, which is the feeling that one’s body part belongs to oneself, and agency, which is the sense of being able to control one’s own body part, can be felt towards fake body parts and those depicted by computer graphics. As part of an attempt to transfer self-body awareness to fake body segments, we investigated whether body ownership and agency are felt towards two visible left hands in an immersive virtual reality environment. One of the two hands shown through virtual reality goggles spatially matched the unseen actual hand. The other hand was fake and displayed at either the 10-cm lateral or medial side of the position of the actual hand. These two left hands moved synchronously with the actual left hand. Participants completed a behavioral test and questionnaire after adapting to the two left hands. In the behavioral test, participants accessed randomly emerging spheres using the seen hands as fast as possible. They used the fake hand to touch a sphere when it appeared near the fake hand 41% of the time when the fake hand was displayed at the medial position of the actual hand. The results of the questionnaire suggest that agency was experienced for the two visible left hands for this condition. By contrast, body ownership was felt mostly against the displayed hand that was spatially consistent with the actual hand. These findings indicate that although agency can be simultaneously felt for two seen left hands, body ownership is felt only for either of the two visible left hands.
A Probabilistic Approach based on Combination of Distance Metrics and Distribution Functions for Human Postures Classification

The article proposes a method for classifying human postures using an improved probabilistic neural network (PNN) with different distance measures and different probabilistic distribution functions (PDF). We found that the PNN with angular distance provides better accuracy, precision, and recall for the postures classification tasks than the PNN with conventional Euclidean distance. The k Nearest Neighbors (kNN) method gives slightly better prediction results than PNN, but our PNN is much faster. Such good computational performance is beneficial for posture recognition tasks that require real-time functions. An example is the needs of cobots or service robots. The article also proposes a method for selecting the distribution smoothing parameter ($\sigma$) using the sub-optimization process based on the improved Gray Wolf optimization (I-GWO) algorithm. It was found that the impact of PDF differences on the quality of the results can be reduced by choosing the best possible $\sigma$. In order to evaluate the developed method, selected human activities were recorded. The datasets were created using two different RGB-D systems located in two different laboratories.
Pimp my language! The influence of robot customization duration on psychological ownership and trust

Involving users in the design of novel technologies is an efficient way to improve attitudes and trust towards them.For example, users may adapt a robots' characteristics to their preferences and needs.To date, research on the customization of robot functionalities is still scarce, with existing work predominantly focusing on the influence of perceived, rather than actual user involvement in the design process.Furthermore, it remains unknown whether the effects of customization depend on merely choosing options, or actively implementing these options.Thus, in the present experiment we investigate whether actual time invested in choosing options (i.e., words that a robot would learn) and implementing these chosen options (i.e., defining the words to a robot in one's own terms) influence psychological ownership and trust towards a customized robot.We found that actual time involved in the design of a robot's functionality may not have the expected influence on psychological ownership, anthropomorphism, and trust towards a robot.Contrary to our hypotheses, time invested in choosing options slightly decreased perceived control and psychological ownership over the robot.The results imply a potential influence of task difficulty, which might hint at perceived self-efficacy being a determinant of psychological ownership.
Robotic Tutors for Nurse Training: Opportunities for HRI Researchers

An ongoing nurse labor shortage has the potential to impact patient care well-being in the entire healthcare system. Moreover, more complex and sophisticated nursing care is required today for patients in hospitals forcing hospital-based nurses to carry out frequent training and assessment procedures, both to onboard new nurses and to validate skills of existing staff that guarantees best practices and safety. In this paper we recognize an opportunity for the development and integration of intelligent robot tutoring technology into nursing education to tackle the growing challenges of nurse deficit. To this end, we identify specific research problems in the area of human-robot interaction that will need to be addressed to enable robot tutors for nurse training.
Connecting without Reaching: How Voice-cloned Robot Can Enhance Mental Health of Isolated People During a Pandemic

Voice cloning techniques using deep neural networks have been used for frauds, such as false financial transactions or fake news, limiting their widespread application. However, when incorporated with communication robots, voice cloning may increase social connectedness for isolated people. Herein, we suggested the concept of a voice-cloned communication robot (VCR), equipped with acquaintances’ voices to allow people in isolation to feel connected with their acquaintances. We developed a prototype VCR, conducted exploratory qualitative and quantitative studies, and verified its potential effectiveness in enhancing the mental health of isolated people.
Redundant Multi-DoF Robot Arm Co-operation Through the Body Integration System

The human arm has seven DoF. If the human arm were more redundant, it would improve workability. However, intuitive manipulation of redundant arms is difficult due to the complexity of the input system or the limitations of the human arm. In this study, we constructed a system in which multiple operators integrate their bodies to intuitively operate a redundant multi-DoF robotic arm. The robotic arm was assembled by connecting two arms, and the roles of each section –distal arm and proximal arm– were assigned to different operators. First, vibrotactile feedback methods to the users based on the operation movement were investigated to compensate for the recognition of the partner’s movement. Three types of feedback methods —the actions of both oneself and one’s partner, the actions of the partner, and none— were compared in order to improve the body awareness. It was shown that feedback of the movement from the partner who operates the proximal robotic arm significantly enhanced the sense of body ownership. Then, the proposed co-operation of the redundant robotic arm was evaluated in the workability. Two types of manipulation methods —a single-operator system that switched the target robotic arm and a multi-DoF body-integration system used by two people— were compared through an object-manipulation task involving obstacle avoidance. It was shown that the two-person co-operation through body integration system to be faster than the single-person operation.
Designing Visual and Auditory Attention-Driven Movements of a Tabletop Robot

This work presents a framework for a visual-auditory attention-driven robot eye-head gaze movement, which combines visual and auditory inputs to determine the direction of gaze movement for a social robot. The framework computes the most salient changes in position by considering both visual and auditory cues. The proposed system was implemented on Haru, a tabletop social robot, where eye-head gaze movement was controlled using visual input from a camera positioned above the eyes and auditory input from a seven-channel microphone. This allowed for eye movement on a two-dimensional flat screen and body rotation towards the person who is speaking. This framework provides a representation of the robot’s attentional gaze that leverages both visual and auditory cues, resulting in more natural and responsive coordinated eye-head gaze movements of the social robot. The potential benefits include improved communication, increased engagement, and a stronger sense of connection with the robot.
Exploring the Personality Design Space of Robots : Personalities and Design Implications for Non-Anthropomorphic Wellness Robots

Non-anthropomorphic robots can be cost-effective and efficient choice in certain context in comparison to social or humanoid robots. However, introduction of nonanthropomorphic robots can evoke uncertainty and anxiety due to novelty of technology. The goal of this paper is to explore personality design space for non-anthropomorphic wellness robots in office environment to foster acceptance among users. Through Participatory Design approach, we explored appropriate personalities for a well-being robot, which would detect employees’ sitting posture and suggest small wellness interventions. We addressed the following research questions: (i) How can personalities be designed and integrated to non-anthropomorphic wellness robots to promote users’ acceptance? (ii) How do the users perceive designed personalities of non-anthropomorphic wellness robot in the office context? We conducted one contextual inquiry (n=5) and one co-design workshop (n=15) followed by evaluation (n=5) in IT office environment with office employees. As a contribution to the paper, we present personalities and design implications for non-anthropomorphic wellness robot in the office context. Our contribution will serve as a guideline for designers to explore and expand their knowledge on designing robot personalities for non-anthropomorphic robots in the context.
Demand-Aware Multi-Robot Task Scheduling with Mixed Reality Simulation

This paper addresses the problem of multi-robot task scheduling by estimating the demand for the tasks in a real-world scenario. Scheduling tasks for multiple robots becomes complex when a human is involved in allocating limited resources. We propose a stochastic multi-agent multi-armed bandit based task scheduler which prioritizes the tasks based on the estimated demand for the tasks. To gain insight into the varying priorities of a human task allocator in a multi-armed bandit scenario, we conducted a user study in a Mixed Reality environment which can be used to customize the resource allocation process. We observe that the users consistently made sub-optimal choices due to their preference to minimize other parameters (such as cumulative distance travelled) of the real world scenario rather than strictly adhering to the optimal strategy. Our proposed method uses the Thompson sampling bandit algorithm with ϵ-greedy approach to solve the multi-agent multi-armed bandit problem. The approach outperformed other methods such as first-come-first-serve, rate monotonic scheduling, and heuristic-based Min-interference approaches in terms of the demand aware performance index.
What’s at Stake? Robot explanations matter for high but not low-stake scenarios

Although the field of Explainable Artificial Intelligence (XAI) in Human-Robot Interaction is gathering increasing attention, how well different explanations compare across HRI scenarios is still not well understood. We conducted an exploratory online study with 335 participants analysing the interaction between type of explanation (counterfactual, feature-based, and no explanation), the stake of the scenario (high, low) and the application scenario (healthcare, industry). Participants viewed one of 12 different vignettes depicting a combination of these three factors and rated their system understanding and trust in the robot. Compared to no explanation, both counterfactual and feature-based explanations improved system understanding and performance trust (but not moral trust). Additionally, when no explanation was present, high-stake scenarios led to significantly worse performance trust and system understanding. These findings suggest that explanations can be used to calibrate users’ perceptions of the robot in high-stake scenarios.
Adaptive Human-Robot Collaboration: Evolutionary Learning of Action Costs Using an Action Outcome Simulator

One of the main challenges for successful human-robot collaborative applications lies in adapting the plan to the human agent’s changing state and preferences. A promising solution is to bridge the gap between agent modelling and AI task planning, which can be done by integrating the agent state as action costs in the task planning domain. This allows for the plan to be adapted to different partners, by influencing the action allocation. The difficulty then lies in setting appropriate action costs. This paper presents a novel framework to learn a set of planning action costs considering the preferred actions for an agent based on their state. An evolutionary optimisation algorithm is used for this purpose, and an action outcome simulator is developed to act as the black-box function, based on both an agent model and an action type model. This addresses the challenge of collecting data in HRC real-world scenarios, accelerating the learning for posterior fine-tuning in real applications. The coherence of the models and the simulator is proven through a conducted survey, and the learning algorithm is shown to learn appropriate action costs, producing plans that satisfy both the agents’ preferences and the prioritised plan requisites. The resulting system is a generic learning framework integrating components that can be easily extended to a wide range of applications, models and planning formalisms.