Reviewerly/exHarmony huggingface dataset:
Federated Continual Learning for Socially Aware Robotics

From learning assistance to companionship, socially aware and socially assistive robotics is promising for enhancing many aspects of daily life. However, socially aware robots face many challenges preventing their widespread public adoption. Two such major challenges are (1) lack in behavior adaptation to new environments, contexts and users, and (2) insufficient capability for privacy protection. The commonly employed centralized learning paradigm, whereby training data is gathered and centralized in a single location (i.e., machine/ server) and the centralized entity trains and hosts the model, contributes to these limitations by preventing online learning of new experiences and requiring storage of privacy-sensitive data. In this work, we propose a decentralized learning paradigm that aims to improve the personalization capability of social robots while also paving the way towards privacy preservation. First, we present a new framework by capitalising on two machine learning approaches, Federated Learning and Continual Learning, to capture interaction dynamics distributed physically across robots and temporally across repeated robot encounters. Second, we introduce four criteria (adaptation quality, adaptation time, knowledge sharing, and model overhead) that should be balanced within our decentralized robot learning framework. Third, we develop a new algorithm – Elastic Transfer – that leverages importance-based regularization to preserve relevant parameters across robots and interactions with multiple humans (users). We show that decentralized learning is a viable alternative to centralized learning in a proof-of-concept Socially-Aware Navigation domain, and demonstrate the efficacy of Elastic Transfer across our proposed evaluation criteria.
Model-based Imitation Learning for Real-time Robot Navigation in Crowds

We are increasingly interacting with robots in our everyday life. To further this development, a critical capability is safe and socially compliant robot navigation in a crowd. In this work, we extract a navigation strategy from past human-to-human interactions with a model-based approach to imitation learning. We propose a hybrid model of crowd dynamics that combines an autoregressive and an inverse autoregressive model for real-time sampling-based planning with respect to human decisions making. Furthermore, we constrain the optimization to allow only admissible velocities for any given robot dynamics that lead to a trajectory on which the robot can safely stop. Experiments are conducted in crowded prerecorded environments, where the robot is placed in a variety of scenarios with varying numbers of humans. The results show that the algorithm is able to navigate in these environments with a lower collision rate and a shorter path than the state-of-the-art.
Human-Robot Interaction using VAHR: Virtual Assistant, Human, and Robots in the Loop

Robots have become ubiquitous tools in various industries and households, highlighting the importance of human-robot interaction (HRI). This has increased the need for easy and accessible communication between humans and robots. Recent research has focused on the intersection of virtual assistant technology, such as Amazon’s Alexa, with robots and its effect on HRI. This paper presents the Virtual Assistant, Human, and Robots in the loop (VAHR) system, which utilizes bidirectional communication to control multiple robots through Alexa. VAHR’s performance was evaluated through a human-subjects experiment, comparing objective and subjective metrics of traditional keyboard and mouse interfaces to VAHR. The results showed that VAHR required 41% less Robot Attention Demand and ensured 91% more Fan-out time compared to the standard method. Additionally, VAHR led to a 62.5% improvement in multi-tasking, highlighting the potential for efficient human-robot interaction in physically- and mentally-demanding scenarios. However, subjective metrics revealed a need for human operators to build confidence and trust with this new method of operation.
Context based Echo State Networks for Robot Movement Primitives

Reservoir Computing, in particular Echo State Networks (ESNs) offer a lightweight solution for time series representation and prediction. An ESN is based on a discrete time random dynamical system that is used to output a desired time series with the application of a learned linear readout weight vector. The simplicity of the learning suggests that an ESN can be used as a lightweight alternative for movement primitive representation in robotics. In this study, we explore this possibility and develop Context-based Echo State Networks (CESNs), and demonstrate their applicability to robot movement generation. The CESNs are designed for generating joint or Cartesian trajectories based on a user definable context input. The context modulates the dynamics represented by the ESN involved. The linear read-out weights then can pick up the context-dependent dynamics for generating different movement patterns for different contexts. To achieve robust movement execution and generalization over unseen contexts, we introduce a novel data augmentation mechanism for ESN training. We show the effectiveness of our approach in a learning from demonstration setting. To be concrete, we teach the robot reaching and obstacle avoidance tasks in simulation and in real-world, which shows that the developed system, CESN provides a lightweight movement primitive representation system that facilitate robust task execution with generalization ability for unseen seen contexts, including extrapolated ones.
Extracting robotic task plan from natural language instruction using BERT and syntactic dependency parser

Natural language encodes rich sequential and contextual information. A task plan for robots can be extracted from natural language instruction through semantic understanding. This information includes sequential actions, target objects and descriptions of working environment. Current systems focus on single-domain understanding such as household or industrial assembly settings, and many rule-based approach have been developed in this context. Thanks to the development of deep learning, data-driven contextual language understanding shows promising results. In this work, an information extraction system is proposed for domain-independent understanding of robotic task plans. The developed approach is based on a pre-trained BERT-model and a syntactic dependency parser. To evaluate the performance, experiments are conducted on three different datasets.
Add-if-Silent Rule-Based Growing Neural Gas for High-Density Topological Structure of Unknown Objects

To realize a super-smart society (Society 5.0) where humans and robots coexist, there is a need for a perceptual system that can recognize unknown objects in various unknown environments quickly and flexibly. In unknown environments, the characteristics of objects cannot be known in advance, so prior learning-based recognition methods such as deep reinforcement learning cannot fully cover the problem. There have been many studies on environment recognition (clustering, etc.) using a combination of RGB images and distance images, but the recognition performance is unstable because it strongly depends on the lighting conditions of the environment. Therefore, in this study, we construct a 3D topological map of the environment in real-time using Growing Neural Gas (GNG), which can learn 3D topological structures even for unlearned objects, using only 3D point cloud data as input. In the real world, due to the characteristics of RGB-D cameras, sample density decreases for more far-away objects and only sparse depth information can be obtained, so conventional GNG cannot generate high-density topological structures of unknown objects. Therefore, if the object category labels of the winner nodes (nearest nodes) for the input vector (3D point cloud) match the unknown object and are within a predefined tolerance area, then it is judged to be useful input information for learning the topological structure of the unknown object, and the topological structure of the unknown object is learned. We propose Add-if-Silent rule-based GNG (AiS-GNG) which can generate high-density topological structures for far-away objects by directly adding input data as a reference vector. We verify the effectiveness of the proposed method through experiments using a 3D dynamics simulator.
An HMM-based Real-time Intervention Methodology for a Social Robot Supporting Learning

To make social robots effective in education, they need to be autonomous both in terms of assessing the student’s engagement state as well as intervening effectively in soft real-time when necessary. Hidden Markov Model (HMM) is an interpretable machine learning technique for modeling temporal data that is commonly used post-hoc to analyse latent learning processes. In this paper, we contribute by proposing an HMM-based intervention methodology for assessing and classifying the state of the student as either productive or unproductive in soft real-time. The system identifies and tracks states and patterns not conducive to learning, and a robot intervention is triggered whenever a too-high non-productive engagement is detected. In a pilot study with 22 children, we evaluate this methodology in terms of both 1) the effectiveness of the interventions on the students’ learning gains and on behaviors found conducive to learning, and 2) the students’ perception of the robotic interventions. Results suggest that the robot interventions have a positive effect on the post-test scores relative to the baseline robot, although there isn’t a significant difference in the learning gains. Moreover, interventions that try to induce reflective behaviors are most effective in inducing the required learning behavior, followed by communication-inducing interventions. Lastly, students’ perception of intervention usefulness does not reflect their actual effectiveness.
Effect of Handshake in VR Environment via Robotic Arm on Psychological Distance

The COVID-19 pandemic has popularized telework using video chat tools. Although video chat tools allow people to communicate each other in remote locations, psychological distance is longer than face-to-face communication, which may lead to a lack of intimacy. Previous studies have shown that interfaces combining video chat, VR, and robots with handshakes can improve the sense of closeness. However, few studies have focused on the sense of closeness in interfaces combining VR and robots. In this study, we proposed a multimodal interface combining VR and a robotic arm and verified whether shaking hands with a remote partner in VR could shorten the psychological distance compared to video chat. The results showed that the psychological distance was predominantly shorter in VR communication than in video chat communication; however, a handshake using robotic arm did not shorten the psychological distance compared to a handshake without a robotic arm. These results indicate that spatial distance is shorter in VR than in video chat and that robotic arms must be made more human-like.
Optimizing Robot Arm Reaching Ability with Different Joints Functionality

During the process of reaching a target, a robotic arm may generate a large number of redundant movements due to the stochastic nature of planning algorithms. We think that each joint of the robotic arm has a unique functionality towards the arm’s end effector. Therefore, during motion planning, we optimized the weights of each joint based on its individual functionality. We introduce the human arm operation mechanism - When a person is reaching for a distant target, the shoulder-near joints take on more actions. We proposed an advanced auxiliary loss function. By utilizing this function, we can filter out multiple solutions from the inverse model, thus reducing the amount of joint change and redundant movements. We conducted comprehensive comparative experiments on the UR3 robotic arm, and the results showed that our approach achieved better efficiency and outcomes.
Vision-based Human Identification with Face and Nametape Recognition in Aerial Casualty Monitoring System

In emergency rescue scenarios, rapid identification of human casualties is a critical first step in enhancing emergency medical response. This task can be limited by the physical and cognitive capacity of rescue personnel, who are exposed to significant risk. The use of small unmanned aerial systems (sUAS) equipped with autonomous casualty assessment abilities can reduce these limitations and risks by enabling remote casualty detection, identification, and vitals assessment, providing standoff protection, and eliminating the need for human personnel to access the potentially hazardous scene. This paper presents a vision-based casualty assessment framework and specifically discusses our casualty identification software, which is designed to recognize the faces of casualties and identify their nametapes in images captured by sUAS under realistic conditions. Our approach addresses the limitations of the sUAS-captured long-distance images to enable accurate identification in challenging casualty monitoring situations. The face and nametape recognition algorithms will be integrated into the larger casualty perception framework and embedded into sUAS platforms to assist with emergency rescue operations. The total casualty perception system will detect, identify, and evaluate the condition of casualties from a remote location, providing standoff protection to first responders and rapid information to inform a suitable medical treatment plan.
Attempting to Aggregate Perceptual Constructs From Deep Neural Networks for Video and Audio Interaction Representation

Socially Assistive Robots are foreseen as having the potential to improve the quality of life of older adults and individuals with mental disabilities. Natural human-robot interaction in everyday settings may require robots that are capable of understanding what is happening in their operating environments so that they can respond appropriately to the experienced situations and engage people in meaningful ways. This paper presents an approach using perceptual constructs to represent what is being observed by the robot. Perceptual constructs are derived from deep neural networks used to process visual and audio data. The objective is to derive a compressed representation of the interactions observed by the robot in real-life settings. Results are provided from observations made by a robot of a room with human activity over a two-week period, outlining what works and remaining challenges.
Neural Network Implementation of Gaze-Target Prediction for Human-Robot Interaction

Gaze cues, which initiate an action or behaviour, are necessary for a responsive and intuitive interaction. Using gaze to signal intentions or request an action during conversation is conventional. We propose a new approach to estimate gaze using a neural network architecture, while considering the dynamic patterns of real world gaze behaviour in natural interaction. The main goal is to provide foundation for robot/avatar to communicate with humans using natural multimodal-dialogue. Currently, robotic gaze systems are reactive in nature but our Gaze-Estimation framework can perform unified gaze detection, gaze-object prediction and object-landmark heatmap in a single scene, which paves the way for a more proactive approach. We generated 2.4M gaze predictions of various types of gaze in a more natural setting (GHIGaze). The predicted and categorised gaze data can be used to automate contextualized robotic gaze-tracking behaviour in interaction. We evaluate the performance on a manually-annotated data set and a publicly available gaze-follow dataset. Compared to previously reported methods our model performs better with the closest angular error to that of a human annotator. As future work, we propose an implementable gaze architecture for a social robot from Furhat robotics <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> . <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> https://furhatrobotics.com/
How Do Human Users Teach a Continual Learning Robot in Repeated Interactions?

Continual learning (CL) has emerged as an important avenue of research in recent years, at the intersection of Machine Learning (ML) and Human-Robot Interaction (HRI), to allow robots to continually learn in their environments over long-term interactions with humans.Most research in continual learning, however, has been robot-centered to develop continual learning algorithms that can quickly learn new information on static datasets.In this paper, we take a human-centered approach to continual learning, to understand how humans teach continual learning robots over the long term and if there are variations in their teaching styles.We conducted an in-person study with 40 participants that interacted with a continual learning robot in 200 sessions.In this betweenparticipant study, we used two different CL models deployed on a Fetch mobile manipulator robot.An extensive qualitative and quantitative analysis of the data collected in the study shows that there is significant variation among the teaching styles of individual users indicating the need for personalized adaptation to their distinct teaching styles.The results also show that although there is a difference in the teaching styles between expert and non-expert users, the style does not have an effect on the performance of the continual learning robot.Finally, our analysis shows that the constrained experimental setups that have been widely used to test most continual learning techniques are not adequate, as real users interact with and teach continual learning robots in a variety of ways.
Exophora Resolution of Linguistic Instructions with a Demonstrative based on Real-World Multimodal Information

To enable a robot to provide support in a home environment through human-robot interaction, exophora resolution is crucial for accurately identifying the target of ambiguous linguistic instructions, which may include a demonstrative, such as “Take that one”. Unlike endophora resolution, which involves predicting the corresponding word from given sentences, exophora resolution necessitates comprehensive utilization of external real-world information to identify and disambiguate the target from the on-site environment. This study aims to resolve ambiguity in language instructions containing a demonstrative through exophora resolution, utilizing real-world multimodal information. The robot accomplishes this by using three types of information: 1) object categories, 2) demonstratives, and 3) pointing, as well as knowledge about objects obtained from the robot’s pre-exploration of the environment. We evaluated the accuracy of object identification under multiple conditions by identifying a user-indicated object in a field that mimics a home environment. Our results demonstrate that our proposed method of exophora resolution using multimodal information can identify the target with two to three times higher accuracy than baseline methods in cases where information is missing.
From Research to Design: Developing the Social Robotic Persuasive Design Cards and Its Techniques

Existing work on social robotic persuasion (SRP) has provided ample knowledge that can benefit the design of social robotics. However, as recent research points out, this knowledge is not presented in a format that effectively supports design practice. Research on translational science provides a theoretical foundation for connecting research to design practice, but the process of knowledge appropriation remains under-explored. In this paper, we present the development and evaluation of the SRPD (social robot persuasive design) cards and the corresponding Roundtable and Spotlight generative methods. Our results show that the SRPD cards can benefit the ideation phase in social robotic design and that our generative methods can optimize participants’ experience in a brainstorming activity.
Improving Sign Language Understanding Introducing Label Smoothing

Sign language is one of the most important communication methods when considering equality, diversity, and inclusion. Sign language understanding implies understanding sign language using machines, and it involves mainly two functions; sign language recognition and sign language translation. To improve sign language understanding performance, this paper proposes to use label smoothing with CTC (Connectionist Temporal Classification) loss as training criteria for the sign language understanding neural network. Experimental results showed the effectiveness of the proposed method in both sign language recognition and translation.
ChatHRC: Personalized Human-Robot Collaboration using Fuzzy Reinforcement Learning with Natural Language Rewards

Collaboration between humans and robots can be challenging because robots may have difficulty understanding a specific person’s intentions, particularly in complicated tasks such as co-manipulation and assembly in computer, communication, and consumer electronics (3C) manufacturing. These tasks require different weights on accuracy and speed for various fabrication steps, making traditional physical interaction inadequate. In this paper, we introduce a fuzzy reinforcement learning-based admittance controller that can infer humans’ intentions not only through physical interaction but also through natural language. During training, the natural language is encoded into a reward term to help the robot reach the human-intended convergence point, allowing us to develop a “personalized” policy. During testing, the language serves as a tool to help the robot understand and obey humans’ intentions when physical interaction alone is insufficient. For example, if the user finds it difficult to push the robot and needs it to move faster, they can say “it’s really slow,” while a request for high-accuracy operation can be conveyed through “the damping is too small.” With this algorithm, the robot can comprehend the intentions and act accordingly in such situations. Further results and videos can be found at: https://sites.google.com/view/hri-nlp.
Exploring the Use of Colored Ambient Lights to Convey Emotional Cues With Conversational Agents: An Experimental Study

Conversational agents (CAs) lack of possibilities to enrich the interaction with emotional cues, although this makes the conversation more human-like and enhances user engagement. Thus, the potential of CAs is not fully exploit and possibilities to convey emotional cues are needed. In this work, CAs use colored ambient lights to display moral emotions during the interaction. To evaluate this approach, a between-subject lab experiment $(N=64)$ was conducted. Participants played a cooperation game with Amazon’s Alexa. Depending on the experimental condition participants received different light expressions: no light, neutral light, or morally emotional light (yellow = joy, blue = sorrow, red = anger matching the game decisions). The effect of the light expressions on the perception of the CA, users’ empathy and cooperation behavior was tested. Against our assumptions, the results indicated no positive effect of the emotional light cues. Limitations, next steps, and implications are discussed.