Reviewerly/exHarmony huggingface dataset:
Prediction of Various Backchannel Utterances Based on Multimodal Information

The listener's backchannels are an important part of dialogues. With appropriate backchannels, people are able to smoothly promote dialogues. Thus, backchannels are considered to be important in dialogues between not only humans but also humans and agents. Progress has been made in studying dialogue agents that perform natural affable dialogue. However, we have not clarified whether the listener's various backchannel types are predictable using the speaker's multimodal information. In this paper, we attempt to predict a listener's various backchannel types on the basis of the speaker's multimodal information in dialogues. First, we construct a dialogue corpus that consists of multimodal information of a speaker's utterances and a listener's backchannels. Second, we construct machine learning models to predict a listener's various backchannel types on the basis of a speaker's multimodal information. Our results suggest that our model was able to predict a listener's various backchannel types on the basis of a speaker's multimodal information.
Now or When?

Interruptions are an important aspect of human-human communication. They help to adjust the conversation flow. Our aim is to equip virtual agents with the ability to handle interruptions, that is to decide when and how to interrupt their human interlocutor. In this paper, we focus on predicting when interruptions may occur during the conversation using multimodal features only from the speaker and propose a model trained on a corpus of dyadic interactions. To assess the model's accuracy, we conduct a perceptual study where we compare different timings (ground truth, randomly chosen or predicted by our model).
A Study of Prediction of Listener's Comprehension Based on Multimodal Information

During dialogues, speakers need to be able to predict whether their partners understand their message. This is important for not only for human-to-human interaction but also human-to-agent interaction. We consider that if the listener's comprehension level can be automatically predicted, interactive agents will be able to communicate appropriately according to the user's comprehension level. However, to the best of our knowledge, there is no case study that reveals how comprehension can be predicted based on multimodal information about the listener. In this study, we attempt to predict comprehension levels on the basis of the listener's multimodal information. First, we construct a dialogue corpus consisting of the listener's comprehension levels and the listener's multimodal information. Next, we construct machine learning models that predict the listener's comprehension levels on the basis of the listener's multimodal information. Our results suggest that our model was able to predict a listener's comprehension level on the basis of a listener's multimodal information. In addition, two movements, the lifting of the cheeks and the pulling up of the corners of the lips, were suggested to be important in assessing the listener's level of comprehension.
Who Said That?

In this paper, we explore the ability to detect personas from conversational data. First, we adapt the Persona-Chat dataset, a well-known dialogue dataset, to support the task of selecting the correct persona out of various candidates. Then, we introduce persona perturbations to create additional identical personas that act as more challenging distractors. We train three different BERT-based models in a multiple-choice fashion to select the correct persona from a group of distractor personas. We show that this approach is able to discern between the group of original persona candidates, however, these models struggle to maintain high performance when we employ very identical distractors obtained from the proposed perturbations.
Prompting for Socially Intelligent Agents with ChatGPT

Socially Intelligent Agents (SIAs) have become increasingly popular in various contexts, including education and entertainment. However, creating complex social scenarios tailored to a designer's specific goals remains a significant challenge. The authoring burden can be substantial, limiting the potential of SIAs to deliver rich, engaging experiences. In this work, we propose leveraging the extensive knowledge stored within Large Language Models and use theory-driven prompting to extract social practices and identify appropriate social affordances for a scenario description. Our prompting approach aims to guide the system into considering the essential components (beliefs and desires) necessary to produce intentions, actions, and emotions1. Results show that our approach produces large amounts of accurate and new information that can add value to the scenario. However, the process can introduce inaccuracies without human supervision.
How Far ahead Can Model Predict Gesture Pose from Speech and Spoken Text?

We investigated how far into the future nonverbal behavior can be predicted from speech and speech text. Specifically, we build a model that generates future behaviors from speech and speech text information and evaluate the quality of the generated behaviors. This helps to clarify how far into the future behavior can be accurately predicted. Our experimental results show that in Gesture Pose Generation using speech and speech text, on the basis of the input speech and text, the nonverbal behavior up to at least 500 ms ahead can be predicted with objective evaluation values that are the same as those when no future prediction is made. This result shows a new possibility for Gesture Pose Generation using speech and speech text to predict the future up to at least 500 ms ahead with no performance degradation.
The influence of conversational agents' role and behaviors on narrative experiences

Conversational agents (CAs) in narrative experiences are defined by the role they endorse and the communication style they adopt when users interact with them. In computer games, users' perception of intelligence and believability ascription influence the positive evaluation of CAs. Yet, the impact of CAs' role and communication style on users' experience remains to be clarified. In this research, the effect of the role and communication style of CAs on users' evaluation is investigated in a crime-solving textual game. Different CAs were created whose roles in the narrative (witness or suspect) and communication style (aggressive or cooperative) were manipulated. A Wizard of Oz method was used to control communication style while users' experience was assessed regarding their interaction with each CA using scales of perceived intelligence and believability. Users also had to indicate a culprit and rate the certainty of their judgments. The results show that both CAs' role and communication style have an influence on users' perception of intelligence and believability, with a higher effect of the role. However, only communication style had a significant influence on the choice of the culprit.
Towards Virtual Audience Simulation For Speech Therapy

The utilization of virtual reality (VR) technology has shown promise in various therapeutic applications, particularly in exposure therapy for reducing fear of certain situations objects or activities, e.g. fear of height, or negative evaluation of others in social situations. VR has been shown to yield positive outcomes in follow-up studies, and provides a safe and ecological therapeutic environment for therapists and their patients. This paper presents a collaborative effort to develop a VR speech therapy system which simulates a virtual audience for users to practice their public speaking skills. We describe a novel web-based graphical user interface that enables therapists to manage the therapy session using a simple timeline. Lastly, we present the results from a qualitative study with therapists and teachers with functional dysphonia, which highlight the potential of such an application to support and augment the therapists' work and the remaining challenges regarding the design of natural interactions, agent behaviours and scenario customisation for patients.
Predicting Interaction Quality Aspects Using Level-Based Scores for Conversational Agents

In order to improve human-agent interaction, it is essential to have good measures of interaction quality. We define interaction quality based on multiple aspects, including usability, likability and perceived conversation quality as subjective measures, and interaction length, completion rate and frequency of unrecognized utterances as objective measures. Determining necessary improvements to a conversational agent is a non-trivial task, because it is difficult to infer from an evaluation of the agent as a whole, which aspects of the agent need to be improved to raise the interaction quality. In this paper, we propose a scoring system for task-oriented conversational agents to predict aspects of interaction quality and to guide an iterative improvement process. Our scoring system does not provide a single score, but leverages structural features of the dialogue management approach and assigns a score on three levels: the utterance, dialogue move, and genre level. Using the agent's scores on separate levels to predict the interaction quality allows making targeted improvements to the conversational agent. In order to evaluate our scoring system, we apply it over the course of multiple crowdsourcing pilot studies, using a recipe recommendation agent. We evaluate the obtained scores in regard to their ability to predict selected objective and subjective interaction quality aspects, as well as their suitability for making informed decisions about necessary improvements.
Virtual Reality Applications for Enhancing Human-Robot Interaction

Human-Robot Interaction (HRI) has become increasingly important as robots have been integrated into various daily life aspects. Gesture recognition plays a crucial role in non-verbal communication in HRI. Indeed, the real-time robot's interpretation of human gestures is essential for enhancing the overall user experience. To this aim, improving the accuracy and efficiency of gesture recognition in HRI could be achieved by integrating Virtual Reality (VR) technology. In this article, we describe the development of virtually-generated avatars and a gesture recognition system that uses machine learning techniques to allow ground robots to recognize gestures performed by both digital agents and actual humans. Nevertheless, we address some existing challenges by presenting a set of gesture definitions, data generation, and system evaluation using virtual reality simulation. Our findings and results highlight the importance of addressing these challenges to enable effective gesture recognition in HRI. In particular, we demonstrate that the adoption of VR can significantly increase the system's accuracy and efficiency, improving the overall user experience.
Conducting Cognitive Behavioral Therapy with an Adaptive Virtual Agent

When conversing, people adapt their behaviors to one another to show their engagement. Virtual agents, acting as interaction partners, should also adapt to their interlocutors in real time. In this paper, we introduce a virtual agent delivering Cognitive Behavioral Therapy (CBT) and adapting its behaviors in real time. The system focuses on the real-time generation of adaptive behavior and management of natural CBT dialogue.
Effect of Avatar Facial Expressiveness on Team Collaboration in Virtual Reality

This paper investigates the impact of displaying the facial expressions of a user in real time on the performance of the task, the social presence and the behavioral changes of the users interacting in a Virtual Reality environment. To evaluate this approach, we conducted a study where the users collaborated to build a TV stand in dyad including a novice and an expert assistant. The collaborative task was divided into two independent phases: a face-to-face discussion phase without object manipulation, and a furniture assembly phase with object manipulation. Our results indicate that the proposed approach can increase social presence and lead to gaze behavior changes in multi-user environments during the face-to-face phase.
Augmented Co-Speech Gesture Generation

Due to their significance in human communication, the automatic generation of co-speech gestures in artificial embodied agents has received a lot of attention. Although modern deep learning approaches can generate realistic-looking conversational gestures from spoken language, they often lack the ability to convey meaningful information and generate contextually appropriate gestures. This paper presents an augmented approach to the generation of co-speech gestures that additionally takes into account given form and meaning features for the gestures. Our framework effectively acquires this information from a small corpus with rich semantic annotations and a larger corpus without such information. We provide an analysis of the effects of distinctive feature targets and we report on a human rater evaluation study demonstrating that our framework achieves semantic coherence and person perception on the same level as human ground truth behavior. We make our data pipeline and the generation framework publicly available.
The Virtual Maze

This demo presents an innovative and validated method for measuring interpersonal trust in virtual humans as social interaction partners. In our design, participants are tasked to navigate through a virtual reality maze and interact with a virtual human, seeking advice on which path to take. By carefully designing the experiments and context, we induced participants to believe that they are interacting with a real person. A validation study, manipulating virtual humans' appearance, tone of voice, and engagement, revealed that the frequency of advice-seeking in the maze correlates with participants' evaluation of interpersonal trust towards virtual humans. Thus, our paradigm represents a valuable research tool for studying trust in virtual reality.
Towards an Adaptive Pedagogical Agent in a Reading Intervention Using Reinforcement Learning

Learning to read can be difficult, but is a crucial skill that is needed for the whole lifespan. Deficiencies in reading capabilities can often be traced back to the initial attempts to learn reading during primary school. While there exist a number of analogue and digital tools to support reading acquisition for primary school children, they often don't account for their heterogeneity. This paper presents work in progress on integrating an adaptive pedagogical agent in a mobile reading app designed for second graders with reading difficulties. We apply Reinforcement Learning to adapt the feedback behavior of the agent to each child's learning needs and elaborate on our methodological approach. With it, we contribute a novel concept for adapting a pedagogical agent's feedback behavior within the critical context of reading development.
XAI to Increase the Effectiveness of an Intelligent Pedagogical Agent

We explore eXplainable AI (XAI) to enhance user experience and understand the value of explanations in AI-driven pedagogical decisions within an Intelligent Pedagogical Agent (IPA). Our real-time and personalized explanations cater to students' attitudes to promote learning. In our empirical study, we evaluate the effectiveness of personalized explanations by comparing three versions of the IPA: (1) personalized explanations and suggestions, (2) suggestions but no explanations, and (3) no suggestions. Our results show the IPA with personalized explanations significantly improves students' learning outcomes compared to the other versions.
CUBE: Conversational User-Interface-Based Embodiment

Our study introduces an open general-purpose platform for the embodiment of conversational AI systems. Conversational User-interface Based Embodiment (CUBE) is designed to streamline the integration of embodied solutions into text-based dialog managers, providing flexibility for customization depending on the specific use case and application. CUBE is responsible for naturally interacting with users by listening, observing, and responding to them.
Alexa, What's Inside of You

Despite the rising prevalence of intelligent voice assistants in people's homes, it remains opaque to most users how they function. With the overall goal to foster informed usage and responsible handling of personal data, the user's understanding increasingly moves into the focus of research. Particularly in the context of black box technologies such as voice assistants, individuals build intuitive mental models (also called folk theories) which can act as a guide for how they understand and interact with such systems. To shed light on individuals' mental models regarding intelligent voice assistants, we applied a visual elicitation method during a citizen science workshop: 26 participants were asked to visualize how they imagine a voice assistant to function. The resulting drawings were categorized based on their theorization complexity level. While basic awareness of the overall functionality (user input -- something happens -- voice assistant output) was found in every drawing, 13 participants revealed mental models on how the input is received (e.g., audio recording and speech processing), six on how the information is then processed (e.g., database requests and data categorization), and five on how the output is generated (e.g., speech synthesis). Overall, the results underline the need to address individuals' gaps in understanding of intelligent technology that is already in widespread use.
Visual Similarity for Socially Interactive Agents that Support Self-Awareness

Self-awareness is a critical factor in social interaction. Teachers being aware of their own emotions and thoughts during class may enable reflection and behavioral change. While inducing self-awareness through mirrors or video is common in face-to-face training, it has been scarcely examined in digital training with virtual avatars. This paper examines the relationship between avatar visual similarity and inducing self-awareness in digital training environments. We developed a theory-based methodology to reliably manipulate perceptually relevant facial features of digital avatars based on human-human identification and emotional predisposition. Manipulating these features allows to create personalized versions of digital avatars with varying degrees of visual similarity.
工業製品に対するPFAS規制と分析に関する注意点

The method of chemical analysis must be selected according to the purpose. Otherwise, proper results cannot be obtained. When analyzing PFOA as an example, the use of pretreatment methods intended for the textiles or leather for the analysis of resin films leads to lower results. Chemical analysis is used as one of the information to demonstrate that industrial products comply with substance restrictions. The information obtained from chemical analysis is important, but there are precautions. This paper describes the results of the chemical analysis of PFOA performed as a verification test and points to precaution regarding chemical analysis.
Performance Augmentation of Base Classifiers Using Adaptive Boosting Framework for Medical Datasets

This paper investigates the performance enhancement of base classifiers within the AdaBoost framework applied to medical datasets. Adaptive boosting (AdaBoost), being an instance of boosting, combines other classifiers to enhance their performance. We conducted a comprehensive experiment to assess the efficacy of twelve base classifiers with the AdaBoost framework, namely, Bayes network, decision stump, ZeroR, decision tree, Naïve Bayes, J-48, voted perceptron, random forest, bagging, random tree, stacking, and AdaBoost itself. The experiments are carried out on five datasets from the medical domain based on various types of cancers, i.e., global cancer map (GCM), lymphoma-I, lymphoma-II, leukaemia, and embryonal tumours. The evaluation focuses on the accuracy, precision, and efficiency of the base classifiers in the AdaBoost framework. The results show that the performance of Naïve Bayes, Bayes network, and voted perceptron is highly improved compared to the rest of the base classifiers, attaining accuracies as high as 94.74%, 97.78%, and 97.78%, respectively. The results also show that in most cases, the base classifiers perform better with AdaBoost compared to their performance, i.e., for voted perceptron, the accuracy is improved up to 13.34%.For bagging, it is improved by up to 7%. This research aims to identify such base classifiers with optimal boosting capabilities within the AdaBoost framework for medical datasets. The significance of these results is that they provide insight into the performance of the base classifiers when used in the boosting framework to enhance the classification performance of classifiers in scenarios where individual classifiers do not perform up to the mark.
Emergency Remote Teaching and Learning and Teachers’ Digital Competence

The challenges posed by the COVID-19 pandemic forced schools to quickly mobilize toward emergency remote teaching and learning (ERTL) practices. Teachers’ digital competences were essential in this context. This study aims to identify the ERTL practices adopted in Portugal and how they were perceived by teachers of students aged 3 to 18. Data were gathered through a questionnaire administered online during April and May 2020 to 329 teachers. ERTL was perceived as an opportunity to develop their own and their students’ digital competences. Teachers expressed a moderately positive view of the process and reported an experience of work overload. The pedagogical practices implemented by teachers during this period were diverse and differed according to gender, self-assessment of digital competence, and level of education taught. Results point to the importance of continued teacher training to achieve quality distance education.
Otonom mobil robotların Voronoi diyagramı ve karınca kolonisi optimizasyonuna dayalı yol planlaması

Path planning aims to enable autonomous robots to navigate safely and efficiently from a starting point to a target point in challenging and dynamic environments. Path planning in robotics is highly significant and still an ongoing subject of research. The increasing use of robots in various applications such as industrial automation, service robotics, and autonomous vehicles has brought forth the need for reliable and efficient path planning algorithms. The inherent capability of Voronoi diagrams to partition space based on proximity makes them an effective framework for research in path planning. Ant colony optimization, a bio-inspired optimization technique, is based on the foraging behavior of ants and is commonly employed to address the traveling salesman problem and various other combinatorial optimization problems. A hybrid method was adopted in this study by combining a Voronoi diagram and an ant colony algorithm. To create paths for the robot where it can stay as far away from obstacles as possible, a Voronoi diagram was utilized. Additionally, to find the shortest path from the starting point to the destination among these paths, ant colony optimization was employed. The main contribution of the study lies in the combination of the Voronoi diagram for obstacle avoidance and ant colony optimization for finding the optimal path. The combination of these techniques makes an effective contribution to robotic path planning by focusing on ensuring safety by avoiding obstacles while optimizing the shortest path. Experimental studies show that the hybrid method produces successful results for the desired purpose.