Reviewerly/exHarmony huggingface dataset:
Eye Movement Differences in Japanese Text Reading between Cognitively Healthy Older and Younger Adults

We analyzed the eye movements of cognitively healthy older adults while reading Japanese text and compared them with those of younger adults. We found that it is essential to match the reading speeds of older and younger adults to accurately compare eye movement parameters during their reading. Cognitively healthy older adults had longer fixation durations, fewer fixations, and fewer extra fixations than younger adults. Meanwhile, cognitively healthy older adults had a length of forward saccades comparable to that of younger adults. These results suggest that the reduced efficiency due to a longer fixation duration is compensated for by fewer fixations in cognitively healthy older adults who read at the same speed as younger adults.
From Natural to Non-Natural Interaction: Embracing Interaction Design Beyond the Accepted Convention of Natural

Natural interactions feel intuitive, familiar, and a good match to the task, user’s abilities, and context. Consequently, a wealth of scientific research has been conducted on natural interaction with computer systems. Contrary to conventional mainstream, we advocate for “non-natural interaction design” as a transformative, creative process that results in highly usable and effective interactions by deliberately deviating from users’ expectations and experience of engaging with the physical world. The non-natural approach to interaction design provokes a departure from the established notion of the “natural,” all the while prioritizing usability—albeit amidst the backdrop of the unconventional, unexpected, and intriguing.
Investigating the Effect of Orientation Variability in Deep Learning-based Human Activity Recognition

Deep Learning (DL) has enabled considerable increases in the accuracy of classification tasks in several domains, including Human Activity Recognition (HAR). It is well-known that when data distribution changes between the training and test datasets, the accuracy can drop, sometimes significantly. However, some variability sources in HAR, such as sensor orientation, are only sometimes considered when evaluating these models. Therefore, we must understand how much such changes could impact current DL architectures. In this paper, under an orientation variability scenario, we evaluate three common DL architectures, DeepConvLSTM, TinyHAR, and Attend-and-Discriminate, to quantify the performance drop attributed to this shift. Our results show that all architectures show performance drops on average, as expected, but participants are affected differently from them, so they would fall short for some in classification accuracy in real-life settings where orientation can change across the wearing sessions of one participant or across participants. The performance change is related to the difference in distribution distance.
Research Methodologies across the Physical - Virtual Reality Spectrum

Over the last couple of years, there has been a big push toward making immersive and mixed technologies available to the general public. Yet, designing for these new technologies is challenging, as users need to position virtual objects in 3D space. The current state-of-the art technologies used to access these virtual environments (e.g., Head-mount displays (HMD)s also presents additional challenges for designers when considering depth perception issues that affect user precision. Moreover, these challenges are exacerbated when designers consider accessibility needs of special populations. To make new immersive and mixed technologies more accessible, we propose a tutorial at UbiComp / ISWC 2023 to discuss design strategies, research methodologies, and implementation practices with special populations using technologies across the physical-virtual reality spectrum. In this tutorial participants will learn how to make these technologies more accessible by (1) teaching students of the tutorial how to design, prototype, and evaluate these technologies using empirical research. We aim to (2) bring together researchers, practitioners, and students who are interested in making immersive and mixed technologies more accessible and (3) identify common problems when designing new user interfaces.
Toward Pioneering Sensors and Features Using Large Language Models in Human Activity Recognition

In this paper, we propose a feature pioneering method using Large Language Models (LLMs). In the proposed method, we use ChatGPT 1 to find new sensor locations and new features. Then we evaluate the machine learning model which uses the found features using an open dataset. In current machine learning, humans make features, for this engineers visit real sites and have discussions with experts and veteran workers. However, this method has the problem that the quality of the features depends on the engineer. In order to solve this problem, we propose a way to make new features using LLMs. As a result, we obtain almost the same level of accuracy as the proposed model which used fewer sensors and the model uses all sensors in the dataset. This indicates that the proposed method is able to extract important features efficiently.
Conversational Grounding in Multimodal Dialog Systems

The process of “conversational grounding” is an interactive process that has been studied extensively in cognitive science, whereby participants in a conversation check to make sure their interlocutors understand what is being referred to. This interactive process uses multiple modes of communication to establish the information between the participants. This could include information provided through eye-gaze, head movements, intonation in speech, along with the content of the speech. While the process is essential to successful communication between humans and between humans and machines, work needs to be done on testing and building the capabilities of the current dialogue system in managing conversational grounding, especially in multimodal medium of communication. Recent work such as Benotti and Blackburn [3] have shown the importance of conversational grounding in dialog systems and how current systems fail in them which is essential for the advancement of Embodied Conversational Agents and Social Robots. Thus my Ph.D. project aims to test, understand and improve the functioning of current dialog models with respect to Conversational Grounding.
Come Fl.. Run with Me: Understanding the Utilization of Drones to Support Recreational Runners' Well Being

The utilization of drones to assist runners in real-time and post-run remains a promising yet unexplored field within human-drone interaction (HDI). Hence, in my doctoral research, I aim to delve into the concepts and relationships surrounding drones in the context of running, than focusing solely on one specific application. I plan on accomplishing this through a three-stage research plan: 1) investigate the feasibility of drones to support outdoor running research, 2) empathize with runners to assess their preferences and experiences running with drone, and 3) implement and test an interactive running with drone scenario. Each stage has specific objectives and research questions aimed at providing valuable insights into the utilization of drones to support runners. This paper outlines the work conducted during my Ph.D. research along with future plans, with the goal of advancing the knowledge in the field of runner drone interaction.
Smart Garments for Immersive Home Rehabilitation Using VR

Adherence to a rehabilitation programme is vital to recover from injury, failing to do so can keep a promising athlete off the field permanently. Although the importance to follow their home exercise programme (HEP) is broadly explained to patients by their physicians, few of them actually complete it correctly. In my PhD research, I focus on factors that could help increase engagement in home exercise programmes for patients recovering from knee injuries using VR and wearable sensors. This will be done through the gamification of the rehabilitation process, designing the system with a user-centered design approach to test different interactions that could affect the engagement of the users.
Captivating the Senses: Crafting a Multisensory Virtual Experience for Enhanced Realism

This paper explores the challenges and opportunities in achieving a high level of realism and presence in Virtual Reality (VR) experiences. Realism and presence, which refer to the level of authenticity of the virtual environment and the user’s subjective perception of being in it, respectively, are fundamental components of engaging and satisfactory VR experiences. Despite advancements in VR technology, users can often distinguish between real and virtual experiences, thereby limiting the potential of VR in various fields. This paper reviews various challenges inherent in achieving indistinguishable realism in VR, encompassing visual, auditory, and tactile components. These insights are critical in guiding the continued development of more immersive and realistic VR experiences.
Toward an Ever-present Extended Reality: Distinguishing Between Real and Virtual

In this short paper we provide some observations on the (im)possibility of distinguishing between real and virtual in augmented reality worlds. Distinguishing may be possible because of imperfect technology, impossibilities, and manipulating perceptual capabilities. The question must be asked, if this distinction cannot always be made, does this place responsibility on the owner or administrator of the Extended Reality (XR) environments? And who might be the owner or administrator in the case of ever-present extended reality?
Where Are the Best Positions of IMU Sensors for HAR? - Approach by a Garment Device with Fine-Grained Grid IMUs -

Many IMU-data-based HAR datasets have been proposed. However, integrating the use of those is difficult because each dataset is measured by its unique sensor positions. Thus, we have developed a garment device with 396 IMU sensors placed in a grid with approximately 7 cm spacing to enable the matching of IMU data from various positions and the integrated use of multiple databases. In this paper, the best and most suitable sensor positions for classification were explored with this device for six basic activities and eight daily activities with the expectation of determining the key IMU positions for the integrated dataset and creating standard sensor positions when building a new dataset. The single sensor positions with the highest classification performance in all activities, basic activities, and daily activities were the upper-middle part of the dominant lower arm, the front of the thigh, and the inner wrist of the dominant arm. The position with the highest classification score for daily activities except Writing and Typing was the inner wrist of the dominant arm. The sensor position with the highest classification score for Writing was the upper part of the dominant upper arm, and Typing was the upper-middle part of the non-dominant lower arm. Furthermore, the combination of the four sensors showed the best macro-F1 scores for all activity sets: 0.90 with all activities, 1.00 with basic activities, and 0.83 with daily activities.
Client Clustering for Energy-Efficient Clustered Federated Learning in Wireless Networks

Clustered Federated Learning (FL) is a novel approach for handling data heterogeneity in FL and training personalized ML models for clients. However, existing research overlooks network constraints and objectives when implementing clustered FL in a wireless network. Since clients experience varying energy costs when connected to different servers, the cluster formation greatly impacts system energy efficiency. To address this, we present an energy-efficient client clustering problem that optimizes FL performance while minimizing energy costs. We introduce a new metric, the effective number of clients, to predict cluster FL performance based on size and data heterogeneity, guiding the clustering optimization. To find optimal client clusters, we propose an energy-efficient clustering algorithm using parallel Gibbs Sampling. Simulations and data experiments demonstrate that our algorithm achieves tunable trade-offs between data similarity (thus FL performance) and energy cost.
Towards a Taxonomy of Human-Building Interactions

Built environments (BE) are increasingly integrating sensing and interactivity capabilities, changing how we co-exist and live. In recent years, HCI, ubiquitous computing, and architecture have contributed to the interdisciplinary field of Human-Building Interaction (HBI). HBI represents the growing complexities of human experience within BE, which includes utilizing sensing capabilities to gracefully support people’s needs. We present an initial taxonomy classifying interactions with HBI devices, the HBI Interactivity Taxonomy, a novel contribution to this field. We employed an integrative research strategy, sampling device descriptions from commercial and academic sources. Taxonomy features were extracted through thematic analysis. The resulting list of characteristics describes how users interact with HBI devices. We offer an initial version of this taxonomy as a tool to facilitate communication and enhance the design and evaluation of future HBI devices in both academia and industry.
An Interactive Workplace for Improving Human Robot Collaboration: Sketch Workpiece Interface for Fast Teaching (SWIFT).

Ubiquitous computing refers to the concept of integrating computers and technology into everyday objects and environments, making them constantly available and seamlessly interconnected. In recent times, there has been a growing adoption of the concept in industrial settings as well. This contribution targets the replacement of an industrial manual repair process through a robotic solution. The focus of this work is on the novel interface utilizing a digital pen and spatial augmented reality, which mimics the original job preparation process but ubiquitously enables automated robot programming. This shifts the work of the human from a dull, dirty and dangerous process to the cognitively more demanding part of inspection and process strategy definition, whereas the robot is used as a dexterous and tenacious intelligent process tool to perform the surface processing operation itself.
Audio-Visual Group-based Emotion Recognition using Local and Global Feature Aggregation based Multi-Task Learning

Audio-video group emotion recognition is a challenging task and has attracted more attention in recent decades. Recently, deep learning models have shown tremendous advances in analyzing human emotion. However, due to its difficulties such as hard to gather a broad range of potential information to obtain meaningful emotional representations and hard to associate implicit contextual knowledge like humans. To tackle these problems, in this paper, we proposed the Local and Global Feature Aggregation based Multi-Task Learning (LGFAM) method to tackle the Group Emotion Recognition problem. The framework consists of three parallel feature extraction networks that were verified in previous work. After that, an attention network using MLP as a backbone with specially designed loss functions was used to fuse features from different modalities. In the experiment section, we present its performance on the EmotiW2023 Audio-Visual Group-based Emotion Recognition subchallenge which aims to classify a video into one of the three emotions. According to the feedback results, the best result achieved 70.63 WAR and 70.38 UAR on the test set. Such improvement proves the effectiveness of our method.
ACE: how Artificial Character Embodiment shapes user behaviour in multi-modal interaction

The ACE - how Artificial Character Embodiment shapes user behavior in multi-modal interactions - workshop aims to bring together researchers, practitioners and experts on the topic of embodiment, to analyze and foster discussion on its effects on user behavior in multi-modal interaction. ACE is aimed at stimulating multidisciplinary discussions on the topic, sharing recent progress, and providing participants with a forum to debate current and future challenges. The workshop includes contributions from computational, neuroscientific and psychological perspectives, as well as technical applications.
AR for the Masses: Attina, the Low-Cost Accessible Headset for Inclusive Learning

Current commercial Augmented Reality (AR) headsets are expensive and limited in availability, posing challenges for researchers and designers in resource-constrained settings. To address this, we introduce Attina, an open-source, low-cost optical see-through AR headset using a smartphone as its display. Overcoming limitations of previous work, the system features stereoscopic rendering for enhanced depth perception and a 3-DoF controller for virtual interactions. We discuss Attina’s design and outline opportunities for a usability study to evaluate its user experience.
Ether-Mark: An Off-Screen Marking Menu For Mobile Devices

Given the computing power of mobile devices, porting feature-rich applications to these devices is increasingly feasible. However, feature-rich applications include large command sets, and providing access to these commands through screen-based widgets results in issues of occlusion and layering. To address this issue, we introduce Ether-Mark, a hierarchical, gesture-based, marking menu inspired, around-device menu for mobile devices enabling both on- and near-device interaction. We investigate the design of such menus and their learnability through three experiments. We first design and contrast three variants of Ether-Mark, yielding a zigzag menu design. We then refine input accuracy via a deformation model of the menu. And, we evaluate the learnability of the menus and the accuracy of the deformation model, revealing an accuracy rate up to 98.28%. We finally, compare in-air Ether-Mark with marking menus.Our results argue for Ether-Mark as a promising effective mechanism to leverage proximal around-device space.
Using Explainability for Bias Mitigation: A Case Study for Fair Recruitment Assessment

In this study, we propose a bias-mitigation algorithm, dubbed ProxyMute, that uses an explainability method to detect proxy features of a given sensitive attribute (e.g., gender) and reduces their effects on decisions by disabling them during prediction time. We evaluate our method for a job recruitment use-case, on two different multimodal datasets, namely, FairCVdb and ChaLearn LAP-FI. The exhaustive set of experiments shows that information regarding the proxy features that are provided by explainability methods is beneficial and can be successfully used for the problem of bias mitigation. Furthermore, when combined with a target label normalization method, the proposed approach shows a good performance by yielding one of the fairest results without deteriorating the performance significantly compared to previous works on both experimental datasets. The scripts to reproduce the results are available at: https://github.com/gizemsogancioglu/expl-bias-mitigation.
TeleVIP: On-site Person Removal and Context Distillation Platform for Dedicated Telepresence Experience

For the application of telepresence robots, the on-site person usually has been regarded as a main element in remote interaction. However, in the context of a remote tour, the visual presence of an on-site person unexpectedly interrupting a robot’s view may worsen the remote visitor’s viewing experience. Moreover, the presence of a telepresence robot with its camera on may arise the privacy concerns of an on-site person. In this sense, we developed TeleVIP, a platform that provides telepresence applications with easy-to-use APIs to remove the figures of on-site people and selectively distill the useful information from the presence or context of on-site people, according to the application’s demand. We implemented a working prototype of TeleVIP and performed pilot experiments in terms of both quantitative metrics and qualitative results, indicating the preliminary usability and feasibility of TeleVIP.
ToozKit: System for Experimenting with Captions on a Head-worn Display

The advent of Automatic Speech Recognition (ASR) has made real-time captioning for the Deaf and Hard-of-Hearing (DHH) community possible, and integration of ASR into Head-worn Displays (HWD) is gaining momentum. We propose a demonstration of an open source, Android-based, captioning toolkit intended to help researchers and early adopters more easily develop interfaces and test usability. Attendees will briefly learn about the the technical architecture, use-cases and features of the toolkit as well as have the opportunity to experience using the captioning glasses on the tooz HWD while engaging in conversation with the demonstrators.
Demonstrating AHA: Boosting Unmodified AI's Robustness by Proactively Inducing Favorable Human Sensing Conditions

Imagine a near-future smart home. Home-embedded visual AI sensors continuously monitor the resident, inferring her activities and internal states that enable higher-level services. Here, as home-embedded sensors passively monitor a free person, good inferences happen inconsistently. The inferences’ confidence highly depends on how congruent her momentary conditions are to the conditions favored by the AI models, e.g., front-facing or unobstructed.
Exploring Feedback Modality Designs to Improve Young Children's Collaborative Actions

Tangible user interfaces offer the benefit of incorporating physical aspects in the interaction with digital systems, enriching how system information can be conveyed. We investigated how visual, haptic, and audio modalities influence young children’s joint actions. We used a design-based research method to design and develop a multi-sensory tangible device. Two kindergarten teachers and 31 children were involved in our design process. We tested the final prototype with 20 children aged 5-6 from three kindergartens. The main findings were: a) involving and getting approval from kindergarten teachers in the design process was essential; b) simultaneously providing visual and audio feedback might help improve children’s collaborative actions. Our study was an interdisciplinary research on human-computer interaction and children’s education, which contributed an empirical understanding of the factors influencing children collaboration and communication.