Reviewerly/exHarmony huggingface dataset:
Validating instructional design and predicting student performance in histology education: Using machine learning via virtual microscopy

As a part of modern technological environments, virtual microscopy enriches histological learning, with support from large institutional investments. However, existing literature does not supply empirical evidence of its role in improving pedagogy. Virtual microscopy provides fresh opportunities for investigating user behavior during the histology learning process, through digitized histological slides. This study establishes how students' perceptions and user behavior data can be processed and analyzed using machine learning algorithms. These also provide predictive data called learning analytics that enable predicting students' performance and behavior favorable for academic success. This information can be interpreted and used for validating instructional designs. Data on the perceptions, performances, and user behavior of 552 students enrolled in a histology course were collected from the virtual microscope, Cytomine®. These data were analyzed using an ensemble of machine learning algorithms, the extra-tree regression method, and predictive statistics. The predictive algorithms identified the most pertinent histological slides and descriptive tags, alongside 10 types of student behavior conducive to academic success. We used these data to validate our instructional design, and align the educational purpose, learning outcomes, and evaluation methods of digitized histological slides on Cytomine®. This model also predicts students' examination scores, with an error margin of <0.5 out of 20 points. The results empirically demonstrate the value of a digital learning environment for both students and teachers of histology.
How Noisy is Too Noisy? The Impact of Data Noise on Multimodal Recognition of Confusion and Conflict During Collaborative Learning

Intelligent systems to support collaborative learning rely on real-time behavioral data, including language, audio, and video. However, noisy data, such as word errors in speech recognition, audio static or background noise, and facial mistracking in video, often limit the utility of multimodal data. It is an open question of how we can build reliable multimodal models in the face of substantial data noise. In this paper, we investigate the impact of data noise on the recognition of confusion and conflict moments during collaborative programming sessions by 25 dyads of elementary school learners. We measure language errors with word error rate (WER), audio noise with speech-to-noise ratio (SNR), and video errors with frame-by-frame facial tracking accuracy. The results showed that the model’s accuracy for detecting confusion and conflict in the language modality decreased drastically from 0.84 to 0.73 when the WER exceeded 20%. Similarly, in the audio modality, the model’s accuracy decreased sharply from 0.79 to 0.61 when the SNR dropped below 5 dB. Conversely, the model’s accuracy remained relatively constant in the video modality at a comparable level (> 0.70) so long as at least one learner’s face was successfully tracked. Moreover, we trained several multimodal models and found that integrating multimodal data could effectively offset the negative effect of noise in unimodal data, ultimately leading to improved accuracy in recognizing confusion and conflict. These findings have practical implications for the future deployment of intelligent systems that support collaborative learning in actual classroom settings.
AirSpec: A Smart Glasses Platform, Tailored for Research in the Built Environment

AirSpec is an extensible, environment-focused, research and development smart glasses platform that evolved from an existing open-source, psychophysiological monitoring system. We created custom supporting software toolkits that allow users to interact with the device, easily view real-time data, and perform remote data collection. In its base configuration, the system includes a variety of sensors that sample physiological and environmental signals and stream that data to a Bluetooth-connected client, either a phone running the AirSpec App or a Bluetooth-equipped computer via our website or a python script. In addition, AirSpecs have been made more extensible with multiple external electrical connections to support more applications and future sensor subsystems.
Crowd Behaviour Prediction using Visual and Location Data in Super-Crowded Scenarios

Predicting the future trajectory of a crowd is important for safety to prevent disasters such as stampedes or collisions. Extensive research has been conducted to explore trajectory prediction in typical crowd scenarios, where the majority of individuals can be easily identified. However, this study focuses on a more challenging scenario known as the super-crowd scene, wherein individuals within the crowd can only be annotated based on their heads. In this particular scenario, people’s re-identification process in tracking does not perform well due to a lack of clear image data. Our research proposes a clustering strategy to overcome people re-identification problems and predict the cluster crowd trajectory. Two-dimensional(2D) maps and multi-cameras will be used to capture full pictures of crowds in a location and extract the venue’s spatial data (see figure 1). The research methodology encompasses several key steps, including evaluating data extraction of the state-of-the-art methods, estimating crowd clusters, integrating 2D maps and multi-view fusion, and evaluating the proposed method on a dataset of multi-view videos collected in a real-world super-crowded scenario.
A Design Framework For Equitable Wearables

Wearables have become ubiquitous technologies that allow people to capture rich data to track their health information, immerse themselves in virtual reality, and stay connected. However, there is a consensus that wearables are only available to a select few due to cost, reducing the potential benefits they can provide to underserved populations. To ensure that wearables are designed more equitably and include the voices of underserved communities in the design process, I conducted a qualitative study with 19 participants from low-income communities and found that safety was a critical issue and that wearables can potentially be used to address their daily safety concerns. In this thesis position paper, I motivate the importance of codesigning wearables with and for underserved communities and propose a novel approach at the hardware and software levels for the development of safety-based wearable devices based on the needs of these communities.
Private, Fair and Secure Collaborative Learning Framework for Human Activity Recognition

Federated learning (FL), a decentralized machine learning technique, enhances privacy by enabling multiple devices to collaboratively train a model without transferring data to a central server. FL is used in Human Activity Recognition (HAR) problems, where multiple users generating private wearable data share models with a server to learn a useful global model. However, FL may compromise data privacy through model information sharing during training. Moreover, it adheres to a one-size-fits-all approach toward data privacy, potentially neglecting varied user preferences in collaborative scenarios such as HAR. In response to these challenges, this paper presents a collaborative learning framework integrating differential privacy (DP) and FL, thus providing a tailored approach to privacy protection. While some existing works integrate DP and FL, they do not allow clients to have different privacy preferences. In this work, we introduce a framework that allows different clients to have different privacy preferences and hence more flexibility in terms of privacy. In our framework, DP adds individualized noise to individual clients’ gradient updates for privacy. However, such noised updates can also be interpreted as an attack on the FL system. Defending against these attacks might result in excluding honest private clients altogether from training, posing a fairness concern. On the other hand, not having any defensive measures might allow malicious users to attack the system, posing a security issue. Thus, to address security and fairness, our framework incorporates a client selection strategy that protects the global model from malicious clients and provides fair model access to honest private clients. We have demonstrated the effectiveness of our system on a HAR dataset and provided insights into our framework’s privacy, utility, and fairness.
Recording multimodal pair-programming dialogue for reference resolution by conversational agents

Pair programming is a collaborative technique which has proven highly beneficial in terms of the code produced and the learning gains for programmers. With recent advances in Programming Language Processing (PLP), numerous tools have been created that assist programmers in non-collaborative settings (i.e., where the technology provides users with a solution, instead of discussing the problem to develop a solution together). How can we develop AI that can assist in pair programming, a collaborative setting? To tackle this task, we begin by gathering multimodal dialogue data which can be used to train systems in a basic subtask of dialogue understanding: multimodal reference resolution, i.e., understanding which parts of a program are being mentioned by users through speech or by using the mouse and keyboard.
Hyper-personalizing Common Norms through Principled Bespoke Generation

In our everyday lives, we often follow standardized norms based on the “standard person”. However, these existing norms, created by averaging data from large populations, often fail to fit an individual. It is also challenging to implement guidelines based on these norms due to various real-life constraints in our life. We propose a hyper-personalization approach, leveraging principled generation to overcome these limitations. We explore a method that deconstructs the standard norms into principles and embodiments, and then generates contextualized guidelines by producing embodiments based on these principles. We initially identified the efficacy of our research by designing a real-life actionable sleep recommendation system. Our next step is the personalization of vocabulary in children’s language learning. In the end, we aim to design and implement a framework that can be applicable to potential cases in the future.
Component attention network for multimodal dance improvisation recognition

Dance improvisation is an active research topic in the arts. Motion analysis of improvised dance can be challenging due to its unique dynamics. Data-driven dance motion analysis, including recognition and generation, is often limited to skeletal data. However, data of other modalities, such as audio, can be recorded and benefit downstream tasks. This paper explores the application and performance of multimodal fusion methods for human motion recognition in the context of dance improvisation. We propose an attention-based model, component attention network (CANet), for multimodal fusion on three levels: 1) feature fusion with CANet, 2) model fusion with CANet and graph convolutional network (GCN), and 3) late fusion with a voting strategy. We conduct thorough experiments to analyze the impact of each modality in different fusion methods and distinguish critical temporal or component features. We show that our proposed model outperforms the two baseline methods, demonstrating its potential for analyzing improvisation in dance.
Smart Contracts for Ethical Mobile Data Collection and Usage

The significant increase in data production resulting from the widespread adoption of mobile and IoT technologies has revolutionized healthcare but also presents significant privacy and ethical challenges. The field of medical data collection is no exception and has limitations in terms of the source, variety and quantity of records from studies on healthcare and wellness. One way to address this dilemma is the use of the Blockchain for patient data collection and use. The anonymity of a centralized network allows the patient’s identity to be protected. The structure formed by nodes allows the information to be always available and does not depend on a main server. The immutability of records in the chain ensures unambiguous traceability of information flow by the healthcare provider. Finally, the network’s consensus and reward mechanisms could motivate new users to participate in active sensing. In this article we will expose the architecture of an application that relies on the Blockchain to meet the above information needs by leveraging the potential of the Ethereum network. In addition, we present a use case where consciously collected data from our platform is used to train a machine learning model automatically, using a P2P Browser-Based Computational Notebook as execution and distribution environment.
Accelerating Knowledge Transfer by Sensing and Actuating Social-Cognitive States

Knowledge Transfer is one of the essential principles in education. The teacher’s knowledge is encoded through speech and writing and transmitted to the student. The student then decodes the transmitted information according to individual capabilities and absorbs it as knowledge. This paper presents an approach to accelerate knowledge transfer using sensor technology and social-cognitive states. So far, we have worked on quantifying meeting discussions, analyzing lecture studies, and estimating domain knowledge from web browsing. The contribution of this study is to estimate the degree of achievement of knowledge transfer and to accelerate it based on the estimated results.
Inflorescence: A Framework for Evaluating Fairness with Clustered Federated Learning

Measuring and ensuring machine learning model fairness is a challenging task that is especially difficult in federated learning (FL) settings where the model developer is not privy to clients’ local data. We propose Inflorescence, a framework that explores how the application of clustered FL strategies, which are designed to handle data distribution skew across federated clients, affects fairness. We share empirical simulation results quantifying the extent to which clustered FL impacts various group and individual fairness metrics, finding that it tends to improve fairness in terms of accuracy, precision, and error entropy, but not in terms of disparate impact or equal odds. We open-source the proposed framework as a Python package to facilitate research on fairness in distributed systems.
Estimating Sampling Rate of Human Activity Data from Accelerometer using Transformer-based Regression Model

We propose a method to estimate the sampling frequency from only the 3-axis acceleration data. The proposed method calculates the absolute difference between two consecutive samples, creates a histogram, and constructs a Transformer-based regression model.
Analysing Fairness of Privacy-Utility Mobility Models

Preserving the individuals’ privacy in sharing spatial-temporal datasets is critical to prevent re-identification attacks based on unique trajectories. Existing privacy techniques tend to propose ideal privacy-utility tradeoffs (PUT), however, largely ignore the fairness implications of mobility models and whether such techniques perform equally for different groups of users. The quantification between fairness and privacy of PUT models is still unclear and there exists limited metrics for measuring fairness in the spatial-temporal context. In this work, we define a set of fairness metrics designed explicitly for human mobility, based on structural similarity and entropy of the trajectories. Under these definitions, we examine the fairness of two state-of-the-art privacy-preserving models that rely on GAN and representation learning to reduce the re-identification rate of users. Our results show that these models violate individual fairness criteria, indicating that users with highly similar trajectories receive disparate privacy gain.
Demonstrating ProxiFit: Proximal Magnetic Sensing using a Single Commodity Mobile toward Holistic Weight Exercise Monitoring

Although many works bring exercise monitoring to smartphones and smartwatches, inertial sensors used in such systems require the device to be in motion to detect exercises. We demonstrate our full paper ProxiFit, a practical on-device exercise monitoring system capable of classifying and counting exercises despite the device being still. ProxiFit remotely detects adjacent exercises with magnetic field fluctuations induced by the motions of ferrous exercise equipment. Novel proximal sensing nature of ProxiFit (1) extends coverage of wearable exercise monitoring to exercises that do not involve device motion such as lower-body machine exercise, and (2) brings a new off-body exercise monitoring mode with line-of-sight screen visibility, namely signage mode, to a smartphone mounted in front of the user.
Binary Social Game: A Principle for Enhancing Desired Behaviors through Asynchronous Social Interactions

Within the HCI community, the impact of social dynamics on behavior change has gained considerable interest. This paper introduces the Binary Social Game (BSG), a principle that utilizes asynchronous social interactions to motivate desired behaviors. Initially, we developed GrowFlower based on the BSG principle, and its effectiveness was evaluated through a two-week wizard-of-oz study. After gaining insights from the preliminary study, we refined our approach and created GENGO, a bingo-style game that is specifically designed to encourage exercise as a targeted behavior. A subsequent two-month user study in real-world conditions, involving 82 participants, validated GENGO’s effectiveness in fostering consistent exercise habits. These findings highlight the potential of BSG as a useful tool for promoting desired behaviors within a group context, especially through asynchronous social interactions. This study contributes to the understanding of how social dynamics can be leveraged for behavior change.
Sandbox AI: We Don't Trust Each Other but Want to Create New Value Efficiently Through Collaboration Using Sensitive Data

This research deals with how to build reliable AI models using shared sensitive data. Confidential computing is gaining attention in AI services for a ubiquitous computing field. It makes legal and social regulations strict, including protecting privacy and ensuring security for sensitive data and AI models. As is also the case in ubiquitous computing research. The research phase also requires a manual trial-and-error process to create value through new combinations of shared data and AI models. Conventional confidential computing cannot realize the human workflow. This paper proposes and verifies “Sandbox AI” that can build precise AI models through efficient manual processes without revealing the shared data or models to one another. Sandbox AI utilizes privacy-preserving synthetic data generation and active learning on a confidential computing architecture. Specifically, Sandbox AI works in a sandbox container and generates synthetic data on the basis of real data, interactively extracts synthetic data that are useful for efficient model training while considering the real data distribution, discloses only the extracted synthetic data for annotation, and re-trains the model on the labeled synthetic data. Experimental results show that the achieved model accuracy is as good as that of conventional learning using real data with privacy violations.
The Third Workshop on Multiple Input Modalities and Sensations for VR/AR Interactions (MIMSVAI)

Rapid technological advances are expanding the practical applicability of virtual reality and/or augmented reality (VR/AR); however, the degree to which new users are able to interact with these technologies is limited by current input modalities. Gaining an intuitive grasp of VR/AR applications requires that users be immersed in the virtual environment, which in turn depends on the integration of multiple realistic sensory feedback mechanisms. This workshop will bring together researchers from the fields of UbiComp and VR/AR to explore alternative input modalities and sensory feedback systems to facilitate the design of coherent and engaging VR/AR experiences comparable to those in the real world.
Towards LLMs for Sensor Data: Multi-Task Self-Supervised Learning

LLMs for vision and NLP domain has been popular by the widespread use of ChatGPT and GPT-4. This paper tackles to build LLMs for sensor domain of one-dimensional signals whose downstream task is activity recognition and emotion detection. We propose a new architecture of Transformer-based self-supervised learner which we name SENvT. This SENvT builds the LLMs for sensor data using 7 pretext objectives in multi-task learning together with contrastive learning. Experimental results show these three. First, we obtained better results for contrastive learning and the masked token task but not for other pretext tasks. Second, the masked token task was better in 60% rather than in 10%. Third, the RGW worked best in accuracy while the masked token task worked best in F1.
11th International Workshop on Human Activity Sensing Corpus and Applications (HASCA)

The recognition of complex and subtle human behaviors from wearable sensors will enable next-generation human-oriented computing in scenarios of high societal value (e.g., dementia care). This will require a large-scale human activity corpus and much-improved methods to recognize activities and the context in which they occur. This workshop deals with the challenges of designing reproducible experimental setups, running large-scale dataset collection campaigns, designing activity and context recognition methods that are robust and adaptive, and evaluating systems in the real world. We wish to reflect on future methods, such as lifelong learning approaches that allow open-ended activity recognition. This year HASCA will welcome papers from participants to the Fifth Sussex-Huawei Locomotion and Transportation Recognition Challenge in a special session.
An Experimental Video Conference Platform to Bridge the Gap Between Digital and In-Person Communication

With many contemporary video conferencing platforms available, there is still a need for platforms that afford a researcher workflow to conduct controlled online experiments. We have developed an open source experimental video conferencing platform that enables researchers to design and conduct remote experiments. Our platform provides a high level of control over the user interface and video streams, which is essential for studying the differences between remote and in-person social interactions. We give an overview of our platform’s usage and architecture and conduct a take-home study (N=9) to evaluate how accessible our system is to potential new contributors. We also follow up with an initial evaluation of technical performance bottlenecks for when our experimental platform is deployed, and show that the computational resources increases per each video stream as well as the type of filters applied to each participant. We end with a short discussion on next steps and the experimental hub’s potential to be extended as a sandbox for testing browser based augmented reality (WebAR) filters to be adopted in interdisciplinary experimental procedures.