Reviewerly/exHarmony huggingface dataset:
Gaze movement operability and sense of spatial presence assessment while operating a robot avatar

The utilization of teleoperated robots has great potential for expanding the range of human social activities, and there has been much anticipation for this technology in recent years. To use teleoperated robots as human surrogates, it is essential to develop a teleoperation system that provides the operator with a high level of presence. Conventional robot avatars are generally operated by synchronizing the head with the operator. However, eye movements are essential for human gaze, and conventional operation (i.e., head movement synchronization only) may reduce perceived presence. In this study, we used a robot avatar that could move its head and eyeballs. We examined the effects of the synchronization of eye movements with the operator on the operability of the robot and the presence perceived by the operator. We conducted an experiment in which participants performed a task where they looked at an object under two conditions: head or head-eye synchronization between the participant and robot avatar. We investigated the effects of eye movement synchronization on task performance, spatial presence, and workload. The results showed that synchronization of eye movements improved operability and reduced workload; however, it did not improve presence. This study suggests that synchronization of eye movements may make the teleoperation of an avatar robot more comfortable and provides meaningful insights into research approaches for the improving presence of an avatar robot.
Ambient Intelligence Telework System to Enhance Concentration for Each Individual: Easy to Concentrate Based on Neuroticism

In recent years, telework has enormously increased because of the COVID-19 pandemic. However, some people believe that teleworkers can concentrate the same as in-person office work, while others do not. Previous studies showed that the work environment necessary for concentration tends to differ depending on the worker's individual attributes. The goal of this study is to propose an Ambient Intelligence (AMI) telework system suited for each individual to enhance concentration. In this paper, we used two videos with different levels of diligence on a task to investigate what kind of environment makes it easier to concentrate, depending on the degree of neuroticism. Concentration was estimated using the auditory steady-state response (ASSR), which is an oscillatory brain signal elicited by repetitive auditory stimulation and used as a hearing test for infants and children. The changes in the power spectral density of the ASSR with concentration were increased by comparing EEG at rest and during concentration. The effect of the two types of videos on concentration was investigated by evaluating the relationship between neuroticism scores and the power spectral density of the ASSR. The results showed that there was a significant difference in concentration influenced by the two types of videos between the high and low neuroticism score groups. In addition, a negative correlation was found between the neuroticism score and the concentration influenced by the two types of videos. We found that people with lower neuroticism tended to easy to concentrate on their work after seeing someone working hard whereas people with higher neuroticism tended to easy to concentrate on their work after seeing someone working lazily. The experimental results suggest the possibility of constructing an AMI system suited to each individual that enhances concentration.
Markerless 3D Pose Estimation System for Mouse Musculoskeletal Model Using DeepLabCut and Multiple RGB-D Cameras

Animal experiments play an important role in basic research as well as in applied research such as new drug development. In particular, the mouse is one of the most valuable laboratory animals in medical research because of its many advantages, such as its small size and ease of breeding. However, evaluations of behavioral animal experiment analysis are often conducted visually by observers, and there are problems with subjective evaluation and human error. An objective and automatic mouse analysis system is needed to solve these problems. Although several analysis systems have been developed in recent years, these systems have problems such as rough behavioral classification and attachment of markers that may affect mouse behavior. In this study, we proposed a detailed behavioral analysis system for a mouse without markers using six RGB-D cameras and video tracking based on deep learning. As a result, estimation with an average error of 5 to 10 mm at keypoints was achieved with this system.
Nursing Care Teaching System Based on Mixed Reality for Effective Caregiver-Patient Interaction

In the nursing care tasks such as assistance for transferring and walking, it is necessary to provide appropriate nursing care movements depending on factors such as the patient's pose and the degree of disability. However, for novice caregivers to practice and learn appropriate nursing care, they must practice for a long time under the guidance of skilled caregivers. To solve this problem, we propose a novel framework for a system that teaches appropriate nursing care actions according to the current situation. The realization of such a teaching system requires technology to recognize the current situation and effectively teach the interaction between the caregiver and the patient. In this article, we propose a system that integrates depth camera-based pose estimation of the patient and Mixed Reality (MR) technology to present the target motion of the patient to a caregiver. To accurately present the patient's target pose to the novice caregivers, our system displays an avatar showing the patient's ideal animation overlaid on the actual patient. Experimental results show that our system can accurately instruct the caregiver about the patient's target pose in each movement procedure.
Reduced Skeleton Representation for Action Recognition on Graph Convolutional Neural Networks

In this paper, we propose a reduced data representation for skeleton-based action recognition on GCNs and examine the method's influence on action recognition performance with respect to computational resources. With the straightforward strategy of omitting skeleton joints that are just mildly relevant, we want to point out that not only general reduction methods are applicable for decreasing a model's complexity but problem-specific options can likewise have a significant influence by also keeping a good accuracy. The trade-off between system accuracy and required computational resources becomes particularly relevant with an increasing focus on green computing. Evaluated on three recent GCN approaches on the NTU RGB+D dataset, our reduced skeleton representation decreases the training time by 27.4% and memory consumption by 28.6%, while decreasing the accuracy by only 1.3 ppt on average. This demonstrates the importance of descriptive joints for skeleton-based action recognition and the ability to optimize the trade-off between system performance and required resources at a representational level.
Activity Monitoring and Management in Avatar-based Interactions

This paper describes the role of activity monitoring and management in a software platform (cybernetic avatar (CA) platform) to support avatar-based interactions. Avatar-based interaction differs in various aspects from conventional face-to-face interaction. For instance, it offers anonymity and parallel operability (i.e., a single operator can control multiple interactions simultaneously). On the other hand, it imposes a large burden on the operator due to the required avatar operations. Reducing the operator's burden is a crucial issue to consider in developing such a system. From the viewpoint of data analysis, it is easy to capture and record the activities of participants in avatar-based interaction because those interactions are carried out by explicit avatar operations via a network. In addition, the activities of human participants in such interaction are observed by sensor networks. Therefore, the data from avatar monitoring should contain many kinds of information on the human operators and interaction partners interacting with each other via the avatars. The activity monitoring and management functions enable various services to support human activities using avatars, including reduction of the operational burden. In this paper, we also address an operation recommendation system, as an example application of activity monitoring, which offers situation-dependent recommendations for the operator's speech and operations to reduce the operator's burden. Experimental results support the effectiveness of the proposed system.
Pointing Gesture Interface with Fovea-Lens Camera by Deep Learning using Skeletal Information

This paper describes a pointing gesture interface based on deep learning using RGB images acquired by a fovea lens camera as input. This interface consists of hand gesture recognition to determine if the operator is pointing and pointing position estimation. The interface first uses OpenPose to estimate the operator's skeleton from images acquired with the fovea lens camera, which mimics the characteristics of human vision. Next, a hand image is cropped based on the estimated wrist coordinates, and hand gesture recognition is performed. Then, the system classifies which of the pre-defined positions the operator is pointing to by deep learning using the estimated joint positions as input. Evaluation experiments showed that hand gesture recognition achieved an accuracy of more than 95 %. The pointing position estimation achieved an accuracy of more than 85 % when the distance between the camera and the operator was 3 m. In addition, the advantages and challenges of using a fovea lens camera compared to using a standard lens camera and a wide-angle lens camera were discussed.
RULe: Relocalization-Uniformization-Landmark Estimation Network for Real-Time Face Alignment in Degraded Conditions

Face alignment refers to the process of estimating the position of a number of salient landmarks on face images or videos, such as mouth and eye corners, nose tip, etc. With the availability of large annotated databases and the rise of deep learning-based methods, face alignment as a domain has matured to a point where it can be applied in more or less unconstrained conditions, e.g. non-frontal head poses, presence of heavy make-up or partial occlusions. However, when considering real-case alignment on videos with possibly low frame rates, we need to make sure that the algorithms are robust to jittering of the face bounding box localization, low-resolution of the face crops, possible bad environmental lighting, brightness, and presence of noise. To tackle these issues, we propose RULe, a three-staged Relocalization-Uniformization-Landmark Estimation network. In the first stage, an initial loosely localized bounding box gets refined to output a well centered face crop, thus reducing the variability of the images prior to passing them to the subsequent stage. Then, in the second stage, the face style is uniformized (using adversarial learning as well as perceptual losses) to correct low resolution or variations of brightness/contrast. Finally, the third stage outputs a precise landmark estimation given such enhanced face crop using a cascaded compact model trained using hint-based knowledge distillation. We show through a variety of experiments that RULe achieves real-time face alignment with state-of-the-art precision in heavily degraded conditions.
Improving Performance of Facial Biometrics With Quality-Driven Dataset Filtering

Advancements in deep learning techniques and availability of large scale face datasets led to significant performance gains in face recognition in recent years. Modern face recognition algorithms are trained on large-scale in-the-wild face datasets. At the same time, many facial biometric applications rely on controlled image acquisition and enrollment procedures (for instance, document security applications). That is why such face recognition approaches can demonstrate the deficiency of the performance in the target scenario (ICAO-compliant images). However, modern approaches for face image quality estimation may help to mitigate that problem. In this work, we introduce a strategy for filtering training datasets by quality metrics and demonstrate that it can lead to performance improvements in biometric applications that rely on face image modality. We filter the main academic datasets using the proposed filtering strategy and present performance metrics.
Investigating Visual Features for Cognitive Impairment Detection Using In-the-wild Data

Early detection of dementia has attracted much research interest due to its crucial role in helping people get suitable treatment or care. Video analysis may provide an effective approach for detection, with low cost and effort compared to current expensive and intensive clinical assessments. This paper investigates the use of a range of visual features - eye blink rate (EBR), head turn rate (HTR) and head movement statistical features (HMSF) - for identifying neurodegenerative disorder (ND), mild cognitive impairment (MCI) and functional memory disorder (FMD). These features are used in a noval multiple thresholds approach, which is applied to an in-the-wild video dataset which includes data recorded in a range of challenging environments. A combination of EBR and HTR gives 78 % accuracy in a three-way classification task (ND/MCI/FMD) and 83%, 83% and 92%, respectively, for the two-way classifications ND/MCI, ND/FMD and MCI/FMD. These results are comparable to related work that uses more features from different modalities. They also provide evidence to support the possibility of an in-the-home detection process for dementia or cognitive impairment.
Are Face Detection Models Biased?

The presence of bias in deep models leads to unfair outcomes for certain demographic subgroups. Research in bias focuses primarily on facial recognition and attribute prediction with scarce emphasis on face detection. Existing studies consider face detection as binary classification into ‘face’ and ‘non-face’ classes. In this work, we investigate possible bias in the domain of face detection through facial region localization which is currently unexplored. Since facial region localization is an essential task for all face recognition pipelines, it is imperative to analyze the presence of such bias in popular deep models. Most existing face detection datasets lack suitable annotation for such analysis. Therefore, we web-curate the Fair Face Localization with Attributes (F2LA) dataset and manually annotate more than 10 attributes per face, including facial localization information. Utilizing the extensive annotations from F2LA, an experimental setup is designed to study the performance of four pre-trained face detectors. We observe (i) a high disparity in detection accuracies across gender and skin-tone, and (ii) interplay of confounding factors beyond demography. The F2LA data and associated annotations can be accessed at http://iab-rubric.org/index.php/F2LA.
Practical Parametric Synthesis of Realistic Pseudo-Random Face Shapes

There is a growing demand for populating virtual worlds with large numbers of realistic-looking characters. Besides hand-crafted characters like the main protagonists in video games, the virtual worlds may also need massive numbers of secondary characters. Manual authoring of their features is not usually practical. For parametric models of human faces, a naive approach randomizes all the parameters of the human face to generate a random one. However, the uniform or hand-crafted distribution of the shape authoring parameters is unlikely to represent value ranges and correlations present naturally in human faces. The paper proposes a simple automated method for generating realistic-looking head shapes via learned mapping between latent space like the FaceNet embedding and the explicit parametric space used by the character modeling tools. Our approach is simple, robust, and can efficiently generate a large variety of head shapes with a predictable dissimilarity.
Activation Template Matching Loss for Explainable Face Recognition

Can we construct an explainable face recognition network able to learn a facial part-based feature like eyes, nose, mouth and so forth, without any manual annotation or additionalsion datasets? In this paper, we propose a generic Explainable Channel Loss (ECLoss) to construct an explainable face recognition network. The explainable network trained with ECLoss can easily learn the facial part-based representation on the target convolutional layer, where an individual channel can detect a certain face part. Our experiments on dozens of datasets show that ECLoss achieves superior explainability metrics, and at the same time improves the performance of face verification without face alignment. In addition, our visualization results also illustrate the effectiveness of the proposed ECLoss.
Human Pose Estimation with Shape Aware Loss

Although the mean square error (mse) of heatmap is an intuitive loss for heatmap-based human pose estimation, the joints localization accuracy may not be improved when heatmap mse reduces. In this paper, we show that a great cause for such misalignment is the unnecessary requirement from heatmap mse on the irrelevant Gaussian parameter, i.e. maximum. The coordinate prediction is precise as long as the probability distribution held by the predicted heatmap is a well-shaped Gaussian distribution and has the same center as the ground truth. However, heatmap mse unnecessarily requires the Gaussian distribution to hold the same maximum as the ground truth. Correspondingly, we introduce mse on the image gradients of the target and predicted heatmap (referred to as gradmap mse) to focus on the shape of the heatmap. Combining heatmap and gradmap mse, we propose a simple yet effective Shape Aware Loss (SAL) method. Being model-agnostic, our method can benefit various existing models. We apply SAL to the three latest network architectures and obtain performance improvements for all of them. Comparisons of the visualized predicted heatmaps further prove the effectiveness of the proposed method.
TransFS: Face Swapping Using Transformer

This paper proposes a Transformer based face swapping model, namely, TransFS. The proposed model mainly solves two current problems of face swapping: 1) the face swapping result does not fully preserve pose and expression of the target face as expected; 2) most of the existing models fail to accomplish high-quality face swapping on high-resolution images. To address these two challenges, we first propose a Cross- Window Face Encoder based on Swin Transformer that learns rich facial features including poses and expressions. Then, we devise an Identity Generator to reconstruct high-resolution images of specific identity with high quality while utilizing the Transformer attention mechanism to increase identity information retention. Finally, a Face Conversion Module is proposed to transform the source identity reconstructed image into the target face image to synthesize the final face swapping result while maintaining the details of pose and expression of the target face. Through extensive experiments, our method not only accomplishes face swapping for low-resolution images with arbitrary identities, but also accomplishes face swapping for high-resolution images. Furthermore, our method achieves the state-of-the-art performance in pose and expression controls compared to other methods.
Self-supervised Learning for Fine-grained Ethnicity Classification under Limited Labeled Data

Human faces are always determined by genes and other external causes, such as geographical environment, which makes it possible for us to predict ethnicity according to the faces. However, it remains a challenging task due to the tiny differences in faces for various ethnicities, which is hard for human beings to tell, especially for ethnicities on the same continent, e.g., East Asia. Although some strongly-supervised methods have demonstrated their feasibility in this task, they cease to be effective when suffering from data-hungry issues in practice. This paper proposes a novel self-supervised model with a polynomial stacked attention mechanism to well excavate distinctions across different nations under limited labeled data. And we also construct a new ethnicity dataset named Cupid which observably extends the scale and categories of ethnic data compared to the existing datasets. Extensive experiments confirm that our method achieves the state-of-the-art results on both the Asian Face dataset and our proposed Cupid dataset.
The Role of Vocal Persona in Natural and Synthesized Speech

The inclusion of voice persona in synthesized voice can be significant in a broad range of human-computer-interaction (HCI) applications, including augmentative and assistive communication (AAC), artistic performance, and design of virtual agents. We propose a framework to imbue compelling and contextually-dependent expression within a synthesized voice by introducing the role of the vocal persona within a synthesis system. In this framework, the resultant ‘tone of voice’ is defined as a point existing within a continuous, contextually-dependent probability space that is traversable by the user of the voice. We also present initial findings of a thematic analysis of 10 interviews with vocal studies and performance experts to further understand the role of the vocal persona within a natural communication ecology. The themes identified are then used to inform the design of the aforementioned framework.
PhygitalNet: Unified Face Presentation Attack Detection via One-Class Isolation Learning

Face biometric systems are shown to be vulnerable to various kinds of presentation attacks including physical and digital attacks. Existing research generally focuses on individual attacks and very few focus on generalizability across digital and physical attacks. In this research, we propose PhygitalNet model that generalizes to both physical and digital presentation attacks on face biometric systems. The proposed model is based on novel one-class iSOLatiOn Learning (SOLO Learning) which is a two-step training process aimed at reducing of the covariate shift between the bonafide samples of the physical as well as digital attack dataset in the pre-training step. In the downstream step, the algorithm introduces a novel single-class iSOLatiOn loss (SOLO loss) function that isolates the samples belonging to the bonafide class away from the samples of the attacked class for both the attack methods. Experimental results show that PhygitalNet achieves a significant performance gain when compared with the baseline techniques, evaluated on a combination of MLFP, MSU-MFSD dataset (for physical attack) and FaceForensics++ (for digital attack) datasets.
Adversarial 3D Face Disentanglement of Identity and Expression

We propose a new framework to decompose 3D facial shape into identity and expression. Existing 3D face disentanglement methods assume the presence of a corresponding neutral (i.e. identity) face for each subject. Our method designs an identity discriminator to obviate this requirement. This is a binary classifier that determines if two input faces are from the same identity, and encourages the synthesised identity face to have the same identity features as the input face and to approach the ‘apathy’ expression. To this end, we take advantage of adversarial learning to train a PointNet-based variational auto-encoder and discriminator. Comprehensive experiments are employed on CoMA, BU3DFE, and FaceScape datasets. Results demonstrate state-of-the-art performance with the option of operating in a more versatile application setting of no known neutral ground truths. Code is available at https://github.com/rmraaron/FaceExpDisentanglement.