Reviewerly/exHarmony huggingface dataset:
Physiological Sensing for Media Perception &amp; Activity Recognition

Wearable sensors have the intriguing potential to continuously evaluate human physiological characteristics in real-time without being obtrusive. This thesis aims to incorporate physiological sensors data to investigate the Media Perception and Activity Recognition. Our primary research goals include (a) neural encoding-based psycho-acoustic attribute analysis for data sonification, (b) empirical evidence for perceptual subjectivity in neural encoding during human-media interactions, the impact of incorporating behavioral ratings, and (c) the efficacy of attention-based transformer models on physiological data on human activity recognition problems.
Non-verbal Signals in Oral History Archives

Oral History Archives (OHA) are a rich source of emotional narratives, encapsulating the personal stories of people across different demographics, historical periods, and cultures. Computational technologies have transformed the oral history archival field by facilitating the transcription and verbal content analysis of interview collections where manual inspection is too time-consuming. However, these methods fail to include the subjective part of the archives. In this project, we explore the potential of automatic breathing patterns and non-verbal cues analysis applied to OHA interviews to gain new insights into the individual and collective emotional responses across different demographics. The proposed framework will investigate if automatic breathing signal prediction enhances the performance of speech emotion recognition models and if a cross-dataset learning approach for breathing signal prediction and paralinguistics analysis will work in OHA. Next, we will further use the emotional information gathered to study cultural differences when it comes to narrating traumatic experiences, focusing on different OHA collections. Lastly, to enhance our research and the literature, we will also design emotion elicitation experiments to create new emotional speech breathing datasets.
A Deep Dive Into Neural Synchrony Evaluation for Audio-visual Translation

We present a comprehensive analysis of the neural audio-visual synchrony evaluation tool SyncNet. We assess the agreement of SyncNet scores vis-a-vis human perception and whether we can use these as a reliable metric for evaluating audio-visual lip-synchrony in generation tasks with no ground truth reference audio-video pair. We further look into the underlying elements in audio and video which vitally affect synchrony using interpretable explanations from SyncNet predictions and analyse its susceptibility by introducing adversarial noise. SyncNet has been used in numerous papers on visually-grounded text-to-speech for scenarios such as dubbing. We focus on this scenario which features many local asynchronies (something that SyncNet isn’t made for).
Towards Accessible Sign Language Assessment and Learning

Recently, a phonology-based sign language assessment approach has been proposed using sign language production acquired in 3D space using Kinect sensor. In order to scale the sign language assessment system to realistic application, there is need to reduce the dependency on Kinect, which is not accessible to wider community, and develop solutions that can potentially work with web-cameras. This paper takes a step in that direction by investigating sign language recognition and sign language assessment in 2D space either by dropping the depth coordinate in Kinect or using methods for skeleton estimation from videos. Experimental studies on Swiss German Sign Language corpus SMILE show that, while loss of depth information leads to considerable drop in sign language recognition performance, high level of sign language assessment performance can still be obtained.
Two-Step Gaze Guidance

One challenge of providing guidance for search tasks consists in guiding the user’s visual attention to certain objects in a potentially large search space. Previous work has tried to guide the user’s attention by providing visual, audio, or haptic cues. The state-of-the-art methods either provide hints pointing towards the approximate direction of the target location for a fast but less accurate search or require the user to perform a fine-grained search from the beginning for a precise yet less efficient search. To combine the advantage of both methods, we propose an interaction concept called Two-Step Gaze Guidance. The first-step guidance focuses on quick guidance toward the approximate direction, and the second-step guidance focuses on fine-grained guidance toward the exact location of the target. A between-subject study (N = 69) with five conditions was carried out to compare the two-step gaze guidance method with the single-step gaze guidance method. Results revealed that the proposed method outperformed the single-step gaze guidance method. More precisely, the introduction of Two-Step Gaze Guidance slightly improves the searching accuracy, and the use of spatial audio as the first-step guidance significantly helps in enhancing the searching efficiency. Our results also indicated several design suggestions for designing gaze guidance methods.
Multimodal Representation Learning For Real-World Applications

Multimodal representation learning has shown tremendous improvements in recent years. An extensive set of works for fusing multiple modalities have shown promising results on the public benchmarks. However, most famous works target unrealistic settings or toy datasets, and a considerable gap exists between the real-world implications of the existing methods. In this work, we aim to bridge the gap between the well-defined benchmark settings and the real-world use cases. We aim to explore architectures inspired by existing promising approaches that have the potential to be implemented in real-world instances. Moreover, we also try to move the research forward by addressing questions that can be solved using multimodal approaches and have a considerable impact on the community. With this work, we attempt to leverage the multimodal representation learning methods, which directly apply to real-world settings.
Investigating the relationship between dialogue and exchange-level impression

Multimodal dialogue systems (MDS) have recently attracted increasing attention. The automatic evaluation of user impression with spoken dialog at the dialog level plays a central role in managing dialog systems. A user usually forms an overall impression through the experience of each exchange of turns in the conversation. Thus, the user’s exchange-level sentiment should be considered when recognizing the user’s overall impression of the dialog. Previous research has focused on modeling user impressions during individual exchanges or during the overall conversation. Thus, the relationship between user sentiment at the exchange level and user impression at the dialog level is still unclear, and appropriately utilizing this relationship in impression analysis remains unexplored. In this paper, we first investigate the relation between sentiment at the exchange level and 18 labels that indicate different aspects of the user impression at the dialog level. Then, we present a multitask learning model (MTL) that uses exchange-level annotations to recognize dialog-level labels. The experimental results demonstrate that our proposed model achieves better performance at the dialog level, outperforming the single-task model by a maximum of 15.7%.
Pull Gestures with Coordinated Graphics on Dual-Screen Devices

A new class of dual-touchscreen device is beginning to emerge, either constructed as two screens hinged together, or as a single display that can fold. The interactive experience on these devices is simply that of two 2D touchscreens, with little to no synergy between the interactive areas. In this work, we consider how this unique, emerging form factor creates an interesting 3D niche, in which out-of-plane interactions on one screen can be supported with coordinated graphics in the other orthogonal screen. Following insights from an elicitation study, we focus on "pull gestures", a multimodal interaction combining on-screen touch input with in air movement. These naturally complement traditional multitouch gestures such as tap and pinch, and are an intriguing and useful way to take advantage of the unique geometry of dual-screen devices.
X-Norm: Exchanging Normalization Parameters for Bimodal Fusion

Multimodal learning aims to process and relate information from different modalities to enhance the model’s capacity for perception. Current multimodal fusion mechanisms either do not align the feature spaces closely or are expensive for training and inference. In this paper, we present X-Norm, a novel, simple and efficient method for bimodal fusion that generates and exchanges limited but meaningful normalization parameters between the modalities implicitly aligning the feature spaces. We conduct extensive experiments on two tasks of emotion and action recognition with different architectures including Transformer-based and CNN-based models using IEMOCAP and MSP-IMPROV for emotion recognition and EPIC-KITCHENS for action recognition. The experimental results show that X-Norm achieves comparable or superior performance compared to the existing methods including early and late fusion, Gradient-Blending (G-Blend) [44], Tensor Fusion Network, [48] and Multimodal Transformer [40], with a relatively low training cost.
Improving Supervised Learning in Conversational Analysis through Reusing Preprocessing Data as Auxiliary Supervisors

Emotions recognition systems are trained using noisy human labels and often require heavy preprocessing during multi-modal feature extraction. Using noisy labels in single-task learning increases the risk of over-fitting. Auxiliary tasks could improve the performance of the primary task learning during the same training – multi-task learning (MTL). In this paper, we explore how the preprocessed data used for creating the textual multimodal description of the conversation, that supports conversational analysis, can be re-used as auxiliary tasks (e.g. predicting future or previous labels and predicting the ranked expressions of actions and prosody), thereby promoting the productive use of data. Our main contributions are: (1) the identification of sixteen beneficially auxiliary tasks, (2) studying the method of distributing learning capacity between the primary and auxiliary tasks, and (3) studying the relative supervision hierarchy between the primary and auxiliary tasks. Extensive experiments on IEMOCAP and SEMAINE data validate the improvements over single-task approaches, and suggest that it may generalize across multiple primary tasks.
Impact of aesthetic movie highlights on semantics and emotions: a preliminary analysis

Aesthetic highlight detection is a challenge for understanding affective processes underlying emotional movie experience. Aesthetic highlights in movies are scenes with aesthetic values and attributes in terms of form and content. Deep understanding of human emotions while watching movies and automatic recognition of emotions evoked by watching movies are critically important for a wide range of applications, such as affective content creation, analysis, and summarization.
Speaker Motion Patterns during Self-repairs in Natural Dialogue

An important milestone for any agent in interaction with humans on a regular basis is to achieve natural and efficient methods of communication. Such strategies should be derived on the hallmarks of human-human interaction. So far, the work in embodied conversational agents (ECAs) implementing such signals has been predominantly through imitating human-like positive back-channels, such as nodding, rather than active interaction. The field of Conversation Analysis (CA) focusing on natural human dialogue suggests that people continuously collaborate on achieving mutual understanding by frequently repairing misunderstandings as they happen. Detecting repairs from speech in real-time is challenging, even with state-of-the-art Natural Language Processing (NLP) models. We present specific human motion patterns during key moments of interaction, namely self initiated self-repairs, which would help agents to recognise and collaboratively solve speaker trouble. The features we present in this paper are the pairwise joint distances of head and hands which are more discriminative than the positions themselves.
Detection-Identification Balancing Margin Loss for One-Stage Multi-Object Tracking

In recent years, one-stage multi-object tracking (MOT) methods, which jointly learn detection and identification in a single network, have attracted extensive attention, due to their efficiency. However, the negative transfer effects caused by the two conflicting objectives of detection and identification have rarely been explored. In this paper, we propose a Detection-Identification Balancing Margin (DIM) loss for minimizing the adverse effects caused by these two different objectives. The proposed DIM loss consists of Detection Margin (DM) loss and Identification Margin (IM) loss. DM loss forces features that are farther from the center of the foreground features than the defined margin due to identification learning to be converged to ensure accurate detection. IM loss enables the various feature representations that are essential for identification by intentionally spreading features that become overly clustered due to detection learning. The proposed DIM loss demonstrates competitive and balanced performance for MOT by providing a positive transfer for features that had a strong negative impact on detection and identification, respectively. (HOTA 61.5, MOTA 75.3, IDF1 75.6 on MOT16, and real-time rates of 25.9 fps were achieved)
Rain-Prior Injected Knowledge Distillation for Single Image Deraining

This paper makes efforts in improving the efficiency of deep networks for single image deraining with a newly proposed knowledge distillation framework. Specifically, we propose a rain-prior injected distillation scheme to transfer the knowledge from a large-scale teacher network to a more compact student network. Previous works directly calculate the distillation loss between the features extracted from the student and teacher networks. Differently, our distillation scheme adaptively removes the noisy background patterns by calculating the distillation loss based on the residual feature, which is inferred from the features extracted from the rain and ground truth images. This residual operation makes the student network focus on transferring only the knowledge on the rain streaks instead of the background, which facilitates more effective distillation results. Furthermore, our method can be applied to reduce both the network size and the deraining recurrence stage, which makes it a plug-and-play module that can be integrated into diverse existing deraining methods. Experimental results prove the efficiency of our method to build an efficient deraining network and the superiority over existing distillation methods.
Simultaneous Learning and Compression for Convolution Neural Networks

Neural network compression techniques almost always operate on pretrained filters. In this paper we propose a sparse training method for simultaneous compression and learning, which operates in the eigen space of the randomly initialized filters and learns to compactly represent the network as it trains from scratch. This eliminates the usual two-step process of having to first train the network, and then compressing it afterwards. To learn the sparse representations we enforce group L1 regularization on the linear combination weights of eigen filters. This results in the recombined filters which have low rank and can be readily compressed with standard pruning and low rank approximation methods. Moreover we show that the L1 norm of the linear combination weights can be used as a proxy for the filter importance for pruning. We demonstrate the effectiveness of our method by applying it to several CNN architectures, and show that our method directly achieves the best compression with competitive performance accuracy as compared to state of the art methods for compressing pre-trained networks.
Estimating Brain Age with Global and Local Dependencies

The brain age has been proven to be a phenotype of relevance to cognitive performance and brain disease. Achieving accurate brain age prediction is an essential prerequisite for optimizing the predicted brain-age difference as a biomarker. As a comprehensive biological characteristic, the brain age is hard to be exploited accurately with models using feature engineering and local processing such as local convolution and recurrent operations that process one local neighborhood at a time. Instead, Vision Transformers learn global attentive interaction of patch tokens, introducing less inductive bias and modeling long-range dependencies. In terms of this, we proposed a novel network for learning brain age interpreting with global and local dependencies, where the corresponding representations are captured by Successive Permuted Transformer (SPT) and convolution blocks. The SPT brings computation efficiency and locates the 3D spatial information indirectly via continuously encoding 2D slices from different views. Finally, we collect a large cohort of 22645 subjects with ages ranging from 14 to 97 and our network performed the best among a series of deep learning methods, yielding a mean absolute error (MAE) of 2.855 in validation set, and 2.911 in an independent test set.
Regional Saliency Map Attack for Medical Image Segmentation

State-of-the-art Deep Neural Networks (DNNs) are promoting medical image processing. However, DNNs are susceptible to adversarial attacks, which could significantly deteriorate model performances and pose a threat to clinical diagnoses. One common method for prevention is through adversarial training, which is highly dependent on harnessing adversarial examples during the training stage. However, adversarial medical examples generated by many existing works are too perceptible to be adversarial examples. To improve the imperceptibility, we proposed a Regional Saliency Map Attack that generates an adversarial example by only perturbing a small number of pixels. Extensive experiments have shown that, on average, our method caused the same degradation in model performance by quantitatively less perceptible perturbations. Visualisations have also verified that the improvement in imperceptibility in an image is both global and regional.
Tuning Neural ODE Networks to Increase Adversarial Robustness in Image Forensics

Although deep-learning-based solutions are pervading different application sectors, many doubts have arisen about their reliability and, above all, their security against threats that can mislead their decision mechanisms. In this work, we considered a particular kind of deep neural network, the Neural Ordinary Differential Equations (N-ODE) networks, which have shown intrinsic robustness against adversarial samples by properly tuning their tolerance parameter at test time. Their behaviour has never been investigated in image forensics tasks such as distinguishing between an original and an altered image. Following this direction, we demonstrate how tuning the tolerance parameter during the prediction phase can control and increase N-ODE’s robustness versus adversarial attacks. We performed experiments on basic image transformations used to generate tampered data, providing encouraging results in terms of adversarial rejection and preservation of the correct classification of pristine images.
Indoor Target-Driven Visual Navigation based on Spatial Semantic Information

Target-driven visual navigation is a widely focused learning-based approach in the field of computer vision. However, it faces two major challenges: poor generalization ability to unknown scenes, and poor navigation performance for increased number of scenes. In this paper, an end-to-end target-driven visual navigation method, which uses Spatial Semantic Information (SSI) to navigate the agent to the target, is presented. To fully integrate the spatial and semantic information in the scene, visual information is encoded into an 8-D spatial context vector. In addition, the size of detected bounding box is used to improve the reward function in end-to-end learning to solve the problem of sparse rewards. Experiments in interactive environment dataset AI2-THOR show that compared with state-of-the-art approaches, our approach has a higher success rate and a better route to target.
A Novel Contrastive Learning Framework for Self-Supervised Anomaly Detection

Anomaly detection is significant in the field of computer vision and refers to identifying those samples in dataset that are different from normal samples. In practice, abnormal products are rare and anomaly detection usually calculates the difference between the inputs and the reconstructed images by reconstruction-based methods. Contrastive learning both maximizes the similarity between a sample and its augmentations, and the differences between different samples, which is suitable for improving the detection capability of the autoencoder. Inspired by this, we design a novel contrastive learning architecture for anomaly detection. In this work, we make reasonable sample pairs to simulate possible real anomalies and maximizes the distance between normal and abnormal samples. Remarkably, our approach improves the vanilla autoencoder model by 14.4% in terms of the AUROC score on the MVTec AD.
A Novel Rank Correlation Measure for Manifold Learning on Image Retrieval and Person Re-ID

Effectively measuring similarity among data samples represented as points in high-dimensional spaces remains a major challenge in retrieval, machine learning, and computer vision. In these scenarios, unsupervised manifold learning techniques grounded on rank information have been demonstrated to be a promising solution. However, various methods rely on rank correlation measures, which often depend on a proper definition of neighborhood size. On current approaches, this definition may lead to a reduction in the final desired effectiveness. In this work, a novel rank correlation measure robust to such variations is proposed for manifold learning approaches. The proposed measure is suitable for diverse scenarios and is validated on a Manifold Learning Algorithm based on Correlation Graph (CG). The experimental evaluation considered 6 datasets on general image retrieval and person Re-ID, achieving results superior to most state-of-the-art methods.
Random Generated Dictionaries for Convolutional Sparse Coding: An ELM Interpretation for Simple CSC Applications

The most basic ELM (extreme learning machines) architecture consists on a single-hidden layer feedforward neural network, with random input weights, plus a densely connected output layer whose weights must be learned. Among other interpretations, it can be understood as using an untrained dictionary (with random entries) along with a non-linear activation function to obtain a representation. Compared to Neural Networks (NN) or Convolutional NN (CNN), ELM is very fast to train.Inspired by the ELM methodology, in this paper we explore the usefulness of using a randomly generated filterbank (FB) as the convolutional dictionary in convolutional sparse coding (CSC) representations and assess its performance for simple applications such denoising and super resolution, when compared to learned FBs. Our main conclusions are that a randomly generated FB (i) has a competitive (restoration) performance when compared to a learned FB, (ii) its performance depends on the actual distribution of its values, i.e. Gaussian, uniform, lognormal, etc., and problem, and (iii) it may ease or potentially eliminate the need for the CDL (convolutional dictionary learning) step in CSR’s applications.