Reviewerly/exHarmony huggingface dataset:
Dynamic Speech Endpoint Detection with Regression Targets

Interactive voice assistants have been widely used as input interfaces in various scenarios, e.g. on smart home devices, wearables and on AR devices. Detecting the end of a speech query, i.e. speech end-pointing, is an important task for voice assistants to interact with users. Traditionally, speech end-pointing is based on pure classification methods along with arbitrary binary targets. In this paper, we propose a novel regression-based speech end-pointing model, which enables an end-pointer to adjust its detection behavior based on the context of user queries. Specifically, we present a pause modeling method and show its effectiveness for dynamic end-pointing. Based on our experiments with vendor-collected smartphone and wearables speech queries, our strategy shows a better trade-off between end-pointing latency and accuracy, compared to the traditional classification-based method. We further discuss the benefits of this model and generalization of the framework in the paper.
Weight-Based Mask For Domain Adaptation

In computer vision, unsupervised domain adaptation (UDA) is an approach to transferring knowledge from a label-rich source domain to a fully-unlabeled target domain. Conventional UDA approaches have two problems. The first problem is that a class classifier can be biased to the source domain because it is trained using only source samples. The second is that previous approaches align image-level features regardless of foreground and background, although the classifier requires foreground features. To solve these problems, we introduce Weight-based Mask Network (WEMNet) composed of Domain Ignore Module (DIM) and Semantic Enhancement Module (SEM). DIM obtains domain-agnostic feature representations via the weight of the domain discriminator and predicts categories. In addition, SEM obtains class-related feature representations using the classifier weight and focuses on the foreground features for domain adaptation. Extensive experimental results reveal that the proposed WEMNet outperforms the competitive accuracy on representative UDA datasets.
In Search of Strong Embedding Extractors for Speaker Diarisation

Speaker embedding extractors (EEs), which map input audio to a speaker discriminant latent space, are of paramount importance in speaker diarisation. However, there are several challenges when adopting EEs for diarisation, from which we tackle two key problems. First, the evaluation is not straightforward because the required features differ between speaker verification and diarisation. We show that better performance on widely adopted speaker verification evaluation protocols does not lead to better diarisation performance. Second, embedding extractors have not seen utterances in which multiple speakers exist. These inputs are inevitably present in speaker diarisation because of overlapped speech and speaker changes; they degrade the performance. To mitigate the first problem, we generate speaker verification evaluation protocols that better mimic the diarisation scenario. We propose two data augmentation techniques to alleviate the second problem, making embedding extractors aware of overlapped speech or speaker change input. One technique generates overlapped speech segments, and the other generates segments where two speakers utter sequentially. Extensive experimental results using three state-of-the-art speaker embedding extractors demonstrate that both proposed approaches are effective.
Bias Identification with RankPix Saliency

Saliency methods are critical tools that allow the estimation of the most important features of an input image that contribute to the network’s prediction. These tools are pivotal in high-stakes applications such as medical diagnosis or autonomous driving. Additionally, these tools can help identify models’ biasedness, such as a strong prior on object placement, easily distinguishable background features, or frequent object co-occurrence. We introduce RankPix, a novel saliency method for visual bias identification in image classification tasks. RankPix is a derivative-free approach that allows the identification of a minimum subset of pixels/features at a given network layer that changes the output of a classifier. Surprisingly, this approaches provides equivalent performance to gradient-based approaches on the standard pointing game benchmark. More interestingly, RankPix outperforms traditional approaches for systematic bias identification.
Boosting Person Re-Identification with Viewpoint Contrastive Learning and Adversarial Training

Person re-identification (ReID) aims at retrieving a person of interest across multiple cameras. Despite significant progress in person ReID, viewpoint variation remains an obstacle to extracting discriminative features for retrieval. To address this problem, we propose a Viewpoint-Robust Network (VRN) based on contrastive learning and adversarial training to boost person ReID. Specifically, a View-point Confusion (VC) module is proposed to conceal viewpoint information to extract viewpoint-agnostic features. We employ viewpoint contrastive learning to discriminate viewpoints, and then conversely ignore the viewpoint information by adversarial training. Besides, an ID Prototype (IDP) module further enhances the network by introducing a confidence-weighted IDP as a viewpoint-robust ID representation and conducting contrastive metric learning with an IDP triplet loss. Extensive experiments demonstrate the proposed method achieves state-of-the-art performance on widely used datasets Market1501 and MSMT17. Visualization of retrieval results illustrates the effectiveness and robustness of the proposed method.
Quantum Graph Transformers

We propose Quantum Graph Transformers (QGT), a novel approach for realizing the Transformer architecture for graph learning with quantum processors. QGT is built on top of the Graph Trans-former (GT) architecture and addresses the main challenge of mapping GT basic functions such as node encodings, graph structure, all-to-all connectivity, and message passing to quantum computing primitives and processors. We empirically demonstrate the training and inference efficacy of our proposed QGT architecture for the graph classification task on quantum devices over various graph datasets.
Can Spoofing Countermeasure And Speaker Verification Systems Be Jointly Optimised?

Spoofing countermeasure (CM) and automatic speaker verification (ASV) sub-systems can be used in tandem with a backend classifier as a solution to the spoofing aware speaker verification (SASV) task. The two sub-systems are typically trained independently to solve different tasks. While our previous work demonstrated the potential of joint optimisation, it also showed a tendency to over-fit to speakers and a lack of sub-system complementarity. Using only a modest quantity of auxiliary data collected from new speakers, we show that joint optimisation degrades the performance of separate CM and ASV sub-systems, but that it nonetheless improves complementarity, thereby delivering superior SASV performance. Using standard SASV evaluation data and protocols, joint optimisation reduces the equal error rate by 27% relative to performance obtained using fixed, independently-optimised subsystems under like-for-like training conditions.
Rethinking Implicit Neural Representations For Vision Learners

Implicit Neural Representations (INRs) are powerful to parameterize continous signals in computer vision. However, almost all INRs methods are limited to low-level tasks, e.g., image/video compression, super-resolution, and image generation. The questions on how to explore INRs to high-level tasks and deep networks are still under-explored. Existing INRs methods suffer from two problems: 1) narrow theoretical definitions of INRs are inapplicable to high-level tasks; 2) lack of representation capabilities to deep networks. Motivated by above facts, we reformulate the definitions of INRs from a novel perspective, and propose an innovative Implicit Neural Representation Network (INRN), which is the first study of INRs to tackle both low-level and high-level tasks. Specifically, we present three key designs for basic blocks in INRN along with two different stacking ways and corresponding loss functions. Extensive experiments with analysis on both low-level task (image fitting) and high-level vision tasks (image classification, object detection, instance segmentation) demonstrate the effectiveness of the proposed method.
Robust Video Anomaly Detection Framework via Prior Knowledge and Multi-Path Frame Prediction

Video anomaly detection aims to automatically detect abnormal objects or behaviors. Most existing methods tackle the problem by minimizing the reconstruction errors stemming from the lack of anomalous data, which leads to poor interpretability and robustness. Focus on the context-dependent nature of anomaly detection, a robust unsupervised Video Anomaly Detection framework based on Knowledge and Frame Prediction is proposed, called VAD-KFP. Prior knowledge which contains the context of anomaly is introduced into the multi-path frame prediction network through multi-layer Graph Convolutional Networks. By integrating the prior knowledge to accurately define anomalies, VAD-KFP is robust to different scenarios and is able to recognize the type of anomaly. An extensive range of experiments have been conducted on three benchmarks, the results of which indicate that our method outperforms strong baselines. Specifically, VAD-KFP obtains an AUROC score of 91.6% for the Avenue dataset.
Towards Reliable Image Outpainting: Learning Structure-Aware Multimodal Fusion with Depth Guidance

Image outpainting technology generates visually plausible content regardless of authenticity, making it unreliable to be applied in practice. Thus, we propose a reliable image outpainting task, introducing the sparse depth from LiDARs (Light Detection And Ranging devices) to extrapolate authentic RGB scenes. The large field view of LiDARs allows it to serve for data enhancement and further multimodal tasks. Concretely, we propose a Depth-Guided Outpainting Network to model different feature representations of two modalities and learn the structure-aware cross-modal fusion. And two components are designed: 1) The Multimodal Learning Module produces unique depth and RGB feature representations from the perspectives of different modal characteristics. 2) The Depth Guidance Fusion Module leverages the complete depth modality to guide the establishment of RGB contents by progressive multimodal feature fusion. Furthermore, we specially design an additional constraint strategy consisting of Cross-modal Loss and Edge Loss to enhance ambiguous contours and expedite reliable content generation. Extensive experiments on KITTI and Waymo datasets demonstrate our superiority over the state-of-the-art method, quantitatively and qualitatively.
Learning to Explain: a Gradient-based Attribution Method for Interpreting Super-Resolution Networks

DNN-based super-resolution(SR) models inherit the black-box nature of DNN and present low transparency. However, few works focus on interpreting low-level SR models and the limited existing gradient-based interpretability methods often require large computational costs and produce spurious/noisy feature attributions with a mixture of positive and negative signals. In this paper, we propose a gradient-based attribution method L2X(Learning to eXplain) to provide post-hoc visualization and interpretation for SR models by quantifying the attribution of individual features with regard to the SR output and generating a heatmap in pixel/input space. L2X relies on the forward-pass activations and efficiently propagates the important signal from the output neuron through the layers to the input in one pass. Besides, both positive and negative attributions are taken into account during the back-propagation process. We conduct cross-architectural analysis on State-of-the-Art SR networks and investigate the inner workings of them. We experimentally demonstrate the potential of L2X as a research tool to diagnose and visualize the most relevant features of the current SR model and offer insights into SR models to make further improvements.
Waveform Boundary Detection for Partially Spoofed Audio

The present paper proposes a waveform boundary detection system for audio spoofing attacks containing partially manipulated segments. Partially spoofed/fake audio, where part of the utterance is replaced, either with synthetic or natural audio clips, has recently been reported as one scenario of audio deepfakes. As deepfakes can be a threat to social security, the detection of such spoofing audio is essential. Accordingly, we propose to address the problem with a deep learning-based frame-level detection system that can detect partially spoofed audio and locate the manipulated pieces. Our proposed method is trained and evaluated on data provided by the ADD2022 Challenge. We evaluate our detection model concerning various acoustic features and network configurations. As a result, our detection system achieves an equal error rate (EER) of 6.58% on the ADD2022 challenge test set, which is the best performance in partially spoofed audio detection systems that can locate manipulated clips.
Using Modified Adult Speech as Data Augmentation for Child Speech Recognition

Data augmentation is a technique which enhances the size and quality of training data such that deep learning or machine learning models can achieve better performance. This paper proposes a novel way of applying data augmentation for child speech recognition in the low data resource scenario. Data augmentation is achieved by modifying existing adult speech signals. The procedure consists of two main parts, resampling, and time scaling. The experiment involves both speech from children aged from kindergarten to grade 10, and adults’ speech. We test the proposed method using both a TDNN-HMM and a GMM-HMM acoustic model. The results show that the proposed data augmentation scheme achieves a relative 7.95% reduction of WERs compared with 4.56% relative reduction when using a traditional bilinear frequency warping approach.
Enhancing Representation Learning with Deep Classifiers in Presence of Shortcut

A deep neural classifier trained on an upstream task can be leveraged to boost the performance of another classifier in a related downstream task through the representations learned in hidden layers. However, presence of shortcuts (easy-to-learn features) in the upstream task can considerably impair the versatility of intermediate representations and, in turn, the downstream performance. In this paper, we propose a method to improve the representations learned by deep neural image classifiers in spite of a shortcut in upstream data. In our method, the upstream classification objective is augmented with a type of adversarial training where an auxiliary network, so called lens, fools the classifier by exploiting the shortcut in reconstructing images. Empirical comparisons in self-supervised and transfer learning problems with three shortcut-biased datasets suggest the advantages of our method in terms of downstream performance and/or training time.
Multi-Aspect Interest Neighbor-Augmented Network for Next-Basket Recommendation

Next-basket recommendation (NBR) is a type of recommendation task that focuses on mining user interests based on the sequential basket records in which users purchase multiple items at a time. Limited by the sparsity brought by short-term user interaction behavior, existing NBR methods typically fail to mine fine-grained and complete representation of user interests, resulting in unsatisfactory recommendation performance. To address this issue, we propose a novel Multi-aspect Interest Neighbor-augmented Network (MINN) to capture fine-grained and complete representation of user interest for the next basket prediction. Specifically, we first design a multi-aspect interest encoder to learn representations of users and items in the multi-aspect interest space. Semantic neighbors are then selected for user to enhance the user’s representation of interest through a semantic neighbor augmentation mechanism. Finally, the user’s representation of interest and his/her repeated purchase behavior are jointly considered to implement the next basket prediction. Extensive experimental studies on two benchmark datasets demonstrate that MINN outperforms several representative NBR methods and achieves new state-of-the-art results.
Improved Training Of Mixture-Of-Experts Language GANs

Despite the dramatic success in image generation, Generative Adversarial Networks (GANs) still face great challenges in text generation. The difficulty in generator training arises from the limited representation capacity and uninformative learning signals obtained from the discriminator. In this work, we (1) first empirically show that the multi-generator approach is able to enhance the representation capacity of the generator for sequence GANs and (2) harness the Feature Statistics Alignment (FSA) paradigm to render fine-grained learning signals to advance the generator training. Specifically, FSA forces the mean statistics of the distribution of fake data to approach that of real samples as close as possible in the finite-dimensional feature space. Empirical study on synthetic and real benchmarks shows the superior performance in quantitative evaluation and demonstrates the effectiveness of our approach to adversarial text generation.
Delivering Speaking Style in Low-Resource Voice Conversion with Multi-Factor Constraints

Conveying the linguistic content and maintaining the source speech’s speaking style, such as intonation and emotion, is essential in voice conversion (VC). However, in a low-resource situation, where only limited utterances from the target speaker are accessible, existing VC methods are hard to meet this requirement and capture the target speaker’s timber. In this work, a novel VC model, referred to as MFC-StyleVC, is proposed for the low-resource VC task. Specifically, speaker timbre constraint generated by clustering method is newly proposed to guide target speaker timbre learning in different stages. Meanwhile, to prevent over-fitting to the target speaker’s limited data, perceptual regularization constraints explicitly maintain model performance on specific aspects, including speaking style, linguistic content, and speech quality. Besides, a simulation mode is introduced to simulate the inference process to alleviate the mis-match between training and inference. Extensive experiments performed on highly expressive speech demonstrate the superiority of the proposed method in low-resource VC.
Multi-Scale Compositional Constraints for Representation Learning on Videos

Combining simple concepts to form structured thoughts and decomposing complex concepts into their constituents is one key characteristic of human cognition. In this work we extract video representations by combining multi-scale processing with compositional constraints, i.e., we constrain the latent space created by the network so that coarse grained video features are composed from a set of fine-grained video features using simple functions. We integrate the proposed constraints in a state-of-the-art contrastive learning frame-work. In our ablations, we evaluate different formulations of the compositional constraints and composition functions. We evaluate the proposed approach for the downstream tasks of action detection in UCF-101, and video summarization in the SumMe dataset. We achieve significant improvements over the baseline, i.e., 3.9% and 6.3% relative improvements for UCF-101 and SumMe respectively, showcasing the importance of compositional video representations.
Scalable Multi-Task Semantic Communication System with Feature Importance Ranking

Semantic communications are expected to be an innovative solution to the emerging intelligent applications in the era of connected intelligence. In this paper, a novel scalable multi-task semantic communication system with feature importance ranking (SMSC-FIR) is explored. Firstly, the multi-task correlations are investigated by a joint semantic encoder to extract relevant features. Then, a new scalable coding method is proposed based on feature importance ranking, which dynamically adjusts the coding rate and guarantees that important features for semantic tasks are transmitted with higher priority. Simulation results show that SMSC-FIR achieves performance gain w.r.t. individual intelligent tasks, especially in the low SNR regime.
Domain Adaptation without Catastrophic Forgetting on a Small-Scale Partially-Labeled Corpus for Speech Emotion Recognition

Hence, developing a corpus for speech emotion recognition (SER) in the target domain is significant; however, this is time-consuming and cost-intensive. In this study, we aim to fully use a partially-labeled corpus in the target domain (target corpus) with the help of an existing fully-labeled corpus (common corpus). To this end, we proposed a method that leverages domain adversarial multi-task learning to reconcile the definitions of emotion classes across domains and noisy student training to utilize unlabeled data. Our experimental results demonstrated that the proposed method improved the SER performance in the target domain when the target corpus was small in size and imbalanced in classes. Furthermore, performance on the common corpus was not deteriorated by the proposed method.
Hierarchical Multi-Agent Reinforcement Learning with Intrinsic Reward Rectification

Hierarchical reinforcement learning (HRL) is a promising approach to solving long-term decision problems and complex tasks, as high-level policy can guide the training procedure of low-level policy with macro actions and intrinsic rewards. However, the amount that macro actions influence decision-making, which affects how much internal rewards should be given to low-level policy, is disregarded by current HRL algorithms. It may be reasonable to provide low-level policy with less intrinsic rewards if macro actions are less important in decision-making. In this paper, we propose a value decomposition based hierarchical multi-agent reinforcement learning method with intrinsic reward rectification, which can determine the effectiveness of macro actions and correct the intrinsic rewards. We show that our proposed method significantly outperforms the state-of-the-art value decomposition approaches on the StarCraft Multi-Agent Challenge platform.
Leveraging Multiple Sources in Automatic African American English Dialect Detection for Adults and Children

This paper <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> presents a novel system which utilizes acoustic, phonological, morphosyntactic, and prosodic information for binary automatic dialect detection of African American English. We train this system utilizing adult speech data and then evaluate on both children’s and adults’ speech with unmatched training and testing scenarios. The proposed system combines novel and state-of-the-art architectures, including a multi-source transformer language model pre-trained on Twitter text data and fine-tuned on ASR transcripts as well as an LSTM acoustic model trained on self-supervised learning representations, in order to learn a comprehensive view of dialect. We show robust, explainable performance across recording conditions for different features for adult speech, but fusing multiple features is important for good results on children’s speech.