Reviewerly/exHarmony huggingface dataset:
Predicting Multi-Codebook Vector Quantization Indexes for Knowledge Distillation

Knowledge distillation (KD) is a common approach to improve model performance in automatic speech recognition (ASR), where a student model is trained to imitate the output behaviour of a teacher model. However, traditional KD methods suffer from teacher label storage issue, especially when the training corpora are large. Although on-the-fly teacher label generation tackles this issue, the training speed is significantly slower as the teacher model has to be evaluated every batch. In this paper, we reformulate the generation of teacher label as a codec problem. We propose a novel Multi-codebook Vector Quantization (MVQ) approach that compresses teacher embeddings to codebook indexes (CI). Based on this, a KD training framework (MVQ-KD) is proposed where a student model predicts the CI generated from the embeddings of a self-supervised pre-trained teacher model. Experiments on the LibriSpeech clean-100 hour show that MVQ-KD framework achieves comparable performance as traditional KD methods (11, 12), while requiring 256 times less storage. When the full LibriSpeech dataset is used, MVQ-KD framework results in 13.8% and 8.2% relative word error rate reductions (WERRs) for non -streaming transducer on test-clean and test-other and 4.0% and 4.9% for streaming transducer. The implementation of this work is already released as a part of the open-source project icefall <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Intent Does Matter! Propagating High-Order Relations for Exploring Interest Preferences

Session-based recommendation (SBR) aims to predict the user’s action at the next timestamp according to an anonymous yet short interaction sequence (i.e., session). Almost all the existing SBR solutions for user preference are only based on the current session without exploiting the high-order relations among other sessions, which may restrict the SBR representation ability and even deteriorate the performance. To this end, we propose a Hyper-relation alignment hyperGraph Convolutional Network, called Hyra-GCN, for better inferring the user preference of the current session. Specifically, we first model session-based data as a hyper-graph capable of representing high-order relationships to exploit item transitions over sessions in a more subtle manner. Subsequently, we explore self-supervised learning on item-session hypergraphs, so as to alleviate the problem of data sparsity. Experimental results on real-world datasets demonstrate the effectiveness of our proposed Hyra-GCN against state-of-the-art baselines.
On Adversarial Robustness of Audio Classifiers

We make three contributions to improve adversarial robustness of audio classifiers. First, most existing works focus on ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</inf> -norm bounded adversarial perturbations. Instead, we consider signal-to-noise ratio (SNR) as a more natural measure of adversarial perturbations for audio data. We show that perturbed examples with a particular SNR can be generated using a corresponding ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf> -norm perturbation, and establish the equivalence of these two metrics in assessing adversarial perturbations. This connection enables direct control of the SNR quality of perturbed examples and allows comparison using perturbations with different ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</inf> -norm constraints. Second, we are among the first to introduce APGD attack for adversarial training on audio data. In our experiments, APGD adversarial training is robust to adversarial attacks without compromising clean accuracy. Last, we improve adversarial robustness by adapting CutMix to audio - cutting and mixing two audio clips together - in conjunction with adversarial training, and observe improvements in robustness on US8K.
TOLD: a Novel Two-Stage Overlap-Aware Framework for Speaker Diarization

Recently, end-to-end neural diarization (EEND) is introduced and achieves promising results in speaker-overlapped scenarios. In EEND, speaker diarization is formulated as a multi-label prediction problem, where speaker activities are estimated independently and their dependency are not well considered. To overcome these disadvantages, we employ the power set encoding to reformulate speaker diarization as a single-label classification problem and propose the overlap-aware EEND (EEND-OLA) model, in which speaker overlaps and dependency can be modeled explicitly. Inspired by the success of two-stage hybrid systems, we further propose a novel Two-stage OverLap-aware Diarization framework (TOLD) by involving a speaker overlap-aware post-processing (SOAP) model to iteratively refine the diarization results of EEND-OLA. Experimental results show that, compared with the original EEND, the proposed EEND-OLA achieves a 14.39% relative improvement in terms of diarization error rates (DER), and utilizing SOAP provides another 19.33% relative improvement. As a result, our method TOLD achieves a DER of 10.14% on the CALLHOME dataset, which is a new state-of-the-art result on this benchmark to the best of our knowledge.
Frequency and Scale Perspectives of Feature Extraction

Convolutional neural networks (CNNs) have achieved superior performance but still lack clarity about the nature and properties of feature extraction. In this paper, by analyzing the sensitivity of neural networks to frequencies and scales, we find that neural networks not only have low- and mediumfrequency biases but also prefer different frequency bands for different classes, and the scale of objects influences the preferred frequency bands. These observations lead to the hypothesis that neural networks must learn the ability to extract features at various scales and frequencies. To corroborate this hypothesis, we propose a network architecture based on Gaussian derivatives, which extracts features by constructing scale space and employing partial derivatives as local feature extraction operators to separate high-frequency information. This manually designed method of extracting features from different scales allows our GSSDNets to achieve comparable accuracy with vanilla networks on various datasets.
Improving Dropout in Graph Convolutional Networks for Recommendation via Contrastive Loss

We propose a novel graph convolutional network (GCN)-based recommendation model that incorporates a contrastive loss. Although GCN-based recommendation models achieve high recommendation performance, existing models suffer from over-fitting since they explicitly encode the interactions as a graph. To mitigate this problem, while the models perform dropout which just randomly drops edges in the graph, this can miss the essential information that represents users’ preferences. The proposed method improves and extends the dropout via the contrastive loss and enhances the robustness while preserving the essential information. The contrastive loss can correlate the representations in different perturbed graphs and promote their consistency. Conceptually, our method captures users’ preferences which are invariant even in the perturbed interactions. The experimental results demonstrate that our method improves the recommendation performance in real-world datasets.
Divcon: Learning Concept Sequences for Semantically Diverse Image Captioning

Human generated image captions contain diverse semantic concepts, while this is still a difficult task for machines. The frequency distribution of semantic concepts in datasets is usually extremely imbalanced, leading to models repeatedly describe frequently occurring semantic concepts, resulting in a decline in the semantic diversity. In this paper, we propose a novel two-step method for diverse image captioning, generating descriptions with more diverse semantic concepts (Di-vCon). Firstly, we developed a concept sequence generator to auto-regressively generate concept sequences. This benefits the model by decoding sequences in a small searching space. Then a sentence generator takes as input the concept sequences and generates descriptions for each sequence. Experiments show that DivCon can generate captions containing diverse semantic concepts and pay more attention to the less occurring concepts. In the diverse image captioning task, Div-Con achieves the state-of-the-art results on MSCOCO dataset with oracle CIDEr and SPICE scores of 1.684 and 0.302.
SINCO: A Novel Structural Regularizer for Image Compression Using Implicit Neural Representations

Implicit neural representations (INR) have been recently proposed as deep learning (DL) based solutions for image compression. An image can be compressed by training an INR model with fewer weights than the number of image pixels to map the coordinates of the image to corresponding pixel values. While traditional training approaches for INRs are based on enforcing pixel-wise image consistency, we propose to further improve image quality by using a new structural regularizer. We present structural regularization for INR compression (SINCO) as a novel INR method for image compression. SINCO imposes structural consistency of the compressed images to the groundtruth by using a segmentation network to penalize the discrepancy of segmentation masks predicted from compressed images. We validate SINCO on brain MRI images by showing that it can achieve better performance than some recent INR methods.
Enhancing the Vocal Range of Single-Speaker Singing Voice Synthesis with Melody-Unsupervised Pre-Training

The single-speaker singing voice synthesis (SVS) usually underperforms at pitch values that are out of the singer's vocal range or associated with limited training samples. Based on our previous work, this work proposes a melody-unsupervised multi-speaker pretraining method conducted on a multi-singer dataset to enhance the vocal range of the single-speaker, while not degrading the timbre similarity. This pre-training method can be deployed to a large-scale multi-singer dataset, which only contains audio-and-lyrics pairs without phonemic timing information and pitch annotation. Specifically, in the pre-training step, we design a phoneme predictor to produce the frame-level phoneme probability vectors as the phonemic timing information and a speaker encoder to model the timbre variations of different singers, and directly estimate the frame-level f0 values from the audio to provide the pitch information. These pre-trained model parameters are delivered into the fine-tuning step as prior knowledge to enhance the single speaker's vocal range. Moreover, this work also contributes to improving the sound quality and rhythm naturalness of the synthesized singing voices. It is the first to introduce a differentiable duration regulator to improve the rhythm naturalness of the synthesized voice, and a bi-directional flow model to improve the sound quality. Experimental results verify that the proposed SVS system outperforms the baseline on both sound quality and naturalness.
Label-Guided Contrastive Learning for Out-of-Domain Detection

Out-of-Domain (OOD) detection from user utterances plays an important part in task-oriented dialogue systems. Recent studies utilize supervised or self-supervised contrastive learning (CL) to learn discriminative semantic features for OOD detection. The supervised contrastive learning (SCL) methods only model class-level features of different in-domain (IND) intents while self-supervised CL (SSCL) methods can model instance-level features. However, the SSCL methods require complex data augmentation and are vulnerable to intrinsic false-negative pairs. To address the issues above and leverage both types of CL, we propose a novel Label-Guided Contrastive Learning (LGCL) framework. LGCL models both instance-level and class-level discriminative semantic representations by employing the sample and its corresponding label simultaneously, such that the prior knowledge of IND can be fully leveraged. Experiment and analysis results on two public benchmark datasets show that the proposed method significantly outperforms baselines on the OOD detection task.
Co-Operative CNN for Visual Saliency Prediction on WCE Images

The physician’s experience is highly correlated with the content interpretation of medical images. Over time, physicians develop their ability to examine the images, and this is usually reflected on gaze patterns they follow to observe visual cues that lead them to diagnostic decisions. In the context of gaze prediction, graph and machine learning methods have been proposed for the visual saliency estimation on generic images. In this work we preset a novel and robust gaze estimation methodology based on physicians’ eye fixations, using convolutional neural networks (CNNs) trained according to a novel co-operative scheme, on medical images acquired during Wireless Capsule Endoscopy (WCE). The proposed training approach considers both the reconstruction accuracy of the estimated saliency maps, and their contribution to the classification process of normal and abnormal findings. The model that was trained with the proposed co-operative procedure was able to achieve an average score of 0.76 Judd’s Area Under the receiver operating Characteristic (AUC-J).
Auxiliary Pooling Layer For Spoken Language Understanding

End-to-end spoken language understanding requires speech data annotated with semantic information and may suffer from the shortage of annotated data. Recent progresses leverage unlabelled speech data to pre-train a speech encoder. However, it remains a challenge for the pre-trained speech encoder to encode semantic information. Existing works explore transferring knowledge from a pre-trained text model with different alignment losses at a fixed granularity. In this paper, we address the variable granularity in transferring knowledge from texts to speech representation via APLY, an auxiliary pooling layer, that fuses the global information with the adaptively encoded local context. We demonstrate the effectiveness of APLY on three benchmarks of spoken language understanding.
Dual-Attention Neural Transducers for Efficient Wake Word Spotting in Speech Recognition

We present dual-attention neural biasing, an architecture designed to boost Wake Words (WW) recognition and improve inference time latency on speech recognition tasks. This architecture enables a dynamic switch for its runtime compute paths by exploiting WW spotting to select which branch of its attention networks to execute for an input audio frame. With this approach, we effectively improve WW spotting accuracy while saving runtime compute cost as defined by floating point operations (FLOPs). Using an in-house de-identified dataset, we demonstrate that the proposed dual-attention network can reduce the compute cost by 90% for WW audio frames, with only 1% increase in the number of parameters. This architecture improves WW F1 score by 16% relative and improves generic rare word error rate by 3% relative compared to the baselines.
The Pipeline System of ASR and NLU with MLM-based data Augmentation Toward Stop Low-Resource Challenge

This paper describes our system for the low-resource domain adaptation track (Track 3) in Spoken Language Understanding Grand Challenge, which is a part of ICASSP Signal Processing Grand Challenge 2023. In the track, we adopt a pipeline approach of ASR and NLU. For ASR, we fine-tune Whisper for each domain with upsampling. For NLU, we fine-tune BART on all the Track3 data and then on low-resource domain data. We apply masked LM (MLM) -based data augmentation, where some of input tokens and corresponding target labels are replaced using MLM. We also apply a retrieval-based approach, where model input is augmented with similar training samples. As a result, we achieved exact match (EM) accuracy 63.3/75.0 (average: 69.15) for reminder/weather domain, and won the 1st place at the challenge.
A Study on Bias and Fairness in Deep Speaker Recognition

With the ubiquity of smart devices that use speaker recognition (SR) systems as a means of authenticating individuals and personalizing their services, fairness of SR systems has becomes an important point of focus. In this paper we study the notion of fairness in recent SR systems based on 3 popular and relevant definitions, namely Statistical Parity, Equalized Odds, and Equal Opportunity. We examine 5 popular neural architectures and 5 commonly used loss functions in training SR systems, while evaluating their fairness against gender and nationality groups. Our detailed experiments shed light on this concept and demonstrate that more sophisticated encoder architectures better align with the definitions of fairness. Additionally, we find that the choice of loss functions can significantly impact the bias of SR models.
DO-FAM: Disentangled Non-Linear Latent Navigation For Facial Attribute Manipulation

Facial attribute manipulation (FAM) aims to edit the semantic attributes of facial images according to the user’s requirements. Unfortunately, the majority of existing FAM methods struggle in meeting at least one of the two requirements: high reconstruction quality and high irrelevance preservation. To alleviate these two limitations, we propose a novel Disentangled nOn-linear latent navigation framework for FAM, termed DO-FAM. To promote the reconstruction quality, we leverage hypernetworks to fine-tune a pre-trained StyleGAN2 generator. To decouple entangled attributes, we propose a novel Disentangled nOn-Linear Latent transformation module, named DOLL, which consists of three components: (1) a decomposer to factorize input latent codes into two parts: attribute-related and attribute-unrelated; (2) a non-linear Latent Transformation Network (LTNet) to navigate the attribute-related latent codes to the target one with respect to the designed attribute(s); and (3) a latent classifier tasked with predicting latent codes’ attributes to guide the latent code navigation. Extensive experimental results on a widely-used benchmark facial editing dataset, CelebA-HQ, demonstrate the superiority of our method over state-of-the-art methods.
Transformer-Based Deep Hashing Method for Multi-Scale Feature Fusion

The deep image hashing aims to map the input image into simply binary hash codes via deep neural networks. Motivated by the recent advancements of Vision Transformers (ViT), many deep hashing methods based on ViT have been proposed. Nevertheless, the ViT has enormous number of model parameters and high computational complexity. Moreover, the last layer of the ViT outputs only the classification tokens as image feature vectors, while the rest of the vectors are discarded. This results in the ineﬃciency of the model computation and the neglect of useful image information. Therefore, this paper proposes a Transformer-based deep hashing method for multi-scale feature fusion (TDH). Specifically, we use a hierarchical Transformer backbone to capture both global and local features of images. The hierarchical Transformer utilizes a local self-attention mechanism to process image blocks in parallel, which reduces computational complexity and promotes computational eﬃciency. Multi-scale feature fusion module captures all image feature vectors of the hierarchical Transformer output to obtain more enriched image feature information. We perform comprehensive experiments on three widely-studied datasets: CIFAR-10, NUS-WIDE and IMAGENET. The experimental results demonstrate that the proposed method in this paper indicates superior results compared to the existing state-of-the-art work. Source code is available https://github.com/shuaichaochao/TDH.
Pseudo-Query Generation For Semi-Supervised Visual Grounding With Knowledge Distillation

Visual grounding is a crucial multi-modal job for locating the objects that the referring queries refer to in images. In recent years, both fully-supervised and weakly-supervised algorithms rely on a large number of query annotations. However, collecting queries in natural language is labor-intensive, which limits the application scenarios of these methods. To overcome this weakness, we propose a novel semi-supervised visual grounding framework. The framework consists of two effective techniques: a prompt enhanced pseudo-query generator utilizing objects without query annotation to produce high-quality pseudo-queries; a knowledge distillation mechanism using a teacher network to stabilize the training process of the student network. Experiment results show that our proposed framework dramatically outperforms the existing methods under the three different label proportions on the three commonly used visual grounding datasets.
Incorporating Reliability in Graph Information Propagation by Fluid Dynamics Diffusion: A case of Multimodal Semisupervised Deep Learning

Classic graph neural networks show some limitations in information extraction performance when applied to multimodal datasets. This is primarily due to such datasets having high volume, variety, and variability. In this paper, we propose structuring graph neural networks on a new graph representation based on fluid dynamics diffusion that allows us to incorporate the reliability of the features used to characterise each sample within the graph structure itself. This approach aims to address some of the major limitations of the classic graph-based learning structures, so to improve accuracy and robustness of the estimates. We show how this approach can help to strongly improve the quality of the analysis of classic graph neural networks. Experimental results are reported to support this point.
Performing Neural Architecture Search Without Gradients

One of the most crucial concerns in Neural Architecture Search (NAS) is the amount of resources and time required for the search. In this paper we address the problem of training-free (or zero-shot) search. We propose a generalized architecture search framework called enhanced training-free neural architecture search (ETE-NAS). It ranks architectures according to a user defined combination of zero-shot ranking functions. We instantiate it with metrics based on neural network Gaussian process (NNGP) kernel and ReLU regions division. This allows much faster search on a single GPU than other training-free approaches without sacrificing accuracy or inducing instability. Code is available at https://github.com/Rufaim/ETENAS
A Slot-Shared Span Prediction-Based Neural Network for Multi-Domain Dialogue State Tracking

There are a large number of candidate values shared among slots in multi-domain dialogue state tracking (DST). The existing span prediction-based DST methods generally adopt slot-independent value extraction architecture, which ignore the value sharing. Besides, the slot-independent design leads to poor scalability. In this paper, we propose a Slot-shared Span Prediction based Network (SSNet) with a general value extraction module for all slots to tackle these problems. To ensure that the value extraction module is able to distinguish different slots, we introduce a Dynamic Fusion Mechanism (DFM) to extract different slot-aware features. DFM plays the routing role, highlighting different dialogue context tokens for different slots. Specifically, DFM firstly calculates similarity matrixes between the dialogue context and different slots, and then determines important dialogue context token with respect to each slot. Experimental results demonstrate that SSNet outperforms the existing start-of-the-art models on both MultiWOZ 2.1 and MultiWOZ 2.2 datasets.