Reviewerly/exHarmony huggingface dataset:
Multi-View Graph Regularized Deep Autoencoder-Like NMF Framework

Many real-world data are composed of different representations or views, thus multi-view clustering (MVC) has attracted more and more attention in recent years. Its key task is how to extract sufficient fusion features from multi-view data. Because the nonnegative matrix factorization (NMF) can favorably explain the extracted features, the NMF based MVC is usually a good choice for multi-view data, and promising results are achieved. Inspired by this, we propose a multi-view graph regularized deep autoencoder-like NMF (MGANMF) framework in this paper for multi-view clustering. MGANMF uses the deep autoencoder-like NMF, which draws lessons from the idea of depth automatic encoder, to learn the hierarchical semantics of multi-view data in a layer-wise manner. Moreover, in order to describe the inherent geometric structure in each view data, graph regulators are introduced to couple the output representation of deep structure. In addition, the self-updating weights are employed to balance the effect of each view. Thus, a new objective function is defined and the optimization processes are presented. Experimental results on several multi-view datasets show the effectiveness of the proposed model.
Probabilistic Back-ends for Online Speaker Recognition and Clustering

This paper focuses on multi-enrollment speaker recognition which naturally occurs in the task of online speaker clustering, and studies the properties of different scoring back-ends in this scenario. First, we show that popular cosine scoring suffers from poor score calibration with a varying number of enrollment utterances. Second, we propose a simple replacement for cosine scoring based on an extremely constrained version of probabilistic linear discriminant analysis (PLDA). The proposed model improves over the cosine scoring for multi-enrollment recognition while keeping the same performance in the case of one-to-one comparisons. Finally, we consider an online speaker clustering task where each step naturally involves multi-enrollment recognition. We propose an online clustering algorithm allowing us to take benefits from the PLDA model such as the ability to handle uncertainty and better score calibration. Our experiments demonstrate the effectiveness of the proposed algorithm.
TriAAN-VC: Triple Adaptive Attention Normalization for Any-to-Any Voice Conversion

Voice Conversion (VC) must be achieved while maintaining the content of the source speech and representing the characteristics of the target speaker. The existing methods do not simultaneously satisfy the above two aspects of VC, and their conversion outputs suffer from a trade-off problem between maintaining source contents and target characteristics. In this study, we propose Triple Adaptive Attention Normalization VC (TriAAN-VC), comprising an encoder-decoder and an attention-based adaptive normalization block, that can be applied to non-parallel any-to-any VC. The proposed adaptive normalization block extracts target speaker representations and achieves conversion while minimizing the loss of the source content with siamese loss. We evaluated TriAAN-VC on the VCTK dataset in terms of the maintenance of the source content and target speaker similarity. Experimental results for one-shot VC suggest that TriAAN-VC achieves state-of-the-art performance while mitigating the trade-off problem encountered in the existing VC methods.
Strategies for Enhanced Signal Modulation Classifications Under Unknown Symbol Rates and Noise Conditions

Radio frequency signal modulation classifications find broad applications in cognitive sensing and RF spectrum coexistence. Recently, deep neural networks have been shown to be a powerful tool for automatic modulation classification (AMC). Accounting for different signal variations is paramount towards reliable classifications. In this paper, we examine the performance of AMC under varying sampling rates and signal-to-noise ratio (SNR). We consider a dynamic environment where the signal modulation and channel conditions can be assumed constant over a number of consecutive observations. We also show that a single ResNet can be used for both modulation classification and estimating SNR which allows network training and testing at the same noise levels. It is shown that significant signal modulation classification accuracy improvement can be achieved using multiple observations and known SNR.
Position-Aware Graph-Based Learning of Whole Slide Images

Encoding whole slide images (WSI) as graphs is well motivated as it allows the gigapixel resolution WSI to be represented in its entirety for learning. To this end, WSIs can be divided into small patches representing the nodes of a graph. Then, graph-based learning approaches can be deployed for cancer grading and classification. Graph-based learning methods such as graph neural network (GNN) are based on message passing among neighboring nodes. However, they do not account for positional information for each patch and if two patches are located in topologically isomorphic neighborhoods, their embedding is almost identical. In this work, classification of cancer from WSI is performed with positional embedding and graph attention. The proposed approach is based on spline convolutional neural network (CNN) to encode the positional embedding of the nodes in graph classification. The algorithm is then tested with the WSI dataset for grading prostate cancer and kidney cancer. A comparison of the proposed method with leading approaches in cancer diagnosis and grading verify improved performance.
GAPter: Gray-Box Data Protector for Deep Learning Inference Services at User Side

The widespread deployment of Deep Learning Inference Services (DLISes) has raised people’s concerns about their data privacy being breached. Although data privacy enhancement has recently attracted a lot of attention, existing solutions all require the cooperation of service providers. Users lose control of their data when making data privacy enhancement decisions. However, it is difficult to enable the user-side control of data abuse prevention because users do not have any programming skills, deep learning knowledge, or rich computing resources. In this work, we propose a fully-automatic userside data privacy enhancement solution, GAPter, for DLISes. Given such a DLIS, GAPter can adaptively fuzz the service for a suitable enhancement strategy, with no cooperation between the DLIS provider and the user. We have implemented and comprehensively evaluated GAPter. The experimental results show that GAPter can find good balance points between privacy enhancement and user data utility.
Towards Robust Audio-Based Vehicle Detection Via Importance-Aware Audio-Visual Learning

Although audio modality has the potential to solve various visually challenging conditions of visual modality, there are few studies on audio-based detection. This is because the audio modality itself contains less accurate spatial information. To alleviate this issue, the existing audio-based methods adopt the visual modality in the training phase to transfer more precise spatial knowledge to the audio modality. However, they do not consider the case where the visual modality is less informative. In this paper, we present a new audio-based vehicle detector that can transfer multimodal knowledge of vehicles to the audio modality during training. To this end, we combine the audio-visual modal knowledge according to the importance of each modality to generate integrated audiovisual feature. Also, we introduce an audio-visual distillation (AVD) loss that guides representation of the audio modal feature to resemble that of the integrated audio-visual feature. As a result, our audio-based detector can perform robust vehicle detection as if it were utilizing both modalities, even if it only receives audio modality as input in the inference. Comprehensive experimental results demonstrate that our method exhibits consistent improvements over the existing methods.
Leveraging Language Embeddings for Cross-Lingual Self-Supervised Speech Representation Learning

In this paper, we propose novel cross-lingual self-supervised speech representation learning methods that explicitly consider language information. Cross-lingual self-supervised speech representation learning has been studied to make effective use of diverse data in various languages. Previous methods train models from multilingual datasets without taking language into account. However, it is difficult to train speech representations from multilingual datasets in the same space without language specification since there are clear differences in the acoustic context between languages. To solve this problem, we propose leveraging language IDs to build self-supervised speech representation learning models that explicitly consider language information. Our proposed models utilize fixed-dimensional language embeddings converted from language IDs for the model learning the relationship between related speech representations in different languages. We investigate two strategies to introduce language embeddings into the models: adding the embeddings to all of the inputs and concatenating to the inputs of the Transformer. We experimentally investigated how the difference between the two strategies affects the downstream tasks. Experimental results on the English and Japanese datasets show that the proposed methods improve the accuracies of downstream automatic speech recognition tasks.
End-to-End Spoken Language Understanding with Tree-Constrained Pointer Generator

End-to-end spoken language understanding (SLU) suffers from the long-tail word problem. This paper exploits contextual biasing, a technique to improve the speech recognition of rare words, in end-to-end SLU systems. Specifically, a tree-constrained pointer generator (TCPGen), a powerful and efficient biasing model component, is studied, which leverages a slot shortlist with corresponding entities to extract biasing lists. Meanwhile, to bias the SLU model output slot distribution, a slot probability biasing (SPB) mechanism is proposed to calculate a slot distribution from TCPGen. Experiments on the SLURP dataset showed consistent SLU-F1 improvements using TCPGen and SPB, especially on unseen entities. On a new split by holding out 5 slot types for the test, TCPGen with SPB achieved zero-shot learning with an SLU-F1 score over 50% compared to baselines which can not deal with it. In addition to slot filling, the intent classification accuracy was also improved.
VF-Taco2: Towards Fast and Lightweight Synthesis for Autoregressive Models with Variation Autoencoder and Feature Distillation

With the development of deep learning, end-to-end neural text-to-speech (TTS) systems have achieved significant improvements in high-quality speech synthesis. However, most of these systems are attention-based autoregressive models, resulting in slow synthesis speed and large model parameter sizes. In this paper, we propose a new fast and lightweight TTS framework named VF-Taco2, which can quickly synthesize speech without GPUs. We first profiled the complexity of decoder process in the current autoregressive model and designed a novel multiple frames prediction module based on variational autoencoder (VAE) to alleviate quality degradation when a larger "reduction factor" is applied. Besides, feature distillation is leveraged to compress a relatively large proposed model to its small version with a minor loss of speech quality. Compared to the original Tacotron 2, our VF-Taco2 achieves a 3.6x-4.4x Mel-spectrum generation acceleration on different performance CPUs, and the parameters are compressed by 1.5x with speech quality maintained.
C<sup>2</sup>BN: Cross-Modality and Cross-Scale Balance Network for Multi-Modal 3D Object Detection

Multi-modal 3D object detection that classifies and locates objects in 3D space by combining point-clouds captured by lidars and RGB images captured by cameras, serves as the basis for autonomous driving. Most of the existing methods aggregate features from point-clouds and images by plain element-wise additions or multiplications. Although these methods improve detection accuracy, such simple operations have difficulties in balancing both modalities. Further, the multi-level features from images also suffer from imbalance problems in receptive fields. To address the above problems, we propose two novel networks: cross-modality balance network (CMN) and cross-scale balance network (CSN). CMN utilizes cross-modality attention mechanisms to balance the importance and receptive field of two modalities. CSN employs cross-scale attention mechanisms to reduce the imbalance in multi-level features. Experiments are performed on the challenging benchmark: KITTI. The experimental results show consistent improvements in different 3D object detection frameworks, which verifies the effectiveness and generality of our proposed networks.
Adaptive Step-Size Methods for Compressed SGD

Compressed Stochastic Gradient Descent (SGD) algorithms have been proposed to address the communication bottleneck in distributed and decentralized optimization problems such as federated machine learning. Many existing compressed SGD algorithms use non-adaptive step-sizes (constant or diminishing) to provide theoretical convergence guarantees. Since non-adaptive step-sizes typically involve unknown system parameters, the step-sizes are fine-tuned in practice to obtain good empirical performance for the given dataset and learning model. Such fine-tuning might be impractical in many scenarios. Thus, it is of interest to study compressed SGD using adaptive step-sizes. Motivated by prior work that use adaptive step-sizes for uncompressed SGD, we develop an Armijo rule based step-size selection method for compressed SGD. In particular, we introduce a scaling technique for the descent step, which we use to establish order-optimal convergence rates for convex-smooth and strong convex-smooth objectives under an interpolation condition, and for non-convex objectives under a strong growth condition. We present experimental results on deep neural networks trained on real-world datasets, and compare the performance of our proposed algorithm with state-of-the-art compressed SGD methods to demonstrate improved performance at various levels of compression.
Simplicial Vector Autoregressive Model For Streaming Edge Flows

Vector autoregressive (VAR) model is widely used to model time-varying processes, but it suffers from prohibitive growth of the parameters when the number of time series exceeds a few hundreds. We propose a simplicial VAR model to mitigate the curse of dimensionality of the VAR models when the time series are defined over higher-order network structures such as edges, triangles, etc. The proposed model shares parameters across the simplicial signals by leveraging the simplicial convolutional filter and captures structure-aware spatio-temporal dependencies of the time-varying processes. Targetting the streaming signals from the real-world nonstationary networks, we develop a group-lasso-based online strategy to learn the proposed model. Using traffic and water distribution networks, we demonstrate that the proposed model achieves competitive signal prediction accuracy with a significantly less number of parameters than the VAR models.
Adaptive Semantic Fusion Framework for Unsupervised Monocular Depth Estimation

Unsupervised monocular depth estimation plays an important role in autonomous driving, and has been received considerable research attention in recent years. Nevertheless, numerous existing methods relying on photometric consistency are excessively susceptible to variations in illumination and suffer in the regions with strong reflection. To overcome this limitation, we propose a novel unsupervised depth estimation framework named ColorDepth, which forces the model to explore object semantic to infer depth. Specifically, we extract pixel-level semantic prior clues of objects using the semantic segmentation network. These priors and the original image are then adaptively fused into color data by a learnable parameter for depth estimation. The incorporation of semantics endows our model with the ability to perceive scene structure information. The fused data effectively alleviates the depth ambiguity within the same semantic block, leading to improved consistency and robustness in challenging scenarios. Extensive experiments on the KITTI and Make3D datasets show that our method surpasses the previous state-of-the-art methods even those supervised by additional constraints, and brings significant performance improvement particularly in the regions of high reflection.
Noise-Aware Target Extension with Self-Distillation for Robust Speech Recognition

Data augmentation using additive noise is a framework for robustly training automatic speech recognition models. To utilize noise information efficiently, previous studies used an additional branch to classify noise conditions. This added branch has a limited effect on the ASR because it performs independently of the ASR branch that classifies senones. In this paper, we propose a noise-aware target extension (NATE) that extends the senone target to contain noise awareness by jointly classifying the senone and noise in a single branch. In the inference stage, the output of the model is processed separately by the noise condition and then aggregated to match the senone posterior distribution. In addition, we combine NATE with self-distillation (NATE <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sd</inf> ) to reduce the model parameters and avoid discrepancies between the outputs of training and inference. The effectiveness of the NATE method is validated on the two benchmark development and evaluation sets and simulated noisy test sets, resulting in significant improvements over the previous methods.
Lexicon-injected Semantic Parsing for Task-Oriented Dialog

Recently, semantic parsing using hierarchical representations for dialog systems has captured substantial attention. Task-Oriented Parse (TOP), a tree representation with intents and slots as labels of nested tree nodes, has been proposed for parsing user utterances. Previous TOP parsing methods are limited on leveraging lexicon resources, which are often used to guide the real dialog system. To mitigate this issue, we first propose a novel span-splitting representation for span-based parser that outperforms existing methods. Then we present a novel lexicon-injected semantic parser, which collects slot labels of tree representation as a lexicon, and injects lexical features to the span representation of parser. An additional slot disambiguation technique is involved to remove inappropriate span match occurrences from the lexicon. Experiments show that our best parser produces a new state-of-the-art result (87.62%) on the TOP dataset, and also confirm the effectiveness of our proposed lexicon-injected parser and slot disambiguation model.
Decoupled Non-Parametric Knowledge Distillation for end-to-End Speech Translation

Existing techniques often attempt to make knowledge transfer from a powerful machine translation (MT) to speech translation (ST) model with some elaborate techniques, which often requires transcription as extra input during training. However, transcriptions are not always available, and how to improve the ST model performance without transcription, i.e., data efficiency, has rarely been studied in the literature. In this paper, we propose Decoupled Non-parametric Knowledge Distillation (DNKD) from data perspective to improve the data efficiency. Our method follows the knowledge distillation paradigm. However, instead of obtaining the teacher distribution from a sophisticated MT model, we construct it from a non-parametric datastore via k-Nearest-Neighbor (kNN) retrieval, which removes the dependence on transcription and MT model. Then we decouple the classic knowledge distillation loss into target and non-target distillation to enhance the effect of the knowledge among non-target logits, which is the prominent "dark knowledge". Experiments on MuST-C corpus show that, the proposed method can achieve consistent improvement over the strong baseline without requiring any transcription.
Whether Contribution of Features Differ Between Video-Mediated and In-Person Meetings in Important Utterance Estimation

This study investigated differences in the contributions of various features to in-person (IP) and video-mediated (VM) meetings. We focused on estimating important utterances using both an IP and a VM meeting corpora as the analysis data. A transformer model with dialogue history was used to estimate important utterances, and five types of input (text, speaker’s audio, others’ audio, speaker’s video, and others’ video) were fed to the model. A comparison of the models for IP and VM revealed that the speaker’s audio has a strong effect on the IP model, the video of the other participants strongly affects the VM model, and the text and others’ audio strongly affects both models in estimating important utterances.
CORSD: Class-Oriented Relational Self Distillation

Knowledge distillation conducts an effective model compression method while holding some limitations: (1) the feature based distillation methods only focus on distilling the feature map but are lack of transferring the relation of data examples; (2) the relational distillation methods are either limited to the handcrafted functions for relation extraction, such as L <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf> norm, or weak in inter- and intra- class relation modeling. Besides, the feature divergence of heterogeneous teacher-student architectures may lead to inaccurate relational knowledge transferring. In this work, we propose a novel training framework named Class-Oriented Relational Self Distillation (CORSD) to address the limitations. The trainable relation networks are designed to extract relation of structured data input, and they enable the whole model to better classify samples by transferring the relational knowledge from the deepest layer of the model to shallow layers. Besides, auxiliary classifiers are proposed to make relation networks capture class-oriented relation that benefits classification task. Experiments demonstrate that CORSD achieves remarkable improvements. Compared to baseline, 3.8%, 1.5% and 4.5% averaged accuracy boost can be observed on CIFAR100, ImageNet and CUB-200-2011, respectively.
Data2vec-Aqc: Search for the Right Teaching Assistant in the Teacher-Student Training Setup

In this paper, we propose a new Self-Supervised Learning (SSL) algorithm called data2vec-aqc, for speech representation learning from unlabeled speech data. Our goal is to improve SSL for speech in domains where both unlabeled and labeled data are limited. Building on the recently introduced data2vec [1], we introduce additional modules to the data2vec framework that leverage the benefit of data augmentations, quantized representations, and clustering. The interaction between these modules helps solve the cross-contrastive loss as an additional self-supervised objective. data2vec-aqc achieves up to 14.1% and 20.9% relative WER improvement over the existing state-of-the-art data2vec system over the test-clean and test-other sets, respectively of LibriSpeech, without the use of any language model (LM). Our proposed model also achieves up to 17.8% relative WER gains over the baseline data2vec when fine-tuned on a subset of the Switchboard dataset. Code: https://github.com/Speech-Lab-IITM/data2vec-aqc.