Reviewerly/exHarmony huggingface dataset:
Lego-Features: Exporting Modular Encoder Features for Streaming and Deliberation ASR

In end-to-end (E2E) speech recognition models, a representational tight-coupling inevitably emerges between the encoder and the decoder. We build upon recent work that has begun to explore building encoders with modular encoded representations, such that encoders and decoders from different models can be stitched together in a zero-shot manner without further fine-tuning. While previous research only addresses full-context speech models, we explore the problem in a streaming setting as well. Our framework builds on top of existing encoded representations, converting them to modular features, dubbed as Lego-Features, without modifying the pre-trained model. The features remain interchangeable when the model is retrained with distinct initializations. Though sparse, we show that the Lego-Features are powerful when tested with RNN-T or LAS decoders, maintaining high-quality downstream performance. They are also rich enough to represent the first-pass prediction during two-pass deliberation. In this scenario, they outperform the N-best hypotheses, since they do not need to be supplemented with acoustic features to deliver the best results. Moreover, generating the Lego-Features does not require beam search or auto-regressive computation. Overall, they present a modular, powerful and cheap alternative to the standard encoder output, as well as the N-best hypotheses.
Interaction-Assisted Multi-Modal Representation Learning for Recommendation

Personalized recommender systems have attracted significant attentions from both industry and academic. Recent studies have shed light on incorporating multi-modal side information into the recommender systems to further boost the performance. Meanwhile, transformer-based multi-modal representation learning has shown great enhancement for downstream visual and textual tasks. However, these self-supervised pre-training methods are not tailored for recommendation and may lead to suboptimal representations. To this end, we propose Interaction-Assisted Multi-Modal Representation Learning for Recommendation (IRL) to inject the information of user interactions into item multi-modal representation learning. Specifically, we extract item graph embedding through user-item interactions and then utilize it to formulate a novel triplet IRL training objective which serves as a behavior-aware pre-training task for the representation learning model. A range of experiments have been conducted on several real-world datasets and extensive results indicate the effectiveness of IRL.
Outlier-Insensitive Kalman Filtering Using NUV Priors

The Kalman filter (KF) is a widely-used algorithm for tracking the latent state of a dynamical system from noisy observations. For systems that are well-described by linear Gaussian state space models, the KF minimizes the mean-squared error (MSE). However, in practice, observations are corrupted by outliers, severely impairing the KF’s performance. In this work, an outlier-insensitive KF (OIKF) is proposed, where robustness is achieved by modeling a potential outlier as a normally distributed random variable with unknown variance (NUV). The NUV’s variance is estimated online, using both expectation-maximization (EM) and alternating maximization (AM). The former was previously proposed for the task of smoothing with outliers and was adapted here to filtering, while both EM and AM obtained the same performance and outperformed the other algorithms, the AM approach is less complex and thus requires 40% less runtime. Our empirical study demonstrates that the MSE of our proposed outlier-insensitive KF outperforms previously proposed algorithms, and that for data clean of outliers, it reverts to the classic KF, i.e., MSE optimality is preserved.
Improving Retrieval-Based Dialogue System Via Syntax-Informed Attention

Multi-turn response selection is a challenging task due to its high demands on efficient extraction of the matching features from abundant information provided by context utterances. Since incorporating syntactic information like dependency structures into neural models can promote a better understanding of the sentences, such a method has been widely used in NLP tasks. Though syntactic information helps models achieved pleasing results, its application in retrieval-based dialogue systems has not been fully explored. Meanwhile, previous works focus on intra-sentence syntax alone, which is far from satisfactory for the task of multi-turn response where dialogues usually contain multiple sentences. To this end, we propose SIA, Syntax-Informed Attention, considering both intra- and inter-sentence syntax information. While the former restricts attention scope to only between tokens and corresponding dependents in the syntax tree, the latter allows attention in cross-utterance pairs for those syntactically important tokens. We evaluate our method on three widely used benchmarks and experimental results demonstrate the general superiority of our method on dialogue response selection.
Graph-Based Point Cloud Color Denoising with 3-Dimensional Patch-Based Similarity

Point clouds are utilized in many 3-D applications such as cross-reality (XR) and realistic 3-D display. They consist of a set of points with 3-D coordinates and associated color signals. These color signals are often perturbed by noise induced by the measurement errors of scanning devices. In this paper, we propose a point cloud denoising method for color signals. Since many conventional methods for point cloud color denoising are based on a low-pass filter in the graph spectral domain, denoising accuracy is affected by the choice of graph. We propose a graph construction method using 3-D patch-based similarity, in which the similarity is calculated with small 3-D patches around the connected points. This is in contrast with conventional graph construction methods for denoising, which are based on point properties such as pairwise point distances and differences in color. Second, we propose a low-pass filtering method where the frequency response is chosen automatically depending on the estimated noise level. Our experimental results show that our proposed method, 3-D patch-based similarity (3DPBS), achieves the best denoising accuracy compared with graph-based state-of-the-art methods.
Knowledge-Aware Few Shot Learning for Event Detection from Short Texts

Event detection in a city is crucial for the government to listen to the voice of the citizens, be aware of the real occurrences in a city, and then make wiser policies. However, in reality some important events with few samples are easily to be overwhelmed by the massive information and hard to be recognized, and additionally the limited word description from the short texts even makes the recognition harder. To address the problems, we propose a knowledge-aware event detector by incorporating the external knowledge to detect the events with few examples. The external knowledge incorporation with different semantic relations is capable to enrich the short texts. In addition, we leverage the representative few shot learning framework to formulate the event detection as the text classification problem. The proposed model is evaluated on two widely event-detection datasets. The experiments show a consistent accuracy improvement. The findings validates that our model with the knowledge infusion is effective to detect the few shot events from the short texts.
A Bidirectional Joint Model for Spoken Language Understanding

Intent detection and slot filling are two fundamental and important tasks in spoken language understanding (SLU). Motivated by the fact that the intent and slots in a user utterance have a strong relationship, joint models that deal with both tasks in a single framework have become a predominant choice in SLU research. Most existing joint models build two different decoders on top of a shared weight encoder or exploit intent information to detect slots. Some joint models transfer information between two tasks implicitly. In this paper, we propose a bidirectional joint model for SLU that explicitly incorporates intent information into slot filling and slot information into intent detection. Specifically, we first predict a soft intent signal, which is fed into a biaffine classifier to recognize slots. Slot features are then employed along with the utterance representation to predict the final intent. We also introduce a loss function that takes into account three types of losses: soft intent detection, final intent detection, and slot filling. Experimental results on three benchmark datasets ATIS, Snips, and PhoATIS show that our model outperforms previous state-of-the-art models in both tasks with relative error reductions ranging from 6% to 22%.
EGAN: A Neural Excitation Generation Model Based on Generative Adversarial Networks with Harmonics and Noise Input

This paper presents a speech synthesis method based on source-filter modeling. The source model is powered by a neural excitation generator trained by GAN to produce an excitation signal with harmonics and noise input and conditioning spectral envelope derived from the WORLD vocoder. An overlap-add FFT filter implements the filter model to generate synthesized speech with the excitation signal and the spectral envelope. The harmonics inputs are 32-channel sine waves at frequencies of F0 and the corresponding harmonic over-tones to provide bases for periodic impulse trains. The noise input is a 3-channel Gaussian noise to infer aperiodicity. The neural excitation generator employs cascaded neural filter blocks to filter 35-channel harmonics and noise signals by amplifying the signal of each input channel with the instantaneous amplitude and adding noise to each channel conditioned by the spectral envelope. Since harmonics directly conditioned by F0 generate the excitation signal, the proposed speech synthesis method is intrinsically flexible in F0 manipulation. Experimental results on CMU ARCTIC and CSTR VCTK speech corpora demonstrate that the proposed method outperforms the conventional speech synthesizers in not only F0 manipulation but also in keeping good synthesis quality against unseen speakers.
Explanations for Automatic Speech Recognition

We address quality assessment for neural network based ASR by providing explanations that help increase our understanding of the system and ultimately help build trust in the system. Compared to simple classification labels, explaining transcriptions is more challenging as judging their correctness is not straightforward and transcriptions as a variable-length sequence is not handled by existing interpretable machine learning models.We provide an explanation for an ASR transcription as a subset of audio frames that is both a minimal and sufficient cause of the transcription. To do this, we adapt existing explainable AI (XAI) techniques from image classification - (1) Statistical Fault Localisation(SFL) [1] and (2) Causal [2]. Additionally, we use an adapted version of Local Interpretable Model-Agnostic Explanations (LIME) [3] for ASR as a baseline in our experiments. We evaluate the quality of the explanations generated by the proposed techniques over three different ASR – Google API [4], the baseline model of Sphinx [5], Deepspeech [6] – and 100 audio samples from the Commonvoice dataset [7].
Semi-Supervised Graph Ultra-Sparsifier Using Reweighted ℓ<sub>1</sub> Optimization

Graph representation learning with the family of graph convolution networks (GCN) provides powerful tools for prediction on graphs. As graphs grow with more edges, the GCN family suffers from sub-optimal generalization performance due to task-irrelevant connections. Recent studies solve this problem by using graph sparsification in neural networks. However, graph sparsification cannot generate ultra-sparse graphs while simultaneously maintaining the performance of the GCN family. To address this problem, we propose Graph Ultra-sparsifier, a semi-supervised graph sparsifier with dynamically-updated regularization terms based on the graph convolution. The graph ultra-sparsifier can generate ultra-sparse graphs while maintaining the performance of the GCN family with the ultra-sparse graphs as inputs. In the experiments, when compared to the state-of-the-art graph sparsifiers, our graph ultra-sparsifier generates ultra-sparse graphs and these ultra-sparse graphs can be used as inputs to maintain the performance of GCN and its variants in node classification tasks.
Cross Modality Knowledge Distillation for Robust Pedestrian Detection in Low Light and Adverse Weather Conditions

RGB-based pedestrian detection is a challenging task in low light and adverse weather conditions because image quality can degrade rather significantly. Including other modalities such as thermal and gated imaging sensors can, on the other hand, significantly improve the detection performance in these conditions. However, these sensors are expensive, and including them may cause design and manufacturing challenges. In this paper, we propose a new framework that utilizes Cross Modality Knowledge Distillation (CMKD) to improve the performance of RGB-only pedestrian detection in low light and adverse weather conditions. Specifically, we develop two CMKD methods that rely on feature-based knowledge distillation and adversarial training to transfer knowledge from a pedestrian detector (teacher) that is trained using multiple modalities to a single modality detector (student) that is trained using RGB images only. Experimental results using the "Seeing Through Fog" dataset show that both of our proposed methods outperform the baseline detector in terms of detection accuracy without increasing computational complexity during inference. In particular, the proposed methods reduce the performance gap between teacher and baseline models by up to 55%.
Speech-Text Based Multi-Modal Training with Bidirectional Attention for Improved Speech Recognition

To let the state-of-the-art end-to-end ASR model enjoy data efficiency, as well as much more unpaired text data by multi-modal training, one needs to address two problems: 1) the synchronicity of feature sampling rates between speech and language (aka text data); 2) the homogeneity of the learned representations from two encoders. In this paper we propose to employ a novel bidirectional attention mechanism (BiAM) to jointly learn both ASR encoder (bottom layers) and text encoder with a multi-modal learning method. The BiAM is to facilitate feature sampling rate exchange, realizing the quality of the transformed features for the one kind to be measured in another space, with diversified objective functions. As a result, the speech representations are enriched with more linguistic information, while the representations generated by the text encoder are more similar to corresponding speech ones, and therefore the shared ASR models are more amenable for unpaired text data pretraining. To validate the efficacy of the proposed method, we perform two categories of experiments with or without extra unpaired text data. Experimental results on Librispeech corpus show it can achieve up to 6.15% word error rate reduction (WERR) with only paired data learning, while 9.23% WERR when more unpaired text data is employed <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Kernel Estimation and Deconvolution for Blind Image Super-Resolution

Blind super-resolution, different from conventional non-blind super-resolution based on the assumption of fixed degradation, handles various unknown Gaussian blur kernels, and thus is closer to real-world application. The accuracy of kernel estimation and deconvolution directly influences the performance of overall super-resolution results, but recent works usually introduce artifacts during the process. In this paper, we propose our methods of a more accurate kernel estimation module (KEM) and deconvolution module (DM). Additionally, KEM and DM are embedded in kernel estimation and deconvolution structure (KEDS), which improves the results to a large extent once combined with non-blind networks.
Learning 3D Human Pose and Shape Estimation Using Uncertainty-Aware Body Part Segmentation

While exploiting body segmentations for supervision, existing 3D human pose and shape estimation methods are plagued by mismatches between clothed body segmentations and skinned SMPL model reprojections. Moreover, noisy pixels introduced by inaccurate segmentation annotations also prevent the model from improving the reconstruction performance further. To address these problems, we propose a novel generalizable framework called Uncertainty-aware body Part Segmentation (UPS), which penalizes different body parts with data uncertainty estimation. Specifically, we use sample-specific segmentations to supervise skinned and clothed body parts separately for realistic human mesh recovery. Furthermore, we leverage data uncertainty to improve the model capacity via learning from representative pixels and resisting noisy ones. Our extensive qualitative and quantitative experiments show that UPS achieves competitive reconstruction results on standard benchmarks.
Bipartite Graph Convolutional Networks with Adversarial Domain Transfer

Bipartite graphs have been widely used in many applications such as recommender systems, search engines and so on. Recent works consider bipartite graphs as homogeneous graphs and apply graph convolution networks for link prediction or node classification. However, in bipartite graphs, there are two types of nodes which are from different domains such as users and items in recommender systems, and cannot be in the same embedding space. In this paper, we proposed a novel graph convolution operation to propagate in bipartite graph with less spatial and temporal complexities, and two mapping functions with adversarial constraints to transfer features between two domains. Experimental results show that the proposed model achieves the improved performance on the tasks of link prediction and recommendation in real-world scenarios.
Dual Meta Calibration Mix for Improving Generalization in Meta-Learning

Meta-learning has achieved remarkable success as a powerful paradigm for transferring learned knowledge from previous tasks. However, the lack of a large number of diverse and qualified tasks is the bottleneck of current meta-learning, which can easily lead to overfitting and therefore seriously hurt the generalization ability. In this paper, to address this challenge, we proposed Dual Meta Calibration Mix (DMCM) to improve the diversity and quality of tasks with "more data". Concretely, we designed a dual augmentation framework and meta calibration mix. The dual augmentation framework augments individual tasks and pairs of tasks by linearly combining samples and labels from both support and query sets, respectively. The meta calibration mix generates new samples by linearly combining image patches and corresponding labels based on the calibrated mixing matrix and calibrated label. Extensive experiments show that our proposed method significantly improves the generalization of meta-learning algorithms and consistently outperforms other state-of-the-art regularization meta-learning methods.
A Lightweight Convolutional Neural Network using Feature Filtering Module

Most lightweight networks adopt simple feature fusion methods, such as element-wise addition, to achieve real-time speed. However, complex background and noise can affect these methods, making it difficult to locate objects accurately. At the same time, if a large number of channel connections are used to fuse the feature layer, the parameter quantity will increase dramatically. In this work, we propose a new network architecture with dense connection and feature filtering to tackle this problem. We introduce a feature filtering module (FFM) designed to implement feature fusion and filter features adaptively according to the global semantic information so as to reduce feature redundancy and aliasing after feature fusion. We verify the proposed network on CIFAR-10, CIFAR-100, Pascal VOC, and MS COCO. Compared with other state-of-the-art works, experimental results show that our network maintains lightweight while improving system accuracy on object recognition. In addition, our FFM can be applied to other models to improve the performance in object detection.
K<sup>2</sup>NN: Self-Supervised Learning with Hierarchical Nearest Neighbors for Remote Sensing

Self-supervised learning aims to learn applicable pre-trained models from massive unlabeled data. Besides image-level pretext tasks, many recent pixel-level studies have been pro-posed to learn dense information in each image. However, most of those methods focus on obtaining pair of matched patches from the same image with different augmentation. At the same time, little effort is devoted to exploiting matched patches from different images. In this work, we develop a novel pixel-level task that leverages an ensemble of nearest neighbors from multiple images to explore diverse objects in each image, especially for remote sensing data. Besides, a sampling strategy with a submodular function is adopted to efficiently update the memory bank consisting of patches. The extensive experiments on remote sensing data confirm the effectiveness of our method.
A Perceptual Neural Audio Coder with a Mean-Scale Hyperprior

This paper proposes an end-to-end neural audio coder based on a mean-scale hyperprior model together with a perceptual optimization using a psychoacoustic model (PAM)-based loss function. The proposed coder estimates the mean and scale hyperpriors using a sub-network after assuming that the probability distribution of latent samples is Gaussian. The main network is an autoencoder based on Resnet-type gated linear units (ResGLUs), each comprising a generalized divisive normalization (GDN) layer. We train both networks to optimize perceptual attributes estimated using a multi-timescale scheme to obtain high perceptual quality. Experimental results show that the proposed model accurately predicts the mean and scale hyperpriors. Also, it obtains consistently higher audio quality than the commercial MP3 audio coder at all bitrates.
Distinguishable Speaker Anonymization Based on Formant and Fundamental Frequency Scaling

Speech data on the Internet are proliferating exponentially because of the emergence of social media, and the sharing of such personal data raises obvious security and privacy concerns. One solution to mitigate these concerns involves concealing speaker identities before sharing speech data, also referred to as speaker anonymization. In our previous work, we have developed an automatic speaker verification (ASV)-model-free anonymization framework to protect speaker privacy while preserving speech intelligibility. Although the framework ranked first place in VoicePrivacy 2022 challenge, the anonymization was imperfect, since the speaker distinguishability of the anonymized speech was deteriorated. To address this issue, in this paper, we directly model the formant distribution and fundamental frequency (F0) to represent speaker identity and anonymize the source speech by the uniformly scaling formant and F0. By directly scaling the formant and F0, the speaker distinguishability degradation of the anonymized speech caused by the introduction of other speakers is prevented. The experimental results demonstrate that our proposed framework can improve the speaker distinguishability and significantly outperforms our previous framework in voice distinctiveness. Furthermore, our proposed method can trade off the privacy-utility by using different scaling factors.