Reviewerly/exHarmony huggingface dataset:
Cumulative Attention Based Streaming Transformer ASR with Internal Language Model Joint Training and Rescoring

This paper presents an approach to improve the performance of streaming Transformer ASR by introducing an internal language model (ILM) as a part of the decoder layers. In the recently pro- posed cumulative attention (CA) based streaming ASR system, only the last or top few decoder layers are equipped with the CA module. Thus in this work, we propose to train the bottom (non-CA) layers as an ILM using an auxiliary LM loss jointly with the rest of the system. During inference, the outputs of the ILM are interpolated with those of the entire Transformer decoder as done in the conventional external language model (ELM) rescoring. The paper also proposes a refinement to the CA algorithm known as CTC look-ahead, in order to improve the precision of endpoint detection. Experiments conducted on AIShell-1, Aidatatang and Librispeech datasets show that the proposed ILM rescoring method achieves on par or better ASR performance when compared to the ELM rescoring baseline. Also, the CTC look-ahead strategy effectively alleviates the early end-of- speech (EOS) triggering issue suffered by the CA module, without bringing noticeable latency degradation.
Gaze Pre-Train For Improving Disparity Estimation Networks

In the process of training Neural Networks, pre-training is an unsupervised training process that uses automatically generated labels for real end-goal task inputs. It usually precedes a supervised training stage, can improve neural network performance, and can reduce training loss. In this work, we used pre-training in the automotive domain where the setup was composed of a camera aimed outside the vehicle and an eye tracking system observing the driver. Our pre-training process used images from the camera as input and eye gaze direction as the automatic generated label. The eye gaze pre-training goal was to initiate and thus improve disparity estimation networks.Selecting eye gaze as labels is the best of both worlds. On one hand, it is somewhat similar to supervised training. Labels are generated by humans, by drivers who have deep understanding of the scenes and driving situation. On the other hand, it is similar to unsupervised training. The labels can be generated automatically. Large quantities of data can be collected easily. Overall, the eye gaze pre-train helped reduce the L1 loss from 0.65 when not using pre-train to 0.45 when using it on the validation set.
Repetition Counting from Compressed Videos Using Sparse Residual Similarity

It is common for modern video codecs to reach triple digit compression ratios, which clearly shows the information redundancy and low information density of the ubiquitous RGB frame video representation. We propose an approach that directly utilizes the components of a compressed video for predicting the count of a repeating action occurring in the video. This complete bypassing of the video decoding step offers significant computational benefits. Furthermore, by leveraging intelligent single I-frame encodings and the sparse nature of accumulated residual vectors, we are able to efficiently capture the frame features even with lightweight feature extraction backbones. On the Countix dataset, our method achieves a considerable 91.5% reduction in model size and 91% reduction in FLOPS, with competitive results compared to the state-of-the-art.
A Synthetic Corpus Generation Method for Neural Vocoder Training

Nowadays, neural vocoders are preferred for their ability to synthesize high-fidelity audio. However, training a neural vocoder requires a massive corpus of high-quality real audio, and the audio recording process is often labor-intensive. In this work, we propose a synthetic corpus generation method for neural vocoder training, which can easily generate synthetic audio with an unlimited number at nearly no cost. We explicitly model the prior characteristics of audio from multiple target domains simultaneously (e.g., speeches, singing voices, and instrumental pieces) to equip the generated audio data with these characteristics. And we show that our synthetic corpus allows the neural vocoder to achieve competitive results without any real audio in the training process. To validate the effectiveness of our proposed method, we performed empirical experiments on both speech and music utterances in subjective and objective metrics. The experimental results show that the neural vocoder trained with the synthetic corpus produced by our method can generalize to multiple target scenarios and has excellent singing voice (MOS: 4.20) and instrumental piece (MOS: 4.00) synthesis results.
Line Segment Matching Based on Intersection-Enhanced Point Correspondences

Line segment matching between two images of the same scene is popularly performed with the assistant of feature point correspondences. However, each existing method only relies on one specific kind of feature points to establish the correspondences, and could yield inferior performance when insufficient high-quality point correspondences are supplied. To address this fundamental problem, a novel line segment matching method is proposed. In our method, ASIFT points are exploited to establish an initial set of correspondences. Then, line intersections are matched between the two input images to establish an additional set of correspondences, i.e., intersection correspondences. Finally, an enhanced set of point correspondences is generated via a union of the ASIFT and intersection correspondences. Based on the yielded intersection-enhanced point correspondences, it will be shown that the performance of line segment matching can be greatly improved.
PRIME: 3D Human Pose and Body Shape Recovery with Perspective Projection

Existing monocular 3D human pose and body shape (HPS) estimation methods make the coplanar assumption and use weak perspective projection in order to simplify the problem setting for images in the wild. However, weak perspective projection inevitably introduce prediction biases. To address this issue, we propose a plug-and-play Perspective Residual Log-likehood on Monocular 3D HPS Estimation (PRIME) module to significantly improve the accuracy of monocular 3D HPS estimation with trivial sacrifice on running time. PRIME applies full perspective projection to construct 2D re-projection loss or extract mesh-alignment features. Specifically, PRIME estimates the distribution of 2D joints and scale to calculate the perspective translation with the focal length. Further, we introduce side view constrain (SVC) of 2D joints to reduce the ambiguity of 3D HPS recovery. Experimental results demonstrate the effectiveness of our method.
Lightweight Fisher Vector Transfer Learning for Video Deduplication

Video deduplication in cloud and on devices is a key challenge for storage and communication efficiency. The lifetime of video content creation, communication/sharing, and consumption can generate multiple versions of the same content with variations in coding and editing effects. In this work, we develop a lightweight and robust deduplication feature based on the fisher vector aggregation of Scale-Invariant Feature Transform (SIFT) keypoints. The fisher vector representation is used for a deduplication transfer learning process that utilizes a lightweight Multilayer Perceptron (MLP) network with center loss to learn a compact and distinctive feature. Simulation on the CC_WEB_VIDEO dataset demonstrated that the proposed feature is extremely robust in deduplication with respect to typical editing effects and coding/transcoding degenerations while being computationally very lightweight compared to other solutions.
Abusive Activity Detection with Multi-Modality Based on Convolutional Neural Network

Abusive activity frequently occurs in various places, including nursing homes for the elderly. However, it is difficult to detect because it has various forms and is not easy to define. Therefore, in this study, we try to detect using the Convolutional Neural Network (CNN). Data based on multimodality (image and sound) was used to improve detection performance. In case of the image, a Motion History Image (MHI) was used to reflect motion information. The developed multi-modality-based detection system showed a performance of 0.807 (Accuracy) and 0.858 (F1 score). By comparing with the system using only image or sound, there is an improvement of 0.132 ~ 0.17 (Accuracy) and 0.094 ~ 0.138 (F1 score)
Space-Time Variable Density Samplings for Sparse Bandlimited Graph Signals Driven by Diffusion Operators

We consider the space-time sampling and reconstruction of sparse bandlimited graph signals driven by a heat diffusion process. In this paper, we develop a sampling framework consisting of selecting a small subset of space-time nodes at random according to some probability distribution, generalizing the classical variable density sampling to the heat diffusion field. We show that the number of space-time samples required to ensure stable recovery depends on an incoherence parameter determined by the interplay between graph topology, temporal dynamics, and sampling probability distributions. In optimal scenarios, as few as $\mathcal{O}\left( {s\log k} \right)$ space-time samples are sufficient to ensure accurate recovery of all k-bandlimited graph signals that are additionally s-sparse. Our proposed sampling method requires much fewer spatial samples than the static case by leveraging temporal information. Finally, we test our sampling techniques on a wide variety of graphs. The numerical results on synthetic and real climate data sets support our theoretical findings and demonstrate the practical applicability.
End-to-End Unsupervised Sketch to Image Generation

A freehand sketch is geometrically distorted and lacks colors, textures, and other visual details. This leads to challenges in freehand sketch-to-image generation tasks. Many existing works take a multistage strategy to generate images from sketches, and they depend on the shape of the input sketch badly and ignore the shape adjustment of the generated image. More importantly, the generated images are of low quality with distorted textures and colors, and shape deformation. In order to overcome the above challenges, we propose an end-to-end method to accomplish the freehand sketch-to-image task, and the proposed architecture is based on an unsupervised network. Our key insight is to propose a shape discriminator to provide shape constraints for the generated image. Besides, we introduce Image-Global Attention(IGA) and Focal Frequency Loss(FFL). IGA and FFL mainly focuses on the whole image and every patch of an image, respectively. We also extend a new dataset called NewGiraffe based on the giraffe class of SketchyCOCO, and our approach is validated on two datasets: Shoes and NewGiraffe. Through qualitative and quantitative results, we demonstrate our method’s ability to generate realistic images from freehand sketches.
Label-Efficient and Robust Learning from Multiple Experts

Learning from multiple sources of labeled data presents unique challenges, including the cost of using many annotators in the training, test, and production time and their limited reliability. In this paper, we analyze the problem of label-efficient learning and propose a method for training a classification system from data labeled by multiple annotators, where only a small subset of them is chosen adaptively to reduce later communication effort and the effect of malicious annotators, especially in the validation and test phase. We define an iterative optimization procedure where the annotation weight vector’s sparsity is enforced while reducing the overall classification error. Using real-world data, we show that our approach can pertain to the classification performance on the complete set of annotators while reducing the need for annotator labels by over 50% on several different tasks. Furthermore, we demonstrate that label sparsity reduces label flip attacks’ effect on performance.
Global and Nodal Mutual Information Maximization in Heterogeneous Graphs

Many real-world graphs involve different types of nodes and edges, being heterogeneous by nature. Heterogeneous graph representation learning embeds their rich structure and semantics into a low-dimensional space to facilitate graph related tasks. In this work, we propose a self-supervised method that learns representations by relying on mutual information maximization among different graph structures (metapaths). Our method, termed HeMI, promotes node-level and global-level shared semantics among nodes with contrastive learning, as well as it leverages interactions among metapaths. Experiments on node classification, node clustering, and link prediction show that HeMI outperforms existing approaches.
Seri: Sketching-Reasoning-Integrating Progressive Workflow for Empathetic Response Generation

Empathy is a key ability for a human-like dialogue system. Inspired by social psychology, empathy includes both affective and cognitive aspects. Previous works on this topic have merely focused on recognizing emotions or modeling cognition with commonsense knowledge. Nevertheless, the generated results of these works still have a big gap with human-like empathetic responses. In this paper, we propose Seri, a SkEtching-Reasoning-Integrating framework for empathetic response generation. In particular, we define an empathy planner to capture and reason about multi-source information that considers cognition and affection. Further, we introduce a dynamic integrator module that allows the model dynamically select the appropriate information to generate empathetic responses. Experimental results on EmpatheticDialogue show that our method outperforms competitive baselines and generates responses with higher diversity and cognitive empathy levels.
Estimating Normalized Graph Laplacians in Financial Markets

Gaussian Markov random fields, a class of graphical models, play an increasingly important role in real-world problems, where they are often applied to uncover conditional correlations between pairs of entities in a network. Motivated by recent applications of graphs in financial markets, we investigate the problem of learning undirected, weighted, normalized, graphical models. More precisely, we design an optimization algorithm to learn precision matrices that are modeled as normalized graph Laplacians. The proposed algorithm takes advantages of frameworks such as the alternating direction method of multipliers and projected gradient descent, which allows us to decompose the original problem into subproblems that can be solved efficiently. We demonstrate the empirical performance of the proposed algorithm, in comparison to state-of-the-art benchmark models, in a number of datasets involving financial time-series.
Self-Convolution for Automatic Speech Recognition

Self-attention plays a significant role in recent automatic speech recognition (ASR) models with promising results. However, it suffers from high computational complexity and weak capability in modeling local information. In contrast, the convolutional neural network (CNN) is computationally effective and superior in learning local information. Whereas it fails in self-interaction and capturing long-range dependence among input tokens. Accordingly, we take their complementary advantages and propose a new module, namely self-convolution, to compensate for each individual limitations. Specifically, self-convolution generates convolution kernels at each token (to model local information) which are then used to convolve itself (for self-interaction). Moreover, we bring in global information during the generation of convolution kernel to enhance the learning of long-range dependencies. In this way, the advantages of self-attention and CNN are both utilized. We conduct rigorous experiments on LibriSpeech, Tedlium2, and AIShell1 datasets and demonstrate that our proposed self-convolution can achieve superior ASR performance than self-attention with less computational cost.
Learning From Positive and Unlabeled Data Using Observer-GAN

In the case of learning from positive and unlabeled data, the input data consist of (1) observations from the positive class and their corresponding labels and (2) unlabeled observations from both positive and negative classes. Generative Adversarial Networks (GANs) have been used to reduce the problem to the supervised setting with the advantage that supervised learning has state-of-the-art accuracy in classification tasks. In order to generate pseudo-negative observations, GANs are trained on positive and unlabeled observations with a modified loss. Using both positive and pseudo-negative observations leads to a supervised learning setting. The generation of pseudo-negative observations that are realistic enough to replace missing negative class samples is a bottleneck for current GAN-based algorithms. By including an additional classifier into the GAN architecture, we describe a novel GAN-based approach. The GAN discriminator instructs the generator to only produce samples that fall into the unlabeled data distribution, while a second classifier (observer) network monitors the GAN training to: (i) prevent the generated samples from falling into the positive distribution; and (ii) learn the features that are the key distinction between the positive and negative observations. Experiments on four image datasets demonstrate that our trained observer network performs better than existing techniques in discriminating between real unseen positive and negative samples.
Improving Learning Objectives for Speaker Verification from the Perspective of Score Comparison

Deep speaker embedding systems are usually trained with classification-based or end-to-end learning objectives. Popular end-to-end approaches utilize deep metric learning, which can be viewed as a few-shot classification objective. In this paper, we investigate the limit of conventional learning objectives in speaker verification and propose a new learning objective designed from the perspective of similarity scores. The proposed method trains a network by score comparison unbound from the classification, which is more suitable for verification tasks. Experiments conducted with popular speaker embedding networks demonstrate the improvements on the VoxCeleb dataset using the proposed loss.
Disentangled Feature Learning for Real-Time Neural Speech Coding

Recently end-to-end neural audio/speech coding has shown its great potential to outperform traditional signal analysis based audio codecs. This is mostly achieved by following the VQ-VAE paradigm where blind features are learned, vector-quantized and coded. In this paper, instead of blind end-to-end learning, we propose to learn disentangled features for real-time neural speech coding. Specifically, more global-like speaker identity and local content features are learned with disentanglement to represent speech. Such a compact feature decomposition not only achieves better coding efficiency by exploiting bit allocation among different features but also provides the flexibility to do audio editing in embedding space, such as voice conversion in real-time communications. Both subjective and objective results demonstrate its coding efficiency and we find that the learned disentangled features show comparable performance on any-to-any voice conversion with modern self-supervised speech representation learning models with far less parameters and low latency, showing the potential of our neural coding framework.
Multilingual End-To-End Spoken Language Understanding For Ultra-Low Footprint Applications

Tiny Signal-to-Interpretation (TinyS2I) has been recently introduced as an ultra low-footprint end-to-end spoken language understanding (SLU) model. This architecture is capable of running in ultra resource constrained environments like voice assistant devices, while at the same time reducing latency. In this work, we propose an extension to TinyS2I and train a multilingual system supporting several languages. Multilingual TinyS2I models show little to no degradation compared to their monolingual counterparts. Increasing the network size in width and depth improves the classification accuracy for mono- and multilingual setups, with the multilingual one improving beyond the monolingual accuracy. This enables users to interact with the device in the language of their choice and dynamically switch between languages without an explicit language setting or accuracy degradation.
MEET: A Monte Carlo Exploration-Exploitation Trade-Off for Buffer Sampling

Data selection is essential for any data-based optimization technique, such as Reinforcement Learning. State-of-the-art sampling strategies for the experience replay buffer improve the performance of the Reinforcement Learning agent. However, they do not incorporate uncertainty in the Q-Value estimation. Consequently, they cannot adapt the sampling strategies, including exploration and exploitation of transitions, to the complexity of the task. To address this, this paper proposes a new sampling strategy that leverages the exploration-exploitation trade-off. This is enabled by the uncertainty estimation of the Q-Value function, which guides the sampling to explore more significant transitions and, thus, learn a more efficient policy. Experiments on classical control environments demonstrate stable results across various environments. They show that the proposed method outperforms state-of-the-art sampling strategies for dense rewards w.r.t. convergence and peak performance by 26% on average.
Benchmarking Cross-Domain Face Recognition with Avatars, Caricatures and Sketches

The accuracy of face recognition skyrocketed in past years and its robustness towards various covariates has been shown, such as variations in pose or age. In recent years, a considerable amount of research efforts has been devoted to cross-domain face recognition aiming at comparing facial images obtained from domains that are different in capture technologies, e.g. visible spectrum versus infrared, or signal representations, e.g. photographs versus sketches. Yet, various relevant domains have hardly been explored and a lack of public databases hampers the development of new algorithms.In this work, we introduce the HDA Cross-Domain (HDA-CD) face image database comprising 1,400 face images from three different domains including avatars, caricatures, and sketches. Said face images were manually generated using popular mobile apps. In a benchmark, we evaluate commercial and open-source state-of-the-art facial analysis methods on the HDA-CD database including face detection and recognition. For the latter task, generated facial images are compared against their original counter-parts. The HDA-CD database is made publicly available at: https://dasec.h-da.de/hda-cdfdb/
Detecting Out-of-Distribution Examples Via Class-Conditional Impressions Reappearing

Out-of-distribution (OOD) detection aims at enhancing standard deep neural networks to distinguish anomalous inputs from original training data. Previous progress has introduced various approaches where the in-distribution training data and even several OOD examples are prerequisites. However, due to privacy and security, auxiliary data tends to be impractical in a real-world scenario. In this paper, we propose a data-free method without training on natural data, called Class-Conditional Impressions Reappearing (C2IR), which utilizes image impressions from the fixed model to recover class-conditional feature statistics. Based on that, we introduce Integral Probability Metrics to estimate layer-wise class-conditional deviations and obtain layer weights by Measuring Gradient-based Importance (MGI). The experiments verify the effectiveness of our method and indicate that C2IR outperforms other post-hoc methods and reaches comparable performance to the full access (ID and OOD) detection method, especially in the far-OOD dataset (SVHN).