Reviewerly/exHarmony huggingface dataset:
LongFNT: Long-Form Speech Recognition with Factorized Neural Transducer

Traditional automatic speech recognition (ASR) systems usually focus on individual utterances, without considering long-form speech with useful historical information, which is more practical in real scenarios. Simply attending longer transcription history for a vanilla neural transducer model shows no much gain in our preliminary experiments, since the prediction network is not a pure language model. This motivates us to leverage the factorized neural transducer structure, containing a real language model, the vocabulary predictor. We propose the LongFNT-Text architecture, which fuses the sentence-level long-form features directly with the output of the vocabulary predictor and then embeds token-level long-form features inside the vocabulary predictor, with a pre-trained contextual encoder RoBERTa to further boost the performance. Moreover, we propose the LongFNT architecture by extending the long-form speech to the original speech input and achieve the best performance. The effectiveness of our LongFNT approach is validated on LibriSpeech and GigaSpeech corpora with 19% and 12% relative word error rate (WER) reduction, respectively.
Suffix Retrieval-Augmented Language Modeling

Causal language modeling (LM) uses word history to predict the next word. BERT, on the other hand, makes use of bi-directional word information in a sentence to predict words at masked positions. While BERT is effective in sequence encoding, it is non-causal by nature and is not designed for sequence generation. In this paper, we propose a novel language model, SUffix REtrieval-Augmented LM (SUREALM), that simulates a bi-directional contextual effect in an autoregressive manner. SUREALM employs an embedding retriever to search for training sentences in a data store that share similar word history during sequence generation. In particular, the suffix portions of the retrieved sentences mimick the "future" context. We evalu-ated our proposed model on the DSTC9 spoken dialogue corpus and showed promising word perplexity reduction on the validation and test set compared to competitive baselines. Our source code is re-leased on GitHub <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speech

The majority of current Text-to-Speech (TTS) datasets, which are collections of individual utterances, contain few conversational aspects. In this paper, we introduce DailyTalk, a high-quality conversational speech dataset designed for conversational TTS. We sampled, modified, and recorded 2,541 dialogues from the open-domain dialogue dataset DailyDialog inheriting its annotated attributes. On top of our dataset, we extend prior work as our baseline, where a non-autoregressive TTS is conditioned on historical information in a dialogue. From the baseline experiment with both general and our novel metrics, we show that DailyTalk can be used as a general TTS dataset, and more than that, our baseline can represent contextual information from DailyTalk. The DailyTalk dataset and baseline code are freely available for academic use with CC-BY-SA 4.0 license <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Temporal Contrastive Learning with Curriculum

We present ConCur, a contrastive video representation learning method that uses curriculum learning to impose a dynamic sampling strategy in contrastive training. More specifically, ConCur starts the contrastive training with easy positive samples (temporally close and semantically similar clips), and as the training progresses, it increases the temporal span effectively sampling hard positives (temporally away and semantically dissimilar). To learn better context-aware representations, we also propose an auxiliary task of predicting the temporal distance between a positive pair of clips. We conduct extensive experiments on two popular action recognition datasets, UCF101, and HMDB51, on which our proposed method achieves superior performance on video action recognition and video retrieval. Detailed ablation studies show the effectiveness of each of the components of our proposed method.
Identifying Source Speakers for Voice Conversion Based Spoofing Attacks on Speaker Verification Systems

An automatic speaker verification system aims to verify the speaker identity of a speech signal. However, a voice conversion system could manipulate a person’s speech signal to make it sound like another speaker’s voice and deceive the speaker verification system. Most countermeasures for voice conversion-based spoofing attacks are designed to discriminate bona fide speech from spoofed speech for speaker verification systems. In this paper, we investigate the problem of source speaker identification – inferring the identity of the source speaker given the voice converted speech. To perform source speaker identification, we simply add voice-converted speech data with the label of source speaker identity to the genuine speech dataset during speaker embedding network training. Experimental results show the feasibility of source speaker identification when training and testing with converted speeches from the same voice conversion model(s). In addition, our results demonstrate that having more converted utterances from various voice conversion model for training helps improve the source speaker identification performance on converted utterances from unseen voice conversion models.
Improving Spoken Language Identification with Map-Mix

The pre-trained multi-lingual XLSR model generalizes well for language identification after fine-tuning on unseen languages. However, the performance significantly degrades when the languages are not very distinct from each other, for example, in the case of dialects. Low resource dialect classification remains a challenging problem to solve. We present a new data augmentation method that leverages model training dynamics of individual data points to improve sampling for the latent mixup. The method works well in low-resource settings where generalization is paramount. Our datamaps-based mixup technique, which we call Map-Mix, improves weighted F1 scores by 2% compared to the random mixup baseline and results in a significantly well-calibrated model. The code for our method is open-sourced on github.
Convolution-Based Channel-Frequency Attention for Text-Independent Speaker Verification

Deep convolutional neural networks (CNNs) have been applied to extracting speaker embeddings with significant success in speaker verification. Incorporating the attention mechanism has shown to be effective in improving the model performance. This paper presents an efficient two-dimensional convolution-based attention module, namely C2D-Att. The interaction between the convolution channel and frequency is involved in the attention calculation by lightweight convolution layers. This requires only a small number of parameters. Fine-grained attention weights are produced to represent channel and frequency-specific information. The weights are imposed on the input features to improve the representation ability for speaker modeling. The C2D-Att is integrated into a modified version of ResNet for speaker embedding extraction. Experiments are conducted on VoxCeleb datasets. The results show that C2D-Att is effective in generating discriminative attention maps and outperforms other attention methods. The proposed model shows robust performance with different scales of model size and achieves state-of-the-art results.
ROI-Based Deep Image Compression with Swin Transformers

Encoding the Region Of Interest (ROI) with better quality than the background has many applications including video conferencing systems, video surveillance and object-oriented vision tasks. In this paper, we propose a ROI-based image compression framework with Swin transformers as main building blocks for the autoencoder network. The binary ROI mask is integrated into different layers of the network to provide spatial information guidance. Based on the ROI mask, we can control the relative importance of the ROI and non-ROI by modifying the corresponding Lagrange multiplier λ for different regions. Experimental results show our model achieves higher ROI PSNR than other methods and modest average PSNR for human evaluation. When tested on models pre-trained with original images, it has superior object detection and instance segmentation performance on the COCO validation dataset.
Semi-Supervised Semantic Segmentation with Structured Output Space Adaption

Semi-supervised semantic segmentation methods rely on dense pixel-level classification with limited data and can thus be developed to adapt source ground truth labels to a target domain. In this paper, we creatively propose a method for semi-supervised semantic segmentation. The key innovation is our adversarial learning method for space adaptation in context, which can be regarded as a structured output that contains spatial similarities between unlabeled data and labeled data. To achieve this, we construct an adversarial learning network to efficiently adapt to the structural output space in labeled and unlabeled similar samples. Furthermore, we introduce two learning strategies into semi-supervised semantic segmentation, one that can selectively capture intra-category and inter-category context dependencies, resulting in robust feature representations. While the other explicitly concatenates the shape information of objects as a separate processing branch to produce sharper predicted boundaries of objects. Experimental results on two well-known benchmark datasets show that our method achieves better performance compared to the previous competitive models.
Code-Switching Speech Synthesis Based on Self-Supervised Learning and Domain Adaptive Speaker Encoder

Recently, end-to-end speech synthesis models based on deep learning have made great progress in speech quality, and gradually replaced traditional speech synthesis methods into the mainstream. However, these methods are still challenging to synthesize highly natural speech. In order to solve the above problems, we introduce self-supervised learning and frame-level domain adversarial training into the speaker encoder based on the speaker verification task, so that the speaker vectors of different languages keep a consistent distribution in the speaker space, and the performance of speech synthesis is improved. In addition, we use a non-autoregressive speech synthesis model in the selection of speech synthesis model, so as to solve the problem of unnatural speech rate caused by cross-language speech synthesis. We first demonstrate that in the mixed language dataset of LibriTTS and AISHELL3, the speaker encoder trained with self-supervised representation has a 4.968% absolute EER reduction compared to the traditional MFCC on the speaker verification task, indicating that self-supervised representation has better generalization for domain-complex datasets. Then we obtain MOS scores of 3.635 and 3.675 for speech naturalness and speaker similarity in the code-switching speech synthesis task, respectively. Our approach simplifies the need to use multiple monolingual encoders to model linguistic information in the past literature, and adds frame-level domain adversarial training to optimize the speaker vectors in the speaker feature space to facilitate the code-switching speech synthesis task.
FedSD: A New Federated Learning Structure Used in Non-iid Data

One of the most challenging problems in federated learning is the convergence speed problem caused by heterogeneity. We propose a novel structure called FedSD, a new method to accelerate the model convergence. We change the one-stage-cycle iteration structure to a 2-stage-cycle one to get the latest global gradient descent direction which can guide the model training direction. We instantiate algorithms using FedSD to improve the performance of experiments on several public datasets. Our empirical studies validate the excellent performance of FedSD.
Class-Incremental Learning on Multivariate Time Series Via Shape-Aligned Temporal Distillation

Class-incremental learning (CIL) on multivariate time series (MTS) is an important yet understudied problem. Based on practical privacy-sensitive circumstances, we propose a novel distillation-based strategy using a single-headed classifier without saving historical samples. We propose to exploit Soft-Dynamic Time Warping (Soft-DTW) for knowledge distillation, which aligns the feature maps along the temporal dimension before calculating the discrepancy. Compared with Euclidean distance, Soft-DTW shows its advantages in overcoming catastrophic forgetting and balancing the stability-plasticity dilemma. We construct two novel MTS-CIL benchmarks for comprehensive experiments. Combined with a prototype augmentation strategy, our framework demonstrates significant superiority over other prominent exemplar-free algorithms.
Towards Real-Time Person Search with Invariant Feature Learning

Person search aims to locate a query person in a gallery of unconstrained scene images, which has many real-world applications. However, existing methods directly build off of advances in object detection for better performance rather than efficiency. Complex designs in heavy-weight detectors are redundant for person search. Furthermore, challenges in person search force existing methods to employ additional modules, which greatly deteriorates models’ efficiency. In this paper, we propose a novel real-time framework for both effective and efficient person search, termed as InvarPS. InvarPS optimizes the over-designed network with invariant feature learning. Specifically, considering the main challenges (i.e., appearance changes, scale variations, and conflicting tasks) in person search, we propose an improved backbone, a Single-Scale Feature Fusion (SSFF) module and a Hierarchical Decoupling Head (HDH) to facilitate the model learning appearance, scale, and task invariant features, respectively. Extensive experiments demonstrate that our method achieves state-of-the-art performance with real-time speed (>100 FPS), which is significantly faster than any previous competitive approach.
Mendam: Multi-Expert Network with Distribution-Aware Momentum for Long-Tailed Recognition

Long-tailed data distribution (i.e., minority classes occupy most of the data, while most classes have very few samples) is a common problem in image classification. The existing deferred re-balancing methods suffer from low accuracy for tail classes due to their bias towards head classes. In this paper, we find that properly adjusting the momentum can significantly improve the performance for class-imbalanced tasks, which provides a novel perspective to solve this thorny problem. Based on this finding, we propose a new way to calculate a distribution-aware momentum (DAM) based on the data distribution to avoid bias towards head classes. Furthermore, we design a multi-expert network (MEN) to designate each expert to focus on different tail classes. We achieve new state-of-the-art on CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT and iNaturalist2018 for image classification.
Enabling Large-Scale Image Search with Co-Attention Mechanism

Content-based image retrieval (CBIR) consists of searching the most similar images to a given query. Most existing attention mechanisms for CBIR are query non-sensitive and are only based on single candidate image’s feature regardless of the actual query content. This can result in incorrect regions especially when the target object is not salient or surrounded by distractors. This paper proposes an efficient and effective query sensitive co-attention mechanism for large scale CBIR tasks. Local feature selection and clustering are employed to reduce the computation cost caused by the query sensitivity. Experimental results indicate that the proposed co-attention method can generate good co-attention maps even under challenging situations leading to a new state of the art performance on several benchmark datasets.
Output-Dependent Gaussian Process State-Space Model

Gaussian process state-space model (GPSSM) is a fully probabilistic state-space model that has attracted much attention over the past decade. However, the outputs of the transition function in the existing GPSSMs are assumed to be independent, meaning that the GPSSMs cannot exploit the inductive biases between different outputs and lose certain model capacities. To address this issue, this paper proposes an output-dependent and more realistic GPSSM by utilizing the well-known, simple yet practical linear model of coregionalization (LMC) framework to represent the output dependency. To jointly learn the output-dependent GPSSM and infer the latent states, we propose a variational sparse GP-based learning method that only gently increases the computational complexity. Experiments on both synthetic and real datasets demonstrate the superiority of the output-dependent GPSSM in terms of learning and inference performance.
Moving Towards Non-Binary Gender Identification Via Analysis of System Errors in Binary Gender Classification

This paper aims to analyse human perceptions of gender in speech signals, focusing on signals that are misclassified by methods for binary gender classification, looking at the features of speech signals that are more likely to be misclassified, or classified as either nonbinary or unclassifiable. The paper also analyses how human subjects perform in classifying such speech signals to gain insight into differences between machine and human performance levels. It is shown that gender classification systems and human ratings lack inter-annotator agreement, as do human ratings considered individually. There is also discussion of the suitability of continuing to use a binary system for gender in the field. This work fits into a larger body of research ongoing in the area of speech technology for trans-gender voice therapy.
Hearing and Seeing Abnormality: Self-Supervised Audio-Visual Mutual Learning for Deepfake Detection

The recent development of deepfakes has resulted in serious threats to society, such as spreading misinformation, defamation, etc. Although recent deepfake detection methods are capable of achieving satisfactory results for seen forgeries, the performance drops significantly for unseen ones. With proper supervised pretraining on auxiliary tasks as prior, the situation can be improved, but the requirement to collect a large number of additional annotations for these tasks may restrict the further development of a generalized deep-fake detector. To address this issue, we propose an Audio-Visual Temporal Synchronization for Deepfake Detection framework for detecting deepfakes that maintains reasonable detection capabilities for unseen ones. The primary objective of our framework is to determine whether there has been a forgery by evaluating the consistency between the sound and the faces in a video clip, together with the relationship between the two features. First, the spatiotemporal feature extraction network is pretrained in a self-supervised manner by exploiting the audio-visual temporal synchronization task to build up a rich representation based on the temporal synchronization relationship between the audio and its corresponding video. For pretraining, we use only real data and carefully selected negative samples with contrastive loss to train the model. A temporal classifier network is used to determine whether or not the video has been manipulated using the representations obtained from the pretrained feature extraction networks. To prevent the model from overfitting to certain manipulation-specific artifacts, we froze the feature extraction networks and only trained the final classifier network on forged data. Extensive experiments on unseen forgery categories and unseen datasets have shown the effectiveness of our method to achieve state-of-the-art results.
BadRes: Reveal the Backdoors Through Residual Connection

Generally, residual connections are indispensable network components in building Convolutional Neural Networks(CNNs) and Transformers for various downstream tasks in Computer Vision(CV), which encourages skip/short cuts between network blocks. However, the layer-by-layer loopback residual connections may also hurt the model’s robustness by allowing unsuspecting input. In this paper, we proposed a simple yet strong backdoor attack method called BadRes, where the residual connections play as a turnstile to be deterministic on clean inputs while unpredictable on poisoned ones. We have performed empirical evaluations on four datasets with ViT and BEiT models, and the BadRes achieves 97% attack success rate without any performance degradation on clean data. Moreover, we analyze BadRes with state-of-the-art defense methods and reveal the fundamental weakness lying in residual connections.
End-to-End Word-Level Disfluency Detection and Classification in Children’s Reading Assessment

Disfluency detection and classification on children’s speech has a great potential for teaching reading skills. Word-level assessment of children’s speech can help teachers to effectively gauge their students’ progress. Hence, we propose a novel attention-based model to perform word-level disfluency detection and classification in a fully end-to-end (E2E) manner making it fast and easy to use. We develop a word-level disfluency annotation scheme using which we annotate a dataset of children read speech, the reading races dataset (READR). We also annotate disfluencies in the existing CMU Kids corpus. The proposed model significantly outperforms traditional cascaded baselines, which use forced alignments, on both datasets. To deal with the inevitable class-imbalance in the datasets, we propose a novel technique called HiDeC (Hierarchical Detection and Classification) which yields a detection improvement of 23% and 16% and a classification improvement of 3.8% and 19.3% relative F1-score on the READR and CMU Kids datasets respectively.
Lit the Darkness: Three-Stage Zero-Shot Learning for Low-Light Enhancement with Multi-Neighbor Enhancement Factors

Low-light images represent an obstacle for computer vision tasks due to the lack of perceptual quality. Also, it is challenging to enhance images and adapt to different illumination conditions. To address this problem, we introduce a zero-shot learning approach. We use a 3-stage model trained without the need for paired or unpaired images to improve the lighting and texture of images. The first stage extracts an enhancement pixel-wise map using depth-wise separable convolution. It also tries to extract enhancement factors so that it can consider results e.g., neighbors from other steps in the following stage. The second stage is a recurrent network that enhances the image iteratively while keeping a small model size. The third stage represents an unsupervised network to preserve semantic information and benefit from it during training. We show extensive experiments on benchmark datasets to compare our model with previous state-of-the-art models quantitatively and qualitatively.
TransLink: Transformer-Based Embedding for Tracklets’ Global Link

Multi-object tracking (MOT) is essential to many tasks related to the smart transportation. Detecting and tracking humans on the road can give a vital feedback for either the moving vehicle or traffic control to ensure better driving safety and traffic flow. However, most trackers face a common problem of identity (ID) switch, resulting in an incomplete human trajectory prediction. In this paper, we propose a Transformer-based tracklet linking method called TransLink to mitigate the association failures. Specifically, the self-attention mechanism is well exploited to get the feature representation for tracklets, followed by a multilayer perceptron to predict the association likelihood, which can be further used in determining the tracklet association. Experiments on the MOT dataset demonstrate the effectiveness of the proposed module in lifting the tracking performances.