Reviewerly/exHarmony huggingface dataset:
On-the-Fly Text Retrieval for end-to-end ASR Adaptation

End-to-end speech recognition models are improved by incorporating external text sources, typically by fusion with an external language model. Such language models have to be retrained whenever the corpus of interest changes. Furthermore, since they store the entire corpus in their parameters, rare words can be challenging to recall. In this work, we propose augmenting a transducer-based ASR model with a retrieval language model, which directly retrieves from an external text corpus plausible completions for a partial ASR hypothesis. These completions are then integrated into subsequent predictions by an adapter, which is trained once, so that the corpus of interest can be switched without incurring the computational overhead of retraining. Our experiments show that the proposed model significantly improves the performance of a transducer baseline on a pair of question-answering datasets. Further, it outperforms shallow fusion on recognition of named entities by about 7% relative; when the two are combined, the relative improvement increases to 13%.
Joint Training and Decoding for Multilingual End-to-End Simultaneous Speech Translation

Recent studies on end-to-end speech translation(ST) have facilitated the exploration of multilingual end-to-end ST and end-to-end simultaneous ST. In this paper, we investigate end-to-end simultaneous speech translation in a one-to-many multilingual setting which is closer to applications in real scenarios. We explore a separate decoder architecture and a unified architecture for joint synchronous training in this scenario. To further explore knowledge transfer across languages, we propose an asynchronous training strategy on the proposed unified decoder architecture. A multi-way aligned multilingual end-to-end ST dataset was curated as a benchmark testbed to evaluate our methods. Experimental results demonstrate the effectiveness of our models on the collected dataset. Our codes and data are available at: https://github.com/XiaoMi/TED-MMST.
UAV Remote Sensing Image Dehazing Based on Multi-Dimensional Saliency Awareness Unequal Network

Current UAV image haze removal methods often suffer from problems of insufficient dehazing and spectrum distortion, especially in regions with rich spectrum and texture information. In this paper, we propose a multi-dimensional saliency awareness unequal network to avoid texture loss and color distortions. First, we design an unequal network structure, which enhances local color and texture detail feature learning. Specifically, we propose a saliency dense block, which employs a saliency map to guide the network to pay attention to the regions with rich spectrum and texture information unequally. Second, a multi-dimensional saliency detection method is proposed. It fuses the features of different dimensions to extract the salient regions, to obtain the spectrum and texture information. Finally, a loss function combining mean square error and edge loss is defined to enhance the learning of the texture. The experimental results demonstrate that the proposed method is capable of restoring color information as well as reserving texture details, especially for salient regions of UAV hazy images.
Mixed Sample Augmentation for Online Distillation

Mixed Sample Regularization (MSR), such as MixUp or CutMix, is a powerful data augmentation strategy to generalize convolutional neural networks. Previous empirical analysis has illustrated an orthogonal performance gain between MSR and conventional offline Knowledge Distillation (KD). To be more specific, student networks can be enhanced with the involvement of MSR in the training stage of sequential distillation. Yet, the interplay between MSR and online knowledge distillation, where an ensemble of peer students learn mutually from each other, remains unexplored. To bridge the gap, we make the first attempt at incorporating CutMix into online distillation, where we empirically observe a significant improvement. Encouraged by this fact, we propose an even stronger MSR specifically for online distillation, named as Cut <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sup> Mix. Furthermore, a novel online distillation framework is designed upon Cut <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sup> Mix, to enhance the distillation with feature level mutual learning and a self-ensemble teacher. Comprehensive evaluations on CIFAR10 and CIFAR100 with six network architectures show that our approach can consistently outperform state-of-the-art distillation methods.
M-CTRL: A Continual Representation Learning Framework with Slowly Improving Past Pre-Trained Model

Representation models pre-trained on unlabeled data show competitive performance in speech recognition, even when fine-tuned on small amounts of labeled data. The continual representation learning (CTRL) framework combines pre-training and continual learning methods to obtain powerful representation. CTRL relies on two neural networks, online and offline models, where the fixed latter model transfers information to the former model with continual learning loss. In this paper, we present momentum continual representation learning (M-CTRL), a framework that slowly updates the offline model with an exponential moving average of the online model. Our framework aims to capture information from the offline model improved on past and new domains. To evaluate our framework, we continually pre-train wav2vec 2.0 with M-CTRL in the following order: Librispeech, Wall Street Journal, and TED-LIUM V3. Our experiments demonstrate that M-CTRL improves the performance in the new domain and reduces information loss in the past domain compared to CTRL.
A Nested Ensemble Method to Bilevel Machine Learning

Modern machine learning problems, such as hyperparameter optimization, meta learning, and adversarial training, adopt a bilevel learning formulation. Such problems involve a nested relation between inner- and outer-level problems, which often have suboptimal solutions with poor generalization ability. To address this issue, this paper proposes an ensemble method tailored to bilevel learning. Our method finds a nested ensemble of inner and outer parameters that improve generalization. We instantiate our general results with meta learning. We show theoretically and empirically that the diversity and the smoother loss landscape of the proposed ensemble methods lead to improved generalization over the state-of-the-art method.
Streaming Stroke Classification of Online Handwriting

Stroke classification for online handwriting aims at providing each stroke with a semantic label so as to fulfill handwriting segmentation. This task has attracted considerable attention due to its significance in online handwriting analysis. Existing methods are designed for the static situation, where stroke classification is conducted on the completion of handwriting. With the popularity of pad devices and electronic whiteboards, streaming stroke classification becomes increasingly important for instant handwriting processing and feedback. However, streaming classification is much more challenging due to the lack of contextual information and is underexplored in the past. In this paper, we propose Multiple Stroke State Transformer (MSST), a novel framework to enable simultaneous real-time classification and modifiability of previous predictions. Particularly, we set multiple states with duration for each stroke and then divide all states into chunks to perform message passing by Transformer. Experiments on handwritten documents and diagrams demonstrate the superiority of our method.
CLIP4VideoCap: Rethinking Clip for Video Captioning with Multiscale Temporal Fusion and Commonsense Knowledge

In this paper, we propose CLIP4VideoCap for video captioning based on large-scale pre-trained CLIP image and text encoders together with multi-scale temporal reasoning and commonsense knowledge. In addition to the CLIP-image encoder operating on successive video frames, we introduce a knowledge distillation-based learning scheme that aims to exploit the CLIP-text encoder to generate rich textual knowledge from the image features. For improved temporal reasoning over the video, we propose a multi-scale temporal fusion scheme that accumulates temporal features from different temporal windows. In addition, we integrate various commonsense aspects in the caption generation which greatly enhances the caption quality by extracting the commonsense features from the video in the intermediate phase. Combining these strategies, we achieve state-of-the-art performance on the benchmark MSR-VTT dataset confirming that our framework significantly outperforms existing approaches.
Minimising Distortion for GAN-Based Facial Attribute Manipulation

Facial Attribute Manipulation (FAM) through GAN-based methods has been an active topic in computer graphics. Existing works show high editing fidelity on randomly generated faces but suffer from distortion on embedded real faces. We alleviate this issue by dividing it into two sub-problems. First, we minimize embedding distortion by introducing a pre-trained Salient Object Detection (SOD) network. Second, we propose a nonlinear transformation network to minimize editing distortion. As a result, our framework, Character Centered Facial Attribute Manipulation (CCFAM), exhibits more disentangled edits on real faces. Moreover, CCFAM is computationally efficient by integrating image complexity into our embedding process. Evaluations demonstrate that our method performs better than the state-of-the-art in terms of both accuracy and fidelity.
Precognition in Contextual Spoken Language Understanding via Knowledge Distillation

Task-oriented dialogue systems have become overwhelmingly popular in recent researches. Spoken Language Understanding (SLU) is widely used to extract the semantics frame of user queries and comprehend users’ intent/emotion/dialogue state in task-oriented dialogue systems. Most previous works on such discriminative tasks only model current query or historical conversations. Even if the entire conversation flow is modeled in some work, it is not suitable for real-world task-oriented dialogue systems, where future contexts are not visible until a response is given based on the current dialogue state. In this paper, we propose to jointly model historical and future information using knowledge distillation methods to address the discrepancy between offline and online information in dialogue understanding. More specifically, by modeling the current utterance and past contexts as student, and the entire dialogue flow as teacher, we optimize the "distance" between these two networks to regularize our student model in both online and offline ways. Our experiments on two dialogue datasets validate the effectiveness of our proposed method, achieving superior results compared with all baseline models.
Do Coarser Units Benefit Cluster Prediction-Based Speech Pre-Training?

The research community has produced many successful self-supervised speech representation learning methods over the past few years. Discrete units have been utilized in various self-supervised learning frameworks, such as VQ-VAE [1], wav2vec 2.0 [2], Hu-BERT [3], and Wav2Seq [4]. This paper studies the impact of altering the granularity and improving the quality of these discrete acoustic units for pre-training encoder-only and encoder-decoder models. We systematically study the current proposals of using Byte-Pair Encoding (BPE) and new extensions that use cluster smoothing and Brown clustering. The quality of learned units is studied intrinsically using zero speech metrics and on the down-stream speech recognition (ASR) task. Our results suggest that longer-range units are helpful for encoder-decoder pre-training; however, encoder-only masked-prediction models cannot yet benefit from self-supervised word-like targets.
High-Acoustic Fidelity Text To Speech Synthesis With Fine-Grained Control Of Speech Attributes

Recently developed neural-based TTS models have focused on robustness and finer control over acoustic features such as phoneme duration, energy, and F <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</inf> , allowing users to have some degree of control over the prosody of the generated speech. We propose a model with fine grained attribute control, which also has better acoustic fidelity (attributes of the output which we want to control do not deviate from the control signals) than previously proposed models as shown in our experiments <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> . Unlike other models, our proposed model does not require fine-tuning the vocoder on its outputs, indicating that it generates higher quality mel-spectrograms that are closer to the ground-truth distribution than that of other models.
Hierarchical Spatiotemporal Feature Fusion Network For Video Saliency Prediction

Current video saliency prediction methods have made great progress relying on the feature extraction capability of CNN, but there are still many defects in hierarchical feature fusion, limiting the further improvement of accuracy. To address this issue, we propose a 3D convolutional Hierarchical Spatiotemporal Feature Fusion Network (HSFF-Net). Specifically, we propose a Bi-directional Temporal-Spatial Feature Pyramid (BiTSFP), the first application of bi-directional fusion architectures in this field, which adds the flow of shallow location information on the flow of deep semantic information. Then, different from addition and concatenation, we design a Hierarchical Adaptive Fusion (HAF) mechanism that can adaptively learn the fusion weights of adjacent features. Moreover, a Frame-wise Attention (FA) module is introduced to augment the temporal features to be fused. Our model is simple yet effective and can run in real-time. Experimental results on the three video saliency benchmarks demonstrate that the HSFF-Net outperforms existing state-of-the-art methods in accuracy.
Hearttoheart: The Arts of Infant Versus Adult-Directed Speech Classification

Psycholinguistics researchers investigate child language exposure by studying children’s language environment. A main factor is whether, in humanistic heart-to-heart dialogue, the speech is directed to the infant (infant-directed speech) versus to another adult (adult-directed speech). The former has been found to better predict children’s lexicon, and therefore constitutes a more relevant part of children’s language environment. Listening to, segmenting and annotating naturalistic long-form recordings collected through infant-worn devices is highly costly and time-consuming, and could be prone to errors in misclassification. We aim to overcome these challenges by automatically classifying speech as infant-directed versus adult-directed. In this research, we exploit multiple datasets, combined to form a larger corpus for training. In addition, we employ four different methods: Multi-task learning, adversarial training, autoencoder multi-task learning and adversarial multi-task learning, the last of which yielded the best results on all datasets.
Sequence-Based Device-Free Gesture Recognition Framework for Multi-Channel Acoustic Signals

Device-free gesture recognition schemes based on acoustic sensing signals are promising solutions for next-generation human-computer interaction systems. However, existing gesture recognition frameworks reuse visual neural networks to perform feature extraction. These approaches ignore the time sequence nature of the acoustic signal and treat acoustic echo profiles solely as 2D images. In this paper, we propose a time-sequence-based deep learning framework that can exploit the spatio-temporal information of sensing signals. The framework first fuses multi-channel acoustic signals to extract spatial gesture features from a single acoustic frame and then uses the Transformer network to discover the timing relations between spatial features. Our extensive evaluations with real-world datasets show that our light-weighted framework outperforms the state-of-the-art in classifying 14 gestures and achieves an average accuracy of 95.85%.
HITSZ TMG at ICASSP 2023 SPGC Shared Task: Leveraging Pre-Training and Distillation Method for Title Generation with Limited Resource

In this paper, we present our proposed method for the shared task of the ICASSP 2023 Signal Processing Grand Challenge (SPGC). We participate in Topic Title Generation (TTG), Track 3 of General Meeting Understanding and Generation (MUG) [1] in SPGC. The primary objective of this task is to generate a title that effectively summarizes the given topic segment. With the constraints of limited model size and external dataset availability, we propose a method as Pre-training - Distillation / Fine-tuning (PDF), which can efficiently leverage the knowledge from large model and corpus. Our method achieves first place during preliminary and final contests in ICASSP2023 MUG Challenge Track 3.
StreamSpeech: Low-Latency Neural Architecture for High-Quality on-Device Speech Synthesis

Neural text-to-speech (TTS) systems have recently demonstrated the ability to synthesize high-quality natural speech. However, the inference latency and real-time factor (RTF) of such systems are still too high for deployment on devices without specialized hardware. In this paper, we describe StreamSpeech – an optimized architecture of a complete TTS system that produces high-quality speech and runs faster than real time with imperceptible latency on resource-constrained devices by utilizing a single CPU core. We divide the standard TTS processing pipeline into three phases with respect to their operating resolution and optimize them separately. Our main novel contribution is the introduction of a lightweight convolutional acoustic model decoder, which enables streaming and low-latency speech generation. Experiments show that the resulting complete TTS system achieves 79 ms latency, 0.155 RTF on a low-power notebook x86 CPU and 276 ms latency, 0.289 RTF on a mid-range mobile ARM CPU with no noticeable difference in the quality of the generated speech.
Optimized Dithering for Quantization Index Modulation

Quantization index modulation (QIM) is a popular data hiding paradigm due to its considerable performance advantages over spread spectrum techniques. Nevertheless, the random dithering procedure in QIM suffers from the synchronization and distortion issues. From the perspective of lattices, this work shows that using fixed optimized dithering is beneficial for achieving a smaller amount of distortion to the cover object. This can be translated into better Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) for the multimedia.
Cross-Domain Object Classification Via Successive Subspace Alignment

Recently, successive subspace learning (SSL)-based methods have shown to be effective for the task of visual object classification with mild data desire and mathematically transparent interpretable capability. However, existing SSL-based methods rely heavily on the data-centric subspace representations, leading to potential performance degradation problem in case of the domain shift between the training (a.k.a., source domain) and testing (a.k.a., target domain) data. To address this limitation, we propose an effective successive subspace learning method based on existing SSL-based methods. Specifically, we introduce a novel linear transformation layer to align eigenvectors in SSL module between source and target domains, as such, the discrepancy between source and target domains will be reduced, resulting in better cross-domain performance. The effectiveness of our proposed method is demonstrated on the Office-Caltech-10 and Office-31 benchmark datasets by using features extracted from pre-trained deep neural networks as input.
An Interpretable Model Using Evidence Information for Multi-Hop Question Answering Over Long Texts

Machine Reading Comprehension (MRC) is a challenging task in natural language understanding, especially multi-hop question answering (QA) in long texts. One of the challenges in multi-hop QA requires models to produce interpretable answers based on evidence that is selected from a given long text. Based on the Retriever-Reader architecture, existing work tackles this problem by using different methods to exploit various evidence information. To better use evidence information, we propose a loss function considering answer groups, which improves the reasoning ability of the reader in the Retriever-Reader architecture. Besides, we introduce the relevance constraint factor containing evidence information to improve the reader’s ability of locating key sentences. Evaluated on the HotpotQA dataset, the proposed methods achieve improvement, demonstrating the effectiveness of our methods and the importance of evidence information.
Facial Texure Perceiver: Towards High-Fidelity Facial Texture Recovery with Input-Level Inductive Biased Perceiver IO

This paper presents a new method, called Facial Texture Perceiver. It deals with the task of facial texture recovery from in-the-wild images without 3D supervision. Motivated by their success in various computer vision tasks, we attempt to use transformers for this task. However, capturing high-fidelity facial details requires a large number of mesh vertices and in this case, naively applying vanilla transformer can incur prohibitively high computational and memory costs. We address this challenge by mapping the input with a large number of mesh vertices to a latent space and performing their attention on this space. Also, we introduce input-level inductive biases by injecting the geometry and appearance embeddings as extra inputs. It helps to data-efficiently learn and generalize in-the-wild domains. The resulting architecture enable the application of Transformers to high-resolution facial meshes. Experiments on CelebA, MICC-Florence and MoFA-test datasets demonstrate that our method can accurately reconstruct facial textures, outperforming state-of-the-art methods.
CM-CS: Cross-Modal Common-Specific Feature Learning For Audio-Visual Video Parsing

The weakly-supervised audio-visual video parsing (AVVP) task aims to parse duration and categories of each snippet when only the video-level event labels are provided. Most methods either leverage attention mechanisms to explore cross-modal and cross-video event semantics or alleviate label noise to improve performance. However, the distributional modality discrepancy caused by the heterogeneity of signals remains a significant challenge. To this end, we propose a novel cross-modal common-specific feature learning method (cm-CS) to map the modal features into modality-common and modality-specific subspaces. The former aims to capture similar high-level scene cue across different modalities, while the later attempts to capture specific cue. The proposed method is applied among and across in-visual 2D-3D modalities, audio-visual modalities, respectively. In addition, we design a training strategy to strengthen the learning of similarity and differences across modalities. Experiments show a large improvement of our method against existing works on the Look, Listen, and Parse (LLP) dataset (e.g. from 58.9% to 62.9% in video-level visual metric).