Reviewerly/exHarmony huggingface dataset:
WL-MSR: Watch and Listen for Multimodal Subtitle Recognition

Video subtitles could be defined as the combination of visualized subtitles in frames and textual content recognized from speech, which play a significant role in video understanding for both humans and machines. In this paper, we propose a novel Watch and Listen for Multimodal Subtitle Recognition (WL-MSR) framework to obtain comprehensive video subtitles, by fusing the information provided by Optical Character Recognition (OCR) and Automatic Speech Recognition (ASR) models. Specifically, we build a Transformer model with mask and crop strategies and multi-level identity embeddings to aggregate both the textual results and features of the two modalities. To pre-filter out the noise items in OCR results before fusion, we adopt an OCR filter based on ASR results and confidence scores of OCR. By combining these techniques, our solution wins the 2nd place in Multimodal Subtitle Recognition Challenge on ICPR2022.
Gator: Graph-Aware Transformer with Motion-Disentangled Regression for Human Mesh Recovery from a 2D Pose

3D human mesh recovery from a 2D pose plays an important role in various applications. However, it is hard for existing methods to simultaneously capture the multiple relations during the evolution from skeleton to mesh, including joint-joint, joint-vertex and vertex-vertex relations, which often leads to implausible results. To address this issue, we propose a novel solution, called GATOR, that contains an encoder of Graph-Aware Transformer (GAT) and a decoder with Motion-Disentangled Regression (MDR) to explore these multiple relations. Specifically, GAT combines a GCN and a graph-aware self-attention in parallel to capture physical and hidden joint-joint relations. Furthermore, MDR models joint-vertex and vertex-vertex interactions to explore joint and vertex relations. Based on the clustering characteristics of vertex offset fields, MDR regresses the vertices by composing the predicted base motions. Extensive experiments show that GATOR achieves state-of-the-art performance on two challenging benchmarks. Code is available at https://github.com/kasvii/GATOR.
Self-Supervised Accent Learning for Under-Resourced Accents Using Native Language Data

In this paper, we propose a novel method to improve the accuracy of an English speech recognizer for a target accent using the corresponding native language data. Collecting labeled data for all accents of English to train an end-to-end neural speech recognizer for English is a difficult and expensive task. Also, finding a pool of representative English speakers for any arbitrary accent to collect unlabeled data can be a difficult task. However, collecting unlabeled speech data for any native language is a much simpler task. It is important to note that the accents of most non-native English speakers are heavily biased by the co-articulation of sounds in their own native language. In view of this, we propose to use unlabeled native language data to learn self-supervised representations during the pre-training stage. The pre-trained model is then fine-tuned using limited labeled English data for the target accent. Experiments using native language data to pre-train an English recognizer followed by fine-tuning using target accented English show significant improvements in word error rates on four different accents (Great Britain, Korean, Chinese, Spanish).
LED: Label Correlation Enhanced Decoder for Multi-Label Text Classification

Multi-label text classification, which aims to predict the relevant labels for each given document, is one of the fundamental tasks of natural language processing. Recent studies have utilized Transformer, which embeds texts and class labels into a joint space to capture the label correlation. However, existing methods tend to take up extra input length and ignore the significance of taxonomic hierarchy. For this reason, we introduce a label correlation enhanced decoder (LED) for multi-label text classification. LED predicts the presence or absence of class labels in parallel with label representation and captures label correlation through multi-task learning. In addition, we propose a hierarchy-aware mask to capture the hierarchical dependency between labels. Comprehensive experiments on four benchmark datasets show that LED outperforms the state-of-the-art baselines. Detailed analysis validates the effectiveness of our proposed method.
Inter-Scale Sure-Let Denoise with Structured Deep Image Prior: Interpretable Self-Supervised Learning

This work proposes a novel image restoration technique inspired by the Ulyanov’s deep image prior (DIP) method. DIP uses a deep convolutional network as an image prior to generate a restored image from a random input one, which brings an advantage of no requirement of training data. However, one problem arises that the interpretability is low and it is not trivial to explain the need for the random input. This paper contributes to train a network by using the Stein’s unbiased risk estimator (SURE) with Monte-Carlo computation for self-supervised learning of image restoration. The Monte-Carlo calculation accepts a random input for computing divergence of the network, and allows us to interpret the reason why a random input is needed. Our framework trains an image restorer instead of an image generator, which simplifies the interpretation of the network structure. Thanks to this, the Luisier’s interscale linear expansion of the thresholding (LET) can become introduced to exploit the cross-correlation among extracted features. In order to avoid using group delay compensation and reduce the number of design parameters, a structured DIP is constructed by using non-separable oversampled lapped transform (NSOLT) instead of U-Net. By showing some results of denoising simulation, the significance of the proposed method is verified.
Flow-Guided Deformable Alignment Network with Self-Supervision for Video Inpainting

Video inpainting aims to utilize plausible contents to fill missing regions in the video. State-of-the-art video inpainting methods typically generate the missing contents of the target frame (current frame) by aggregating the temporal information of reference frames (neighboring frames) aligned using deformable convolution. However, these deformable convolution alignment networks often suffer from offset overflow during training, resulting in unsatisfactory alignment, thereby obtaining compromised inpainting performance. In this paper, we propose a self-supervised Flow-Guided Deformable Alignment (FGDA) network for aligning reference frames at the feature level. FGDA computes the residual of the optical flow as the offsets. This design can effectively reduce the burden of offsets learning, thereby avoiding offset overflow. Furthermore, a gradient-weighted reconstruction loss for supervised completed frame reconstruction is designed, which can use the gradients in all directions of the video frame to emphasize the difficultly reconstructed texture regions, so that detail textures get more attention during training. Experiments show that FGDA-based video inpainting model trained with gradient-weighted reconstruction loss outperforms the state-of-the-art by a significant margin in terms of PSNR and SSIM with relative improvements of 6.2% and 2.1%, respectively.
Bayesian Methods for Optical Flow Estimation Using a Variational Approximation, with Applications to Ultrasound

We develop a unified Bayesian framework for optical flow (OF) estimation that uses a variational lower bound to obtain a variational approximation of the posterior probability distribution. Our framework enables the incorporation of domain-specific knowledge as well as a quantification of the uncertainty of OF estimation, and it encompasses existing maximum a posteriori (MAP) and variational Bayes (VB) methods as special cases. We leverage this flexibility for the ultrasound modality by using ultrasound-specific likelihood functions within both MAP and VB methods. Numerical results for the problem of cardiac motion estimation demonstrate that VB methods outperform MAP methods, in addition to providing a more faithful uncertainty measure.
Exploration Into Translation-Equivariant Image Quantization

This is an exploratory study that discovers the current image quantization (vector quantization) do not satisfy translation equivariance in the quantized space due to aliasing. Instead of focusing on anti-aliasing, we propose a simple yet effective way to achieve translation-equivariant image quantization by enforcing orthogonality among the codebook embeddings. To explore the advantages of translation-equivariant image quantization, we conduct three proof-of-concept experiments with a carefully controlled dataset: (1) text-to-image generation, where the quantized image indices are the target to predict, (2) image-to-text generation, where the quantized image indices are given as a condition, (3) using a smaller training set to analyze sample efficiency. From the strictly controlled experiments, we empirically verify that the translation-equivariant image quantizer improves not only sample efficiency but also the accuracy over VQGAN up to +11.9% in text-to-image generation and +3.9% in image-to-text generation.
Compose &amp; Embellish: Well-Structured Piano Performance Generation via A Two-Stage Approach

Even with strong sequence models like Transformers, generating expressive piano performances with long-range musical structures remains challenging. Meanwhile, methods to compose well-structured melodies or lead sheets (melody + chords), i.e., simpler forms of music, gained more success. Observing the above, we devise a two-stage Transformer-based framework that Composes a lead sheet first, and then Embellishes it with accompaniment and expressive touches. Such a factorization also enables pretraining on non-piano data. Our objective and subjective experiments show that Compose & Embellish shrinks the gap in structureness between a current state of the art and real performances by half, and improves other musical aspects such as richness and coherence as well.
Ensemble of Deep Neural Network Models for MOS Prediction

Automatic evaluation of the quality of synthetic speech has the potential to serve as a cheaper and less time-consuming alternative to standard listening tests. In this paper, we present our contribution to the ongoing research: a system for automatic prediction of the mean opinion score (MOS) given by human listeners. The system was specifically developed for the recent VoiceMOS Challenge. Following the success of fusion systems in similar challenges, our contribution is an ensemble that interpolates the outputs of seven different models: four different wav2vec models, a CNN-RNN model, QuartzNet, and the LDNet baseline. During the VoiceMOS challenge, our system achieved the second-best utterance-level MSE of 0.171 and ranged from 2nd to 8th place among all 22 participating teams in terms of other evaluation metrics.
Using Emotion Embeddings to Transfer Knowledge between Emotions, Languages, and Annotation Formats

The need for emotional inference from text continues to diversify as more and more disciplines integrate emotions into their theories and applications. These needs include inferring different emotion types, handling multiple languages, and different annotation formats. A shared model between different configurations would enable the sharing of knowledge and a decrease in training costs, and would simplify the process of deploying emotion recognition models in novel environments. In this work, we study how we can build a single model that can transition between these different configurations by leveraging multilingual models and Demux, a transformer-based model whose input includes the emotions of interest, enabling us to dynamically change the emotions predicted by the model. Demux also produces emotion embeddings, and performing operations on them allows us to transition to clusters of emotions by pooling the embeddings of each cluster. We show that Demux can simultaneously transfer knowledge in a zero-shot manner to a new language, to a novel annotation format and to unseen emotions. Code is available at https://github.com/gchochla/Demux-MEmo. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
FFEDCL: Fair Federated Learning with Contrastive Learning

Federated Learning (FL) is a new paradigm of distributed machine learning, which can effectively solve the problem of data islands with privacy protection. Typical aggregation in FedAvg causes the global model bias to local models of clients with more local data. When clients’ data is Non-IID, the global model can not fully learn the data distribution of clients with less data, which incurs unfairness to these clients and then uninspired them to participate in FL. To this end, we propose a real-time fairness adjustment algorithm for the global model based on model-level contrastive learning, called FFedCL. We calculate our special-designed contrastive learning loss and use Stochastic Gradient Descent (SGD) to adjust the global model. It aims to improve the similarity of the global model to the model of clients with less data, thereby improving the fairness of FL. We evaluate FFedCL on three Non-IID datasets. The experimental results present that FFedCL improves up to 7% accuracy while maintaining almost the lowest variance compared with the state-of-the-art baseline, demonstrating its effectiveness on FL’s fairness enhancement.
Transwnet: Integrating Transformers into CNNS via Row and Column Attention for Abdominal Multi-Organ Segmentation

Learning how to model global relationships and extract local details is crucial in improving the performance of multi-organ segmentation. Most existing U-shaped structure methods use feature fusion to address these two challenges, but still lack the ability to balance capturing global relationships and local details. To address these issues, we propose a novel multi-organ segmentation framework called TransWnet to mine global relationships and local details from both intra- and inter-scale perspectives. To achieve this, we innovatively design a Row and Column Swin Transformer (RCST) module that can efficiently capture global contextual features and construct local information. Specifically, we design a parallel structure of Row and Column Attention to model the global relationships of multi-scale encoded features, and further mine local information from the global relationships through a local window mechanism. Extensive experiments on the Synapse dataset show that our method outperforms state-of-the-art approaches and achieves accurate segmentation of abdominal multi-organs.
Rigid-Body Sound Synthesis with Differentiable Modal Resonators

Physical models of rigid bodies are used for sound synthesis in applications from virtual environments to music production. Traditional methods, such as modal synthesis, often rely on computationally expensive numerical solvers, while recent deep learning approaches are limited by post-processing of their results. In this work, we present a novel end-to-end framework for training a deep neural network to generate modal resonators for a given 2D shape and material using a bank of differentiable IIR filters. We demonstrate our method on a dataset of synthetic objects but train our model using an audio-domain objective, paving the way for physically-informed synthesisers to be learned directly from recordings of real-world objects.
Speechlmscore: Evaluating Speech Generation Using Speech Language Model

While human evaluation is the most reliable metric for evaluating speech generation systems, it is generally costly and time-consuming. Previous studies on automatic speech quality assessment address the problem by predicting human evaluation scores with machine learning models. However, they rely on supervised learning and thus suffer from high annotation costs and domain-shift problems. We propose SpeechLMScore, an unsupervised metric to evaluate generated speech using a speech language model. SpeechLMScore computes the average log-probability of a speech signal by mapping it into discrete tokens and measures the average probability of generating the sequence of tokens. Therefore, it does not require human annotation and is a highly scalable framework. Evaluation results demonstrate that the proposed metric shows a promising correlation with human evaluation scores on different speech generation tasks including voice conversion, text-to-speech, and speech enhancement.
Improving Contextual Spelling Correction by External Acoustics Attention and Semantic Aware Data Augmentation

We previously proposed contextual spelling correction (CSC) to correct the output of end-to-end (E2E) automatic speech recognition (ASR) models with contextual information such as name, place, etc. Although CSC has achieved reasonable improvement in the biasing problem, there are still two drawbacks for further accuracy improvement. First, due to information limitation in text only hypothesis or weak performance of ASR model on rare domains, the CSC model may fail to correct phrases with similar pronunciation or anti-context cases where all biasing phrases are not present in the utterance. Second, there is a discrepancy between the training and inference of CSC. The bias list in training is randomly selected but in inference there may be more similarity between ground truth phrase and other phrases. To solve above limitations, in this paper we propose an improved non-autoregressive (NAR) spelling correction model for contextual biasing in E2E neural transducer-based ASR systems to improve the previous CSC model from two perspectives: Firstly, we incorporate acoustics information with an external attention as well as text hypotheses into CSC to better distinguish target phrase from dissimilar or irrelevant phrases. Secondly, we design a semantic aware data augmentation schema in training phrase to reduce the mismatch between training and inference to further boost the biasing accuracy. Experiments show that the improved method outperforms the baseline ASR+Biasing system by as much as 20.3% relative name recall gain and achieves stable improvement compared to the previous CSC method over different bias list name coverage ratio.
A Comparison of Semi-Supervised Learning Techniques for Streaming ASR at Scale

Unpaired text and audio injection have emerged as dominant methods for improving ASR performance in the absence of a large labeled corpus. However, little guidance exists on deploying these methods to improve production ASR systems that are trained on very large supervised corpora and with realistic requirements like a constrained model size and CPU budget, streaming capability, and a rich lattice for rescoring and for downstream NLU tasks. In this work, we compare three state-of-the-art semi-supervised methods encompassing both unpaired text and audio as well as several of their combinations in a controlled setting using joint training. We find that in our setting these methods offer many improvements beyond raw WER, including substantial gains in tail-word WER, decoder computation during inference, and lattice density.
MASKED-AP: Attention Pyramid Convolutional Neural Network with Mask for Cervical Cell Classification

The automatic and effective cervical cell classification technique is critical for cervical cytology screening and cervical cancer prevention. We notice that cervical cell classification is a fine-grained classification task. The difference between classes is small while the difference within a class is large, so it is difficult to capture the discriminative features between different classes of cells for classification. To address this problem, this paper proposes an attention pyramid model (Masked-AP) used for cervical cell classification. Our Masked-AP effectively combines high-level semantic features with low-level detailed features extracted from images, which are both important for classification. Further, we also use the attention mask to drive our model to focus more on the cell nuclei containing substantial discriminative information. For model evaluation, we built a cervical cell dataset named LDCC including 17476 images from 507 subjects. LDCC is available at http://ldcc3.biodwhu.cn/LDCC3/. Our method yielded 74.11% accuracy, 74.11% recall, 74.19% precision, and 74.07% F1-score, and outperforms the state-of-the-art methods.
Learning to Reconnect Interrupted Trajectories for Weakly Supervised Multi-Object Tracking

Recently, some weakly supervised multi-object tracking (MOT) methods learn identity embedding features with pseudo identity labels rather than the high-cost manual ones. However, these pseudo identity labels may contain many false or missing identities, which adversely affect the optimization of tracking networks, resulting in interrupted trajectories of occluded targets. To effectively reconnect the interrupted trajectories caused by noisy pseudo labels, we propose a novel weakly supervised MOT method based on a Trajectory-Reconnecting Transformer (TRTMOT). TRT-MOT performs feature decoupling to extract discriminative embedding features for reconnecting trajectories of occluded targets. Experimental results show that TRTMOT outperforms previous weakly supervised MOT methods by at least +3.6 and +5.6 on MOTA for the MOT17 and MOT20 datasets, respectively.
Joint Compression and Demosaicking For Satellite Images

Image sensors used in real camera systems are equipped with colour filter arrays which sample the light rays in different spectral bands. Each colour channel can thus be obtained sep-arately by considering the corresponding colour filter. While existing compression solutions mostly assume that the captured raw data has been demosaicked prior to compression, in this paper, we describe an end-to-end trainable neural network for joint compression and demosaicking of satellite images. We first introduce a training loss combining a perceptual loss with the classical mean square error, which is shown to better preserve the high-frequency details present in satellite images. We then present a multi-loss balancing strategy which significantly improves the performance of the proposed joint demosaicking-compression solution.
Evaluating Speech–Phoneme Alignment and its Impact on Neural Text-To-Speech Synthesis

In recent years, the quality of text-to-speech (TTS) synthesis vastly improved due to deep-learning techniques, with parallel architectures, in particular, providing excellent synthesis quality at fast inference. Training these models usually requires speech recordings, corresponding phoneme-level transcripts, and the temporal alignment of each phoneme to the utterances. Since manually creating such fine-grained alignments requires expert knowledge and is time-consuming, it is common practice to estimate them using automatic speech–phoneme alignment methods. In the literature, either the estimation methods’ accuracy or their impact on the TTS system’s synthesis quality is evaluated. In this study, we perform experiments with five state-of-the-art speech–phoneme aligners and evaluate their output with objective and subjective measures. As our main result, we show that small alignment errors (below 75 ms error) do not decrease the synthesis quality, which implies that the alignment error may not be the crucial factor when choosing an aligner for TTS training.