Reviewerly/exHarmony huggingface dataset:
Sine: Similarity-Regularized Intra-Class Exploitation for Cross-Granularity Few-Shot Learning

Few-shot learning aims for rapid adaptation with few samples. Recently, cross-granularity few-shot learning has emerged as a promising research area, where models observe coarse labels but target fine-grained recognition among novel classes. As coarse supervision tends to eliminate feature discrimination among underlying sub-classes, existing methods commonly utilize self-supervision as a complement to explore intra-class variation. However, current methods suffer from an intrinsic conflict between contrastive learning and coarse supervision. In this paper, we locate the root cause of the intrinsic conflict. Then, we resolve it by exploiting the similarity among augmented views while ignoring the unreasonable constraint between negative pairs. Besides, we decouple contrastive learning and coarse supervision into parallel branches to better regularize the latent space. Albeit simple, our approach consistently outperforms state-of-the-art methods across different benchmarks.
Estimating Uncertainty On Video Quality Metrics

Video Quality Metrics (VQM) are models used to predict the score that a user would give to the quality of a video visualization. They are widely used in video processing systems, for monitoring end-to-end quality or system troubleshooting for example. In these scenarios, the improvement is quantified based on a certain enhancement of a VQM score and trouble-detection is done based on a certain drop or threshold computed based on a VQM. Yet, whether such improvement or fault-detection is worth a significant increase in power consumption is questionable. Therefore, the goal of this work is to propose a method to predict the uncertainty of the quality metric. In this paper, we propose a framework to evaluate the confidence interval of a VQM for a given content using simple video features. We assess the performance of the framework by using the confidence intervals to predict if two videos are of similar or different quality and show that in most cases our approach performs better than just using a constant confidence interval.
High‐performance graphics processing unit‐based strategy for tuning a unmanned aerial vehicle controller subject to time‐delay constraints

Abstract Recently, high‐performance computing strategies have been implemented to improve performance analysis and reduce the development time of new solutions in robotic applications, such as path planning, machine learning, and vision, which require massive matrix computations. In this sense, this work aims to study the aerial robots' behavior during their mission execution. Due to the large search space in the set of parameter combinations and the high computational cost required to perform such an analysis after sequentially executing thousands of simulations, this work proposes an open‐source graphics processing unit (GPU)‐based implementation to simulate the robot behavior. A GPU‐accelerated flight route analysis for multi‐unmanned aerial vehicle (UAV) systems is proposed for the tuning control problem in the parameters‘ space considering the problem of delay in sending information to a ground control station. Considering our implementation, the experimental results show a speedup up to 325, 629, and 5959 in comparison to the parallel version with 16 threads, C coder converter, and native Matlab code, respectively. The implementation is available in the Colab Google platform and it can easily be expanded for analyses involving larger amounts of different parameters, robot models, strategies, and controllers.
Skillnet-NLG: General-Purpose Natural Language Generation with a Sparsely Activated Approach

We present SkillNet-NLG, a sparsely activated approach that handles many natural language generation tasks with one model. Different from traditional dense models that always activate all the parameters, SkillNet-NLG selectively activates relevant parts of the parameters to accomplish a task, where the relevance is controlled by a set of predefined skills. The strength of such model design is that it provides an opportunity to precisely adapt relevant skills to learn new tasks effectively. We evaluate on Chinese natural language generation tasks. Results show that, with only one model file, SkillNet-NLG outperforms previous best performance methods on four of five tasks. SkillNet-NLG performs better than two multitask learning baselines (a dense model and a Mixture-of-Expert model) and achieves comparable performance to task-specific models. Lastly, SkillNet-NLG surpasses baseline systems when adapted to new tasks.
Federated Self-Learning with Weak Supervision for Speech Recognition

Automatic speech recognition (ASR) models with low-footprint are increasingly being deployed on edge devices for conversational agents, which enhances privacy. We study the problem of federated continual incremental learning for recurrent neural network-transducer (RNN-T) ASR models in the privacy-enhancing scheme of learning on-device, without access to ground truth human transcripts or machine transcriptions from a stronger ASR model. In particular, we study the performance of a self-learning based scheme, with a paired teacher model updated through an exponential moving average of ASR models. Further, we propose using possibly noisy weak-supervision signals such as feedback scores and natural language understanding semantics determined from user behavior across multiple turns in a session of interactions with the conversational agent. These signals are leveraged in a multitask policy-gradient training approach to improve the performance of self-learning for ASR. Finally, we show how catastrophic forgetting can be mitigated by combining on-device learning with a memory-replay approach using selected historical datasets. These innovations allow for 10% relative improvement in WER on new use cases with minimal degradation on other test sets in the absence of strong-supervision signals such as ground-truth transcriptions.
Surrogate Based Post-HOC Calibration for Distributional Shift

This paper focuses on improving the calibration performance of the post-hoc approach for the distributional shift. Taking the popular temperature scaling (TS) as a case in point, the key task is finding a matched temperature for the shifted test set. To address this issue, we pose an insight that temperature is strongly correlated with the shifting intensity by a tiny experiment. Based on the finding, we propose a simple yet effective approach named Surrogate Based Temperature Scaling (SBTS), where the surrogate model is trained to map the relationship between temperature and the shifting intensity. Empirical experimental results of various shift types on the CIFAR-10 and CIFAR-100 demonstrate that SBTS can significantly improve the calibration performance under distributional shift.
Raising The Limit of Image Rescaling Using Auxiliary Encoding

Normalizing flow models using invertible neural networks (INN) have been widely investigated for successful generative image super-resolution (SR) by learning the transformation between the normal distribution of latent variable z and the conditional distribution of high-resolution (HR) images gave a low-resolution (LR) input. Recently, image rescaling models like IRN utilize the bidirectional nature of INN to push the performance limit of image upscaling by optimizing the downscaling and upscaling steps jointly. While the random sampling of latent variable z is useful in generating diverse photo-realistic images, it is not desirable for image rescaling when accurate restoration of the HR image is more important. Hence, in places of random sampling of z, we propose auxiliary encoding modules to further push the limit of image rescaling performance. Two options to store the encoded latent variables in downscaled LR images, both readily supported in existing image file format, are proposed. One is saved as the alpha-channel, the other is saved as meta-data in the image header, and the corresponding modules are denoted as suffixes -A and -M respectively. Optimal network architectural changes are investigated for both options to demonstrate their effectiveness in raising the rescaling performance limit on different baseline models including IRN and DLV-IRN.
MTFD: Multi-Teacher Fusion Distillation for Compressed Video Action Recognition

As an important work in computer vision, some recent representative works such as Two-stream networks, 3D ConvNets, and Transformer-based networks have achieved outstanding performance. However, due to the high computational cost, the explosion of computation time and parameters, they cannot meet the needs of real-time applications. The current work utilizes the keyframes, residual and motion information retained by compressed video for computation, which greatly reduces the computational effort but still cannot satisfy real-time applications. Therefore, we propose a multi-teacher fusion distillation framework for compressed video action recognition (MTFD). Unlike the traditional method of transferring the knowledge of single or multiple teachers directly into the student model, we also perform knowledge transfer between teachers. MTFD achieves better knowledge distillation through mutual guidance and information fusion between teachers. Furthermore, we improve the network’s ability to extract motion information, which ultimately reduces the computational effort while maintaining high accuracy.
Joint Unsupervised and Supervised Learning for Context-Aware Language Identification

Language identification (LID) recognizes the language of a spoken utterance automatically. According to recent studies, LID models trained with an automatic speech recognition (ASR) task perform better than those trained with a LID task only. However, we need additional text labels to train the model to recognize speech, and acquiring the text labels is a cost high. In order to overcome this problem, we propose context-aware language identification using a combination of unsupervised and supervised learning without any text labels. The proposed method learns the context of speech through masked language modeling (MLM) loss and simultaneously trains to determine the language of the utterance with supervised learning loss. The proposed joint learning was found to reduce the error rate by 15.6% compared to the same structure model trained by supervised-only learning on a subset of the VoxLingua107 dataset consisting of sub-three-second utterances in 11 languages.
Signal Processing And Quantum State Tomography on Noisy Devices

Quantum State Tomography (QST) is a fundamental tool for quantum signal processing. However, in real noisy quantum devices construction of the state’s density matrix via QST can utilize a large amount of resources. Here, we discuss some signal processing techniques that are currently applied to this resource issue, and implement on current quantum chips a modification that can assist in reducing resources. An application of QST to quantum entanglement distillation is provided for further insight.
Domain Adaptation with External Off-Policy Acoustic Catalogs for Scalable Contextual End-to-End Automated Speech Recognition

Despite improvements to the generalization performance of automated speech recognition (ASR) models, specializing ASR models for downstream tasks remains a challenging task, primarily due to reduced data availability (necessitating increased data collection), and rapidly shifting data distributions (requiring more frequent model fine-tuning). In this work, we investigate the potential of leveraging external knowledge, particularly through off-policy generated text-to-speech key-value stores, to allow for flexible post-training adaptation to new data distributions. In our approach, audio embeddings captured from text-to-speech are used, along with semantic text embeddings, to bias ASR via an approximate k-nearest-neighbor (KNN) based attentive fusion step. Our experiments on LibiriSpeech and in-house voice assistant/search datasets show that the proposed approach can reduce domain adaptation time by up to 1K GPU-hours while providing up to 3% WER improvement compared to a fine-tuning baseline, suggesting a promising approach for adapting production ASR systems in challenging zero and few-shot scenarios.
DPP-Based Client Selection for Federated Learning with NON-IID DATA

This paper proposes a client selection (CS) method to tackle the communication bottleneck of federated learning (FL) while concurrently coping with FL’s data heterogeneity issue. Specifically, we first analyze the effect of CS in FL and show that FL training can be accelerated by adequately choosing participants to diversify the training dataset in each round of training. Based on this, we lever-age data profiling and determinantal point process (DPP) sampling techniques to develop an algorithm termed Federated Learning with DPP-based Participant Selection (FL-DP <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> S). This algorithm effectively diversifies the participants’ datasets in each round of training while preserving their data privacy. We conduct extensive experiments to examine the efficacy of our proposed method. The results show that our scheme attains a faster convergence rate, as well as a smaller communication overhead than several baselines.
Leveraging Label Correlations in a Multi-Label Setting: a Case Study in Emotion

Detecting emotions expressed in text has become critical to a range of fields. In this work, we investigate ways to exploit label correlations in multi-label emotion recognition models to improve emotion detection. First, we develop two modeling approaches to the problem in order to capture word associations of the emotion words themselves, by either including the emotions in the input, or by leveraging Masked Language Modeling (MLM). Second, we inte-grate pairwise constraints of emotion representations as regularization terms alongside the classification loss of the models. We split these terms into two categories, local and global. The former dynamically change based on the gold labels, while the latter remain static during training. We demonstrate state-of-the-art performance across Spanish, English, and Arabic in SemEval 2018 Task 1 E-c using monolingual BERT-based models. On top of better performance, we also demonstrate improved robustness. Code is available at https://github.com/gchochla/Demux-MEmo. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
SafeDeep: A Scalable Robustness Verification Framework for Deep Neural Networks

The state-of-the-art machine learning techniques come with limited, if at all any, formal correctness guarantees. This has been demonstrated by adversarial examples in the deep learning domain. To address this challenge, here, we propose a scalable robustness verification framework for Deep Neural Networks (DNNs). The framework relies on Linear Programming (LP) engines and builds on decades of advances in the field for analyzing convex approximations of the original network. The key insight is in the on-demand incremental refinement of these convex approximations. This refinement can be parallelized, making the framework even more scalable. We have implemented a prototype tool to verify the robustness of a large number of DNNs in epileptic seizure detection. We have compared the results with those obtained by two state-of-the-art tools for the verification of DNNs. We show that our framework is consistently more precise than the over-approximation-based tool ERAN and more scalable than the SMT-based tool Reluplex.
Improving Non-Autoregressive Speech Recognition with Autoregressive Pretraining

Autoregressive (AR) automatic speech recognition (ASR) models predict each output token conditioning on the previous ones, which slows down their inference speed. On the other hand, non-autoregressive (NAR) models predict tokens independently and simultaneously within a constant number of decoding iterations, which brings high inference speed. However, NAR models generally have lower accuracy than AR models. In this work, we propose AR pretraining to the NAR encoder to reduce the accuracy gap between AR and NAR models. The experiment results show that our AR-pretrained MaskCTC reaches the same accuracy as AR Conformer on Aishell-1 (both 4.9% CER) and reduce the performance gap with AR Conformer on LibriSpeech by relatively 50%. Moreover, our AR-pretrained MaskCTC only needs single decoding iteration, which reduces inference time by 50%. We also investigate multiple masking strategies in training the masked language model of MaskCTC.
Bridging Speech and Textual Pre-Trained Models With Unsupervised ASR

Spoken language understanding (SLU) is a task aiming to extract high-level semantics from spoken utterances. Previous works have investigated the use of speech self-supervised models and textual pre-trained models, which have shown reasonable improvements to various SLU tasks. However, because of the mismatched modalities between speech signals and text tokens, previous methods usually need complex designs of the frameworks. This work proposes a simple yet efficient unsupervised paradigm that connects speech and textual pre-trained models, resulting in an unsupervised speech-to-semantic pre-trained model for various tasks in SLU. To be specific, we propose to use unsupervised automatic speech recognition (ASR) as a connector that bridges different modalities used in speech and textual pre-trained models. Our experiments show that unsupervised ASR itself can improve the representations from speech self-supervised models. More importantly, it is shown as an efficient connector between speech and textual pre-trained models, improving the performances of five different SLU tasks. Notably, on spoken question answering, we reach the state-of-the-art result over the challenging NMSQA benchmark.
DVQVC: An Unsupervised Zero-Shot Voice Conversion Framework

Zero-shot voice conversion (VC) is to convert speech from one speaker to a target speaker while preserving the original linguistic information, given only one reference speech clip of the unseen target speaker. This work proposes a new VC model, and its key idea is to conduct thorough speaker and content disentanglement by adopting an advanced speech encoder plus vector quantization (VQ) as a content encoder, and an advanced speaker encoder for accurate speaker embedding. In addition, we propose a perceptual loss, a speaker constrative loss and an adversarial loss to compensate the content imperfection caused by VQ and to further improve the speech quality/intelligibility. Overall, the proposed model uses only unsupervised features/losses, and achieves excellent VC performance in terms of both speech quality/intelligibility and speaker similarity, for both seen and unseen speakers.
SMCL: Saliency Masked Contrastive Learning for Long-Tailed Visual Recognition

Real-world data often follow a long-tailed distribution with a high imbalance in the number of samples between classes. The problem with training from imbalanced data is that some background features, common to all classes, can be unobserved in classes with scarce samples. As a result, this background correlates to biased predictions into "major" classes. In this paper, we propose saliency masked contrastive learning, a new method that uses saliency masking and contrastive learning to mitigate the problem and improve the generalizability of a model. Our key idea is to mask the important part of an image using saliency detection and use contrastive learning to move the masked image towards minor classes in the feature space, so that background features present in the masked image are no longer correlated with the original class. Experiment results show that our method achieves state-of-the-art level performance on benchmark long-tailed datasets.
Unified Prompt Learning Makes Pre-Trained Language Models Better Few-Shot Learners

Language prompting induces the model to produce a textual output during the training phase, which achieves remarkable performance in few-shot learning scenarios. However, current prompt-based methods either use the same task-specific prompts for each instance, losing the particularity of instance-dependent information, or generate an instance-dependent prompt for each instance, lacking shared information about the task. In this paper, we propose an efficient few-shot learning method to dynamically decide the degree to which task-specific and instance-dependent information are incorporated according to different task and instance characteristics, enriching the prompt with task-specific and instance-dependent information. Extensive experiments on a wide range of natural language understanding tasks demonstrate that our approach obtains significant improvements compared to prompt-based fine-tuning baselines in a few-shot setting with about 0.1% parameters tuned. Moreover, our approach outperforms existing state-of-the-art efficient few-shot learning methods on several natural language understanding tasks.
CNN Filter for Super-Resolution with RPR Functionality in VVC

High resolution (HR) videos are easy to go beyond the band-width limit during transmission after encoding. Downsampling followed by upsampling is a well-known strategy for compressing HR videos with limited bandwidth. Versatile video coding (VVC) provides a reference picture resampling (RPR) functionality to consider the limited bandwidth. In this paper, we propose a convolutional neural network (CNN) filter for super-resolution (SR) with the RPR functionality in VVC. We design a lightweight SR network that combines CNN with the RPR functionality in VVC, called lightweight network of multi-level mixed scale and depth information with attention mechanism (LMSDANet). For LMSDANet, we provide a lightweight block of multi-mixed scale and depth information with attention (LMSDAB) to extract multi-scale and convolutional layer depth information while enhancing the representation of features. Compared with VTM-11.0_NNVC-2.0 anchor, LMSDANet achieves {-9.16% (Y), 17.03% (U), -7.61% (V)} and {-4.14% (Y), 6.34% (U), -2.25% (V)} BD-rate changes (average on A1 and A2) in AI and RA configurations, respectively.
Source-Free Unsupervised Domain Adaptation for Question Answering

Based on the assumption that samples in the source and target domains are freely accessible during training, unsupervised domain adaptation (UDA) of question answering (QA) aims to transfer knowledge learned from labeled source datasets to similar tasks in the unlabeled target domains. However, such assumption can easily lead to privacy violation issues in real-world applications, especially when the source domain data involves privacy-intensive domains such as finance and healthcare. In this paper, we introduce Source-Free Domain Adaptation Framework for QA (denoted as SFQA), which only allows access to trained source models for target learning, making data privacy protection more promising. Specifically, the proposed SFQA model consists of a feature extractor module (Bert Encoder) and a classifier module (Answer Classifier). We first transfer the trained source model to the target model while keeping the source classifier module frozen. Then we adopt the question generation model to generate questions and answers for the target domain. Taking the generated questions and target domain context as inputs, and the generated answers as pseudo-labels, we train the target model with joint entropy to learn a target domain-specific feature extractor. Experimental results demonstrate the superiority and effectiveness of the proposed SFQA, and show that SFQA outperforms the state-of-the-art methods.