Reviewerly/exHarmony huggingface dataset:
Internal Language Model Personalization of E2E Automatic Speech Recognition Using Random Encoder Features

End-to-end (E2E) speech-to-text models generally require transcribed audio for training and personalization. We introduce the use of random audio encoder features, rather than speech, to fine-tune the final model layers and acquire new vocabulary from text-only data. This technique can be used for on-device personalization before the user has provided any speech data. We show improvements in the recall of new vocabulary and word error rate (WER) on held-out test sets using simulated user experiments on hybrid autoregressive transducer (HAT) models using conformer-based encoders and simple text embeddings for label processing. We compare this approach to the use of synthetic audio, finding random encoder features to be more beneficial with lower computational cost. Experiments show that the maximum benefit is gained by updating specific network components comprising a subset of those expressing the internal language model.
Residual Adapters for Targeted Updates in RNN-Transducer Based Speech Recognition System

This paper investigates an approach for adapting RNN-Transducer (RNN-T) based automatic speech recognition (ASR) model to improve the recognition of unseen words during training. Prior works have shown that it is possible to incrementally finetune the ASR model to recognize multiple sets of new words. However, this creates a dependency between the updates which is not ideal for the hot-fixing use-case where we want each update to be applied independently of other updates. We propose to train residual adapters on the RNN-T model and combine them on-the-fly through adapter-fusion. We investigate several approaches to combine the adapters so that they maintain the ability to recognize new words with only a minimal degradation on the usual user requests. Specifically, the sum-fusion which sums the outputs of the adapters inserted in parallel shows over 90% recall on the new words with less than 1% relative WER degradation on the usual data compared to the original RNN-T model.
GAN You Hear Me? Reclaiming Unconditional Speech Synthesis from Diffusion Models

We propose AudioStyleGAN (ASGAN), a new generative adversarial network (GAN) for unconditional speech synthesis. As in the StyleGAN family of image synthesis models, ASGAN maps sampled noise to a disentangled latent vector which is then mapped to a sequence of audio features so that signal aliasing is suppressed at every layer. To successfully train ASGAN, we introduce a number of new techniques, including a modification to adaptive discriminator augmentation to probabilistically skip discriminator updates. ASGAN achieves state-of-the-art results in unconditional speech synthesis on the Google Speech Commands dataset. It is also substantially faster than the top-performing diffusion models. Through a design that encourages disentanglement, ASGAN is able to perform voice conversion and speech editing without being explicitly trained to do so. ASGAN demonstrates that GANs are still highly competitive with diffusion models. Code, models, samples: https://github.com/RF5/simple-asgan/.
Efficient Dynamic Filter For Robust and Low Computational Feature Extraction

The unseen noise signal is difficult to anticipate, and various approaches have been developed to address this issue. In our earlier work, we proposed a lightweight dynamic filter by splitting the filter into kernel and spatial parts. This small footprint model showed robust results in an unseen noisy environment. However, a simple pooling process for dividing the feature would limit the performance. In this paper, we propose an efficient dynamic filter to enhance the performance of the existing dynamic filter. Instead of the simple feature mean, we separate the input features as non-overlapping chunks, and separable convolutions take place for each feature direction. We also propose a dynamic filter based attention pooling method. These methods are applied to the kernel part in our previous work, and experiments are carried out for keyword spotting and speaker verification. We confirm that our proposed method performs better in unseen environments than the recently developed models.
Building Markovian Generative Architectures Over Pretrained LM Backbones for Efficient Task-Oriented Dialog Systems

Recently, Transformer based pretrained language models (PLMs), such as GPT2 and T5, have been leveraged to build generative task-oriented dialog (TOD) systems. A drawback of existing PLM-based models is their non-Markov architectures across turns, i.e., the whole history is used as the conditioning input at each turn. First, this brings inefficiencies in memory and computation. Furthermore, using the whole history increases model complexity and may hurt the training efficiency, especially when facing small amounts of labeled training data (the low-resource setting). In this paper, motivated by the observation that dialog states could be viewed as Markov states, we propose to build Markovian Generative Architectures (MGA) over PLM backbones for efficient TOD systems. Experiments on MultiWOZ2.1 show that in the rich-resource setting, the proposed Markov models reduce memory and time costs without performance degradation; in the low-resource setting, the training efficiency of the Markov models is more significant.
Flickering Reduction with Partial Hypothesis Reranking for Streaming ASR

Incremental speech recognizers start displaying results while the users are still speaking. These partial results are beneficial to users who like the responsiveness of the system. However, as new partial results come in, words that were previously displayed can change or disappear. The results appear unstable and this unwanted phenomenon is called flickering. Typical remediation approaches can increase latency and reduce the quality of the partials results, but little work has been done to measure these effects. We first introduce two new metrics that allow us to measure the quality and latency of the partials. We propose the new, lightweight approach of reranking the partial results in favor of a more stable prefix without changing the beam search. This allows us to reduce flickering without impacting the final result. We show that we can roughly halve the amount of flickering with negligible impact on the quality and latency of the partial results.
Efficient Text Analysis with Pre-Trained Neural Network Models

This paper investigates the application of pre-trained BERT model in three classic text analysis tasks: Chinese grapheme-to-phoneme(G2P), text normalization(TN) and sentence punctuation annotation. Even though the full-sized BERT has prominent modeling power, there are two challenges for it in real applications: the requirement for annotated training data and the considerable computational cost. In this paper, we propose BERT-based low-latency solutions. To collect sufficient training corpus for G2P, we transfer knowledge from existing rule-based system to BERT through a large amount of unlabeled corpus. The new model could convert all characters directly from raw texts with higher accuracy. We also propose a hybrid two-stage text normalization pipeline which reduces the sentence error rate by 25% compared to the rule-based system. We offer both supervised and weakly supervised versions and find that the latter has only 1% accuracy drop from the former.
VTTS: Visual-Text To Speech

This paper proposes a visual-text to speech (vTTS) method, a method for synthesizing speech directly from visual text (i.e., text as an image). vTTS can use visual features in visual text that should be important for speech synthesis such as emphasis and radicals (components in Chinese characters), but they are not available in conventional TTS using discrete symbols as the input. The proposed vTTS method extracts visual features with a convolutional neural network and then generates acoustic features with a non-autoregressive model in an end-to-end manner. Experimental results show that 1) the vTTS method is capable of generating speech with naturalness comparable to or better than a conventional TTS, 2) it can transfer emphasis and emotion attributes in visual text to speech without additional labels and architectures, and 3) it can synthesize more natural and intelligible speech from unseen and rare characters than conventional TTS.
Improving Noise Robustness for Spoken Content Retrieval Using Semi-Supervised ASR and N-Best Transcripts for BERT-Based Ranking Models

BERT-based re-ranking and dense retrieval (DR) systems have been shown to improve search effectiveness for spoken content retrieval (SCR). However, both methods can still show a reduction in effectiveness when using ASR transcripts in comparison to accurate manual transcripts. We find that a known-item search task on the How2 dataset of spoken instruction videos shows a reduction in mean reciprocal rank (MRR) scores of 10-14%. As a potential method to reduce this disparity, we investigate the use of semi-supervised ASR transcripts and N-best ASR transcripts to mitigate ASR errors for spoken search using BERT-based ranking. Semi-supervised ASR transcripts brought 2-5.5% MRR improvements over standard ASR transcripts and our N-best early fusion methods for BERT DR systems improved MRR by 3-4%. Combining semi-supervised transcripts with N-best early fusion for BERT DR reduced the MRR gap in search effectiveness between manual and ASR transcripts by more than 50% from 14.32% to 6.58%.
Textual Data Augmentation for Arabic-English Code-Switching Speech Recognition

The pervasiveness of intra-utterance code-switching (CS) in spoken content requires that speech recognition (ASR) systems handle mixed language. Designing a CS-ASR system has many challenges, mainly due to data scarcity, grammatical structure complexity, and domain mismatch. The most common method for addressing CS is to train an ASR system with the available transcribed CS speech, along with monolingual data. In this work, we propose a zero-shot learning methodology for CS-ASR by augmenting the monolingual data with artificially generating CS text. We based our approach on random lexical replacements and Equivalence Constraint (EC) while exploiting aligned translation pairs to generate random and grammatically valid CS content. Our empirical results show a 65.5% relative reduction in language model perplexity, and 7.7% in ASR WER on two ecologically valid CS test sets. The human evaluation of the generated text using EC suggests that more than 80% is of adequate quality.
Exact Prosody Cloning in Zero-Shot Multispeaker Text-to-Speech

The cloning of a speaker's voice using an untranscribed reference sample is one of the great advances of modern neural text-to-speech (TTS) methods. Approaches for mimicking the prosody of a transcribed reference audio have also been proposed recently. In this work, we bring these two tasks together for the first time through utterance level normalization in conjunction with an utterance level speaker embedding. We further introduce a lightweight aligner for extracting fine-grained prosodic features, that can be finetuned on individual samples within seconds. We show that it is possible to clone the voice of a speaker as well as the prosody of a spoken reference independently without any degradation in quality and high similarity to both original voice and prosody, as our objective evaluation and human study show. All of our code and trained models are available, alongside static and interactive demos.
Cost-Sensitive Learning for Medical Insurance Fraud Detection With Temporal Information

Fraudulent activities within the U.S. healthcare system cost billions of dollars each year and harm the wellbeing of many qualifying beneficiaries. The implementation of an effective fraud detection method has become imperative to secure the welfare of the general public. In this article, we focus on the problem of fraud detection using the current year's Medicare claims data from the perspective of utilizing temporal information from the previous years. We group the data into temporal trajectories of the key covariates and base our feature engineering around these trajectories. For effective feature engineering on the temporal data, we propose to use the functional principal component analysis (FPCA) method for analyzing the temporal covariates’ trajectory as well as the distributional FPCA for extracting features from the empirical probability density curve of the covariates. Moreover, we introduce the framework of cost-sensitive learning for analyzing the Medicare database to allow for asymmetrical losses in the confusion matrix, such that the classification rule reflects the realistic tradeoff between the fixed cost and the fraud cost. The issue of class imbalance in the database is tackled through the random undersampling scheme. Our results confirm that the trained classifier has a reasonably good prediction performance and a significant percentage of cost savings can be achieved by taking into account the financial cost.
Scoring the Original Soundtrack of an Escape Room

Through the example of Nobody – Vis et ressens (Nice, France, 2021), this article sheds light on the musical creation process involved in the conception of an automatized escape room, where participants experience a multimodal experience (sound, light, scenery, video) driven by an original scenario. As the composer, sound designer, and computer music designer, I propose to study Nobody as a playful game. I show how current ludomusicology insights can be used and adapted to a game that is played in a physical space. After first exposing considerations of both technical and aesthetic aspects, I explain the artistic choices made for creating the soundtrack of six interactive puzzles. Through practice-led research of these different mini music games, I analyze how the influence of electronic dance music (EDM) and related club culture are melded together into the development of the plot and its retro-futuristic theme. I show how constraints forced me to adapt the audio content yet also opened a wide range of musical possibilities. The analysis helps clarify how what I call “game music” differs from “background music”. Moreover, the discussion highlights compositional issues that are similar to video game music-making and shows that escape rooms need to be studied further within the field of ludomusicology. Although some of the analytical tools remain valid because of their technical aspect (transition types, typology of music games, transition speed quantization) or their theoretical aspects (ALI model, branching-layering concept, triple lock of synchronization), traditional concepts such as “immersion” or “game feel” need to be questioned for escape room games, as that type of game takes place in a physical space.
Business Environment and Inclusive Growth Nexus: Evidence from African Countries

It is increasingly recognized that economic growth alone is not enough to guarantee the well-being of all.This is why the current debates on sustainable development are highlighting the need to promote inclusive growth in the world's countries in general, and in developing countries in particular.And one way to achieve inclusive growth would be an improved business environment through better business regulations.This paper aims to analyze the relationship between business environment and inclusive growth in a sample of African countries over the period 2010-2018.The business environment is measured by the World Bank's Ease of Doing Business index.Inclusive growth is measured by two proxies: 1) GDP per person employed and 2) an inclusive growth index constructed using Principal Component Analysis (PCA) method.In addition, we analyzed the above relationship in a static and dynamic framework.In the static framework, the business environment-inclusive growth relationship was analyzed by estimating a Panel Corrected Standard Error (PCSE) model as the baseline model and several other estimation models such as random-effects, fixed-effect Driscoll-Kraay, and pooled Ordinary Least Squares (pooled OLS).As for the dynamic framework, the study used the Generalized Method of Moments in system (System-GMM) estimation technique.The results show a positive and significant effect of business environment on inclusive growth.This result remains robust to the different estimation techniques used on the one hand and regardless of the measure of inclusive growth used on the other hand and suggests that a regulatory environment conducive to business activities is one way for African countries to achieve inclusive growth.From this result, policy recommendations have been formulated accordingly.
Robust Hyperspectral Inpainting via Low-Rank Regularized Untrained Convolutional Neural Network

Over the past decade, many low-rank models, factorizations, or approximations have been applied to the restoration of hyperspectral images (HSIs) (e.g., denoising, inpainting, and super-resolution) from their incomplete and/or noisy measurements. Recently, deep learning (DL) has been shown to be a powerful method for solving inverse problems (including HSI restoration), but a large amount of training data is required. Since this is not possible for HSIs, unlike red green blue (RGB) images, in this work, a novel unsupervised framework for hyperspectral inpainting (HI) is proposed that can be implemented using an untrained convolutional neural network (CNN) for deep image prior (DIP), together with a recently reported differentiable regularization for the data rank and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell _{2}$ </tex-math></inline-formula> -norm squared loss function. Based on the proposed framework, we come up with a novel HI algorithm [denoted as deep low-rank hyperspectral inpainting (DLRHyIn)] and a robust DLRHyIn (denoted as R-DLRHyIn) which is robust against outliers, where the latter differs from the former only in the Huber loss function (HLF) (which has been justified robust to mixed noise) used instead. Then some simulation results and real-data experiments are provided to demonstrate the effectiveness of the proposed DLRHyIn and R-DLRHyIn. Finally, we draw some conclusions.
Implicit Embeddings via GAN Inversion for High Resolution Chest Radiographs

Generative models allow for the creation of highly realistic artificial samples, opening up promising applications in medical imaging. In this work, we propose a multi-stage encoder-based approach to invert the generator of a generative adversarial network (GAN) for high resolution chest radiographs. This gives direct access to its implicitly formed latent space, makes generative models more accessible to researchers, and enables to apply generative techniques to actual patient's images. We investigate various applications for this embedding, including image compression, disentanglement in the encoded dataset, guided image manipulation, and creation of stylized samples. We find that this type of GAN inversion is a promising research direction in the domain of chest radiograph modeling and opens up new ways to combine realistic X-ray sample synthesis with radiological image analysis.
A Multimodal Aggregation Network With Serial Self-Attention Mechanism for Micro-Video Multi-Label Classification

Currently, micro-videos have attracted increasing attention due to their unique properties and great commercial value. Considering that micro-videos naturally incorporate multimodal information, a powerful representation method for distinct joint multimodal representations is essential for real applications. Inspired by the potential of attention neural network architectures over various tasks, we propose a multimodal aggregation network (MANET) with a serial self-attention mechanism to perform tasks of micro-video multi-label classification. Specifically, we first propose a parallel content-dependent graph neural networks (CDGNN) module, which explores category-related embeddings of micro-videos by disentangling category relations into modality-specific and modality-shared category dependency patterns. Then we introduce a serial self-attention (SSA) module to transmit the multimodal information in sequential order, in which an aggregation bottleneck is incorporated to better collect and condense the significant information. Experiments conducted on a large-scale multi-label micro-video dataset demonstrate that our proposed method has achieved competitive results compared with several state-of-the-art methods.
Heterogeneous Differential-Private Federated Learning: Trading Privacy for Utility Truthfully

Differential-private federated learning (DP-FL) has emerged to prevent privacy leakage when disclosing encoded sensitive information in model parameters. However, the existing DP-FL frameworks usually preserve privacy homogeneously across clients, while ignoring the different privacy attitudes and expectations. Meanwhile, DP-FL is hard to guarantee that uncontrollable clients (i.e., stragglers) have truthfully added the expected DP noise. To tackle these challenges, we propose a heterogeneous differential-private federated learning framework, named HDP-FL, which captures the variation of privacy attitudes with truthful incentives. First, we investigate the impact of the HDP noise on the theoretical convergence of FL, showing a tradeoff between privacy loss and learning performance. Then, based on the privacy-utility tradeoff, we design a contract-based incentive mechanism, which encourages clients to truthfully reveal private attitudes and contribute to learning as desired. In particular, clients are classified into different privacy preference types and the optimal privacy-price contracts in the discrete-privacy-type model and continuous-privacy-type model are derived. Our extensive experiments with real datasets demonstrate that HDP-FL can maintain satisfactory learning performance while considering different privacy attitudes, which also validate the truthfulness, individual rationality, and effectiveness of our incentives.
Distributed Algorithms for Boolean Equations Over Networks

In this article, we study systems of Boolean equations over a network, where each node in the network possesses only one Boolean equation from the system. The Boolean equation assigned at any particular node is a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">private</i> equation known to this node only, and the aim of this article is to develop distributed algorithms that allow all the nodes to obtain solutions to the network Boolean equations without exchanging their local Boolean equations. First, we observe that the Boolean equations can be locally lifted to a system of linear algebraic equations under a basis of Boolean vectors, which is distributedly solvable using existing distributed linear equation algorithms as a subroutine. Next, we construct a distributed Boolean equation solver by the nodes solving the lifted linear network equation for a number of randomly selected initial values, and then converting the algebraic solutions into solutions to the original Boolean equations by a novel Boolean vector search algorithm. We prove that for solvable Boolean equations, when the initial values of the nodes for the distributed linear equation solving step are independent identically distributed selected according to a uniform distribution in a high-dimensional cube, such an algorithm returns the exact solution set of the Boolean equations at each node with high probability. We also present an algorithm for distributed verification of the satisfiability of Boolean equations when the solvability is not known beforehand, and prove its correctness. Finally, we show that by utilizing linear equation solvers with differential privacy to replace the in-network computing routines, the distributed Boolean equation algorithms can be made differentially private