Miracle-dz/UniFAR-FTUs huggingface dataset:
Adversarial Latent Representation Learning for Speech Enhancement

This paper proposes a novel adversarial latent representation learning (ALRL) method for speech enhancement. Based on adversarial feature learning, ALRL employs an extra encoder to learn an inverse mapping from the generated data distribution to the latent space. The encoder builds an inner connection with the generator, and provides relevant latent information for adversarial feature modelling. A new loss function is proposed to implement the encoder mapping simultaneously. In addition, the multi-head self-attention is also applied to the encoder for learning of long-range dependencies and further effective adversarial representations. The experimental results demonstrate that ALRL outperforms current GAN-based speech enhancement methods.
CAM: Uninteresting Speech Detector

Voice assistants such as Siri, Alexa, etc. usually adopt a pipeline to process users’ utterances, which generally include transcribing the audio into text, understanding the text, and ﬁ-nally responding back to users. One potential issue is that some utterances could be devoid of any interesting speech, and are thus not worth being processed through the entire pipeline. Examples of uninteresting utterances include those that have too much noise, are devoid of intelligible speech, etc. It is therefore desirable to have a model to ﬁlter out such useless utterances before they are ingested for downstream processing, thus saving system resources. Towards this end, we propose the Combination of Audio and Metadata (CAM) detector to identify utterances that contain only uninteresting speech. Our experimental results show that the CAM detector considerably outperforms using either an audio model or a metadata model alone, which demonstrates the effectiveness of the proposed system.
An Unsupervised Method to Select a Speaker Subset from Large Multi-Speaker Speech Synthesis Datasets

Large multi-speaker datasets for TTS typically contain diverse speakers, recording conditions, styles and quality of data. Although one might generally presume that more data is better, in this paper we show that a model trained on a carefully-chosen subset of speakers from LibriTTS provides signiﬁcantly better quality synthetic speech than a model trained on a larger set. We propose an unsupervised methodology to ﬁnd this subset by clustering per-speaker acoustic representations.
Deep Neural Network-Based Generalized Sidelobe Canceller for Robust Multi-Channel Speech Recognition

The elastic spatial ﬁlter (ESF) proposed in recent years is a popular multi-channel speech enhancement front end based on deep neural network (DNN). It is suitable for real-time processing and has shown promising automatic speech recognition (ASR) results. However, the ESF only utilizes the knowledge of ﬁxed beamforming, resulting in limited noise reduction capabilities. In this paper, we propose a DNN-based generalized sidelobe canceller (GSC) that can automatically track the target speaker’s direction in real time and use the blocking technique to generate reference noise signals to further reduce noise from the ﬁxed beam pointing to the target direction. The coefﬁcients in the proposed GSC are fully learnable and an ASR criterion is used to optimize the entire network. The 4-channel experiments show that the proposed GSC achieves a relative word error rate improvement of 27.0% compared to the raw observation, 20.6% compared to the oracle direction-based traditional GSC, 10.5% compared to the ESF and 7.9% compared to the oracle mask-based generalized eigenvalue (GEV) beamformer.
Confidence Measure for Speech-to-Concept End-to-End Spoken Language Understanding

Recent studies have led to the introduction of Speech-to-Concept End-to-End (E2E) neural architectures for Spoken Language Understanding (SLU) that reach state of the art performance. In this work, we propose a way to compute conﬁ-dence measures on semantic concepts recognized by a Speech-to-Text E2E SLU system. We investigate the use of the hidden representations of our CTC-based SLU system to train an ex-ternal simple classiﬁer. We experiment two kinds of external simple classiﬁers to analyze subsequences of hidden representations involved in recognized semantic concepts. The ﬁrst ex-ternal classiﬁer is based on a MLP while the second one is based on a bLSTM neural network. We compare them to a baseline conﬁdence measure computed directly from the softmax outputs of the E2E system. On the French challenging MEDIA corpus, when the conﬁdence measure is used to reject, experiments show that using an external BLSTM signiﬁcantly out-performs the other approaches in terms of precision/recall. To evaluate the additional information provided by this conﬁdence measure, we compute the value of Normalised Cross-Entropy (NCE). Reaching a value equal to 0.288, we show that our best proposed conﬁdence measure brings relevant information about the reliability of a recognized concept.
Forward and Backward Private Conjunctive Searchable Symmetric Encryption

Consider, for instance, a client that offloads an encrypted database of (potentially sensitive) emails to an untrusted server. At a later point of time, the client might want to issue a query of the form “retrieve all emails received from xyz@foobar.org or “retrieve all emails with the keyword “research” in the subject field”. Ideally, the client should be able to perform this task without revealing any sensitive information to the server, such as the sources and contents of the emails, the keywords underlying a given query, the distribution of keywords across emails, etc. Unfortunately, techniques such as fully homomorphic encryption [19], that potentially allow achieving such an “ideal” notion of privacy, are unsuitable for practical deployment due to large performance overheads.
Proteinuria and Bilirubinuria as Potential Risk Indicators of Acute Kidney Injury during Running in Outpatient Settings

Background and objectives: The purpose of this study was to explore which urinary markers could indicate acute kidney injury (AKI) during prolonged trail running in outpatient settings. Materials and Methods: Twenty-nine experienced trail runners (age 39.1 ± 8.8 years, weight 71.9 ± 11 kg, height 171.9 ± 8.3 cm) completed a 35 km event (cumulative positive ascend of 1815 m, altitude = 906 to 1178 m.a.s.l.) under a temperature of 25.52 ± 1.98 °C and humidity of 79.25 ± 7.45%). Two participant groups (AKI = 17 and No-AKI = 12) were made according to AKI diagnosis criteria based on pre- and post-race values of serum creatinine (sCr) (an increase of 1.5 times from baseline). Blood and urinalysis were performed immediately pre- and post-race. Results: Pre- vs. post-race differences in sCr and sBUN were found in both AKI and No-AKI groups (p < 0.01). Differences in post-race values were found between groups (p = 0.03). A total of 52% of AKI runners presented significant increases in proteinuria (χ2 = 0.94, p = 0.01) and 47% in bilirubinuria (χ2 = 0.94, p = 0.04). Conversely, No-AKI participants presented no significant increases in urine markers. Conclusions: These study’s findings may suggest the potential use of urinalysis as an accessible alternative in the outpatient setting to early identify transitional AKI until a clinical confirmation is performed.
Audiovisual Correspondence Learning in Humans and Machines

Audiovisual correspondence learning is the task of acquiring the association between images and its corresponding audio. In this paper, we propose a novel experimental paradigm in which unfamiliar pseudo images and pseudowords in audio form are introduced to both humans and machine systems. The task is to learn the association between the pairs of image and audio which is later evaluated with a retrieval task. The machine sys-tem used in the study is pretrained with the ImageNet corpus along with the corresponding audio labels. This model is transfer learned for the new image-audio pairs. Using the proposed paradigm, we perform a direct comparison of one-shot, two-shot and three-shot learning performance for humans and machine systems. The human behavioral experiment conﬁrms that the majority of the correspondence learning happens in the ﬁrst exposure of the audio-visual pair. This paper proposes a machine model which performs on par with the humans in audio-visual correspondence learning. But compared to the machine model, humans exhibited better generalization ability for new input samples with a single exposure.
On the Robustness and Training Dynamics of Raw Waveform Models

We investigate the robustness and training dynamics of raw waveform acoustic models for automatic speech recognition (ASR). It is known that the ﬁrst layer of such models learn a set of ﬁlters, performing a form of time-frequency analysis. This layer is liable to be under-trained owing to gradient vanishing, which can negatively affect the network performance. Through a set of experiments on TIMIT, Aurora-4 and WSJ datasets, we investigate the training dynamics of the ﬁrst layer by measuring the evolution of its average frequency response over different epochs. We demonstrate that the network efﬁciently learns an optimal set of ﬁlters with a high spectral resolution and the dynamics of the ﬁrst layer highly correlates with the dynamics of the cross entropy (CE) loss and word error rate (WER). In addition, we study the robustness of raw waveform models in both matched and mismatched conditions. The accuracy of these models is found to be comparable to, or better than, their MFCC-based counterparts in matched conditions and notably improved by using a better alignment. The role of raw wave-form normalisation was also examined and up to 4.3% absolute WER reduction in mismatched conditions was achieved.
Improved RawNet with Filter-wise Rescaling for Text-independent Speaker Verification using Raw Waveforms

Recent advances in deep learning have facilitated the design of speaker verification systems that directly input raw waveforms. For example, RawNet extracts speaker embeddings from raw waveforms, which simplifies the process pipeline and demonstrates competitive performance. In this study, we improve RawNet by scaling feature maps using various methods. The proposed mechanism utilizes a scale vector that adopts a sigmoid non-linear function. It refers to a vector with dimensionality equal to the number of filters in a given feature map. Using a scale vector, we propose to scale the feature map multiplicatively, additively, or both. In addition, we investigate replacing the first convolution layer with the sinc-convolution layer of SincNet. Experiments performed on the VoxCeleb1 evaluation dataset demonstrate the effectiveness of the proposed methods, and the best performing system reduces the equal error rate by half compared to the original RawNet. Expanded evaluation results obtained using the VoxCeleb1-E and VoxCeleb-H protocols marginally outperform existing state-of-the-art systems.
Attentive Convolutional Recurrent Neural Network Using Phoneme-Level Acoustic Representation for Rare Sound Event Detection

A well-trained Acoustic Sound Event Detection system captures the patterns of the sound to accurately detect events of interest in an auditory scene, which enables applications across domains of multimedia, smart living, and even health monitoring. Due to the scarcity and the weak labelling nature of the sound event data, it is often challenging to train an accurate and robust acoustic event detection model directly, especially for those rare occurrences. In this paper, we proposed an architecture which takes the advantage of integrating ASR network representations as additional input when training a sound event detector. Here we used the convolutional bi-directional recurrent neural network (CBRNN), which includes both spectral and temporal attentions, as the SED classiﬁer and further combined the ASR feature representations when performing the end-to-end CBRNN training. Our experiments on the TUT 2017 rare sound event detection dataset showed that with the inclusion of ASR features, the overall discriminative performance of the end-to-end sound event detection system has improved; the average performance of our proposed framework in terms of f-score and error rates are 97 % and 0.05 % respectively.
Finding Intelligible Consonant-Vowel Sounds Using High-Quality Articulatory Synthesis

In this study, a state-of-the-art articulatory speech synthesiser was used as the basis for simulating the exploration of CV sounds imitating speech stimuli. By adopting a relevant kinematic model and systematically reducing the search space of consonant articulatory targets, intelligible CV sounds can be found. Derivative-free optimisation strategies were evaluated to speed up the process of exploring articulatory space and the possibility of using automatic speech recognition as a means of evaluating intelligibility was explored.
Two-Stage Polyphonic Sound Event Detection Based on Faster R-CNN-LSTM with Multi-Token Connectionist Temporal Classification

We propose a two-stage sound event detection (SED) model to deal with sound events overlapping in time-frequency. In the first stage which consists of a faster R-CNN and an attention-LSTM, each log-mel spectrogram segment is divided into one or more proposed regions (PRs) according to the coordinates of a region proposal network. To efficiently train polyphonic sound, we take only one PR for each sound event from a bounding box regressor associated with the attention-LSTM. In the second stage, the original input image and the difference image between adjacent segments are separately pooled according to the coordinate of each PR predicted in the first stage. Then, two feature maps using CNNs are concatenated and processed further by LSTM. Finally, CTC-based n-best SED is conducted using the softmax output from the CNN-LSTM, where CTC has two tokens for each event so that the start and ending time frames are accurately detected. Experiments on SED using DCASE 2019 Task 3 show that the proposed two-stage model with multi-token CTC achieves an F1-score of 97.5%, while the first stage alone and the two-stage model with a conventional CTC yield F1-scores of 91.9% and 95.6%, respectively.
Learning Higher Representations from Pre-Trained Deep Models with Data Augmentation for the COMPARE 2020 Challenge Mask Task

Human hand-crafted features are always regarded as expensive, time-consuming, and difﬁcult in almost all of the machine-learning-related tasks. First, those well-designed features extremely rely on human expert domain knowledge, which may restrain the collaboration work across ﬁelds. Second, the features extracted in such a brute-force scenario may not be easy to be transferred to another task, which means a series of new features should be designed. To this end, we introduce a method based on a transfer learning strategy combined with data augmentation techniques for the C OM P AR E 2020 Challenge Mask Sub-Challenge. Unlike the previous studies mainly based on pre-trained models by image data, we use a pre-trained model based on large scale audio data, i.e., AudioSet. In addition, the SpecAugment and mixup methods are used to improve the generalisation of the deep models. Experimental results demonstrate that the best-proposed model can signiﬁcantly ( p < . 001 , by one-tailed z -test) improve the unweighted average recall (UAR) from 71.8% (baseline) to 76.2% on the test set. Finally, the best result, i.e., 77.5% of the UAR on the test set, is achieved by a late fusion of the two best proposed models and the best single model in the baseline.
Phonetic Entrainment in Cooperative Dialogues: A Case of Russian

It has been shown for a number of languages that speakers accommodate to each other in conversation. Such accommodation, or entrainment, reveals itself in many modalities including speech: interlocutors are found to entrain in intensity, fundamental frequency, tempo and other acoustic features. This paper presents data on speech entrainment in Russian using the standard measures for speech entrainment: proximity, convergence and synchrony. The research uses 49 dialogues from the SibLing speech corpus where speakers played a card-matching game. The list of acoustic features includes various measures of pitch, energy, spectral slope, HNR, jitter, and shimmer. The results for Russian are compared with those published previously for other languages.
Enhancing Formant Information in Spectrographic Display of Speech

Formants are resonances of the time varying vocal tract sys-tem, and their characteristics are reﬂected in the response of the system for a sequence of impulse-like excitation sequence originated at the glottis. This paper presents a method to enhance the formants information in the display of spectrogram of the speech signal, especially for high pitched voices. It is well known that in the narrowband spectrogram, the presence of pitch harmonics masks the formant information, whereas in the wideband spectrogram, the formant regions are smeared. Using single frequency ﬁltering (SFF) analysis, we show that the wideband equivalent SFF spectrogram can be modiﬁed to enhance the formant information in the display by improving the frequency resolution. For this, we obtain two SFF spectrograms by using single frequency ﬁltering of the speech signal at two closely spaced roots on the real axis in the z-plane. The ratio or difference of the two SFF spectrograms is processed to enhance the formant information in the spectrographic display. This will help in tracking rapidly changing formants and in resolving closely spaced formants. The effect is more pronounced in the case of high-pitched voices, like female and children speech.
Multilingual Speech Recognition Using Language-Specific Phoneme Recognition as Auxiliary Task for Indian Languages

This paper proposes a multilingual acoustic modeling approach for Indian languages using a Multitask Learning (MTL) framework. Language-speciﬁc phoneme recognition is explored as an auxiliary task in MTL framework along with the primary task of multilingual senone classiﬁcation. This auxiliary task regularizes the primary task with both the context-independent phonemes and language identities induced by language-speciﬁc phoneme. The MTL network is also extended by structuring the primary and auxiliary task outputs in the form of a Structured Output Layer (SOL) such that both depend on each other. The experiments are performed using a database of the three Indian languages Gujarati, Tamil, and Telugu. The experimental re-sults show that the proposed MTL-SOL framework performed well compared to baseline monolingual systems with a relative reduction of 3.1-4.4 and 2.9-4.1 % in word error rate for the development and evaluation sets, respectively.
Real-Time Single-Channel Deep Neural Network-Based Speech Enhancement on Edge Devices

In this paper, we present a deep neural network architecture comprising of both convolutional neural network (CNN) and recurrent neural network (RNN) layers for real-time single-channel speech enhancement (SE). The proposed neural network model focuses on enhancing the noisy speech magnitude spectrum on a frame-by-frame process. The developed model is implemented on the smartphone (edge device), to demonstrate the real-time usability of the proposed method. Perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI) test results are used to compare the proposed algorithm to previously published conventional and deep learning-based SE methods. Subjective ratings show the performance improvement of the proposed model over the other baseline SE methods.
Efficient WaveGlow: An Improved WaveGlow Vocoder with Enhanced Speed

Neural vocoder, such as WaveGlow, has become an important component in recent high-quality text-to-speech (TTS) systems. In this paper, we propose Efﬁcient WaveGlow (EWG), a ﬂow-based generative model serving as an efﬁcient neural vocoder. Similar to WaveGlow, EWG has a normalizing ﬂow backbone where each ﬂow step consists of an afﬁne coupling layer and an invertible 1 × 1 convolution. To reduce the number of model parameters and enhance the speed without sacriﬁcing the quality of the synthesized speech, EWG improves WaveGlow in three aspects. First, the WaveNet-style transform network in Wave-Glow is replaced with an FFTNet-style dilated convolution network. Next, to reduce the computation cost, group convolution is applied to both audio and local condition features. At last, the local condition is shared among the transform network layers in each coupling layer. As a result, EWG can reduce the number of ﬂoating-point operations (FLOPs) required to generate one-second audio and the number of model parameters both by more than 12 times. Experimental results show that EWG can reduce real-world inference time cost by more than twice, without any obvious reduction in the speech quality.
Sequence-Level Self-Learning with Multiple Hypotheses

In this work, we develop new self-learning techniques with an attention-based sequence-to-sequence (seq2seq) model for automatic speech recognition (ASR). For untranscribed speech data, the hypothesis from an ASR system must be used as a label. However, the imperfect ASR result makes unsupervised learning difficult to consistently improve recognition performance especially in the case that multiple powerful teacher models are unavailable. In contrast to conventional unsupervised learning approaches, we adopt the \emph{multi-task learning} (MTL) framework where the $n$-th best ASR hypothesis is used as the label of each task. The seq2seq network is updated through the MTL framework so as to find the common representation that can cover multiple hypotheses. By doing so, the effect of the \emph{hard-decision} errors can be alleviated. We first demonstrate the effectiveness of our self-learning methods through ASR experiments in an accent adaptation task between the US and British English speech. Our experiment results show that our method can reduce the WER on the British speech data from 14.55\% to 10.36\% compared to the baseline model trained with the US English data only. Moreover, we investigate the effect of our proposed methods in a federated learning scenario.
Controlling the Strength of Emotions in Speech-Like Emotional Sound Generated by WaveNet

This paper proposes a method to enhance the controllability of a Speech-like Emotional Sound (SES). In our previous study, we proposed an algorithm to generate SES by employing WaveNet as a sound generator and conﬁrmed that SES can successfully convey emotional information. The proposed algorithm generates SES using only emotional IDs, which results in having no linguistic information. We call the generated sounds “speech-like” because they sound as if they are uttered by human beings although they contain no linguistic information. We could synthesize natural sounding acoustic signals that are fairly different from vocoder sounds to make the best use of WaveNet. To ﬂexibly control the strength of emotions, this paper proposes to use a state of voiced, unvoiced, and silence (VUS) as auxiliary features. Three types of emotional speech, namely, neutral, angry, and happy, were generated and subjectively evaluated. Experimental results reveal the following: (1) VUS can control the strength of SES by changing the durations of VUS states, (2) VUS with narrow F0 distribution can express stronger emotions than that with wide F0 distribution, and (3) the smaller the unvoiced percentage is, the stronger the emotional impression is.