Reviewerly/exHarmony huggingface dataset:
Modified Apriori Based Data Sanitization for Cloud Data Security: An Optimization Assisted Model

This study proposes a new approach to cloud data security, with two distinct processes: proposed data sanitization and restoration. By hiding it from unwanted users and access, the data sanitization procedure ensures the protection of sensitive data stored in massive databases. In this case, the Modified Apriori approach is used to clean the data. In this scenario, the central goal is to generate a key, which is done using an optimization algorithm. Recovery or retrieval of cleaned data is a reversible process in data recovery. The optimization key for both sanitization and restoration will be discovered using the proposed MFUWOA. The optimal key for both restorations as well as sanitization will be determined using the proposed MFUWOA. While resolving the optimization challenge, the targets stated include the data retention ratio, hidden ratio, and level of alteration. Moreover, the adopted MFUWOA model for dataset 1 attains the best fitness value (∼0.2) to extant schemes including WOA (∼0.3), MFO (∼0.7), PRO (∼0.34), SSO (∼0.8), CMBO (∼0.4), and ALO (∼1.0), correspondingly.
Test‐Time Optimization for Video Depth Estimation Using Pseudo Reference Depth

Abstract In this paper, we propose a learning‐based test‐time optimization approach for reconstructing geometrically consistent depth maps from a monocular video. Specifically, we optimize an existing single image depth estimation network on the test example at hand. We do so by introducing pseudo reference depth maps which are computed based on the observation that the optical flow displacement for an image pair should be consistent with the displacement obtained by depth‐reprojection. Additionally, we discard inaccurate pseudo reference depth maps using a simple median strategy and propose a way to compute a confidence map for the reference depth. We use our pseudo reference depth and the confidence map to formulate a loss function for performing the test‐time optimization in an efficient and effective manner. We compare our approach against the state‐of‐the‐art methods on various scenes both visually and numerically. Our approach is on average 2.5× faster than the state of the art and produces depth maps with higher quality.
The Ihara expression of a generalization of the weighted zeta function on a finite digraph

We define a new weighted zeta function for a finite digraph and obtain its determinant expression called the Ihara expression. The graph zeta function is a generalization of the weighted graph zeta function introduced in previous research. That is, our result makes it possible to derive the Ihara expressions of the previous graph zeta functions for any finite digraphs.
Anomaly detection method of segment-powered linear induction motor based on AE-GAN

Segment-powered linear motors (SLM) have high reliability, unbalanced operating data samples and few abnormal samples, making it difficult to detect anomalies through supervised classification models. Therefore, we propose an anomaly detection method based on the reconstruction model, which uses the difference between small reconstruction error of normal samples and large reconstruction error of abnormal samples to detect anomalies. Meanwhile, in order to make the reconstruction model better reconstruct the input samples, this paper introduces adversarial regularization to improve the training effect. The proposed model uses normal samples for training, and unknown state samples for testing. The anomaly score is obtained through the reconstruction error between the input sample and the reconstructed sample, so as to judge whether the input sample is abnormal. Experiments are carried out on the actual operation data of the SLM, which proves the effectiveness of the proposed method in the anomaly detection task of the SLM.
Unsupervised Domain Adaptation of Neural PLDA Using Segment Pairs for Speaker Verification

Probabilistic linear discriminant analysis (PLDA) is a popular tool in speaker recognition. PLDA parameters are estimated from labeled data, and domain shift causes performance degradation. Since obtaining a labeled dataset for target domain is costly, we propose an unsupervised domain adaptation of PLDA which does not require clustering or alignment. Our proposed method relies on discriminative training of PLDA using pseudo target/non-target labels. Segment pairs extracted from the same utterance that is likely to contain a single speaker are labeled as target, and pairs from randomly sampled utterances that are likely to belong to different speakers are labeled as non-target. We applied this approach to Neural PLDA. Our approach significantly improved the performance of the out-of-domain PLDA on the target domain with a relatively small amount of unlabeled data, performing on par with a baseline adaptation approach on SRE18 full-full scenario while outperforming it on full-10s and 10s-10s cases.
On Granularity of Prosodic Representations in Expressive Text-to-Speech

In expressive speech synthesis it is widely adopted to use latent prosody representations to deal with variability of the data during training. Same text may correspond to various acoustic realizations, which is known as a one-to-many mapping problem in text-to-speech. Utterance, word, or phoneme-level representations are extracted from target signal in an auto-encoding setup, to complement phonetic input and simplify that mapping. This paper compares prosodic embeddings at different levels of granularity and examines their prediction from text. We show that utterance-level embeddings have insufficient capacity and phoneme-level tend to introduce instabilities when predicted from text. Word-level representations impose balance between capacity and predictability. As a result, we close the gap in naturalness by 90% between synthetic speech and recordings on LibriTTS dataset, without sacrificing intelligibility.
Improving Luxembourgish Speech Recognition with Cross-Lingual Speech Representations

Luxembourgish is a West Germanic language spoken by roughly 390,000 people, mainly in Luxembourg. It is one of Europe's under-described and under-resourced languages, not extensively investigated in the context of speech recognition. We explore the self-supervised multilingual learning of Luxembourgish speech representations for the speech recognition downstream task. We show that learning cross-lingual representations is essential for low-resourced languages such as Luxembourgish. Learning cross-lingual representations and rescoring the output transcriptions with language modelling while using only 4 hours of labelled speech achieves a word error rate of 15.1% and improves our Transfer Learning baseline model relatively by 33.1% and absolutely by 7.5%. Increasing the amount of labelled speech to 14 hours yields a significant performance gain resulting in a 9.3% word error rate. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Models and datasets are available at https://hugging£ace.co/lemswasabi
G-Augment: Searching for the Meta-Structure of Data Augmentation Policies for ASR

Data augmentation is a ubiquitous technique used to provide robustness to automatic speech recognition (ASR) training. However, even as so much of the ASR training process has become automated and more “end-to-end,” the data augmentation policy (what augmentation functions to use, and how to apply them) remains hand-crafted. We present G(raph)-Augment, a technique to define the augmentation space as directed acyclic graphs (DAGs) and search over this space to optimize the augmentation policy itself. We show that given the same computational budget, policies produced by G-Augment are able to perform better than SpecAugment policies obtained by random search on fine-tuning tasks on CHiME-6 and AMI. G-Augment is also able to establish a new state-of-the-art ASR performance on the CHiME-6 evaluation set (30.7% WER). We further demonstrate that G- Augment policies show better transfer properties across warm-start to cold-start training and model size compared to random-searched SpecAugment policies.
Fine Grained Spoken Document Summarization Through Text Segmentation

Podcast transcripts are long spoken documents of conversational dialogue. Challenging to summarize, podcasts cover a diverse range of topics, vary in length, and have uniquely different linguistic styles. Previous studies in podcast summarization have generated short, concise dialogue summaries. In contrast, we propose a method to generate long fine-grained summaries, which describe details of sub-topic narratives. Leveraging a readability formula, we curate a data subset to train a long sequence transformer for abstractive summarization. Through text segmentation, we filter the evaluation data and exclude specific segments of text. We apply the model to segmented data, producing different types of fine grained summaries. We show that appropriate filtering creates comparable results on ROUGE and serves as an alternative method to truncation. Experiments show our model outperforms previous studies on the Spotify podcast dataset when tasked with generating longer sequences of text.
An Attention-Based Backend Allowing Efficient Fine-Tuning of Transformer Models for Speaker Verification

In recent years, self-supervised learning paradigm has received extensive attention due to its great success in various down-stream tasks. However, the fine-tuning strategies for adapting those pre-trained models to speaker verification task have yet to be fully explored. In this paper, we analyze several feature extraction approaches built on top of a pre-trained model, as well as regularization and a learning rate scheduler to stabilize the fine-tuning process and further boost performance: multi-head factorized attentive pooling is proposed to factorize the comparison of speaker representations into multiple phonetic clusters. We regularize towards the parameters of the pre-trained model and we set different learning rates for each layer of the pre-trained model during fine-tuning. The experimental results show our method can significantly shorten the training time to 4 hours and achieve SOTA performance: 0.59%, 0.79% and 1.77% EER on Vox1-O, Vox1-E and Vox1-H, respectively. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Code is available at https://github.com/JunyiPeng00/IEEE-SLT22-Pretrained-Model-for-SV.
Towards Visually Prompted Keyword Localisation for Zero-Resource Spoken Languages

Imagine being able to show a system a visual depiction of a keyword and finding spoken utterances that contain this keyword from a zero-resource speech corpus. We formalise this task and call it visually prompted keyword localisation (VPKL): given an image of a keyword, detect and predict where in an utterance the keyword occurs. To do VPKL, we propose a speech-vision model with a novel localising attention mechanism which we train with a new keyword sampling scheme. We show that these innovations give improvements in VPKL over an existing speech-vision model. We also compare to a visual bag-of-words (BoW) model where images are automatically tagged with visual labels and paired with unlabelled speech. Although this visual BoW can be queried directly with a written keyword (while our's takes image queries), our new model still outperforms the visual BoW in both detection and localisation, giving a 16% relative improvement in localisation F1.
Fully Unsupervised Training of Few-Shot Keyword Spotting

For training a few-shot keyword spotting (FS-KWS) model, a large labeled dataset containing massive target keywords has known to be essential to generalize to arbitrary target keywords with only a few enrollment samples. To alleviate the expensive data collection with labeling, in this paper, we propose a novel FS-KWS system trained only on synthetic data. The proposed system is based on metric learning enabling target keywords to be detected using distance metrics. Exploiting the speech synthesis model that generates speech with pseudo phonemes instead of texts, we easily obtain a large collection of multi-view samples with the same semantics. These samples are sufficient for training, considering metric learning does not intrinsically necessitate labeled data. All of the components in our framework do not require any supervision, making our method unsupervised. Experimental results on real datasets show our proposed method is competitive even without any labeled and real datasets.
On Compressing Sequences for Self-Supervised Speech Models

Compressing self-supervised models has become increasingly necessary, as self-supervised models become larger. While previous approaches have primarily focused on compressing the model size, shortening sequences is also effective in reducing the computational cost. In this work, we study fixed-length and variable-length subsampling along the time axis in self-supervised learning. We explore how individual downstream tasks are sensitive to input frame rates. Subsampling while training self-supervised models not only improves the overall performance on downstream tasks under certain frame rates, but also brings significant speed-up in inference. Variable-length subsampling performs particularly well under low frame rates. In addition, if we have access to phonetic boundaries, we find no degradation in performance for an average frame rate as low as 10 Hz.
Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detection

Audio-visual active speaker detection (AVASD) is well-developed, and now is an indispensable front-end for several multi-modal applications. However, to the best of our knowledge, the adversarial robustness of AVASD models hasn't been investigated, not to mention the effective defense against such attacks. In this paper, we are the first to reveal the vulnerability of AVASD models under audio-only, visual-only, and audio-visual adversarial attacks through extensive experiments. What's more, we also propose a novel audio-visual interaction loss (AVIL) for making attackers difficult to find feasible adversarial examples under an allocated attack budget. The loss aims at pushing the inter-class embeddings to be dispersed, namely non-speech and speech clusters, sufficiently disentangled, and pulling the intra-class embeddings as close as possible to keep them compact. Experimental results show the AVIL outperforms the adversarial training by 33.14 mAP (%) under multi-modal attacks.
Learning Invariant Representation and Risk Minimized for Unsupervised Accent Domain Adaptation

Unsupervised representation learning for speech audios attained impressive performances for speech recognition tasks, particularly when annotated speech is limited. However, the unsupervised paradigm needs to be carefully designed and little is known about what properties these representations acquire. There is no guarantee that the model learns meaningful representations for valuable information for recognition. Moreover, the adaptation ability of the learned representations to other domains still needs to be estimated. In this work, we explore learning domain-invariant representations via a direct mapping of speech representations to their corresponding high-level linguistic informations. Results prove that the learned latents not only capture the articulatory feature of each phoneme but also enhance the adaptation ability, outperforming the baseline largely on accented benchmarks.
Automatic Prediction of Intelligibility of Words and Phonemes Produced Orally by Japanese Learners of English

The practical goal for language learning is smooth communication with others, and many teachers have a strong focus on measurement of not accentedness but intelligibility, often regarded as correctness of actual understanding. However, automatic prediction of intelligibility has not been well developed especially for smaller units such as words and phonemes. This is mainly because of difficulty of measuring while-listening behaviors of listeners, and thus it was difficult to build an L2 speech corpus of a sufficient size with intelligibility annotation to train a network-based predictor. In this paper, we annotate intelligibility using oral dictation with a small delay, i.e., shadowing, to collect a large enough corpus from two raters with different language backgrounds. Since perceived intelligibility depends on their language background, inter-rater difference should be taken into account. Therefore with this corpus, a multi-rater neural model is built to predict each rater's intelligibility of the individual words and phonemes in L2 speech. Two tasks are examined, i.e., regression of intelligibility scores and classification of a given segment to be intelligible or not. Results show that our model has higher F1 scores than intra-rater agreements, indicating that our model can simulate the two raters accurately well although they have different language background.
Sub-8-Bit Quantization for On-Device Speech Recognition: A Regularization-Free Approach

For on-device automatic speech recognition (ASR), quantization aware training (QAT) is ubiquitous to achieve the trade-off between model predictive performance and efficiency. Among existing QAT methods, one major drawback is that the quantization centroids have to be predetermined and fixed. To overcome this limitation, we introduce a regularization-free, “soft-to-hard” compression mechanism with self-adjustable centroids in a <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mu$</tex> -Law constrained space, resulting in a simpler yet more versatile quantization scheme, called General Quantizer (GQ). We apply GQ to ASR tasks using Recurrent Neural Network Transducer (RNN-T) and Conformer architectures on both LibriSpeech and de-identified far-field datasets. Without accuracy degradation, GQ can compress both RNN-T and Conformer into sub-8-bit, and for some RNN-T layers, to 1-bit for fast and accurate inference. We observe a 30.73% memory footprint saving and 31.75% user-perceived latency reduction compared to 8-bit QAT via physical device benchmarking.
How to Boost Anti-Spoofing with X-Vectors

With the development of speech synthesis or voice conversion, speech spoofing countermeasures are increasingly required for protecting automatic speaker verification system. In our daily life, if we are familiar with the speaker, we tend to seek her/his traits in our memory to distinguish between bona fide and spoofed speech of her/him. Speaker label can not be directly used to guide the training of anti-spoofing network because it is difficult to obtain in real scenes. Motivated by this, we use x-vectors to represent speaker information and propose two novel methods by introducing x-vectors on the acoustic feature and embedding level into the two mainstream anti-spoofing methods (LightCNN and SeNet). An attention module is also added on the embedding level for further improvement. Experimental results on the ASVspoof 2019 logical access (LA) database show that the best EER and mintDCF in our methods are 0.98% and 0.0294, outperforming state-of-the-art single systems as far as we know.
Diarisation Using Location Tracking with Agglomerative Clustering

Previous works have shown that spatial location information can be complementary to speaker embeddings for a speaker diarisation task. However, the models used often assume that speakers are fairly stationary throughout a meeting. This paper proposes to relax this assumption, by explicitly modelling the movements of speakers within an Agglomerative Hierarchical Clustering (AHC) diarisation framework. Kalman filters, which track the locations of speakers, are used to compute log-likelihood ratios that contribute to the cluster affinity computations for the AHC merging and stopping decisions. Experiments show that the proposed approach is able to yield improvements on a Microsoft rich meeting transcription task, compared to methods that do not use location information or that make stationarity assumptions.
How do Phonological Properties Affect Bilingual Automatic Speech Recognition?

Multilingual Automatic Speech Recognition (ASR) for Indian languages is an obvious technique for leveraging their similarities. We present a detailed analysis of how phonological similarities and differences between languages affect Time Delay Neural Network (TDNN) and End-to-End (E2E) ASR. To study this, we select genealogically similar pairs from five Indian languages and train bilingual acoustic models. We compare these against corresponding monolingual acoustic models and find similar phoneme distributions within speech to be the primary factor for improving model performance, with phoneme overlap being secondary. The influence of phonological properties on performance is visible in both cases. Word Error Rate (WER) of E2E decreased by a median of 2.35%, and upto 8.5% when the phonological similarity was greatest. WER of TDNN increased by 11.69% when the similarity was lowest. Thus, it is clear that the choice of supplementary language is important for model performance.
Learning to Jointly Transcribe and Subtitle for End-To-End Spontaneous Speech Recognition

TV subtitles are a rich source of transcriptions of many types of speech, ranging from read speech in news reports to conversational and spontaneous speech in talk shows and soaps. However, subtitles are not verbatim (i.e. exact) transcriptions of speech, so they cannot be used directly to improve an Automatic Speech Recognition (ASR) model. We propose a multitask dual-decoder Transformer model that jointly performs ASR and automatic subtitling. The ASR decoder (possibly pre-trained) predicts the verbatim output and the subtitle decoder generates a subtitle, while sharing the encoder. The two decoders can be independent or connected. The model is trained to perform both tasks jointly, and is able to effectively use subtitle data. We show improvements on regular ASR and on spontaneous and conversational ASR by incorporating the additional subtitle decoder. The method does not require preprocessing (aligning, filtering, pseudo-labeling,…) of the subtitles.
Match to Win: Analysing Sequences Lengths for Efficient Self-Supervised Learning in Speech and Audio

Self-supervised learning (SSL) has proven vital in speech and audio-related applications. The paradigm trains a general model on unlabeled data that can later be used to solve specific downstream tasks. This type of model is costly to train as it requires manipulating long input sequences that can only be handled by powerful centralised servers. Surprisingly, despite many attempts to increase training efficiency through model compression, the effects of truncating input sequence lengths to reduce computation have not been studied. In this paper, we provide the first empirical study of SSL pre-training for different specified sequence lengths and link this to various downstream tasks. We find that training on short sequences can dramatically reduce resource costs while retaining a satisfactory performance for all tasks. This simple one-line change would promote the migration of SSL training from data centres to user-end edge devices for more realistic and personalised applications.