Reviewerly/exHarmony huggingface dataset:
Numerically Stable Centered Error Entropy and Mixture Minimum Error Entropy Estimators for Bearings-Only Target Tracking Problem

This research focuses on tracking a moving target using only bearing measurements obtained by a moving observer, commonly termed bearings-only tracking (BoT) or target motion analysis (TMA). The accuracy of popular nonlinear state estimators such as the unscented Kalman filter (UKF) is affected by non-Gaussian noises in the measurements and uncertainties in the observer path. To address this, maximum correntropy (MC) and minimum error entropy (MEE) criteria have been recently reported in the literature. Even though the MEE criteria have the potential to produce more accurate estimates, the estimation framework tends to become unstable. Hence, to deal with the complexities of noise in the model and the instability in numerical computations, a robust mixture minimum error entropy criterion (MMEE) with UKF is proposed in this paper. In addition to MMEE, a robust centered error entropy-based estimator (RCEE) is also proposed. The RCEE-UKF and MMEE-UKF methods are compared with MEE-UKF and MC-UKF estimators through simulations, and it is found that the RCEE-UKF provides the best estimation accuracy.
CAPTuring Accents: An Approach to Personalize Pronunciation Training for Learners with Different L1 Backgrounds

This paper presents a novel approach to addressing the often-overlooked issue of pronunciation instruction in language learning through a Computer-Assisted Pronunciation Training (CAPT) system. While traditional CAPT systems are based on Automatic Speech Recognition (ASR) models trained on native speakers, we argue that this approach results in low accuracy when applied to non-native speakers. To address this limitation, we propose integrating advancements in ASR and accent recognition technology to create a more tailored and effective system. Specifically, our innovation lies in incorporating an accent recognition model into our mobile applications, allowing us to identify learners’ first language (L1) backgrounds and subsequently provide personalized exercises and feedback. By doing so, we enable course content creators to design exercises that are linguistically context-aware, and we employ ASR technology to enhance the accuracy of speech detection and accelerate transcription generation during the content creation phase. Furthermore, we make use of neural style transfer techniques to adapt learners’ accents before comparing them to reference pronunciations. The evaluation scores are then generated using the Dynamic Time Warping (DTW) algorithm. The key contribution of this paper lies in demonstrating how the integration of ASR-based and accent-targeted solutions can significantly enhance the effectiveness of CAPT systems. This integrated approach offers learners a more precise and personalized learning experience, thereby optimizing pronunciation training.
Significance of Indic Self-supervised Speech Representations for Indic Under-Resourced ASR

Automatic speech recognition is a mature speech technology, almost able to attend human label recognition performance conditioned on the availability of sufficient labeled training data. However, the performance of the system struggles to achieve deployable performance in the under-resourced scenario. In such a scenario, most of the work suggests traditional frameworks are preferable over state-of-the-art deep learning frameworks. This work creates a dataset for the Lambani language of 6 hours duration, and attempts to develop an ASR system. The system provides a character error rate (CER) of $$39.1\%$$ and $$24.1\%$$ using the GMM-HMM framework and TDNN framework, respectively for Lambani dataset. The language doesn’t have enough publicly available speech and corresponding text transcription resources of its own. Motivating by the same, this work uses the publicly available wav2vec2.0 (W2V) pre-trained model (trained on 23 Indian languages’ unlabeled speech data) and fine-tuned it with the labeled data of the Lambani language. After that using the fine-tuned framework as a non-linear feature extractor, the ASR task is performed with GMM-HMM and TDNN framework. The proposed approach provides a relative improvement of $$53.4\%$$ and $$32.1\%$$ for the GMM-HMM and TDNN frameworks, respectively.
Multi-task Learning over Mixup Variants for the Speaker Verification Task

Performance of the pseudo-label (PL)-based self-supervised training depends greatly on the quality of estimated PLs. Recent studies have shown that label noise can remarkably impact downstream performance. Recently, research has demonstrated that mixup regularization is effective against noise memorization. In this work, we extend this previous study by exploring several recent forms of mixup, namely 2-step interpolation double mixup to enhance model robustness, mixup over speech frames for better recognition at the frame-level, moment exchange mixup to encourage utilization of moment information of speaker speech as they can reveal speaker style, and virtual mixup training to regularize the areas in-between training points to be locally-Lipschitz and enforce consistent predictions. We analyze their effect on the generalization of some state-of-the-art speaker verification (SV) systems and explore their combination via different multi-task learning-based approaches. Our results show that the proposed mixup formulations are aligned with the SV task and that our proposed multi-task learning-based approach can be beneficial to improve the performance and robustness of SV systems.
Direct Vs Cascaded Speech-to-Speech Translation Using Transformer

Direct speech-to-speech translation (DS2ST) is a process of translating speech from one language to another without using a written form of the language. Most of the works attempted for DS2ST utilized the auxiliary network and knowledge from the written form of the language directly or indirectly to improve the performance. This work proposes a transformer-based sequence-to-sequence model to perform the DS2ST task without an auxiliary network. Also, a comparative study is made with a cascaded system. The experiments are performed with the Prabhupadavani dataset in two languages (Hindi and English). The result shows that with our proposed DS2ST model, a BLEU score of 16.46 is achieved without using any auxiliary information. We also augmented the data with speed perturbation and improved the DS2ST performance BLEU score to 18.58.
E-TTS: Expressive Text-to-Speech Synthesis for Hindi Using Data Augmentation

Current state-of-the-art text-to-speech (TTS) systems trained on read-speech have reduced issues with repetition or skipping of words and can produce natural-sounding speech. However, E2E systems have difficulty producing conversational speech, especially in generating out-of-domain terms, and lack appropriate prosody. This paper proposes a novel data augmentation approach to build intelligible and expressive speech using FastSpeech2 (FS2). Two different studies are performed. Conversational-style phrases/short interrogative sentences are synthesized using a baseline FS2 system and a hidden Markov model-based speech synthesis (HTS) system. Both systems are trained on 8.5 h of read speech in Hindi. This results in DS1 (FS2) and DH1 (HTS) synthetic datasets, respectively. Using DS1 and DH1, we train FS2 models, namely S1 and H1. While S1 sounds natural, H1 is more intelligible on OOD words. An attempt is made to further adapt these systems with as little as 11 min of original prosodically-rich story data from the same speaker to produce systems S2 and H2, respectively. We evaluate three FS2-based models: Baseline FS2 (vanilla FS2), the proposed models, S2, and H2. The subjective evaluation shows that systems S2 and H2 outperform the baseline FS2 system with an average MOS of 4.16 and 4.42, respectively. Further, we observe that H2 is better than S2 in terms of both MOS and intelligibility. We also do the objective evaluation tests and analyze the synthesized speech based on prosodic attributes to support our claim.
Cross Lingual Style Transfer Using Multiscale Loss Function for Soliga: A Low Resource Tribal Language

Voice conversion is the art of mimicking different speaker voices and styles. In this paper, we present a cross-lingual speaker style adaptation based on a multi-scale loss function, using a deep learning framework for syntactically similar languages Kannada and Soliga, under a low resource setup. The existing speaker adaptation methods usually depend on monolingual data and cannot be directly adopted for cross-lingual data. The proposed method calculates multi-scale reconstruction loss on the generated mel-spectrogram with that of the original mel-spectrogram and adopts its weights based on the loss function for various scales. Extensive experimental results illustrate that the multi-scale reconstruction resulted in a significant reduction of generator noise compared to the baseline model and faithfully transfers Soliga speaker styles to Kannada speakers while retaining the linguistic aspects of Soliga.
Curriculum Learning Based Approach for Faster Convergence of TTS Model

With the advent of deep learning, Text-to-Speech technology has been revolutionized, and current state-of-the-art models are capable of synthesizing almost human-like speech. Recent Text-to-Speech models use a sequence-to-sequence architecture that directly converts text or phoneme sequence into low-level acoustic representation such as spectrogram. These end-to-end models need a large dataset for training, and with conventional learning methodology, they need days of training to generate intelligible and natural voice. ‘How to use a large dataset to efficiently train a TTS model?’ has not been studied in the past. ‘Curriculum learning’ has been proven to speed up the convergence of models in other machine learning areas. For TTS task, the challenge in creating curriculum is to establish the difficulty criteria for the training samples. In this paper, we have experimented with various scoring functions based on text and acoustic features and achieved faster convergence of the end-to-end TTS model. We found ’text-length’ or the number of phonemes/characters in text to be a simple yet most effective measure of difficulty for designing curriculum for Text-to-Speech task. Using text-length based curriculum, we validated the faster convergence of TTS model using three datasets of different languages.
OBMO: One Bounding Box Multiple Objects for Monocular 3D Object Detection

Compared to typical multi-sensor systems, monocular 3D object detection has attracted much attention due to its simple configuration. However, there is still a significant gap between LiDAR-based and monocular-based methods. In this paper, we find that the ill-posed nature of monocular imagery can lead to depth ambiguity. Specifically, objects with different depths can appear with the same bounding boxes and similar visual features in the 2D image. Unfortunately, the network cannot accurately distinguish different depths from such non-discriminative visual features, resulting in unstable depth training. To facilitate depth learning, we propose a simple yet effective plug-and-play module, One Bounding Box Multiple Objects (OBMO). Concretely, we add a set of suitable pseudo labels by shifting the 3D bounding box along the viewing frustum. To constrain the pseudo-3D labels to be reasonable, we carefully design two label scoring strategies to represent their quality. In contrast to the original hard depth labels, such soft pseudo labels with quality scores allow the network to learn a reasonable depth range, boosting training stability and thus improving final performance. Extensive experiments on KITTI and Waymo benchmarks show that our method significantly improves state-of-the-art monocular 3D detectors by a significant margin (The improvements under the moderate setting on KITTI validation set are 1.82 ~ 10.91% mAP in BEV and 1.18 ~ 9.36% mAP in 3D). Codes have been released at https://github.com/mrsempress/OBMO.
From System Models to Class Models: An In-Context Learning Paradigm

Is it possible to understand the intricacies of a dynamical system not solely from its input/output pattern, but also by observing the behavior of other systems within the same class? This central question drives the study presented in this paper. In response to this query, we introduce a novel paradigm for system identification, addressing two primary tasks: one-step-ahead prediction and multi-step simulation. Unlike conventional methods, we do not directly estimate a model for the specific system. Instead, we learn a meta model that represents a class of dynamical systems. This meta model is trained on a potentially infinite stream of synthetic data, generated by simulators whose settings are randomly extracted from a probability distribution. When provided with a context from a new system–specifically, an input/output sequence–the meta model implicitly discerns its dynamics, enabling predictions of its behavior. The proposed approach harnesses the power of Transformers, renowned for their in-context learning capabilities. For one-step prediction, a GPT-like decoder-only architecture is utilized, whereas the simulation problem employs an encoder-decoder structure. Initial experimental results affirmatively answer our foundational question, opening doors to fresh research avenues in system identification.
Enhancing Children’s Short Utterance Based ASV Using Data Augmentation Techniques and Feature Concatenation Approach

The task of developing an automatic speaker verification (ASV) system for children’s speech is a challenging one due to a number of reasons. The dearth of domain-specific data is one among them. The challenge further intensifies with the introduction of short utterances of speech, a relatively unexplored domain in the case of children’s ASV. To circumvent the issue arising due to data scarcity, the work in this paper extensively explores various in-domain and out-of-domain data augmentation techniques. A data augmentation approach is proposed that encompasses both in-domain and out-of-domain data augmentation techniques. The out-of-domain data used are from adult speakers which are known to have acoustic attributes in stark contrast to child speakers. Consequently, various techniques like prosody modification, formant modification and voice-conversion are employed in order to modify the adult acoustic features and render it acoustically similar to children’s speech prior to augmentation. The in-domain data augmentation approach, on the other hand, involved speed perturbation of children’s speech. The proposed data augmentation approach helps not only in increasing the amount of training data but also in effectively capturing the missing target attributes which helps in boosting the verification performance. A relative improvement of $$43.91\%$$ in equal error rate (EER) with respect to the baseline system is a testimony of it. Furthermore, the commonly used Mel-frequency cepstral coefficients (MFCC) average out the higher-frequency components due to the larger bandwidth of the filter-bank. Therefore, effective preservation of higher-frequency contents in children’s speech is another challenge which must be appropriately tackled for the development of a reliable and robust children’stion techniques and Feature Concatenation A ASV system. The feature concatenation of MFCC and IMFCC is carried out with the sole intention of effectively preserving the higher-frequency contents in the children’s speech data. The feature concatenation approach, when combined with proposed data augmentation, helps in further improvement of the verification performance and results in an overall relative reduction of $$48.51\%$$ for equal error rate.
Phone Durations Modeling for Livvi-Karelian ASR

This paper presents the results of experiments conducted during development of an automatic speech recognition system for the low-resource Karelian language (Livvi-Karelian dialect). The main issues addressed within this work are related to acoustic modeling, viz. the treatment of long and short phonemes. There are two approaches to modeling phonological duration in the so-called quantity languages: representation of long and short phonemes as distinct units, and interpretation of long phonemes as reduplicated. There is currently no consensus on which strategy is the most promising. The Livvi-Karelian case is further complicated by the fact that the phonology of Karelian was heavily influenced by Russian, so that a direct transfer of the methods applied to other Balto-Finnic languages is questionable. In the course of the study, experiments were conducted with both approaches, showing that treating long phonemes as reduplicated outperforms the approaches implying introduction of long and short counterparts in the phoneme set. The usage of alternative transcriptions for words with long consonants further improved the recognition accuracy. In addition, the present study contributes to the application of DNN approaches to the tasks of language and acoustic modeling in low-resource languages. In the future works, it is planned to improve the performance of the developed system with transfer techniques and advanced data augmentation procedures.
A Novel Scheme to Classify Read and Spontaneous Speech

The COVID-19 pandemic has led to an increased use of remote telephonic interviews, making it important to distinguish between scripted and spontaneous speech in audio recordings. In this paper, we propose a novel scheme for identifying read and spontaneous speech. Our approach uses a pre-trained $$\texttt{DeepSpeech}$$ audio-to-alphabet recognition engine to generate a sequence of alphabets from the audio. From these alphabets, we derive features that allow us to discriminate between read and spontaneous speech. Our experimental results show that even a small set of self-explanatory features can effectively classify the two types of speech very effectively.
Analysis of a Hinglish ASR System’s Performance for Fraud Detection

Speech technology has made significant advancement in the last few decades. Recently it has also made in-roads into the households of the people through products like virtual assistants and voice-controlled devices. At the same time, in the shadows, and away from the eyes of the people, increasingly speech technology has been used gainfully for business tasks such as voice analytic and fraud detection in different industries. If the speech signal needs to be used for down-stream tasks such as developing a conversational voice bot, processing speech signals through an automatic speech recognition (ASR) engine is often an important and first step. Despite significant research in this space, typically the performance of an ASR engine is far from being perfect for all the test conditions it may encounter in the real world. The problem gets more complex when speech data has code-switching. Another equally important aspect while doing deployment of speech technology based products is that it is rather difficult to know if the performance of an ASR engine is adequate for its output to be used for a down-stream task. In this paper, we present our study of how the performance of an ASR engine developed for Hinglish (Hindi code switched with English) for the food domain affects the performance of the final down-stream task of detecting fraud patterns. In addition, we show how the performance of the final task can be improved by using state-of-the-art natural language processing (NLP) techniques like Bidirectional Encoder Representations from Transformers (BERT). The end-to-end system (ASR engine for Hinglish followed by NLP techniques) is successfully deployed in production for fraud detection at Swiggy, one of the largest food delivery platforms in India and the world.
Developing a Question Answering System on the Material of Holocaust Survivors’ Testimonies in Russian

The paper makes use of the annotated task-oriented corpus of Holocaust testimonies in Russian (ruOHQA) to train a question-answer neural network model. We start from data preprocessing, present statistical analysis of the collected corpus for approximately 1500 pairs of questions and answers and describe its strengths and limitations. Also, we carry out experiments on automatic processing of the ruOHQA corpus using pre-trained transformer-based neural network models. Finally, we explore the capability of several models to generate simplified high-quality answers to questions and compare their results. The kind of research we present allows us to extract knowledge from oral history archives more productively.
A CRF-Based POS Tagging Approach for Bishnupriya Manipuri Language

Bishnupriya Manipuri is an endangered language. Due to very limited resources, it becomes very challenging for carrying out any computational tasks for this type of endangered language. Bishnupriya Manipuri is also among the less computationally explored languages. In this paper, we present an automated part-of-speech tagging approach based on conditional random field using the CRF-Suite library, which is an important and basic task for any natural language processing task. We have carried out experiments on a corpus annotated by us using gold standard tags and achieved a satisfactory tagging accuracy of 86.21%. Further, the results are compared with some existing state-of-the-art approaches.
Effect of Noise in Khasi Speech Recognition System

In this paper, we investigated the effect of noise in Khasi automatic speech recognition system (ASR). Acoustic models (AMs) were built with clean speech data and tested with or without noise to analyze the performance. The AMs were trained with 39–dimensions cepstral mean-variance normalization(CM VN) for Mel-frequency cepstral coefficients (MFCC) and perceptual linear prediction (PLP). Linear discriminative analysis (LDA) was applied and then de-correlated by using maximum likelihood linear transformation (MLLT) to assign the integrated frames to 40 dimensions. Speaker adaption was also used to produce word lattices for each utterance generated by hidden Markov model (HMM). The study shows that increasing the signal to noise ratio (SNR) leads to the improvement of the recognition performance. Further, subspace-Gaussian mixture model (SGMM) outperformed the other conventional models irrespective of the data used. Additionally, with the increase of SNR level from 10–30 dB, reduction of word error rate (WER) in the range of 37.75–14.30% for MFCC and 37.70–14.91% for PLP were achieved, respectively.
A Meta-consensus Strategy for Binarization of Dendritic Spines Images

Image binarization is the process of separation of pixel values into two groups, foreground and background. Thresholding can be categorized into global thresholding and local thresholding. This paper describes a consensus-based strategy that takes into account the predictions of some classic thresholding methods like Otsu, Niblack, Savoula, and a deep learning model called UNet and takes a decision based on majority voting. In this work, we present different ways of combining the classic binarization methods. The qualitative and quantitative results show that it outperforms individual binarization method. The quantitative result is presented in terms of statistical parameters: F-measure, recall, precision.