Reviewerly/exHarmony huggingface dataset:
Pseudo-Label Based Supervised Contrastive Loss for Robust Speech Representations

The self supervised learning (SSL) of speech, with discrete tokenization (pseudo-labels), while illustrating performance improvements in low-resource speech recognition, has faced challenges in achieving context invariant and noise robust representations. In this paper, we propose a self-supervised framework based on contrastive loss of the pseudo-labels, obtained from an offline k-means quantizer (tokenizer). We refer to the proposed setting as pseudo-con. The pseudo-con loss, within a batch of training, allows the model to cluster the instances of the same pseudo-label while separating the instances of a different pseudo-label. The proposed pseudo-con loss can also be combined with the cross entropy loss, commonly used in self-supervised learning schemes. We demonstrate the effectiveness of the pseudo-con loss applied for various SSL techniques, like hidden unit bidirectional encoder representations from transformers (HuBERT), best random quantizer (BEST-RQ) and hidden unit clustering (HUC). Our evaluations using the proposed pseudo-con framework achieves state of art results on various sub-tasks of ZeroSpeech 2021 challenge as well as on the context invariance benchmarks. Further, we show significant performance improvements over existing SSL approaches on the TIMIT phoneme recognition task as well as the Librispeech (100h) ASR experiments.
Enhancing Expressivity Transfer in Textless Speech-to-Speech Translation

Textless speech-to-speech translation systems are rapidly advancing, thanks to the integration of self-supervised learning techniques. However, existing state-of-the-art systems fall short when it comes to capturing and transferring expressivity accurately across different languages. Expressivity plays a vital role in conveying emotions, nuances, and cultural subtleties, thereby enhancing communication across diverse languages. To address this issue this study presents a novel method that operates at the discrete speech unit level and leverages multilingual emotion embeddings to capture language-agnostic information. Specifically, we demonstrate how these embeddings can be used to effectively predict the pitch and duration of speech units in the target language. Through objective and subjective experiments conducted on a French-to-English translation task, our findings highlight the superior expressivity transfer achieved by our approach compared to current state-of-the-art systems.
No Pitch Left Behind: Addressing Gender Unbalance In Automatic Speech Recognition Through Pitch Manipulation

Automatic speech recognition (ASR) systems are known to be sensitive to the sociolinguistic variability of speech data, in which gender plays a crucial role. This can result in disparities in recognition accuracy between male and female speakers, primarily due to the under-representation of the latter group in the training data. While in the context of hybrid ASR models several solutions have been proposed, the gender bias issue has not been explicitly addressed in end-to-end neural architectures. To fill this gap, we propose a data augmentation technique that manipulates the fundamental frequency $(f0)$ and formants. This technique reduces the data unbalance among genders by simulating voices of the under-represented female speakers and increases the variability within each gender group. Experiments on spontaneous English speech show that our technique yields a relative WER improvement up to 9.87% for utterances by female speakers, with larger gains for the least-represented $f0$ ranges.
Mechanical Properties Test of Graphene Concrete Based on Fuzzy Control Algorithm

Natural resources such as sand and stone in concrete are non-renewable resources in the short term, and unlimited exploitation and utilization will cause irreversible damage and impact on the ecological environment. Composite material is a kind of multiphase material, and its thermodynamic properties and failure mechanism are not only dependent on the macro service environment, but also closely related to the material properties, structural morphology and interface characteristics of each component. Multi-scale science is a basic, interdisciplinary and forward-looking science, which is used to study the phenomenon of coupling and correlation between different time or space scales. In this paper, a test method of mechanical properties of graphene concrete based on fuzzy control algorithm is proposed, which provides a reference for further research on mechanical properties of rockfill concrete. Judging from the correlation coefficient, the strength of mortar matrix should be based on the compressive strength when calculating the concrete micromechanics. Under the condition of the same mass mix ratio, the compressive strength of concrete increases with the strength of coarse aggregate used in the preparation of concrete.
A Token-Wise Beam Search Algorithm for RNN-T

Standard Recurrent Neural Network Transducers (RNN-T) decoding algorithms for speech recognition are iterating over the time axis, such that one time step is decoded before moving on to the next time step. Those algorithms result in a large number of calls to the joint network, which were shown in previous work to be an important factor that reduces decoding speed. We present a decoding beam search algorithm that batches the joint network calls across a segment of time steps, which results in 20 %-96 % decoding speedups consistently across all models and settings experimented with. In addition, aggregating emission probabilities over a segment may be seen as a better approximation to finding the most likely model output, causing our algorithm to improve oracle word error rate by up to 11 % relative as the segment size increases, and to slightly improve general word error rate.
A Comparison of Optimization Methods for Path Finding Problem

Abstract—Optimizing travel time on a multi-lane highway with dynamically changing traffic density poses a 2-dimensional path optimization challenge. The temporal evolution of traffic density introduces a quasi-quadratic nature to the optimization problem, rendering quadratic optimization methods applicable. This research compares prevalent quadratic optimization techniques—specifically, the interior point method (IPM), sequential quadratic programming (SQP), and active set (AS) methods—in terms of both travel times and computational efficiency. A Monte-Carlo (MC) study is conducted, incorporating variables like the number of lanes, road length, and constraints on car speed (i.e., speed limits).Additionally, a novel path optimization approach, termed ’segmented path optimization,' is introduced. This method involves dividing the road into segments, optimizing each segment, merging their end and start points, and discretizing the optimal path. Through our comparative analysis, we find that the efficacy of these methods is context-dependent. There is no universally superior method, as each demonstrates its advantages under different conditions. However, the pipeline and the optimization flowchart may help the structuring of future works that contain the optimization processes, especially in artificial intelligence, smart recommendation, and GPS-based systems.
Deriving Translational Acoustic Sub-Word Embeddings

There is a growing interest in understanding the representational geometry of acoustic word embeddings (AWEs), which are fixed-dimensional representations of spoken words. However, not much research has been conducted on acoustic sub-word embeddings (ASWEs), which can provide a better understanding of the AWE space. This work focuses on decomposing AWEs to obtain ASWEs while retaining the ability to reconstruct AWEs by translating ASWEs in the embedding space, under constrained settings. Initially, high-quality AWEs are obtained with an Average Precision (AP) score of 0.97 on the word discrimination task. Subsequently, ASWEs are derived through the decomposition of AWEs. Three adapted versions of the AP metric, utilized for evaluating the quality of the derived ASWEs and their translational properties, are proposed. The results demonstrate that the derived ASWEs exhibit high quality, and the reconstruction of AWEs from the ASWEs is achievable by translating them in the embedding space.
Speaker Adaptation for End-to-End Speech Recognition Systems in Noisy Environments

We analyze the impact of speaker adaptation in end-to-end automatic speech recognition models based on transformers and wav2vec 2.0 under different noise conditions. By including speaker embeddings obtained from x-vector and ECAPA-TDNN systems, as well as i-vectors, we achieve relative word error rate improvements of up to 16.3% on LibriSpeech and up to 14.5% on Switchboard. We show that the proven method of concatenating speaker vectors to the acoustic features and supplying them as auxiliary model inputs remains a viable option to increase the robustness of end-to-end architectures. The effect on transformer models is stronger, when more noise is added to the input speech. The most substantial benefits for systems based on wav2vec 2.0 are achieved under moderate or no noise conditions. Both x-vectors and ECAPA-TDNN embeddings outperform i-vectors as speaker representations. The optimal embedding size depends on the dataset and also varies with the noise condition.
H<sub>EVAL</sub>: A New Hybrid Evaluation Metric for Automatic Speech Recognition Tasks

Many studies have examined the shortcomings of word error rate (WER) as an evaluation metric for automatic speech recognition (ASR) systems. Since WER considers only literal word-level correctness, new evaluation metrics based on semantic similarity such as semantic distance (SD) and BERTScore have been developed. However, we found that these metrics have their own limitations, such as a tendency to overly prioritise keywords. We propose H <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EVAL</inf> , a new hybrid evaluation metric for ASR systems that considers both semantic correctness and error rate and performs significantly well in scenarios where WER and SD perform poorly. Due to lighter computation compared to BERTScore, it offers 49 times reduction in metric computation time. Furthermore, we show that H <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EVAL</inf> correlates strongly with downstream NLP tasks. Also, to reduce the metric calculation time, we built multiple fast and lightweight models using distillation techniques without significant reduction in performance.
GRAFRAUD: Fraud Detection using Graph Databases and Neural Networks

The issue of fraud has become a significant concern for many companies, particularly in the finance sector, but the traditional methods of detecting fraud are no longer adequate. Innovative technologies are necessary to identify complex fraudulent activities, and RedisGraph, a high-performance graph database, may offer a solution. With the assistance of neural networks, RedisGraph can accurately and efficiently detect fraudulent transactions in vast and intricate environments. Companies typically use a combination of Python and Oracle Databases to design fraud detection systems. which provide robust data management and real time AI processing capabilities. These technologies allow to create fraud detection systems that can determine fraudulent activities in real-time. But according to advancements of fraud methods only using of these systems not efficient nowadays. This article presents a proof of concept based on an essential use case of RedisGraph-powered neural networks in detecting financial fraud. It demonstrates the value of carefully employing Python and Oracle Database to construct and deploy real-time systems that can efficiently detect fraudulent activities.
Swarm Bug Algorithms for Path Generation in Unknown Environments

In this paper, we consider the problem of a swarm traveling between two points as fast as possible in an unknown environment cluttered with obstacles. Potential applications include search-and-rescue operations where damaged environments are typical. We present swarm generalizations, called SwarmCom, SwarmBugl, and SwarmBug2, of the classical path generation algorithms Com, Bugl, and Bug2. These algorithms were developed for unknown environments and require low computational power and memory storage, thereby freeing up resources for other tasks. We show the upper bound of the worst-case travel time for the first agent in the swarm to reach the target point for SwarmBugl. For SwarmBug2, we show that the algorithm underperforms in terms of worst-case travel time compared to SwarmBugl. For SwarmCom, we show that there exists a trivial scene for which the algorithm will not halt, and it thus has no performance guarantees. Moreover, by comparing the upper bound of the travel time for SwarmBugl with a universal lower bound for any path generation algorithm, it is shown that in the limit when the number of agents in the swarm approaches infinity, no other algorithm has strictly better worst-case performance than SwarmBugl and the universal lower bound is tight.
Residual Connections Improve Prediction Performance

Click-through rate (CTR) prediction is a critical task in online advertising and recommendation systems, where the depth and complexity of learning models have become increasingly challenging. This paper addresses the challenges associated with CTR prediction by introducing residual connections to enhance CTR prediction models. In this paper, we investigate the integration of residual connections into CTR prediction models. Experiments involve the application of plain MaskNet and MaskNet enhanced with residual connections on benchmark datasets from both the company and Avazu. Our findings demonstrate that residual connections can be effectively integrated into CTR prediction models, increasing the AUC by 0.77% and decreasing the loss by 2.8%.
Sinhala Sign Language Learning System for Hearing Impaired Community

This research aims to develop a comprehensive system for Sinhala Sign Language (SSL) that includes a learning system, dynamic sign detection, audio/video to sign conversion, and vocal training. SSL plays a crucial role in facilitating communication for individuals who are deaf or hard of hearing in Sri Lanka. The learning system provides a platform for learning SSL and includes a text-to-sign language interpreter. The dynamic sign detection system uses computer vision techniques to identify and interpret dynamic signs accurately. The audio/video to sign conversion system bridges the gap between spoken language and SSL by converting auditory information into visual representations. The vocal training system focuses on enhancing the vocal skills of cochlear implanted children. This research contributes to the development of effective communication and language skills for SSL users.
Helpchain: A Blockchain based Disaster Management System

The motivation of this study is to design a blockchain-based, decentralized, and trustworthy disaster management system. The proposed system provides an inventory of residents in the disaster areas such as earthquake zones. This information may be used by the relief agencies in case of natural disasters to track the people living in the area and initiate a rescue operation if needed. The system involves a mobile application for individual users to register and indicate their status, blockchain network to keep the data, and a desktop application to be used by disaster relief agencies to manage the whereabouts of the possible victims and organize their operations. We have built and tested a working system with Sepolia Blockchain (Ethereum Test Network) with test users, but a specific blockchain network designed for this purpose should be implemented for the project to be more feasible.
Leveraging Transformer-based Language Models for Enhanced Service Insight in Tourism

Customer feedback is a valuable resource for enhancing customer experience and identifying areas that require improvement. Utilizing user insights allows a tourism company to identify and address problematic points in its service delivery, provide feedback to partner companies regarding their product offerings, and even reconsider agreements by incorporating these opinions when curating their product portfolio. Setur implemented a systematic approach to collecting customer feedback by distributing "after-stay surveys'' to its customers via email following the completion of the agency services provided. Guest answers to open-ended questions that gather opinions about travel experience are analyzed by four tasks: user intention for answering, the sentiment of the review, subjects touched upon, and whom it concerned. For these tasks, transformer-based natural language processing (NLP) models BERT, DistilBERT, RoBERTa, and Electra are fine-tuned to classify customer reviews. Based on the test results, it is observed that best practices could be gathered using Bert. In addition, we showed that different insights can be obtained from text comments made for two hotels in Aydın, Türkiye. Some users made complaints using neutral sentences. In some cases, people gave high scores to the numerical rating questions, but their open-ended questions could have a negative meaning.
Robust End-to-End Diarization with Domain Adaptive Training and Multi-Task Learning

Due to the scarcity of publicly available diarization data, the model performance can be improved by training a single model with data from different domains. In this work, we propose to incorporate domain information to train a single end-to-end diarization model for multiple domains. First, we employ domain adaptive training with parameter-efficient adapters for on-the-fly model reconfiguration. Second, we introduce an auxiliary domain classification task to make the diarization model more domain-aware. For seen domains, the combination of our proposed methods reduces the absolute DER from 17.66% to 16.59% when compared with the baseline. During inference, adapters from ground-truth domains are not available for unseen domains. We demonstrate our model exhibits a stronger generalizability to unseen domains when adapters are removed. For two unseen domains, this improves the DER performance from 39.91% to 23.09% and 25.32% to 18.76% over the baseline, respectively.
MUST: A Multilingual Student-Teacher Learning Approach for Low-Resource Speech Recognition

Student-teacher learning or knowledge distillation (KD) has been previously used to address data scarcity issue for training of speech recognition (ASR) systems. However, a limitation of KD training is that the student model classes must be a proper or improper subset of the teacher model classes. It prevents distillation from even acoustically similar languages if the character sets are not same. In this work, the aforementioned limitation is addressed by proposing a MUltilingual Student-Teacher (MUST) learning which exploits a posteriors mapping approach. A pre-trained mapping model is used to map posteriors from a teacher language to the student language ASR. These mapped posteriors are used as soft labels for KD learning. Various teacher ensemble schemes are experimented to train an ASR model for low-resource languages. A model trained with MUST learning reduces relative character error rate (CER) up to $9.5 \%$ in comparison with a baseline monolingual ASR.
Thai-Dialect: Low Resource Thai Dialectal Speech to Text Corpora

We release a speech-to-text benchmark dataset containing 10 Thai dialects that cover different regions of Thailand. Our corpora consists of the standard dialect, Thai-central (THA); the northern dialects (Khummuang (NOD), Nan (KHB) and Yno (YNO)); the northeastern dialects (Korat (TTS), Khmer (KXM) and Laos (TTS)); and the southern dialects (Krabi (SOU), Pattani (MFA) and Phangnga (SOU)). All transcriptions are based on the Thai writing standard. We constructed baseline models by fine-tuning from self-supervised pre-trained models. Results show that multilingual/multidialectal systems outperform monolingual ones, and different dialect combinations can affect the performance of multilingual/multidialectal training.
Audio-Adapterfusion: A Task-Id-Free Approach for Efficient and Non-Destructive Multi-Task Speech Recognition

Adapters are an efficient, composable alternative to full fine-tuning of pre-trained models and help scale the deployment of large ASR models to many tasks. In practice, a task ID is commonly prepended to the input during inference to route to single-task adapters for the specified task. However, one major limitation of this approach is that the task ID may not be known during inference, rendering it unsuitable for most multi-task settings. To address this, we propose three novel task-ID-free methods to combine single-task adapters in multi-task ASR and investigate two learning algorithms for training. We evaluate our methods on 10 test sets from 4 diverse ASR tasks and show that our methods are non-destructive and parameter-efficient. While only updating 17 % of the model parameters, our methods can achieve an 8 % mean WER improvement relative to full fine-tuning and are on-par with task-ID adapter routing.
HIGNN-TTS: Hierarchical Prosody Modeling With Graph Neural Networks for Expressive Long-Form TTS

Recent advances in text-to-speech, particularly those based on Graph Neural Networks (GNNs), have significantly improved the expressiveness of short-form synthetic speech. However, generating human-parity long-form speech with high dynamic prosodic variations is still challenging. To address this problem, we expand the capabilities of GNNs with a hierarchical prosody modeling approach, named HiGNNTTS. Specifically, we add a virtual global node in the graph to strengthen the interconnection of word nodes and introduce a contextual attention mechanism to broaden the prosody modeling scope of GNNs from intra-sentence to inter-sentence. Additionally, we perform hierarchical supervision from acoustic prosody on each node of the graph to capture the prosodic variations with a high dynamic range. Ablation studies show the effectiveness of HiGNN-TTS in learning hierarchical prosody. Both objective and subjective evaluations demonstrate that HiGNN-TTS significantly improves the naturalness and expressiveness of long-form synthetic speech <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> . <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Speech samples: https://dukguo.github.io/HiGNN-TTS/
Generative Linguistic Representation for Spoken Language Identification

Effective extraction and application of linguistic features are central to the enhancement of spoken Language IDentification (LID) performance. With the success of recent large models, such as GPT and Whisper, the potential to leverage such pre-trained models for extracting linguistic features for LID tasks has become a promising area of research. In this paper, we explore the utilization of the decoder-based network from the Whisper model to extract linguistic features through its generative mechanism for improving the classification accuracy in LID tasks. We devised two strategies - one based on the language embedding method and the other focusing on direct optimization of LID outputs while simultaneously enhancing the speech recognition tasks. We conducted experiments on the large-scale multilingual datasets MLS, VoxLingua107, and CommonVoice to test our approach. The experimental results demonstrated the effectiveness of the proposed method on both in-domain and out-of-domain datasets for LID tasks.