Reviewerly/exHarmony huggingface dataset:
Multi-accent pronunciation assessment based on domain adversarial training

Second language learners are usually influenced by their first languages and thus produce varied and complex accents difficult for pronunciation assessment. Meanwhile, obtaining sufficient human-labeled data for accented speech is tedious and costly, which limits the robustness and accuracy of pronunciation assessment. In this paper, we propose an end-to-end (E2E) method for multi-accent pronunciation assessment. We utilize a cross-lingual pre-trained acoustic model to ensure the discriminative capability of feature representations for the assessment of different accents. Moreover, to improve the robustness of pronunciation assessment on low-resource or unannotated accented speech, we employ domain adversary training to make representations accent-invariant for assessment. Experimental results show the proposed method outperforms the baselines across multiple accents of English, including Chinese, Korean, German, and Indonesian, in the Pearson correlation coefficient (PCC). When assessing unseen English pronunciation with Japanese accents, it also shows superiority to other baselines.
Naval Wargame Prototyping: Multiplayer Real-Time Strategy Game Simulation Using Unreal Engine

We propose an integration framework for a wargame prototype using the modern game engine - Unreal Engine. This wargame functions as a simulation tool for strategy training, strategy testing, and simulating enemy forces like warships, aircraft, and weapons. Existing wargames come in proprietary and free versions. The former is often expensive and exclusive due to security reasons, while developing a functional wargame is complex, requiring various technological components such as a physics system and artificial intelligence (AI). To overcome these challenges, we propose a rapid prototype using Unreal Engine. This approach leverages advanced technology and ensures the prototype is ready for future upgrades when new versions of the game engine are released. We evaluate the prototype's system capabilities and expert assessments.
Analysis of the Interaction Effect on Pruning and Transfer Learning in Model Training

Pruning and transfer learning are extensively applied in developing convolutional neural network algorithms, and at times, they are concurrently employed as part of the training model strategy. However, the existing literature has only focused on studying these techniques separately, with no research focusing on the interaction effect between the two technologies. Therefore, we designed several scenarios to test and explore this issue in different target tasks. The experimental results showed significant performance differences among various strategies employed for the same task.
Residual, Mixer, and Attention: The Three-way Combination for Streaming Wake Word Detection Framework

Speech interactions and digital assistants rely on effective wake word detection models to identify predefined words. In this study, we address the need for an efficient wake word detection model and propose a "three-way residual separable convolution network" (3W-ResSC) inspired by human multi-perspective learning. Our 3W-ResSC model combines three information: residual learning (ResNet), point-wise pattern, and depth-wise patterns from a CNN mixer (Mi). In addition, to conclude and emphasize information, our independent multi-view attention (iMVA) is proposed based attention mechanism (At). Based on ResNet, Mi, and At, we propose three different combinations for our model; ResNet+Mi, ResNet+At, and ResNet+Mi+At. By leveraging the strengths of these three combinations, our combinations are investigated to find the outstanding performance and pattern. To evaluate our model, we conduct experiments on multiple datasets, including Gowajee, HeyFireFox, HeySnips, and GSCv2. The results demonstrate that our 3W-ResSC outperforms baseline models in terms of equal error rate (EER), Matthews correlation coefficient (MCC), and false rejection rate (FRR) in multi-class classification. Additionally, we introduce a wake word detection framework specifically designed for streaming processing. This framework leverages an independent speech window, resembling a buffer-like streaming production process. Overall, our proposed 3W-ResSC model and wake word detection framework offer significant advancements in wake word detection, showcasing improved performance and efficiency, which discuss in various datasets and training sizes. These findings contribute to the development of more effective wake word detection and speech processing when using ResNet, Mi, and At in deep learning.
Evaluating Methods for Ground-Truth-Free Foreign Accent Conversion

Foreign accent conversion (FAC) is a special application of voice conversion (VC) which aims to convert the accented speech of a non-native speaker to a native-sounding speech with the same speaker identity. FAC is difficult since the native speech from the desired non-native speaker to be used as the training target is impossible to collect. In this work, we evaluate three recently proposed methods for ground-truth-free FAC, where all of them aim to harness the power of sequence-to-sequence (seq2seq) and non-parallel VC models to properly convert the accent and control the speaker identity. Our experimental evaluation results show that no single method was significantly better than the others in all evaluation axes, which is in contrast to conclusions drawn in previous studies. We also explain the effectiveness of these methods with the training input and output of the seq2seq model and examine the design choice of the non-parallel VC model, and show that intelligibility measures such as word error rates do not correlate well with subjective accentedness. Finally, our implementation is open-sourced to promote reproducible research and help future researchers improve upon the compared systems.
AVATAR: Robust Voice Search Engine Leveraging Autoregressive Document Retrieval and Contrastive Learning

Voice, as input, has progressively become popular on mobiles and seems to transcend almost entirely text input. Through voice, the voice search (VS) system can provide a more natural way to meet user’s information needs. However, errors from the automatic speech recognition (ASR) system can be catastrophic to the VS system. Building on the recent advanced lightweight autoregressive retrieval model, which has the potential to be deployed on mobiles, leading to a more secure and personal VS assistant. This paper presents a novel study of VS leveraging autoregressive retrieval and tackles the crucial problems facing VS, viz. the performance drop caused by ASR noise, via data augmentations and contrastive learning, showing how explicit and implicit modeling the noise patterns can alleviate the problems. A series of experiments conducted on the Open-Domain Question Answering (ODSQA) confirm our approach’s effectiveness and robustness in relation to some strong baseline systems.
Improved One-class Learning for Voice Spoofing Detection

Most of the studies on one-class learning for voice spoofing detection have poor generalization ability for unknown attacks and lack flexibility for different tasks. To address these problems, we propose an improved one-class softmax (IOC-Softmax) by reformulating the scale factor of one-class softmax (OC-Softmax) into two parts such that it down-weights the loss assigned to bona fide samples. Our IOC-Softmax is a novel dynamically scaled OC-Softmax, which can select different scale factors for different tasks, so that it can prevent a large number of bona fide samples from overwhelming the anti-spoofing model during training. To verify the effectiveness of the proposed IOC-Softmax, we perform numerous experiments on the ASVspoof 2019 dataset. The results show that our IOC-Softmax achieves better performance when compared with OC-Softmax.
A Study on Eliminating Biased Node in Federated Learning

Federated Learning (FL) is a method to perform a deep learning technique with distributed nodes without aggregating their individually collected data. Each node can keep its own dataset secret, and a central server aggregates the updated parameters in a deep learning model from each node. However, it is reported that FL is vulnerable to poisoning attacks on training data. In this study, we propose a method to exclude biased nodes by analyzing the statistical characteristics of the weight parameters sent by each node to the central server. In the computer simulation, we evaluate the detection accuracy of biased node, and conduct discussions.
Exploring Residual Cepstral Features for Spoken Language Identification

In this paper, we introduce a Spoken Language Identification (SLI) system, which uses Linear Prediction (LP) residual, decomposed with linear filterbank and followed by log and Discrete Cosine Transform (DCT). This feature set is called Linear Frequency Residual Cepstral Coefficients (LFRCC). Mel Frequency Cepstral Coefficients (MFCC) and Linear Frequency Cepstral Coefficients (LFCC) were also extracted and kept as baseline features. The Attentive Time Delay Neural Network (Attentive - TDNN) was employed to evaluate the efficiency, relevancy, and sufficiency of said LFRCC features. For experiments, a computationally challenging VoxLingua107 dataset was used. From experiments, we achieved 9.22%, 7.8%, and 6.86% EER for MFCC, LFCC, and LFRCC, respectively. Significant improvements in accuracies and equal error rate (EER) were observed when comparing the performance of MFCC and LFCC with data fusion strategies. The feature-level fusion of MFCC and LFRCC resulted in a notable enhancement of 3.16%, 2.12%, and 1.16%, accuracies compared to other combinations. Moreover, each combination of feature sets in score-level fusion has outperformed LFRCC and baselines. This paper provides a significant study on how prosodic information provided by LP residual contributes to language-specific information. We present the pitch strength analysis to understand better the contribution of LP residual in capturing language-specific information.
High-Quality Triggers Based Fragile Watermarking for Optical Character Recognition Model

Deep neural networks have demonstrated exceptional performance in numerous domains, yet they also face significant security issues. To safeguard the integrity of deep neural networks and prevent unauthorized parameter manipulation, researchers have proposed fragile model watermarking techniques. However, current methods concentrate on image classification models and are not suitable for text image recognition models. Consequently, this paper proposes a remote black-box integrity authentication method for text image recognition models. Specifically, we embed a trigger set into the model to be protected as its watermark. This trigger set is carefully constructed by adding tiny noise to meticulously selected training set samples, with each trigger set sample labeled using model-specific sentence. Utilizing the principle of information entropy theory, we employ an optimal approach to fine-tuning the model, aiming to ensure that the outputs of the trigger set exhibit a pronounced sensitivity to variations within the model. Experiments indicate that with the structural similarity between the trigger set samples and the original samples exceeding 0.9, the embedded watermark has a negligible impact on model performance. The minimum decrease in accuracy on the test set is less than 0.5%. Moreover, even after undergoing modifications limited to fine-tuning only the last two layers and altering a mere one ten-thousandth of the parameters, the model watermarking is effective in identifying such changes.
Rotation Angle Detection Using a Pilot Signal from Rotated Stego-Image

Robust watermarking methods are necessary to resist attacks. With conventional watermarking methods, watermarks are extracted and their errors are corrected regardless of the type of attack. For example, error-correcting codes and spread spectrum are used to correct watermark errors. To reduce watermark errors, embedding methods in the frequency domain, such as DCT and DFT, and the use of scale invariant feature transform have been proposed. These methods are effective for non-geometric attacks such as additive noise and distortion. However, they do not work well against geometric attacks due to the lack of synchronization. It is necessary to perform optimal synchronization for geometric attacks such as scaling, rotation, cropping, etc. We focus on the rotation attack and propose a method of estimating the rotation angle using a grid-shaped pilot signal. While suppressing the original image-derived slant, the slant of the pilot signal can be emphasized and detected. Thus, the rotation angle of a stego-image can be accurately estimated.
Energy-Efficient Continual Learning for Autonomous Driving

Our work highlighted the primary challenges of Autonomous Driving (AD), namely the Catastrophic Forgetting (CF) of previous knowledge by the AD system upon new scenario encounters. Considering the infeasible model retraining with past data given computational, power, and storage constraints on the embedded device, we proposed an experiment featuring Avalanche Continual Learning (CL) training strategies to investigate which strategies excel in this task and combine the promising ones in the hope for a more balanced and efficient trade-off between performance and energy consumption. Our experiment unprecedentedly validated the candidates against a new benchmark introducing natural distribution change and time correlation between input images. We found that although a synergy of CL strategies yields higher resistance towards CF, the slight accuracy gain is not worth the additional computation when we account for energy consumption, rendering a simple Replay strategy the best solution for the Continual Learning benchmark for Autonomous Driving: Online Continual Classification (CLAD-C). Our proposal delivers a 65.80% improvement over the baseline at our proposed accuracy-power ratio metric.
Trajectory Generation Method Based on DDP for 6-DOF Collaborative Robot

Nonlinear Model Predictive Control (NMPC) is one of the optimal control techniques. Solving for the optimal solution of a nonlinear model can be approached through three main methods, and recently, research on Differential Dynamic Programming (DDP) based on Dynamic Programming (DP) has gained significant attention. In this study, we applied DDP to generate trajectories for the RB5-850, a 6-DOF collaborative robot developed by Rainbow Robotics, and its performance was validated through simulations. The results demonstrate the effectiveness and applicability of the DDP algorithm in generating optimal trajectories for collaborative robots.
In-Air Handwriting for Chinese Character Recognition from Monocular Camera: A Deep Learning based Approach with Fingertip Detection and Virtual Strokes Elimination

This study presents a deep learning-based approach for recognizing Chinese characters written in the air. Chinese characters, which are composed of numerous strokes in a square shape, pose a challenge for recognizing real and virtual strokes when writing in the air. Existing Optical Character Recognition (OCR) models often result in inaccurate recognition due to this difficulty. Depth cameras can improve recognition accuracy, but they are more expensive than traditional optical cameras.To address these challenges, this study employs a deep learning model to track fingertips from a single optical camera. The approach incorporates an intuitive interface that distinguishes between real and virtual strokes while writing. Gesture recognition serves as commands to enable users to start and stop writing and to make corrections during input. This approach makes in-air handwriting behavior similar to writing on paper. To evaluate the effectiveness of the proposed approach, a comparative experiment was conducted to evaluate the model's performance for recognizing characters with and without virtual strokes. Eliminating virtual strokes enabled the OCR model to achieve over 90% recognition accuracy, which was 37% higher than the accuracy with virtual strokes. Moreover, the recognition accuracy of characters with virtual strokes fluctuated significantly in a stability test conducted one week later, while the input method without virtual strokes remained stable at over 90% recognition accuracy. These experimental results indicate that the proposed method can not only effectively remove virtual strokes in air writing but also make in-air handwriting easier for people.
Dictionary-driven Chinese ASR Entity Correction with Controllable Decoding

The mainstream ASR error correction system is mainly based on the encoder-decoder structure, by learning the mapping of incorrect text to correct text. Although this approach showed good results in ASR error correction, it mainly solves the spelling errors that occur at high frequencies, while struggling with low-frequency ASR entity errors, especially those entities unseen in the training set. Another downside of the structure is that it may produce undesirable hallucination entities. In this paper, we introduce a DiCODER model for Chinese ASR entity error correction. Specifically, we first retrieve relevant entities from the entity dictionary based on the similarity of the text and phonetic information and then fuse them with ASR text for joint encoding. Next, a controlled decoding strategy utilizing dynamic vocabulary is proposed to generate more reasonable results. Experimental results on three publicly available datasets demonstrate the effectiveness of our proposed method.
Decentralized Trusted Database Approach to Online Product Reviews

Nowadays, Blockchain technology can address risks by providing a decentralized database that ensures data integrity and prevents unauthorized alterations. Traditional centralized database systems are the preferred choice for organizing large collections of data due to their rapid and systematic data accessibility. However, the concentration of access authority in the hands of administrators or individuals with power introduces potential risks to data integrity. This research proposes a decentralized approach to online product reviews that integrates a centralized database with Blockchain technology. By integrating these two systems, users can access and review content data with the assurance that the information remains unaltered. The research proposes a method to verify the integrity of data stored in the centralized database and a strategy to optimize the balance between trust and gas fees when storing data on the Blockchain. The proposed approach has the potential to improve the trustworthiness and security of online product reviews. It can also help to reduce the risks of data manipulation and fraud. Our method serves to enhance the credibility of the database.
Fractal Dimension in Deep Learning

In the rapidly evolving field of deep learning, architectural models have grown increasingly complex, delivering impressive performance. However, the more complex models require more processing resources. Furthermore, it requires huge amounts of data to provide high-quality performance results. In this study, we have examined the strengths of the fractal dimension which is a powerful tool for describing self-similarity and complexity of data and for effectively reducing data dimension. Our investigation explores the methods for integrating fractal dimensions into the training of convolutional neural networks (CNNs). We assess this investigation from three key perspectives: performance, training time, and computational resource utilization.
Learning Semantic Information from Machine Translation to Improve Speech-to-Text Translation

End-to-end speech translation (ST) directly translates the source speech to the target text, following a typical encoder-decoder framework. However, it has shown that the conventional ST encoder is mainly used to extract long but locally attentive acoustic features, which may lead to a lack of global semantic features. In this work, we therefore propose to integrate a semantic decoder into the speech translation (SD-ST) model, where the semantic decoder can generate text-like features with more global semantic information analogously to the machine translation system (MT). We also investigate different strategies to ensure length consistency between text-like features and text sequences. Experimental results show that the proposed SD-ST model achieves the best BLEU score on the 40-hour subset of the Fisher Spanish English dataset and a comparable BLEU score on the MuST-C dataset. Furthermore, it is shown that the SD-ST model can even perform zero-shot ST.
Enhancing Depth Image with Richer Convolution Feature for Improved Perception

Depth cameras are widely used in various fields, particularly in robotics, for tasks such as robot path planning, object avoidance, target tracking, and autonomous driving. However, the incomplete depth sensing technology of depth cameras results in depth images with significant noise, including errors in depth values and detection failures due to environmental instability. This noise adversely affects the performance and stability of robotic applications. To address this issue, various filtering methods have been proposed, but the corrected depth images still contain residual noise. This paper utilize a deep learning-based edge detection method, Richer Convolution Feature, to accurately estimate planar regions in depth images. The RANSAC algorithm is used to enhance the planes of objects. Experimental results demonstrate the effectiveness of the proposed approach in reducing noise and improving depth image quality.
Enhancing Denoising Models Performance Through Timestep-Aware Predictor for Latent Diffusion-Based Image Generation

In various deep learning domains, transformer-based models have shown high performance, particularly in the field of generative modeling, where stable diffusion models have become prominent. In diffusion models, the process begins by applying noise to the original images using a scheduler. The denoising process is then trained to generate clean images from the noisy inputs. And the noise application step is not learned, focusing solely on training the denoising process to improve image generation. One notable denoising model is the time-embedded U-Net, which incorporates time embedding into its architecture. In this study, we propose enhancing the performance of the time-embedded denoising U-Net by introducing a predictor specifically designed for timesteps. The input images are compared based on the timestep error, while the output images are compared with a fixed timestep of 0, analyzing the timestep features of the images. By incorporating the predictor, we observed that even for unlearned timesteps, the similarity with the learned timesteps remained consistent. This modification enables the denoising U-Net to better seize the sequential flow of timesteps.