Reviewerly/exHarmony huggingface dataset:
Language modeling for spontaneous speech recognition based on disfluency labeling and generation of disfluent text

Disfluencies in spontaneous speech such as fillers and hesitations are major causes of automatic speech recognition (ASR) errors. In our previous work, we proposed a "disfluency labeling" method allows an end-to-end (E2E) ASR model to recognize such disfluencies as symbolized recognition targets. This study proposes a method of further improving spontaneous speech recognition by integrating a language model (LM) that can predict disfluencies as pre-defined symbols. First, we fine-tuned a pre-trained Bidirectional Encoder Representations from Transformers (BERT) to predict word boundaries where disfluency symbols ("#" for a filler and "@" for a hesitation) should be inserted within a corpus of normal written text. We then used it to generated a large corpus of disfluent text. Finally, we trained an LM to predict the disfluency symbols with the BERT-generated corpus, and integrated the LM into an E2E ASR model using Shallow Fusion. Our experimental results show that the LM trained using the generated disfluent text corpus improved disfluency prediction and, as a result, spontaneous speech recognition performance.
A Lightweight Speaker Verification Model For Edge Device

In this paper, we investigate lightweight models for automatic speaker verification (ASV) that can be feasibly implemented on edge devices. The proposed model architecture includes an improved and re-parameterized VGG-based front-end stem (RepVGG) and an ECAPA-TDNNLite-like backbone. We refer the proposed model by IM ECAPA Rep-TDNNLite. We further apply the decoupled knowledge distillation approach to improve the performance of the lightweight model. On the open VoxCeleb1-O test set, the proposed IM ECAPA Rep-TDNNLite achieves 1.76% equal error rate (EER) and 0.13 minimum decision cost function (minDCF) with 1.31M model parameters and 1.45G FLOPs. The performance can be improved to 1.66% EER and 0.12 minDCF by a different RepVGG front-end with 1.46M parameters and 2.67G FLOPs. Thus, the proposed light-weight model is capable of providing good performance on edge devices with limited computation resources.
A Biased Mixed-Precision Convolution Engine for Hardware-Efficient Computational Imaging CNN

Mixed-precision networks have demonstrated significant advantages in CNN acceleration for edge inference. They can achieve high throughput and energy efficiency while maintaining quality in resource-constrained environments. However, existing high-performance accelerators for bit-flexible computation suffer from significant hardware overheads due to the extra sign extension in bit-level decomposition, severely confining the performance enhancement in mixed-precision networks. In this paper, we aim to facilitate efficient inference for mixed-precision networks in computational imaging applications. First, we propose a biased multiplier to optimize hardware performance for bit-flexible architectures. Unlike the conventional bit-level decomposition, the proposed biased representation (BR) avoids sign extension to enhance hardware efficiency. On top of that, we build 4/8-bit biased mixed-precision networks and convolution engines with highly-parallel eCNN accelerator for evaluation. We implement convolution engines with three types of multipliers, uniform-precision, sign-extended, and biased multipliers, on TSMC 40 nm technology to support advanced super-resolution and denoising at 4K UHD 30fps, FHD 60fps and FHD 30fps. Compared to the sign-extended counterpart, synthesis results show that our biased mixed-precision convolution engine can provide area overhead and energy reduction by 50%. At the same level of image quality, it can also deliver 33% energy reduction on average over the uniform-precision method.
Generative Adversarial Network-Based Frame Interpolation with Multi-Perspective Discrimination

Video frame interpolation plays an important role in video applications. This paper presents a new deep learning-based frame interpolation model under the generative adversarial networks (GANs) framework. The devised generator cascades three multi-scale sub-modules to capture coarse-to-fine visual characteristics. Furthermore, the generator is trained with an adequate loss function consisting of two adversarial and four reconstruction losses. Additional to the conventional discriminator that derives the spatial discrepancy, a new discriminator is employed to strengthen the temporal consistency among adjacent frames. The proposed method not only inherits the merit of GANs in processing speed but also produces high-quality video frames. Experimental results show that the proposed method achieves the most favorable image quality in PSNR and SSIM with a considerably short runtime compared to other state-of-the-art frame interpolation approaches.
Streaming End-to-End ASR Using CTC Decoder and DRA for Linguistic Information Substitution

This study proposes a method of streaming automatic speech recognition (ASR) for Japanese speech using domain adaptation. The development of ASR technology has led to its employment in various fields, however several issues, such as the decline recognition accuracy which occurs due to domain mismatches between training and test data. In particular, the accuracy of online speech recognition, also known as streaming ASR, which is becoming popular in real-world applications, also suffers from this issue. To realize domain-adapted streaming ASR, in this study we utilize a method of domain adaptation known as the Density Ratio Adaptation (DRA), which uses Language Models (LMs) trained with extensive text data. Our proposed method employs DRA for streaming ASR of Japanese speech, in a model using a CTC decoder for recognition. To facilitate streaming ASR processing, the CTC decoder successively replaces linguistic information on a frame-by-frame basis, obtaining recognition results through greedy searches. Considering CTC’s assumption of conditional independence, we used unigram LMs for the subtraction and addition of language information when using DRA, during which selected linguistic information is replaced. Experimental results demonstrate that the proposed method enhances speech recognition accuracy when there is data domain mismatch.
Preemptive Image Protection against Steganography

Image steganography plays a vital role in the realm of information hiding, involving the discreet embedding of confidential data within an image. However, the misuse of this technology by malicious actors on social media platforms poses a significant threat to public security. Presently, the primary defense against image steganography is steganalysis technology, which effectively distinguishes steganographic images from regular ones by detecting subtle modifications. Nevertheless, there is considerable room for improvement in existing steganalysis techniques, especially when dealing with small payloads. In this paper, we propose a proactive strategy to enhance the detection of steganographic images with small payloads. Specifically, as the image is captured, we apply a meticulous-designed and imperceptible mask to it. This mask is aimed at increasing the success rate of detection. We have developed three convolutional neural networks (CNNs) to facilitate this proactive image processing: preemptive-CNN, steganography network (Stego-CNN), and steganalysis discriminator. During the training process, Stego-CNN remains unaltered, while preemptive-CNN and the discriminator are jointly trained to ensure that the applied mask enhances the detection accuracy. Experimental results demonstrate the effectiveness of our proposed approach in enhancing the detection accuracy, all without compromising the visual quality of the images.
Integrity Heuristic as Tie-Breaking for Classical Task Planning

There is a growing interest in automated planning (also known as AI planning) as a promising technique for autonomous agents, such as robots, to achieve a given goal (mission) by devising their own action plans. In AI planning as heuristic search, informative and fast heuristics are crucial for performance. Recently, AI planning systems using state-of-the-art heuristics have made remarkable progress compared to the past, mostly achieved by using a combination of multiple heuristics. In this paper, we propose a novel search heuristic called the integrity heuristic, which is based on the popular FF heuristic. The integrity heuristic is used in conjunction with the FF heuristic through a tie-breaking mechanism to improve the performance of planners. In this paper, we present the theoretical grounds for defining the integrity of states and empirically verify our integrity heuristic by implementing a greedy best-first search algorithm using the FF heuristic and our integrity heuristic, and show that the proposed integrity heuristic leads to performance improvement.
An Image Registration Technique for Brain MR Images Using Linear Transform by 3DCNN

In this paper, we focus on brain atrophy in Alzheimer's disease and propose a 3D image registration method of brain MR images for the purpose of calculating the atrophy rate by temporal subtraction technique. The nonlinear 3D image registration model of Stergios et al. is recombined into a linear image registration model, and image registration is performed with isotropic affine and rigid body transformations. In addition, the accuracy is improved by the Instance-specific Optimization, which performs the optimization process for each case. For the training of the model, we used pseudo-data obtained by arbitrarily linear transformation of the ADNI brain MR images. Synthetic data and actual ADNI time series data (AD, MCI) were used for the test. The experimental results show that the addition of the optimization process improves the normalized cross-correlation (NCC) by more than 0.1. The loss function of the model is compared between Grid loss and MSE, and it is confirmed that MSE is effective for NCC. Furthermore, the output images confirm the usefulness of the image registration of the skull in the linear transformation.
Robust Measurements for RF Fingerprinting with Constellation Patterns of Radiated Waveforms

We introduce an RF fingerprint for non-destructive cellular device identification. The new proposed fingerprinting algorithm is a data-driven technique based on a singular value decomposition of a user equipment's symbol-constellation points. We explore the effectiveness of the fingerprint technique with a test set of real devices and show experimentally that this fingerprint is robust to device positioning errors and measurement noise.
Potential of Unsupervised Deep Learning for Detection of EM Side-Channel Attacks

The growing concern over side-channel emissions and their impact on the confidentiality of computer applications and systems has led to increased attention towards the vulner-ability of the backside of the Si substrate. This research paper proposes an innovative potential approach to bolster hardware security and mitigate vulnerabilities against electromagnetic side-channel attacks. The approach involves the fusion of Multi-physics simulation and deep learning techniques to accurately identify the most susceptible locations for side-channel leakage on the backside of silicon substrates within integrated circuits. By employing contextual feature extraction and an attention mechanism, the study aims to enhance chip-level security analysis and automate the detection of points of interest that are prone to side-channel attacks. The presents potential method overcomes limitations present in previous works, leading to improved quality and accuracy of the results.
Effectiveness of Model and Data Scale Contrastive Learning in Non-IID Federated Learning

All involved clients are guaranteed data privacy in a collaborative machine learning environment via Federated Learning. The lack of generalization in local client models brought on by data heterogeneity, however, is one of Federated Learning ’s major challenges that lead to slow model convergence and communication latency. In this study, we use contrastive learning techniques that have been shown effective in both centralized and federated learning environments. For local models to have a stronger capacity for generalization, we suggest adopting contrastive loss at the model and data scales. Utilizing the CIFAR10 and CIFAR-100 datasets, we assess and contrast our suggested strategy with other industry standard approaches.
LSR++: An Efficient and Tiny Model for Image Super-Resolution

A single-image lightweight super-resolution (LSR) method was presented in our earlier work. LSR predicts the residual image between the interpolated low-resolution (ILR) and high-resolution (HR) images using a self-supervised framework. A further enhanced version of LSR, called LSR++, is proposed in this work. The improvement of LSR++ over LSR primarily lies in a new patch alignment scheme, where the spatially geometrical property of patches is exploited. Patches can be categorized into two main classes through alignment. Then, one regressor is needed for each class. Compared with LSR, LSR++ lowers the computational complexity from 3.83K to 1.71K FLOPs/pixel and the model size from 770K to 200K parameters while preserving close visual performance measured by PSNR/SSIM. LSR++ offers an attractive super-resolution method for mobile applications.
A Reversible Image Processing Method for Color Tone Control Using Data Hiding

Reversible image processing methods have been actively researched by using data hiding techniques. In these methods, additional information for reversibility should be embedded into the processed image. Their main targets for image processing are luminance and saturation. In this paper, we focus on tone control that specifically enhances blue or red tone with reversibility. Through our simulations, we prove that our method can effectively enhance color tones and the data embedding process does not severely affect the quality of output images.
Low-Light Image Enhancement via Distillation of NIR-to-RGB Conversion Knowledge

In this paper, we propose a novel approach for low-light image enhancement via distillation of NIR-to-RGB conversion network’s knowledge. NIR images contain abundant detail information that is independent of lighting conditions. We leverage the knowledge of the NIR-to-RGB conversion network, which effectively preserves detail information from NIR while generating accurate colors, as it shares similarities with low-light image enhancement. By using the rich detail information provided by the NIR-to-RGB conversion for low-light image enhancement, we are able to reduce color distortion and noise while preserving detail information. Our experimental results show the superior performance of our proposed network.
Speech Synthesis Using Ambiguous Inputs From Wearable Keyboards

This paper proposes a new application in speech communication using text-to-speech (TTS), and the goal is to enable dysarthria, articulation disorder, or persons who have difficulty in speaking to communicate anywhere and anytime using speech to express their thoughts and feelings. To achieve this goal, an input method is required. Thus, we propose a new text-entry method based on three concepts. First, from an easy-to-carry perspective, we used a wearable keyboard that inputs digits from 0 to 9 in decimal notation according to 10-finger movements. Second, from a no-training perspective, users input sentences in a way of touch typing using the wearable keyboard. Following this method, we obtained a sequence of numbers corresponding to the sentence. Third, a neural machine translation (NMT) method is applied to estimate texts from the sequence of numbers. The NMT was trained using two datasets; one is a Japanese-English parallel corpus containing 2.8 million pairs of sentences, which were extracted from TV and movie subtitles, while the other is a Japanese text dataset containing 32 million sentences, which were extracted from a question-and-answer platform. Using the model, phonemes and accent symbols were estimated from a sequence of numbers. Thus, the result accuracy in symbol levels was 91.48% and 43.45% of all the sentences were completely estimated with no errors. To subjectively evaluate feasibility of the NMT model, a two-person word association game was conducted; one gave hints using synthesized speech that is generated from symbols estimated by NMT, while the other guessed answers. As a result, 67.95% of all the quizzes were correctly answered, and experiment results show that the proposed method has the potential for dysarthria to communicate with TTS using a wearable keyboard.
Design of Interactive System for Acupoint Analysis Based on Augmented Reality

In this paper, we utilize the edge computing of Jetson Nano J1010 development board and employ MediaPipe technology and acupoint mapping algorithm to display acupoints on the screen. By using a custom deep neural network (DNN) hand recognition model, we achieve augmented reality (AR) interaction with acupoints. Users can view the corresponding acupoint locations on the body and acupoint descriptions through gestures. Additionally, we develop a mobile app that enables interaction with the system via Bluetooth technology. Users can select symptoms they wish to understand, and the system will display acupoints that can alleviate those symptoms. Our team also develops an AI-based diagnostic system using the open-source conversational AI Rasa, which utilizes natural language understanding (NLU) to determine the user's intents from input on the mobile app. When the intent is related to medical content, the information is passed to a BERT model for diagnosis analysis, determining possible diseases or symptoms and providing corresponding recommended acupoints for symptom relief. This aids users in making preliminary judgments about their own conditions. All AI diagnostic information is transmitted to the server-side, where physicians can view user information through a website. By visualizing the data through charts, physicians can quickly understand the user's condition and further save medical resources. This acupoint-based healthcare system offers a safe, convenient, and efficient solution for health management, particularly valuable for individuals lacking experience and medical knowledge.
Low-light is More Than Darkness: An Empirical Study on Illumination Types and Enhancement Methods

Low-light images challenge both human perception and computer vision algorithms. Despite notable progress in this field, there are still various gaps that are yet to be investigated, such as the significance of low-light illumination characteristics towards image enhancement and object classification. Therefore, this paper details various analyses to study this phenomenon and provide insights for future developments of algorithms and solutions. Specifically, comparative analysis was done to investigate human and machine perception towards "low-light types", followed by empirical studies on the effect of illumination types towards state-of-the-art image enhancement quality and also their pre-processing capability for downstream task, namely object classification. It is found that illumination types significantly influences the performance of enhancement algorithms that tend to cater for a "general" type of low-light illumination. This lack of illumination type awareness therefore leads models to perform well in certain conditions, but severely underperforms in others. Thus, it is imperative for upcoming works to incorporate such illumination information for potential breakthroughs in this area.
Augmented-Ensemble TD3: Overcoming the Shackles of Constant Action Delay

Reinforcement Learning has experienced significant advances in various domains. However, delayed feedback in RL environments poses challenges due to the violation of the Markovian property. In this paper, we propose an approach to address the issues of Markov Decision Process(MDP) with delayed feedback. The proposed approach, called “Augmented-Ensemble Twin-Delayed Deep Deterministic Policy Gradient(TD3),” aims to mitigate the performance degradation caused by delayed feedback.
Continuous Sign Language Recognition Using MediaPipe

According to WHO, more than 5% of the world’s population suffer from hearing loss. They estimate that by 2050, the figure will reach over 700 million people. Therefore, the need for a sign language translation technique is essential. However, that has not been an effective approach to recognize dynamic sign language in real-time. This paper proposed the new method to continuous translate sign language recognition in real-time using Convolutional Neural Network and Long-Short term Memory. With the help of extracting coordinates of body joints from MediaPipe, it reaches an overall accuracy up to 99.5%.
Multi-granularity Semantic and Acoustic Stress Prediction for Expressive TTS

Stress, as the perceptual prominence within sentences, plays a key role in expressive text-to-speech (TTS). It can be either the semantic focus in text or the acoustic prominence in speech. However, stress labels are always annotated by listening to the speech, lacking semantic information in the corresponding text, which may degrade the accuracy of stress prediction and the expressivity of TTS. This paper proposes a multi-granularity stress prediction method for expressive TTS. Specifically, we first build Chinese Mandarin datasets with both coarse-grained semantic stress and fine-grained acoustic stress. Then, the proposed model progressively predicts semantic stress and acoustic stress. Finally, a TTS model is adopted to synthesize speech with the predicted stress. Experimental results on the proposed model and synthesized speech show that our proposed model achieves good accuracy in stress prediction and improves the expressiveness and naturalness of the synthesized speech.
An Multi-evidence Fusion Based on C-Distance with Uncertain Reasoning for Classification

Dempster-Shafer (D-S) evidence theory has been extensively used to handle uncertain information. However, evidence fusion remains a challenging problem because counter-intuitive results would occur when combining highly conflicting evidence. To address this issue, we propose a weighted fusion model with multi-evidence uncertainty reasoning based on a novel similarity quantification function. Specifically, we first define a dissimilarity representation function C-distance. This function considers two factors, the angle and the norm of the evidence vector, so that the similarity between evidences can be described more comprehensively. In particular, unlike previous weights defined as the ratio between distances, we use a nonlinear function to fit the weight and the C-distance, which expands the learning space of weights. Based on the above work, we propose a weighted fusion framework. Experiments on real datasets illustrate the excellent performance of the proposed method in information fusion.
FID-RPRGAN-VC: Fréchet Inception Distance Loss based Region-wise Position Normalized Relativistic GAN for Non-Parallel Voice Conversion

Voice conversion (VC) is the speech-to-speech (STS) synthesis process that converts the vocal identity of a source speaker to a target speaker by keeping the linguistic content unaltered. In recent years, VC research has been explored using generative adversarial network (GAN) models. However, a substantial difference exists between the real and the state-of-the-art (SOTA) VC model-generated speech samples as far as naturalness is concerned. This work proposes an improved GAN model for non-parallel VC to enhance the naturalness of the generated speech samples. The improved GAN model is integrated with a region-wise positional normalization technique in the generator, a relativistic mechanism-based discriminator, and a Fréchet inception distance (FID) based loss function. We tested the proposed model on VCC 2018, CMU Arctic, and a dysarthric speech dataset. The experimental results revealed the superiority of the proposed FID-RPRGAN-VC model over the SOTA MaskCycleGAN-VC model.