Reviewerly/exHarmony huggingface dataset:
An Analysis of Semantically-Aligned Speech-Text Embeddings

Embeddings play an important role in end-to-end solutions for multi-modal language processing problems. Although there has been some effort to understand the properties of single-modality embedding spaces, particularly that of text, their cross-modal counterparts are less understood. In this work, we study some intrinsic properties of a joint speech-text embedding space, constructed by minimizing the distance between paired utterance and transcription inputs in a teacher-student model setup, that are informative for several prominent use cases. We found that incorporating automatic speech recognition through both pretraining and multitask scenarios aid semantic alignment significantly, resulting in more tightly coupled embeddings. To analyse cross-modal embeddings we utilise a quantitative retrieval accuracy metric for semantic alignment, zero-shot classification for generalis-ability, and probing of the encoders to observe the extent of knowledge transfer from one modality to another.
Empirical Analysis of Training Strategies of Transformer-Based Japanese Chit-Chat Systems

In recent years, several high-performance conversational systems have been proposed based on the Transformer encoder-decoder model. Although previous studies analyzed the effects of the model parameters and the decoding method on subjective dialogue evaluations with overall metrics, it is not analyzed enough how the differences of fine-tuning datasets affect the user's detailed impressions. In addition, the Transformer-based approach has mostly been verified for English, not for such languages as Japanese that have large inter-language distances. In this study, we developed large-scale Transformer-based Japanese dialogue models and Japanese chit-chat datasets and examined their effectiveness. We analyzed the relationships between users' multifaceted impressions and fine-tuning datasets.
A Study on the Integration of Pre-Trained SSL, ASR, LM and SLU Models for Spoken Language Understanding

Collecting sufficient labeled data for spoken language understanding (SLU) is expensive and time-consuming. Recent studies achieved promising results by using pre-trained models in low-resource scenarios. Inspired by this, we aim to ask: which (if any) pre-training strategies can improve performance across SLU benchmarks? To answer this question, we employ four types of pre-trained models and their combinations for SLU. We leverage self-supervised speech and language models (LM) pre-trained on large quantities of un-paired data to extract strong speech and text representations. We also explore using supervised models pre-trained on larger external automatic speech recognition (ASR) or SLU corpora. We conduct extensive experiments on the SLU Evaluation (SLUE) benchmark and observe self-supervised pre-trained models to be more powerful, with pre-trained LM and speech models being most beneficial for the Sentiment Analysis and Named Entity Recognition task, respectively. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Our code and models will be publicly available as part of the ESPnet-SLU toolkit.
Streaming, Fast and Accurate on-Device Inverse Text Normalization for Automatic Speech Recognition

Automatic Speech Recognition (ASR) systems typically yield output in lexical form. However, humans prefer a written form output. To bridge this gap, ASR systems usually employ Inverse Text Normalization (ITN). In previous works, Weighted Finite State Transducers (WFST) have been employed to do ITN. WFSTs are nicely suited to this task but their size and run-time costs can make deployment on embedded applications challenging. In this paper, we describe the development of an on-device ITN system that is streaming, lightweight & accurate. At the core of our system is a streaming transformer tagger, that tags lexical tokens from ASR. The tag informs which ITN category might be applied, if at all. Following that, we apply an ITN-category-specific WFST, only on the tagged text, to reliably perform the ITN conversion. We show that the proposed ITN solution performs equivalent to strong base-lines, while being significantly smaller in size and retaining customization capabilities.
Anonymizing Speech with Generative Adversarial Networks to Preserve Speaker Privacy

In order to protect the privacy of speech data, speaker anonymization aims for hiding the identity of a speaker by changing the voice in speech recordings. This typically comes with a privacy-utility trade-off between protection of individuals and usability of the data for downstream applications. One of the challenges in this context is to create non-existent voices that sound as natural as possible. In this work, we propose to tackle this issue by generating speaker embeddings using a generative adversarial network with Wasserstein distance as cost function. By incorporating these artificial embeddings into a speech-to-text-to-speech pipeline, we outperform previous approaches in terms of privacy and utility. According to standard objective metrics and human evaluation, our approach generates intelligible and content-preserving yet privacy-protecting versions of the original recordings.
A Comprehensive Study on Self-Supervised Distillation for Speaker Representation Learning

In real application scenarios, it is often challenging to obtain a large amount of labeled data for speaker representation learning due to speaker privacy concerns. Self-supervised learning with no labels has become a more and more promising way to solve it. Compared with contrastive learning, self-distilled approaches use only positive samples in the loss function and thus are more attractive. In this paper, we present a comprehensive study on self-distilled self-supervised speaker representation learning, especially on critical data augmentation. Our proposed strategy of audio perturbation augmentation has pushed the performance of the speaker representation to a new limit. The experimental results show that our model can achieve a new SoTA on Voxceleb 1 speaker verification evaluation benchmark (i.e., equal error rate (EER) 2.505%, 2.473%, and 4.791 % for trial Vox1-O, Vox1-E and Vox1-H, respectively), discarding any speaker labels in the training phase.
Inter-KD: Intermediate Knowledge Distillation for CTC-Based Automatic Speech Recognition

Recently, the advance in deep learning has brought a considerable improvement in the end-to-end speech recognition field, simplifying the traditional pipeline while producing promising results. Among the end-to-end models, the connectionist temporal classification (CTC)-based model has attracted research interest due to its non-autoregressive nature. However, such CTC models require a heavy computational cost to achieve outstanding performance. To mitigate the computational burden, we propose a simple yet effective knowledge distillation (KD) for the CTC framework, namely Inter-KD, that additionally transfers the teacher's knowledge to the intermediate CTC layers of the student network. From the experimental results on the LibriSpeech, we verify that the Inter-KD shows better achievements compared to the conventional KD methods. Without using any language model (LM) and data augmentation, Inter-KD improves the word error rate (WER) performance from 8.85% to 6.30% on the test-clean.
Four-in-One: a Joint Approach to Inverse Text Normalization, Punctuation, Capitalization, and Disfluency for Automatic Speech Recognition

Features such as punctuation, capitalization, and formatting of entities are important for readability, understanding, and natural language processing tasks. However, Automatic Speech Recognition (ASR) systems produce spoken-form text devoid of formatting, and tagging approaches to formatting address just one or two features at a time. In this paper, we unify spoken-to-written text conversion via a two-stage process: First, we use a single transformer tagging model to jointly produce token-level tags for inverse text normalization (ITN), punctuation, capitalization, and disfluencies. Then, we apply the tags to generate written-form text and use weighted finite state transducer (WFST) grammars to format tagged ITN entity spans. Despite joining four models into one, our unified tagging approach matches or outperforms task-specific models across all four tasks on benchmark test sets across several domains.
An Ultralightweight Object Detection Network for Empty-Dish Recycling Robots

The emergence of empty-dish recycling robots has alleviated problems, such as labor shortages, caused by an aging population. The detection and grasping of dishes play a crucial role in empty-dish recycling robots. However, due to the limited resources of edge devices, traditional object detection models require more space to store parameters and much computational overhead, limiting the development of empty-dish recycling robots. Therefore, this article proposes an ultralightweight dish detection model YOLO-GS for an empty-dish recycling robot. We use the modified CSPDarknet as the backbone structure and design an ultralightweight neck structure for efficient feature fusion. Meanwhile, we design a lightweight head structure for object classification and bounding box coordinate regression by combining ghost shuffle convolution (GSConv2D) and the anchor-free method. For the empty-dish recycling robot to grasp the dishes, we design a dish grasp point extraction algorithm using image processing. Finally, TensorRT is used to optimize and accelerate the model for efficient and intelligent detection of dishes on the NVIDIA Jetson Xavier NX. The experimental results show that YOLO-GS achieves 99.380% mean average precision (mAP) with a parameter amount of 0.606 M. The inference speed of the TensorRT-optimized YOLO-GS algorithm reaches 31.371 FPS, which meets the needs of real-time dish detection by the empty-dish recycling robot. The image of the empty-dish recycling robot demo is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://www.youtube.com/watch?v=pCBo1nzm3qU&amp;t=22s</uri> .
Communitychain: Toward a Scalable Blockchain in Smart Home

Rapid development of smart homes in recent years has led to the production of an increasing number of smart devices that meet people’s daily needs. However, these smart devices may belong to different vendors and provide different functions and services, making it difficult to apply a centralized solution to a smart home system. Although blockchain has the potential to address the potential problems of smart homes because of its features such as nontampering, decentralization, and security, scalability remains a key challenge when integrating blockchain and smart homes, which is mainly reflected in the horizontal expansion, throughput and latency of the system. To address this challenge, Communitychain–a new scalable blockchain architecture suitable for smart home systems composing heterogeneous devices–is proposed in this study. A community model based on sharding that enables the architecture to adapt to an increase in the number of nodes is designed, which includes new miner nodes and leader node selection algorithms. Efficient cross-shard routing and sideBlock schemes are adopted to improve the throughput and latency and enhance the scalability of the system. Furthermore, lightweight authorization and authentication processes running in each community ensure secure access to device resources. The experimental results indicate that the proposed blockchain architecture can effectively improve the scalability of blockchain-based smart homes and adapt well to the system dynamics.
Illumination-Adaptive Unpaired Low-Light Enhancement

Supervised networks address the task of low-light enhancement using paired images. However, collecting a wide variety of low-light/clean paired images is tedious as the scene needs to remain static during imaging. In this paper, we propose an unsupervised low-light enhancement network using context-guided illumination-adaptive norm (CIN). Inspired by coarse to fine methods, we propose to address this task in two stages. In stage- I, a pixel amplifier module (PAM) is used to generate a coarse estimate with an overall improvement in visibility and aesthetic quality. Stage- II further enhances the saturated dark pixels and scene properties of the image using CIN. Different ablation studies show the importance of PAM and CIN in improving the visible quality of the image. Next, we propose a region-adaptive single input multiple output (SIMO) model that can generate multiple enhanced images from a single low-light image. The objective of SIMO is to let users choose the image of their liking from a pool of enhanced images. Human subjective analysis of SIMO results shows that the distribution of preferred images varies, endorsing the importance of SIMO-type models. Lastly, we propose a low-light road scene (LLRS) dataset having an unpaired collection of low-light and clean scenes. Unlike existing datasets, the clean and low-light scenes in LLRS are real and captured using fixed camera settings. Exhaustive comparisons on publicly available datasets, and the proposed dataset reveal that the results of our model outperform prior art quantitatively and qualitatively.
Exploiting 3D fractal cube and chaos for effective multi-image compression and encryption

At present, some image compression and encryption algorithms have been presented to protect digital images by combining chaos and compressed sensing (CS). However, some of them still suffer from security and efficiency shortcomings. To solve these issues, a multi-image compression and encryption based on 3D fractal cube, CS and chaos is introduced in this paper. Firstly, the multiple plain images are sparsified by the Haar Wavelet Transform (HWT), and measured by CS to obtain the measurement value matrices. Herein, a four-wing hyperchaotic system is adopted to generate the measurement matrix for CS. Subsequently, a 3D fractal cube (3D-FC) is defined and generated iteratively by constructing the initial 3D-FC. Based on it, the obtained measurement value matrices are shuffled to reduce the high correlation between the adjacent pixels. Next, the cipher images are obtained after the cipher-associated adaptive diffusion based on GF (28) is performed on the confused image. Moreover, we have conducted extensive experiments to verify the encryption efficiency, compression performance, and security. Both simulation results and performance analysis confirm that the present algorithm has satisfactory performance in these aspects.
Bottom-up 2D pose estimation via dual anatomical centers for small-scale persons

In multi-person 2D pose estimation, the bottom-up methods simultaneously predict poses for all persons, and unlike the top-down methods, do not rely on human detection. However, the SOTA bottom-up methods' accuracy is still inferior compared to the existing top-down methods. This is due to the predicted human poses being regressed based on the inconsistent human bounding box center and the lack of human-scale normalization, leading to the predicted human poses being inaccurate and small-scale persons being missed. To push the envelope of the bottom-up pose estimation, we firstly propose multi-scale training to enhance the network to handle scale variation with single-scale testing, particularly for small-scale persons. Secondly, we introduce dual anatomical centers (i.e., head and body), where we can predict the human poses more accurately and reliably, especially for small-scale persons. Moreover, existing bottom-up methods use multi-scale testing to boost the accuracy of pose estimation at the price of multiple additional forward passes, which weakens the efficiency of bottom-up methods, the core strength compared to top-down methods. By contrast, our multi-scale training enables the model to predict high-quality poses in a single forward pass (i.e., single-scale testing). Our method achieves 38.4% improvement on bounding box precision and 39.1% improvement on bounding box recall over the state of the art (SOTA) on the challenging small-scale persons subset of COCO. For the human pose AP evaluation, we achieve a new SOTA (71.0 AP) on the COCO test-dev set with the single-scale testing. We also achieve the top performance (40.3 AP) on the OCHuman dataset in cross-dataset evaluation.
WMCP-EM: An integrated dehazing framework for visibility restoration in single image

Adopting local patches for varying haze conditions is crucial for optimizing the performance in single image dehazing. We propose a novel two-fold method with a prime focus on self-adaptive prior named Weighted Median Channel Prior (WMCP) to resolve the problems introduced by using a fixed size local-patch in the dehazing process. WMCP works by leveraging the spatially changing haze statistics such as inclusion–exclusion of related pixels for estimating depth-map in varying haze conditions. It is a scale-invariant technique that retains most of the information present in the local neighbourhood of the hazy input image for estimating scene depth, which traditional methods generally fail to preserve. In addition, an unsharp-masking based technique called edge-modulation (EM) promotes hidden or missing details such as microscopic edges and textures lost due to haze, making this scheme beneficial in ensuring a visually aesthetic and realistic dehazed image. This research also includes a set of ablation tests to assess the contributions of each module engaged in the dehazing process. We performed a comparative evaluation of our method with several state-of-the-art techniques, revealing its superiority in terms of visibility improvement and edge preservation, especially when the dense haze regions are taken into consideration.
Automated Eye Movement Classification Based on EMG of EOM Signals Using FBSE-EWT Technique

The accurate automated eye movement classification is gaining importance in the field of human–computer interaction (HCI). The present article aims at the classification of six types of eye movements from electromyogram (EMG) of extraocular muscles (EOM) signals using the Fourier–Bessel series expansion-based empirical wavelet transform (FBSE-EWT) with time and frequency-domain (TAFD) features. The FBSE-EWT of EMG signals results in Fourier–Bessel intrinsic mode functions (FBIMFs), which correspond to the frequency contents in the signal. A hybrid approach is used to select the prominent FBIMFs followed by the statistical and signal complexity-based feature extraction. Furthermore, metaheuristic optimization algorithms are employed to reduce the feature space dimension. The discrimination ability of the reduced feature set is verified by Kruskal–Wallis statistical test. Multiclass support vector machine (MSVM) has been employed for classification. First, the classification has been performed with TAFD features followed by the combination of TAFD and FBSE-EWT-based reduced feature set. The combination of TAFD and FBSE-EWT-based feature set has provided good classification performance. This study demonstrates the efficacy of FBSE-EWT and subsequent metaheuristic feature selection algorithms in classifying the eye movements from EMG of EOM signals. The combination of TAFD and the selected features through salp swarm optimization algorithm has provided maximum classification accuracy of 98.91% with MSVM employing Gaussian and radial basis function kernels. Thus, the proposed approach has the potential to be used in HCI applications involving biomedical signals.
Feature selection method based on quantum inspired genetic algorithm for Arabic signature verification

The signature is a behavioral-based human characteristics that is extensively used as legal evidence of identification on bank checks, credit cards, and wills. Developing a good offline signature verification (OSV) system is essential to avoid fraud. Due to the plurality of signature styles and sample contexts, feature extraction (FE) is challenging for OSV systems. Current signature verification methods present promising results in distinguishing between genuine and forged signatures, but they are far from satisfactory. Regarding feature selection (FS), stable and appropriate features substantially impact signature verification. The issue with current approaches is that they apply combination features without addressing correlation and discriminant. A genetic algorithm (GA) is popularly deployed as a method for optimization to identify the best features. Regrettably, GA has drawbacks when the search dimension is expanded, such as a lack of comprehensive exploitation, difficulties in achieving good convergence, and therefore increased computation. This study addresses the aforementioned challenges by introducing multifeature fusion and discriminant FS using a novel quantum inspired GA (QIGA). QIGAs are constructed using quantum computing concepts such as qubits (Q-bits) and superposition of states. QIGA is able to express a linear superposition of solutions with the objective of increasing gene diversity by utilizing the qubit as a representation. Quantum rotation computing (QRC) enhances population diversity and tackles computing complexity and early convergence issues in GA. The superposition concept in QRC utilizes both mutation (divergence) and crossover (convergence) procedures to expand diversity and improve population selection. QIGA performs better in a parallel structure because of its quick convergence and robust global search capabilities. This implies that QIGA embodies features of both exploration and exploitation. The studied approach is applied to three different databases of signatures: SID-Arabic handwritten signatures, CEDAR, and UTSIG. Consequently, the proposed OSV based on the QIGA improved equal error rate (EER) between 10% and 20% over the GA without influencing computation complexity.
The Economics of Ransomware Attacks on Integrated Supply Chain Networks

We explore the economics of ransomware on production supply chains. Integrated supply chains result in a mutual-dependence between firms that can be exploited by cyber-criminals. For instance, we show that by targeting one firm in the network the criminals can potentially hold multiple firms to ransom. Overlapping security systems may also allow the criminals to strike at weak points in the network. For instance, it may be optimal for the attacker to target a supplier in order to ransom a large producer at the heart of the production network. We introduce a game theoretic model of an attack on a supply chain and solve for two types of Nash equilibria. We then study a hub and spoke example before providing simulation results for a general case. We find that the total ransom the criminals can demand is increasing in the average path length of the network. Thus, the ransom is lowest for a hub and spoke network and highest for a line network. Mitigation strategies are discussed.