Reviewerly/exHarmony huggingface dataset:
Adaspeech 2: Adaptive Text to Speech with Untranscribed Data

Text to speech (TTS) is widely used to synthesize personal voice for a target speaker, where a well-trained source TTS model is fine-tuned with few paired adaptation data (speech and its transcripts) on this target speaker. However, in many scenarios, only untranscribed speech data is available for adaptation, which brings challenges to the previous TTS adaptation pipelines (e.g., AdaSpeech). In this paper, we develop AdaSpeech 2, an adaptive TTS system that only leverages untranscribed speech data for adaptation. Specifically, we introduce a mel-spectrogram encoder to a well-trained TTS model to conduct speech reconstruction, and at the same time constrain the output sequence of the mel-spectrogram encoder to be close to that of the original phoneme encoder. In adaptation, we use untranscribed speech data for speech reconstruction and only fine-tune the TTS decoder. AdaSpeech 2 has two advantages: 1) Pluggable: our system can be easily applied to existing trained TTS models without re-training. 2) Effective: our system achieves on-par voice quality with the transcribed TTS adaptation (e.g., AdaSpeech) with the same amount of untranscribed data, and achieves better voice quality than previous untranscribed adaptation methods <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining

Much recent work on Spoken Language Understanding (SLU) is limited in at least one of three ways: models were trained on oracle text input and neglected ASR errors, models were trained to predict only intents without the slot values, or models were trained on a large amount of in-house data. In this paper, we propose a clean and general framework to learn semantics directly from speech with semi-supervision from transcribed or untranscribed speech to address these issues. Our framework is built upon pretrained end-to-end (E2E) ASR and self-supervised language models, such as BERT, and fine-tuned on a limited amount of target SLU data. We study two semi-supervised settings for the ASR component: supervised pretraining on transcribed speech, and unsupervised pretraining by replacing the ASR encoder with self-supervised speech representations, such as wav2vec. In parallel, we identify two essential criteria for evaluating SLU models: environmental noise-robustness and E2E semantics evaluation. Experiments on ATIS show that our SLU framework with speech as input can perform on par with those using oracle text as input in semantics understanding, even though environmental noise is present and a limited amount of labeled semantics data is available for training.
Two-Stage Attention-Based Model for Code Search with Textual and Structural Features

Searching and reusing existing code from a large scale codebase can largely improve developers’ programming efficiency. To support code reuse, early code search models leverage information retrieval (IR) techniques to index a large-scale code corpus and return relevant code according to developers’ search query. However, IR-based models fail to capture the semantics in code and query. To tackle this issue, developers applied deep learning (DL) techniques to code search models. However, these models either are too complex to determine an effective method efficiently or learning for semantic correlation between code and query inadequately.To bridge the semantic gap between code and query effectively and efficiently, we propose a code search model TabCS (Two-stage Attention-Based model for Code Search) in this study. TabCS extracts code and query information from the code textual features (i.e., method name, API sequence, and tokens), the code structural feature (i.e., abstract syntax tree), and the query feature (i.e., tokens). TabCS performs a two-stage attention net-work structure. The first stage leverages attention mechanisms to extract semantics from code and query considering their semantic gap. The second stage leverages a co-attention mechanism to capture their semantic correlation and learn better code/query representation. We evaluate the performance of TabCS on two existing large-scale datasets with 485k and 542k code snippets, respectively. Experimental results show that TabCS achieves an MRR of 0.57 on Hu et al.’s dataset, outperforming three state-of-the-art models CARLCS-CNN, DeepCS, and UNIF by 18%, 70%, 12%, respectively. Meanwhile, TabCS gains an MRR of 0.54 on Husain et al.’s, outperforming CARLCS-CNN, DeepCS, and UNIF by 32%, 76%, 29%, respectively.
MBNET: MOS Prediction for Synthesized Speech with Mean-Bias Network

Mean opinion score (MOS) is a popular subjective metric to assess the quality of synthesized speech, and usually involves multiple human judges to evaluate each speech utterance. To reduce the labor cost in MOS test, multiple methods have been proposed to automatically predict MOS scores. To our knowledge, for a speech utterance, all previous works only used the average of multiple scores from different judges as the training target and discarded the score of each individual judge, which did not well exploit the precious MOS training data. In this paper, we propose MBNet, a MOS predictor with a mean subnet and a bias subnet to better utilize every judge score in MOS datasets, where the mean subnet is used to predict the mean score of each utterance similar to that in previous works, and the bias subnet to predict the bias score (the difference between the mean score and each individual judge score) and capture the personal preference of individual judges. Experiments show that compared with MOSNet baseline that only leverages mean score for training, MBNet improves the system-level spearmans rank correlation coefficient (SRCC) by 2.9% on VCC 2018 dataset and 6.7% on VCC 2016 dataset.
Hand Gesture Recognition Based on Trajectories Features and Computation-Efficient Reused LSTM Network

Touchless hand gesture recognition using radar sensor is considered to be an attractive and effective technique for human-machine interaction. In this paper, we propose a hand gesture recognition approach based-on range-Doppler-angle trajectories and a reused long short-term memory (RLSTM) network using a 77GHz frequency modulated continuous wave (FMCW) multiple-input-multiple-output (MIMO) radar. To overcome amounts of gesture interferences, a gesture desktop is designed and a potential hand gesture detection approach is followed to determine whether it is a potential hand gesture. If the gesture happens in the assumed gesture desktop, the range-Doppler-angle trajectories are extracted by using the discretize Fourier transform (DFT), multiple signal classification (MUSIC) algorithm and the Kalman filtering. These signatures cannot only significantly reduce the dimension for the following neural network but also exploit the three dimensional information provided by the MIMO radar. Finally, a LSTM network with the reused forward propagation approach is proposed to exploit the spatial, temporal and Doppler information and simplify the network architecture. The experimental results show that the proposed approach can achieve an average accuracy of 99.4% and recognize a new person's hand gestures with an average accuracy of 98.0% for 9 hand gestures.
Motion U-Net: Multi-cue Encoder-Decoder Network for Motion Segmentation

Detection of moving objects is a critical component of many computer vision tasks. Recently, deep learning architectures have been developed for supervised learning based moving object change detection. Some top performing architectures, like FgSegNet are single frame spatial appearance cue-based detection and tend to overfit to the training videos. We propose a novel compact multi-cue autoencoder deep architecture, Motion U-Net (MU-Net) for robust moving object detection that generalizes much better than FgSegNet and requires nearly 30 times fewer weight parameters. Motion and change cues are estimated using a multi-modal background subtraction module combined with flux tensor motion estimation. MU-Net was trained and evaluated on the CDnet-2014 change detection challenge video sequences and had an overall F-measure of 0.9369. We used the unseen SBI-2015 video dataset to assess generalization capacity where MU-Net had an F-measure of 0.7625 while FgSegNet_v2 was 0.3519, less than half the MU-Net accuracy. The source code of the Motion U-Net is available at https://github.com/CIVA-Lab/Motion-U-Net.
PRICE: Privacy and Reliability-Aware Real-Time Incentive System for Crowdsensing

Crowdsensing is regarded as a critical component of the Internet of Things (IoT) and has been widely applied in smart city services. Incentive mechanism design, data reliability evaluation, and privacy preservation are the research focuses of crowdsensing. However, most existing incentive mechanisms fail to protect data privacy and evaluate data credibility, simultaneously. Moreover, traditional privacy and reliability-aware incentive schemes are usually challenging to realize real-time reward distribution. To this end, we first point out a single-time slice of failure problem in real-time incentive mechanisms and propose a two-layer truth discovery model (TLTD) to resolve this problem. Then, a reliability-aware real-time incentive mechanism (RRIM) is designed based on the proposed TLTD. In order to evaluate data reliability in a privacy-preserving manner, we build a privacy-preserving truth discovery solution ( <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PriTD</monospace> ) based on secure computation protocols. Finally, our proposed system [privacy and reliability-aware real-time incentive system for crowdsensing (PRICE)] integrating the aforementioned protocols realizes real-time reward distribution, data reliability evaluation, and privacy protection, simultaneously. Theoretical analysis and experimental evaluations on a synthetic and real-world data set demonstrate the feasibility and efficiency of the proposed PRICE.
e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks

Recently, there has been an increasing number of efforts to introduce models capable of generating natural language explanations (NLEs) for their predictions on vision-language (VL) tasks. Such models are appealing, because they can provide human-friendly and comprehensive explanations. However, there is a lack of comparison between existing methods, which is due to a lack of re-usable evaluation frameworks and a scarcity of datasets. In this work, we introduce e-ViL and e-SNLI-VE. e-ViL is a benchmark for explainable vision-language tasks that establishes a unified evaluation framework and provides the first comprehensive comparison of existing approaches that generate NLEs for VL tasks. It spans four models and three datasets and both automatic metrics and human evaluation are used to assess model-generated explanations. e-SNLI-VE is currently the largest existing VL dataset with NLEs (over 430k instances). We also propose a new model that combines UNITER [15], which learns joint embeddings of images and text, and GPT-2 [38], a pre-trained language model that is well-suited for text generation. It surpasses the previous state of the art by a large margin across all datasets. Code and data are available here: https://github.com/maximek3/e-ViL.
Taking an (Embodied) Cue From Community Health: Designing Dementia Caregiver Support Technology to Advance Health Equity

Dementia affects >50 million worldwide, causing progressive cognitive and physical disabilities. Its caregiving burden falls largely onto informal caregivers, who experience their own health problems, and face tremendous stress with little support–all exacerbated during COVID-19. In this paper, we present a new caregiver support perspective, where the lenses of health equity and community health can shape future technology design. Through a 1.5 year long, in-depth research process with dementia community health workers, we learned how caregiving support technology can reflect key concepts in dementia community health practice. This paper makes two contributions: 1) We propose employing embodied cueing, such as imitation or action mimicry, as a communication modality that can align technology with community caregiving approaches, promote agency in people with dementia, and relieve caregiver burden, and 2) We suggest new avenues for HCI research to advance health equity in the context of dementia technology design.
Review on Reinforcement Learning, Research Evolution and Scope of Application

Machine learning is considered as the study of computer algorithms that enables the machine to learn and adapt to new data without any human intervention. Reinforcement learning is a paradigm of machine learning by which a self-governing agent utilizes its experience communicating with an environmental situation to improve its behavior. The paper summarizes different reinforcement learning algorithms, merits and demerits of the existing reviewed methods along with the applications and challenges that gives a future research direction.
Tools, Tricks, and Hacks: Exploring Novel Digital Fabrication Workflows on #PlotterTwitter

As digital fabrication machines become widespread, online communities have provided space for diverse practitioners to share their work, troubleshoot, and socialize. These communities pioneer increasingly novel fabrication workflows, and it is critical that we understand and conceptualize these workflows beyond traditional manufacturing models. To this end, we conduct a qualitative study of #PlotterTwitter, an online community developing custom hardware and software tools to create artwork with computer-controlled drawing machines known as plotters. We documented and analyzed emergent themes where the traditional interpretation of digital fabrication workflows fails to capture important nuances and nascent directions. We find that #PlotterTwitter makers champion creative exploration of interwoven digital and physical materials over a predictable series of steps. We discuss how this challenges long-running views of digital fabrication and propose design implications for future frameworks and toolkits to account for this breadth of practice.
Evaluation of Sentiment Analysis via Word Embedding and RNN Variants for Amazon Online Reviews

Consumer feedback is highly valuable in business to assess their performance and is also beneficial to customers as it gives them an idea of what to expect from new products. In this research, the aim is to evaluate different deep learning approaches to accurately predict the opinion of customers based on mobile phone reviews obtained from Amazon.com. The prediction is based on analysing these reviews and categorizing them as positive, negative, or neutral. Different deep learning algorithms have been implemented and evaluated such as simple RNN with its four variants, namely, Long Short-Term Memory Networks (LRNN), Group Long Short-Term Memory Networks (GLRNN), gated recurrent unit (GRNN), and update recurrent unit (UGRNN). All evaluated algorithms are combined with word embedding as feature extraction approach for sentiment analysis including Glove, word2vec, and FastText by Skip-grams. The five different algorithms with the three feature extraction methods are evaluated based on accuracy, recall, precision, and F1-score for both balanced and unbalanced datasets. For the unbalanced dataset, it was found that the GLRNN algorithms with FastText feature extraction scored the highest accuracy of 93.75%. This result achieved the highest accuracy on this dataset when compared with other methods mentioned in the literature. For the balanced dataset, the highest achieved accuracy was 88.39% by the LRNN algorithm.
Dynamic Sparsity Neural Networks for Automatic Speech Recognition

In automatic speech recognition (ASR), model pruning is a widely adopted technique that reduces model size and latency to deploy neural network models on edge devices with resource constraints. However, multiple models with different sparsity levels usually need to be separately trained and deployed to heterogeneous target hardware with different resource specifications and for applications that have various latency requirements. In this paper, we present Dynamic Sparsity Neural Networks (DSNN) that, once trained, can instantly switch to any predefined sparsity configuration at run-time. We demonstrate the effectiveness and flexibility of DSNN using experiments on internal production datasets with Google Voice Search data, and show that the performance of a DSNN model is on par with that of individually trained single sparsity networks. Our trained DSNN model, therefore, can greatly ease the training process and simplify deployment in diverse scenarios with resource constraints.
Security and Privacy for Edge Intelligence in 5G and Beyond Networks: Challenges and Solutions

5G and beyond (B5G) networks are leading a digital revolution in telecommunication in both academia and industry. It brings new paradigms in many aspects of people's daily lives due to its advantages. However, it still leaves some issues in terms of security and privacy as challenges. Blockchain, the public database, is an alternative to the traditional centralized systems, serving as the backbone technique in many systems, including manufacturing, economics, and industry. Blockchain is promising in solving the security issues in the sense that it provides desirable properties including decentralization, transparency, immutability, and so on. In this article, we investigate typical security and privacy issues in edge intelligence in B5G networks and devise a framework to integrate blockchain with such systems, which can provide guaranteed security as well as privacy. We also illustrate several possible solutions to these security and privacy issues in edge intelligence in B5G systems based on blockchain and Ethereum to show how blockchain contributes to the coming B5G networks.
Attention Mechanisms in CNN-Based Single Image Super-Resolution: A Brief Review and a New Perspective

With the advance of deep learning, the performance of single image super-resolution (SR) has been notably improved by convolution neural network (CNN)-based methods. However, the increasing depth of CNNs makes them more difficult to train, which hinders the SR networks from achieving greater success. To overcome this, a wide range of related mechanisms has been introduced into the SR networks recently, with the aim of helping them converge more quickly and perform better. This has resulted in many research papers that incorporated a variety of attention mechanisms into the above SR baseline from different perspectives. Thus, this survey focuses on this topic and provides a review of these recently published works by grouping them into three major categories: channel attention, spatial attention, and non-local attention. For each of the groups in the taxonomy, the basic concepts are first explained, and then we delve deep into the detailed insights and contributions. Finally, we conclude this review by highlighting the bottlenecks of the current SR attention mechanisms, and propose a new perspective that can be viewed as a potential way to make a breakthrough.
Blockchain technology and smart contract for civil structural health monitoring system

Abstract The rapid development of blockchain technology is exerting a profound impact on all walks of life. Blockchain, which is a decentralized and distributed technology architecture, has inspired people to think about the social development model. However, blockchain technology has not been applied to health monitoring. Therefore, this study proposes a distributed health monitoring system based on blockchain technology, which realizes data security, information transparency, efficient sharing, and independent decision making with smart contracts. First, the system is designed as a blockchain network for structural health monitoring (SHM), and a series of smart contracts for health monitoring is developed to achieve monitoring authority verification, structural damage identification, and other functions. Based on the strength of the Ethereum private chain and Guangzhou New TV Tower benchmark model, smart contracts for health monitoring were deployed, and a series of tests of the proposed system were performed. The results indicate that the proposed system can provide monitoring authority verification, generate abnormal alerts, immutability of data, attack resistance, and traceability query.
When Video meets Inertial Sensors

Ubiquitous finger motion tracking enables a number of exciting applications in augmented reality, sports analytics, rehabilitation-healthcare etc. While finger motion tracking with cameras is very mature, largely due to availability of massive training datasets, there is a dearth of training data for developing robust machine learning (ML) models for wearable IoT devices with Inertial Measurement Unit (IMU) sensors. Towards addressing this problem, this paper presents ZeroNet, a system that shows the feasibility of developing ML models for IMU sensors with zero training overhead. ZeroNet harvests training data from publicly available videos for performing inferences on IMU. The difference in data among video and IMU domains introduces a number of challenges due to differences in sensor-camera coordinate systems, body sizes of users, speed/orientation changes during gesturing, sensor position variations etc. ZeroNet addresses these challenges by systematically extracting motion data from videos and transforming them into acceleration and orientation information measured by IMU sensors. Furthermore, data-augmentation techniques are exploited that create synthetic variations in the harvested training data to enhance the generalizability and robustness of the ML models to user diversity. Evaluation with 10 users demonstrates a top-1 accuracy of 82.4% and a top-3 accuracy of 94.8% for recognition of 50 finger gestures thus indicating promise. While we have only scratched the surface, we outline a number of interesting possibilities for extending this work in the cross-disciplinary areas of computer vision, machine learning, and wearable IoT for enabling novel applications in finger motion tracking.
Confidence Estimation for Attention-Based Sequence-to-Sequence Models for Speech Recognition

For various speech-related tasks, confidence scores from a speech recogniser are a useful measure to assess the quality of transcriptions. In traditional hidden Markov model-based automatic speech recognition (ASR) systems, confidence scores can be reliably obtained from word posteriors in decoding lattices. However, for an ASR system with an auto-regressive decoder, such as an attention-based sequence-to-sequence model, computing word posteriors is difficult. An obvious alternative is to use the decoder softmax probability as the model confidence. In this paper, we first examine how some commonly used regularisation methods influence the softmax-based confidence scores and study the overconfident behaviour of end-to-end models. Then we propose a lightweight and effective approach named confidence estimation module (CEM) on top of an existing end-to-end ASR model. Experiments on LibriSpeech show that CEM can mitigate the overconfidence problem and can produce more reliable confidence scores with and without shallow fusion of a language model. Further analysis shows that CEM generalises well to speech from a moderately mismatched domain and can potentially improve downstream tasks such as semi-supervised learning.