Reviewerly/exHarmony huggingface dataset:
The Coral++ Algorithm for Unsupervised Domain Adaptation of Speaker Recognition

State-of-the-art speaker recognition systems are trained with a large amount of human-labeled training data set. Such a training set is usually composed of various data sources to enhance the modeling capability of models. However, in practical deployment, unseen condition is almost inevitable. Domain mismatch is a common problem in real-life applications due to the statistical difference between the training and testing data sets. To alleviate the degradation caused by domain mismatch, we propose a new feature-based unsupervised domain adaptation algorithm. The algorithm we propose is a further optimization based on the well-known CORrelation ALignment (CORAL), so we call it CORAL++. On the NIST 2019 Speaker Recognition Evaluation (SRE19), we use SRE18 CTS set as the development set to verify the effectiveness of CORAL++. With the typical x-vector/PLDA setup, the CORAL++ outperforms the CORAL by 9.40% relatively on EER.
Contrastive Siamese Network for Semi-Supervised Speech Recognition

This paper introduces contrastive siamese (c-siam) network, an architecture for leveraging unlabeled acoustic data in speech recognition. c-siam is the first network that extracts high-level linguistic information from speech by matching outputs of two identical transformer encoders. It contains augmented and target branches which are trained by: (1) masking inputs and matching outputs with a contrastive loss, (2) incorporating a stop gradient operation on the target branch, (3) using an extra learnable transformation on the augmented branch, (4) introducing new temporal augment functions to prevent the shortcut learning problem. We use the Libri-light 60k unsupervised data and the LibriSpeech 100hrs/960hrs supervised data to compare c-siam and other best-performing systems. Our experiments show that c-siam provides 20% relative word error rate improvement over wav2vec baselines. A c-siam network with 450M parameters achieves competitive results compared to the state-of-the-art networks with 600M parameters.
Multi-Modal Emotion Recognition with Self-Guided Modality Calibration

Multi-modal emotion recognition aims to extract sentiment-related information from multiple sources and integrate different modal representations for sentiment analysis. Alignment is an effective strategy to achieve semantically consistent representations for multi-modal emotion recognition, while the current alignment models are jointly unable to maintain the dependence of word-to-sentence and independence of unimodal learning. In this paper, we propose a Self-guided Modality Calibration Network (SMCN) to realize multi-modal alignment which can capture the global connections without interfering with unimodal learning. While preserving unimodal learning without interference, our model leverages semantic sentiment-related features to guide modality-specific representation learning. On one hand, SMCN simulates human thinking by deriving a branch for acquiring knowledge of other modalities in unimodal learning. This branch aims to lean high-level semantic information of other modalities for realizing semantic alignment between modalities. On the other hand, we also provide an indirect interaction manner to integrate unimodal feature and calibrate features in different levels for avoiding unimodal features mixed with other clues. Experiments demonstrate that our approach outperforms the state-of-the-art methods on both IEMOCAP and MELD datasets.
MLP-SVNET: A Multi-Layer Perceptrons Based Network for Speaker Verification

Convolution and self-attention based neural networks have both obtained excellent performance in automatic speaker verification. However, the convolution model often lacks the ability of long-term dependency modeling due to the limitation of receptive field, while the self-attention model is insufficient to model local information. To tackle this limitation, we propose a new multi-layer perceptrons based speaker verification network (MLP-SVNet) which can apply MLPs across temporal and frequency dimensions to capture the local and global information at the same time. The experimental results conducted on Voxceleb show that the proposed model is very competitive when compared to other systems based on convolution or self-attention. In addition, we demonstrate that MLP-SVNet based on multi-layer per-ceptrons can produce complementary embeddings, which can be fused with the state-of-the-art system to further improve the performance.
Self-Supervised Learning for Sentiment Analysis via Image-Text Matching

There is often a resemblance in the sentiment expressed in social media posts (text) and their accompanying images. In this paper, We leverage this sentiment congruence for self-supervised representation learning for sentiment analysis. By teaching the model to pair an image with its corresponding social media post, the model can learn a representation capturing sentiment features from the image and text without supervision. We then use the pre-trained encoder for feature extraction for sentiment analysis in downstream tasks. We show significant improvement and good transferability for sentiment classification in addition to robustness in performance when available data decreases on public datasets (B-T4SA and IMDb Movie Review). With this work, we demonstrate the effectiveness of self-supervised learning through cross-modal matching for sentiment analysis.
Anno-MI: A Dataset of Expert-Annotated Counselling Dialogues

Research on natural language processing for counselling dialogue analysis has seen substantial development in recent years, but access to this area remains extremely limited due to the lack of publicly available expert-annotated therapy conversations. In this work, we introduce AnnoMI, the first publicly and freely accessible dataset of professionally transcribed and expert-annotated therapy dialogues. It consists of 133 conversations that demonstrate high- and low-quality motivational interviewing (MI), an effective counselling technique, and the annotations by domain experts cover key MI attributes. We detail the data collection process including dialogue selection, transcription and annotation. We also present analyses of AnnoMI and discuss its potential applications.
Underwater Small Target Detection Based on Deformable Convolutional Pyramid

Due to the problem of severe deformation, occlusion, diversified scenarios, general object detection methods cannot achieve satisfactory results in underwater object detection tasks. In this paper, we propose a two-stage Underwater Small Target Detection (USTD) network. In the proposed USTD, the Deformable Convolutional Pyramid(DCP) is proposed to deal with the problems of deformation, occlusion, and various object sizes effectively. Besides, we also propose a strategy of domain generalization based on curriculum learning to improve generalization in multi-domain environments, which is named as Phased Learning. Afterward, we construct an underwater target detection set (UTDS) to evaluate the accuracy of our method in underwater target detection tasks. Our method shows superior detection performance in experiments and reaches state-of-the-art for underwater target detection. Finally, in the 2020 China Underwater Robot Professional Contest (URPC), our method reached third place in terms of accuracy.
Are 3D convolutional networks inherently biased towards appearance?

3D convolutional networks, as direct inheritors of 2D convolutional networks for images, have placed their mark on action recognition in videos. Combined with pretraining on large-scale video data, high classification accuracies have been obtained on numerous video benchmarks. In an effort to better understand why 3D convolutional networks are so effective, several works have highlighted their bias towards static appearance and towards the scenes in which actions occur. In this work, we seek to find the source of this bias and question whether the observed biases towards static appearances are inherent to 3D convolutional networks or represent limited significance of motion in the training data. We resolve this by presenting temporality measures that estimate the data-to-model motion dependency at both the layer-level and the kernel-level. Moreover, we introduce two synthetic datasets where motion and appearance are decoupled by design, which allows us to directly observe their effects on the networks. Our analysis shows that 3D architectures are not inherently biased towards appearance. When trained on the most prevalent video sets, 3D convolutional networks are indeed biased throughout, especially in the final layers of the network. However, when training on data with motions and appearances explicitly decoupled and balanced, such networks adapt to varying levels of temporality. To this end, we see the proposed measures as a reliable method to estimate motion relevance for activity classification in datasets and use them to uncover the differences between popular pre-training video collections, such as Kinetics, IG-65M and Howto100 m.
Semi-Supervised Classification for Intra-Pulse Modulation of Radar Emitter Signals Using Convolutional Neural Network

Intra-pulse modulation classification of radar emitter signals is beneficial in analyzing radar systems. Recently, convolutional neural networks (CNNs) have been used in classification of intra-pulse modulation of radar emitter signals, and the results proved better than the traditional methods. However, there is a key disadvantage in these CNN-based methods: the CNN requires enough labeled samples. Labeling the modulations of radar emitter signal samples requires a tremendous amount of prior knowledge and human resources. In many circumstances, the labeled samples are quite limited compared with the unlabeled samples, which means that the classification will be semi-supervised. In this paper, we propose a method which could adapt the CNN-based intra-pulse classification approach to the case where a very limited number of labeled samples and a large number of unlabeled samples are provided, to classify the intra-pulse modulations of radar emitter signals. The method is based on a one-dimensional CNN and uses pseudo labels and self-paced data augmentation, which could improve the accuracy of intra-pulse classification. Extensive experiments show that our proposed method can improve the intra-pulse modulation classification performance in the semi-supervised situations.
An Adapter Based Pre-Training for Efficient and Scalable Self-Supervised Speech Representation Learning

We present a method for transferring pre-trained self-supervised (SSL) speech representations to multiple languages. There is an abundance of unannotated speech, so creating self-supervised representations from raw audio and fine-tuning on small annotated datasets is a promising direction to build speech recognition systems. SSL models generally perform SSL on raw audio in a pre-training phase and then fine-tune on a small fraction of annotated data. Such models have produced state of the art results for ASR. However, these models are very expensive to pre-train. We use an existing wav2vec 2.0 model and tackle the problem of learning new language representations while utilizing existing model knowledge. Crucially we do so without catastrophic forgetting of the existing language representation. We use adapter modules to speed up pre-training a new language task. Our model can decrease pre-training times by 32% when learning a new language task, and learn this new audio-language representation without forgetting previous language representation. We evaluate by applying these language representations to automatic speech recognition.
Synthesizing Dysarthric Speech Using Multi-Speaker Tts For Dysarthric Speech Recognition

Dysarthria is a motor speech disorder often characterized by reduced speech intelligibility through slow, uncoordinated control of speech production muscles. Automatic Speech recognition (ASR) systems may help dysarthric talkers communicate more effectively. To have robust dysarthria-specific ASR, sufficient training speech is required, which is not readily available. Recent advances in Text-To-Speech (TTS) synthesis multi-speaker end-to-end systems suggest the possibility of using synthesis for data augmentation. In this paper, we aim to improve multi-speaker end-to-end TTS systems to synthesize dysarthric speech for improved training of a dysarthria-specific DNN-HMM ASR. In the synthesized speech, we add dysarthria severity level and pause insertion mechanisms to other control parameters such as pitch, energy, and duration. Results show that a DNN-HMM model trained on additional synthetic dysarthric speech achieves WER improvement of 12.2% compared to the baseline, the addition of the severity level and pause insertion controls decrease WER by 6.5%, showing the effectiveness of adding these parameters. Audio samples are available at https://mohammadelc.github.io/SpeechGroupUKY/
A Two-Stream Information Fusion Approach to Abnormal Event Detection in Video

Human abnormal activity detection for automatic surveillance systems is to detect abnormal objects and human behaviours in videos. In this paper, we propose to explicitly address different kinds of abnormal events by developing a two-stream fusion approach that integrates both geometry and image texture information. To be concrete, we firstly propose to utilize an object detector to divide the abnormal events into two catalogues: abnormal human behaviors and abnormal objects. For the detection of abnormal human behaviours, we exploit a spatial-temporal graph convolutional network (ST-GCN) which considers both spatial and temporal domains to capture the geometrical features from human pose graphs. The extracted geometric feature embeddings are further adapted with a clustering step to cluster the temporal graphs and output normality scores. For the detection of abnormal objects, the obtained from the object detector are reused to assist with generating normality scores of possible anomalies. Finally, a late fusion is performed to integrate normality scores from both screams for final decision. The experimental results on the datasets of UCSD PED2 and ShanghaiTech Campus demonstrate the effectiveness of our proposed approach and the improved performance compared to other state-of-the-art approaches.
Encryption Resistant Deep Neural Network Watermarking

Deep neural network (DNN) watermarking is one of the main techniques to protect the DNN. Although various DNN watermarking schemes have been proposed, none of them is able to resist the DNN encryption. In this paper, we propose an encryption resistent DNN watermarking scheme, which is able to resist the parameter shuffling based DNN encryption. Unlike the existing schemes which use the kernels separately for watermarking embedding, we propose to embed the watermark into the fused kernels to resist the parameter shuffling. We further propose a MappingNet to map the the fused kernels into a higher dimension to increase the watermarking capacity. The MappingNet and the DNN are jointly trained to conduct final watermark embedding. Experimental results indicate the effectiveness of our proposed scheme for resisting the DNN encryption.
Neufa: Neural Network Based End-to-End Forced Alignment with Bidirectional Attention Mechanism

Although deep learning and end-to-end models have been widely used and shown their superiority in automatic speech recognition (ASR) and text-to-speech (TTS) synthesis, state-of-the-art forced alignment (FA) models are still based on hidden Markov model (HMM). HMM has limited view of contextual information and is developed with long pipelines, leading to error accumulation and unsatisfactory performance. Inspired by the capability of attention mechanism in capturing long term contextual information and learning alignments in ASR and TTS, we propose a neural network based end-to-end forced aligner called NeuFA, in which a novel bidirectional attention mechanism plays an essential role. NeuFA integrates the alignment learning of both ASR and TTS tasks in a unified framework by learning bidirectional alignment information from a shared attention matrix in the proposed bidirectional attention mechanism. Alignments are extracted from the learnt attention weights and optimized by the ASR, TTS and FA tasks in a multi-task learning manner. Experimental results demonstrate the effectiveness of our proposed model, with mean absolute error (MAE) on test set drops from 25.8 ms to 23.7 ms at word level, and from 18.0 ms to 15.7 ms at phoneme level compared with state-of-the-art HMM based model.
Voice-Enabled Intelligent Virtual Agents for People With Amnesia: Systematic Review

Older adults often have increasing memory problems (amnesia), and approximately 50 million people worldwide have dementia. This syndrome gradually affects a patient over a period of 10-20 years. Intelligent virtual agents may support people with amnesia.This study aims to identify state-of-the-art experimental studies with virtual agents on a screen capable of verbal dialogues with a target group of older adults with amnesia.We conducted a systematic search of PubMed, SCOPUS, Microsoft Academic, Google Scholar, Web of Science, and CrossRef on virtual agent and amnesia on papers that describe such experiments. Search criteria were (Virtual Agent OR Virtual Assistant OR Virtual Human OR Conversational Agent OR Virtual Coach OR Chatbot) AND (Amnesia OR Dementia OR Alzheimer OR Mild Cognitive Impairment). Risk of bias was evaluated using the QualSyst tool (University of Alberta), which scores 14 study quality items. Eligible studies are reported in a table including country, study design type, target sample size, controls, study aims, experiment population, intervention details, results, and an image of the agent.A total of 8 studies was included in this meta-analysis. The average number of participants in the studies was 20 (SD 12). The verbal interactions were generally short. The usability was generally reported to be positive. The human utterance was seen in 7 (88%) out of 8 studies based on short words or phrases that were predefined in the agent's speech recognition algorithm. The average study quality score was 0.69 (SD 0.08) on a scale of 0 to 1.The number of experimental studies on talking about virtual agents that support people with memory problems is still small. The details on the verbal interaction are limited, which makes it difficult to assess the quality of the interaction and the possible effects of confounding parameters. In addition, the derivation of the aggregated data was difficult. Further research with extended and prolonged dialogues is required.
Invisible and Efficient Backdoor Attacks for Compressed Deep Neural Networks

Compressed deep neural network (DNN) models have been widely deployed in many resource-constrained platforms and devices. However, the security issue of the compressed models, especially their vulnerability against backdoor attacks, is not well explored yet. In this paper, we study the feasibility of practical backdoor attacks for the compressed DNNs. More specifically, we propose a universal adversarial perturbation (UAP)-based approach to achieve both high attack stealthiness and high attack efficiency simultaneously. Evaluation results across different DNN models and datasets with various compression ratios demonstrate our approach’s superior performance compared with the existing solutions.
Application of LDA and word2vec to detect English off-topic composition

This paper presents an off-topic detection algorithm combining LDA and word2vec aiming at the problem of the lack of accurate and efficient off-topic detection algorithms in the English composition-assisted review system. The algorithm uses the LDA model to model the document and train the document through the word2vec, and uses the semantic relationship between the document's topics and words to calculate the probability weighted sum for each topic and its feature words in the document, and finally selects the off-topic composition by setting a reasonable threshold. Different F values are obtained by changing the number of topics in the document, and the best number of topics is determined. Experimental results show that the proposed method is more effective than vector space model, can detect more off-topic compositions, and the accuracy is higher, the F value is more than 88%, which realizes the intelligent processing of off-topic detection of composition, and can be effectively applied in English composition teaching.
Owning Mistakes Sincerely: Strategies for Mitigating AI Errors

Interactive AI systems such as voice assistants are bound to make errors because of imperfect sensing and reasoning. Prior human-AI interaction research has illustrated the importance of various strategies for error mitigation in repairing the perception of an AI following a breakdown in service. These strategies include explanations, monetary rewards, and apologies. This paper extends prior work on error mitigation by exploring how different methods of apology conveyance may affect people's perceptions of AI agents; we report an online study (N=37) that examines how varying the sincerity of an apology and the assignment of blame (on either the agent itself or others) affects participants' perceptions and experience with erroneous AI agents. We found that agents that openly accepted the blame and apologized sincerely for mistakes were thought to be more intelligent, likeable, and effective in recovering from errors than agents that shifted the blame to others.
Saliency-aware color harmony models for outdoor signboard

This paper introduces a geometric approach for assessing color harmony of a signboard, and color coherence of a signboard with the environment. We propose to incorporate visual saliency as an inherent color characteristic residing in the image space, to better cope with the attention mechanism when people view a scene. In doing so, our color harmony models consider saliency-weighted color differences and area balance in CIELab color space. We collect 5.2 K valid subjective ratings on 375 diverse signboards in the real world, and translate them into quantitative measures for model construction. Experimental results show that our models improve the overall performance, especially for modeling color coherence between a signboard and the environment. The study also reveals that color combinations with similar chroma but distinctive lightness lead to harmonic signboards, while simple color patches in accordance with local context contribute to environment-coherent signboards.
Lithology Identification of Uranium-Bearing Sand Bodies Using Logging Data Based on a BP Neural Network

Lithology identification is an essential fact for delineating uranium-bearing sandstone bodies. A new method is provided to delineate sandstone bodies by a lithological automatic classification model using machine learning techniques, which could also improve the efficiency of borehole core logging. In this contribution, the BP neural network model for automatic lithology identification was established using an optimized gradient descent algorithm based on the neural network training of 4578 sets of well logging data (including lithology, density, resistivity, natural gamma, well-diameter, natural potential, etc.) from 8 boreholes of the Tarangaole uranium deposit in Inner Mongolia. The softmax activation function and the cross-entropy loss function are used for lithology classification and weight adjustment. The lithology identification prediction was carried out for 599 samples, with a prediction accuracy of 88.31%. The prediction results suggest that the model is efficient and effective, and that it could be directly applied for automatic lithology identification in sandstone bodies for uranium exploration.