Reviewerly/exHarmony huggingface dataset:
Fine-Grained Emotional Control of Text-to-Speech: Learning to Rank Inter- and Intra-Class Emotion Intensities

State-of-the-art Text-To-Speech (TTS) models are capable of producing high-quality speech. The generated speech, however, is usually neutral in emotional expression, whereas very often one would want fine-grained emotional control of words or phonemes. Although still challenging, the first TTS models have been recently proposed that are able to control voice by manually assigning emotion intensity. Unfortunately, due to the neglect of intra-class distance, the intensity differences are often unrecognizable. In this paper, we propose a fine-grained controllable emotional TTS, that considers both inter- and intra-class distances and be able to synthesize speech with recognizable intensity difference. Our subjective and objective experiments demonstrate that our model exceeds two state-of-the-art controllable TTS models for controllability, emotion expressiveness and naturalness.
Joint Robust Representation And Generalization Enhancement For Cross-Modality Person Re-Identification

Cross-modality person re-identification (cm-ReID) aims to match pedestrian images from visible and infrared cameras. Most existing methods ignore data bias due to different cameras and views and overlook the strong dependence between feature maps that hinders modal alignment. In this paper, we propose a unified method named Joint Robust Representation and Generalization Enhancement (RRGE) to alleviate the above issues. First, we propose a robust representation module (RRM), which can improve the model’s robustness for the global context, camera, and view change perturbations. Second, we propose a generalization enhancement module (GEM), which uses channel-level dropout to alleviate the dependencies between feature maps to improve the model’s generalization. Moreover, we balance the number of different modalities in each batch. Our method outperforms other state-of-the-art methods in terms of cross-modality person re-identification tasks.
Weavspeech: Data Augmentation Strategy For Automatic Speech Recognition Via Semantic-Aware Weaving

A cut-and-paste type of data augmentation strategy has attracted considerable attention in the vision community due to its simplicity and effectiveness in improving generalization performance. However, it is challenging for Automatic Speech Recognition (ASR) tasks to apply this type of augmentation since segments corresponding to specific output tokens (e.g. words or sub-words) have various lengths. Furthermore, if speech signals are indiscriminately mixed without considering semantics, the risk of generating nonsensical sentences arises. To address these issues, in this paper, we propose WeavSpeech, still a simple yet effective cut-and-paste augmentation method for ASR tasks that weaves a pair of speech data considering semantics. Our method can be applied to any language without requiring language-specific knowledge and seamlessly integrated with other verified augmentations. We validate the superiority of our method on representative ASR benchmark datasets, including LibriSpeech and WSJ.
Perceptual Analysis of Speaker Embeddings for Voice Discrimination between Machine And Human Listening

This study investigates the information captured by speaker embeddings with relevance to human speech perception. A Convolutional Neural Network was trained to perform one-shot speaker verification under clean and noisy conditions, such that high-level abstractions of speaker-specific features were encoded in a latent embedding vector. We demonstrate that robust and discriminative speaker embeddings can be obtained by using a training loss function that optimizes the embeddings for similarity scoring during inference. Computational analysis showed that such speaker embeddings predicted various hand-crafted acoustic features, while no single feature explained substantial variance of the embeddings. Moreover, the relative distances in the speaker embedding space moderately coincided with voice similarity, as inferred by human listeners. These findings confirm the overlap between machine and human listening when discriminating voices and motivate further research on the remaining disparities for improving model performance.
History, Present and Future: Enhancing Dialogue Generation with Few-Shot History-Future Prompt

Dialogue history and response in open-domain dialogue are loosely coupled. Generating informative responses solely based on the original dialogue history is not easy, as dialogue history may not contain enough information or it may contain irrelevant noises. Intuitively, if a generation model can foresee possible dialogue future, or obtain real useful histories, it could generate more informative responses. In this paper, we propose a novel lightweight dialogue generation framework named few-shot history-future prompt that utilizes useful histories and simulated futures to help generate informative responses, without the need for fine-tuning or adding extra parameters. To obtain useful histories, we retrieve and combine relevant utterances from noisy multi- turn histories. Then we adopt a retrieval-generation hybrid approach to obtain diversified simulated futures. Such that our model could learn to condition on history combinations and simulated futures via few-shot learning. Experiments over publicly available datasets demonstrate that our method can help models generate better responses.
Meeting Action Item Detection with Regularized Context Modeling

Meetings are increasingly important for collaborations. Action items in meeting transcripts are crucial for managing post-meeting to-do tasks, which usually are summarized laboriously. The Action Item Detection task aims to automatically detect meeting content associated with action items. However, datasets manually annotated with action item detection labels are scarce and in small scale. We construct and release the first Chinese meeting corpus with manual action item annotations <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> . In addition, we propose a Context-Drop approach to utilize both local and global contexts by contrastive learning, and achieve better accuracy and robustness for action item detection. We also propose a Lightweight Model Ensemble method to exploit different pre-trained models <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> . Experimental results on our Chinese meeting corpus and the English AMI corpus demonstrate the effectiveness of the proposed approaches.
To drop or not to drop? Predicting the omission of the infinitival marker in a Swedish future construction

Abstract We investigate the optional omission of the infinitival marker in a Swedish future tense construction. During the last two decades the frequency of omission has been rapidly increasing, and this process has received considerable attention in the literature. We test whether the knowledge which has been accumulated can yield accurate predictions of language variation and change. We extracted all occurrences of the construction from a very large collection of corpora. The dataset was automatically annotated with language-internal predictors which have previously been shown or hypothesized to affect the variation. We trained several models in order to make two kinds of predictions: whether the marker will be omitted in a specific utterance and how large the proportion of omissions will be for a given time period. For most of the approaches we tried, we were not able to achieve a better-than-baseline performance. The only exception was predicting the proportion of omissions using autoregressive integrated moving average models for one-step-ahead forecast, and in this case time was the only predictor that mattered. Our data suggest that most of the language-internal predictors do have some effect on the variation, but the effect is not strong enough to yield reliable predictions.
CONSEN: Complementary and Simultaneous Ensemble for Alzheimer’s Disease Detection and MMSE Score Prediction

This paper proposes a novel method for Alzheimer’s disease detection and MMSE prediction using a complementary and simultaneous ensemble (CONSEN) algorithm based on multilingual spontaneous speech. We define pause and intervention of speech to form disfluency features, as well as several acoustic features to train generalized models. With the help of the proposed CONSEN algorithm, our model achieves the best performance of 86.69% for AD detection and 3.727 RMSE for MMSE prediction, which is placed first rank in both tasks in ICASSP Signal Processing Grand Challenge: ADReSS-M Challenge 2023.
Rethink Pair-Wise Self-Supervised Cross-Modal Retrieval From A Contrastive Learning Perspective

Cross-modal retrieval often faces the challenges of eliminating modality gap, learning robust modality invariance and semantic discrimination. Existing self-supervised crossmodal approaches still suffer from the faulty negative sample selection strategy and the lack of reliable high-level semantic discriminative guidance. Therefore, we propose a robust self-supervised co-training instance and semantic discrimination learning method (RCL) for cross-modal retrieval. Specifically, by the k-reciprocal nearest neighbor to generate pairwise pseudo-labels, we can correctly select negative samples and better filter the false negative ones, and thus pull semantically similar instances closer in a similar supervised contrastive learning. In addition, we use prototype contrastive learning to learn high-level semantic discriminative representations from different semantic groups, which pull instances and prototype vectors closer to better learn the semantic structure of multimodal data. Extensive experiments demonstrate the effectiveness of our method on cross-modal datasets.
Compressing Cross-Domain Representation via Lifelong Knowledge Distillation

Most Knowledge Distillation (KD) approaches focus on the discriminative information transfer and assume that the data is provided in batches during training stages. In this paper, we address a more challenging scenario in which different tasks are presented sequentially, at different times, and the learning goal is to transfer the generative factors of visual concepts learned by a Teacher module to a compact latent space represented by a Student module. In order to achieve this, we develop a new Lifelong Knowledge Distillation (LKD) framework where we train an infinite mixture model as the Teacher which automatically increases its capacity to deal with a growing number of tasks. In order to ensure a compact architecture and to avoid forgetting, we propose to measure the relevance of the knowledge from a new task for a set of experts making up the Teacher module, guiding each expert to capture the probabilistic characteristics of several similar domains. The network architecture is expanded only when learning an entirely different task. The Student is implemented as a lightweight probabilistic generative model. The experiments show that LKD can train a compressed Student module that achieves the state of the art results with fewer parameters.
Enhanced Dcf Tracker Regularized by Reliable Sample Construction

Discriminative correlation filter (DCF) is a highly efficient tracking technique using the circulant shifted samples of search images to update the template, so the reliability of input samples determines template quality. In this paper, we rethink the reliability problem of input samples in advance during template updating and propose an enhanced DCF tracking method regularized by a novel sparse representation based reliable sample construction term, called enhanced sparse correlation filter (ESCF). Specifically, the reconstructed reliable samples are the sparse representation of circulant shifted samples of unfiltered input samples, in which the target will approach the center to preserve target visual cues into the template when using the cosine window. Besides, we jointly perform template learning and reliable sample construction into a unified learning paradigm to benefit from each other, which further can be carried out in the frequency domain without incurring excessive time cost by skillful decomposition. Experiments on several popular visual tracking datasets verify the efficacy of ESCF and show that ESCF performs favorably against several well-established representative counterparts.
Dewarping Documents Using C<sup>2</sup> Continuous Boundary Estimation

Hand-held mobile devices are commonly used to capture documents. Physical distortions in such captured document images make it difficult to extract information automatically. Uneven illumination and different lighting condition make the process more challenging. The problem is even more severe when we capture book pages. In this paper, we propose a novel document dewarping algorithm that works as follows. First, a document boundary (BD) mask is computed from an input image using a deep learning model which is further used to compute the document corners. Using the document corners, several equidistant points are estimated along each of the four document sides. These equidistant points are used to compute smooth outlines independently along four document sides using C <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> continuous Bézier curve. Finally, a mesh is computed using the Bézier curves and the document is dewarped. The proposed algorithm is compared with the stateof-the-art algorithms in the literature and the results show the efficiency of the proposed algorithm. The low computational complexity of the algorithm makes it suitable to be used in any mobile devices.
Unsupervised Fine-Tuning Data Selection for ASR Using Self-Supervised Speech Models

Self-supervised learning (SSL) has been able to leverage unlabeled data to boost the performance of automatic speech recognition (ASR) models when we have access to only a small amount of transcribed speech data. However, this raises the question of which subset of the available unlabeled data should be selected for transcription. Our work investigates different unsupervised data selection techniques for fine-tuning the HuBERT model under a limited transcription budget. We investigate the impact of speaker diversity, gender bias, and topic diversity on the downstream ASR performance. We also devise two novel techniques for unsupervised data selection: pre-training loss based data selection and the perplexity of byte pair encoded clustered units (PBPE) and we show how these techniques compare to pure random data selection. Finally, we analyze the correlations between the inherent characteristics of the selected fine-tuning subsets as well as how these characteristics correlate with the resultant word error rate. We demonstrate the importance of token diversity, speaker diversity, and topic diversity in achieving the best performance in terms of WER.
Exploring Binary Classification Loss for Speaker Verification

The mismatch between close-set training and open-set testing usually leads to significant performance degradation for speaker verification task. For existing loss functions, metric learning-based objectives depend strongly on searching effective pairs which might hinder further improvements. And popular multi-classification methods are usually observed with degradation when evaluated on unseen speakers. In this work, we introduce SphereFace2 framework which uses several binary classifiers to train the speaker model in a pair-wise manner instead of performing multi-classification. Benefiting from this learning paradigm, it can efficiently alleviate the gap between training and evaluation. Experiments conducted on Voxceleb show that the SphereFace2 outperforms other existing loss functions, especially on hard trials. Besides, large margin fine-tuning strategy is proven to be compatible with it for further improvements. Finally, SphereFace2 also shows its strong robustness to class-wise noisy labels which has the potential to be applied in the semi-supervised training scenario with inaccurate estimated pseudo labels.
Local-Global Progressive U-Transformers for Accurate Hepatic and Portal Veins Segmentation in Abdominal MR Images

Segmentation of hepatic and portal veins in abdominal magnetic resonance images plays an essential role in surgical planning of liver tumor ablation and resection. Accurately extracting these blood vessels is a challenging task due to the complex vessel structures with high noise and irregular vessel shapes caused by nearby tumors. This work presents a new deep learning method called local-global progressive U-Transformers for precise extraction of hepatic and portal veins. Specifically, our method embeds convolution into the Transformer frame to extract features progressively and uses window attention to achieve full fusion of local and global features, as well as it only requires a small number of training parameters close to lightweight networks. We evaluated the proposed method on 30 clinical abdominal scans, with the experimental results showing that our method works better than the other segmentation approaches, improving the dice similarity coefficient from 0.7885 to 0.8132 and significantly reducing the number of parameters from 93.19M to 7.57M. We also found that our method can address the problem of voxel intensity variations and irregular vessel structures.
Training Graph Neural Networks on Growing Stochastic Graphs

Graph Neural Networks (GNNs) rely on graph convolutions to exploit meaningful patterns in networked data. Based on matrix multiplications, convolutions incur in high computational costs leading to scalability limitations in practice. To overcome these limitations, proposed methods rely on training GNNs in smaller number of nodes, and then transferring the GNN to larger graphs. Even though these methods are able to bound the difference between the output of the GNN with different number of nodes, they do not provide guarantees against the optimal GNN on the very large graph. In this paper, we propose to learn GNNs on very large graphs by leveraging the limit object of a sequence of growing graphs, the graphon. We propose to grow the size of the graph as we train, and we show that our proposed methodology – learning by transference – converges to a neighborhood of a first order stationary point on the graphon data. A numerical experiment validates our proposed approach.
Pseudo Multi-Source Domain Extension and Selective Pseudo-Labeling for Unsupervised Domain Adaptive Medical Image Segmentation

Unsupervised domain adaptation (UDA) attracts extra attention in medical image processing because no additional labels are required when adapting to different distributions. In this work, we propose a novel unsupervised domain adaptation framework named as Domain Expansion and PseudoLabeling (DEPL). We extend the domain of a labeled source domain data to four different distributed domains and use adversarial learning to align the image appearance level and feature level from the four different domains to the unlabeled target domain. In addition, we propose a selective pseudolabeling mechanism, namely using strong confidence pseudolabeling to boost model performance. We evaluate our model for the MR to CT adaptation segmentation task on the public dataset MMWHS. Compared to seven other state-of-the-art segmentation methods, our DEPL achieves the best Dice similarity coefficient by 82.4%, which is at least 3.9% higher than the other UDA segmentation methods.
Hint-Dynamic Knowledge Distillation

Knowledge Distillation (KD) transfers the knowledge from a high-capacity teacher model to promote a smaller student model. Existing efforts guide the distillation by matching their prediction logits, feature embedding, etc., while leaving how to efficiently utilize them in junction less explored. In this paper, we propose Hint-dynamic Knowledge Distillation, dubbed HKD, which excavates the knowledge from the teacher’s hints in a dynamic scheme. The guidance effect from the knowledge hints usually varies in different instances and learning stages, which motivates us to customize a specific hint-learning manner for each instance adaptively. Specifically, a meta-weight network is introduced to generate the instance-wise weight coefficients about knowledge hints in the perception of the dynamical learning progress of the student model. We further present a weight ensembling strategy to eliminate the potential bias of coefficient estimation by exploiting the historical statics. Experiments on standard benchmarks of CIFAR-100 and Tiny-ImageNet manifest that the proposed HKD well boost the effect of knowledge distillation tasks.
Graph Representation Learning For Stroke Recurrence Prediction

Stroke is one of the leading causes of death worldwide, and its mortality rate is drastically higher for patients who suffer recurrent strokes. Motivated by the recent success of graph learning methods on medical tasks, we introduce a graph representation framework for stroke recurrence prediction (GraSReP) based on patient data. In a nutshell, GraSReP sequentially consists of: i) a procedure for converting tabular, time-series patient data to a series of graphs, ii) a graph deep learning architecture that provides embeddings for the patients and, iii) a random forest classifier that predicts the patients’ risk of recurrent stroke based on these graph embeddings. We demonstrate GraSReP’s effectiveness for predicting recurrent strokes using real-world electronic health records, and discuss how it can be leveraged for the efficient application of preventive care.
An algorithmic debugging approach for belief-desire-intention agents

Abstract Debugging agent systems can be rather difficult. It is often noted as one of the most time-consuming tasks during the development of cognitive agents. Algorithmic (or declarative) debugging is a semi-automatic technique, where the developer is asked questions by the debugger in order to locate the source of an error. We present how this can be applied in the context of a BDI agent language, demonstrate how it can speed up or simplify the debugging process and reflect on its advantages and limitations.
Side-Channel Power Analysis Based on SA-SVM

Support vector machines (SVMs) have been widely used in side-channel power analysis. The selection of the penalty factor and kernel parameter heavily influences how well support vector machines work. Setting reasonable SVM hyperparameters is a key issue in side-channel power analysis. The novel side-channel power analysis method SA-SVM, which combines simulated annealing (SA) and support vector machines (SVMs) to analyze the power traces and crack the key, is proposed in this paper as a solution to this issue. This method differs from other approaches in that it integrates SA and SVMs, enabling us to more effectively explore the search space and produce superior results. In this paper, we conducted experiments on SA-SVM and SVM models from three different aspects: the selection of kernel functions, the number of parameters, and the number of eigenvalues. To compare our results with previous research, we performed experimental evaluations on open datasets. The results indicate that, compared with the SVM model, the SA-SVM model improved the accuracy by 0.25% to 3.25% and reduced the required time by 39.96% to 98.02% when the point of interest was 53, recovering the key using only three power traces. The SA-SVM model outperforms existing methods in terms of accuracy and computation time.