Reviewerly/exHarmony huggingface dataset:
ESCL: Equivariant Self-Contrastive Learning for Sentence Representations

Previous contrastive learning methods for sentence representations often focus on insensitive transformations to produce positive pairs, but neglect the role of sensitive transformations that are harmful to semantic representations. Therefore, we propose an Equivariant Self-Contrastive Learning (ESCL) method to make full use of sensitive transformations, which encourages the learned representations to be sensitive to certain types of transformations with an additional equivariant learning task. Meanwhile, in order to improve practicability and generality, ESCL simplifies the implementations of traditional equivariant contrastive methods to share model parameters from the perspective of multi-task learning. We evaluate our ESCL on semantic textual similarity tasks. The proposed method achieves better results while using fewer learning parameters compared to previous methods.
Repackagingaugment: Overcoming Prediction Error Amplification in Weight-Averaged Speech Recognition Models Subject to Self-Training

Representation-based speech recognition models have demonstrated state-of-the-art performance on downstream tasks. These models are pre-trained on large-scale unlabeled data, fine-tuned on a small amount of labeled data, and subsequently advanced via the self-training procedure by leveraging pseudo-labels. However, a self-trained representation model produces prediction errors caused by training with incorrect labels in the pseudo-labeled data. Weight-averaging methods have been employed to refine the pseudo-labels in a variety of studies; however, these methods amplify the prediction errors of each self-trained model. To alleviate this problem, we propose RepackagingAugment, a data augmentation method that improves the diversity of models while preventing the same incorrect labels from recursively occurring in every epoch. Our data augmentation deconstructs the paired speech–text data into word units and repackages them into a randomly determined number of word sequences. This strategy induces the models to produce different prediction errors by mitigating the problem of incorrect label over-fitting. Through various experiments on representation models, such as wav2vec 2.0 and data2vec, we demonstrate that our approach improves the performance of weight-averaged models.
PQLM - Multilingual Decentralized Portable Quantum Language Model

With careful manipulation, malicious agents can reverse engineer private information encoded in pre-trained language models. Security concerns motivate the development of quantum pre-training. In this work, we propose a highly portable quantum language model (PQLM) that can easily transmit information to downstream tasks on classical machines. The framework consists of a cloud PQLM built with random Variational Quantum Classifiers (VQC) and local models for downstream applications. We demonstrate the ad hoc portability of the quantum model by extracting only the word embeddings and effectively applying them to downstream tasks on classical machines. Our PQLM exhibits comparable performance to its classical counterpart on both intrinsic evaluation (loss, perplexity) and extrinsic evaluation (multilingual sentiment analysis accuracy) metrics. We also perform ablation studies on the factors affecting PQLM performance to analyze model stability. Our work establishes a theoretical foundation for a portable quantum pre-trained language model that could be trained on private data and made available for public use with privacy protection guarantees.
Semantics-Disentangled Contrastive Embedding for Generalized Zero-Shot Learning

Generalized zero-shot learning (GZSL) is a challenging class of vision and knowledge transfer problems in which both seen and unseen classes appear during testing. Most existing GZSL methods achieve knowledge transfer based on the original features of samples that inevitably contain information irrelevant to recognition, resulting in negative influence for the performance. In this paper, we propose a novel contrastive disentanglement learning framework for the GZSL task (SDCE-GZSL), where the original and generated visual features are factorized into semantic-consistent and semantic-unrelated representations via a novel mutual information (MI)-based constraint. In addition, we propose a contrastive learning framework that leverages class-level and instance-level supervision to further facilitate disentanglement. Extensive experiments show that our approach achieves significant improvements over the state-of-the-art approaches.
The Ajmide Topic Segmentation System for the ICASSP 2023 General Meeting Understanding and Generation Challenge

This paper describes our topic segmentation (TS) system submitted to the ICASSP2023 Signal Processing Grand Challenge - General Meeting Understanding and Generation challenge (MUG). We make three improvements to the official baseline system of the TS track. Firstly, considering that meeting transcriptions are usually long-form documents, we propose a PoNet-Svec-Transformers network to learn both sentence representations and document-level context. Secondly, we introduce a training data synthesis method that significantly increase the size of the training dataset. Finally, we leverage focal loss and adversarial training methods to improve system performance. Our best submission achieves a first-place score of 48.56/48.84 on the Eval/Test set in the TS task.
Enhancing Ontology Translation Through Cross-Lingual Agreement

Ontology serves as the foundation for the underlying representation of knowledge. In order to achieve the sharing of knowledge across languages, ontologies that are typically only represented in English must be translated into different languages. Building a domain-specific translation system is necessary for ontology due to the extremely specialized vocabulary and absence of contextual information. In this paper, we introduce cross-lingual agreement to alleviate the aforementioned issues. We propose a method for representing ontology labels that constructs agreement constraint objects by minimizing the distances between source and target sides. We also integrate with adversarial learning to reduce the difference between the ontology label and the hypothesis generated by the translation model during the fine-tuning. Finally, the agreement modeling strategy is incorporated into the decoding phase to guide the generation of translation candidates. Experiments on the four domains English-to-German ontology show that the proposed method achieves significant improvements over the baselines.
Possibilistic Bernoulli Filter for Extended Target Tracking

An extended object in target tracking refers to the object which produces a time-varying number of noisy detections (measurements) from its scattering or feature points. The optimal sequential Bayesian state estimator for an appearing/disappearing extended object in the presence of false and missed detections is known as the Bernoulli Filter Ext (BF-X) [1]. Bayesian estimation methods rely on probabilistic models. When probabilistic models are known only partially or imprecisely, quantitative modeling of uncertainty can be carried out using possibility functions. This paper formulates the analog of the BF-X in the framework of possibility theory, where uncertainty is represented using possibility functions, rather than probability distributions. Possibility functions have the capacity to model with integrity the partial or imprecise probabilistic specifications and thus the proposed possibilistic BF-X is characterised by an enhanced robustness in the absence of precise measurement or dynamic models.
Asymptotically Optimal Nonparametric Classification Rules for Spike Train Data

Spike train data find a growing list of applications in computational neuroscience, streaming data and finance. Statistical analysis of spike trains is based on various probabilistic and neural network models. The statistical approach relies on parametric or nonparametric specifications of the underlying model. In this paper we consider the nonparametric classification problem for a class of spike train data characterized by nonparametricaly specified intensity functions. We derive the optimal Bayes rule and next formulate the plug-in non-parametric kernel classifiers. Asymptotical properties of the rules are established including the limit with respect to the increasing observation time interval and the size of a training set. The obtained results are supported by a finite sample simulation studies.
LeanSpeech: The Microsoft Lightweight Speech Synthesis System for Limmits Challenge 2023

This paper describes the Microsoft Text-to-Speech (TTS) system: LeanSpeech for LIMMITS (Lightweight, Multi-speaker, Multi-lingual Indic TTS) Challenge 2023 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> , which is part of ICASSP2023 to encourage the advance of TTS in Indian Languages. We propose a lightweight encoder-decoder acoustic model composed of 1-D convolution and LSTM blocks, which is trained with knowledge distillation from a multi-speaker multi-lingual teacher model, DelightfulTTS [1]. The speech corpus is reprocessed and used in both AM training and vocoder fine-tuning. In Track-2 of the challenge, our system achieves MOS 4.56 and SMOS 3.98, which indicates the efficiency of the proposed model and training strategy.
Hierarchical Softmax for End-To-End Low-Resource Multilingual Speech Recognition

Low-resource speech recognition has been long-suffering from insufficient training data. In this paper, we propose an approach that leverages neighboring languages to improve low-resource scenario performance, founded on the hypothesis that similar linguistic units in neighboring languages exhibit comparable term frequency distributions, which enables us to construct a Huffman tree for performing multilingual hierarchical Softmax decoding. This hierarchical structure enables cross-lingual knowledge sharing among similar tokens, thereby enhancing low-resource training outcomes. Empirical analyses demonstrate that our method is effective in improving the accuracy and efficiency of low-resource speech recognition.
Alignment Entropy Regularization

Existing training criteria in automatic speech recognition (ASR) permit the model to freely explore more than one time alignments between the feature and label sequences. In this paper, we use entropy to measure a model’s uncertainty, i.e. how it chooses to distribute the probability mass over the set of allowed alignments. Furthermore, we evaluate the effect of entropy regularization in encouraging the model to distribute the probability mass only on a smaller subset of allowed alignments. Experiments show that entropy regularization enables a much simpler decoding method without sacrificing word error rate, and provides better time alignment quality.
Defense Against Black-Box Adversarial Attacks Via Heterogeneous Fusion Features

This paper presents an effective approach for the adversarial defense task named a heterogeneous feature fusion network (HFFN). Inspired by the fact that humans can utilize multimodal information to help themselves perceive objects, we introduce the caption features into the classic convolutional neural networks (CNNs) and fuse them with traditional image features. To reduce the "modal gap" between heterogeneous features, we introduce the hetero-center loss that can constrain the distance between the class centers of different modalities. In addition, we further integrate the guided complement entropy loss into HFFN, which can restrict the prediction probability of the model on non-ground truth classes, so that the adversarial robustness of fused heterogeneous features can be deeply improved. We further adopt three state-of-the-art comparison approaches and design four ablation methods to evaluate the effectiveness and adversarial robustness of HFFN. Moreover, in order to compare the performance of HFFN with the comparison methods and ablation methods, we also utilize a substitute model to generate a variety range of adversarial examples. Extensive experiments based on these adversarial examples for the CIFAR-10 and CIFAR-100 datasets exhaustively and strongly demonstrate the superiority of our method.
Spatial Correlation Fusion Network for Few-Shot Segmentation

Few-shot semantic segmentation aims to learn new knowledge rapidly with very few annotated data to segment novel classes. Recent methods follow a metric learning framework with prototypes for foreground representation [1]. However, representing support images by one or more prototypes may face problems caused by inadequate representation for segmentation, noise in complex scenes, and close semantic relation to background features. We propose a Spatial Correlation Fusion Network(SCFNet) for few-shot segmentation to address the issues. Firstly, to better capture fine-grained features, we design a Spatial Correlation Fusion module to address the loss of spatial information in support images, thus improving the performance of Few-shot segmentation. Secondly, a Prototype Contrastive Transformation(PCT) module is proposed to learn a transformation matrix for the prototype, which is capable of alleviating close semantic information and noise by adopting transformation loss. Experiments on PASCAL-5 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> [2] and COCO-20 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> [3] validate the effectiveness of our network for few-shot semantic segmentation and show our approach achieves state-of-the-art results.
LP-IOANet: Efficient High Resolution Document Shadow Removal

Document shadow removal is an integral task in document enhancement pipelines, as it improves visibility, readability and thus the overall quality. Assuming that the majority of practical document shadow removal scenarios require real-time, accurate models that can produce high-resolution outputs in-the-wild, we propose Laplacian Pyramid with Input/Output Attention Network (LP-IOANet), a novel pipeline with a lightweight architecture and an upsampling module. Furthermore, we propose three new datasets which cover a wide range of lighting conditions, images, shadow shapes and viewpoints. Our results show that we outperform the state-of-the-art by a 35% relative improvement in mean average error (MAE), while running real-time in four times the resolution (of the state-of-the-art method) on a mobile device.
Convex Optimization of Deep Polynomial and ReLU Activation Neural Networks

We consider training multi-layer neural networks with polynomial and ReLU activation functions. We develop exact convex optimization formulations for three-layer and deeper architectures. Our formulations are based on semidefinite lifting and recent results on the hidden convexity of two-layer ReLU networks. We show that certain deep neural network architectures can be trained to global optimality in polynomial time by solving equivalent convex optimization problems.
Progressive Diversifying Policy for Multi-Agent Reinforcement Learning

Multi-Agent Reinforcement Learning (MARL) has recently achieved promising performance in many collaborative decision making tasks. However, one of the main bottleneck challenges for MARL is the sparsity of the team reward, which can lead to the homogenization of agents’ behaviors. To address these issues, we propose a Progressive Diversifying Policy (PDP) algorithm in this paper. Specifically, we propose to actively amplify the diversity between agents’ policies during learning and exploit diversity as an additional intrinsic reward for MARL. Furthermore, we propose a progressive diversity boosting policy to find a better team policy. Leveraging the aforementioned improvements, our method can handle sparse team rewards and alleviate the homogeneous behaviors of agents. We conduct experiments on widely-used MARL environments and the results show that PDP can provide state-of-the-art performance while maintaining a competitive convergence speed.
Visual Information Matters for ASR Error Correction

Aiming to improve the Automatic Speech Recognition (ASR) outputs with a post-processing step, ASR error correction (EC) techniques have been widely developed due to their efficiency in using parallel text data. Previous works mainly focus on using text or/ and speech data, which hinders the performance gain when not only text and speech information, but other modalities, such as visual information are critical for EC. The challenges are mainly two folds: one is that previous work fails to emphasize visual information, thus rare exploration has been studied. The other is that the community lacks a high-quality benchmark where visual information matters for the EC models. Therefore, this paper provides 1) simple yet effective methods, namely gated fusion and image captions as prompts to incorporate visual information to help EC; 2) large-scale benchmark datasets, namely Visual-ASR-EC, where each item in the training data consists of visual, speech, and text information, and the test data are carefully selected by human annotators to ensure that even humans could make mistakes when visual information is missing. Experimental results show that using captions as prompts could effectively use the visual information and surpass state-of-the-art methods by upto 1.2% in Word Error Rate(WER), which also indicates that visual information is critical in our proposed Visual-ASR-EC dataset.
Spatio-Temporal Structure Consistency for Semi-Supervised Medical Image Classification

Intelligent medical diagnosis has shown remarkable progress on the large-scale datasets with full annotations. However, very few labeled images are available due to significantly expensive annotations by experts. To efficiently leverage abundant unlabeled data, we propose a novel Spatio-Temporal Structure Consistent (STSC) learning framework to combine both spatial and temporal structure consistency. Specifically, a gram matrix is derived to capture the structural similarity among different training samples in the representation space. At the spatial level, our framework explicitly enforces the consistency of structural similarity among different samples under perturbations. At the temporal level, we desire to maintain the consistency of the structural similarity in different training iterations by digging out the stable sub-structures in a relation graph. Experiments on two medical image datasets (i.e., ISIC 2018 and ChestX-ray14) show that our method outperforms state-of-the-art Semi-Supervised Learning (SSL) methods. Furthermore, extensive qualitative analysis on the Gram matrices and heatmaps by Grad-CAM are presented to validate the effectiveness of our method.
Adaptive Scale and Spatial Aggregation for Real-Time Object Detection

Cutting-edge real-time detectors usually reach real-time performance by adopting lightweight architectures. The accuracy of detection may be limited by their insufficient capabilities to obtain powerful feature representation, which is a notoriously onerous task in machine vision applications. Aiming at this problem, this study proposes a method of adaptive aggregation of features at both scale and spatial levels in an anchor-free framework: 1) at the scale level, a Multi-scale Point Feature Fusion (MPFF) module has been proposed to fuse point features from multiple scales via a self-adaptive re-weighting manner; 2) at the spatial level, a Restrained Deformable Convolution (R-DCN) has been designed to focus on the most informative features in a pre-defined region while avoiding the remote feature distraction. Based on R-DCN, an Adaptive Spatial Aggregation (ASA) module has been presented to alleviate the feature misalignment problem in classification and regression tasks via their respective spatial divisions. Extensive experimental results on MS COCO indicate that Adaptive Aggregation Detector (AADet) achieves a state-of-the-art detection performance, i.e., 41.8 AP at 60 FPS.
Is Multi-Task Learning an Upper Bound for Continual Learning?

Continual learning and multi-task learning are commonly used machine learning techniques for learning from multiple tasks. However, existing literature assumes multi-task learning as a reasonable performance upper bound for various continual learning algorithms, without rigorous justification. Additionally, in a multi-task setting, a small subset of tasks may behave as adversarial tasks, negatively impacting overall learning performance. On the other hand, continual learning approaches can avoid the negative impact of adversarial tasks and maintain performance on the remaining tasks, resulting in better performance than multi-task learning. This paper introduces a novel continual self-supervised learning approach, where each task involves learning an invariant representation for a specific class of data augmentations. We demonstrate that this approach results in naturally contradicting tasks and that, in this setting, continual learning often outperforms multi-task learning on benchmark datasets, including MNIST, CIFAR-10, and CIFAR-100.
Augmentation Robust Self-Supervised Learning for Human Activity Recognition

Human Activity Recognition (HAR) is widely applied on wearable devices in our daily lives. However, acquiring high-quality wearable sensor data set with ground-truths is challenging due to the high cost in collecting data and necessity of domain experts. In order to achieve generalization from limited data, we study augmentation-based Self-Supervised Learning (SSL) for data from wearable devices. However, there is an issue in one of the most popular SSL approaches, contrastive learning: it is sensitive to the choice of data augmentations. To resolve this, we first propose to combine contrastive learning with generative learning, which is robust to augmentations. Second, we propose an automatic augmentation policy search method to discover the most promising augmentation policy. We empirically verify our approaches on three public HAR datasets. Experimental results show that our proposed SSL approach is robust to augmentations, and delivers higher accuracy than contrastive learning. Additionally, with the searched augmentation policy we are able to further improve the accuracy of HAR task.
Simulating Realistic Speech Overlaps Improves Multi-Talker ASR

Multi-talker automatic speech recognition (ASR) has been studied to generate transcriptions of natural conversation including over-lapping speech of multiple speakers. Due to the difficulty in acquiring real conversation data with high-quality human transcriptions, a naïve simulation of multi-talker speech by randomly mixing multiple utterances was conventionally used for model training. In this work, we propose an improved technique to simulate multi-talker overlap-ping speech with realistic speech overlaps, where an arbitrary pattern of speech overlaps is represented by a sequence of discrete tokens. With this representation, speech overlapping patterns can be learned from real conversations based on a statistical language model, such as N-gram, which can be then used to generate multi-talker speech for training. In our experiments, multi-talker ASR models trained with the proposed method show consistent improvement on the word error rates across multiple datasets.