Reviewerly/exHarmony huggingface dataset:
Boosting Bert Subnets with Neural Grafting

Pre-trained Language Models in Natural Language Processing have become increasingly computationally expensive and memory demanding. The recently proposed computation-adaptive BERT models facilitate their deployment in practical applications. Training such a BERT model involves jointly optimizing subnets of varying sizes, which is not easy due to their mutual interference with one another. The larger-size subnets in particular could deteriorate when there is a large performance gap between the smallest subnet and the super-net. In this work, we propose Neural grafting to boost BERT subnets, especially the larger ones. Specifically, we regard the less important sub-modules of a BERT model as less active and reactivate them via layer-wise Neural grafting. Experimental results show that the proposed method improves the average performance of BERT subnets on six datasets of GLUE benchmark. The subnet performing comparable to the supernet BERT-Base reduces around 67% and 70% inference latency on GPU and CPU, respectively. Moreover, we compare two Neural grafting strategies under varied experimental settings, hoping to shed light on the application scenarios of Neural grafting.
Neural Networks with Quantization Constraints

Enabling low precision implementations of deep learning models, without considerable performance degradation, is necessary in resource and latency constrained settings. Moreover, exploiting the differences in sensitivity to quantization across layers can allow mixed precision implementations to achieve a considerably better computation performance trade-off. However, backpropagating through the quantization operation requires introducing gradient approximations, and choosing which layers to quantize is challenging for modern architectures due to the large search space. In this work, we present a constrained learning approach to quantization aware training. We formulate low precision supervised learning as a constrained optimization problem, and show that despite its non-convexity, the resulting problem is strongly dual and does away with gradient estimations. Furthermore, we show that dual variables indicate the sensitivity of the objective with respect to constraint perturbations. We demonstrate that the proposed approach exhibits competitive performance in image classification tasks, and leverage the sensitivity result to apply layer selective quantization based on the value of dual variables, leading to considerable performance improvements.
SENER: Sentiment Element Named Entity Recognition for Aspect-Based Sentiment Analysis

Aspect-based sentiment analysis (ABSA) is a task of identifying fine-grained sentiment entities in a given sentence, which is generally formulated as a sequence labeling problem. Recently, advancements in large pre-trained language models (PLMs) led to generative ABSA, where the task is treated as text-to-text transition resolved by fine-tuning PLMs. Although the generative methods are designed to capture sentence-level semantic information, they are inappropriate for explicit comprehension of sentiment structure. In order to address this issue, we propose sentiment element named entity recognition (SENER) for ABSA. SENER integrates the concepts of named entity recognition (NER) and generative ABSA to retrieve the sentiment entities with predefined sentiment element names, leading to better semantic and sentiment structure understanding. Experimental results on several ABSA tasks show that the proposed SENER significantly outperforms previous works on ASQP and ASTE.
Make Your Enemy Your Friend: Improving Image Rotation Angle Estimation with Harmonics

It is well known that rotation introduces periodic artifacts into the resulting image. By measuring such periodicities, the rotation angle θ can be estimated from the rotated image without the availability of the original unrotated image. However, existing methods suffer from harmonics, especially when θ is small. This paper revisits the harmonics caused by rotation in the cyclic spectrum and points out that such harmonics can help our rotation angle estimation if used effectively. Based on this observation, we propose aggregating the magnitudes of the candidate peaks and their harmonics in detecting the rotation-specific peak. Both theoretical analyses and experimental results demonstrate the advantage of the proposed scheme over previous methods.
Knowledge-Augmented Frame Semantic Parsing with Hybrid Prompt-Tuning

Frame semantics-based approaches have been widely used in semantic parsing tasks and have become mainstream. It remains challenging to disambiguate frame representations evoked by target lexical units under different contexts. Pre-trained Language Models (PLMs) have been used in semantic parsing and significantly improve the accuracy of neural parsers. However, the PLMs-based approaches tend to favor collocated patterns presented in the training data, leading to inaccurate outcomes. The intuition here is to design a mechanism to optimally use knowledge captured in semantic frames in conjunction with PLMs to disambiguate frames. We propose a novel Knowledge-Augmented Frame Semantic Parsing Architecture (KAF-SPA) to enhance semantic representation by incorporating accurate frame knowledge into PLMs during frame semantic parsing. Specifically, a Memory-based Knowledge Extraction Module (MKEM) is devised to select accurate frame knowledge and construct the continuous templates in the high dimensional vector space. Moreover, we design a Task-oriented Knowledge Probing Module (TKPM) using hybrid prompts (in terms of continuous and discrete prompts) to incorporate the selected knowledge into the PLMs and adapt PLMs to the tasks of frame and argument identification. Experimental results on two public FrameNet datasets demonstrate that our method significantly outperforms strong baselines (by more than +3% in F1), achieving state-of-art results on the current benchmark. Ablation studies verify the effectiveness of KAF-SPA.
Forecasting of Breathing Events from Speech for Respiratory Support

When a patient using a breathing support system such as a portable oxygen concentrator (POC) talks, the flow of oxygen to the lungs is disturbed. The ideal moment to administer oxygen-rich air during talking would be the brief inhale moments between utterances. However, the detection of the inhale moment is difficult and the latency of the air transfer from the pump, through the hose, to the nasal cannula may be larger than the inspiratory period in normal speech. The prediction of the next inhale moment could be used to compensate the latency and therefore provide a significantly better support for a patient who needs oxygen-rich air but want to communicate normally. In this paper we provide the first evidence that it is possible to forecasts the next inhale moment from the speech of the talker using deep learning techniques. Breathing forecasting from speech has also other potential applications that we briefly discuss in the paper.
Named Entity Detection and Injection for Direct Speech Translation

In a sentence, certain words are critical for its semantic. Among them, named entities (NEs) are notoriously challenging for neural models. Despite their importance, their accurate handling has been neglected in speech-to-text (S2T) translation research, and recent work has shown that S2T models perform poorly for locations and notably person names, whose spelling is challenging unless known in advance. In this work, we explore how to leverage dictionaries of NEs known to likely appear in a given context to improve S2T model outputs. Our experiments show that we can reliably detect NEs likely present in an utterance starting from S2T encoder outputs. Indeed, we demonstrate that the current detection quality is sufficient to improve NE accuracy in the translation with a 31% reduction in person name errors.
Certified Robustness of Quantum Classifiers Against Adversarial Examples Through Quantum Noise

Recently, quantum classifiers have been known to be vulnerable to adversarial attacks, where quantum classifiers are fooled by imperceptible noises to have misclassification. In this paper, we propose one first theoretical study that utilizing the added quantum random rotation noise can improve the robustness of quantum classifiers against adversarial attacks. We connect the definition of differential privacy and demonstrate the quantum classifier trained with the natural presence of additive noise is differentially private. Lastly, we derive a certified robustness bound to enable quantum classifiers to defend against adversarial examples supported by experimental results.
FindAdaptNet: Find and Insert Adapters by Learned Layer Importance

Adapters are lightweight bottleneck modules introduced to assist pre-trained self-supervised learning (SSL) models to be customized to new tasks. However, searching the appropriate layers to insert adapters on large models has become difficult due to the large number of possible layers and thus a vast search space (2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</sup> possibilities for N layers). In this paper, we propose a technique that achieves automatic insertion of adapters for downstream automatic speech recognition (ASR) and spoken language understanding (SLU) tasks. Our approach is based on two-stage training. First, we train our model for a specific downstream task with additional shallow learnable layers and weight parameters to obtain the weighted summation over the output of each layer in SSL. This training method is established by the SUPERB baseline [1]. This first-stage training determines the most important layers given their respective weights. In the second stage, we proceed to insert adapters to the most important layers, retaining both performance and neural architecture search efficiency. On the CommonVoice dataset[2] we obtain 20.6% absolute improvement in Word Error Rate (WER) on the Welsh language against the conventional method, which inserts the adapter modules into the highest layers without search. In the SLURP SLU task, our method yields 4.0% intent accuracy improvement against the same conventional baseline.
Associative Learning Network for Coherent Visual Storytelling

Visual storytelling task aims to generate relevant and coherent story for an ordered stream of images. Although visual storytelling methods have made promising improvement in recent years, existing methods pay little attention to the association ability and divergent thinking of the model, which are essential for humanistic stories. This paper introduces a novel Associative Learning Network for Coherent Visual Storytelling to explore the model’s association ability while telling a new story. Specifically, we first build a graph based on the pointwise mutual information and learn association degree of word pairs with Graph Convolutional Network. Besides, an auxiliary hierarchical decoder is designed to combine the words together to generate coherent story. In this way, our model can recall information using associative mem-ory, enhancing the coherence and informativeness of the generated story. Extensive experiments on VIST dataset demonstrate that the proposed framework substantially outperforms the state-of-the-art methods across multiple evaluation metrics.
A Bayesian Perspective for Determinant Minimization Based Robust Structured Matrix Factorization

We introduce a Bayesian perspective for the structured matrix factorization problem. The proposed framework provides a probabilistic interpretation for existing geometric methods based on determinant minimization. We model input data vectors as linear transformations of latent vectors drawn from a distribution uniform over a particular domain reflecting structural assumptions, such as the probability simplex in Nonnegative Matrix Factorization and polytopes in Polytopic Matrix Factorization. We represent the rows of the linear transformation matrix as vectors generated independently from a normal distribution whose covariance matrix is inverse Wishart distributed. We show that the corresponding maximum a posteriori estimation problem boils down to the robust determinant minimization approach for structured matrix factorization, providing insights about parameter selections and potential algorithmic extensions.
Variational Inference Aided Estimation of Time Varying Channels

One way to improve the estimation of time varying channels is to incorporate knowledge of previous observations. In this context, Dynamical VAEs (DVAEs) build a promising deep learning (DL) framework which is well suited to learn the distribution of time series data. We introduce a new DVAE architecture, called k-MemoryMarkovVAE (k-MMVAE), whose sparsity can be controlled by an additional memory parameter. Following the approach in [1] we derive a k-MMVAE aided channel estimator which takes temporal correlations of successive observations into account. The results are evaluated on simulated channels by QuaDRiGa and show that the k-MMVAE aided channel estimator clearly outperforms other machine learning (ML) aided estimators which are either memoryless or naively extended to time varying channels without major adaptions.
Learning Dependencies of Discrete Speech Representations with Neural Hidden Markov Models

While discrete latent variable models have had great success in self-supervised learning, most models assume that frames are independent. Due to the segmental nature of phonemes in speech perception, modeling dependencies among latent variables at the frame level can potentially improve the learned representations on phonetic-related tasks. In this work, we assume Markovian dependencies among latent variables, and propose to learn speech representations with neural hidden Markov models. Our general framework allows us to compare to self-supervised models that assume independence, while keeping the number of parameters fixed. The added dependencies improve the accessibility of phonetic information, phonetic segmentation, and the cluster purity of phones, showcasing the benefit of the assumed dependencies.
Meta Learning with Adaptive Loss Weight for Low-Resource Speech Recognition

Model Agnostic Meta-Learning (MAML) is an effective meta-learning algorithm for low-resource automatic speech recognition (ASR). It uses gradient descent to learn the initialization parameters of the model through various languages, making the model quickly adapt to unseen low-resource languages. But MAML is unstable due to its unique bilevel loss backward structure, which significantly affects the stability and generalization of the model. Since various languages have different contributions to the target language, the loss weights corresponding to the effects of diverse languages require costly manual adjustment in the training stage. Proper selection of these weights will influence the performance of the entire model. In this paper, we propose to apply a loss weight adaption method to MAML using Convolutional Neural Network (CNN) with Homoscedastic Uncertainty. The results of experiments showed that the proposed method outperformed previous gradient-based meta-learning methods and other loss weights adaption methods, and it further improved the stability and effectiveness of MAML.
Forensics for Adversarial Machine Learning Through Attack Mapping Identification

This paper considers the problem of performing post-attack forensic analysis for a test-time attack on a machine learning model. A test-time attack can be represented as a mapping that receives a benign test example as the input and outputs a falsified version of it. Given a set of attacked examples in the post-attack time, our objective is to identify the correct attack mapping among the collection of candidate attack strategies with diverse objectives and constraints. We present an attack mapping identification method that utilizes a pre-attack example recovery mechanism as a feature extraction method. Using numerical experiments, we demonstrate the effectiveness of the proposed approach in detecting the correct attack mapping among a number of different candidate attack strategies.
Background-Weakening Consistency Regularization for Semi-Supervised Video Action Detection

Consistency-based techniques have produced state-of-the-art results in semi-supervised action detection. When the model false detects the dynamic information in the background as an action, spatio-temporal consistency calculations can hardly reflect this false detection result. We consider weakening the dynamic information in the augmented video background to reduce its spatio-temporal consistency with the dynamic information in the original video background. Thus we propose a Background-Weakening with Calibration Constraint (BWCC) framework, which highlights the negative impact of information in the background of false detection by calculating the consistency of the predictions of the background weakened video and the original video. Specifically, Background Weaken (BW) module judges the foreground and background of the video based on the initial predictions of the model and makes adjustments to the video background. To mitigate the effects of the misjudgments result in weakened action pixels, we additionally introduce a model that does not undergo background weakening to aid training through Calibration Constraint (CC) module. Our approach achieves competitive performance over existing leading approaches on two action detection datasets, UCF101-24 and JHMDB-21.
Dynamic Scalable Self-Attention Ensemble for Task-Free Continual Learning

Continual learning represents a challenging task for modern deep neural networks due to the catastrophic forgetting following the adaptation of network parameters to new tasks. In this paper, we address a more challenging learning paradigm called Task-Free Continual Learning (TFCL), in which the task information is missing during the training. To deal with this problem, we introduce the Dynamic Scalable Self-Attention Ensemble (DSSAE) model, which dynamically adds new Vision Transformer (ViT) based-experts to deal with the data distribution shift during the training. To avoid frequent expansions and ensure an appropriate number of experts for the model, we propose a new dynamic expansion mechanism that evaluates the novelty of incoming samples as expansion signals. Furthermore, the proposed expansion mechanism does not require knowing the task information or the class label, which can be used in a realistic learning environment. Empirical results demonstrate that the proposed DSSAE achieves state-of-the-art performance in a series of TFCL experiments.
Sharing Low Rank Conformer Weights for Tiny Always-On Ambient Speech Recognition Models

Continued improvements in machine learning techniques offer exciting new opportunities through the use of larger models and larger training datasets. However, there is a growing need to offer these new capabilities on-board low-powered devices such as smart-phones, wearables and other embedded environments where only low memory is available. Towards this, we consider methods to reduce the model size of Conformer-based speech recognition models which typically require models with greater than 100M parameters down to just 5M parameters while minimizing impact on model quality. Such a model allows us to achieve always-on ambient speech recognition on edge devices with low-memory neural processors. We propose model weight reuse at different levels within our model architecture: (i) repeating full conformer block layers, (ii) sharing specific conformer modules across layers, (iii) sharing sub-components per conformer module, and (iv) sharing decomposed sub-component weights after low-rank decomposition. By sharing weights at different levels of our model, we can retain the full model in-memory while increasing the number of virtual trans-formations applied to the input. Through a series of ablation studies and evaluations, we find that with weight sharing and a low-rank architecture, we can achieve a WER of 2.84 and 2.94 for Librispeech dev-clean and test-clean respectively with a 5M parameter model.
Möbius Total Variation for Directed Acyclic Graphs

We propose a novel definition of total variation (TV) specifically defined for directed acyclic graphs (DAGs). It is a generalization of the classical definition for discrete-time signals (noting that discrete-time is a DAG) and obtained by inverting an integration operator on DAGs, leveraging the theory of Möbius inversion from combinatorics. We demonstrate the performance of our Möbius TV against prior TV definitions when used for denoising a set of real-world DAG signals.
Input-Dependent Dynamical Channel Association For Knowledge Distillation

Feature-map based knowledge distillation has exhibited its significance in improving the performance of student model. Existing works mainly focus on the formulation of knowledge, but ignore the number difference of channels due to heterogeneous architectures of teacher-student pair. They generally adopt handcrafted matching or input-independent association matrix, which would lead to the semantic mismatch, thus suboptimal performance. To resolve this problem, we present an input-dependent channel association module. This module automatically generates an allocation matrix in a cross-attention manner, which enables each student channel to be dynamically connected to its semantic-related teacher channel based on its learning state. An alternative training scheme is applied for stable optimization. Extensive experiments on image classification with a variety of settings based on the popular network architectures well demonstrate the effectiveness of our proposed strategy.
Instance-Aware Hierarchical Structured Policy for Prompt Learning in Vision-Language Models

In recent years, learnable prompts have emerged as a major prompt learning paradigm, enhancing the performance of large-scale vision-language pre-trained models in few-shot image classification. However, enhancing methods are often time-consuming and inflexible because 1) class-specific prompts are inefficient in certain situations; 2) instance-specific prompts are put in a fixed position. To address these issues, inspired by the coarse-to-fine decision-making paradigm of human, we propose an Instance-Aware Hierarchical-Structured Policy (IAHSP) that integrates instance-specific prompt selection and appropriate position selection using a reinforcement learning fashion. Specifically, IAHSP consists of two sub-policies: 1) the root policy selects the most suitable prompt from the prompts pool, and 2) the leaf policy identifies the optimal position for inserting the selected prompt. We train these two policies iteratively with rewards constraining the prompts while maintaining their diversity. Extensive experiments on 11 public benchmarks demonstrate that our IAHSP significantly boosts the few-shot image classification performance of vision-language pre-trained models, while also exhibiting superior generalization performance.
SAN: A Robust End-to-End ASR Model Architecture

In this paper, we propose a novel Siamese Adversarial Network (SAN) architecture for automatic speech recognition, which aims at solving the difficulty of fuzzy audio recognition. Specifically, SAN constructs two sub-networks to differentiate the audio feature input and then introduces a loss to unify the output distribution of these sub-networks. Adversarial learning enables the network to capture more essential acoustic features and helps the models achieve better performance when encountering fuzzy audio input. We conduct numerical experiments with the SAN model on several datasets for the automatic speech recognition task. All experimental results show that the siamese adversarial nets significantly reduce the character error rate (CER). Specifically, we achieve a competitive 4.37 CER result on the AISHELL-1 dataset without any language models. To reveal the generality of the siamese adversarial net, we also conduct experiments on the phoneme recognition task, which also shows the superiority of the siamese adversarial network.