Reviewerly/exHarmony huggingface dataset:
Dynamic Chunk Convolution for Unified Streaming and Non-Streaming Conformer ASR

Recently, there has been an increasing interest in unifying streaming and non-streaming speech recognition models to reduce development, training and deployment cost. The best-known approaches rely on either window-based or dynamic chunk-based attention strategy and causal convolutions to minimize the degradation due to streaming. However, the performance gap still remains relatively large between non-streaming and a full-contextual model trained independently. To address this, we propose a dynamic chunk-based convolution replacing the causal convolution in a hybrid Connectionist Temporal Classification (CTC)-Attention Conformer architecture. Additionally, we demonstrate further improvements through initialization of weights from a full-contextual model and parallelization of the convolution and self-attention modules. We evaluate our models on the open-source Voxpopuli, LibriSpeech and in-house conversational datasets. Overall, our proposed model reduces the degradation of the streaming mode over the non-streaming full-contextual model from 41.7% and 45.7% to 16.7% and 26.2% on the LibriSpeech test-clean and test-other datasets respectively, while improving by a relative 15.5% WER over the previous state-of-the-art unified model.
Towards Bandwidth Estimation for Graph Signal Reconstruction

In numerous graph signal processing applications, data is often missing for a variety of reasons, and predicting the missing data is essential. In this paper, we consider data on graphs modeled as bandlimited graph signals. Predicting or reconstructing the unknown signal values for such a model requires an estimate of the signal bandwidth. In this paper, we address the problem of estimating the reconstruction errors, minimizing which would thereby provide an estimate of the signal bandwidth. In doing so, we design a cross-validation approach needed for stable graph signal reconstruction and propose a method for estimating the reconstruction errors for different choices of signal bandwidth. Using this technique, we are able to estimate the reconstruction error on a variety of real-world graphs.
Stuart: Individualized Classroom Observation of Students with Automatic Behavior Recognition And Tracking

Each student matters, but it is hardly for instructors to observe all the students during the courses and provide helps to the needed ones immediately. In this paper, we present StuArt, a novel automatic system designed for the individualized classroom observation, which empowers instructors to concern the learning status of each student. StuArt can recognize five representative student behaviors (hand-raising, standing, sleeping, yawning, and smiling) that are highly related to the engagement and track their variation trends during the course. To protect the privacy of students, all the variation trends are indexed by the seat numbers without any personal identification information. Furthermore, StuArt adopts various user-friendly visualization designs to help instructors quickly understand the individual and whole learning status. Experimental results on real classroom videos have demonstrated the superiority and robustness of the embedded algorithms. We expect our system promoting the development of large-scale individualized guidance of students. More information is in https://github.com/hnuzhy/StuArt.
A Privacy-Preserving Trajectory Mining Model

In this paper, we consider user privacy issues in location-based social networks (LBSNs). To encourage users to upload personal trajectory data for the benefit of society, we propose a novel privacy-preserving model to protect user trajectory data during the transmission and in the data center (the cloud). The proposed encryption model consists of a Gaussian-whiten encoder and a rotation matrix for the encryption key. The advantage of this model is that the encrypted data can be used for machine learning tasks and generative tasks, even it is not decrypted. We design a feasible interaction strategy to share the model between users and the cloud, and protect the encryption keys from cloud and external eavesdroppers. We apply our model to the trajectory dataset and feed the encrypted data to two classification tasks. Simulation results show that the proposed privacy-preserving model can protect user privacy at an acceptable cost of accuracy loss for classification tasks. Meanwhile, it outperforms classical differential privacy (DP) methods in terms of classifier accuracy loss and privacy preservation ability.
Shuffled Autoregression for Motion Interpolation

This work aims to provide a deep-learning solution for the motion interpolation task. Previous studies solve it with geometric weight functions. Some other works propose neural networks for different problem settings with consecutive pose sequences as input. However, motion interpolation is a more complex problem that takes isolated poses (e.g., only one start pose and one end pose) as input. When applied to motion interpolation, these deep learning methods have limited performance since they do not leverage the flexible dependencies between interpolation frames as the original geometric formulas do. To realize this interpolation characteristic, we propose a novel framework, referred to as Shuffled AutoRegression, which expands the autoregression to generate in arbitrary (shuffled) order and models any inter-frame dependencies as a directed acyclic graph. We further propose an approach to constructing a particular kind of dependency graph, with three stages assembled into an end-to-end spatial-temporal motion Transformer. Experimental results on one of the current largest datasets show that our model generates vivid and coherent motions from only one start frame to one end frame and outperforms competing methods by a large margin. The proposed model is also extensible to multiple keyframes’ motion interpolation tasks and other areas’ interpolation.
Quantifying Catastrophic Forgetting in Continual Federated Learning

The deployment of Federated Learning (FL) systems poses various challenges such as data heterogeneity and communication efficiency. We focus on a practical FL setup that has recently drawn attention, where the data distribution on each device is not static but dynamically evolves over time. This setup, referred to as Continual Federated Learning (CFL), suffers from catastrophic forgetting, i.e., the undesired forgetting of previous knowledge after learning on new data, an issue not encountered with vanilla FL. In this work, we formally quantify catastrophic forgetting in a CFL setup, establish links to training optimization and evaluate different episodic replay approaches for CFL on a large scale real-world NLP dataset. To the best of our knowledge, this is the first such study of episodic replay for CFL. We show that storing a small set of past data boosts performance and significantly reduce forgetting, providing evidence that carefully designed sampling strategies can lead to further improvements.
Backdoor Attack Against Automatic Speaker Verification Models in Federated Learning

Speaker verification has been widely and successfully adopted in many mission-critical fields for user identification. Similar to other classical machine learning tasks, the training performance of speaker verification relies heavily on the number and diversity of data. Therefore third-party data (e.g. data from public dataset benchmarks, the Internet or other cooperating companies) need to be included during the training process. This raises two major questions: How to use multi-party data while ensuring data security and privacy? Whether adopting entrusted third-party data can threaten the security of the speaker verification model? In this paper, we first demonstrate that federated learning (FL) provides an alternative way for training the speaker verification model without collecting data from multi-parties together. We then validate that it is possible to perform backdoor attack under a looser threat assumption, namely poisoning partial speakers instead of all of the speakers. Specifically, we study the security of speaker verification models in FL for the first time. During the training process of FL, we make full use of the advantages of FL, and design a two stage training strategy. Besides, we propose Global Spectral Cluster (GSC) method to alleviate insufficient trigger generalization problem, which cased by the constrain that the attacker can only reach and poison its own data. We also adopt Personalized Federated Aggregation (PFAgg) to avoid modeling pollution in other parties, enhancing the invisibility of backdoor attack in FL. Experimental results on the TIMIT dataset show that our proposed framework can not only achieve satisfying attack results, but also have an acceptable error rate.
Towards Practical Edge Inference Attacks Against Graph Neural Networks

Graph Neural Networks (GNNs) have demonstrated superior performance in numerous real-world applications. Despite their success, recent studies have shown that GNNs are vulnerable under edge inference attacks aimed to infer the connectivity of a given pair of nodes. However, existing methods primarily focus on the scenario when properties of target nodes are revealed. In this paper, we propose an edge inference attack in a more realistic and practical setting. In our threat model, the adversary cannot obtain properties of target nodes but can inject a single probing node and query the target GNN for its prediction. By connecting the probing and target nodes, the adversary can infer the connectivity of the target node pair based on the prediction of the probing node. Extensive experiments show that our attack performs comparably to ones that require properties of target nodes. And when given such auxiliary knowledge, our attack outperforms state-of-the-art methods.
Multi-View K-Means with Laplacian Embedding

Most of the existing multi-view clustering algorithms are performed in the original feature space, and their performance in heavily reliant on the quality of the raw data. Besides, some two-stage strategies cannot achieve ideal results due to the absence of capturing the correlation between views. In view of this, we propose Multi-View K-means with Laplacian Embedding (MVKLE), which is capable of clustering multi-view data in the learned embedding space. Specifically, we employ local structure-preserving dimensionality reduction to obtain the underlying representation of each view, and obtain the clustering results directly through an effective optimization formulation. Experiments on several common multi-view datasets demonstrate the superiority of the proposed method.
A Bayesian Perspective on Noise2Noise: Theory and Extensions

The time and resource costs of obtaining pristine training data in machine learning are high. In signal recovery tasks, Noise2Noise proposed by Lehtinen et al. aims to reduce the data cost by learning the regression over two noisy measurements corresponding to the same latent variable. Close examination shows that Lehtinen’s original derivation requires a strictly "frequentist" conjecture—i.e. deterministic treatment of the latent variable. This paper presents a Bayesian counter-piece to the original Noise2Noise formulation, with a fully stochastic treatment of the latent variable. We propose to extend Noise2Noise further to unbiased estimate of risk (Noise2Noise2MSE), covariance analysis (Noise2Noise2Cov), and minimum mean squared error estimate (Noise2Noise2MMSE), all derived from pairs of noisy measurements only.
Enhancing Unsupervised Speech Recognition with Diffusion GANS

We enhance the vanilla adversarial training method for unsupervised Automatic Speech Recognition (ASR) by a diffusionGAN. Our model (1) injects instance noises of various intensities to the generator’s output and unlabeled reference text which are sampled from pretrained phoneme language models with a length constraint, (2) asks diffusion timestep-dependent discriminators to separate them, and (3) back-propagates the gradients to update the generator. Word/phoneme error rate comparisons with wav2vec-U under Librispeech (3.1% for test-clean and 5.6% for test-other), TIMIT and MLS datasets, show that our enhancement strategies work effectively.
Convergence Analysis of Graphical Game-Based Nash Q−Learning using the Interaction Detection Signal of N−Step Return

The graphical game provides an effective method for modeling different kinds of sparse interactions in multi-agent reinforcement learning. Most previous work on game abstraction lacks theoretical guarantees of convergence. In this paper, we adopt the ${\mathcal{N}}$-step return signal to detect interactions between agents and build the Markov graphical game based on it. We analyze that the solution of the Markov graphical game is an ϵ-Nash equilibrium which guarantees the convergence of the proposed NSR-G <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> NashQ algorithm theoretically. Also, we have done experiments in different multi-agent reinforcement learning tasks with both tabular and function approximation solutions. The results show the NSR-G <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> NashQ algorithm accelerates the convergence of agents to the optimal policy.
Recognizability of Demographically Altered Computerized Facial Approximations in an Automated Facial Recognition Context for Potential Application in Unidentified Persons Data Repositories

This study examined the recognizability of demographically altered facial approximations for potential utility in unidentified persons tracking systems. Five computer-generated approximations were generated for each of 26 African male participants using the following demographic parameters: (i) African male (true demographics), (ii) African female, (iii) Caucasian male, (iv) Asian male, and (v) Hispanic male. Overall, 62% of the true demographic facial approximations for the 26 African male participants examined were matched to a corresponding life photo within the top 50 images of a candidate list generated from an automated blind search of an optimally standardized gallery of 6159 photographs. When the African male participants were processed as African females, the identification rate was 50%. In contrast, less congruent identification rates were observed when the African male participants were processed as Caucasian (42%), Asian (35%), and Hispanic (27%) males. The observed results suggest that approximations generated using the opposite sex may be operationally informative if sex is unknown. The performance of approximations generated using alternative ancestry assignments, however, was less congruent with the performance of the true demographic approximation (African male) and may not yield as operationally constructive data as sex-altered approximations.
Self-Supervised Guided Hypergraph Feature Propagation for Semi-Supervised Classification with Missing Node Features

Graph neural networks (GNNs) with missing node features have recently received increasing interest. Such missing node features seriously hurt the performance of the existing GNNs. Some recent methods have been proposed to reconstruct the missing node features by the information propagation among nodes with known and unknown attributes. Although these methods have achieved superior performance, how to exactly exploit the complex data correlations among nodes to reconstruct missing node features is still a great challenge. To solve the above problem, we propose a self-supervised guided hypergraph feature propagation (SGHFP). Specifically, the feature hypergraph is first generated according to the node features with missing information. And then, the reconstructed node features produced by the previous iteration are fed to a two-layer GNNs to construct a pseudo-label hypergraph. Before each iteration, the constructed feature hypergraph and pseudo-label hypergraph are fused effectively, which can better preserve the higher-order data correlations among nodes. After then, we apply the fused hypergraph to the feature propagation for reconstructing missing features. Finally, the reconstructed node features by multi-iteration optimization are applied to the downstream semi-supervised classification task. Extensive experiments demonstrate that the proposed SGHFP outperforms the existing semi-supervised classification with missing node feature methods.
Efficiently Fusing Sparse Lidar for Enhanced Self-Supervised Monocular Depth Estimation

Monocular self-supervised depth estimation with a low-cost sensor is the mainstream solution to gathering dense depth maps for robots and autonomous driving. In this paper, based on the philosophy "less is more" (i.e., focusing only on valid pixels in sparse LiDAR), we propose a novel framework, Efficient Sparse Depth (EffisDepth), for predicting dense depth. The Sparse Feature Extractor (SFE) embedded in the proposed framework effectively handles sparse LiDAR by forming sparse tensors. The Slender Group Block (SGB) is the main building block in SFE, which extracts features from sparse tensors via a structure of two branches. Extensive experiments show that our method achieves state-of-the-art performance on the KITTI benchmark, demonstrating the effectiveness of each proposed component and the self-supervised learning framework.
Data Driven Joint Sensor Fusion and Regression Based on Geometric Mean Squared Error

This paper explores the problem of estimating a temporal series measured from multiple independent sensors with unequal and stationary measurement errors with unknown variances. By formulating the data fusion problem as a joint Maximum Likelihood estimation of sensor covariances and a fusion rule, a batch data driven method is derived involving a residual covariance determinant minimization of a diagonal matrix. It is shown that yielding useful learning from data with good generalization properties in the joint regression and fusion approach requires the assumption of some structure on the sensor noises and/or on the temporal series to be estimated. An efficient data driven algorithm is proposed to obtain the best linear sensor combiner, whose performance is numerically analyzed and compared with the Cramer-Rao Lower Bound of the estimated parameters.
Dual Path Modeling for Semantic Matching by Perceiving Subtle Conflicts

Transformer-based pre-trained models have achieved great improvements in semantic matching. However, existing models still suffer from insufficient ability to capture subtle differences. The modification, addition and deletion of words in sentence pairs may make it difficult for the model to predict their relationship. To alleviate this problem, we propose a novel Dual Path Modeling Framework to enhance the model’s ability to perceive subtle differences in sentence pairs by separately modeling affinity and difference semantics. Based on dual-path modeling framework we design the Dual Path Modeling Network (DPM-Net) to recognize semantic relations. And we conduct extensive experiments on 10 well-studied semantic matching and robustness test datasets, and the experimental results show that our proposed method achieves consistent improvements over baselines.
Evaluating Parameter-Efficient Transfer Learning Approaches on SURE Benchmark for Speech Understanding

Fine-tuning is widely used as the default algorithm for transfer learning from pre-trained models. Parameter inefficiency can however arise when, during transfer learning, all the parameters of a large pre-trained model need to be updated for individual downstream tasks. As the number of parameters grows, fine-tuning is prone to overfitting and catastrophic forgetting. In addition, full fine-tuning can become prohibitively expensive when the model is used for many tasks. To mitigate this issue, parameter-efficient transfer learning algorithms, such as adapters and prefix tuning, have been proposed as a way to introduce a few trainable parameters that can be plugged into large pre-trained language models such as BERT, HuBERT. In this paper, we introduce the Speech UndeRstanding Evaluation (SURE) benchmark for parameter-efficient learning for various speech processing tasks. Additionally, we introduce a new adapter, ConvAdapter, based on 1D convolution. We show that ConvAdapter outperforms the standard adapters while showing comparable performance against prefix tuning and Low-Rank Adaptation with only 0.94% of trainable parameters.
Dynamic Signed Graph Learning

An important problem in graph signal processing (GSP) is to infer the topology of an unknown graph from a set of observations on the nodes of the graph, i.e. graph signals. Recently, graph learning (GL) approaches have been extended to learn dynamic graphs from temporal graph signals. However, existing work primarily focuses on unsigned graphs and cannot learn signed graphs, which are important data structures that can represent the similarity and dissimilarity of the nodes. In this paper, we propose a dynamic signed GL (dynSGL) method based on the assumptions that (i) at each time point signals are smooth with respect to the signed graph, i.e. signal values at two nodes connected with a positive (negative) edge are similar (dissimilar) and (ii) evolution of the graph structures is smooth across time. The performance of dynSGL is evaluated on simulated data and shown to have higher accuracy compared to static signed and dynamic unsigned GL techniques. Application of the proposed method to a financial dataset gives important insights to the time-varying changes to the interactions between stocks.
Enhancing and Adversarial: Improve ASR with Speaker Labels

ASR can be improved by multi-task learning (MTL) with domain enhancing or domain adversarial training, which are two opposite objectives with the aim to increase/decrease domain variance towards domain-aware/agnostic ASR, respectively. In this work, we study how to best apply these two opposite objectives with speaker labels to improve conformer-based ASR. We also propose a novel adaptive gradient reversal layer for stable and effective adversarial training without tuning effort. Detailed analysis and experimental verification are conducted to show the optimal positions in the ASR neural network (NN) to apply speaker enhancing and adversarial training. We also explore their combination for further improvement, achieving the same performance as i-vectors plus adversarial training. Our best speaker-based MTL achieves 7% relative improvement on the Switchboard Hub5’00 set. We also investigate the effect of such speaker-based MTL w.r.t. cleaner dataset and weaker ASR NN.
Trust Your Partner’s Friends: Hierarchical Cross-Modal Contrastive Pre-Training for Video-Text Retrieval

Video-text retrieval has greatly benefited from the massive web video in recent years, while the performance is still limited to the weak supervision from the uncurated data. In this work, we propose to leverage the well-represented information of each original modality and exploit complementary information in two views of the same video, i.e., video clips and captions, by using one view to obtain positive samples with the neighboring samples of the other. Respecting the hierarchical organization of real-world data, we further design a hierarchical cross-modal pre-training method (HCP) to learn good representations in the common embedding space. We evaluate the pre-trained model on three downstream tasks, i.e. text-to-video retrieval, action step localization and video question answering and our method outperforms previous works under the same setting.