Reviewerly/exHarmony huggingface dataset:
TOPO-MLP : A Simplicial Network without Message Passing

Due to their ability to model meaningful higher order relations among a set of entities, higher order network models have emerged recently as a powerful alternative for graph-based network models which are only capable of modeling binary relationships. Message passing paradigm is still dominantly used to learn representations even for higher order network models. While powerful, message passing can have disadvantages during inference, particularly when the higher order connectivity information is missing or corrupted. To overcome such limitations, we propose Topo-MLP, a purely MLP-based simplicial neural network algorithm to learn the representation of elements in a simplicial complex without explicitly relying on message passing. Our framework utilizes a novel Higher Order Neighborhood Contrastive (HONC) loss which implicitly incorporates the simplicial structure into representation learning. Our proposed model’s simplicity makes it faster during inference. Moreover, we show that our model is robust when faced with missing or corrupted connectivity structure.
Adaptive Large Margin Fine-Tuning For Robust Speaker Verification

Large margin fine-tuning (LMFT) is an effective strategy to improve the speaker verification system’s performance and is widely used in speaker verification challenge systems. Because the large margin in the loss function could make the training task too difficult, people usually use longer training segments to alleviate this problem in LMFT. However, the LMFT model could have a duration mismatch with the real scenario verification, where the verification speech may be very short. In our experiments, we also find that LMFT fails in short duration and other verification scenarios. To solve this problem, we propose the duration-based and similarity-based adaptive large margin fine-tuning (ALMFT) strategy. To verify its effectiveness, we constructed fixed, variable length, and asymmetric verification trials based on VoxCeleb1. Experimental results demonstrate that ALMFT algorithms are very effective and robust, which not only achieve comparable improvement with LMFT in official VoxCeleb evaluation trials but also overcome performance degradation problems in short-duration and asymmetric scenarios respectively.
Regularized EM Algorithm

Expectation-Maximization (EM) algorithm is a widely used iterative algorithm for computing (local) maximum likelihood estimate (MLE). It can be used in an extensive range of problems, including the clustering of data based on the Gaussian mixture model (GMM). Numerical instability and convergence problems may arise in situations where the sample size is not much larger than the data dimensionality. In such low sample support (LSS) settings, the covariance matrix update in the EM-GMM algorithm may become singular or poorly conditioned, causing the algorithm to crash. On the other hand, in many signal processing problems, a priori information can be available indicating certain structures for different cluster covariance matrices. In this paper, we present a regularized EM algorithm for GMM-s that can make efficient use of such prior knowledge as well as cope with LSS situations. The method aims to maximize a penalized GMM likelihood where regularized estimation may be used to ensure positive definiteness of covariance matrix updates and shrink the estimators towards some structured target covariance matrices. We show that the theoretical guarantees of convergence hold, leading to better performing EM algorithm for structured covariance matrix models or with low sample settings.
Improving the Modality Representation with multi-view Contrastive Learning for Multimodal Sentiment Analysis

Modality representation learning is an important problem for multimodal sentiment analysis (MSA), since the highly distinguishable representations can contribute to improving the analysis effect. Previous works of MSA have usually focused on internal fusion strategies for different modalities within one sample, and the external usage of cross reference relations among different samples was given less attention. Recently, the rise of contrastive learning provides powerful clues for us to learn modal representation with stronger discriminative ability. In this study, we explore the approach of representations improvement and devise a three-stages framework with multi-view contrastive learning to refine representations for the specific objectives. Firstly, for each modality, we employ the supervised contrastive learning to pull samples within the same class together while the other samples are pushed apart. Then, a self-supervised contrastive learning is designed for the distilled cross-modal representations after a novel Transformer-based interaction module. At last, we leverage again the supervised contrastive learning to enhance the fused multimodal representation. We conduct extensive experiments on three open datasets, and results show the advance of our model.
Distributed Bayesian Tracking on the Special Euclidean Group Using Lie Algebra Parametric Approximations

This paper proposes new distributed particle filters for tracking the state of a dynamic system that evolves on the Special Euclidean Group. The algorithms are based on the Random Exchange diffusion technique and build compressed parametric approximations to the particles using Lie algebras. Via numerical simulations, we observe that the proposed methods perform similarly to a centralized particle filter, surpassing an extended Kalman filter by a large margin.
Measuring the Transferability of ℓ<sub>∞</sub> Attacks by the ℓ<sub>2</sub> Norm

Deep neural networks could be fooled by adversarial examples with trivial differences to original samples. To keep the difference imperceptible in human eyes, researchers bound the adversarial perturbations by the ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</inf> norm, which is now commonly served as the standard to align the strength of different attacks for a fair comparison. However, we propose that using the ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</inf> norm alone is not sufficient in measuring the attack strength, because even with a fixed ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</inf> distance, the ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf> distance also greatly affects the attack transferability between models. Through the discovery, we reach more in-depth understandings towards the attack mechanism, i.e., several existing methods attack black-box models better partly because they craft perturbations with 70% to 130% larger ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf> distances. Since larger perturbations naturally lead to better transferability, we thereby advocate that the strength of attacks should be simultaneously measured by both the ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</inf> and ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf> norm. Our proposal is firmly supported by extensive experiments on ImageNet dataset from 7 attacks, 4 white-box models, and 9 black-box models.
ICEL: Learning with Inconsistent Explanations

Generating the heatmaps is one of the explanation methods to show what regions the model use to predict in vision tasks. GradCAM is a popular approach to provide such heatmaps. However, GradCAM is post-hoc, and its heatmaps are not always meeting human annotations. Inspired by CGC (Contrastive GradCAM consistency), we propose ICEL (InConsistent Explanation Learning) method which introduces inconsistent explanation loss measured by cosine similarity on heatmaps. We show that our method can preserve classification accuracy, while the heatmaps generated by explanation methods are more consistent with human annotations, and the computational complexity is reduced from O(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) to O(n), compared with CGC.
Relational Representation Learning for Zero-Shot Relation Extraction with Instance Prompting and Prototype Rectification

Zero-shot relation extraction aims to extract novel relations that are not observed beforehand. However, existing representation methods are not pre-trained for relational representations and embeddings contain much linguistic information, the distances between them are not consistent with relational semantic similarity. In this paper, we propose a novel method based on Instance Prompting and Prototype Rectification (IPPR) to conduct relational representation learning for zeroshot relation extraction. Instance prompting is designed to reduce the gap between pre-training and fine-tuning, and guide the pre-trained model to generate relation-oriented instance representations. Prototype rectification aims to push the prototype embeddings away from each other and makes the instance embeddings closer to its corresponding prototype embeddings for dynamically rectifying the prototype embeddings. Experimental results on two public datasets demonstrate that our proposed method achieves new state-of-the-arts performance <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Weakly-Supervised Scene-Specific Crowd Counting Using Real-Synthetic Hybrid Data

Due to the domain gap between the public large-scale datasets and actual scenes, the crowd counting models trained on the common datasets have a significant performance degradation when applying in practical applications. To address the above issue, one of the solution is to label additional data from the novel scenes, which is time-consuming and impractical for multiple scenes. Another solution is to utilize domain adaptation approaches to adapt a well-trained model to novel scenes. However, most of these approaches focus on appearance adaptation while the background and the crowd distribution is not adapted. In this paper, we propose a weakly-supervised method with real-synthetic hybrid data which only requires a small portion of unlabelled real images and auto-generated synthetic labelled images for training. First, the hybrid data is generated based on background from the real scene and random distributed synthetic persons. Second, an initialized counter is trained based on the hybrid data and the crowd distribution is predicted based on the predictions on real images. Then, a better crowd counter is trained based on new hybrid data generated from updated crowd distribution. The process is iterated until convergence. Extensive experiments demonstrate the effectiveness of the proposed method.
Conditioning and Sampling in Variational Diffusion Models for Speech Super-Resolution

Recently, diffusion models (DMs) have been increasingly used in audio processing tasks, including speech super-resolution (SR), which aims to restore high-frequency content given low-resolution speech utterances. This is commonly achieved by conditioning the network of noise predictor with low-resolution audio. In this paper, we propose a novel sampling algorithm that communicates the information of the low-resolution audio via the reverse sampling process of DMs. The proposed method can be a drop-in replacement for the vanilla sampling process and can significantly improve the performance of the existing works. Moreover, by coupling the proposed sampling method with an unconditional DM, i.e., a DM with no auxiliary inputs to its noise predictor, we can generalize it to a wide range of SR setups. We also attain state-of-the-art results on the VCTK Multi-Speaker benchmark with this novel formulation.
Receptive Field Reliant Zero-Cost Proxies for Neural Architecture Search

Neural Architecture Search (NAS) is a fast growing technology for automatic design of deep-learning architectures. NAS includes three stages: search space design, search strategy, and evaluation criterion. Among these, the evaluation of various architectures is very cost-intensive task. In this work, we have proposed a set of receptive field reliant zero-cost proxies which need only one iteration of training and thereby reduce the computational time associated with evaluation criterion during the NAS. The proposed zero-cost proxies are based on layer-wise binding of the prune-at-initialization score with its receptive field for more effective measure as compared to the vanilla counterparts to achieve generalizability. The proposed zero-cost proxies are validated on the set of PyTorchCV models, and NAS-Bench-201 benchmarking datasets. The proposed zero-cost proxies have performed better for set of PyTorchCV models and competitively with vanilla counterparts for NAS-Bench-201. The efficiency of the proposed method is also demonstrated in NAS on NAS-Bench-201 using Aging Evolution as controller.
Emodiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidance

Although current neural text-to-speech (TTS) models are able to generate high-quality speech, intensity controllable emotional TTS is still a challenging task. Most existing methods need external optimizations for intensity calculation, leading to suboptimal results or degraded quality. In this paper, we propose EmoDiff, a diffusion-based TTS model where emotion intensity can be manipulated by a proposed soft-label guidance technique derived from classifier guidance. Specifically, instead of being guided with a one-hot vector for the specified emotion, EmoDiff is guided with a soft label where the value of the specified emotion and Neutral is set to α and 1 − α respectively. The α here represents the emotion intensity and can be chosen from 0 to 1. Our experiments show that EmoDiff can precisely control the emotion intensity while maintaining high voice quality. Moreover, diverse speech with specified emotion intensity can be generated by sampling in the reverse denoising process.
Spectral Clustering-Aware Learning of Embeddings for Speaker Diarisation

In speaker diarisation, speaker embedding extraction models often suffer from the mismatch between their training loss functions and the speaker clustering method. In this paper, we propose the method of spectral clustering-aware learning of embeddings (SCALE) to address the mismatch. Specifically, besides an angular prototypical (AP) loss, SCALE uses a novel affinity matrix loss which directly minimises the error between the affinity matrix estimated from speaker embeddings and the reference. SCALE also includes p-percentile thresholding and Gaussian blur as two important hyper-parameters for spectral clustering in training. Experiments on the AMI dataset showed that speaker embeddings obtained with SCALE achieved over 50% relative speaker error rate reductions using oracle segmentation, and over 30% relative diarisation error rate reductions using automatic segmentation when compared to a strong baseline with the AP-loss-based speaker embeddings.
A Study on Partial Shape Matching between Flake Surface and Surface of Joining Material using Measured Point Cloud

石器とは，先史時代に作成・使用されていた，生活のための道具である. 出土した石器や剥片を接合して, 母岩を再現したものを石器の接合資料と呼ぶ．接合資料を発掘調査報告書に記載する場合，実測して実測図を作成するか，または3D計測した点群に基づいて図化するが，接合資料を調査・研究するには，レーザー計測に基づいた接合資料表面の点群だけでは不十分で，内部の情報を持つ完全な接合資料の三次元点群が必要になる．完全な接合資料を生成するためには，接合資料を計測して得られた表面点群と， 接合資料を構成する石器の点群とのマッチングが必要である．しかし，接合資料を計測して得られた表面点群は，一部の石器が他の石器に部分的に隠されており，表面点群と石器剥離面を表す点群とをマッチングさせる場合，石器の一部分に着目してマッチングできる手法が要求される．そこで本稿では，接合資料を計測して得られた表面点群と，石器剥離面を表す点群との部分マッチングを行う，新しい手法を提案する．本手法により，接合資料表面点群と，石器の計測点群から，接合資料を構成する石器の空間配置を復元することが可能となる．
TrimTail: Low-Latency Streaming ASR with Simple But Effective Spectrogram-Level Length Penalty

In this paper, we present TrimTail, a simple but effective emission regularization method to improve the latency of streaming ASR models. The core idea of TrimTail is to apply length penalty (i.e., by trimming trailing frames, see Fig. 1-(b)) directly on the spectrogram of input utterances, which does not require any alignment. We demonstrate that TrimTail is computationally cheap and can be applied online and optimized with any training loss or any model architecture on any dataset without any extra effort by applying it on various end-to-end streaming ASR networks either trained with CTC loss [1] or Transducer loss [2]. We achieve 100 ~ 200ms latency reduction with equal or even better accuracy on both Aishell-1 and Librispeech. Moreover, by using TrimTail, we can achieve a 400ms algorithmic improvement of User Sensitive Delay (USD) with an accuracy loss of less than 0.2.
Sample-Adapt Fusion Network for RGB-D Hand Detection in the Wild

RGB and depth modalities provide complementary information, which can be effectively utilized to improve the performance of hand detection in the wild. Most existing fusion-based methods model the channel-wise or spatial-wise cross-modal correlation to exploit the complementary RGB-D information, in which the modeling operations are shared across all input samples. However, the input images show various modes due to the high diversity of scenes in the wild. This inter-sample variance cannot be effectively perceived by static modeling operations shared across all samples. To address this problem, we propose a Sample-Adapt Fusion Network (SAFNet) with Channel Dynamic Refinement Module (CDRM) and Spatial Dynamic Aggregation Module (SDAM) to adaptively model the channel-wise and spatial-wise cross-modal correlation. Specifically, we propose a Multi-kernel Attention Module (MAM) to individually generate attention maps for each input sample by applying learnable weighting operations to multiple convolutional kernels. Our method outperforms state-of-the-art methods on CUG Hand dataset.
Adaptive Non-Local Generative Adversarial Networks for Low-Dose CT Image Denoising

Low-dose computed tomography (CT) has been widely used in medical diagnosis and treatment. Many deep networks have been proposed for low-dose CT denoising. The local receptive field of the convolution affects the network performance. For different input images, conventional neural networks always adopt a fixed number of channels which limits the performance of deep networks. To address these problems, we propose a channel-adaptive convolution and patch selection (CAPS) module to enhance the feature extraction of our network. CAPS enables our network to adaptively adjust the number of channels according to different inputs. Moreover, the concatenation of patches can expand the receptive field globally, so the shallow layer of our network can extract more global information. To further ensure the clarity of denoised images, we present a new wavelet loss function to the generator of our generative adversarial network. Compared with state-of-the-art methods, our network can obtain superior denoising results.
Cocktail Hubert: Generalized Self-Supervised Pre-Training for Mixture and Single-Source Speech

Self-supervised learning leverages unlabeled data effectively, improving label efficiency and generalization to domains without labeled data. While recent work has studied generalization to more acoustic/linguistic domains, languages, and modalities, these investigations are limited to single-source speech with one primary speaker in the recording. This paper presents Cocktail HuBERT, a self-supervised learning framework that generalizes to mixture speech using a masked pseudo source separation objective. This objective encourages the model to identify the number of sources, separate and understand the context, and infer the content of masked regions represented as discovered units. Cocktail HuBERT outperforms state-of-the-art results with 69% lower WER on multi-speaker ASR, 31% lower DER on diarization, and is competitive on single-and multi-speaker tasks from SUPERB.
Variable Attention Masking for Configurable Transformer Transducer Speech Recognition

This work studies the use of attention masking in transformer transducer based speech recognition for building a single configurable model for different deployment scenarios. We present a comprehensive set of experiments comparing fixed masking, where the same attention mask is applied at every frame, with chunked masking, where the attention mask for each frame is determined by chunk boundaries, in terms of recognition accuracy and latency. We then explore the use of variable masking, where the attention masks are sampled from a target distribution at training time, to build models that can work in different configurations. Finally, we investigate how a single configurable model can be used to perform both first pass streaming recognition and second pass acoustic rescoring. Experiments show that chunked masking achieves a better accuracy vs latency trade-off compared to fixed masking, both with and without FastEmit. We also show that variable masking improves the accuracy by up to 8% relative in the acoustic re-scoring scenario.
CPA: Compressed Private Aggregation for Scalable Federated Learning Over Massive Networks

Federated learning (FL) allows a central server to train a model using remote users’ data. FL faces challenges in preserving the local datasets privacy and in its communication overhead; which is considerably dominant in large-scale networks. These limitations are often mitigated individually by local differential privacy (LDP) mechanisms, compression, and user-selection techniques, which often come at the cost of accuracy. In this work we present compressed private aggregation (CPA), which allows massive deployments to simultaneously communicate at extremely low bit-rates while achieving privacy, anonymity, and resilience to malicious users. CPA randomizes a code-book for compressing the data into a few bits, ensuring anonymity and robustness, with a subsequent perturbation to hold LDP. We provide both a theoretical analysis and a numerical study, demonstrating the performance gains of CPA compared with separate mechanisms for compression and privacy.
Prompt Makes mask Language Models Better Adversarial Attackers

Generating high-quality synonymous perturbations is a core challenge for textual adversarial tasks. However, candidates generated from the masked language model often contain many words that are antonyms or irrelevant to the original words, which limit the perturbation space and affect the attack’s effectiveness. We present ProAttacker <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> which uses Prompt to make the mask language models better adversarial Attackers. ProAttacker inverts the prompt paradigm by leveraging the prompt with the class label to guide the language model to generate more semantically-consistent perturbations. We present a systematic evaluation to analyze the attack performance on 6 NLP datasets, covering text classification and inference. Our experiments demonstrate that ProAttacker outperforms state-of-the-art attack strategies in both success rate and perturb rate.
NCL: Textual Backdoor Defense Using Noise-Augmented Contrastive Learning

At present, backdoor attacks attract attention as they do great harm to deep learning models. By poisoning the training data, the adversary makes the model trained based on this dataset being injected with a backdoor. In the field of text, however, existing works do not provide sufficient defense against backdoor attacks. In this paper, we propose a Noise-augmented Contrastive Learning (NCL) framework to defend against textual backdoor attacks when training models with untrustworthy data. With the aim of mitigating the mapping between triggers and the target label, we add appropriate noise perturbing possible backdoor triggers, augment the training dataset, and then pull homology samples in the feature space utilizing contrastive learning objective. Experiments demonstrate the effectiveness of our method in defending three types of textual backdoor attacks, outperforming the prior works.