Reviewerly/exHarmony huggingface dataset:
Earning Extra Performance from Restrictive Feedbacks

Many machine learning applications encounter situations where model providers are required to further refine the previously trained model so as to gratify the specific need of local users. This problem is reduced to the standard model tuning paradigm if the target data is permissibly fed to the model. However, it is rather difficult in a wide range of practical cases where target data is not shared with model providers but commonly some evaluations about the model are accessible. In this paper, we formally set up a challenge named Earning eXtra PerformancE from restriCTive feEDdbacks (EXPECTED) to describe this form of model tuning problems. Concretely, EXPECTED admits a model provider to access the operational performance of the candidate model multiple times via feedback from a local user (or a group of users). The goal of the model provider is to eventually deliver a satisfactory model to the local user(s) by utilizing the feedbacks. Unlike existing model tuning methods where the target data is always ready for calculating model gradients, the model providers in EXPECTED only see some feedbacks which could be as simple as scalars, such as inference accuracy or usage rate. To enable tuning in this restrictive circumstance, we propose to characterize the geometry of the model performance with regard to model parameters through exploring the parameters' distribution. In particular, for deep models whose parameters distribute across multiple layers, a more query-efficient algorithm is further tailor-designed that conducts layerwise tuning with more attention to those layers which pay off better. Our theoretical analyses justify the proposed algorithms from the aspects of both efficacy and efficiency. Extensive experiments on different applications demonstrate that our work forges a sound solution to the EXPECTED problem, which establishes the foundation for future studies towards this direction.
Macroscopic-and-Microscopic Rain Streaks Disentanglement Network for Single-Image Deraining

Single-image deraining aims to restore the image that is degraded by the rain streaks, where the long-standing bottleneck lies in how to disentangle the rain streaks from the given rainy image. Despite the progress made by substantial existing works, several crucial questions - e.g., How to distinguish rain streaks and clean image, while how to disentangle rain streaks from low-frequency pixels, and further prevent the blurry edges - have not been well investigated. In this paper, we attempt to solve all of them under one roof. Our observation is that rain streaks are bright stripes with higher pixel values that are evenly distributed in each color channel of the rainy image, while the disentanglement of the high-frequency rain streaks is equivalent to decreasing the standard deviation of the pixel distribution for the rainy image. To this end, we propose a self-supervised rain streaks learning network to characterize the similar pixel distribution of the rain streaks from a macroscopic viewpoint over various low-frequency pixels of gray-scale rainy images, coupling with a supervised rain streaks learning network to explore the specific pixel distribution of the rain streaks from a microscopic viewpoint between each paired rainy and clean images. Building on this, a self-attentive adversarial restoration network comes up to prevent the further blurry edges. These networks compose an end-to-end Macroscopic-and-Microscopic Rain Streaks Disentanglement Network, named [Formula: see text]RSD-Net, to learn rain streaks, which is further removed for single image deraining. The experimental results validate its advantages on deraining benchmarks against the state-of-the-arts. The code is available at: https://github.com/xinjiangaohfut/MMRSD-Net.
Behavioral Predictors of Intention to Use a Text Messaging Reminder System Among People Living With HIV in Rural Uganda: Survey Study

The expansion of cellular phones in sub-Saharan Africa spurred the development of SMS text message-based mobile health (mHealth) technology. Numerous SMS text message-based interventions have attempted to increase retention in care for people living with HIV in sub-Saharan Africa. Many of these interventions have failed to scale. Understanding theory-grounded factors leading to mHealth acceptability is needed to create scalable, contextually appropriate, and user-focused interventions to improve longitudinal HIV care for people living with HIV in sub-Saharan Africa.In this study, we aimed to understand the relationship between constructs from the Unified Theory of Acceptance and Use of Technology (UTAUT), constructs identified in previous qualitative research, and behavioral intention to use a novel SMS text message-based mHealth intervention designed to improve care retention among people living with HIV initiating treatment in rural Uganda.We conducted a survey of people living with HIV who were newly initiating HIV care in Mbarara, Uganda, and had agreed to use a novel SMS text message-based system that notified them of abnormal laboratory results and reminded them to return to the clinic. Survey items assessed behavioral intention to use the SMS text messaging system; constructs from UTAUT; and demographics, literacy, SMS text messaging experience, HIV status disclosure, and social support. We used factor analysis and logistic regression to estimate the relationships between UTAUT constructs and the behavioral intention to use the SMS text messaging system.A total of 249 participants completed the surveys, of whom 115 (46.2%) expressed high behavioral intention to use the SMS text messaging intervention. In a multivariable analysis, we found that performance expectancy (adjusted odds ratio [aOR] of the scaled factor score 5.69, 95% CI 2.64-12.25; P<.001), effort expectancy (aOR of the scaled factor score 4.87, 95% CI 1.75-13.51; P=.002), and social influence (measured as a 1-unit Likert score increase in the perception that clinical staff have been helpful in the use of the SMS text messaging program; aOR 3.03, 95% CI 1.21-7.54; P=.02) were significantly associated with high behavioral intention to use the SMS text messaging program. SMS text messaging experience (aOR/1-unit increase 1.48, 95% CI 1.11-1.96; P=.008) and age (aOR/1-year increase 1.07, 95% CI 1.03-1.13; P=.003) were also significantly associated with increased odds of high intention to use the system.Performance expectancy, effort expectancy, and social influence, as well as age and SMS experience, were drivers of high behavioral intention to use an SMS text messaging reminder system among people living with HIV initiating treatment in rural Uganda. These findings highlight salient factors associated with SMS intervention acceptability in this population and indicate attributes that are likely to be key to the successful development and scaling of novel mHealth interventions.
Biologically-Inspired Continual Learning of Human Motion Sequences

This work proposes a model for continual learning on tasks involving temporal sequences, specifically, human motions. It improves on a recently proposed brain-inspired replay model (BI-R) by building a biologically-inspired conditional temporal variational autoencoder (BI-CTVAE), which instantiates a latent mixture-of-Gaussians for class representation. We investigate a novel continual-learning-to-generate (CL2Gen) scenario where the model generates motion sequences of different classes. The generative accuracy of the model is tested over a set of tasks. The final classification accuracy of BI-CTVAE on a human motion dataset after sequentially learning all action classes is 78%, which is 63% higher than using no-replay, and only 5.4% lower than a state-of-the-art offline trained GRU model.
Generic Dependency Modeling for Multi-Party Conversation

To model the dependencies between utterances in multi-party conversations, we propose a simple and generic framework based on the dependency parsing results of utterances. Particularly, we present an approach to encoding the dependencies in the form of relative dependency encoding (ReDE) and illustrate how to implement it in Transformers by modifying the computation of self-attention. Experimental results on four multi-party conversation benchmarks show that this framework successfully boosts the general performance of two Transformer-based language models and leads to comparable or even superior performance compared to the state-of-the-art methods. The code is available at https://github.com/shenwzh3/ReDE.
I-Tuning: Tuning Frozen Language Models with Image for Lightweight Image Captioning

Image Captioning is a traditional vision-and-language task that aims to generate the language description of an image. Recent studies focus on scaling up the model size and the number of training data, which significantly increase the cost of model training. Different to these heavy-cost models, we introduce a lightweight image captioning framework (I-Tuning), which contains a small number of trainable parameters. We design a novel I-Tuning cross-attention module to connect the non-trainable pre-trained language decoder GPT2 and vision encoder CLIP-ViT. Since most parameters are not required to be updated during training, our framework is lightweight and fast. Experimental results conducted on three image captioning benchmarks reveal that our frame-work achieves comparable or better performance than the large-scale baseline systems. But our models contain up to 10 times fewer trainable parameters and require much fewer data for training compared with state-of-the-art baselines.
Modaldrop: Modality-Aware Regularization for Temporal-Spectral Fusion in Human Activity Recognition

Although most of existing works for sensor-based Human Activity Recognition rely on the temporal view, we argue that the spectral view also provides complementary prior and accordingly benchmark a standard multi-view framework with extensive experiments to demonstrate its consistent superiority over single-view opponents. We then delve into the intrinsic mechanism of the multi-view representation fusion, and propose ModalDrop as a novel modality-aware regularization method to learn and exploit representations of both views effectively. We demonstrate its advantage over existing representation fusion alternatives with comprehensive experiments and ablations. The improvements are consistent for various settings and are orthogonal with different backbones. We also discuss its potential application for other related tasks regarding representation or modality fusion. The source code is available on https://github.com/studyzx/ModalDrop.git.
Gated Enhanced RPN and Hybrid-View for Few-Shot Object Detection

Few-Shot Object Detection (FSOD) is designed to detect unseen objects using a few examples. Matching between query image and support instance via metric learning has been shown to be an effective FSOD method. In previous work, the quality of proposals generated by the RPN is not yet high due to the lack of a fine-grained matching strategy, and the detector performs feature integration only at the spatial location level is limited. Therefore, in this paper, we propose a new method for few-shot object detection which obtains high-quality proposals by the Gated Enhanced RPN (GRPN). To further improve the detection performance from different views, we propose a Hybrid-View Detector to categorize proposals more comprehensively. We perform extensive experiments on the MS-COCO benchmark and the experimental results demonstrate the effectiveness of the proposed method and outperform the state-of-the-art on most shot cases.
Towards Explainable Recommendation Via Bert-Guided Explanation Generator

Explainable recommender system has recently drawn increasing attention due to its capability of providing justification to recommendation. Rather than focusing on certain topics or specific item features, the explanation generated by existing works are too general without the guidance of aspects. However, such information is not given in the practical scenario. To address this issue, we propose a novel Explainable recommender system with BERT-guided explanation generator, named ExBERT to generate reliable explanation with finer granularity. More specifically, a multi-head self-attention based encoder is employed to incorporate pseudo user and item profiles into semantic representation. Moreover, we propose a novel matched explanation prediction task with discriminative ability to enable personalization of the generated sentence. Extensive experiments conducted on two real-world explainable recommendation datasets significantly outperform the state-of-the-art in generation.
Long-Tailed Recognition with Causal Invariant Transformation

Standard classification models rely on the assumption that all the classes of interest are equally represented in training datasets. However, visual phenomena exhibit a long-tailed distribution, such that many standard approaches fail to properly model and result in a considerable degeneration on accuracy. The recent methods have produced encouraging results, but their efforts only seek to simulate the statistical relationship between data and labels and compensate for imbalanced data-related issues, without addressing the underlying causal mechanisms. In this paper, a comprehensive structural causal model is developed to excavate the intrinsic causal mechanism between data and labels. Specifically, we assume that each input is constructed from a mix of causal factors and non-causal factors, and only the causal factors cause the classification judgments. In order to extract such causal factors from inputs and then reconstruct the invariant causal mechanisms, we propose a Causal Invariant Transformation algorithm for Long-tailed recognition (CITL), which generates diverse data to avoid the over-fitting on the tail classes and enforces the learnt representations to maintain the causal factors and eliminate the non-causal factors. Our extensive experimental results on several widely used datasets have demonstrated the effectiveness of our proposed CITL approach.
Privacy-Preserving Automatic Speaker Diarization

Automatic Speaker Diarization (ASD) is an enabling technology with numerous applications, which deals with recordings of multiple speakers, raising special concerns in terms of privacy. In fact, in remote settings, where recordings are shared with a server, clients relinquish not only the privacy of their conversation, but also of all the information that can be inferred from their voices. However, to the best of our knowledge, the development of privacy-preserving ASD systems has been overlooked thus far. In this work, we tackle this problem using a combination of two cryptographic techniques, Secure Multiparty Computation (SMC) and Secure Modular Hashing, and apply them to the two main steps of a cascaded ASD system: speaker embedding extraction and agglomerative hierarchical clustering. Our system is able to achieve a reasonable trade-off between performance and efficiency, presenting real-time factors of 1.1 and 1.6, for two different SMC security settings.
Federated Learning for ASR Based on wav2vec 2.0

This paper presents a study on the use of federated learning to train an ASR model based on a wav2vec 2.0 model pre-trained by self supervision. Carried out on the well-known TED-LIUM 3 dataset, our experiments show that such a model can obtain, with no use of a language model, a word error rate of 10.92% on the official TEDLIUM 3 test set, without sharing any data from the different users. We also analyse the ASR performance for speakers depending to their participation to the federated learning. Since federated learning was first introduced for privacy purposes, we also measure its ability to protect speaker identity. To do that, we exploit an approach to analyze information contained in exchanged models based on a neural network footprint on an indicator dataset. This analysis is made layer-wise and shows which layers in an exchanged wav2vec 2.0based model bring the speaker identity information.
An Auto-Encoder Based Method for Camera Fingerprint Compression

Camera fingerprint links a picture to its camera sensor, which is widely applied in sensor device identification, social network tracing and forgery detection. However, such fingerprints are in high dimensionality and cost substantial memory and computing resources, limiting their uses in real-time processing on embedded devices. In this paper, we introduce a new method to compress high-dimensional floating-point fingerprints to low-dimensional binary features to save storage as well as maintaining their representative abilities. Also, we present a much faster approach to sensor device matching with hamming distance, compared with the commonly used Peak to Correlation Energy (PCE) distance. Our method contains two stages. First, raw fingerprints are compressed into low-dimensional features with our proposed grouping strategy and auto-encoder based model. Then, the compressed floating-point features are further converted into more compact binary features. Experiments show that our method achieves superior performance over several competitive compression methods in both identification and verification tasks.
Intermpl: Momentum Pseudo-Labeling With Intermediate CTC Loss

This paper presents InterMPL, a semi-supervised learning method of end-to-end automatic speech recognition (ASR) that performs pseudo-labeling (PL) with intermediate supervision. Momentum PL (MPL) trains a connectionist temporal classification (CTC)-based model on unlabeled data by continuously generating pseudo-labels on the fly and improving their quality. In contrast to autoregressive formulations, such as the attention-based encoder-decoder and transducer, CTC is well suited for MPL, or PL-based semi-supervised ASR in general, owing to its simple/fast inference algorithm and robustness against generating collapsed labels. However, CTC generally yields inferior performance than the autoregressive models due to the conditional independence assumption, thereby limiting the performance of MPL. We propose to enhance MPL by introducing intermediate loss, inspired by the recent advances in CTC-based modeling. Specifically, we focus on self-conditional and hierarchical conditional CTC, that apply auxiliary CTC losses to intermediate layers such that the conditional independence assumption is explicitly relaxed. We also explore how pseudo-labels should be generated and used as supervision for intermediate losses. Experimental results in different semi-supervised settings demonstrate that the proposed approach outperforms MPL and improves an ASR model by up to a 12.1% absolute performance gain. In addition, our detailed analysis validates the importance of the intermediate loss.
A Game of Snakes and Gans

Generative adversarial networks (GANs) comprise generator and discriminator networks trained adversarially to learn the underlying distribution of a dataset. Recently, we have shown that the optimal GAN discriminator can be obtained in closed-form as the solution to the Poisson partial differential equation (PDE). While existing approaches either train a network or solve the PDE in closed-form, we propose training the generator through the gradient field of the optimal discriminator. In this paper, we establish a connection between active contour models (snakes) and GANs. We evolve a set of snake points over the gradient field of radial basis function (RBF) Coulomb GAN. The generator is then trained to follow the trajectory of the snake. The proposed approach benefits from both the sample diversity seen in flow-based approaches and the fast sampling capability of GANs. Experimental validation on 2-D synthetic data shows that the proposed approach leads to accelerated convergence, compared against the baseline approaches that either employ network or kernel-based discriminators.
SLBERT: A Novel Pre-Training Framework for Joint Speech and Language Modeling

We propose SLBERT (Speech and Language pre-training framework for BERT), an end-to-end trainable framework for learning joint representations of speech and language modalities. We enhance the well-known BERT architecture to provide a dual-stream multimodal architecture that processes both speech and language input. To enable effective information exchange between the two modalities, we introduce a novel attention fusion mechanism via AF-Blocks. To acquire robust contrastive representations for speech and language processing applications, we pre-train SLBERT on three auxiliary tasks: Masked Language Modeling, Masked Speech Modeling, and Speech-Language Matching. We evaluate our proposed model on two well-known multimodal tasks: intent classification and sentiment analysis. Our model achieves state-of-the-art results on both benchmarks while surpassing even larger baselines.
Cross-Modal Matching and Adaptive Graph Attention Network for RGB-D Scene Recognition

Despite the significant advances in RGB-D scene recognition, there are several major limitations that need further investigation. For example, simply extracting modal-specific features neglects the complex relationships among multiple modalities of features. Moreover, cross-modal features have not been considered in most existing methods. To address these concerns, we propose to integrate the tasks of cross-modal matching and modal-specific recognition, termed as Matching-to-Recognition Network (MRNet). Specifically, the cross-modal matching network enhances the descriptive power of the recognition network via a layer-wise semantic loss. The recognition network obtains multi-modal features from a two-stream CNN: global features are obtained by a higher-layer of a CNN to preserve the semantic content, and local layout features are learned by the graph attention network, thus better capturing the key object regions and modelling their relationships. Extensive experiments results demonstrate the MRNet achieves superior performance to state-of-the-art methods, especially for recognition solely based on single modality.
Boosting Signal Modulation Few-Shot Learning with Pre-Transformation

The recent flourish of deep learning on various tasks is largely accredited to the rich and high-quality labeled data. Nonetheless, collecting sufficient labeled samples is not very practical for many real applications. Few-shot Learning (FSL) provides a promising solution that allows a model to learn the concept of novel classes with a few labeled samples. However, many existing FSL methods are only designed for computer vision tasks and are not suitable for radio signal recognition. This paper calls for a radically different approach to FSL: in contrast to developing a new FSL model, we should focus on transforming the radio signal to be better processed by the state-of-the-art (SOTA) FSL model. We propose Modulated Signal Pre-transformation (MSP), a parameterized radio signal transformation framework that encourages the signals having the same semantics to have similar representations. MSP currently adapts to various SOTA FSL models for signal modulation recognition and can support the mainstream deep learning backbone. Evaluation results show that MSP improves the performance gains for many SOTA FSL models while maintaining flexibility.
Measure and Countermeasure of the Capsulation Attack Against Backdoor-Based Deep Neural Network Watermarks

Backdoor-based watermarking schemes were proposed to protect the intellectual property of deep neural networks under the black-box setting. However, additional security risks emerge after the schemes have been published for as forensics tools. This paper reveals the capsulation attack that can easily invalidate most established backdoor-based watermarking schemes without sacrificing the pirated model’s functionality. By encapsulating the deep neural network with a filter, an adversary can block abnormal queries and reject the ownership verification. We propose a metric to measure a backdoor-based watermarking scheme’s security against the capsulation attack, and design a new backdoor-based deep neural network watermarking scheme that is secure against the capsulation attack by inverting the encoding process.