Reviewerly/exHarmony huggingface dataset:
Scalable and Secure Federated XGBoost

Federated learning (FL) is the distributed machine learning framework that enables collaborative training across multiple parties while ensuring data privacy. Practical adaptation of XGBoost, the state-of-the-art tree boosting framework, to FL remains nascent due to high cost incurred by conventional privacy-preserving methods. Such limitations can be attributed to the lack of formal analytical model to enable new privacy methods well customized to federated XGBoost. To this end, we propose a novel formulation, termed splitting matrix, in the context of federated XGBoost that mathematically characterizes the role of passive party (PP) having been neglected in the literature. This new formulation facilitates our novel adoption of secure matrix multiplication protocol into federated XGBoost to propose FedXGBoost as a framework for secure XGBoost in federated setting with lossless accuracy and negligible overhead. Extensive experiments on both synthetic and real datasets exhibit our algorithm’s empirical outperformance over known methods in the literature.
Interweaved Graph and Attention Network for 3D Human Pose Estimation

Despite substantial progress in 3D human pose estimation from a single-view image, prior works rarely explore global and local correlations, leading to insufficient learning of human skeleton representations. To address this issue, we propose a novel Interweaved Graph and Attention Network (IGANet) that allows bidirectional communications between graph convolutional networks (GCNs) and attentions. Specifically, we introduce an IGA module, where attentions are provided with local information from GCNs and GCNs are injected with global information from attentions. Additionally, we design a simple yet effective U-shaped multi-layer perceptron (uMLP), which can capture multi-granularity information for body joints. Extensive experiments on two popular benchmark datasets (i.e. Human3.6M and MPI-INF-3DHP) are conducted to evaluate our proposed method. The results show that IGANet achieves state-of-the-art performance on both datasets. Code is available at https://github.com/xiu-cs/IGANet.
Federated Intelligent Terminals Facilitate Stuttering Monitoring

Stuttering is a complicated language disorder. The most common form of stuttering is developmental stuttering, which begins in childhood. Early monitoring and intervention are essential for the treatment of children with stuttering. Automatic speech recognition technology has shown its great potential for non-fluent disorder identification, whereas the previous work has not considered the privacy of users’ data. To this end, we propose federated intelligent terminals for automatic monitoring of stuttering speech in different contexts. Experimental results demonstrate that the proposed federated intelligent terminals model can analyze symptoms of stammering speech by taking personal privacy protection into account. Furthermore, the study has explored that the Shapley value approach in the federated learning setting has comparable performance to data-centralised learning.
Efficient Practices for Profile-to-Frontal Face Synthesis and Recognition

Despite the great progress of deep learning and generative adversarial networks, face frontalization (i.e., profile-to-frontal synthesis) and profile (i.e., non-frontal) face recognition still remain challenging tasks under uncontrolled environments. In this study, we propose three efficient practices to improve the performance of profile-to-frontal face synthesis and recognition. Firstly, the identity preserving module is embedded to constrain synthesized frontal images similar to true frontal faces in feature space. Secondly, facial consistency loss is employed to reduce the artifact of the generated frontal face in pixel space. Lastly, the multi-model embedded scheme enhances the representation learning through diverse facial features extracted by multiple facial feature extractors. The proposed practices are general, though specifically deployed to CR-GAN in this study for performance verification. Experimental results on Multi-PIE and VGGFace2 demonstrate that the proposed practices qualitatively generate more realistic photography frontal faces and quantitatively obtain better face recognition accuracy.
Classification of Synthetic Facial Attributes by Means of Hybrid Classification/Localization Patch-Based Analysis

Facial attributes editing, that is the manipulation of some specific attributes of a face image, is a new trend in the generation of synthetic images by GANs. Several recent studies have shown the possibility to detect the synthetic nature of such images by training a DL-based binary classifier. At the same time, the question about the specific face attributes that have been altered is typically disregarded, yet this may be a crucial information for forensic analysts. In this paper, we propose a new architecture whose objective is to identify the altered facial attributes of synthetic face images. To do so, we developed a hybrid classification-and-localization architecture. The local and global features are first extracted from the full image and from specific image patches, and then merged by using an attentional feature fusion module. The extensive experiments we have carried out involving 19 different facial attributes, manipulated by a StyleGAN2 network, show the good accuracy of the proposed method and its robustness against several image post-processing operators.
Hierarchical Diffusion Models for Singing Voice Neural Vocoder

Recent progress in deep generative models has improved the quality of neural vocoders in speech domain. However, generating a high-quality singing voice remains challenging due to a wider variety of musical expressions in pitch, loudness, and pronunciations. In this work, we propose a hierarchical diffusion model for singing voice neural vocoders. The proposed method consists of multiple diffusion models operating in different sampling rates; the model at the lowest sampling rate focuses on generating accurate low-frequency components such as pitch, and other models progressively generate the waveform at higher sampling rates on the basis of the data at the lower sampling rate and acoustic features. Experimental results show that the proposed method produces high-quality singing voices for multiple singers, outperforming state-of-the-art neural vocoders with a similar range of computational costs.
Picking the Underused Heads: A Network Pruning Perspective of Attention Head Selection for Fusing Dialogue Coreference Information

The Transformer-based models with the multi-head self-attention mechanism are widely used in natural language processing, and provide state-of-the-art results. While the pre-trained language backbones are shown to implicitly capture certain linguistic knowledge, explicitly incorporating structure-aware features can bring about further improvement on the downstream tasks. However, such enhancement often requires additional neural components and increases training parameter size. In this work, we investigate the attention head selection and manipulation strategy for feature injection from a network pruning perspective, and conduct a case study on dialogue summarization. We first rank attention heads in a Transformer-based summarizer with layer-wise importance. We then select the underused heads through extensive analysis, and inject structure-aware features by manipulating the selected heads. Experimental results show that the importance-based head selection is effective for feature injection, and dialogue summarization can be improved by incorporating coreference information via head manipulation.
Pyramid Dynamic Inference: Encouraging Faster Inference Via Early Exit Boosting

Transformer-based models demonstrate state of the art results on several natural language understanding tasks. However, their deployment comes at the cost of increased footprint and inference latency, limiting their adoption to real-time applications. Early exit strategies are designed to speed-up the inference by routing out a subset of samples at the earlier layers of the model. Exiting early causes losing model accuracy. In order to optimize the trade-off between model accuracy and latency, we propose Pyramid Dynamic Inference (PDI), a scheme that encourages fast inference via boosting the performance of early exit heads. PDI allows for more confident early inference by injecting stronger classifiers at earlier layers. It also prevents a significant increase in the model footprint by gradually shrinking the classifiers as the semantic capacity of the deeper transformer layers increase. Experiment results show that PDI outperforms the baselines on both accuracy and latency on the GLUE benchmark.
TSpeech-AI System Description to the 5th Deep Noise Suppression (DNS) Challenge

This report presents the development of Tencent AI Lab’s personalized speech enhancement system for the 2023 ICASSP Signal Processing Grand Challenge – deep noise suppression (DNS) challenge <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> , which includes the use of a modified band-split recurrent neural network (BSRNN) and a multi-resolution spectrogram discriminator to improve perceptual quality metrics. The proposed system outperforms the baseline system by 0.047 and 0.036 final scores in the headset track and speaker phone track, respectively, and ranks among the top-3 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> in both tracks.
Conditional LS-GAN Based Skylight Polarization Image Restoration and Application in Meridian Localization

Skylight polarization images (SPIs) contain crucial spatial information that can be used for navigation purposes. Under most circumstances, the quality of the images becomes a major concern, especially when there is blocking between the perception equipment and the sky. This paper introduces a deep learning-based methodology for restoring SPIs and utilizing the restored images for navigation. First, an adversarial training paradigm is adopted to restore the SPIs collected in a severe blocking environment. We utilize a self-developed simulation method that uses solar altitude and azimuth angle to generate ground truth, and no prior knowledge of the masking information for noise is used. Second, we show how to locate the meridian based on the restored SPIs using a residual neural network. In experiments, we demonstrate the superiority of the proposed model in restoring SPIs, and the enhanced meridian localization precision by using the restored SPIs.
Fast and Efficient Model Serving Using Multi-GPUs with Direct-Host-Access

As deep learning (DL) inference has been widely adopted for building user-facing applications in many domains, it is increasingly important for DL inference servers to achieve high throughput while preserving bounded latency. DL inference requests can be immediately served if the corresponding model is already in the GPU memory. Otherwise, it needs to load the model from host to GPU, adding a significant delay to inference. This paper proposes DeepPlan to minimize inference latency while provisioning DL models from host to GPU in server environments. First, we take advantage of the direct-host-access facility provided by commodity GPUs, allowing access to particular layers of models in the host memory directly from GPU without loading. Second, we parallelize model transmission across multiple GPUs to reduce the time for loading models from host to GPU. We show that a single inference can achieve a 1.94× speedup compared with the state-of-the-art pipelining approach for BERT-Base. When deploying multiple BERT, RoBERTa, and GPT-2 instances on a DL inference serving system, DeepPlan shows a significant performance improvement compared to the pipelining technique and stable 99% tail latency.
Learning to Auto-Correct for High-Quality Spectrograms

Non-autoregressive text-to-speech (TTS) has achieved impressive inference speedup but at the cost of inferior voice quality. The fundamental reason lies in the gap between the complexity of data distributions and the capability of modeling methods. Previous works utilize either simplifying data distributions or enhancing modeling methods to alleviate the problem. In this work, we propose a new architecture ReAct-Speech to explicitly learn to "auto-correct" for high-quality spectrograms. Specifically, ReActSpeech utilizes a redistribution module to improve (correct) extracted alignments automatically, and an iterative decoder called revisor to refine (correct) spectrograms iteratively. Extensive experiments conducted on several benchmarks show that ReActSpeech can greatly alleviate the above problem and achieve a nice tradeoff between training time, inference speed, and output quality.
Deep Feature Aggregation for Lightweight Single Image Super-Resolution

In recent years, a number of lightweight single-image super-resolution (SISR) network methods heave been proposed. However, most existing approaches do not make full use of the information before and after the convolution and the high-frequency information of the image. In this paper, we propose a lightweight deep feature aggregation network (DFAnet), which fuses the outputs of all the deep feature aggregation blocks (DFAB) through the designed nonlinear global feature fusion (NGFF) module. The DFAB includes deep feature aggregation structure (DFAS) and non-local sparse attention mechanism (NLSA), where DFAS consists of several aggregation convolutions and information rearrangement operations. Then the output of DFAS is assessed by non-local sparse attention module to form our basic block DFAB. Furthermore, we design a nonlinear global feature fusion (NGFF) module to learn the nonlinear relationship between the output of each DFAB, which encourages every DFAB to pay attention to different patterns of the image. The qualitative and quantitative experimental results on several benchmark datasets show the proposed method achieves the state-of-the-art results in term of reconstruction accuracy, computational complexity and memory consumption.
URM4DMU: An User Representation Model for Darknet Markets Users

Darknet markets provide a large platform for trading illicit goods and services due to their anonymity. Learning an invariant representation of each user based on their posts on different markets makes it easy to aggregate user information across different platforms, which helps identify anonymous users. Traditional user representation methods mainly rely on modeling the text information of posts and cannot capture the temporal content and the forum interaction of posts. While recent works mainly use CNN to model the text information of posts, failing to effectively model posts whose length changes frequently in an episode. To address the above problems, we propose a model named URM4DMU(User Representation Model for Darknet Markets Users) which mainly improves the post representation by augmenting convolutional operators and self-attention with an adaptive gate mechanism. It performs much better when combined with the temporal content and the forum interaction of posts. We demonstrate the effectiveness of URM4DMU on four darknet markets. The average improvements on MRR value and Recall@10 are 22.5% and 25.5% over the state-of-the-art method respectively.
Enhanced Low-Resolution LiDAR-Camera Calibration via Depth Interpolation and Supervised Contrastive Learning

Motivated by the increasing application of low-resolution LiDAR, we target the problem of low-resolution LiDAR-camera calibration in this work. The main challenges are two-fold: sparsity and noise in point clouds. To address the problem, we propose to apply depth interpolation to increase the point density and supervised contrastive learning to learn noise-resistant features. The experiments on RELLIS-3D demonstrate that our approach achieves an average mean absolute rotation/translation errors of 0.15cm/0.33° on 32-channel LiDAR point cloud data, which significantly outperforms all reference methods.
Estimation of Time-Varying Graph Topologies from Graph Signals

In science and engineering, we often deal with signals that are acquired from time-varying systems represented by dynamic graphs. We observe these signals, and the interest is in finding the time-varying topology of the graphs. We propose two Bayesian methods for estimating these topologies without assuming any specific functional relationships among the signals on the graphs. The two methods exploit Gaussian processes, where the first method uses the length scale of the kernel and relies on variational inference for optimization, and the second method is based on derivatives of the functions and Monte Carlo sampling. Both methods estimate the time-varying topologies of the graphs sequentially. We provide numerical tests that show the performance of the methods in two settings.
Study on the Fairness of Speaker Verification Systems Across Accent and Gender Groups

Speaker verification (SV) systems are currently used for consequential tasks like giving access to bank accounts or making forensic decisions. Ensuring that these systems are fair and do not disfavor any particular group is crucial. In this work, we analyze the performance of two X-vector-based SV systems across groups defined by gender and accent of the speakers when speaking English. To this end, we created a new dataset based on the VoxCeleb corpus by selecting samples from speakers with accents from different countries. We used this dataset to evaluate system performance of SV systems trained with VoxCeleb data. We show that performance, measured with a calibration-sensitive metric, is markedly degraded on groups that are underrepresented in training: females and speakers with nonnative accents in English. Finally, we show that a simple data balancing approach mitigates this undesirable bias on the minority groups without degrading performance on the majority groups.
M<sup>3</sup>ST: Mix at Three Levels for Speech Translation

How to solve the data scarcity problem for end-to-end speech-to-text translation (ST)? It’s well known that data augmentation is an efficient method to improve performance for many tasks by enlarging the dataset. In this paper, we propose Mix at three levels for Speech Translation (M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> ST) method to increase the diversity of the augmented training corpus. Specifically, we conduct two phases of fine-tuning based on a pre-trained model using external machine translation (MT) data. In the first stage of fine-tuning, we mix the training corpus at three levels, including word level, sentence level and frame level, and fine-tune the entire model with mixed data. At the second stage of fine-tuning, we take both original speech sequences and original text sequences in parallel into the model to fine-tune the network, and use Jensen-Shannon divergence to regularize their outputs. Experiments on MuST-C speech translation benchmark and analysis show M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> ST outperforms current strong baselines and achieves state-of-the-art results on eight directions with an average BLEU of 29.9.
Transformer-based tracking Network for Maneuvering Targets

For strong maneuvering targets, the drastic change of target motion models makes the tracking methods hard to adapt and provide accurate state estimation promptly. To solve the state estimation problem of strong maneuvering targets, we propose a transformer-based tracking network, named TrTNet. The TrTNet model learns a new residual mapping from the observation trajectory to the real trajectory. Through the parallel manner, the TrTNet model extracts single state features of the observation trajectory at all time steps simultaneously. The correlation information between states can be explored and utilized to enhance state features of single time step, benefiting to learn the transition law between states and generate high-quality tracking model. Simulation results show that the proposed TrTNet model achieves better performance in maneuvering target tracking than other competitive approaches.
Distance-Based Online Label Inference Attacks Against Split Learning

Split learning is a promising paradigm for distributed learning at resource-constrained devices, where the learning model is split to be trained at the participants collaboratively. Unlike federated learning that shares the entire gradients among participants, split learning only exchanges the intermediate learning results (i.e., the extracted features/smashed data and gradients) at the cut layer between participants and server. This requires different/new attack designs to understand the security performance of various privacy-sensitive applications using split learning. This paper focuses on private labels and proposes three label inference attacks based on the similarities between exchanged gradients/smashed data and sample points. We mathematically analyze and unify these similarities (for retrieving accurate labels) as Euclidean distance, such that the attack can be conducted by finding the nearest sample point from target data in Euclidean space. We also show that transfer learning can help retrieve private labels directly from raw data. Experimental results demonstrate that our attacks can recover private labels against three state-of-the-art label protection methods.
Accidental Learners: Spoken Language Identification in Multilingual Self-Supervised Models

In this paper, we extend previous self-supervised approaches for language identification by experimenting with Conformer based architecture in a multilingual pre-training paradigm. We find that pre-trained speech models optimally encode language discriminatory information in lower layers. Further, we demonstrate that the embeddings obtained from these layers are significantly robust to classify unseen languages and different acoustic environments without additional training. After fine-tuning a pre-trained Conformer model on the VoxLin-gua107 dataset, we achieve results similar to current state-of-the-art systems for language identification. More, our model accomplishes this with 5x less parameters. We open-source the model through the NVIDIA NeMo toolkit.
Active Learning of non-Semantic Speech Tasks with Pretrained models

Pretraining neural networks with massive unlabeled datasets has become popular as it equips the deep models with a better prior to solve downstream tasks. However, this approach generally assumes that the downstream tasks have access to annotated data of sufficient size. In this work, we propose ALOE, a novel system for improving the data- and label-efficiency of non-semantic speech tasks with active learning (AL). ALOE uses pretrained models in conjunction with active learning to label data incrementally and learn classifiers for downstream tasks, thereby mitigating the need to acquire labeled data beforehand. We demonstrate the effectiveness of ALOE on a wide range of tasks, uncertainty-based acquisition functions, and model architectures. Training a linear classifier on top of a frozen encoder with ALOE is shown to achieve performance similar to several baselines that utilize the entire labeled data.