Reviewerly/exHarmony huggingface dataset:
AMC-Net: An Effective Network for Automatic Modulation Classification

Automatic modulation classification (AMC) is a crucial stage in the spectrum management, signal monitoring, and control of wireless communication systems. The accurate classification of the modulation format plays a vital role in the subsequent decoding of the transmitted data. End-to-end deep learning methods have been recently applied to AMC, outperforming traditional feature engineering techniques. However, AMC still has limitations in low signal-to-noise ratio (SNR) environments. To address the drawback, we propose a novel AMC-Net that improves recognition by denoising the input signal in the frequency domain while performing multi-scale and effective feature extraction. Experiments on two representative datasets demonstrate that our model performs better in efficiency and effectiveness than the most current methods.
FED-3DA: A Dynamic and Personalized Federated Learning Framework

In federated learning, the non-IID data generated from heterogeneous clients may reduce the global model efficiency. Previous studies use personalization as a common approach to adapt the global model to these clients (called the local model). However, client’s data distribution may change dynamically with its location or environment, which can degrade the performance of the local model, leading to a new Dynamic Personalized Federated Learning (DPFL) problem. This paper proposes a novel approach to reduce the impact of the dynamic distribution on the local model based on metalearning and distribution distance measurement named Fed-3DA. It calculates the distribution distance periodically to perceive the distribution change on the client and adjust the local model preferences from a global meta-model through the distribution representation. Our experiments on public datasets show that Fed-3DA can effectively reduce the performance fluctuation of the local model in DPFL scenarios.
Text-To-Speech Synthesis Based on Latent Variable Conversion Using Diffusion Probabilistic Model and Variational Autoencoder

Text-to-speech synthesis (TTS) is a task to convert texts into speech. Two of the factors that have been driving TTS are the advancements of probabilistic models and latent representation learning. We propose a TTS method based on latent variable conversion using a diffusion probabilistic model and the variational autoencoder (VAE). In our TTS method, we use a waveform model based on VAE, a diffusion model that predicts the distribution of latent variables in the waveform model from texts, and an alignment model that learns alignments between the text and speech latent sequences. Our method integrates diffusion with VAE by modeling both mean and variance parameters with diffusion, where the target distribution is determined by approximation from VAE. This latent variable conversion framework potentially enables us to flexibly incorporate various latent feature extractors. Our experiments show that our method is robust to linguistic labels with ambiguous orthography and alignment errors.
Multi-Speaker Data Augmentation for Improved end-to-end Automatic Speech Recognition

Publicly available datasets traditionally used to train E2E ASR models for conversational telephone speech recognition are based on clean, short duration, single speaker utterances collected on separate channels. While E2E ASR models achieve state-of-the-art performance on recognition tasks that match well with such training data, they are observed to fail on test recordings that contain multiple speakers, significant channel or background noise or span longer durations than training data utterances. To mitigate these issues, we propose an on-the-fly data augmentation strategy that transforms single speaker training data into multiple speaker data by appending together multiple single speaker utterances. The proposed technique encourages the E2E model to become robust to speaker changes and also process longer utterances effectively. During training, the model is also guided by a teacher model trained on single speaker utterances to map its multi-speaker encoder embeddings to better performing single speaker representations. With the proposed technique we obtain 7-14% relative improvement on various single speaker and multiple speaker test sets. We also show how this technique is able to improve recognition performance by up to 14% by capturing useful information from preceding spoken utterances used as dialog history.
Flowpose: Conditional Normalizing Flows for 3D Human Pose and Shape Estimation from Monocular Videos

Human motion modeling is essential for video-based 3D human pose and shape estimation. Most existing methods model human motion by learning a deterministic mapping from the input videos to the human body parameters, while the uncertainties such as occlusions and depth ambiguities are ignored. To address this problem, we propose a probabilistic model based on conditional normalizing flows called FlowPose to learn the distribution of feasible 3D human motion. This model allows access to the most likely 3D human poses given a video input, which means that more accurate and temporally coherent human poses can be obtained. Additionally, a contrastive training strategy is utilized to maximize the mutual information between video features and their 3D human poses, resulting in an improvement on feature extraction of the conditional flow model. Experimental results on two benchmarks 3DPW and Human3.6M demonstrate that our method outperforms the state-of-the-art video-based methods.
Deep Manifold Graph Auto-Encoder For Attributed Graph Embedding

Representing graph data in a low-dimensional space for subsequent tasks is the purpose of attributed graph embedding. Most existing neural network approaches learn latent representations by minimizing reconstruction errors. Rare work considers the data distribution and the topological structure of latent codes simultaneously, which often results in inferior embeddings in real-world graph data. This paper proposes a novel Deep Manifold (Variational) Graph Auto-Encoder (DMVGAE/DMGAE) method for attributed graph data to improve the stability and quality of learned representations to tackle the crowding problem. The node-to-node geodesic similarity is preserved between the original and latent space under a pre-defined distribution. The proposed method surpasses state-of-the-art baseline algorithms by a significant margin on different downstream tasks across popular datasets, which validates our solutions. We promise to release the code after acceptance.
Learning Quantum Entanglement Distillation With Noisy Classical Communications

An important primitive for quantum networking is entanglement distillation, whose goal is to enhance the fidelity of entangled qubits through local operations and classical communication (LOCC). Existing distillation protocols assume the availability of ideal, noiseless, communication channels. In this paper, we study the case in which communication takes place over noisy binary symmetric channels. We propose to implement local processing through parameterized quantum circuits (PQCs) that are optimized to maximize the average fidelity, while accounting for communication errors. The introduced approach, Noise Aware-LOCCNet (NA-LOCCNet), is shown to have significant advantages over existing protocols designed for noiseless communications.
A Database for Multi-Modal Short Video Quality Assessment

The short video has gained increasing attention in information sharing and commercial promotions due to the fast development of social platforms. Accompanying, it introduces great requirements for assessing the quality of short videos for efficient information acquirement and propagation. However, existing video quality assessment researches focus on assessing video content with five rating scores, limiting the assessment to a one-dimension and simplified criterion. In this paper, we establish a novel database dubbed MMSVD-Douyin for assessing multi-modal short video quality under consideration of three evaluation criteria. It includes 4,684 short videos, three kinds of modalities, six kinds of data formats, and three assessment criteria. To conduct the short video quality assessment, we set up an all-around multi-modal short video quality assessment benchmark (MulSVQA) that dynamically fuses representations from three modalities and produces numbers of "likes", "shares" and "comments" of short videos.
Affinity Learning With Blind-Spot Self-Supervision for Image Denoising

In this paper, we extend the blind-spot based self-supervised denoising by using affinity learning to remove noise from affected pixels. Inspired by inpainting, we introduce a novel Mask Guided Residual Convolution (MGRConv) to learn a neighboring image pixel affinity map that gradually removes noise and refines blind-spot denoising process. We show that mask convolution plays an important role in blind-spot denoising since it is theoretically aligned with $\mathcal{J} - invariance$, which blind-spot based self-supervised denoising frameworks are built upon. The theoretical analysis further shows the motivation behind using more adaptive mask convolutions. Our MGRConv not only enables dynamic mask learning without external trainable parameters, but also preserves appropriate mask constraints by sigmoid activation and residual summation. Our MGRConv is a balance between partial convolution and learnable attention maps, and boosts denoising performance better than other inpainting convolutions with similar or even less parameters, memory, and training/inference time. Extensive experiments show that our proposed plug-and-play MGRConv can assist blind-spot based denoising networks to reach promising results on both existing single-image based and dataset based benchmarks.
Locale Encoding for Scalable Multilingual Keyword Spotting Models

A Multilingual Keyword Spotting (KWS) system detects spoken keywords over multiple locales. Conventional monolingual KWS approaches do not scale well to multilingual scenarios because of high development/maintenance costs and lack of resource sharing. To overcome this limit, we propose two locale-conditioned universal models with locale feature concatenation and feature-wise linear modulation (FiLM). We compare these models with two baseline methods: locale-specific monolingual KWS, and a single universal model trained over all data. Experiments over 10 localized language datasets show that locale-conditioned models substantially improve accuracy over baseline methods across all locales in different noise conditions. FiLM performed the best, improving on average FRR by 61% (relative) compared to monolingual KWS models of similar sizes.
Improving Sentence Similarity Estimation for Unsupervised Extractive Summarization

Unsupervised extractive summarization aims to extract salient sentences from a document as the summary without labeled data. Recent literatures mostly research how to leverage sentence similarity to rank sentences in the order of salience. However, sentence similarity estimation using pre-trained language models mostly takes little account of document-level information and has a weak correlation with sentence salience ranking. In this paper, we proposed two novel strategies to improve sentence similarity estimation for unsupervised extractive summarization. We use contrastive learning to optimize a document-level objective that sentences from the same document are more similar than those from different documents. Moreover, we use mutual learning to enhance the relationship between sentence similarity estimation and sentence salience ranking, where an extra signal amplifier is used to refine the pivotal information. Experimental results demonstrate the effectiveness of our strategies. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Thermal Infrared Image Inpainting Via Edge-Aware Guidance

Image inpainting has achieved fundamental advances with deep learning. However, almost all existing inpainting methods aim to process natural images, while few target Thermal Infrared (TIR) images, which have widespread applications. When applied to TIR images, conventional inpainting methods usually generate distorted or blurry content. In this paper, we propose a novel task—Thermal Infrared Image Inpainting, which aims to reconstruct missing regions of TIR images. Crucially, we propose a novel deep-learning-based model TIR-Fill. We adopt the edge generator to complete the canny edges of broken TIR images. The completed edges are projected to the normalization weights and biases to enhance edge awareness of the model. In addition, a refinement network based on gated convolution is employed to improve TIR image consistency. The experiments demonstrate that our method outperforms state-of-the-art image inpainting approaches on FLIR thermal dataset.
Relevance Propagation through Deep Conditional Random Fields

Conditional random fields (CRFs), a particular type of graph neural networks (GNNs), can be used to make structured predictions in machine learning, with various applications from image processing and natural language processing to recommender systems. CRFs refine the prediction of a sample by taking into account its context information. However, there is a lack of work on post-hoc explanation approaches to CRFs, especially when the model is softmax-activated like the deep mean field network (DMFN). In this paper, we bridge this gap by proposing a layer-wise relevance propagation (LRP) method based on deep Taylor decomposition to explain CRFs, especially the DMFN model. The method considers the intermediate softmax activation layers in DMFN. We use two evaluation settings: top K% deletion and insertion to evaluate the method. Experimental studies on fake news detection using the DMFN model prove the effectiveness of our explanation method compared to the other baseline methods.
Good Neighbors are All You Need for Chinese Grapheme-To-Phoneme Conversion

Most Chinese Grapheme-to-Phoneme (G2P) systems employ a three-stage framework that first transforms input sequences into character embeddings, obtains linguistic information using language models, and then predicts the phonemes based on global context about the entire input sequence. However, linguistic knowledge alone is often inadequate. Language models frequently encode overly general structures of a sentence and fail to cover specific cases needed to use phonetic knowledge. Also, a handcrafted post-processing system is needed to address the problems relevant to the tone of the characters. However, the system exhibits inconsistency in the segmentation of word boundaries which consequently degrades the performance of the G2P system. To address these issues, we propose the Reinforcer that provides strong inductive bias for language models by emphasizing the phonological information between neighboring characters to help disambiguate pronunciations. Experimental results show that the Reinforcer boosts the cutting-edge architectures by a large margin. We also combine the Reinforcer with a large-scale pre-trained model and demonstrate the validity of using neighboring context in knowledge transfer scenarios.
Towards Controllable Audio Texture Morphing

In this paper, we propose a data-driven approach to train a Generative Adversarial Network (GAN) conditioned on "soft-labels" distilled from the penultimate layer of an audio classifier trained on a target set of audio texture classes. We demonstrate that interpolation between such conditions or control vectors provide smooth morphing between the generated audio textures, and show similar or better audio texture morphing capability compared to the state-of-the-art methods. The proposed approach results in a well-organized latent space that generates novel audio outputs while remaining consistent with the semantics of the conditioning parameters. This is a step towards a general data-driven approach to designing generative audio models with customized controls capable of traversing out-of-distribution regions for novel sound synthesis.
On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networks

OTFusion, or layer-wise model fusion via optimal transport, applies soft neuron association to unify different pre-trained networks. Despite its effectiveness in saving computational resources, OTFusion requires the input networks to have the same number of layers. To address this issue, we propose a novel model fusion framework, named CLAFusion, to fuse neural networks with different numbers of layers, which we refer to as heterogeneous neural networks, via cross-layer alignment. We demonstrate that the cross-layer alignment problem, which is an unbalanced assignment problem, can be solved efficiently using dynamic programming. Based on the cross-layer alignment, our framework balances the number of layers of neural networks before applying layer-wise model fusion. Our experiments indicate that CLAFusion, with an extra finetuning process, improves the accuracy of residual networks on the CIFAR10, CIFAR100, and Tiny-ImageNet datasets. Furthermore, we explore its practical usage for model compression and knowledge distillation when applied to the teacher-student setting.
Volumetric Attribute Compression for 3D Point Clouds Using Feedforward Network with Geometric Attention

We study 3D point cloud attribute compression using a volumetric approach: given a target volumetric attribute function f : ℝ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> → ℝ, we quantize and encode parameter vector θ that characterizes f at the encoder, for reconstruction ${f_{\hat \theta }}\left( {\mathbf{x}} \right)$ at known 3D points x’s at the decoder. Extending a previous work Region Adaptive Hierarchical Transform (RAHT) that employs piecewise constant functions to span a nested sequence of function spaces, we propose a feedforward linear network that implements higher-order B-spline bases spanning function spaces without eigen-decomposition. Feedforward network architecture means that the system is amenable to end-to-end neural learning. The key to our network is space-varying convolution, similar to a graph operator, whose weights are computed from the known 3D geometry for normalization. We show that the number of layers in the normalization at the encoder is equivalent to the number of terms in a matrix inverse Taylor series. Experimental results on real-world 3D point clouds show up to 2-3 dB gain over RAHT in energy compaction and 20-30% bitrate reduction.
Egocentric Action Anticipation for Personal Health

The egocentric action anticipation task consists in predicting future (unobserved) actions based on input from a wearable first-person view camera. In this work, we are focusing on applying egocentric action anticipation methods to the personal health domain, i.e., utilizing them for the analysis of dietary and hygienic activities routine and for the prevention of undesirable behavior (such as tasting food before washing hands or adding sugar). We collect a dataset of egocentric videos, capturing specific activities related to food preparation; fine-tune existing action anticipation models on this dataset and analyse effects caused by domain shift.
Audio-Driven High Definetion and Lip-Synchronized Talking Face Generation Based on Face Reenactment

Generating audio-driven photo-realistic talking face has received intensive attention due to its ability to bring more new human-computer interaction experiences. However, previous works struggled to balance high definition, lip synchronization, and low customization costs, which would degrade the user experience. In this paper, a novel audio-driven talking face generation method was proposed, which subtly converts the problem of improving video definition into the problem of face reenactment to produce both lip-synchronized and high- definition face video. The framework is decoupled, meaning that the same trained model can be used on arbitrary characters and audio without further customizing training for specific people, thus significantly reducing costs. Experiment results show that our proposed method achieves the high video definition, and comparable lip synchronization performance with the existing state-of-the-art methods.
Speaker Change Detection For Transformer Transducer ASR

Speaker change detection (SCD) is an important feature that improves the readability of the recognized words from an automatic speech recognition (ASR) system by breaking the word sequence into paragraphs at speaker change points. Existing SCD solutions either require additional ensemble for the time based decisions and recognized word sequences, or implement a tight integration between ASR and SCD, limiting the potential optimum performance for both tasks. To address these issues, we propose a novel framework for the SCD task, where an additional SCD module is built on top of an existing Transformer Transducer ASR (TT-ASR) network. Two variants of the SCD network are explored in this framework that naturally estimate speaker change probability for each word, while allowing the ASR and SCD to have independent optimization scheme for the best performance. Experiments show that our methods can significantly improve the F1 score on LibriCSS and Microsoft call center data sets without ASR degradation, compared with a joint SCD and ASR baseline.
Spherical Vector Quantization for Spatial Direction Coding

We address in this paper the problem of designing high bit-rate spherical vector quantization (VQ) to achieve transparent joint coding of angular values (azimuth and elevation) representing source directions in parametric spatial audio coding. We present the design of three coding methods using an optimized 3D grid and evaluate their performance/complexity at a bitrate of 16 bits per spatial direction. A comparison with VQ based on existing 3D spherical grids is provided for benchmarking purposes.
Ultrasound Image Quality Control Using Speech-Assisted Switchable CycleGAN

Unlike computed tomography (CT) and magnetic resonance imaging (MRI) in which the image quality (IQ) is controlled by predefined acquisition setups, the IQ of ultrasound (US) is heavily dependent upon operators. In particular, an operator often adjusts the system parameters in a real-time manner based on his/her preference. Unfortunately, there are many cases where such real-time control of IQ is difficult, especially in the intensive care unit (ICU) or operating room (OR), since the operator should simultaneously treat patients in sterile status and adjust the system parameters. To address this, inspired by the recent success of Switchable CycleGAN using Adaptive Instance Normalization (AdaIN) layers, here we propose a novel speech-assisted Switchable CycleGAN architecture that can be controlled by operator’s verbal commands. Specifically, we employ a Speech Recognition Module (SRM) to generate AdaIN codes for Switchable CycleGAN. In particular, our SRM is based on the pre-trained model so that it can be less effected by the operator’s voice and robustly generate AdaIN codes. Various experimental results confirm that the proposed method can successfully control IQ of US by speech.