Reviewerly/exHarmony huggingface dataset:
An Adaptive Enhancement Method for Gastrointestinal Low-Light Images of Capsule Endoscope

Balancing image local detail enhancement with brightness enhancement has been a challenge. The images captured by wireless capsule endoscopy (WCE) are low-light and unclear. To this end, we propose an adaptive enhancement method for WCE images. Firstly, we use the guided filter to filter and smooth the WCE images to approximate its illumination component, and then the reflection component is obtained by decomposing it based on the Retinex model. Secondly, an adaptive Sigmoid function is obtained according to the positive correlation between the just-noticeable difference (JND) threshold of the illumination component and the gain parameter of the Sigmoid function, to adaptively enhance the illumination component, and then based on the Retinex model fusion with the reflection component. Finally, we combine with Gamma correction algorithm to enhance the contrast of the above results. Experimental results show that the proposed method can adaptively enhance the overall effect and local details of WCE images; and the feature extraction and matching effects are better than the classical enhancement algorithms, with an average increase of 88.8% and 59.1% respectively.
Second-Order Statistic Deviation to Model Anomalies in the Design of Unsupervised Detectors

Anomaly Detection is a challenging task due to the limited knowledge about possible anomalies. This issue can be tackled by modeling anomalies through domain expertise or collecting sufficient anomalous data. However, some domains, such as monitoring systems, require detectors that are capable of detecting any potential alteration in the observed phenomenon. Hereby we propose a tool to generate anomalies as a statistical deviation from the characterization of the signal representing the normal behavior. Two families of deviation models are presented, and the effectiveness of the tool is proven using well-known unsupervised detectors. The effects of a possible intermediate data compression stage on the detection capabilities are also considered.
Personalized Federated Learning on Long-Tailed Data via Adversarial Feature Augmentation

Personalized Federated Learning (PFL) aims to learn personalized models for each client based on the knowledge across all clients in a privacy-preserving manner. Existing PFL methods generally assume that the underlying global data across all clients are uniformly distributed without considering the long-tail distribution. The joint problem of data heterogeneity and long-tail distribution in the FL environment is more challenging and severely affects the performance of personalized models. In this paper, we propose a PFL method called Federated Learning with Adversarial Feature Aug-mentation (FedAFA) to address this joint problem in PFL. FedAFA optimizes the personalized model for each client by producing a balanced feature set to enhance the local minority classes. The local minority class features are generated by transferring the knowledge from the local majority class features extracted by the global model in an adversarial example learning manner. The experimental results on benchmarks under different settings of data heterogeneity and long-tail distribution demonstrate that FedAFA significantly improves the personalized performance of each client compared with the state-of-the-art PFL algorithm. The code is available at https://github.com/pxqian/FedAFA.
Vani: Very-Lightweight Accent-Controllable TTS for Native And Non-Native Speakers With Identity Preservation

We introduce VANI, a very lightweight multi-lingual accent controllable speech synthesis system. Our model builds upon disentanglement strategies proposed in RADMMM[1] and supports explicit control of accent, language, speaker and fine-grained F <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</inf> and energy features for speech synthesis. We utilize the Indic languages dataset, released for LIMMITS 2023 as part of ICASSP Signal Processing Grand Challenge, to synthesize speech in 3 different languages. Our model supports transferring the language of a speaker while retaining their voice and the native accent of the target language. We utilize the large-parameter RADMMM model for Track 1 and lightweight VANI model for Track 2 and 3 of the competition.
Pyramid Spatial Feature Transform and Shared-Offsets Deformable Alignment Based Convolutional Network for HDR Imaging

To generate ghost-free high dynamic range (HDR) images by merging multiple differently exposed low dynamic range (LDR) images, the key is to handle ill-exposed areas in the input LDR images and misalignment among them. In this paper, a Pyramid Spatial Feature Transform and shared-offsets Deformable convolutional Network (PSFTDNet) is proposed to achieve this target. The pyramid spatial feature transform module tackles ill-exposed areas, which modulates the features to exploit complementary information of them in a coarse-to-fine manner. The shared-offsets deformable alignment handles misalignment among the input images, which applies the offsets used to align exposure-aligned images to align all the features. Experiments on the NTIRE HDR challenge dataset and Kalantari dataset show that the proposed PSFTDNet outperforms all the conventional methods with PSNR-L scores of 42.31 dB and 41.54 dB, and PSNR-T scores of 34.78 dB and 43.56 dB.
Frequency Reciprocal Action and Fusion for Single Image Super-Resolution

Frequency-based methods have recently received much attention due to their impressive restoration of detail and structure in single image super-resolution (SISR). However, most of these methods mainly use frequency information as auxiliary means but ignore exploring the correlations and pixel distribution differences among various frequencies. To address the limitations, we propose a novel Frequency Reciprocal Action and Fusion Network (FRAF) that explores various frequency correlations and differences. Specifically, we design a Frequency Reciprocal Action (FRA) module, which safely enhances valid spatial information and decreases un-necessary repetition by reciprocal action among various spatial frequencies, to generate refined high- and low-frequency features. These refined frequency features are then progressively to guide the details and structure recovery, respectively. Furthermore, we develop a Detail and Structure Fusion (DSF) module to adaptively select, enhance and fuse the features to output the final HR image. This way ensures the final image is a high-quality product with rich details and a clear structure. Experimental results demonstrate that our method achieves superior performance over state-of-the-art (SOTA) approaches on both quantitative and qualitative evaluations.
Towards Building Text-to-Speech Systems for the Next Billion Users

Deep learning based text-to-speech (TTS) systems have been evolving rapidly with advances in model architectures, training methodologies, and generalization across speakers and languages. However, these advances have not been thoroughly investigated for Indian language speech synthesis. Such investigation is computationally expensive given the number and diversity of Indian languages, relatively lower resource availability, and the diverse set of advances in neural TTS that remain untested. In this paper, we evaluate the choice of acoustic models, vocoders, supplementary loss functions, training schedules, and speaker and language diversity for Dravidian and Indo-Aryan languages. Based on this, we identify monolingual models with FastPitch and HiFi-GAN V1, trained jointly on male and female speakers to perform the best. With this setup, we train and evaluate TTS models for 13 languages and find our models to significantly improve upon existing models in all languages as measured by mean opinion scores. We open-source all models on the Bhashini platform.
Metric Learning for User-Defined Keyword Spotting

The goal of this work is to detect new spoken terms defined by users. While most previous works address Keyword Spotting (KWS) as a closed-set classification problem, this limits their transferability to unseen terms. The ability to define custom keywords has advantages in terms of user experience.In this paper, we propose a metric learning-based training strategy for user-defined keyword spotting. In particular, we make the following contributions: (1) we construct a large-scale keyword dataset with an existing speech corpus and propose a filtering method to remove data that degrade model training; (2) we propose a metric learning-based two-stage training strategy, and demonstrate that the proposed method improves the performance on the user-defined keyword spotting task by enriching their representations; (3) to facilitate the fair comparison in the user-defined KWS field, we propose unified evaluation protocol and metrics.Our proposed system does not require an incremental training on the user-defined keywords, and outperforms previous works by a significant margin on the Google Speech Commands dataset using the proposed as well as the existing metrics.
Background Disturbance Mitigation for Video Captioning Via Entity-Action Relocation

Video captioning aims to generate sentences to accurately describe the video content, in which video background plays the role of prompts. State-of-the-art methods tend to explore richer video representations adequately, fusing with language to improve caption quality, which has shown great success. However, they focus on exploiting foreground semantics, ignoring the potential negative impact of video background disturbance to caption generation, i.e., the entities and the actions are misjudged by a similar video background. To ameliorate this issue, we propose Entity-Action Relocation (EAR) to enhance the adaptability of entities and actions to various backgrounds by giving them the background. Specifically, for an extracted original video feature, we construct a mixed background for all entities and actions to form a distracting video feature sample. After that, contrastive learning is applied to pull the generated caption of the original representations and of the distracting representations closer, and to push the former away from the generated caption of other videos, explicitly concentrating on the entities and actions of the current video scene. Extensive experiments on two public datasets (MSR-VTT and MSVD) demonstrate that dealing with background disturbance for video can obtain a competitive caption generation effect.
Attention Localness in Shared Encoder-Decoder Model For Text Summarization

Text summarization is to generate a brief version of a given article while maintaining its essential meaning. Most existing solutions typically relied on the standard attention-based encoder-decoder framework, where each token in the source article, including redundancy, would be contributed to the de-coder through the attention mechanism. It follows that how to filter out the redundant content becomes an important issue in the text summarization task. In this study, we propose a localness attention network, with simplicity and feasibility in mind, which circles different local regions in the source article as contributors in different decoding steps. To further strengthen the localness model, we share the semantic space of the encoder and decoder. The experimental results conducted on two benchmark datasets demonstrate the effectiveness and applicability of the proposed method in relation to several well-practiced works.
Notice of Removal: Exploiting Semantic Attributes for Transductive Zero-Shot Learning

Removed.
Zephyr: Zero-Shot Punctuation Restoration

Punctuation restoration can be crucial for the cascade speech translation system. Traditional approaches typically treat it as a sequential tagging problem, predicting which punctuation mark should follow a given word. However, this often requires significant computational and storage resources for full-stage training or fine-tuning. Our argument is that pre-trained language models (PLMs) can directly leverage their learned knowledge for punctuation generation, making additional training unnecessary. In this paper, we propose the Zephyr algorithm, which utilizes PLMs to perform zero-shot and few-shot punctuation restoration for both offline and streaming scenarios. Our experimental results demonstrate that, in comparison to fine-tuning-based baselines, Zephyr achieves competitive performance while requiring little to no training cost and exhibiting better generalizability in zeroshot and few-shot settings.
Towards Zero-Shot Personalized Table-to-Text Generation with Contrastive Persona Distillation

Existing neural methods have shown great potentials towards generating informative text from structured tabular data as well as maintaining high content fidelity. However, few of them shed light on generating personalized expressions, which often requires well-aligned persona-table-text datasets that are difficult to obtain. To overcome these obstacles, we explore personalized table-to-text generation under a zero-shot setting, by assuming no well-aligned persona-table-text triples are required during training. To this end, we firstly collect a set of unpaired persona information and then propose a semi-supervised approach with contrastive persona distillation (S <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> P-CPD) to generate personalized context. Specifically, tabular data and persona information are firstly represented as latent variables separately. Then, we devise a latent space fusion technique to distill persona information into the table representation. Besides, a contrastive-based discriminator is employed to guarantee the style consistency between the generated context and its corresponding persona. Experimental results on two benchmarks demonstrate S <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> P-CPD’s ability on keeping both content fidelity and personalized expressions.
A Knowledge-Driven Vowel-Based Approach of Depression Classification from Speech Using Data Augmentation

We propose a novel explainable machine learning (ML) model that identifies depression from speech, by modeling the temporal dependencies across utterances and utilizing the spectrotemporal information at the vowel level. Our method first models the variable-length utterances at the local-level into a fixed-size vowel-based embedding using a convolutional neural network with a spatial pyramid pooling layer (vowel CNN). Following that, the depression is classified at the global-level from a group of vowel CNN embeddings that serve as the input to another 1D CNN (depression CNN). Different data augmentation methods are designed for both the training of vowel CNN and depression CNN. We investigate the performance of the proposed system at various temporal granularities when modeling short, medium, and long analysis windows, corresponding to 10, 21, and 42 utterances, respectively. The proposed method reaches comparable performance with previous state-of-the-art approaches and depicts explainable properties with respect to the depression outcome. The findings from this work may benefit clinicians by providing additional intuitions during joint human-ML decision-making tasks.
Scale-Adaptive Tiny Object Detection Enhanced by Across-Scale and Shape-Preserved Semantic Location

In tiny object detection, the main challenges are tiny objects’ weak feature responses and possible semantic disappearance in deep networks. To address the problems, we proposed an Instance-level, Scale-adaptive, Shape-preserved, and Semantic-consistent Supervision (I4S) module for better locating tiny objects. It models across-scale feature responses of an instance as an elliptic cone, whose axis indicates the instance’s semantic center in different scales. By the cone supervision on across-scale feature maps, it not only exploits the classification semantic from multi-scale feature maps, but also preserves and explores the information of instance shape in consecutive scales. Experiment results on public datasets proved that our method can effectively improve the location accuracy and significantly reduce the missed detection rate comparing with the method of directly fusing multi-scale features.
Anchored Speech Recognition with Neural Transducers

Neural transducers have achieved human level performance on standard speech recognition benchmarks. However, their performance significantly degrades in the presence of cross-talk, especially when the primary speaker has a low signal-to-noise ratio. Anchored speech recognition refers to a class of methods that use information from an anchor segment (e.g., wake-words) to recognize device-directed speech while ignoring interfering background speech. In this paper, we investigate anchored speech recognition to make neural transducers robust to background speech. We extract context information from the anchor segment with a tiny auxiliary network, and use encoder biasing and joiner gating to guide the transducer towards the target speech. Moreover, to improve the robustness of context embedding extraction, we propose auxiliary training objectives to disentangle lexical content from speaking style. We evaluate our methods on synthetic LibriSpeech-based mixtures comprising several SNR and overlap conditions; they improve relative word error rates by 19.6% over a strong baseline, when averaged over all conditions.
Boosting Face Recognition Performance with Synthetic Data and Limited Real Data

Face recognition is one of the most precise and straightforward methods to establish individual identity, and is important in our daily life. To solve the issues of privacy, bias, and collection difficulty caused by face recognition relying heavily on collecting a huge number of real face images from the Internet, a seemingly promising idea is to employ GAN-generated synthetic faces as the training data. However, there are obvious surface gaps and domain gaps between real and synthetic face images, and cannot be replaced directly. In this paper, we attempt to boost face recognition simultaneously using synthetic data and limited real data. Specifically, we first design an augmented space for auto augmentation methods to augment synthetic images to alleviate the surface gap, then propose to disentangle the underlying style distributions through dual batch normalization layers so that both synthetic and real images can be learned jointly by convolution layers without mixing across domains. Extensive experiments demonstrate our method can achieve better results than training with large quantities of real data.
Incorporating Uncertainty from Speaker Embedding Estimation to Speaker Verification

Speech utterances recorded under differing conditions exhibit varying degrees of confidence in their embedding estimates, i.e., uncertainty, even if they are extracted using the same neural network. This paper aims to incorporate the uncertainty estimate produced in the xi-vector network front-end with a probabilistic linear discriminant analysis (PLDA) back-end scoring for speaker verification. To achieve this we derive a posterior covariance matrix, which measures the uncertainty, from the frame-wise precisions to the embedding space. We propose a log-likelihood ratio function for the PLDA scoring with the uncertainty propagation. We also propose to replace the length normalization pre-processing technique with a length scaling technique for the application of uncertainty propagation in the back-end. Experimental results on the VoxCeleb-1, SITW test sets as well as a domain-mismatched CNCeleb1-E set show the effectiveness of the proposed techniques with 14.5%–41.3% EER reductions and 4.6%–25.3% minDCF reductions.
Personalized Lightweight Text-to-Speech: Voice Cloning with Adaptive Structured Pruning

Personalized TTS is an exciting and highly desired application that allows users to train their TTS voice using only a few recordings. However, TTS training typically requires many hours of recording and a large model, making it unsuitable for deployment on mobile devices. To overcome this limitation, related works typically require fine-tuning a pre-trained TTS model to preserve its ability to generate high-quality audio samples while adapting to the target speaker’s voice. This process is commonly referred to as "voice cloning." Although related works have achieved significant success in changing the TTS model’s voice, they are still required to fine-tune from a large pre-trained model, resulting in a significant size for the voice-cloned model. In this paper, we propose applying trainable structured pruning to voice cloning. By training the structured pruning masks with voice-cloning data, we can produce a unique pruned model for each target speaker. Our experiments demonstrate that using learnable structured pruning, we can compress the model size to 7 times smaller while achieving comparable voice-cloning performance.
Improving the Stochastic Gradient Descent’s Test Accuracy by Manipulating the ℓ<sub>∞</sub> Norm of its Gradient Approximation

The stochastic gradient descent (SGD) is a simple yet very influential algorithm used to find the minimum of a loss (cost) function which is dependent on datasets with large cardinality, such in cases typically associated with deep learning (DL). There exists several variants/improvements over the "vanilla" SGD, which from a high-level perspective, may be understood as using an adaptive element-wise step-size (SS). Moreover, from an algorithmic point of view, there is a clear "incremental improvement path" which relates all of them, i.e. from simple alternative such SG Clipping (SGC) to the well-known variance correction (Adagrad), follow by an (EMA) exponential moving average (RMSprop) to alternative furtherance such Newton (AdaDelta) or bias correction along with different EMA options for the gradient itself (Adam, AdaMAx, AdaBelief, etc.).In this paper, inspired by previous non-stochastic results on how to avoid divergence for ill chosen SS (for the accelerated proximal gradient algorithm), instead of directly using the standard SGD gradient’s EMA ${{\mathbf{\bar g}}_k}$, we propose to modify its entries so as to force $\left\{ {{{\left\| {{{{\mathbf{\bar g}}}_k}} \right\|}_\infty }} \right\}{\text{'s}}$ moving average to be non-increasing. Our reproducible computational results compare our proposed algorithm, called SGD-ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</inf> , with several optimizers (such Adam, AdaMax, SGC, etc.); while, as expected, SGD-ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</inf> allows us to use larger SS without divergence problems, (i) it also matches a well-tuned Adam’s performance (superior to "default parameters" Adam), and (ii) heuristically, its convergence properties (rate, oscillations, etc.) are superior when compared to other well-known algorithms.
Prosody Is Not Identity: A Speaker Anonymization Approach Using Prosody Cloning

Prosody is closely linked to the identity of a speaker, leading to individual pitch and intonation patterns. Therefore, it is challenging in speaker anonymization to generate speech utterances that both keep the original audio’s main prosodic structure and preserve the speaker’s privacy. In this paper, we present a system that extends a speech-to-text-to-speech anonymization pipeline with prosody cloning and show how to control the cloning by multiplying pitch and energy sequences with random offset values. Using automatic and human evaluation, we find this combination to successfully overcome the privacy-utility trade-off for prosody by achieving high privacy and high pitch correlation scores. At the same time, the anonymized utterances prove to reproduce the original voice distinctiveness and content with high intelligibility and only a small loss in naturalness, making them suitable for downstream applications.