Reviewerly/exHarmony huggingface dataset:
IoU-Aware Multi-Expert Cascade Network Via Dynamic Ensemble for Long-Tailed Object Detection

Object detection over a long-tailed large-scale dataset is practical, challenging, and comprehensively under-explored. Recently proposed methods mainly focus on eliminating the imbalanced classification problem. However, only a few attempts have been made to consider the quality of the predicted bounding boxes. Inspired by the observation of existing Cascade architecture, "detectors with specific IoU thresholds excel at different label frequencies of bounding boxes," this paper first pinpoints the issue in long-tailed distribution. A detector may predict inaccurate bounding boxes on the categories of fewer training data such that the corresponding extracted visual features could further degrade the classification accuracy. Thus, the predicted accuracy of bounding boxes becomes substantially different among categories in the long-tailed distribution. We introduce a Multi-Expert Cascade (MEC) framework that readjusts the weight of each category in the training process via a multi-expert loss. Furthermore, we leverage dynamic ensemble mechanisms at inference time to fully utilize expert detectors and achieve better performance. Extensive experiments on the recent long-tailed large vocabulary object detection dataset show that the proposed MEC framework significantly improves the performance of most widely-used detectors over various backbones on object detection and instance segmentation tasks.
Efficient Learning of Balanced Signature Graphs

The novel concept of signature graphs extends signed graphs by admitting multiple types of partial similarity/agreement or dissimilarity/disagreement. Extending the concept of balancedness to signature graphs yields an explicit and efficient basis for multi-class clustering and classification. Contrary to existing two-stage approaches that consist of graph learning followed by graph clustering, we propose a one-step procedure that directly learns a perfectly clustered graph. We describe the algorithmic constituents for our approach and illustrate its superiority via numerical simulations.
Filterbank Learning for Noise-Robust Small-Footprint Keyword Spotting

In the context of keyword spotting (KWS), the replacement of handcrafted speech features by learnable features has not yielded superior KWS performance. In this study, we demonstrate that filterbank learning outperforms handcrafted speech features for KWS whenever the number of filterbank channels is severely decreased. Reducing the number of channels might yield certain KWS performance drop, but also a substantial energy consumption reduction, which is key when deploying common always-on KWS on low-resource devices. Experimental results on a noisy version of the Google Speech Commands Dataset show that filterbank learning adapts to noise characteristics to provide a higher degree of robustness to noise, especially when dropout is integrated. Thus, switching from typically used 40-channel log-Mel features to 8channel learned features leads to a relative KWS accuracy loss of only 3.5% while simultaneously achieving a 6.3× energy consumption reduction.
Efficient Feature Extraction for Non-Maximum Suppression in Visual Person Detection

Non-Maximum Suppression (NMS) is a post-processing step in almost every visual object detector, tasked with rapidly pruning the number of overlapping detected candidate rectangular Regions-of-Interest (RoIs) and replacing them with a single, more spatially accurate detection (in pixel coordinates). The common Greedy NMS algorithm suffers from drawbacks, due to the need for careful manual tuning. In visual person detection, most NMS methods typically suffer when analyzing crowded scenes with high levels of in-between occlusions. This paper proposes a modification on a deep neural architecture for NMS, suitable for such cases and capable of efficiently cooperating with recent neural object detectors. The method approaches the NMS problem as a rescoring task, aiming to ideally assign precisely one detection per object. The proposed modification exploits the extraction of RoI representations, semantically capturing the region’s visual appearance, from information-rich feature maps computed by the detector’s intermediate layers. Experimental evaluation on two common public person detection datasets shows improved accuracy against competing methods, with acceptable inference speed.
Improving Scheduled Sampling for Neural Transducer-Based ASR

The recurrent neural network-transducer (RNNT) is a promising approach for automatic speech recognition (ASR) with the introduction of a prediction network that autoregressively considers linguistic aspects. To train the autoregressive part, the ground-truth tokens are used as substitutions for the previous output token, which leads to insufficient robustness to incorrect past tokens; a recognition error in the decoding leads to further errors. Scheduled sampling (SS) is a technique to train autoregressive model robustly to past errors by randomly replacing some ground-truth tokens with actual outputs generated from a model. SS mitigates the gaps between training and decoding steps, known as exposure bias, and it is often used for attentional encoder-decoder training. However SS has not been fully examined for RNNT because of the difficulty in applying SS to RNNT due to the complicated RNNT output form. In this paper we propose SS approaches suited for RNNT. Our SS approaches sample the tokens generated from the distiribution of RNNT itself, i.e. internal language model or RNNT outputs. Experiments in three datasets confirm that RNNT trained with our SS approach achieves the best ASR performance. In particular, on a Japanese ASR task, our best system outperforms the previous state-of-the-art alternative.
Voice-Preserving Zero-Shot Multiple Accent Conversion

Most people who have tried to learn a foreign language would have experienced difficulties understanding or speaking with a native speaker’s accent. For native speakers, understanding or speaking a new accent is likewise a difficult task. An accent conversion system that changes a speaker’s accent but preserves that speaker’s voice identity, such as timbre and pitch, has the potential for a range of applications, such as communication, language learning, and entertainment. Existing accent conversion models tend to change the speaker identity and accent at the same time. Here, we use adversarial learning to disentangle accent dependent features while retaining other acoustic characteristics. What sets our work apart from existing accent conversion models is the capability to convert an unseen speaker’s utterance to multiple accents while preserving its original voice identity. Subjective evaluations show that our model generates audio that sound closer to the target accent and like the original speaker.
Video Captioning via Relation-Aware Graph Learning

Recent neural models for video captioning usually employed an encoder-decoder framework. However, most approaches either neglected the spatial and temporal interactions between objects in a video or implicitly modelled the interactions, resulting in less desired performance. In this paper, we propose a novel relation-aware graph learning framework. It explicitly models both spatial and temporal relations for objects. In particular, a relation-aware graph is designed to depict the spatial relations between different objects in a scene. Parallelly, a temporal graph network is designed to perform relational reasoning for the same objects in adjacent frames. Features of both types of relations are learned and fused for the follow-up language decoder. Experiments on two bench-mark datasets show the effectiveness of our framework. It achieves state-of-the-art performance with CIDEr scores on MSVD and MSR-VTT.
Central Nodes Detection from Partially Observed Graph Signals

This paper focuses on detecting the central nodes in a graph from partially observed graph signals with unknown graph topology. We follow a general model which considers observed data as filtered graph signals with excitation driven from some external sources shaped by a possibly low-rank influence matrix. To identify the full centrality vector, we rely on the key observation that the vector is embedded in a linear system with the influence matrix. We provide the necessary and sufficient conditions for the centrality vector to be uniquely recovered. Notably, among other requirements, our conditions show that the number of external sources needs to be larger than the number of hidden nodes. Finally, we design an alternating minimization algorithm to estimate centrality vectors from the partially observed signals. Numerical results support our findings.
QOS-Aware Task Scheduling in An Edge Environment

In Edge frameworks, Virtual Machines (VM) are used for the asset usage as per the planning by the users. The most common example for this is accessible RAM, disregarding about the long haul usage. The planning and situation processes are computationally of high cost and the influence of execution is conveyed. In this paper, we propose a Edge VM planning calculation that considers previously running VM asset usage over the long run by the dissecting past VM usage levels to plan the current VMs. This is done by advancing the execution in the utilization of K-Means Nearest Neighbor (KNN) and Naive Bayes characterization procedure. The Euclidean distance of KNN is estimated and then the virtual machine is booked on the actual machine. The Edge in the board processes the VM situation & influences the previously sent frameworks so the point is to limit such execution corruption. In addition to this, over-burden of VMs will quite often take the assets from the adjoining VMs, so that the work boosts VMs genuine CPU usage. The outcome of our experimental setup shows the refinement in the conventional instant-based actual machine determination which learns the framework conduct as well as it adjusts over the long run. It also improvises the counts of the actual machines which gets diminished by the utilization of K-NN and NB classifier rather than traditional usage of Support Vector Machine (SVM) classifier.
Human Pose Estimation from Ambiguous Pressure Recordings with Spatio-Temporal Masked Transformers

Despite the impressive performance of vision-based pose estimators, they generally fail to perform well under adverse vision conditions and often don’t satisfy the privacy demands of customers. As a result, researchers have begun to study tactile sensing systems as an alternative. However, these systems suffer from noisy and ambiguous recordings. To tackle this problem, we propose a novel solution for pose estimation from ambiguous pressure data. Our method comprises a spatio-temporal vision transformer with an encoder-decoder architecture. Detailed experiments on two popular public datasets reveal that our model outperforms existing solutions in the area. Moreover, we observe that increasing the number of temporal crops in the early stages of the network positively impacts the performance while pre-training the network in a self-supervised setting using a masked auto-encoder approach also further improves the results.
EfficientSpeech: An On-Device Text to Speech Model

State of the art (SOTA) neural text to speech (TTS) models can generate natural-sounding synthetic voices. These models are characterized by large memory footprints and substantial number of operations due to the long-standing focus on speech quality with cloud inference in mind. Neural TTS models are generally not designed to perform standalone speech syntheses on resource-constrained and no Internet access edge devices. In this work, an efficient neural TTS called EfficientSpeech that synthesizes speech on an ARM CPU in real-time is proposed. EfficientSpeech uses a shallow non-autoregressive pyramid-structure transformer forming a U-Network. EfficientSpeech has 266k parameters and consumes 90 MFLOPS only or about 1% of the size and amount of computation in modern compact models such as Mixer-TTS. EfficientSpeech achieves an average mel generation real-time factor of 104.3 on an RPi4. Human evaluation shows only a slight degradation in audio quality as compared to FastSpeech2.
Pair DETR: Toward Faster Convergent DETR

The DETR object detection approach applies the Transformer encoder and decoder architecture to detect objects from images and achieves promising performance. In this paper, we present a simple approach to address the main problem of DETR, the slow convergence, by using representation learning technique. In this approach, we detect an object bounding box as a pair of keypoints, the top-left corner and the center, using two decoders. By detecting objects as paired keypoints, the model builds up a joint classification and pair association on the output queries from two decoders. For the pair association we propose utilizing contrastive self-supervised learning algorithm without requiring specialized architecture. Experimental results on MS COCO dataset show that Pair DETR can reduce training epoch by 10x from original DETR and 1.5x from Conditional DETR, while having consistently higher Average Precision scores.
BIRD-PCC: Bi-Directional Range Image-Based Deep Lidar Point Cloud Compression

The large amount of data collected by LiDAR sensors brings the issue of LiDAR point cloud compression (PCC). Previous works on LiDAR PCC have used range image representations and followed the predictive coding paradigm to create a basic prototype of a coding framework. However, their prediction methods give an inaccurate result due to the negligence of invalid pixels in range images and the omission of future frames in the time step. Moreover, their handcrafted design of residual coding methods could not fully exploit spatial redundancy. To remedy this, we propose a coding framework BIRD-PCC. Our prediction module is aware of the coordinates of invalid pixels in range images and takes a bidirectional scheme. Also, we introduce a deep-learned residual coding module that can further exploit spatial redundancy within a residual frame. Experiments conducted on SemanticKITTI and KITTI-360 datasets show that BIRD-PCC outperforms other methods in most bitrate conditions and generalizes well to unseen environments.
CFFMixer: Multi-Dimensional Feature Fusion for Object Detection

Object detection is a fundamental task in the field of computer vision, and one of its essential requirements is high-quality feature fusion. Previous works have made various efforts in this regard: CNN-based detectors use convolutional blocks to fuse local features and dense prior knowledge to predict objects, while query-based detectors fuse global features by self-attention then decode features with object queries. However, their feature fusion methods are relatively monotonous. Considering that different modules are applicable to different dimensions, we proposed an object detector named CFFMixer which used hybrid architecture to achieve multi-dimensional feature fusion. The sampling strategy to extract abundant local and global features was first introduced then the Comprehensive Feature Fusion Network (CFFN) was proposed to integrate them. CFFN not only achieved local and global features interaction in the spatial dimension, but also fused semantics in the channel dimension. Furthermore, we conducted experiments and made a comparison with competitive models, our model finally got 43.0 mAP on COCO 2017 dataset within 12 epochs. Experimental results showed that the model’s accuracy benefits from the powerful feature fusion capability of CFFN. Besides, we performed ablation studies on our modules to evaluate their effectiveness.
Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samples

The objective of this work is to explore the learning of visually grounded speech models (VGS) from multilingual perspective. Bilingual VGS models are generally trained with an equal number of spoken captions from both languages. However, in reality, there can be an imbalance among the languages for the available spoken captions. Our key contribution in this work is to leverage the power of a high-resource language in a bilingual visually grounded speech model to improve the performance of a low-resource language. We introduce two methods to distill the knowledge of high-resource language into low-resource languages: (1) incorporating a strong pre-trained high-resource language encoder and (2) using semantically similar spoken captions. Our experiments show that combining these two approaches effectively enables the low-resource language to surpass the performances of monolingual and bilingual counterparts for cross-modal retrieval tasks.
Implicitly Rotation Equivariant Neural Networks

Convolutional Neural Networks (CNN) are inherently equivariant under translations, however, they do not have an equivalent embedded mechanism to handle other transformations such as rotations. The existing solutions require redesigning standard networks with filters mapped from combinations of predefined basis involving complex analytical functions. Such formulations are hard to implement as well as the imposed restrictions in the choice of basis can lead to model weights that are sub-optimal for the primary deep learning task (e.g. classification). We propose Implicitly Equivariant Network (IEN) which induces approximate equivariance in the different layers of a standard CNN by optimizing a multi-objective loss function. We show for ResNet models on Rot-MNIST and Rot-TinyImageNet that even with its simple formulation, IEN performs at par or even better than steerable networks. Also, IEN facilitates construction of heterogeneous filter groups allowing reduction in the number of channels in CNNs by a factor of over 30%. Further, we demonstrate that for the hard problem of visual object tracking, IEN outperforms the state-of-the-art rotation equivariant tracking method while providing faster inference speed.
A Two-Stage System for Spoken Language Understanding

In this paper, we propose a Two-Stage system for SLU, which consists of Automatic Speech Recognition (ASR) tasks and Natural Language Understanding (NLU) tasks. Our work is done in the context of an ICASSP 2023 Spoken Langauge Understanding Grand Challenge. In the first stage, we use a model based on the encoder-decoder structure to recognize the speech utterance into text. In the second stage, we combine Bidirectional Encoder Representations for Transformers (BERT) and Conditional Random Field (CRF) to train intent determination and slot filling jointly. In addition, we compress the data using Byte Pair Encoding (BPE) and construct the BERT word list by pre-training to save a large number of parameters, thus extending the model depth and obtaining an Exact Match boost. The final experiments demonstrate the effectiveness of our proposed model.
AE-Flow: Autoencoder Normalizing Flow

Recently normalizing flows have been gaining traction in text-to-speech (TTS) and voice conversion (VC) due to their state-of-the-art (SOTA) performance. Normalizing flows are unsupervised generative models. In this paper, we introduce supervision to the training process of normalizing flows, without the need for parallel data. We call this training paradigm AutoEncoder Normalizing Flow (AE-Flow). It adds a reconstruction loss forcing the model to use information from the conditioning to reconstruct an audio sample. Our goal is to understand the impact of each component and find the right combination of the negative log-likelihood (NLL) and the reconstruction loss in training normalizing flows with coupling blocks. For that reason we will compare flow-based mapping model trained with: (i) NLL loss, (ii) NLL and reconstruction losses, as well as (iii) reconstruction loss only. Additionally, we compare our model with SOTA VC baseline. The models are evaluated in terms of naturalness, speaker similarity, intelligibility in many-to-many and many-to-any VC settings. The results show that the proposed training paradigm systematically improves speaker similarity and naturalness when compared to regular training methods of normalizing flows. Furthermore, we show that our method improves speaker similarity and intelligibility over the state-of-the-art.
MvCo-DoT: Multi-View Contrastive Domain Transfer Network for Medical Report Generation

In clinical scenarios, multiple medical images with different views are usually generated at the same time, and they have high semantic consistency. However, the existing medical report generation methods cannot exploit the rich multi-view mutual information of medical images. Therefore, in this work, we propose the first multi-view medical report generation model, called MvCo-DoT. Specifically, MvCo-DoT first propose a multi-view contrastive learning (MvCo) strategy to help the deep reinforcement learning based model utilize the consistency of multi-view inputs for better model learning. Then, to close the performance gaps of using multi-view and single-view inputs, a domain transfer network is further proposed to ensure MvCo-DoT achieve almost the same performance as multi-view inputs using only single-view inputs. Extensive experiments on the IU X-Ray public dataset show that MvCo-DoT outperforms the SOTA medical report generation baselines in all metrics.
Twitter Stance Detection via Neural Production Systems

Stance detection is an important task, which aims to classify the attitude of an opinionated text toward a given target. In this paper, we develop an interpretable neural production system for stance detection (NPS4SD). NPS4SD is an end-to-end deep learning model, which consists of a set of knowledge rules that are applied by binding with specific entities. NPS4SD consists of two main components: a pretrained model for learning the text representation and a variable binding network (VBN) to bind the knowledge rules with text entities. Extensive experiments are conducted to evaluate the effectiveness of the proposed NPS4SD model on three real-world datasets with in-domain, cross-target and zero-shot setups. Experimental results demonstrate that NPS4SD achieves substantially better performance than the strong competitors for the stance detection task.
A Unified One-Shot Prosody and Speaker Conversion System with Self-Supervised Discrete Speech Units

We present a unified system to realize one-shot voice conversion (VC) on the pitch, rhythm, and speaker attributes. Existing works generally ignore the correlation between prosody and language content, leading to the degradation of naturalness in converted speech. Additionally, the lack of proper language features prevents these systems from accurately preserving language content after conversion. To address these issues, we devise a cascaded modular system leveraging self-supervised discrete speech units as language representation. These discrete units provide duration information essential for rhythm modeling. Our system first extracts utterance-level prosody and speaker representations from the raw waveform. Given the prosody representation, a prosody predictor estimates pitch, energy, and duration for each discrete unit in the utterance. A synthesizer further reconstructs speech based on the predicted prosody, speaker representation, and discrete units. Experiments show that our system outperforms previous approaches in naturalness, intelligibility, speaker transferability, and prosody transferability. Code and samples are publicly available. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>