Reviewerly/exHarmony huggingface dataset:
Movienet-PS: A Large-Scale Person Search Dataset in the Wild

Person search (PS) aims to jointly localize and identify a query person from natural, uncropped images. Existing works unintentionally adopt pedestrians (with similar poses and unchanging clothing) as the query and restrict the application scenarios in surveillance. This is due to that most PS datasets are collected from surveillance cameras with a limited diversity of views, scenes, appearances, etc. In this paper, we study a more general and realistic task in the wild, where we aim to search target persons with a much higher degree of diversity. To this end, we introduce a new PS dataset, namely MovieNet-PS, based on an existing large-scale movie dataset. MovieNet-PS is currently the largest and most diverse PS dataset, consisting of 160K images (100K for training), 274K bounding boxes, and 3K identities. It stands out from existing counterparts from two levels of diversities, i.e., scene-level and identity-level, with 92,043 scenes and significant variations in poses, clothing, scales, etc. for the same identity. To validate the rich context information on our dataset and make full use of it, we propose a novel global-local context network, which exploits scene and group context to boost the search performance. Extensive experiments demonstrate that MovieNet-PS is more challenging and comprehensive than existing datasets, and our approach further pushes the state of the art by a large margin (relatively 34% in mAP) on this dataset. Codes, models, and the dataset are available at: https://github.com/ZhengPeng7/GLCNet.
Yolo-Based Lightweight Object Detection With Structure Simplification And Attention Enhancement

In this paper we propose a lightweight object detector by optimizing the structure of YOLOv3. The optimization is conducted in two aspects: simplifying the structural components by lightweight substitutes and introducing the attention mechanism to increase the detection accuracy. For the simplification, we remodel the backbone based on MobileNet v2 and replace every 3×3 convolution in the detection neck and head by the fusion of a 3 × 3 depthwise separable convolution and a squeeze and excitation block (DSConv+SE); for the attention enhancement, we introduce the high-frequency wavelets of the original image to the input, a simplified non- local block to the simplified backbone and convolutional block attention modules to the simplified detection neck. In addition, local 3×3 convolution branches are introduced to the simplified backbone for enhanced learning capability. Experiments demonstrate that the proposed detector outperforms each compared state-of-the-art work in one or more aspects.
Face Recognition on Point Cloud with Cgan-Top for Denoising

Face recognition using 3D point clouds is gaining growing interest, while raw point clouds often contain a significant amount of noise due to imperfect sensors. In this paper, an end-to-end 3D face recognition on a noisy point cloud is proposed, which synergistically integrates the denoising and recognition modules. Specifically, a Conditional Generative Adversarial Network on Three Orthogonal Planes (cGAN-TOP) is designed to effectively remove the noise in the point cloud, and recover the underlying features for subsequent recognition. A Linked Dynamic Graph Convolutional Neural Network (LDGCNN) is then adapted to recognize faces from the processed point cloud, which hierarchically links both the local point features and neighboring features of multiple scales. The proposed method is validated on the Bosphorus dataset. It significantly improves the recognition accuracy under all noise settings, with a maximum gain of 14.81%.
Deep Autoencoding One-Class time Series Anomaly Detection

Time-series Anomaly Detection(AD) is widely used in monitoring and security applications in various industries and has become a hot spot in the field of deep learning. Normality-representation-based methods perform well in certain scenarios but may ignore some aspects of the overall normality. Feature-extraction-based methods always take a process of pre-training, whose target differs from AD, leading to a decline in AD performance. In this paper, we propose a new AD method called deep Autoencoding One-Class (AOC), which learns features with AutoEncoder(AE). Meanwhile, the normal context vectors from AE are constrained into a hypersphere small enough, similar to one-class methods. With an objective function that optimizes the two assumptions simultaneously, AOC learns various aspects of normality, which is more effective for AD. Experiments on public datasets show that our method outperforms existing baseline approaches.
Choice Fusion As Knowledge For Zero-Shot Dialogue State Tracking

With the demanding need for deploying dialogue systems in new domains with less cost, zero-shot dialogue state tracking (DST), which tracks user’s requirements in task-oriented dialogues without training on desired domains, draws attention increasingly. Although prior works have leveraged question-answering (QA) data to reduce the need for in-domain training in DST, they fail to explicitly model knowledge transfer and fusion for tracking dialogue states. To address this issue, we propose CoFunDST, which is trained on domain-agnostic QA datasets and directly uses candidate choices of slot-values as knowledge for zero-shot dialogue-state generation, based on a T5 pre-trained language model. Specifically, CoFunDST selects highly-relevant choices to the reference context and fuses them to initialize the decoder to constrain the model outputs. Our experimental results show that our proposed model achieves outperformed joint goal accuracy compared to existing zero-shot DST approaches in most domains on the MultiWOZ 2.1. Extensive analyses demonstrate the effectiveness of our proposed approach for improving zero-shot DST learning from QA.
Pretraining Conformer with ASR for Speaker Verification

This paper proposes to pretrain Conformer with automatic speech recognition (ASR) task for speaker verification. Conformer combines convolution neural network (CNN) and Transformer model for modeling local and global features, respectively. Recently, multi-scale feature aggregation Conformer (MFA-Conformer) has been proposed for automatic speaker verification. MFA-Conformer concatenates frame-level outputs from all Conformer blocks for further pooling. However, our experiments show that Conformer can be easily overfitted with limited speaker recognition training data. To avoid overfitting, we propose to transfer the knowledge learned from ASR to speaker verification. Specifically, an ASR pretrained Conformer is used to initialize the training of MFA-Conformer for speaker verification. Our experiments show that pretraining Conformer with ASR leads to significant performance gains across model sizes. The best model achieves 0.48%, 0.71% and 1.54% EER on Voxceleb1-O, Voxceleb1-E, and Voxceleb1-H, respectively.
Efficient Super-Resolution for Compression Of Gaming Videos

Due to the increasing demand for game-streaming services, efficient compression of computer-generated video is more critical than ever, especially when the available bandwidth is low. This paper proposes a super-resolution framework that improves the coding efficiency of computer-generated gaming videos at low bitrates. Most state-of-the-art super-resolution networks generalize over a variety of RGB inputs and use a unified network architecture for frames of different levels of degradation, leading to high complexity and redundancy. Since games usually consist of a limited number of fixed scenarios, we specialize one model for each scenario and assign appropriate network capacities for different QPs to perform super-resolution under the guidance of reconstructed high-quality luma components. Experimental results show that our framework achieves a superior quality-complexity trade-off compared to the ESRnet baseline, saving at most 93.59% parameters while maintaining comparable performance. The compression efficiency compared to HEVC is also improved by more than 17% BD-rate gain.
Multi-modal ASR error correction with joint ASR error detection

To tackle the recognition error problem for Automatic speech recognition (ASR), one common approach is to utilize text-based ASR error correction methods focusing on text error pat-terns. To include the audio information for better error correction, we propose a sequence-to-sequence multi-modal ASR error correction model. The multi-modal representations from pre-trained audio and text encoders are fused and aligned based on an attention mechanism. The decoder then searches for the corresponding correction results based on the fused representations. To better explore the correlations between different modalities, an additional ASR error detection task is applied on top of the fused representations. We optimize the network by a multi-task learning method combining both ASR error detection and correction tasks. Experimental results based on a 200-hour dataset recorded by Chinese English-as-second-language (ESL) learners show the proposed correction model can achieve significant improvement compared to the baselines with or without other ASR error correction methods.
Benchmarking White Blood Cell Classification under Domain Shift

Recognizing the types of white blood cells (WBCs) in microscopic images of human blood smears is a fundamental task in the fields of pathology and hematology. Although previous studies have made significant contributions to the development of methods and datasets, few papers have investigated benchmarks or baselines that others can easily refer to. For instance, we observed notable variations in the reported accuracies of the same Convolutional Neural Network (CNN) model across different studies, yet no public implementation exists to reproduce these results. In this paper, we establish a benchmark for WBC recognition. Our results indicate that CNN-based models achieve high accuracy when trained and tested under similar imaging conditions. However, their performance drops significantly when tested under different conditions. Moreover, the ResNet classifier, which has been widely employed in previous work, exhibits an unreasonably poor generalization ability under domain shifts due to batch normalization. We investigate this issue and suggest some alternative normalization techniques that can mitigate it. We make fully-reproducible code publicly available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Self-Attention for Enhanced OAMP Detection in MIMO Systems

Multiple-Input Multiple-Output (MIMO) systems are essential for wireless communications. Since classical algorithms for symbol detection in MIMO setups require large computational resources or provide poor results, data-driven algorithms are becoming more popular. Most of the proposed algorithms, however, introduce approximations leading to degraded performance for realistic MIMO systems. In this paper, we introduce a neural-enhanced hybrid model, augmenting the analytic backbone algorithm with state-of-the-art neural network components. In particular, we introduce a self-attention model for the enhancement of the iterative Orthogonal Approximate Message Passing (OAMP)-based decoding algorithm. In our experiments, we show that the proposed model can outperform existing data-driven approaches for OAMP while having improved generalization to other SNR values at limited computational overhead.
CANDY: Category-Kernelized Dynamic Convolution for Instance Segmentation

Instance segmentation has been dominated by the paradigm that predicts masks using local RoI features and simplicity frameworks based on global mask prediction. Despite the comparable performance between local-based and global-based approaches, the AP results of objects on different scales vary significantly. In this paper, we first point out that the key factor to bridging such a gap lies in the utilization of local RoI information for global mask prediction. Then, we observe a ’class-agnostic segmentation’ problem exists in the nearby region of interesting objects after implementing the above combination. To overcome this issue, we further propose a CAtegory-kerNelized DYnamic (CANDY) convolution. Benefiting from it, the discriminative ability of the resulting instance segmentation framework, i.e., CANDY-Mask, on foreground objects is significantly enhanced. Extensive experiments on the MS-COCO dataset are conducted to verify the performance of CANDY-Mask. Our proposed CANDY-Mask obtains 48.1% boxes AP and 40.7% masks AP on the MS-COCO test-dev set with ResNet50 backbone, achieving state-of-the-art among various models.
Sora: Scalable Black-Box Reachability Analyser on Neural Networks

The vulnerability of deep neural networks (DNNs) to input perturbations has posed a significant challenge. Recent work on robustness verification of DNNs not only lacks scalability but also requires severe restrictions on the architecture (layers, activation functions, etc.). To address these limitations, we propose a novel framework, SORA, for scalable blackbox reachability analysis of DNNs. SORA can work on a broad class of neural network structures, including those networks with very deep layers and a huge number of neurons with nonlinear activation functions. Based on the Lipschitz continuity, SORA verifies the reachability property of DNNs with a novel optimisation algorithm and has global convergence guarantee. Our method does not require access to the inner structures of the DNNs, hence a black-box method. Experimental results show that, compared to existing verification methods, SORA shows superior performance in terms of both efficiency and scalability, especially when handling a deep neural network with very deep layers and a large number of neurons with various types of nonlinear activation functions.
One-Shot Medical Action Recognition With A Cross-Attention Mechanism And Dynamic Time Warping

In this paper, we address the classification of medical actions with only one single sample by developing a novel one-shot learning framework which contains both cross-attention and dynamic time warping (DTW) modules. To be concrete, we firstly transform the raw skeleton sequence into the signal-level image representation. We exploit a metric learning approach, which is the prototypical network for the proposed one-shot learning framework and choose the residual network (ResNet18) as the backbone which is widely used in recent years. Cross-attention is applied for guiding the network to focus on the more important joints from each specific action. The cross-attention mechanism that applies between the support and query set will be adapted for mining and matching the relationships with the human body. Furthermore, a DTW module is introduced to mitigate the temporal information mismatching issue between the actions from the support and query sets. The experimental results on the NTU RGB+D 120 dataset demonstrate the effectiveness of our proposed approach and the improved performance compared to the baseline approach. The code of this work is available at <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Audio-Visual Inpainting: Reconstructing Missing Visual Information with Sound

We tackle audio-visual inpainting, the problem of completing an image in such a way to be consistent with the sound associated to the scene. To this end, we propose a multimodal, audio-visual inpainting method (AVIN), and show how to leverage sound to reconstruct semantically consistent images. AVIN is a 2-stage algorithm, which first learns the scene semantics and reconstructs low resolution images based on a conditional probability distribution of pixels in the space conditioned to audio, and then refines such result with a GAN-based network to increase the resolution of the reconstructed image. We show that AVIN is able to recover the original content, especially in the hard cases where the missing area heavily degrades the scene semantics: it can perform cross-modal generation whenever no visual context is observed at all, reconstructing visual data from sound only. Code will be made available upon acceptance.
More Speaking or More Speakers?

Self-training (ST) and self-supervised learning (SSL) methods have demonstrated strong improvements in automatic speech recognition (ASR). In spite of these advances, to the best of our knowledge, there is no analysis of how the composition of the labelled and unlabelled datasets used in these methods affects the results. In this work we aim to analyse the effect of number of speakers in the training data on a recent SSL algorithm (wav2vec 2.0), and a recent ST algorithm (slimIPL). We perform a systematic analysis on both labeled and unlabeled data by varying the number of speakers while keeping the number of hours fixed and vice versa. Our findings suggest that SSL requires a large amount of unlabeled data to produce high accuracy results, while ST requires a sufficient number of speakers in the labelled data, especially in the low-regime setting. In this manner these two approaches improve supervised learning in different regimes of data composition.
Embedding a Differentiable Mel-Cepstral Synthesis Filter to a Neural Speech Synthesis System

This paper integrates a classic mel-cepstral synthesis filter into a modern neural speech synthesis system towards end-to-end controllable speech synthesis. Since the mel-cepstral synthesis filter is explicitly embedded in neural waveform models in the proposed system, both voice characteristics and the pitch of synthesized speech are highly controlled via a frequency warping parameter and fundamental frequency, respectively. We implement the mel-cepstral synthesis filter as a differentiable and GPU-friendly module to enable the acoustic and waveform models in the proposed system to be simultaneously optimized in an end-to-end manner. Experiments show that the proposed system improves speech quality from a baseline system maintaining controllability. The core PyTorch modules used in the experiments are publicly available on GitHub <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Ideal: Improved Dense Local Contrastive Learning For Semi-Supervised Medical Image Segmentation

Due to the scarcity of labeled data, Contrastive Self-Supervised Learning (SSL) frameworks have lately shown great potential in several medical image analysis tasks. However, the existing contrastive mechanisms are sub-optimal for dense pixel-level segmentation tasks due to their inability to mine local features. To this end, we extend the concept of metric learning to the segmentation task, using a dense (dis)similarity learning for pre-training a deep encoder network, and employing a semi-supervised paradigm to fine-tune for the downstream task. Specifically, we propose a simple convolutional projection head for obtaining dense pixel-level features, and a new contrastive loss to utilize these dense projections thereby improving the local representations. A bidirectional consistency regularization mechanism involving two-stream model training is devised for the downstream task. Upon comparison, our IDEAL method outperforms the SoTA methods by fair margins on cardiac MRI segmentation. Our source codes are publicly accessible at: https://github.com/Rohit-Kundu/IDEAL-ICASSP23.
Reducing Language Confusion for Code-Switching Speech Recognition with Token-Level Language Diarization

Code-switching (CS) occurs when languages switch within a speech signal and leads to language confusion for automatic speech recognition (ASR). We address the problem of language confusion for improving CS-ASR from two perspectives: incorporating and disentangling language information. We incorporate language information within the CS-ASR model by dynamically biasing the model with token-level language posteriors corresponding to outputs of a sequence-to-sequence auxiliary language diarization (LD) module. In contrast, the disentangling process reduces the difference between languages via adversarial training so as to normalize two languages. We conduct experiments on the SEAME dataset. Compared to the baseline model, both the joint optimization with LD and the language posterior bias achieve performance improvement. Comparison of the proposed methods indicates that incorporating language information is more effective than disentangling for reducing language confusion in CS speech.
A3S: Adversarial Learning of Semantic Representations for Scene-Text Spotting

Scene-text spotting is a task that predicts a text area on natural scene images and recognizes its text characters simultaneously. It has attracted much attention in recent years due to its wide applications. Existing research has mainly focused on improving text region detection, not text recognition. Thus, while detection accuracy is improved, the end-to-end accuracy is insufficient. Texts in natural scene images tend to not be a random string of characters but a meaningful string of characters, a word. Therefore, we propose adversarial learning of semantic representations for scene text spotting (A3S) to improve end-to-end accuracy, including text recognition. A3S simultaneously predicts semantic features in the detected text area instead of only performing text recognition based on existing visual features. Experimental results on publicly available datasets show that the proposed method achieves better accuracy than other methods.
Terminology-Aware Medical Dialogue Generation

Medical dialogue generation aims to generate responses according to a history of dialogue turns between doctors and patients. Unlike open-domain dialogue generation, this requires background knowledge specific to the medical domain. Existing generative frameworks for medical dialogue generation fall short of incorporating domain-specific knowledge, especially with regard to medical terminology. In this paper, we propose a novel framework to improve medical dialogue generation by considering features centered on domain-specific terminology. We leverage an attention mechanism to incorporate terminologically centred features, and fill in the semantic gap between medical background knowledge and common utterances by enforcing language models to learn terminology representations with an auxiliary terminology recognition task. Experimental results demonstrate the effectiveness of our approach, in which our proposed framework outperforms SOTA language models. Additionally, we provide a new dataset with medical terminology annotations to support research on medical dialogue generation. Our dataset and code are available at https://github.com/tangg555/meddialog.
Towards Adversarially Robust Continual Learning

Recent studies show that models trained by continual learning can achieve the comparable performances as the standard supervised learning and the learning flexibility of continual learning models enables their wide applications in the real world. Deep learning models, however, are shown to be vulnerable to adversarial attacks. Though there are many studies on the model robustness in the context of standard supervised learning, protecting continual learning from adversarial attacks has not yet been investigated. To fill in this research gap, we are the first to study adversarial robustness in continual learning and propose a novel method called Task-Aware Boundary Augmentation (TABA) to boost the robustness of continual learning models. With extensive experiments on CIFAR-10 and CIFAR-100, we show the efficacy of adversarial training and TABA in defending adversarial attacks.