Reviewerly/exHarmony huggingface dataset:
Sparse Black-Box Inversion Attack with Limited Information

Existing black-box model inversion attacks mainly focus on training and attacking surrogate models. However, due to the deployment process of face recognition models, training surrogate models becomes extremely difficult in practice. At the same time, query-based black-box inversion attacks still suffer from low image quality and high computational costs. To bridge these gaps, in this paper, we propose BMI-S, a sparse black-box inversion attack against face recognition models. BMI-S first introduces evolution strategies to perform efficient black-box gradient estimation and achieve query-based attacks. Meanwhile, BMI-S performs sparse attacks on the key styles that contribute most to the face recognition process. By only optimizing key style control vectors, BMI-S further narrows the dimensions of the search space and accelerates the inversion attacks.
LEAPT: Learning Adaptive Prefix-to-Prefix Translation For Simultaneous Machine Translation

Simultaneous machine translation, which aims at a realtime translation, is useful in many live scenarios but very challenging due to the trade-off between accuracy and latency. To achieve the balance for both, the model needs to wait for appropriate streaming text (READ policy) and then generates its translation (WRITE policy). However, WRITE policies of previous work either are specific to the method itself due to the end-to-end training or suffer from the input mismatch between training and decoding for the non-end-to-end training. Therefore, it is essential to learn a generic and better WRITE policy for simultaneous machine translation. Inspired by strategies utilized by human interpreters and "wait" policies, we propose a novel adaptive prefix-to-prefix training policy called LEAPT, which allows our machine translation model to learn how to translate source sentence prefixes and make use of the future context. Experiments show that our proposed methods greatly outperform competitive baselines and achieve promising results.
A Perturbation-Based Policy Distillation Framework with Generative Adversarial Nets

We study the problem of imitation learning in automated decision systems, in which a learner is trained to imitate an expert demonstrator. A widely used method is adversarial imitation learning that alternately optimizes a generator (learner) and a discriminator (reward function). However, the discriminator is biased during the initial and intermediate training stages. Consequently, the gradient descent direction of the learner is misguided, which leads to unstable training and sample complexity. In this paper, we propose deep imitation learning through a guidance-based policy distillation (GIL) algorithm. First, GIL proposes a teacher model, the guidance-based variational autoencoder, which is pre-trained with expert demonstrations. Then, GIL proposes a perturbation-based policy distillation method that uses the teacher model to guide the learner in the correct optimization direction, enabling the learner to imitate the expert policy with fewer detours. The experimental results show that our approach achieve higher sample efficiency compared with multiple baselines.
Test-Time Training-Free Domain Adaptation

Deploying deep learning models to new environments is very challenging. Domain adaptation (DA) is a promising paradigm to solve the problem by collecting and adapting to unlabeled data in new environments. Though research efforts have led to steady performance improvement over the past decade, DA algorithms are still hard to deploy, as training on unlabeled new data makes tuning difficult and not feasible for inference-only devices. To make DA practical, in this paper we study a new problem named Test-time Training-Free Domain Adaptation (TTDA), where trained models must adapt to a single input (mimicking the test-time scenario) without training. By exploiting spatial activation that was previously overlooked and simply averaged out, we propose a simple method based on Feature Statistics Transformation (FST) on-the-fly for each test example. The proposed algorithm is tested in the TTDA setting on two standard DA benchmarks. Surprisingly, it surpasses or performs on par with state-of-the-art DA methods, even though they require additional training. We envision that this training-free paradigm has the potential to bring DA to embedded devices and would be of interest to audience of community.
Boosting Transferability of Adversarial Example via an Enhanced Euler’s Method

Adversarial examples are intentionally designed images to force convolution neural networks to give error classification outputs. Existing attacks have constructed transferable adversarial examples from the base attack algorithm, data augmentation, ensemble model, etc. Nevertheless, under the black-box case especially facing defense models, the transferability of adversarial examples still needs to be improved. In this paper, we try to develop a better base attack to boost the transferability of adversarial examples. Through analyzing the baseline gradient-based attacks, we found their iterative procedures of updating gradients are similar to numerical Euler’s methods. From the perspective of numerical analysis, we employ an enhanced Euler’s method, with less approximate errors and thus more accurate, to search a better approximate optimal solution to construct a more transferable gradient-based attack. To this end, we apply two-step gradient calculations of the enhanced Euler’s method to correct gradient descent directions. As a base attack, our attacks can be easily integrated with data augmentations and ensemble model augmentations. Experimental results show the proposed augmented attack significantly improves the transferability of adversarial examples and achieves an average attack success rate at least 3% higher than state-of-the-arts under black-box settings with defense mechanisms.
Learning Graph Laplacian from Intrinsic Patterns via Gaussian Process

In this paper, we present a novel scheme to learn a graph topology named Laplacian constrained Gaussian process (LCGP). Previous graph learning methods directly use raw observed signals, which often degrades estimation accuracy due to the noise effects and the sample imbalances. LCGP tackles this problem by introducing a small number of intrinsic patterns, that is, LCGP assumes representative signals on a graph. These signals are derived as Bayesian latent variables, and we estimate the graph topology with objective after marginalizing them. Furthermore, the number of the intrinsic patterns is automatically determined by a Bayesian inference procedure. In the experiment, we compared LCGP with baseline and state-of-the-art methods in graph learning and confirmed the effectiveness of LCGP.
Joint Discriminator and Transfer Based Fast Domain Adaptation For End-To-End Speech Recognition

Adapting End-to-End (E2E) models to unseen domains is still a big challenge since training E2E models requires lots of paired audio and text training data. We propose a novel domain adaptation framework for the E2E model, which only uses the text of the target domain. Moreover, the proposed methods can keep the performance on the source domain intact while greatly improving the performance on the target domain. The proposed framework consists of two parts: the discriminator and the transfer which were optimized separately. Finally, optimized discriminator and transfer were combined and evaluated on two domain adaption tasks. In the experiments of adapting the English Librispeech to Gigaspeech, we obtained an average relative 11.6% and 11.8% on word error rate (WER) reduction for the target domain dev and test sets, respectively, while almost without WER degradation on the source domain. For the inhouse Chinese corpus aviation and TV, the character error rate (CER) of the source domain increased within 5%, while the CER on the target domain achieved around relative 85% and 42% improvement, respectively. In addition, our approach is also more effective in the mixed domain scenarios in the evaluation.
Dual Collaborative Visual-Semantic Mapping for Multi-Label Zero-Shot Image Recognition

Multi-label zero-shot learning (ML-ZSL), with the difficulty of both multi-label learning and zero-shot learning, aims to recognize various unseen objects that are not observed during training. Previous methods mainly use a single directional visual-semantic mapping to associate the visual and semantic embedding space, which is not sufficient to adequately realize knowledge transfer from seen to unseen classes. In this paper, we propose a novel dual collaborative visual-semantic mapping framework, constructing abundant connection relationships by exploring two aspects of mapping streams, i.e., the visual-to-semantic (V2S) mapping and the semantic-to-visual (S2V) mapping. Through the collaborative learning of these two effective mappings, our method achieves state-of-the-art performance on the MS-COCO and PASCAL-VOC, two benchmarks for ML-ZSL.
Monocular 3D Human Pose Estimation Based on Global Temporal-Attentive and Joints-Attention In Video

Learning to capture human motion is essential to 3D human pose and shape estimation from monocular video, which is widely used in many 3D applications. However, the existing methods mainly rely on recurrent or convolutional operation to model such temporal information, which limits the ability to capture non-local contextual relations of human motion and ignores human joint hierarchies. To address this problem, we propose a Global Temporal-Attentive and Joints-Attention network (GTAJA-Net). This method introduces a Global Attention Feature Integration (GAFI) module and a Motion Tree Fusion Decoder (MTFD) module on the basis of a temporally consistent mesh recovery system (TCMR). A GAFI consisting of a collection of temporal features obtains final temporal features carrying spatial information that enhances temporal correlation and refine the features of the current frame. Meanwhile, MTFD aims at modeling the joint level attention. MTFD considers pose estimation as a top-down hierarchical process similar to SMPL kinematic tree. Though conceptually simple, our GTAJA-Net outperforms the state-of-the-art methods on the 3DPW, MPI-INF-3DHP, and Human3.6M benchmark datasets. Our code is available at https://github.com/xiangcece/GTAJA-Net.
SADE: A Self-Adaptive Expert for Multi-Dataset Question Answering

Multi-dataset question answering (QA) aims to combine multiple QA datasets to build models that not only perform well on training distributions, but also transfer and generalize well to new distributions. Some prior work considered building a collection of dataset-specific experts upon a shared Transformer, so as to simultaneously encode both regularities across datasets and specificities of each dataset. This approach, however, has its limitations when generalized to an unseen new distribution, and the number of extra parameters will increase with the number of training datasets. In this paper, we devise Self-ADaptive Expert (SADE), the key idea of which is to train a single expert that can be automatically adapted to each individual instance according to its gradients. This gradient-based, instance-level modulation scheme makes our approach easily adaptable to any instance from unseen new distributions, and keeps the number of extra parameters as a constant. We further design a contrastive learning mechanism to enhance the discriminability of modulation signals across different datasets. Experimental results on twelve QA datasets demonstrate that SADE consistently outperforms previous state-of-the-art in all the three settings including in-domain learning, few-shot transfer learning, and zero-shot generalization.
Multi-Lingual Pronunciation Assessment with Unified Phoneme Set and Language-Specific Embeddings

Automatic pronunciation assessment is commonly trained and applied for a specific language, which is not practical in multi-lingual or low-resource scenarios. In this paper, we propose a unified method to take advantage of multi-lingual data for multi-lingual pronunciation assessment. To this end, we first construct a concise unified phoneme set for multi-lingual phoneme recognition based on a pre-trained acoustic model. In this way we can not only share language-independent knowledge but also try to discriminate language-specific information for pronunciation assessment. Second, we employ language-specific embeddings for different languages, which act like language-specific assessment criteria to adaptively adjust the feature weights based on an attention mechanism. The whole network is optimized in a unified framework. Experimental results based on multi-lingual datasets demonstrate its superiority to different baselines in Pearson correlation coefficient (PCC). We also illustrate the generalizability of the proposed method for both seen and unseen data.
One-Shot Action Detection via Attention Zooming In

Hinted by a modest support set, few-shot action detection (FSAD) aims at localizing the action instances of unseen classes within an untrimmed query video. Existing FSAD techniques mostly rely on generating a set of class-agnostic action proposals from the query video and then finding the most plausible ones by assessing their correlation to the support set. Such two-stage approaches are feasible but not efficient, largely due to neglecting the support information in generating the proposals. This work focuses on the one-shot image scenario and introduces the attention zooming in strategy to effectively and progressively carry out support-query cross-attention while generating proposals. The resulting one-stage model yields high-quality action proposals for boosting one-shot action detection (OSAD) performance. Our extensive experiments on the ActivityNet-1.3 and THUMOS-14 datasets demonstrate that the proposed framework can achieve state-of-the-art performance in tack-ling challenging image-based OSAD tasks.
Analysis of Computer Vision for Graphics and Animation

The field of computer vision is rapidly evolving. Pictures and videos can be obtained and processed to model, duplicate, and occasionally introduce additional visuals to complete valuable tasks. This Paper outlines a method for gathering, refining, and comprehending video and images. In the future, computer vision will be used in a wide range of products and services, including cameras, movies, smartphones, drones, and much more. Video sequences, multiple camera views, multi-dimensional data from medical scanning, and 3D scanner devices can all be used to represent various types of image data. the purpose of this paper is to analyze the role of computer vision for graphics and animations in order to analyze the role perception of virtual machine and computer animation and vision has been studied in detail. Multidimensional data, video sequences, and multiple camera images from a medical scanner are all examples of image data. The goal of computer vision as a technological discipline is to create computer viewing devices using its models and theories.
Adaptive Multi-Corpora Language Model Training for Speech Recognition

Neural network language model (NNLM) plays an essential role in automatic speech recognition (ASR) systems, especially in adaptation tasks when text-only data is available. In practice, an NNLM is typically trained on a combination of data sampled from multiple corpora. Thus, the data sampling strategy is important to the adaptation performance. Most existing works focus on designing static sampling strategies. However, each corpus may show varying impacts at different NNLM training stages. In this paper, we introduce a novel adaptive multi-corpora training algorithm that dynamically learns and adjusts the sampling probability of each corpus along the training process. The algorithm is robust to corpora sizes and domain relevance. Compared with static sampling strategy baselines, the proposed approach yields remarkable improvement by achieving up to relative 7% and 9% word error rate (WER) reductions on in-domain and out-of-domain adaptation tasks, respectively.
ERBNet: An Effective Representation Based Network for Unbiased Scene Graph Generation

The scene graph generation (SGG) task has attracted increasing attention in recent years. The goal of SGG is to predict relations between pairs of objects within an image. Due to the long-tailed distribution of the dataset annotations, the performance of SGG is still far from satisfactory. To address the long-tailed problem, existing methods try various ways to conduct unbiased learning. However, we argue that the essence of the long-tailed problem in SGG is that the classifier is seriously affected by the long-tailed data. To handle this issue, we propose a novel network named ERBNet, which contains a relation feature fusion (RFF) encoder to construct effective representations of relations between objects, and a nearest class mean (NCM) classifier to conduct relation prediction based on relation feature similarities. Extensive experimental results show that the proposed ERBNet outperforms several state-of-the-art methods on the challenging Visual Genome dataset.
Commdre: Document-Level Relation Extraction with Self-Supervised Commonsense Learning

Document-level relation extraction (DocRE) is a more challenging task for which multi-label and multi-entity problems need to be resolved effectively than its sentence-level counterpart. It aims at extracting relationships between two entities at once while taking into account significant cross-sentence features and long-distance semantic representation. In this paper, we propose a self-supervised commonsense-enhanced DocRE model, called CommDRE, without external knowledge. First, we introduce self-supervised learning to represent the commonsense knowledge of each entity in an entity pair. Second, we convert the cross-sentence entity pairs into anonymous entity pairs with a coreference commonsense alternative. Finally, we perform semantic relation representation learning on the anonymous entity pairs and automatically convert them into target entity pairs. Experimental results show that it performs significantly better than strong baselines by 2.76% F1, and commonsense knowledge has an important contribution to the DocRE through the ablation study.
TRUSTERA: A Live Conversation Redaction System

We introduce Trustera <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> , the first functional system that redacts personally identifiable information (PII) in real-time spoken conversations to remove agents’ need to hear sensitive information while preserving the naturalness of live customer-agent conversations. As opposed to post-call redaction, audio masking starts as soon as the customer begins speaking to a PII entity. This significantly reduces the risk of PII being intercepted or stored in insecure data storage. Trustera’s architecture consists of a pipeline of automatic speech recognition, natural language understanding, and a live audio redactor module. The system’s goal is three-fold: redact entities that are PII, mask the audio that goes to the agent, and at the same time capture the entity, so that the captured PII can be used for a payment transaction or caller identification. Trustera is currently being used by thousands of agents to secure customers’ sensitive information.
CNN Filter for RPR-Based SR in VVC with Wavelet Decomposition

In this paper, we propose a convolutional neural network (CNN) filter for reference picture resampling (RPR)-based super-resolution (SR) with wavelet decomposition. The proposed CNN filter takes the low resolution (LR) reconstructed frame (Rec <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LR</inf> ), LR prediction frame (Pre <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LR</inf> ) and high resolution (HR) RPR upsampled frame (RPR <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">output</inf> ) as the input for RPR-based SR. Thus, the proposed CNN filter not only learns a mapping function between LR and HR images, but also effectively removes blocking artifacts in the reconstructed frame. We adopt wavelet decomposition to make RPR <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">output</inf> the same size as Rec <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LR</inf> and Pre <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LR</inf> as well as obtain the relationship between high frequency (HF) and low frequency (LF) components. To maximize feature reuse under the limited parameters, we design a residual spatial and channel attention block (RSCB) that combines residual blocks with spatial attention and channel attention to learn the weighted local information and global information in different receptive fields. Experimental results show that the proposed CNN filter achieves -8.98% and -4.05% BD-rate reductions on Y channel in AI and RA configurations over VTM-11.0_NNVC-2.0 anchor, respectively.
Rate-Distortion Optimization with Alternative References for UGC Video Compression

User generated content (UGC) refers to videos that are uploaded by users and shared over the Internet. UGC may have low quality due to noise and previous compression. When re-encoding UGC for streaming or downloading, a traditional video coding pipeline will perform rate-distortion (RD) optimization to choose coding parameters. However, in the UGC video coding case, since the input is not pristine, quality “saturation” (or even degradation) can be observed, i.e., increased bitrate only leads to improved representation of coding artifacts and noise present in the UGC input. In this paper, we study the saturation problem in UGC compression, where the goal is to identify and avoid during encoding, the coding parameters and rates that lead to quality saturation. We proposed a geometric criterion for saturation detection that works with rate-distortion optimization, and only requires a few frames from the UGC video. In addition, we show how to combine the proposed saturation detection method with existing video coding systems that implement rate-distortion optimization for efficient compression of UGC videos.
Self-Distillation Hashing for Efficient Hamming Space Retrieval

Deep hashing-based approaches have become the optimal solutions for large-scale image retrieval task due to their high computational efficiency and low storage burden. Some methods leverage a large teacher network to improve the retrieval performance of the small student network through knowledge distillation, which incurs high computational and time costs. In this paper, we propose Self-Distillation Hashing (SeDH), which improves the image retrieval performance without introducing a complex teacher model and significantly reduces the overall computation costs. Specifically, we generate the soft targets via ensembling the logits of other similar images among the mini-batch. The ensembled soft targets can model the relations between different image samples, which can act as additional supervision for classification. Besides, to learn more compact features and accurate inter-sample similarities, we propose a similarity-preserving loss on the learned hashing features, which aligns the softened similarity distribution with the pairwise soft similarity. Extensive experiments demonstrate that our approach can yield state-of-the-art performance on deep supervised hashing retrieval.