Reviewerly/exHarmony huggingface dataset:
Learning Expressive And Generalizable Motion Features For Face Forgery Detection

Previous face forgery detection methods mainly focus on appearance features, which may be easily attacked by sophisticated manipulation. Considering the majority of current face manipulation methods generate fake faces based on a single frame, which do not take frame consistency and coordination into consideration, artifacts on frame sequences are more effective for face forgery detection. However, current sequence-based face forgery detection methods use general video classification networks directly, which discard the special and discriminative motion information for face manipulation detection. To this end, we propose an effective sequence-based forgery detection framework based on an existing video classification method. To make the motion features more expressive for manipulation detection, we propose an alternative motion consistency block instead of the original motion features module. To make the learned features more generalizable, we propose an auxiliary anomaly detection block. With these two specially designed improvements, we make a general video classification network achieve promising results on three popular face forgery datasets.
A security-friendly privacy-preserving solution for federated learning

Federated learning is a privacy-aware collaborative machine learning method where the clients collaborate on constructing a global model by performing local model training using their training data and sending the local model updates to the server. Although it enhances privacy by letting the clients collaborate without sharing their training data, it is still prone to sophisticated privacy attacks because of possible information leakage from the local model updates sent to the server. To prevent such attacks, generally secure aggregation protocols are proposed so that the server will not be able to access the individual local model updates but the aggregated result. However, such secure aggregation approaches may not allow the execution of security mechanisms against some security attacks to model training, such as poisoning and backdoor attacks, because the server cannot access the individual local model updates and; therefore, cannot analyze them to detect anomalies resulting from these attacks. Thus, solutions that satisfy privacy and security at the same time or new privacy-preserving solutions that allow the server to execute some analysis on the local model updates without violating privacy are needed for federated learning. In this paper, we introduce a novel security-friendly privacy solution for federated learning based on multi-hop communication to hide clients' identities. Our solution ensures that the forwardee clients in the path between the source client and the server cannot execute malicious activities by altering model updates and contributing to the global model construction with more than one local model update in one FL round. We then propose two different approaches to make the solution also robust against possible malicious packet drop behaviors by the forwardee clients.
Semi-Supervised Domain Generalization with Graph-Based Classifier

Semi-supervised domain generalization (SSDG) has recently emerged as a potential research topic. Compared to domain generalization, SSDG represents a realistic and challenging goal, which only requires a few labels from source domains. To tackle this problem, this work presents a novel pseudo-labeling method that facilitates incremental learning on a large amount of unlabeled data. With edge weighting optimization, the proposed method utilizes the graph Laplacian regularizer (GLR) in a multi-class setting that relies on the generated similarity graph. The proposed overall SSDG scheme mitigates the overfitting problem by an adaptive threshold module based on a two-stage GLR denoiser. Our experiments on PACS and OfficeHome verify that the proposed method effectively improves the quality of pseudo-labeling and domain generalization, achieving top performance in terms of accuracy.
Variable Rate Allocation for Vector-Quantized Autoencoders

Vector-quantized autoencoders have recently gained interest in image compression, generation and self-supervised learning. However, as a neural compression method, they lack the possibility to allocate a variable number of bits to each image location, e.g. according to the semantic content or local saliency. In this paper, we address this limitation in a simple yet effective way. We adopt a product quantizer (PQ) that produces a set of discrete codes for each image patch rather than a single index. This PQ-autoencoder is trained end-to-end with a structured dropout that selectively masks a variable number of codes at each location. These mechanisms force the decoder to reconstruct the original image based on partial information and allow us to control the local rate. The resulting model can compress images on a wide range of operating points of the rate-distortion curve and can be paired with any external method for saliency estimation to control the compression rate at a local level. We demonstrate the effectiveness of our approach on the popular Kodak and ImageNet datasets by measuring both distortion and perceptual quality metrics.
Improving Transformer-Based End-to-End Speaker Diarization by Assigning Auxiliary Losses to Attention Heads

Transformer-based end-to-end neural speaker diarization (EEND) models utilize the multi-head self-attention (SA) mechanism to enable accurate speaker label prediction in overlapped speech regions. In this study, to enhance the training effectiveness of SA-EEND models, we propose the use of auxiliary losses for the SA heads of the transformer layers. Specifically, we assume that the attention weight matrices of an SA layer are redundant if their patterns are similar to those of the identity matrix. We then explicitly constrain such matrices to exhibit specific speaker activity patterns relevant to voice activity detection or overlapped speech detection tasks. Consequently, we expect the proposed auxiliary losses to guide the transformer layers to exhibit more diverse patterns in the attention weights, thereby reducing the assumed redundancies in the SA heads. The effectiveness of the proposed method is demonstrated using the simulated and CALLHOME datasets for two-speaker diarization tasks, reducing the diarization error rate of the conventional SA-EEND model by 32.58% and 17.11%, respectively.
Neural Transducer Training: Reduced Memory Consumption with Sample-Wise Computation

The neural transducer is an end-to-end model for automatic speech recognition (ASR). While the model is well-suited for streaming ASR, the training process remains challenging. During training, the memory requirements may quickly exceed the capacity of state-of-the-art GPUs, limiting batch size and sequence lengths.In this work, we analyze the time and space complexity of a typical transducer training setup. We propose a memory-efficient training method that computes the transducer loss and gradients sample by sample. We present optimizations to increase the efficiency and parallelism of the sample-wise method. In a set of thorough benchmarks, we show that our sample-wise method significantly reduces memory usage, and performs at competitive speed when compared to the default batched computation. As a highlight, we manage to compute the transducer loss and gradients for a batch size of 1024, and audio length of 40 seconds, using only 6 GB of memory.
Targeted Adversarial Attacks Against Neural Machine Translation

Neural Machine Translation (NMT) systems are used in various applications. However, it has been shown that they are vulnerable to very small perturbations of their inputs, known as adversarial attacks. In this paper, we propose a new targeted adversarial attack against NMT models. In particular, our goal is to insert a predefined target keyword into the translation of the adversarial sentence while maintaining similarity between the original sentence and the perturbed one in the source domain. To this aim, we propose an optimization problem, including an adversarial loss term and a similarity term. We use gradient projection in the embedding space to craft an adversarial sentence. Experimental results show that our attack outperforms Seq2Sick, the other targeted adversarial attack against NMT models, in terms of success rate and decrease in translation quality. Our attack succeeds in inserting a keyword into the translation for more than 75% of sentences while similarity with the original sentence stays preserved <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Optimizing Distributed Multi-Sensor Multi-Target Tracking Algorithm Based On Labeled Multi-Bernoulli Filter

In this paper, we propose an improved distributed fusion algorithm under the Labeled multi-Bernoulli (LMB) filter framework. Firstly, the LMB parameter set is augmented by a new group variable, which is able to record the matching information of the neighbour sensors. Then the matching LMB components of the survival targets between the sensors can be fused directly by checking the group variable, greatly reducing the time cost of the matching calculation and the interference from the newborn targets. While for the newborn targets, the Murty algorithm is employed and only performed once to find the best matching relation between the sensors. Finally, experimental results show that the proposed algorithm offers a better tracking performance than the state-of-the-art R-GCI-LMB algorithm with lower computational complexity and higher tracking accuracy.
G2PL: Lexicon Enhanced Chinese Polyphone Disambiguation Using Bert Adapter with a New Dataset

Polyphone disambiguation is the core of grapheme-to-phoneme(G2P) module for the Chinese speech synthesis system. However, there is a lack of datasets and only one public for polyphone disambiguation. Moreover, due to the double long-tail distribution of polyphones, the ratio of pronunciation data for most polyphones is extremely unbalanced after sampling. To solve these problems, we propose a new dataset with 57,000 sentences from various domains by a new strategy for sampling. In addition, we propose the G2PL, which integrates word features into the bottom of BERT to assist in predicting the correct pronunciation of polyphone. In the experiment, we train the G2PL model to outperform other methods on our and public datasets. Our dataset, codes and user-friendly package are freely available.
NSV-TTS: Non-Speech Vocalization Modeling And Transfer In Emotional Text-To-Speech

This paper addresses the problem of non-speech vocalization (NSV) modeling and transfer in emotional TTS. We propose an emotion TTS system (NSV-TTS) to model NSV and emotional speech. The model utilizes self-supervised learning to extract unsupervised linguistic units (ULUs) for NSV labeling and zero-shot NSV transfer. Furthermore, we propose token mixing and random masking to boost the performance. We evaluate the proposed method on various NSV types and emotion classes. The experimental results reveal that the proposed method performs well in the zero-shot NSV transfer task. Lastly, we conduct ablation studies to investigate the proposed method further.
Contrastive Learning with Dialogue Attributes for Neural Dialogue Generation

Designing an effective learning method remains a challenge in neural dialogue generation systems as it requires the training objective to well approximate the intrinsic human-preferred dialogue properties. Conventional training approaches such as maximum likelihood estimation focus on modeling general syntactic patterns and may fail to capture intricate conversational characteristics. Contrastive dialogue learning offers an effective training schema by explicitly training a neural dialogue model on multiple positive and negative conversational pairs. However, constructing contrastive learning pairs is non-trivial, and multiple dialogue attributes have been found to be crucial for governing the human judgments of conversations. This paper proposes to guide the response generation with attribute-aware contrastive learning to improve the overall quality of the generated responses, where contrastive learning samples are generated according to various important dialogue attributes each specializing in a different principle of conversation. Extensive experiments show that our proposed techniques are crucial to achieving superior model performance.
Semantic-Aware Gated Fusion Network For Interactive Colorization

Deep neural networks boost many successful colorization methods, including automatic, interactive, and exemplar-based methods. Among them, interactive methods with global and/or local inputs are probably the most flexible to accurately add colors to a gray image. However, due to the sparseness of input-semantic correspondences, existing methods encounter difficulties in distributing inputs into correct regions. Moreover, they simply add or concatenate the features of different inputs to the network before color reconstruction, which cannot balance the influences of different inputs. To this end, we propose a novel interactive colorization network, which explicitly builds input-semantic correspondences with an attention mechanism and proposes a gated feature fusion module to balance the influences of global and local inputs. We further apply a differentiable histogram loss to impose a smooth impact of the global inputs. Extensive experiments demonstrate that our method can flexibly control the results and outperforms other state-of-the-art interactive methods.
Self-Similarity is all You Need for Fast and Light-Weight Generic Event Boundary Detection

The self-similarity matrix (SSM) is becoming more prevalent in temporal representation understanding; it has been utilized for various video understanding tasks, such as classifying human actions, counting repetitions, and identifying generic event boundaries. Recently proposed methods for Generic Event Boundary Detection (GEBD) [1] based on SSM have obtained impressive results on the Kinetics-GEBD dataset. However, they demand a large model size and an immense number of computations to achieve good performance, making them challenging to realize on edge devices. We introduce a projected SSM with cosine distance that produces an efficient representation of SSM that can be interpreted using lighter transformer decoders. This paper presents a lightweight novel architecture with just 3M trainable parameters that utilize projected SSM to solve GEBD. In addition to the low computation regime of the model, the experiments demonstrate that the architecture is invariant to the feature extractor model while inferencing. The proposed method achieves a boost of 13.92% on the Kinetics-GEBD validation dataset with 3.5X fewer model parameters and 19.5X fewer multi-add operations compared to the baseline [1]. We report competitive F1 results at unprecedented efficiency with 22X fewer model parameters than state-of-the-art methods and achieve the lowest inference time on GPU and mobile device.
Multiple Contrastive Learning for Multimodal Sentiment Analysis

Multimodal sentiment analysis has received extensive attention with the explosion of multimodal data. For multimodal data, representations should have disparate distributions in the feature space under different labels. The paired multi-modal image-text posts should be closer than unpaired. We propose Multimodal fine-grained interaction with the Multiple Contrastive Learning (M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> CL) model for image-text multi-modal sentiment detection. Specifically, we first obtain the reinforced global representation of one modality with the assistance of fine-grained information from another via the Multimodal Interaction Component. Then, we introduce the Multiple Contrastive Learning Component, including Supervised Contrastive Learning (SCL) and Dual Multimodal Contrastive Learning (DMCL). SCL accomplishes pushing the posts with the same sentiment closer and pulling the instances of different sentiments apart within each modality. DMCL pushes the paired image-text features together and pulls the unpaired apart with multiple stages. Extensive experiments conducted on three datasets confirm the effectiveness of our approach.
Rethinking Learning-Based Method for Lossless Genome Compression

Lossless genome compression plays a vital role in genomic analysis procedure. The main challenges are from long range of the genome sequence and high frequency of genome variants. However, existing learning-based methods almost all rely on local DNA fragments in small windows, which makes them unable to capture deep regularities of genome sequence and may lead to unsatisfactory performance because of the large variability in different individuals. In this paper, we redesign the deep learning model and propose a simple yet effective position-driven transformer for genome data compression. Our approach, called CompressBERT, is based on two core designs. First, we introduce global position of the complete genome sequence into our deep model, which can make the genome sequence distinguishable in base level. Second, we pre-train our deep model by identifying SNP genome variants, which can further facilitate genome compression task. Furthermore, the proposed CompressBERT is validated on three datasets from different species. Experimental results show that our approach outperforms state-of-the-art methods.
Masked Token Similarity Transfer for Compressing Transformer-Based ASR Models

Recent self-supervised automatic speech recognition (ASR) models based on transformers are showing best performance, but their footprint is too large to be trained on low-resource environments or deployed to edge devices. Knowledge distillation (KD) can be employed to reduce the model size. However, setting embedding dimension of teacher and student network to different values makes it difficult to transfer token embeddings for better performance. To mitigate this issue, we present a novel KD method in which student mimics the prediction vector of teacher under our proposed masked token similarity transfer (MTST) loss where the temporal relation between a token and the other unmasked ones is encoded into a dimension-agnostic token similarity vector. Under our transfer learning setting with a fine-tuned teacher, our proposed methods reduce the model size of student to 28.3% of teacher’s while word error rate on test-clean subset in LibriSpeech corpus is 4.93%, which surpasses prior works. Our source code will be made available.
Saliency-Driven Hierarchical Learned Image Coding for Machines

We propose to employ a saliency-driven hierarchical neural image compression network for a machine-to-machine communication scenario following the compress-then-analyze paradigm. By that, different areas of the image are coded at different qualities depending on whether salient objects are located in the corresponding area. Areas without saliency are transmitted in latent spaces of lower spatial resolution in order to reduce the bitrate. The saliency information is explicitly derived from the detections of an object detection network. Furthermore, we propose to add saliency information to the training process in order to further specialize the different latent spaces. All in all, our hierarchical model with all proposed optimizations achieves 77.1 % bitrate savings over the latest video coding standard VVC on the Cityscapes dataset and with Mask R-CNN as analysis network at the decoder side. Thereby, it also outperforms traditional, non-hierarchical compression networks.
Deep Fusion of Multi-Object Densities Using Transformer

The fusion of multiple probability densities has important applications in many fields, including, for example, multi-sensor signal processing, robotics, and smart environments. In this paper, we demonstrate that deep learning based methods can be used to fuse multi-object densities. Given a scenario with several sensors with possibly different field-of-views, tracking is performed locally in each sensor by a tracker, which produces random finite set multi-object densities. To fuse outputs from different trackers, we adapt a recently proposed transformer-based multi-object tracker, where the fusion result is a global multi-object density, describing the set of all alive objects at the current time. We compare the performance of the transformer-based fusion method with a well-performing model-based Bayesian fusion method in several simulated scenarios with different parameter settings using synthetic data. The simulation results show that the transformer-based fusion method outperforms the model-based Bayesian method in our experimental scenarios. The code is available at https://github.com/Lechili/DeepFusion.
Boosting Fine-Grained Sketch-Based Image Retrieval with Self-Supervised Learning

Fine-grained sketch-based image retrieval (FG-SBIR) aims at aligning images and sketches at the instance level. It is a challenging task as there are significant differences between sketch and image. Existing methods usually produce less desired performance due to the lack of large-scale fine-grained image-sketch datasets and the strong dependence on the classification models pretrained on ImageNet. In this paper, we propose a better self-supervised pre-trained FG-SBIR model which does not depend on large-scale annotated datasets. Only images and their corresponding edge maps are used at the pre-training stage. Mixed modal transformation is designed to generate different mixed-up views. The FG-SBIR model is pre-trained by minimizing the distance between the views of the same instance and then fine-tuned by a simple triplet loss. With a plain downstream network, it achieves generally better performance than state-of-the-art models on three widely used FG-SBIR datasets.
Deep Triple-Supervision Learning Unannotated Surgical Endoscopic Video Data for Monocular Dense Depth Estimation

Surface reconstruction is an essential way to expand surgical field of view during endoscopic surgery, but it certainly requires dense depth estimation of endoscopic video sequences. Unfortunately, such a dense depth recovery suffers from illumination variation, weak texture, and occlusion. To address these problems, this work proposes a new triple-supervision self-learning strategy that uses unannotated endoscopic video data to predict monocular endoscopic dense depth information. This strategy first employs an effective conventional method to estimate camera poses and sparse depth maps to establishing a sparse data self-supervision. Furthermore, our strategy still combines two consistency measures to supervise dense depth and photometric information. We evaluated our method on collected colonoscopic videos, with the experimental results showing that our triple-supervision learning framework works more effective and accurate than some current self-supervised and unsupervised learning methods.