Reviewerly/exHarmony huggingface dataset:
Vehicle View Synthesis by Generative Adversarial Network

In recent years, novel view synthesis methods has been proposed and combined with many models in different computer vision tasks. Previous works solve the problem by using additional 3D information. In this paper, a novel view synthesis method is proposed based on Generative Adversarial Networks (GANs), named PTGAN. PTGAN generates new views by specifying keypoints of vehicles from other views. This makes the pose transformation of vehicle practical when 3D information is unavailable. The proposed PTGAN first extracts identity-related and pose-unrelated feature representations from input images and then concatenates the representation with the pose information to generate the fake image with the assigned pose to deal with the pose variation problem. Experimental results demonstrate that the proposed method achieves very competitive results to the existing view synthesis methods.
Cross-Modal Mutual Learning for Cued Speech Recognition

Automatic Cued Speech Recognition (ACSR) provides an intelligent human-machine interface for visual communications, where the Cued Speech (CS) system utilizes lip movements and hand gestures to code spoken language for hearing-impaired people. Previous ACSR approaches often utilize direct feature concatenation as the main fusion paradigm. However, the asynchronous modalities (i.e., lip, hand shape and hand position) in CS may cause interference for feature concatenation. To address this challenge, we propose a transformer based cross-modal mutual learning framework to prompt multi-modal interaction. Compared with the vanilla self-attention, our model forces modality-specific information of different modalities to pass through a modality-invariant codebook, concatenating linguistic representations with tokens of each modality. Then the shared linguistic knowledge is used to re-synchronize multi-modal sequences. Moreover, we establish a novel large-scale multi-speaker CS dataset for Mandarin Chinese. To our knowledge, this is the first work on ACSR for Mandarin Chinese. Extensive experiments are conducted for different languages (i.e., Chinese, French, and British English). Results demonstrate that our model exhibits superior recognition performance to the state-of-the-art by a large margin.
Do Prosody Transfer Models Transfer Prosodyƒ

Some recent models for Text-to-Speech synthesis aim to transfer the prosody of a reference utterance to the generated target synthetic speech. This is done by using a learned embedding of the reference utterance, which is used to condition speech generation. During training, the reference utterance is identical to the target utterance. Yet, during synthesis, these models are often used to transfer prosody from a reference that differs from the text or speaker being synthesized.To address this inconsistency, we propose to use a different, but prosodically-related, utterance during training too. We believe this should encourage the model to learn to transfer only those characteristics that the reference and target have in common. If prosody transfer methods do indeed transfer prosody they should be able to be trained in the way we propose. However, results show that a model trained under these conditions performs significantly worse than one trained using the target utterance as a reference. To explain this, we hypothesize that prosody transfer models do not learn a transferable representation of prosody, but rather a utterance-level representation which is highly dependent on both the reference speaker and reference text.
Permutation Invariant Training for Paraphrase Identification

Identifying sentences sharing similar meanings is crucial to speech and text understandings. Although currently popular cross-encoder solutions with pre-trained language models as backbone have achieved remarkable performance, they suffer from the lack of the permutation invariance or symmetry that is one of the most important inductive biases to such task. To alleviate this issue, in this research we propose a permutation invariant training framework, in which a symmetry regularization is introduced during training that forces the model to produce the same predictions for input sentence pairs in both forward and backward directions. Empirical studies exhibit improved performance over competitive baselines.
Audio-Driven Talking Head Video Generation with Diffusion Model

Synthesizing high-fidelity talking head videos by fitting input audio sequences is a highly anticipated technique in many applications, such as digital humans, virtual video conferences, and human-computer interaction. Popular GAN-based methods aim to align speech audio with lip motions and head poses. However, existing methods are prone to training instability and even mode collapse, resulting in low-quality video generation. In this paper, we propose a novel audio-driven diffusion method for generating high-resolution realistic videos of talking heads with the help of the denoising diffusion model. Specifically, the face attribute disentanglement module is proposed to disentangle eye blinking and lip motion features, where the lip motion features are synchronized with audio features via the contrastive learning strategy, and the disentangled motion features are aligned well with the talking head. Furthermore, the denoising diffusion model takes the source image and the warped motion features as input to generate the high-resolution realistic talking head with diverse head poses. Extensive evaluations using multiple metrics demonstrate that our method outperforms the current techniques both qualitatively and quantitatively.
Deformable Cross Attention for Learning Optical Flow

Optical flow is the process of estimating motion in scenes. Each object in the scene has a homogeneous motion, i.e., moves in the same direction with the same velocity. Therefore, connecting the parts of an image globally provides an essential cue for learning accurate motion. Convolution-based methods estimate the motion features from the local regions, which miss this important cue. Recently, some methods used Transformer to model global dependencies to improve optical flow. However, Transformer suffers from excessive attention computations and still brings irrelevant parts into the region of interest. Therefore, we propose a deformable cross-attention for optical flow estimation, which provides two important advantages: connecting the parts of the image globally while deforming the attention to the objects’ shapes in the image and reducing the memory consumption. Our proposed method achieved competitive performance on Sintel and KITTI 2015 datasets in terms of accuracy and efficiency.
Internal Language Model Estimation Based Adaptive Language Model Fusion for Domain Adaptation

ASR model deployment environment is ever-changing, and the incoming speech can be switched across different domains during a session. This brings a challenge for effective domain adaptation when only target domain text data is available, and our objective is to obtain obviously improved performance on the target domain while the performance on the general domain is less undermined. In this paper, we propose an adaptive LM fusion approach called internal language model estimation based adaptive domain adaptation (ILME-ADA). To realize such an ILME-ADA, an interpolated log-likelihood score is calculated based on the maximum of the scores from the internal LM and the external LM (ELM) respectively. We demonstrate the efficacy of the proposed ILME-ADA method with both RNN-T and LAS modeling frameworks employing neural network and n-gram LMs as ELMs respectively on two domain specific (target) test sets. The proposed method can achieve significantly better performance on the target test sets while it gets minimal performance degradation on the general test set, compared with both shallow and ILME-based LM fusion methods.
M<sup>2</sup>-CTTS: End-to-End Multi-Scale Multi-Modal Conversational Text-to-Speech Synthesis

Conversational text-to-speech (TTS) aims to synthesize speech with proper prosody of reply based on the historical conversation. However, it is still a challenge to comprehensively model the conversation, and a majority of conversational TTS systems only focus on extracting global information and omit local prosody features, which contain important fine-grained information like keywords and emphasis. Moreover, it is insufficient to only consider the textual features, and acoustic features also contain various prosody information. Hence, we propose M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> -CTTS, an end-to-end multi-scale multi-modal conversational text-to-speech system, aiming to comprehensively utilize historical conversation and enhance prosodic expression. More specifically, we design a textual context module and an acoustic context module with both coarse-grained and fine-grained modeling. Experimental results demonstrate that our model mixed with fine-grained context information and additionally considering acoustic features achieves better prosody performance and naturalness in CMOS tests.
Efficient Compressed Video Action Recognition Via Late Fusion with a Single Network

Compressed video action recognition is an action recognition approach that can achieve efficient inference by directly classifying video data obtained from multiple features stored in compressed videos. Most conventional methods use multiple networks to process compressed video features. They explore the use of lightweight networks without affecting the classification performance to reduce the computational complexity of compressed video action recognition. This study explores another approach to reduce the computational complexity, by using a single network instead of multiple networks to process compressed video features. Although training a single network cannot yield the practical classification performance of conventional methods, we propose an extended MIMO training method for action recognition to simultaneously process different features. Our experiments demonstrate that our method is the most efficient in terms of computational complexity and can achieve classification performances comparable to conventional methods.
Structural Reparameterization Lightweight Network for Video Action Recognition

3D convolution networks play an important role in extracting spatiotemporal features in video action recognition. However, it usually brings a large number of paramters, which results in deployment difficulty in edge devices with limited memory space. Although lightweight 3DCNNs can reduce the mode size significantly, it causes a serious loss of accuracy. This paper proposes a novel approach to reduce the model size while preserves accuracy by combining lightweight networks with structural reparameterization. To reduce the model size, we propose 3D-DBB module, based on 2D Diverse Branch Block(DBB). Furthermore, we propose three structures based on 3D-DBB: (1) 3D depthwise convolution (called 3D-DBB-DepthWise), (2) the 3D pointwise convolution (called 3D-DBB-PointWise), and (3) reparameterizable depthwise separable structure (called DP3DBB), which is the concatenation of the two previous structures. We design and compare the effect of two different replacements for replacing depthwise separable structures in lightweight networks. Our method achieves 93.33% with only 0.42% loss in accuracy when the model size is only 1/50 of that of 3D-ResNeXt101.
Prior-Enhanced Temporal Action Localization Using Subject-Aware Spatial Attention

Temporal action localization (TAL) aims to detect the boundary and identify the class of each action instance in a long untrimmed video. Current approaches treat video frames homogeneously, and tend to give background and key objects excessive attention. This limits their sensitivity to localize action boundaries. To this end, we propose a prior-enhanced temporal action localization method (PETAL), which only takes in RGB input and incorporates action subjects as priors. This proposal leverages action subjects’ information with a plug-and-play subject-aware spatial attention module (SA-SAM) to generate an aggregated and subject-prioritized representation. Experimental results on THUMOS-14 and ActivityNet-1.3 datasets demonstrate that the proposed PETAL achieves competitive performance using only RGB features, e.g., boosting mAP by 2.41% or 0.25% over the state-of-the-art approach that uses RGB features or with additional optical flow features on the THUMOS-14 dataset.
LSTM-Based Video Quality Prediction Accounting for Temporal Distortions in Videoconferencing Calls

Current state-of-the-art video quality models, such as VMAF, give excellent prediction results by comparing the degraded video with its reference video. However, they do not consider temporal distortions (e.g., frame freezes or skips) that occur during videoconferencing calls. In this paper, we present a data-driven approach for modeling such distortions automatically by training an LSTM with subjective quality ratings labeled via crowdsourcing. The videos were collected from live videoconferencing calls in 83 different network conditions. We applied QR codes as markers on the source videos to create aligned references and compute temporal features based on the alignment vectors. Using these features together with VMAF core features, our proposed model achieves a PCC of 0.99 on the validation set. Furthermore, our model outputs per-frame quality that gives detailed insight into the cause of video quality impairments. The VCM model and dataset are open-sourced at https://github.com/microsoft/Video_Call_MOS.
Effect of Lossy Compression Algorithms on Face Image Quality and Recognition

Lossy face image compression can degrade the image quality and the utility for the purpose of face recognition. This work investigates the effect of lossy image compression on a state-of-the-art face recognition model, and on multiple face image quality assessment models. The analysis is conducted over a range of specific image target sizes. Four compression types are considered, namely JPEG, JPEG 2000, downscaled PNG, and notably the new JPEG XL format. Frontal color images from the ColorFERET database were used in a Region Of Interest (ROI) variant and a portrait variant. We primarily conclude that JPEG XL allows for superior mean and worst case face recognition performance especially at lower target sizes, below approximately 5kB for the ROI variant, while there appears to be no critical advantage among the compression types at higher target sizes. Quality assessments from modern models correlate well overall with the compression effect on face recognition performance.
ICStega: Image Captioning-based Semantically Controllable Linguistic Steganography

Nowadays, social media has become the preferred communication platform for web users but brought security threats. Linguistic steganography hides secret data into text and sends it to the intended recipient to realize covert communication. Compared to edit-based linguistic steganography, generation-based approaches largely improve the payload capacity. However, existing methods can only generate stego text alone. Another common behavior in social media is sending semantically related image-text pairs. In this paper, we put forward a novel image captioning-based stegosystem, where the secret messages are embedded into the generated captions. Thus, the semantics of the stego text can be controlled and the secret data can be transmitted by sending semantically related image-text pairs. To balance the conflict between payload capacity and semantic preservation, we proposed a new sampling method called Two-Parameter Semantic Control Sampling to cutoff low-probability words. Experimental results have shown that our method can control diversity, payload capacity, security, and semantic accuracy at the same time.
Visual Prompting for Adversarial Robustness

In this work, we leverage visual prompting (VP) to improve adversarial robustness of a fixed, pre-trained model at test time. Compared to conventional adversarial defenses, VP allows us to design universal (i.e., data-agnostic) input prompting templates, which have plug-and-play capabilities at test time to achieve desired model performance without introducing much computation overhead. Although VP has been successfully applied to improving model generalization, it remains elusive whether and how it can be used to defend against adversarial attacks. We investigate this problem and show that the vanilla VP approach is not effective in adversarial defense since a universal input prompt lacks the capacity for robust learning against sample-specific adversarial perturbations. To circumvent it, we propose a new VP method, termed Class-wise Adversarial Visual Prompting (C-AVP), to generate class-wise visual prompts so as to not only leverage the strengths of ensemble prompts but also optimize their interrelations to improve model robustness. Our experiments show that C-AVP outperforms the conventional VP method, with 2.1× standard accuracy gain and 2× robust accuracy gain. Compared to classical test-time defenses, C-AVP also yields a 42× inference time speedup. Code is available at https://github.com/Phoveran/vp-for-adversarial-robustness.
SFEMGN: Image Denoising with Shallow Feature Enhancement Network and Multi-Scale ConvGRU

Image denoising methods based on convolutional neural networks have been popular and achieved relatively excellent performance. However, most of the existing methods cannot fully obtain and use the shallow feature information when removing noise, and cannot better combine information between various network layers. In this paper, we propose an image denoising algorithm based on a feature enhancement network and multi-scale convGRU, named a shallow feature enhancement and multi-scale convGRU denoising network (SFEMGN), through an in-depth study of convolutional networks and GRU networks. We first propose a feature enhancement block to extract richer shallow features and enhance the protection of image details. Furthermore, the proposed SFEMGN integrates a multi-scale convolution GRU module, which can combine spatial features and temporal features at the same time. Comparative experiments and ablation studies demonstrate that our proposed model can achieve competitive performance in both gray and color image denoising tasks.
Matrix Resolvent Eigenembeddings for Dynamic Graphs

Eigenvector embeddings have been widely used to study graph properties in signal processing, mining, and learning tasks. However, if a graph is changing dynamically, these embeddings have to be recomputed. In this work we introduce a novel matrix resolvent expansion-based projection scheme to update eigenvector embeddings of dynamic graphs. The proposed method can tackle graph updates where both new vertices and edges are added, and its potential is illustrated via numerical tests on real data.
Data-Aware Zero-Shot Neural Architecture Search for Image Recognition

Zero-shot neural architecture search (NAS) has shown great potential in designing image recognition networks for its high efficiency and low resource consumption. However, most of the existing zero-shot NAS methods fail to utilize prior information in datasets when calculating the score of candidate networks, leading to inferior performance. Our theoretical analysis and experimental results reveal that utilizing the samples in datasets as input for calculating scores can obtain better search results. Besides, we notice some samples in the dataset have larger score variance than the others. Based on these findings, we design data-aware zero-shot (DAZS) NAS. We introduce a generator to generate data for a score calculation with affordable overhead, and adopt contrastive learning to optimize the generator for a more stable score. Experiments show that our DAZS achieves superior results against the state-of-the-art method on both CIFAR and ImageNet-1k, and has good transferability.
Graph Learning from Gaussian and Stationary Graph Signals

Graphs have become pervasive tools to represent information and datasets with irregular support. However, in many cases, the underlying graph is either unavailable or naively obtained, calling for more advanced methods to its estimation. Indeed, graph topology inference methods that estimate the network structure from a set of signal observations have a long and well established history. By assuming that the observations are both Gaussian and stationary in the sought graph, this paper proposes a new scheme to learn the network from nodal observations. Consideration of graph stationarity overcomes some of the limitations of the classical Graphical Lasso algorithm, which is constrained to a more specific class of graphical models. On the other hand, Gaussianity allows us to regularize the estimation, requiring less samples than in existing graph stationarity-based approaches. While the resultant estimation (optimization) problem is more complex and non-convex, we design an alternating convex approach able to find a stationary solution. Numerical tests with synthetic and real data are presented, and the performance of our approach is compared with existing alternatives.
Spike-Based Optical Flow Estimation Via Contrastive Learning

Spiking cameras have shown promising advantages for optical flow estimation in high-speed scenarios. The recent work SCFlow [1] attempts to train an optical flow model using spike frames based on a multi-scale flow reconstruction loss. However, only using the flow reconstruction loss is unable to effectively deal with the details of motion, which may lead to noise and blur in the estimated flow fields. To address this issue, we introduce a contrastive loss into spike-based optical flow estimation, which exploits both the information of positive samples and negative samples. Moreover, we propose a refinement step with flexible reception fields to effectively refine the initial flow fields. Experiments on the spiking optical flow dataset PHM demonstrate that the proposed network is effective for spike-based optical flow estimation. In addition, our method achieves competitive performance compared to recent spike-based, frame-based, and event-based methods.
Learning Silhouettes with Group Sparse Autoencoders

Sparse coding has been extensively used in neuroscience to model brain-like computation by drawing analogues between neurons’ firing activity and the nonzero elements of sparse vectors. Contemporary deep learning architectures have been used to model neural activity, inspired by signal processing algorithms; however sparse coding architectures are not able to explain the higher-order categorization that has been empirically observed at the neural level. In this work, we propose a novel model-based architecture, termed group-sparse autoencoder, that produces sparse activity patterns in line with neural modeling, but showcases a higher-level order in its activation maps. We evaluate a dense model of our architecture on MNIST and CIFAR-10 and show that it learns dictionaries that resemble silhouettes of the given class, while its activations have a significantly higher level order compared to sparse architectures. Source code is available at: https://github.com/manosth/silhouette-learning.