Reviewerly/exHarmony huggingface dataset:
UFS-Net: Unsupervised Network For Fashion Style Editing And Generation

AI-aided fashion design has attracted growing interest because it eliminates tedious manual operations. However, existing methods are costly because they require abundant labeled data or paired images for training. In addition, they have low flexibility in attribute editing. To overcome these limitations, we propose UFS-Net, a new unsupervised network for fashion style editing and generation. Specifically, we initially design a coarse-to-fine embedding process to embed the user-defined sketch and the real clothing into the latent space of StyleGAN. Subsequently, we propose a feature fusion scheme to generate clothing with attributes provided by the sketch. In this way, our network requires neither labels nor sketches during the training but can perform flexible attribute editing and conditional generation. Extensive experiments reveal that our method significantly outperforms state-of-the-art approaches. In addition, we introduce a new dataset, Fashion-Top, to address the limitations in the existing fashion datasets.
Hierarchical Class Level Attribute Guided Generative Meta Learning for Pest Image Zero-shot Learning

Existing pest image classification models require a large number of labeled training images. However, labels for most pest images in the real world do not exist. Therefore, the zero-shot learning method based on generative meta-learning provides an effective solution, which first uses attributes to transfer knowledge from seen classes to unseen classes, and then synthesizes the features of unseen classes. We observe that seen and unseen classes share the same high-level attributes, which can be used to learn a shared set of optimal parameters for seen and unseen classes. Therefore, we propose a novel Hierarchical Class level Attribute guided Generative meta model for pest image Zero-shot Learning (HCAG-ZSL). HCAG-ZSL uses the pre-built Taxonomic Attribute Tree to get the high-level attributes corresponding to the class attributes. These attributes are then fed into a well-designed generator to generate visual features. Extensive experiments show that the proposed model outperforms state-of-the-art generative meta models.
RF-based Multi-view Pose Machine for Multi-Person 3D Pose Estimation

In this paper, we present RF-based Multi-view Pose machine (RF-MvP) for multi-person 3D pose estimation using RF signals. Specifically, we first develop a lightweight anchor-free detector module to locate and crop regions of interest from horizontal and vertical RF signals. Afterward, we propose a Multi-view Fusion Network to unproject the RF signals from the horizontal and vertical millimeter-wave radars into a unified latent space, and then calculate the correlation for weighted fusion. Finally, a Spatio-Temporal Attention Network is designed to reconstruct the multi-person 3D skeleton sequences, in which the spatial attention module is proposed to recover invisible body parts using non-local correlations among joints and the temporal attention module refines the 3D pose sequences using temporal coherency learned from frame queries. We evaluate the performance of the proposed RF-MvP and state-of-the-art methods on a large-scale dataset with multi-person 3D pose labels and corresponding radar signals. The experimental results show that RF-MvP outperforms all of the baseline methods, which locates multi-person 3D key points with an average error of 73mm and generalizes well in new data such as occlusion, low illumination.
Injecting-Diffusion: Inject Domain-Independent Contents into Diffusion Models for Unpaired Image-to-Image Translation

Diffusion models have shown remarkable performance in the task of image synthesis. However, we notice that existing methods fail to preserve domain-independent contents of the input images, making it challenging for unpaired image-to-image translation. To address this issue, we proposed a diffusion model for domain-independent content injecting. We propose a domain-independent content extractor to obtain domain-independent contents from the source domain. After that, we inject the extracted contents into the diffusion model and fuse them with domain-specific appearances of the target domain through our proposed cross-domain attention mechanism. The qualitative and quantitative experiments demonstrate that our proposed method can generate high-fidelity images of the target domain while preserving domain-independent contents of the source domain.
Multimodal Sentiment Analysis with Preferential Fusion and Distance-aware Contrastive Learning

Recent efforts on multimodal sentiment analysis (MSA) leverage data from multiple modalities, among which the text modality is heavily relied on. However, the text modality often contains false correlations between text tokens and sentiment labels, leading to errors in sentiment analysis. To address this issue, we propose a new framework, PriSA, which incorporates the preferential fusion and distance-aware contrastive learning. Specifically, we first propose a preferential inter-modal fusion method, which utilizes the text modality to guide the calculation of the inter-modal correlations. Then the resulting inter-modal features are further used to calculate mixed-modal correlations through our proposed distance-aware contrastive learning, which leverages the distance information of the sentiment labels. At last, we identify the sentiment information based on both the mixed-modal correlations and the discriminative intra-modal features extracted from the visual and audio modalities via a self-attention module. Experimental results show that our proposed PriSA achieves the state-of-the-art performance on four datasets, including MOSEI, MOSI, SIMS, and UR-FUNNY. The code is available at https://github.com/FeipengMa6/PriSA.
Improving Multi-generation Robustness of Learned Image Compression

Benefiting from flexible network designs and end-to-end joint optimization approach, learned image compression (LIC) has demonstrated excellent coding performance in recent years. However, existing compression models suffer from serious multi-generation loss, which always occurs during image editing and transcoding. During the process of repeatedly encoding and decoding, the image quality will rapidly degrade, resulting in various types of distortion, which significantly limits the practical application of LIC. In this paper, a thorough analysis is carried out to determine the source of generative loss in successive image compression (SIC). We point out and solve the quantization drift problem that affects SIC, reversibility loss function as well as channel relaxation method are proposed to further reduce the generation loss. Experiments show that by using our proposed solutions, LIC can achieve comparable performance to BPG even after 50 times reencoding. Our code is available at https://github.com/leelitian/Multi-Generation-Robust-Coding.
Who, What and Where: Composite-semantic Instance Search for Story Videos

This paper studies Who-What-Where (3W) composite-semantic video instance search (INS) problem, which aims to find a specific person doing a queried action in a particular place. Mainstream approaches adopt a complete decomposition strategy, which divides a composite-semantic query into multiple single-semantic queries. However, due to the lack of necessary correlation analysis among constituent semantics, these methods cannot always generate identity-matching and semantics-consistent 3W INS results. To address the above challenges, we propose a partial decomposition scheme with action as the link. Specifically, we selectively split the 3W INS as person-action INS and action-location INS. The former ensures the retrieved person and action share the same identity by modeling their relative spatial positions at the frame level, while the latter improves the semantic consistency between action and location with a cross-semantic attention mechanism at the shot level. Particularly, we build a large-scale 3W INS dataset, containing over 470k video shots, on basis of NIST TRECVID 2016-2021 INS tasks and verify the effectiveness of the proposed method with both quantitative and qualitative experiments.
Noisy-to-Clean Label Learning for Medical Image Segmentation

In the field of medical image processing, accurate segmentation is of great importance to assist doctors in diagnosis. However, existing machine learning methods are hardly effective for medical image segmentation in the absence of large and accurate datasets. Existing methods of learning with noisy labels rarely try to explore the correlation between noisy and clean labels. We found that some error corrections are learnable in the process of noisy labels corrected by medical experts. In this work, we propose a novel method to improve the performance of medical image segmentation. The method consists of two main networks: segmentation network segments the image and label correction network records and learns the denoising process of noisy labels, denoises the noisy labels. In addition, we introduce a feature fusion branch between the two networks. We compare with several state-of-the-art methods which learning with noisy label on the gastric wall dataset and notice that our method has strong competitiveness.
Labeled Probability Hypothesis Density Filtering for Track-Before-Detect Strategy

Weak target recognition, tracking and track management with a low signal-to-noise ratio (SNR) are always tricky problems. Probability hypothesis density (PHD) filtering propagates the first-order multi-target moment to obtain the best Poisson approximation to multi-target density. The PHD filtering does not consider explicit associations between measurements and targets, which is computationally efficient. But it cannot distinguish different targets or extract the time series of track states. Based on track-before-detect (TBD) strategies, this paper proposes labeled PHD (LPHD) filtering and derives its close-form solution, which identifies targets with a unique label. It is derived based on rigorous Bayes criteria, finite set statistics and Kullback-Leibler divergence minimization approximation. The separable TBD-based observation likelihood is conjugate to the Poisson mixture prior for LPHD filtering. Under the point-target assumption, the multi-hypothesis assignments of pixel-to-target are implemented with Murty’s K-shortest path algorithm for LPHD filtering. Additionally, sequential Monte Carlo (SMC) implementations under the nonlinear non-Gaussian assumption are devised. Finally, simulations exhibit good performance in low SNR scenarios.
CCLAP: Controllable Chinese Landscape Painting Generation Via Latent Diffusion Model

With the development of deep generative models, recent years have seen great success of Chinese landscape painting generation. However, few works focus on controllable Chinese landscape painting generation due to the lack of data and limited modeling capabilities. In this work, we propose a controllable Chinese landscape painting generation method named CCLAP, which can generate painting with specific content and style based on Latent Diffusion Model. Specifically, it consists of two cascaded modules, i.e., content generator and style aggregator. The content generator module guarantees the content of generated paintings specific to the input text. While the style aggregator module is to generate paintings of a style corresponding to a reference image. Moreover, a new dataset of Chinese landscape paintings named CLAP is collected for comprehensive evaluation. Both the qualitative and quantitative results demonstrate that our method achieves state-of-the-art performance, especially in artfully-composed and artistic conception. Codes are available at https://github.com/Robin-WZQ/CCLAP.
Cross-domain Prototype Contrastive loss for Few-shot 2D Image-Based 3D Model Retrieval

2D image-based 3D model retrieval (IBMR) usually relies on abundant explicit supervision on 2D images, together with unlabeled 3D models to learn domain-aligned yet class-discriminative features for the retrieval task. However, collecting large-scale 2D labels is cost-effective and time-consuming. Therefore, we explore a challenging IBMR task, where only few-shot labeled 2D images are available while the rest of the 2D and 3D samples remain unlabeled. Limited annotation of 2D images further increases the difficulty of domain-aligned yet discriminative feature learning. Therefore, we propose cross-domain prototype contrastive loss (CPCL) for the few-shot IBMR task. Specifically, we capture semantic information to learn class-discriminative features in each domain by minimizing intra-domain prototype contrastive loss. Besides, we perform inter-domain transferable contrastive learning to align the features between instances and prototypes of the same class across domains. Comprehensive experiments on popular benchmarks, MI3DOR and MI3DOR-2, validate the superiority of CPCL.
A Unified MRC Framework with Multi-Query for Multi-modal Relation Triplets Extraction

Relation triplets extraction (RTE) aims to extract all potential triplets of subject-object entities and the corresponding relation in a sentence, which is an extremely essential procedure in information extraction and knowledge graph construction. In the literature, dominant studies normally focus on text only, which neglects the fact that other modalities may have effective information available for entities and relations (e.g., the image accompanying the text on Twitter). Therefore, in this paper, we introduce a multi-modal task for relation triplets extraction (MRTE), which considers both text and image modalities. To well handle the multi-modality and achieve all potential triplets extraction, we propose a novel multi-modal approach by resorting to a unified machine reading comprehension (MRC) framework with multiple queries, namely MUMRC. Specifically, we design a unified framework to conduct multi-modal entity extraction, and multi-modal relation classification depending on all potential subject-object entity pairs, which recasts both sub-tasks as a machine reading comprehension (MRC) problem. In this unified MRC model, we incorporate multiple questions into original text data and leverage visual information to enhance the textual semantics. Extensive experiments on the MNRE dataset demonstrate that the proposed approach significantly outperforms the state-of-the-art baselines of textual RTE and multi-modal information extraction.
Underwater Image Enhancement with an Adaptive Self Supervised Network

High-quality underwater images are a vital source of modern marine vision and multimedia applications. But low visibility with color and contrast distortions in uneven underwater illumination conditions poses several challenges to these applications (e.g., visual servoing, long-range navigation, unmanned underwater vehicles, and SONAR imaging, etc.) working in robust conditions from dawn to dusk. Direct enhancement deteriorates the structure and texture with objectionable color casts due to distance-dependent attenuation and light scattering. This paper presents an adaptive self-supervised network for underwater image enhancement (ASSU-Net), which can work without relying on the type and quantity of training images, rather, it is capable of working with only a few input images. In this network, we split the image into reflection and illumination components to simplify the solution space, allowing us to handle the artifacts associated with each component independently. A structure and texture awareness scheme is introduced while following the mutual consistency of decomposition in combination with the histogram distribution. This scheme is embedded in the network to constrain objectionable artifacts (i.e., structure texture and edge distortions). The network loss is optimized to force the network to learn with a few shots of unpaired data, where we preserve visual details for low-lit underwater images. Extensive experiments are performed to demonstrate the superiority of the proposed method.
Fine-Grained Alignment for Boundary Samples under Open Set Domain Adaptation

Open set domain adaptation aims to transfer knowledge in the presence of unknown samples in the target domain. Previous approaches use additional classifiers or threshold-based methods to identify unknown samples and try to investigate the information of class diversity within the unknown samples. Despite achieving excellent adaptation results, these methods ignore those samples that lie on the cluster boundaries, especially the clustering-based methods. In this paper, we propose a novel Neighbor Prototype Contrastive Clustering (NPC <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) method, which uses the Local Semantic Structure (LSS) to help these low-confidence samples located on the boundary of clusters to return to their own clusters. Further, we propose Local Semantic Consistency (LSC) to evaluate the clustering result and apply it to the domain adaptation process as a metric to assess the reliability of the samples. Results on four benchmarks show that our NPC <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> significantly outperforms most state-of-the-art methods with higher LSC.
Knowledge Enhanced Model for Live Video Comment Generation

Live video commenting is popular on video media platforms, as it can create a chatting atmosphere and provide supplementary information for users while watching videos. Automatically generating live video comments can improve user experience and enable human-like generation for bot chatting. Existing works mostly focus on short video datasets while ignoring other important video types such as long videos like movies. In this work, we collect a new Movie Live Comments (MovieLC) dataset to support research on live video comment generation for long videos. We also propose a knowledge enhanced generation model inspired by the divergent and informative nature of live video comments. Our model adopts a pre-training encoder-decoder framework and incorporates external knowledge. Extensive experiments show that both objective metrics and human evaluation demonstrate the effectiveness of our proposed model. The MovieLC dataset and our code will be released.
Social Context-aware GCN for Video Character Search via Scene-prior Enhancement

With the increasing demand for intelligent services of online video platforms, video character search task has attracted wide attention to support downstream applications like fine-grained retrieval and summarization. However, traditional solutions only focus on visual or coarse-grained social information and thus cannot perform well when facing complex scenes, such as changing camera view or character posture. Along this line, we leverage social information and scene context as prior knowledge to solve the problem of character search in complex scenes. Specifically, we propose a scene-prior-enhanced framework, named SoCoSearch. We first integrate multimodal clues for scene context to estimate the prior probability of social relationships, and then capture characters’ co-occurrence to generate an enhanced social context graph. Afterwards, we design a social context-aware GCN framework to achieve feature passing between characters to obtain robust representation for the character search task. Extensive experiments have validated the effectiveness of SoCoSearch in various metrics.
Audio-Visual Generalized Zero-Shot Learning Based on Variational Information Bottleneck

Audio-visual generalized zero-shot learning (GZSL) aims to train a model on seen classes for classifying data samples from both seen classes and unseen classes. Due to the absence of unseen training samples, the model tends to misclassify unseen class samples into seen classes. To mitigate this problem, in this paper, we propose a method based on variational information bottleneck for audio-visual GZSL. Specifically, we model the joint representations as a product-of-experts over marginal representations to integrate the information of audio and visual. Besides, we introduce variational information bottleneck to the learning of audio-visual joint representations and marginal representations of audio, visual, and text label modalities. This helps our model reduce the negative impact of information that cannot be generalized to unseen classes. Experimental results conducted on the UCF-GZSL, VGGSound-GZSL, and ActivityNet-GZSL benchmarks demonstrate the effectiveness and superiority of the proposed model for audio-visual GZSL.
Cross-Level Attention Based Adaptive Feature Alignment Network for Arbitrary-Shaped Text Detection

Over the past few years, segmentation-based text detection methods have achieved great progress in the scene text detection field. However, most of the existing methods with FPN tend to ignore the feature misalignment issues caused by semantic gap between features at different levels, leading to inaccurate segmentation prediction. Therefore, we propose an end-to-end trainable text detector to alleviate the above dilemma. Specifically, we propose an Adaptive Feature Transform Module (AFTM) to adaptively align features of different levels. Furthermore, a Cross-Level Attention (CLA) block is developed to capture the cross-level information, and selectively enhance element-wise features of multi-level feature maps. Experiments on four benchmark datasets, MSRA-TD500, Total-Text, CTW1500 and ICDAR2015, demonstrate that our proposed method has competitive performance and strong robustness.
MuDPT: Multi-modal Deep-symphysis Prompt Tuning for Large Pre-trained Vision-Language Models

Prompt tuning, like CoOp, has recently shown promising vision recognizing and transfer learning ability on various downstream tasks with the emergence of large pre-trained vision-language models like CLIP. However, we identify that existing uni-modal prompt tuning approaches may result in sub-optimal performance since this uni-modal design breaks the original alignment of textual and visual representations in the pre-trained model. Inspired by the nature of pre-trained vision-language models, we aim to achieve completeness in prompt tuning and propose a novel approach called Multi-modal Deep-symphysis Prompt Tuning, dubbed as MuDPT, which extends independent multi-modal prompt tuning by additionally learning a model-agnostic transformative network to allow deep hierarchical bi-directional prompt fusion. We evaluate the effectiveness of MuDPT on few-shot vision recognition and out-of-domain generalization tasks. Compared with the state-of-the-art methods, MuDPT achieves better recognition and generalization ability with an apparent margin thanks to synergistic alignment of textual and visual representations. Our code is available at: https://github.com/Mechrev0/MuDPT.
Multi-Level Feature-Guided Stereoscopic Video Quality Assessment Based on Transformer and Convolutional Neural Network

Stereoscopic video (3D video) has been increasingly applied in industry and entertainment. And the research of stereoscopic video quality assessment (SVQA) has become very important for promoting the development of stereoscopic video system. Many CNN-based models have emerged for SVQA task. However, these methods ignore the significance of the global information of the video frames for quality perception. In this paper, we propose a multi-level feature-fusion model based on Transformer and convolutional neural network (MFFTCNet) to assess the perceptual quality of the stereoscopic video. Firstly, we use global information from Transformer to guide local information from convolutional neural network (CNN). Moreover, we utilize low-level features in the CNN branch to guide high-level features. Besides, considering the binocular rivalry effect in the human vision system (HVS), we use 3D convolution to achieve rivalry fusion of binocular features. The proposed method is tested on two public stereoscopic video quality datasets. The result shows that this method correlates highly with human visual perception and outperforms state-of-the-art (SOTA) methods by a significant margin.