Reviewerly/exHarmony huggingface dataset:
AMM-GAN: Attribute-Matching Memory for Person Text-to-Image Generation

Using GANs to generate realistic images from text descriptions has achieved great progress. However, it is difficult to represent attributes such as gender, clothing, hairstyle, and accessories completely and accurately, so generating full-body person images remains a challenge. In this paper, we propose the Attribute-Matching Memory Generative Adversarial Network(AMM-GAN) to generate high-quality full-body person images directly from the text. The AMM-GAN uses an attribute text feature extraction module to extract relevant attribute information from the text, and the encoder is dynamically updated to extract more accurate features. We also propose an attribute-matching memory (AMM) generator to refine the image by improving the attention to the whole attribute in a memory network composed of image and text information. Furthermore, the real-result-driven discriminators guide the network to produce more realistic and natural images and achieve better image-text matching. Experimental results demonstrate that our AMM-GAN achieves the best performance in FID and R-Precison metrics compared to existing models.
Discovering Multimodal Hierarchical Structures with Graph Neural Networks for Multi-modal and Multi-hop Question Answering

Multimodal reasoning is a challenging task that requires understanding and integrating information from different modalities, such as text and image. Existing methods for multimodal reasoning often fail to capture the rich structural information among visual and textual semantics in different modalities, which is crucial for generating accurate answers. In this paper, we propose a novel method that leverages graph neural networks to model the structural information to enhance multimodal reasoning. Specifically, we first use a Multimodal and Multi-hop reader to attend to different chunks in the context based on the question, and then search for multi-hop candidate tokens within these chunks. Next, we construct a graph to represent the relations among the chunks. Then we apply a Sparse Matrix-Tree algorithm to learn a hierarchical informative structure. Then, we use a Hierarchy-aware Message Passing mechanism to perform multi-hop reasoning on the selected edges and update the node representations. Finally, we use a graph-selection decoder to generate the answer based on the structure-enriched chunk representation. We conduct experiments on the WebQA dataset, which is a large-scale multimodal question answering dataset [1]. The results show that our method outperforms the baseline methods in terms of reasoning and the overall answer accuracy. We also provide some qualitative analysis to illustrate how our method benefits from the structural information among different modalities.
RGB-D Road Segmentation Based on Geometric Prior Information

Deep data can provide rich spatial structure information, which can effectively exclude the interference of lighting and road texture in road segmentation. This paper proposes a road segmentation model based on two kinds of a priori knowledge: disparity information, and surface normal vector information. Then, a two-branch neural network is used to process the color image and the processed depth image separately, and an effective fusion module is designed to make full use of the complementary features of the two modalities. The experimental results on the KITTI road detection dataset and Cityscape dataset show that the method in this paper has good road segmentation performance.
Enhancing Geological Understanding and Identifying Gold Anomalies in Ailaoshan Orogen

Abstract The Ailaoshan Orogen, situated between the Yangtze and Simao Blocks, underwent a complex structural, magmatic, and metamorphic evolution resulting in different tectonic subzones with varying structural lineaments and elemental concentrations. These can conceal or reduce anomalies due to the mutual effect between different anomaly areas. This study divided the whole zone into subzones based on tectonic settings, ore‐cluster areas, or sample catchment basins (SCB). To identify geochemical and structural anomalies associated with gold mineralization, the mean plus twice standard deviations (Mean+2STD), factor analysis (FA), concentration–area (C‐A) modeling of stream sediment geochemical data, and lineament density were utilized in both the Ailaoshan Orogen and the individual subzones. The FA in the divided 98 SCBs with 6 SCBs containing gold deposits can roughly ascertain unknown rock types, identify specific element associations of known rocks and discern the porphyry or skarn‐type gold mineralization. Compared with methods of Mean + 2STD and C–A model of data in the whole Ailaoshan Orogen, which mistake the anomalies as background or act the background as anomalies, the combined methods of FA and C‐A in the separate subzones or SCBs works well in regional metallogenic potential analysis. Mapping of lineament densities with a 10 km circle diameter is not suitable to locate gold deposits due to the delineated large areas of medium‐high lineament density. In contrast, the use of circle diameters of 1.3 km or 1.7 km in the ore‐cluster scale delineates areas with a higher concentration of lineament density, consistent with the locations of known gold deposits. By analyzing the map of faults and gold anomalies, two potential prospecting targets, SCBs 1 and 63 with a sandstone as a potential host rock for Au, were identified in the Ailaoshan Orogen. The use of combined methods in the divided subzones proved to be more effective in improving geological understanding and identifying mineralization anomalies associated with Au, rather than analyzing the entire large area.
Efficient Point-Based Single Scale 3D Object Detection from Traffic Scenes

In the field of 3D object detection, the point-based method faces significant limitations due to the need to process large-scale collections of irregular point clouds, resulting in reduced inference speed. To address this issue, we propose an efficient single-scale single-stage 3D object detection algorithm called SS-3DSSD. Our method eliminates the time-consuming multi-scale feature extraction module used in PointNet++ and adopts an efficient single-scale feature extraction method based on neighborhood-attention, significantly improving the model’s inference speed. Additionally, we introduce a learning-based sampling method to overcome the limited receptive fields of single-scale methods and a multi-level context feature grouping module to meet varying feature requirements at different levels. On the KITTI test set, our method achieves an inference speed of 66.7 frames per second on the RTX 2080Ti, with an average precision of 81.35% for the moderate difficulty car category. This represents a better balance between inference speed and detection accuracy, offering promising implications for real-time 3D object detection applications.
Deep Stereo Matching with Superpixel Based Feature and Cost

Previous stereo methods achieved state-of-the-art performances but are still difficult to handle the well-known edge-fattening issue at depth discontinuity regions. In this paper, we proposed differentiable superpixel-based feature and cost aggregation (DSFCA) networks for stereo matching. More specifically, we generated the superpixel maps using the simple linear iterative clustering (SLIC) method and used the efficient feature extraction network to extract the dynamic scale unary features of stereo pair images. Next, we exploited the edge or contour feature information of superpixel maps to aggregate the unary features using the differentiable superpixel-based feature aggregation (DSFA) module. The aggregated features better represent the similarity of stereo pair image feature maps and produce high-quality initial cost volumes. Furthermore, the matching cost is also aggregated by the proposed differentiable superpixel-based cost aggregation (DSCA) module. The experimental results demonstrate that the proposed method outperforms previous methods on the KITTI 2012, KITTI 2015, and SceneFlow datasets.
Segmenting Key Clues to Induce Human-Object Interaction Detection

Two-stage HOI detectors have made great progress in training and inference, but still suffer from loss of original image features and ambiguous human-object relationships. To address the above issues, this paper proposes a network based on segmentation to extract key features to induce human-object interaction detection, which consists of two parts. First, a segmentation-based module is designed to extract fine-grained interaction features from original images, which is then refined by a feature learning encoder. Secondly, a graph-based module is proposed to encode the spatial relationships of detected human and object instances, which is to learn the interaction contexts from pair-wised human-object contexts. A transformer decoder is then utilized to equip the interaction features from the original images with the interaction contexts. The proposed method can directly learn fine-grained interaction features under the guidance of spatial relationships, achieving state-of-the-art performance on two standard benchmarks for HOI detection, HICO-DET and V-COCO.
Cross-Task Physical Adversarial Attack Against Lane Detection System Based on LED Illumination Modulation

Lane detection is one of the fundamental technologies for autonomous driving, but it faces many security threats from adversarial attacks. Existing adversarial attacks against lane detection often simplify it as a certain type of computer vision task and ignore its cross-task characteristic, resulting in weak transferability, poor stealthiness, and low executability. This paper proposes a cross-task physical adversarial attack scheme based on LED illumination modulation (AdvLIM) after analyzing the decision behavior coexisting in lane detection models. We generate imperceptible brightness flicker through fast intensity modulation of LED illumination and utilize the rolling shutter effect of the CMOS image sensor to inject brightness information perturbations into the captured scene image. According to different modulation parameters, this paper proposes two attack strategies: scatter attack and dense attack. Cross-task attack experiments are conducted on the Tusimple dataset on five lane detection models of three different task types, including segmentation-based models (SCNN, RESA), point detection-based model (LaneATT), and curve-based models (LSTR, BézierLaneNet). The experimental results show that AdvLIM causes a detection accuracy drop of 77.93%, 75.86%, 51.51%, 89.41%, and 73.94% for the above models respectively. Experiments are also conducted in the physical world, and the results indicate that AdvLIM can pose a real and significant security threat to diverse lane detection systems in real-world scenarios.
Contrastive Perturbation Network for Weakly Supervised Temporal Sentence Grounding

The purpose of temporal sentence grounding is to find the most relevant temporal period corresponding to the natural language query in an unmodified video. In recent years, the weak supervision paradigm, which does not require tedious annotations of starting and ending positions of the corresponding video segments, has gained significant attention due to its low annotation cost and reasonable efficiency. However, its effectiveness is seriously affected by the low-quality negative samples generated with random strategies. In this paper, we propose a Contrastive Perturbation Network (CPN), which introduces perturbation schemes into contrastive learning of weak supervised temporal sentence grounding. The perturbation involves both the proposal generation module and the reconstruction module of the CPN. In the proposal generation module, we introduce the KL divergence loss to minimize the distribution differences between the perturbed positive and real positive proposals, to force the network to be robust to the redundant information and learn fine-grained alignments between the text and video modalities. The reconstruction module leverages the perturbed features to generate a highly challenging negative proposal and strengthens the supervision to the proposal generation module by distinguishing the positive and negative proposals with the use of contrastive learning. Extensive experiments on two public benchmarks, i.e., ActivityNet Captions and Charades-STA, demonstrate that the proposed CPN could effectively improve the performance of weakly supervised temporal sentence grounding.
Lightweight Multispectral Skeleton and Multi-stream Graph Attention Networks for Enhanced Action Prediction with Multiple Modalities

Human action recognition methods often focus on extracting structural and temporal information from skeleton-based graphs. However, these approaches struggle with effectively capturing and processing extensive information during action transitions. To overcome this limitation, we propose LMS-GAT, a novel approach that facilitates information exchange through node concentration and diffusion across structural and temporal dimensions. By selectively suppressing and reinstating the representations of structural nodes for each specific action, and utilizing hierarchical shifted temporal windows for assessing temporal information, LMS-GAT addresses the challenge of dynamic changes in action recognition. Experimental evaluation on NTU RGB+D 60 and 120 datasets shows that LMS-GAT outperforms state-of-the-art methods in terms of prediction accuracy. This highlights the efficacy of our approach in capturing and recognizing human actions with improved performance.
An Efficient Momentum Framework for Face-Voice Association Learning

Cross-modal face-voice association is an active field that utilizes biometric features for cross-modal information retrieval. The primary approach for addressing this task involves utilizing contrastive learning to construct a modality-agnostic subspace. However, many existing contrastive learning methods in cross-modal research tend to neglect the significance of symmetrical information within heterogeneous data. This oversight leads to the generation of different negative examples for each identity in a random mini-batch. Furthermore, the length of negative examples in contrastive learning is coupled with the mini-batch size and is limited by the GPU memory size. To address these issues, this paper introduces an innovative Cross-Modal Momentum Contrast (CMMC) algorithm, which leverages queues to provide sufficient and symmetric information. Moreover, we propose an update strategy to maintain the consistency of negative example information throughout the training process. By combining the operations mentioned above, our proposed CMMC can effectively improve the correlation between face and voice data. Extensive experiments conducted on two datasets confirm the superiority of our framework and demonstrate its competitive performance compared to state-of-the-art methods.
FGFusion: Fine-Grained Lidar-Camera Fusion for 3D Object Detection

Lidars and cameras are critical sensors that provide complementary information for 3D detection in autonomous driving. While most prevalent methods progressively downscale the 3D point clouds and camera images and then fuse the high-level features, the downscaled features inevitably lose low-level detailed information. In this paper, we propose Fine-Grained Lidar-Camera Fusion (FGFusion) that make full use of multi-scale features of image and point cloud and fuse them in a fine-grained way. First, we design a dual pathway hierarchy structure to extract both high-level semantic and low-level detailed features of the image. Second, an auxiliary network is introduced to guide point cloud features to better learn the fine-grained spatial information. Finally, we propose multi-scale fusion (MSF) to fuse the last N feature maps of image and point cloud. Extensive experiments on two popular autonomous driving benchmarks, i.e. KITTI and Waymo, demonstrate the effectiveness of our method.
Learning Adapters for Text-Guided Portrait Stylization with Pretrained Diffusion Models

This paper presents a framework for text-guided face portrait stylization using a pre-trained large-scale diffusion model. To balance style transformation and content preservation, we introduce an adapter that modifies specific components of the diffusion model. By training the adapter to only modify these components, we reduce the tuning parameter space, resulting in an efficient solution for face portrait stylization. Our approach captures the target style and at the same time, preserves the source portrait content, making it an effective method for personalized image editing. Experimental results show its superiority over state-of-the-art techniques in various stylization tasks.
Enhancing Text-Image Person Retrieval Through Nuances Varied Sample

Text-image person retrieval is a task that involves searching for a specific individual based on a corresponding textual description. However, a key challenge in this task is achieving modal alignment while conducting fine-grained retrieval. Current methods utilize classification and metric losses to enhance discrimination and alignment. Nevertheless, the substantial dissimilarities between samples often impede the network’s capacity to learn discriminative fine-grained information. To tackle this issue and enable the network to focus on intricate details, we introduce the Nuanced Variation Module (NVM). This module generates artificially difficult negative samples, which serve as a guide for directing the network’s attention towards discerning nuances. The incorporation of NVM-constructed hard-negative samples enhances the alignment loss and facilitates the network’s attentiveness to details. Additionally, we leverage the image text matching task to explicitly augment the network’s fine-grained ability. By adopting our NVM method, the network can extract an ample amount of fine-grained features, thereby mitigating the interference caused by challenging negative samples. Extensive experiments demonstrate that our proposed method achieves competitive performance compared to state-of-the-art approaches on publicly available datasets.
Autoencoder and Masked Image Encoding-Based Attentional Pose Network

Despite recent advances in single-image-based 3D human pose and shape estimation, partial occlusion remains a major challenge for many methods, leading to significant prediction errors. Some existing methods fail to provide satisfactory performance for 3D human body reconstruction in occluded outdoor environments. To address these issues, we propose an autoencoder for feature extraction that integrates image masking methods to improve training stability. Our approach utilizes an attention mechanism to effectively capture the features of partially visible body parts, addressing partial occlusion. We further employ a partial attention mechanism to obtain the final features and use a regressor to estimate human model parameters. Experimental results on outdoor 3D poses in benchmark datasets demonstrate that our method outperforms state-of-the-art image-based methods in terms of robustness and efficiency. Qualitative evaluation shows that our method achieves more accurate and robust reconstruction results than existing methods, not only in occluded scenarios but also on standard benchmarks. Our approach exhibits excellent model robustness and training stability.
HFGCN-Based Action Recognition System for Figure Skating

With the rising popularity of winter sports, there is a growing interest in figure skating, and an action recognition algorithm for figure skating is increasingly needed. The figure skating action recognition algorithm is not only good for intelligent event understanding but also for objective competition judging. However, figure skating actions are difficult to distinguish by one frame, and the errors caused by the keypoint detection may be propagated in the action classification, and affect the performance of the figure skating action recognition algorithm. In this paper, a figure skating hierarchical dataset FSHD-10 and a Figure Skating Action Recognition System (FSARS) are established. The FSARS adopts a multi-stream structure and a decision fusion module to learn features at different dimensions. A Hierarchical Fine-Grained Graph Convolutional Neural Network (HFGCN) is also proposed in this paper. The HFGCN extracts the temporal features by a temporal modeling module and an attention module. The hierarchical classification adopted in HFGCN takes advantage of the hierarchy of figure skating actions and improves the precision of the action classification. The experimental results show that these improvements contribute to FSARS and make it achieve a final accuracy of 93.70% on the FSHD-10 dataset.
Enhancing Image Comprehension for Computer Science Visual Question Answering

Computer science visual question answering is a fundamental task in the intelligent education. However, current models have poor performance in this task. There are mainly two issues in these models. Firstly, they cannot accurately capture the fine-grained objects and relations in images. Secondly, these models lack the computer domain knowledge. To address the issues, we propose an Image Comprehension Enhancing Model. Specifically, it uses object detection technique to capture fine-grained features of images and utilizes Optical Character Recognition (OCR) to transform fine-grained features into the text information. The model adopts the text information to prompt the large language model, which generates the image caption with the computer domain knowledge. The fine-grained features and image caption can enhance the model’s image comprehension and compensate the lack of knowledge. Additionally, the model utilizes the cross-modal attention mechanism to integrate the image features and fine-grained features of the image with the text features. The experimental results on the CSDQA dataset demonstrate that our proposed model outperforms the baselines, and the accuracy improves at least 4.80%.
An Effective Dynamic Reweighting Method for Unbiased Scene Graph Generation

Despite the remarkable advancements in Scene Graph Generation (SGG) in recent years, the precise capture and modeling of long-tail object relationships remain persistent challenges in the field. Conventional methods generally employ resampling and reweighting techniques to achieve unbiased predictions. Existing reweighting methods in SGG calculate weights based on the class distribution of the dataset. And they focus on the reweighting of the related samples while overlooking the reweighting of the samples whose objects are unrelated. However, the sample distribution during the training process is inconsistent with the class distribution of the dataset, and the reweighting of samples whose objects are unrelated should not be overlooked. In this paper, we propose a novel method named Dynamic Reweighting based on the Sample Distribution (DRSD). The DRSD method calculates the weights of classes based on the sample distribution during the training process and incorporates reweighting for the samples whose objects are unrelated. Specifically, we utilize a sample queue mechanism to record and update the sample distribution and introduce a transition mechanism to ensure training stability. The experiments conducted on the Visual Genome dataset demonstrate the effectiveness of our method. Our method exhibits model-agnostic characteristics and yields significant performance improvements on three benchmark models (Motif, VCTree, and Transformer). Specifically, it achieves an increase of $$23.4\%$$ , $$25.1\%$$ , and $$27.6\%$$ on the mR@100 metric for the Predicate Classification task, achieving $$40.9\%$$ , $$41.2\%$$ , and $$43.4\%$$ , respectively. Moreover, our method outperforms the state-of-the-art reweighting method in SGG, i.e. FGPL, by $$3\%$$ .