bluuebunny/crossref_metadata_2025 huggingface dataset:
Improving Non-Autoregressive Sign Language Translation with Random Ordering Progressive Prediction Pretraining
Authors: Yu Pei , Lai Changhao , Hu Cong , Liu Shan , Zhang Liang , Fu Biao , Chen Yidong
Published: October 2024

Recently, the Non-AutoRegressive (NAR) decoding mechanism, effectively reducing the inference latency of text generation, has been applied to Sign Language Translation (SLT). Typically, the current best NAR SLT model using a Curriculum-based Non-autoregressive Decoder (CND) outperforms AutoRegressive (AR) baselines in speed and performance. Although it has been proven that AutoRegressive Pre-trained Language Models (AR-PLMs) further boost the performance of AR SLT models, combining NAR Pretrained Language Models (NAR-PLMs) with NAR SLT model remains challenge due to (1) existing NAR-PLMs’ inability to model token dependencies between decoder layers, crucial for NAR SLT models using CND; (2) the modality gap between the decoder’s inputs of the NAR-PLMs and NAR SLT models. To address these, we propose a Random Ordering Progressive Prediction Pre-training task for NAR SLT models using CND, enabling the decoder to predict target sequences in diverse orderings and enhancing the modeling of target token dependencies between layers. Moreover, we propose a CTC-enhanced Soft Copy method to incorporate target-side information in the decoder’s inputs, alleviating the modality gap. Experimental results on PHOENIX-2014T and CSL-Daily demonstrate that our model consistently outperforms all strong baselines and achieves competitive performance with AR SLT models equipped with AR-PLMs.
Generalized Face Anti-Spoofing via Finer Domain Partition and Disentangling Liveness-Irrelevant Factors
Authors: Yang Jingyi , Yu Zitong , Ni Xiuming , He Jia , Li Hui
Published: October 2024

Face anti-spoofing techniques based on domain generalization have recently been studied widely. Adversarial learning and meta-learning techniques have been adopted to learn domain-invariant representations. However, prior approaches often consider the dataset gap as the primary factor behind domain shifts. This perspective is not fine-grained enough to reflect the intrinsic gap among the data accurately. In our work, we redefine domains based on identities rather than datasets, aiming to disentangle liveness and identity attributes. We emphasize ignoring the adverse effect of identity shift, focusing on learning identity-invariant liveness representations through orthogonalizing liveness and identity features. To cope with style shifts, we propose Style Cross module to expand the stylistic diversity and Channel-wise Style Attention module to weaken the sensitivity to style shifts, aiming to learn robust liveness representations. Furthermore, acknowledging the asymmetry between live and spoof samples, we introduce a novel contrastive loss, Asymmetric Augmented Instance Contrast. Extensive experiments on four public datasets demonstrate that our method achieves state-of-the-art performance under cross-dataset and limited source dataset scenarios. Additionally, our method has good scalability when expanding diversity of identities. Code is available at https://github.com/yjyddq/DLIF.
TP-GMOT: Tracking Generic Multiple Object by Textual Prompt with Motion-Appearance Cost (MAC) SORT
Authors: Anh Duy Le Dinh , Tran DKim Hoang , Le DNgan Hoang
Published: October 2024

While Multi-Object Tracking (MOT) has made substantial advancements, it is limited by heavy reliance on prior knowledge and limited to predefined categories. In contrast, Generic Multiple Object Tracking (GMOT), tracking multiple objects with similar appearance, requires less prior information about the targets but faces challenges with variants like viewpoint, lighting, occlusion, and resolution. Our contributions commence with the introduction of the Refer-GMOT dataset a collection of videos, each accompanied by fine-grained textual descriptions of their attributes. Subsequently, we introduce a novel text prompt-based open-vocabulary GMOT framework, called TP-GMOT, which can track never-seen object categories with zero training examples. Within TP-GMOT framework, we introduce two novel components: (i) TP-OD, an object detection by a textual prompt, for accurately detecting unseen objects with specific characteristics. (ii) Motion-Appearance Cost SORT MAC-SORT, a novel object association approach that adeptly integrates motion and appearance-based matching strategies to tackle the complex task of tracking multiple generic objects with high similarity. Our contributions are benchmarked on the Refer-GMOT dataset for GMOT task. Additionally, to assess the generalizability of the proposed TP-GMOT framework and the effectiveness of MAC-SORT tracker, we conduct ablation studies on the DanceTrack and MOT20 datasets for the MOT task. Our dataset, code, and models will be publicly available at: https://fsoft-aic.github.io/TP-GMOT
Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Model
Authors: Kim Taehee , Cho Yeongjae , Shin Heejun , Jo Yohan , Shin Dongmyung
Published: October 2024

Visual question answering (VQA) is a task where an image is given, and a series of questions are asked about the image. To build an efficient VQA algorithm, a large amount of QA data is required which is very expensive. Generating synthetic QA pairs based on templates is a practical way to obtain data. However, VQA models trained on those data do not perform well on complex, human-written questions. To address this issue, we propose a new method called chain of QA for human-written questions (CoQAH). CoQAH utilizes a sequence of QA interactions between a large language model and a VQA model trained on synthetic data to reason and derive logical answers for human-written questions. We tested the effectiveness of CoQAH on two types of human-written VQA datasets for 3D-rendered and chest X-ray images and found that it achieved state-of-the-art accuracy in both types of data. Notably, CoQAH outperformed general vision-language models, VQA models, and medical foundation models with no finetuning. The source code for CoQAH is available [10].
PromptCD: Coupled and Decoupled Prompt Learning for Vision-Language Models
Authors: Wu Junjie , Sun Mingjie , Gong Chen , Yu Nan , Fu Guohong
Published: October 2024

Large-scale pre-trained vision-language models (VLMs), like CLIP, have presented striking generalizability for adapting to image classification in a few shot setting. Most existing methods explore a set of learnable tokens, such as prompt learning, on data-efficient utilization for task adaptation. However, they focus on either the coupled-modality property by prompt projection or decoupled-modality characteristic by prompt consistency, which ignores effective interaction between prompts. To model the deep yet sufficient cross-modal interaction and enhance the generalization between both seen and unseen tasks, in this paper, we propose a novel coupled and decoupled prompt learning framework, dubbed PromptCD, for vision-language models. Specifically, we introduce a bi-directional coupled-modality mechanism to intensify the interaction between both vision and language branches. Additionally, we propose mixture consistency to further improve the generalization and discrimination of the models on unseen tasks. The integration of such a mechanism and consistency facilitates the proposed framework adaptation for various downstream tasks. We conduct extensive experiments on 11 image classification datasets under a range of evaluation protocols, including base-to-novel and domain generalization, and cross-dataset recognition. Experimental results demonstrate that our proposed PromptCD overall outperforms state-of-the-art methods.
Dual Attention Encoder with Joint Preservation for Medical Image Segmentation
Authors: Li Shijie , Tu Yunbin , Gong Yu , Zhong Bowen , Li Zheng
Published: October 2024

Transformers have recently gained considerable popularity for capturing long-range dependencies in the medical image segmentation. However, most transformer-based segmentation methods primarily focus on modeling global dependencies and fail to fully explore the complementary nature of different dimensional dependencies within features. These methods simply treat the aggregation of multi-dimensional dependencies as auxiliary modules for incorporating context into the Transformer architecture, thereby limiting the model’s capability to learn rich feature representations. To address this issue, we introduce the Dual Attention Encoder with Joint Preservation (DANIE) for medical image segmentation, which synergistically aggregates spatial-channel dependencies across both local and global areas through attention learning. Additionally, we design a lightweight aggregation mechanism, termed Joint Preservation, which learns a composite feature representation, allowing different dependencies to complement each other. Without bells and whistles, our DANIE significantly improves the performance of previous state-of-the-art methods on five popular medical image segmentation benchmarks, including Synapse, ACDC, ISIC 2017, ISIC 2018 and GlaS.
RFDFM: A Deep Factorization Machine Network Model for Invasive Lung Adenocarcinoma Screening in CT Images
Authors: Zhou Jing , Guo Chengcheng , Ji Ying
Published: October 2024

As a common subtype of lung cancer, the diagnosis of lung adenocarcinoma has significant importance in clinical practice, particularly in distinguishing between pre-invasive adenocarcinoma (Pre-IA) and invasive adenocarcinoma (IAC). This distinction is critical because the two types of lesions correspond to different clinical treatment strategies: Pre-IAs typically require only regular observation, while IACs necessitate immediate surgical removal. In this article, we propose a novel deep learning model, the Radiomic Feature Deep Factorization Machine (RFDFM) network model, for distinguishing IACs from Pre-IAs in CT images, leveraging both radiomic features and deep learning features. Our novelty resides in pioneering the application of recommendation system model structures to computer-aided diagnosis of pulmonary nodules, demonstrating feasibility and effectively addressing the limitations of traditional methods in handling radiomic features. Moreover, the use of low-level feature fusion convolutional neural networks minimizes the information loss, and an element-wise attention mechanism in feature fusion stage to accentuate key features and improve model fitting. For extensive validation, 1,052 nodule samples were gathered from a total of 791 patients that were diagnosed with lung adenocarcinoma across two top-tier hospitals. The proposed RFDFM method can achieve a sota performance of 94.2% in terms of AUC. Results of extensive ablation studies demonstrate its contribution to improved performance. Finally, to promote more efficient academic communication, the analysis code is publicly available at https://github.com/Chengcheng-Guo/RFDFM.
Decoupled Competitive Framework for Semi-Supervised Medical Image Segmentation
Authors: Chen Jiahe , Ying Jiahe , Wang Shen , Zheng Jianwei
Published: October 2024

Confronting the critical challenge of insufficiently annotated samples in medical domain, semi-supervised medical image segmentation (SSMIS) emerges as a promising solution. Specifically, most methodologies following the Mean Teacher (MT) or Dual Students (DS) architecture have achieved commendable results. However, to date, these approaches face a performance bottleneck due to two inherent limitations, e.g., the over-coupling problem within MT structure owing to the employment of exponential moving average (EMA) mechanism, as well as the severe cognitive bias between two students of DS structure, both of which potentially lead to reduced efficacy, or even model collapse eventually. To mitigate these issues, a Decoupled Competitive Framework (DCF) is elaborated in this work, which utilizes a straightforward competition mechanism for the update of EMA, effectively decoupling students and teachers in a dynamical manner. In addition, the seamless exchange of invaluable and precise insights is facilitated among students, guaranteeing a better learning paradigm. The DCF introduced undergoes rigorous validation on three publicly accessible datasets, which encompass both 2D and 3D datasets. The results demonstrate the superiority of our method over previous cutting-edge competitors. Code will be available at https://github.com/Fly-away20/DCF.
Sinogram-Image Dual-Domain Network for Robust Metal Artifact Reduction in CT Image
Authors: Liu Chong , Huang Yuhan , Li Bo , Ding Hui
Published: October 2024

Computed tomography (CT) utilizes X-ray technology for internal body imaging. However, the presence of metal objects often results in artifacts due to their significant absorption and scattering of X-rays, thus obstructing lesion diagnosis, especially in the presence of multiple metals. Existing artifact reduction methods often suffer from deficiencies in completeness and preservation of fine detail. To address this limitation, we propose a novel sinogram and image dual-domain network. Specifically, in the sinogram domain, two enhancement modules are designed: one for extracting information from regions affected by metal traces, and the other for learning to restore the sinogram corresponding to these metal traces. Subsequently, utilizing filtered back projection (FBP), artifact removal images are reconstructed in the image domain. Quantitative and qualitative analyses of synthetic images show our framework’s superiority over conventional Metal Artifact Reduction (MAR) methods in both synthetic and clinical settings.
TwinDiffusion: Enhancing Coherence and Efficiency in Panoramic Image Generation with Diffusion Models
Authors: Zhou Teng , Tang Yongchuan
Published: October 2024

Diffusion models have emerged as effective tools for generating diverse and high-quality content. However, their capability in high-resolution image generation, particularly for panoramic images, still faces challenges such as visible seams and incoherent transitions. In this paper, we propose TwinDiffusion, an optimized framework designed to address these challenges through two key innovations: the Crop Fusion for quality enhancement and the Cross Sampling for efficiency optimization. We introduce a training-free optimizing stage to refine the similarity of adjacent image areas, as well as an interleaving sampling strategy to yield dynamic patches during the cropping process. A comprehensive evaluation is conducted to compare TwinDiffusion with the prior works, considering factors including coherence, fidelity, compatibility, and efficiency. The results demonstrate the superior performance of our approach in generating seamless and coherent panoramas, setting a new standard in quality and efficiency for panoramic image generation.
Solving PDEs on Point Clouds by Physics-Informed Learning with Graph Neural Networks
Authors: Hwang Rakhoon , Ryu Junseung , Park Seungtae , Hwang Hyung Ju
Published: October 2024

Scientific machine learning (SciML) explores the development of neural network models to approximate solutions to Partial Differential Equations (PDEs). However, there exists a significant research gap when computational domains are arbitrary manifolds, which are common in real-world scientific and engineering applications. The inherent challenge arises when calculating differential operators defined on curved surfaces, particularly in scenarios where surface parameterization is unavailable. In this paper, we present a neural network-based method for solving PDEs on surfaces described only by point clouds, without any other geometrical priors. Our method comprises two steps—local surface approximation based on graph neural networks and solving PDEs on point clouds. For surface reconstruction, our graph neural networks can be generalized based on the predictions of simple geometries during training to significantly more complicated surfaces for evaluation. The proposed approach demonstrates its capacity to learn geometric features from point cloud data without requiring external datasets, offers superior performance compared to benchmark models across various PDE types, and exhibits robustness in handling complex surfaces, non-uniform point distributions, and noise.
Anatomical Consistency Distillation and Inconsistency Synthesis for Brain Tumor Segmentation with Missing Modalities
Authors: Zhang Zheyu , Liu Xinzhao , Chen Zheng , Zhang Yueyi , Yue Huanjing , Ou Yunwei , Sun Xiaoyan
Published: October 2024

Multi-modal Magnetic Resonance Imaging (MRI) is imperative for accurate brain tumor segmentation, offering indispensable complementary information. Nonetheless, the absence of modalities poses significant challenges in achieving precise segmentation. Recognizing the shared anatomical structures between mono-modal and multi-modal representations, it is noteworthy that mono-modal images typically exhibit limited features in specific regions and tissues. In response to this, we present Anatomical Consistency Distillation and Inconsistency Synthesis (ACDIS), a novel framework designed to transfer anatomical structures from multi-modal to mono-modal representations and synthesize modality-specific features. ACDIS consists of two main components: Anatomical Consistency Distillation (ACD) and Modality Feature Synthesis Block (MFSB). ACD incorporates the Anatomical Feature Enhancement Block (AFEB), meticulously mining anatomical information. Simultaneously, Anatomical Consistency ConsTraints (ACCT) are employed to facilitate the consistent knowledge transfer, i.e., the richness of information and the similarity in anatomical structure, ensuring precise alignment of structural features across mono-modality and multi-modality. Complementarily, MFSB produces modality-specific features to rectify anatomical inconsistencies, thereby compensating for missing information in the segmented features. Through validation on the BraTS2018 and BraTS2020 datasets, ACDIS substantiates its efficacy in the segmentation of brain tumors with missing MRI modalities.
Unidirectional Cross-Modal Fusion for RGB-T Tracking
Authors: Guo Xiao , Li Hangfei , Zha Yufei , Zhang Peng
Published: October 2024

The key issue of RGB-T tracking is to obtain an effective multimodal representation of targets by utilizing complementary RGB and TIR modality information. Previous methods of template fusion or bidirectional search-template interaction potentially diminish the target representation, resulting from noise information of both templates and search regions. Meanwhile, the direct fusion of sole search features without interacting with templates cannot fully utilize target-relevant contextual information. To mitigate these issues, we present UCTrack, which fuses complementary multimodal search features conditioned on undisturbed RGB and TIR template features. Specifically, we design a Unidirectional Cross-modal Fusion (UCF) module to effectively minimize the influence of background noise on templates by pruning the unnecessary template-to-search cross-modal interaction and to mutually enhance RGB and TIR search features with target-relevant information through multimodal spatial fusion. Furthermore, this module is seamlessly integrated into different layers of a ViT backbone to facilitate feature extraction and cross-modal fusion for RGB-T tracking. Benefiting from the UCF module, UCTrack can effectively and accurately represent multimodal target features without unnecessary template-to-search interaction flow and direct template fusion, making the first proposal of unidirectional cross-modal fusion paradigm for RGB-T tracking to our best knowledge. Extensive experiments on three popular RGB-T tracking benchmarks demonstrate that our method achieves state-of-the-art performance.
IPA-NeRF: Illusory Poisoning Attack Against Neural Radiance Fields
Authors: Jiang Wenxiang , Zhang Hanwei , Zhao Shuo , Guo Zhongwen , Wang Hao
Published: October 2024

Neural Radiance Field (NeRF) represents a significant advancement in computer vision, offering implicit neural network-based scene representation and novel view synthesis capabilities. Its applications span diverse fields including robotics, urban mapping, autonomous navigation, virtual reality/augmented reality, etc., some of which are considered high-risk AI applications. However, despite its widespread adoption, the robustness and security of NeRF remain largely unexplored. In this study, we contribute to this area by introducing the Illusory Poisoning Attack against Neural Radiance Fields (IPA-NeRF). This attack involves embedding a hidden backdoor view into NeRF, allowing it to produce predetermined outputs, i.e. illusory, when presented with the specified backdoor view while maintaining normal performance with standard inputs. Our attack is specifically designed to deceive users or downstream models at a particular position while ensuring that any abnormalities in NeRF remain undetectable from other viewpoints. Experimental results demonstrate the effectiveness of our Illusory Poisoning Attack, successfully presenting the desired illusory on the specified viewpoint without impacting other views. Notably, we achieve this attack by introducing small perturbations solely to the training set. The code can be found at https://github.com/jiang-wenxiang/IPA-NeRF.
Sparse Convolutional Neural Network for Localization and Orientation Prediction and Application to Drone Control
Authors: Rodziewicz-Bielewicz Jan , Korzeń Marcin
Published: October 2024

Depth cameras used in visual feedback loops typically produce point clouds or their voxelized images. In the paper, we introduce an original sparse convolutional neural network structure tailored to work with 3D point clouds taken from RGB-D camera. The proposed structure is particularly effective for localization and orientation tasks and operates on the entire scene. We demonstrate its application by using the proposed neural network in a computer vision feedback system to track and control the drone indoors. The visual feedback loop measures the drone’s state and uses the information about its position and orientation to control the drone and allow it to fly along a given trajectory. In the experimental part we show the effectiveness of this approach to control the drone position in indoor spaces. We achieve trajectory-following accuracy comparable to that of the drone’s inbuilt controller’s ability to maintain a fixed position. We compare the proposed solution with a YOLO-based detector and sparse convolutional network to angle prediction operating on cropped and aligned drone point clouds.
MoSt-DSA: Modeling Motion and Structural Interactions for Direct Multi-Frame Interpolation in DSA Images
Authors: Xu Ziyang , Zhao Huangxuan , Cui Ziwei , Liu Wenyu , Zheng Chuansheng , Wang Xinggang
Published: October 2024

Artificial intelligence has become a crucial tool for medical image analysis. As an advanced cerebral angiography technique, Digital Subtraction Angiography (DSA) poses a challenge where the radiation dose to humans is proportional to the image count. By reducing images and using AI interpolation instead, the radiation can be cut significantly. However, DSA images present more complex motion and structural features than natural scenes, making interpolation more challenging. We propose MoSt-DSA, the first work that uses deep learning for DSA frame interpolation. Unlike natural scene Video Frame Interpolation (VFI) methods that extract unclear or coarse-grained features, we devise a general module that models motion and structural context interactions between frames in an efficient full convolution manner by adjusting optimal context range and transforming contexts into linear functions. Benefiting from this, MoSt-DSA is also the first method that directly achieves any number of interpolations at any time steps with just one forward pass during both training and testing. We conduct extensive comparisons with 7 representative VFI models for interpolating 1 to 3 frames, MoSt-DSA demonstrates robust results across 470 DSA image sequences (each typically 152 images), with average SSIM over 0.93, average PSNR over 38 (standard deviations of less than 0.030 and 3.6, respectively), comprehensively achieving state-of-the-art performance in accuracy, speed, visual effect, and memory usage. Our code is available at https://github.com/ZyoungXu/MoSt-DSA.
CorrAdaptor: Adaptive Local Context Learning for Correspondence Pruning
Authors: Zhu Wei , Liu Yicheng , He Yuping , Liao Tangfei , Zheng Kang , Xu Xiaoqiu , Wang Tao , Lu Tong
Published: October 2024

In the fields of computer vision and robotics, accurate pixel-level correspondences are essential for enabling advanced tasks such as structure-from-motion and simultaneous localization and mapping. Recent correspondence pruning methods usually focus on learning local consistency through k-nearest neighbors, which makes it difficult to capture robust context for each correspondence. We propose CorrAdaptor, a novel architecture that introduces a dual-branch structure capable of adaptively adjusting local contexts through both explicit and implicit local graph learning. Specifically, the explicit branch uses KNN-based graphs tailored for initial neighborhood identification, while the implicit branch leverages a learnable matrix to softly assign neighbors and adaptively expand the local context scope, significantly enhancing the model’s robustness and adaptability to complex image variations. Moreover, we design a motion injection module to integrate motion consistency into the network to suppress the impact of outliers and refine local context learning, resulting in substantial performance improvements. The experimental results on extensive correspondence-based tasks indicate that our CorrAdaptor achieves state-of-the-art performance both qualitatively and quantitatively.
MEFusion: Unsupervised Mutual Enhancement for Multimodal Image Fusion
Authors: Cao Yushe , Jiao Siwen , Sun Penghao , Peng Baoyun , Shi Dianxi , Shi Yuanchun
Published: October 2024

Image fusion aims to extract valuable information from each modality to create a fused image. Currently, state-of-the-art image fusion approaches tend to initially decompose each modality into distinct yet complementary features, and transfer beneficial information through carefully hand-crafted or learned fusion rules to the target. Nevertheless, previous approaches treat each modality in isolation before fusion, potentially under-utilising the complementary information available across modalities. In this word, we introduce a novel method called MEFusion that pioneers cross-modality mutual enhancement before feature decomposition. By harnessing the individual strengths of each modality, MEFusion elevates the overall quality and comprehensiveness of the fusion outcome. To facilitate a bidirectional enhancement for each feature across modalities, we have designed a pluggable co-attention mechanism that seamlessly integrates into a lightweight dual-path transformer. Furthermore, to enrich the details of each modality, we propose an unsupervised cross-modality mutual enhancement loss, which overcomes the limitations of requiring paired training data for enhancement tasks. Extensive experiments conducted on several benchmark datasets demonstrate the superiority of our proposed MEFusion method in terms of traditional fusion metrics and perceptual quality improvement of fused images.
Class-Aware Sample Weight Learning for Cross-Modal Unsupervised Domain Adaptation in Cross-User Wearable Human Activity Recognition
Authors: Wu Tong , Liu Yanbin , Yongchareon Sira
Published: October 2024

Existing unsupervised domain adaption approaches for cross-user Wearable Human Activity Recognition (WHAR) typically assume that users utilize the uni-modal sensor deployment configuration and cannot transfer across different sensor modalities. In this paper, we consider the more realistic cross-modal wearable human activity recognition setting to investigate the unsupervised domain adaptation task. This new context presents two formidable challenges: (1) how to alleviate modality heterogeneity across users, and (2) how to explore cross-modal domain correlation for better unsupervised domain adaptation. We propose a cross-modal unsupervised domain Adaptation model with Class-Aware Sample Weight Learning (CASWL-Adapt) to address both challenges. First, a spherical modality discriminator is designed to capture modal-specific discriminative features of each user during domain adaptation, thus achieving a reduction of sample variance caused by modal heterogeneity. Given a user-specific modal, modality-independent domain-invariant features can be efficiently generated by the well-developed modality discrimination loss and adversarial training. Second, a class-aware weight network is devised to calculate sample weights through classification loss and activity class similarity for each sample. Furthermore, the network leverage end-to-end learning and meta-optimization update rules to explore inter-domain correlations. Cross-modal activity classes are expected to adaptively implement different weighting schemes based on their intrinsic bias characteristics to select the most appropriate samples for domain knowledge transfer. We demonstrate that CASWL-Adapt achieves state-of-the-art results on three challenging benchmarks: Epic-Kitchens, Multimodal-EA and RealWorld, especially effective for new users of unseen modality.
Robust Monocular Depth Estimation in Adverse Weather Conditions by Unsupervised Domain Adaptation
Authors: Lee Jihui , Lai-Dang Quoc-Vinh , Sengar Neha , Har Dongsoo
Published: October 2024

Robust monocular depth estimation is essential for various applications relying on visual cues to understand the real world. To ensure robustness, unsupervised domain adaptation is widely used for monocular depth estimation. Despite recent advances, existing methods often struggle in outdoor environments due to adverse environmental conditions and limited datasets. Intentionally corrupted images obtained from real images captured in clear weather conditions for unsupervised domain adaptation often fail to accurately represent the complex characteristics of diverse environments, leading to unrealistic training data. From this viewpoint, simulation data offering more plausible representation of adverse weather conditions are used. However, it still presents drawbacks due to potentially degrading adaptation capabilities. To address the limitations of using simulation data, we propose a wild-condition pass filtering module that extracts wild-condition features and captures cross-domain relationships from both real and simulation datasets. This enables comprehensive learning of different conditions from each dataset and improved performance on real adversarial target images. The proposed method achieves a notable 22% improvement over the baseline on the Foggy Cityscapes dataset, highlighting the importance of employing realistic domain adaptation techniques to effectively address the challenges posed by adverse environmental conditions. The code is available at https://github.com/JH2-LEE/wide.
Reducing Texture Bias of Deep Neural Networks via Edge Enhancing Diffusion
Authors: Heinert Edgar , Rottmann Matthias , Maag Kira , Kahl Karsten
Published: October 2024

Convolutional neural networks (CNNs) for image processing tend to focus on localized texture patterns, commonly referred to as texture bias. While most of the previous works in the literature focus on the task of image classification, we go beyond this and study the texture bias of CNNs in semantic segmentation. In this work, we propose to train CNNs on pre-processed images with less texture to reduce the texture bias. Therein, the challenge is to suppress image texture while preserving shape information. To this end, we utilize edge enhancing diffusion (EED), an anisotropic image diffusion method initially introduced for image compression, to create texture reduced duplicates of existing datasets. Extensive numerical studies are performed with both CNNs and vision transformer models trained on original data and EED-processed data from the Cityscapes dataset and the CARLA driving simulator. We observe strong texture-dependence of CNNs and moderate texture-dependence of transformers. Training CNNs on EED-processed images enables the models to become completely ignorant with respect to texture, demonstrating resilience with respect to texture re-introduction to any degree. Additionally we analyze the performance reduction in depth on a level of connected components in the semantic segmentation and study the influence of EED pre-processing on domain generalization as well as adversarial robustness.
u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model
Authors: Xu Jinjin , Xu Liwu , Yang Yuzhe , Li Xiang , Wang Fanyi , Xie Yanchun , Huang Yi-Jie , Li Yaqian
Published: October 2024

Recent advancements in multi-modal large language models (MLLMs) have led to substantial improvements in visual understanding, primarily driven by sophisticated modality alignment strategies. However, predominant approaches prioritize global or regional comprehension, with less focus on fine-grained, pixel-level tasks. To address this gap, we introduce u-LLaVA, an innovative unifying multi-task framework that integrates pixel, regional, and global features to refine the perceptual faculties of MLLMs. We commence by leveraging an efficient modality alignment approach, harnessing both image and video datasets to bolster the model’s foundational understanding across diverse visual contexts. Subsequently, a joint instruction tuning method with task-specific projectors and decoders for end-to-end downstream training is presented. Furthermore, this work contributes a novel mask-based multi-task dataset comprising 277K samples, crafted to challenge and assess the fine-grained perception capabilities of MLLMs. The overall framework is simple, effective, and achieves state-of-the-art performance across multiple benchmarks. We make model, data, and code publicly accessible at https://github.com/OPPOMKLab/u-LLaVA.
Night-to-Day: Unpaired Image-to-Image Translation for Nighttime Pedestrian Detection
Authors: Althoupety Afnan , Wang Li-Yun , Feng Wu-Chi , Rekabdar Banafsheh
Published: October 2024

In this paper, we show that exploiting Generative Adversarial Networks (GANs) to transform nighttime images into daytime representation increases the robustness of pedestrian detection in low-light conditions. Our work aims at first learning the image translation to transfer the style from daytime images to nighttime images with unpaired GAN training. Second, we use our end-to-end trained GAN model to translate night images as a pre-processing step before feeding them into an object detector that is pre-trained on daytime images only. To demonstrate the effectiveness of our translation approach, we conducted experiments on two real-world pedestrian datasets using both one-stage and two-stage object detectors. Our results outperform the baseline in all experiments and show highly competitive detection performance compared with other GAN-based approaches while holding the most lightweight architecture. We believe that our approach is an effective pre-processing first step that helps in bridging the performance gap between day and night at no expense of re-training object detector networks with more night images.
MC-SORT: A Motion Correction-Based Framework for Long-Term Multiple Object Tracking
Authors: Li Xiangyu , Qin Yunchuan , Li Ruihui , Tan Guanghua , Tang Zhuo , Li Kenli
Published: October 2024

Long-term occlusion is one of the most formidable challenges in Multi-Object Tracking (MOT). The motion models of existing SORT-based trackers are unreliable in estimating the motion states of long-term occluded targets. This is mainly because as the occlusion period increases, the increases speed of estimation errors in the motion model increases faster. In practical applications, we believe that the estimation error of the tracker during long-term occlusion is mainly concentrated in the estimation error of the motion model on the velocity of the occluded target. In this work, we have demonstrated that in the long-term occlusion period, appropriately correcting the estimated values of the motion model on the target motion velocity and fully utilizing the temporal and attribute information of the target’s historical trajectory as calculation indicators of correlation are beneficial for improving the robustness of the tracker in long-term occlusion. We refer to our proposed motion correction-based framework as MC-SORT, which mainly consists of a Momentum Compensation Module (MCM) and a Backtracking Re-association (BRA) module. The former can correct the estimated value of the target’s motion state during long-term occlusion, the latter uses the temporal and attribute information of the target’s historical trajectory during long-term occlusion as correlation indicators to measure the degree of correlation between the target and trajectory. Our proposed MC-SORT has the characteristics of simplicity, online, real-time, and plug-and-play, particularly improving the robustness of the tracker in long-term occlusion. The extensive experimental results on the MOT17 and MOT20 datasets demonstrate the robustness and superiority of our framework.
Cliff: Leveraging Ambiguous Samples for Enhanced Test-Time Adaptation
Authors: Chen Xiao , Zhang Qihui , Wang Yan
Published: October 2024

Given the common scenario where a trained model confronts significant variations in data distributions different from the training data at test time, Test Time Adaptation (TTA) has emerged as a crucial field of study. Traditional methods in TTA have focused on filtering low-entropy samples to improve model performance, primarily through entropy minimization techniques. However, these approaches exhibit limitations as they often overlook the potential classes of high-entropy samples. This oversight can result in an inadequate utilization of available data, particularly under challenging conditions where model adaptability is critical. In contrast to conventional approaches, our work diverges from the sole emphasis on low-entropy samples by leveraging the rich information contained within ambiguous samples. We demonstrate that reliance solely on entropy minimization is detrimental when dealing with ambiguous samples. To address this, we introduce Cliff, a novel framework designed to learn from ambiguous samples effectively. Concretely, Cliff comprises two innovative components: Dynamic Recognition (DR) and Gap Raising Loss (GRL). DR proposes a method for identifying ambiguous samples and dynamically assigning weights to them, enhancing the model’s focus on potentially informative discrepancies. Whereas the proposed GRL, indeed theoretically proven to be beneficial to the model, guides the model in effectively distinguishing among potential classes by emphasizing the differences in their predictive probabilities. Extensive experiments conducted on CIFAR-10-C and CIFAR-100-C datasets demonstrate Cliff’s state-of-the-art performance. Our results show an average accuracy improvement of 20.24% and 21.12% over the direct use of source domain models on target domains, respectively.
Instruction Following with Goal-Conditioned Reinforcement Learning in Virtual Environments
Authors: Volovikova Zoya , Skrynnik Alexey , Kuderov Petr , Panov Aleksandr I.
Published: October 2024

In this study, we address the issue of enabling an artificial intelligence agent to execute complex language instructions within virtual environments. In our framework, we assume that these instructions involve intricate linguistic structures and multiple interdependent tasks that must be navigated successfully to achieve the desired outcomes. To effectively manage these complexities, we propose a hierarchical framework that combines the deep language comprehension of large language models with the adaptive action-execution capabilities of reinforcement learning agents: the language module (based on LLM) translates the language instruction into a high-level action plan, which is then executed by a pre-trained reinforcement learning agent.We have demonstrated the effectiveness of our approach in two different environments: in IGLU, where agents are instructed to build structures, and in Crafter, where agents perform tasks and interact with objects in the surrounding environment according to language commands.
Prompt-Based Domain Incremental Learning with Modular Classification Layer
Authors: Wang Boyu , Ma Yue , Qiu Qinru
Published: October 2024

Continual learning is referred to as machine learning model’s ability to learn a sequence of tasks or data over time without forgetting previously learned knowledge. In particular, we focus on domain incremental learning where the model trained on one domain, e.g., real-life photos, has to be augmented to accommodate data from new domains, e.g., cartoon images. Typical incremental learning relies on rehearsal-based methods, which store trained samples in a buffer and replay them during training alongside new data. This results in significant memory overhead and raise concerns about data privacy. Recently, prompt-based methods address these challenges and outperform them by utilizing the pre-trained Vision Transformer (ViT) and replacing the replay buffer with a prompt pool. However, existing prompt-based models fail to capture domain-specific knowledge and perform poorly in domain incremental learning. In this paper, we propose “domain prompt incremental learning via dynamic neural network”, which combines the advantages of architecture-based and prompt-based methods. Specifically, our framework maintains two types of prompt: a instance-level prompt that improves the model’s generalization ability is shared across all input samples; and a domain prompt that encodes domain-specific knowledge is assigned for each task. Furthermore, a separated classification head is trained for each domain so that the model has a pre-trained ViT and an ensemble of classification layers, one for each domain. The experimental results shows that our approach outperforms state-of-the-art methods by 2.3% in average on two domain incremental learning (DIL) benchmarks.
Unified Video and Image Representation for Boosted Video Face Forgery Detection
Authors: Liu Haotian , Pan Chenhui , Liu Yang , Zhao Guoying , Li Xiaobai
Published: October 2024

Face forgery detection is crucial in preserving the security and integrity of facial data amidst the rapid developments in face manipulation techniques and deep generative models. Existing methods for video face forgery detection typically assume that all frames in a forged video are manipulated, while identifying partially forged videos with only a subset of altered frames is still a challenge to be solved. To address this issue, we propose a novel framework, i.e., the UVIF, that utilizes additional annotated images to provide fine-grained supervision for detecting partial forgeries in videos. The UVIF integrates a unified encoder and a multi-task learning paradigm to model both facial videos and images for boosted video face forgery detection. A 2D backbone with temporal fusion modules is employed for the unified encoder. A pseudo labeling process is also designed for facial video frames to bridge the representation of individual video frames and static images. Extensive experiments on benchmark datasets demonstrate the effectiveness of our framework, outperforming state-of-the-art methods in detecting partially forged videos while introducing no additional computational overhead. Our code is available at https://github.com/haotianll/UVIF.
Recurrent Task Specialization Network for Segmentation-aided Vascular Landmarks Detection in Retinal Images
Authors: Hervella Álvaro S. , Rouco José , Novo Jorge , Sánchez Clara I. , Ortega Marcos
Published: October 2024

The detection of vessel crossings and bifurcations in eye fundus images plays an important role in numerous applications, including the diagnosis of ophthalmic and systemic diseases, biometric authentication, and retinal image registration. Nowadays, deep neural networks are successfully used for the detection of these vascular landmarks. However, existing approaches could be limited by the lack of understanding of the retinal anatomy and the intricate retinal vasculature. In this context, we propose Recurrent Task Specialization, a novel approach that performs a recurrent forward process with two forward passes through the same network, each of them specialized in a different task. We apply the proposed approach to the detection of vessel crossings and bifurcations in the retina via heatmap regression, using the segmentation of the retinal vasculature as the auxiliary task. To validate our proposal, we perform comparative experiments on two public datasets, including common alternatives to leverage auxiliary tasks, such as standard multi-task learning and transfer learning. The proposed approach outperforms existing alternatives and achieves the best results in the state-of-the-art for the detection of vessel crossings and bifurcations in retinal images. In this regard, our experiments demonstrate the potential of the proposed approach to improve the performance of deep neural networks in applications where adequate auxiliary tasks can be constructed.
StyleMamba: State Space Model for Efficient Text-Driven Image Style Transfer
Authors: Wang Zijia , Liu Zhi-Song
Published: October 2024

We present StyleMamba, an efficient image style transfer framework that translates text prompts into corresponding visual styles while preserving the content integrity of the original images. Existing text-guided stylization requires hundreds of training iterations and takes a lot of computing resources. To speed up the process, we propose a conditional State Space Model for Efficient Text-driven Image Style Transfer, dubbed StyleMamba, that sequentially aligns the image features to the target text prompts. To enhance the local and global style consistency between text and image, we propose masked and second-order directional losses to optimize the stylization direction to significantly reduce the training iterations by 5× and the inference time by 3×. Extensive experiments and qualitative evaluation confirm the robust and superior stylization performance of our methods compared to the existing baselines. Full code of this paper can be found in https://github.com/OliverDOU776/StyleMamba.