GenAI4ELab/papercli-papers huggingface dataset:
Domain Shifting: A Generalized Solution for Heterogeneous Cross-Modality Person Re-Identification
Authors: Yan Jiang; Xu Cheng; Hao Yu; Xingyu Liu; Haoyu Chen; Guoying Zhao
Venue: ECCV 2024 (main)

Abstract: "Cross-modality person re-identification (ReID) is a challenging task that aims to match cross-modality pedestrian images across multiple camera views. Existing methods are tailored to specific tasks and perform well for visible-infrared or visible-sketch ReID. However, the performance exhibits a notable decline when the same method is utilized for multiple cross-modality ReIDs, limiting its generalization and applicability. To address this issue, we propose a generalized domain shifting method (DNS) for cross-modality ReID, which can address the generalization and perform well in both visible-infrared and visible-sketch modalities. Specifically, we propose the heterogeneous space shifting and common space shifting modules to augment specific and shared representations in heterogeneous space and common space, respectively, thereby regulating the model to learn the consistency between modalities. Further, a domain alignment loss is developed to alleviate the cross-modality discrepancies by aligning the patterns across modalities. In addition, a domain distillation loss is designed to distill identity-invariant knowledge by learning the distribution of different modalities. Extensive experiments on two cross-modality ReID tasks (, visible-infrared ReID, visible-sketch ReID) demonstrate that the proposed method outperforms the state-of-the-art methods by a large margin."
Self-Supervised Video Desmoking for Laparoscopic Surgery
Authors: Renlong Wu; Zhilu Zhang; Shuohao Zhang; Longfei Gou; Haobin Chen; Lei Zhang; Hao Chen; Wangmeng Zuo
Venue: ECCV 2024 (main)

Abstract: "Due to the difficulty of collecting real paired data, most existing desmoking methods train the models by synthesizing smoke, generalizing poorly to real surgical scenarios. Although a few works have explored single-image real-world desmoking in unpaired learning manners, they still encounter challenges in handling dense smoke. In this work, we address these issues together by introducing the self-supervised surgery video desmoking (SelfSVD). On the one hand, we observe that the frame captured before the activation of high-energy devices is generally clear (named pre-smoke frame, PS frame), thus it can serve as supervision for other smoky frames, making real-world self-supervised video desmoking practically feasible. On the other hand, in order to enhance the desmoking performance, we further feed the valuable information from PS frame into models, where a masking strategy and a regularization term are presented to avoid trivial solutions. In addition, we construct a real surgery video dataset for desmoking, which covers a variety of smoky scenes. Extensive experiments on the dataset show that our SelfSVD can remove smoke more effectively and efficiently while recovering more photo-realistic details than the state-of-the-art methods. The dataset, codes, and pre-trained models are available at https: //github.com/ZcsrenlongZ/SelfSVD."
Removing Rows and Columns of Tokens in Vision Transformer enables Faster Dense Prediction without Retraining
Authors: Diwei Su; cheng fei; Jianxu Luo
Venue: ECCV 2024 (main)

Abstract: "In recent years, vision transformers based on self-attention mechanisms have demonstrated remarkable abilities in various tasks such as natural language processing, computer vision (CV), and multimodal applications. However, due to the high computational costs and the structural nature of images, the application of transformers to CV tasks faces challenges, particularly when handling ultra-high-resolution images. Recently, several token reduction methods have been proposed to improve the computational efficiency of transformers by reducing the number of tokens without the need for retraining. These methods primarily involve fusion based on matching or clustering. The former exhibits faster speed but suffers more accuracy loss compared to the latter. In this work, we propose a simple matching-based fusion method called Token Adapter, which achieves comparable accuracy to the clustering-based fusion method with faster speed and demonstrates higher potential in terms of robustness. Our method was applied to Segmenter, MaskDINO and SWAG, exhibiting promising performance on four tasks, including semantic segmentation, instance segmentation, panoptic segmentation, and image classification. Specifically, our method can be applied to Segmenter on ADE20k, providing 41% frames per second (FPS) acceleration while maintaining full performance without retraining or fine-tuning off-the-shelf weights. Our code will be released at https://github.com/MilknoCandy/ Token-Adapter."
Continuity Preserving Online CenterLine Graph Learning
Authors: Yunhui Han; Kun Yu; Zhiwei Li
Venue: ECCV 2024 (main)

Abstract: "Lane topology, which is usually modeled by a centerline graph, is essential for high-level autonomous driving. For a high-quality graph, both topology connectivity and spatial continuity of centerline segments are critical. However, most of existing approaches pay more attention to connectivity while neglect the continuity. Such kind of centerline graph usually cause problem to planning of autonomous driving. To overcome this problem, we present an end-to-end network, CGNet, with three key modules: 1) Junction Aware Query Enhancement module, which provides positional prior to accurately predict junction points; 2) Bézier Space Connection module, which enforces continuity constraints on any two topologically connected segments in a Bézier space; 3) Iterative Topology Refinement module, which is a graph-based network with memory to iteratively refine the predicted topological connectivity. CGNet achieves state-of-the-art performance on both nuScenes and Argoverse2 datasets. Our code is available at https://github.com/XiaoMi/CGNet."
Decomposition of Neural Discrete Representations for Large-Scale 3D Mapping
Authors: Minseong Park; Suhan Woo; Euntai Kim
Venue: ECCV 2024 (main)

Abstract: "Learning efficient representations of local features is a key challenge in feature volume-based 3D neural mapping, especially in large-scale environments. In this paper, we introduce Decomposition-based Neural Mapping (DNMap), a storage-efficient large-scale 3D mapping method that employs a discrete representation based on a decomposition strategy. This decomposition strategy aims to efficiently capture repetitive and representative patterns of shapes by decomposing each discrete embedding into component vectors that are shared across the embedding space. Our DNMap optimizes a set of component vectors, rather than entire discrete embeddings, and learns composition rather than indexing the discrete embeddings. Furthermore, to complement the mapping quality, we additionally learn low-resolution continuous embeddings that require tiny storage space. By combining these representations with a shallow neural network and an efficient octree-based feature volume, our DNMap successfully approximates signed distance functions and compresses the feature volume while preserving mapping quality. Our source code is available at https://github.com/minseong-p/dnmap."
MirrorGaussian: Reflecting 3D Gaussians for Reconstructing Mirror Reflections
Authors: Jiayue Liu; Xiao Tang; Freeman Cheng; Zihao Yang; Zhihao Li; Jianzhuang Liu; Yi Huang; Jiaqi Lin; Shiyong Liu; Xiaofei Wu; Songcen Xu; Chun Yuan
Venue: ECCV 2024 (main)

Abstract: "3D Gaussian Splatting showcases notable advancements in photo-realistic and real-time novel view synthesis. However, it faces challenges in modeling mirror reflections, which exhibit substantial appearance variations from different viewpoints. To tackle this problem, we present MirrorGaussian, the first method for mirror scene reconstruction with real-time rendering based on 3D Gaussian Splatting. The key insight is grounded on the mirror symmetry between the real-world space and the virtual mirror space. We introduce an intuitive dual-rendering strategy that enables differentiable rasterization of both the real-world 3D Gaussians and the mirrored counterpart obtained by reflecting the former about the mirror plane. All 3D Gaussians are jointly optimized with the mirror plane in an end-to-end framework. MirrorGaussian achieves high-quality and real-time rendering in scenes with mirrors, empowering scene editing like adding new mirrors and objects. Comprehensive experiments on multiple datasets demonstrate that our approach significantly outperforms existing methods, achieving state-of-the-art results. Project page: https://mirror-gaussian.github.io/."
Leveraging Representations from Intermediate Encoder-blocks for Synthetic Image Detection
Authors: Christos Koutlis; Symeon Papadopoulos
Venue: ECCV 2024 (main)

Abstract: "The recently developed and publicly available synthetic image generation methods and services make it possible to create extremely realistic imagery on demand, raising great risks for the integrity and safety of online information. State-of-the-art Synthetic Image Detection (SID) research has led to strong evidence on the advantages of feature extraction from foundation models. However, such extracted features mostly encapsulate high-level visual semantics instead of fine-grained details, which are more important for the SID task. On the contrary, shallow layers encode low-level visual information. In this work, we leverage the image representations extracted by intermediate Transformer blocks of CLIP’s image-encoder via a lightweight network that maps them to a learnable forgery-aware vector space capable of generalizing exceptionally well. We also employ a trainable module to incorporate the importance of each Transformer block to the final prediction. Our method is compared against the state-of-the-art by evaluating it on 20 test datasets and exhibits an average +10.6% absolute performance improvement. Notably, the best performing models require just a single epoch for training (∼8 minutes). Code available at https://github.com/mever-team/rine."
Exploring Vulnerabilities in Spiking Neural Networks: Direct Adversarial Attacks on Raw Event Data
Authors: Yanmeng Yao; Xiaohan Zhao; Bin Gu
Venue: ECCV 2024 (main)

Abstract: "In the field of computer vision, event-based Dynamic Vision Sensors (DVSs) have emerged as a significant complement to traditional pixel-based imaging due to their low power consumption and high temporal resolution. These sensors, particularly when combined with Spiking Neural Networks (SNNs), offer a promising direction for energy-efficient and fast-reacting vision systems. Typically, DVS data are converted into grid-based formats for processing with SNNs, with this transformation process often being an opaque step in the pipeline. As a result, the grid representation becomes an intermediate yet inaccessible stage during the implementation of attacks, highlighting the importance of attacking raw event data. Existing attack methodologies predominantly target grid-based representations, hindered by the complexity of three-valued optimization and the broad optimization space associated with raw event data. Our study addresses this gap by introducing a novel adversarial attack approach that directly targets raw event data. We tackle the inherent challenges of three-valued optimization and the need to preserve data sparsity through a strategic amalgamation of methods: 1) Treating Discrete Event Values as Probabilistic Samples: This allows for continuous optimization by considering discrete event values as probabilistic space samples. 2) Focusing on Specific Event Positions: We prioritize specific event positions that merge original data with additional target label data, enhancing attack precision. 3) Employing a Sparsity Norm: To retain the original data’s sparsity, a sparsity norm is utilized, ensuring the adversarial data’s comparability. Our empirical findings demonstrate the effectiveness of our combined approach, achieving noteworthy success in targeted attacks and highlighting vulnerabilities in models based on raw event data."
HSR: Holistic 3D Human-Scene Reconstruction from Monocular Videos
Authors: Lixin Xue; Chen Guo; Chengwei Zheng; Fangjinhua Wang; Tianjian Jiang; Hsuan-I Ho; Manuel Kaufmann; Jie Song; Otmar Hilliges
Venue: ECCV 2024 (main)

Abstract: "An overarching goal for computer-aided perception systems is the holistic understanding of the human-centric 3D world, including faithful reconstructions of humans, scenes, and their global spatial relationships. While recent progress in monocular 3D reconstruction has been made for footage of either humans or scenes alone, the joint reconstruction of both humans and scenes, along with their global spatial information, remains an unsolved challenge. To address this, we introduce a novel and unified framework that simultaneously achieves temporally and spatially coherent 3D reconstruction of static scenes with dynamic humans from monocular RGB videos. Specifically, we parameterize temporally consistent canonical human models and static scene representations using two neural fields in a shared 3D space. Additionally, we develop a global optimization framework that considers physical constraints imposed by potential human-scene interpenetration and occlusion. Compared to separate reconstructions, our framework enables detailed and holistic geometry reconstructions of both humans and scenes. Furthermore, we introduce a synthetic dataset for quantitative evaluations. Extensive experiments and ablation studies on both real-world and synthetic videos demonstrate the efficacy of our framework in monocular human-scene reconstruction. Code and data are publicly available on our project page."
Online Video Quality Enhancement with Spatial-Temporal Look-up Tables
Authors: Zefan Qu; Xinyang Jiang; Yifan Yang; Dongsheng Li; Cairong Zhao
Venue: ECCV 2024 (main)

Abstract: "Low latency rates are crucial for online video-based applications, such as video conferencing and cloud gaming, which make improving video quality in online scenarios increasingly important. However, existing quality enhancement methods are limited by slow inference speed and the requirement for temporal information contained in future frames, making it challenging to deploy them directly in online tasks. In this paper, we propose a novel method, STLVQE, specifically designed to address the rarely studied online video quality enhancement (Online-VQE) problem. Our STLVQE designs a new VQE framework which contains a Module-Agnostic Feature Extractor that greatly reduces the redundant computations and redesign the propagation, alignment, and enhancement module of the network. A Spatial-Temporal Look-up Tables (STL) is proposed, which extracts spatial-temporal information in videos while saving substantial inference time. To the best of our knowledge, we are the first to exploit the LUT structure to extract temporal information in video tasks. Extensive experiments on the MFQE 2.0 dataset demonstrate our STLVQE achieves a satisfactory performance-speed trade-off."
PARIS3D: Reasoning-based 3D Part Segmentation Using Large Multimodal Model
Authors: Amrin Kareem; Jean Lahoud; Hisham Cholakkal
Venue: ECCV 2024 (main)

Abstract: "Recent advancements in 3D perception systems have significantly improved their ability to perform visual recognition tasks such as segmentation. However, these systems still heavily rely on explicit human instruction to identify target objects or categories, lacking the capability to actively reason and comprehend implicit user intentions. We introduce a novel segmentation task known as reasoning part segmentation for 3D objects, aiming to output a segmentation mask based on complex and implicit textual queries about specific parts of a 3D object. To facilitate evaluation and benchmarking, we present a large 3D dataset comprising over 60k instructions paired with corresponding ground-truth part segmentation annotations specifically curated for reasoning-based 3D part segmentation. We propose a model that is capable of segmenting parts of 3D objects based on implicit textual queries and generating natural language explanations corresponding to 3D object segmentation requests. Experiments show that our method achieves competitive performance to models that use explicit queries, with the additional abilities to identify part concepts, reason about them, and complement them with world knowledge. Our source code, dataset, and trained models are accessible at this url."
Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance
Authors: Donghoon Ahn; Hyoungwon Cho; Jaewon Min; Jungwoo Kim; Wooseok Jang; SeonHwa Kim; Hyun Hee Park; Kyong Hwan Jin; Seungryong Kim
Venue: ECCV 2024 (main)

Abstract: "Recent studies have demonstrated that diffusion models can generate high-quality samples, but their quality heavily depends on sampling guidance techniques, such as classifier guidance (CG) and classifier-free guidance (CFG). These techniques are often not applicable in unconditional generation or various downstream tasks such as the solving inverse problems. In this paper, we propose novel sampling guidance, called Perturbed-Attention Guidance (PAG), which improves diffusion sample quality across both unconditional and conditional settings, achieving this without requiring additional training or the integration of external modules. PAG progressively enhances the structure of samples throughout the denoising process by generating intermediate samples with degraded structures and guiding the denoising process away from these degraded samples. These degraded samples are created by substituting selected self-attention maps in the diffusion U-Net, which capture structural information between image patches, with an identity matrix. In both ADM and Stable Diffusion, PAG surprisingly improves sample quality in conditional and even unconditional generation. Moreover, PAG significantly enhances baseline performance in various downstream tasks where existing guidance methods such as CG or CFG cannot be fully utilized, including ControlNet with empty prompts and solving inverse problems such as inpainting and deblurring. To the best of our knowledge, this is the first approach to apply guidance in solving inverse problems using diffusion models."
Localization and Expansion: A Decoupled Framework for Point Cloud Few-shot Semantic Segmentation
Authors: Zhaoyang Li; Yuan Wang; Wangkai Li; Rui Sun; Tianzhu Zhang
Venue: ECCV 2024 (main)

Abstract: "Point cloud few-shot semantic segmentation (PC-FSS) aims to segment targets of novel categories in a given query point cloud with only a few annotated support samples. The current top-performing prototypical learning methods employ prototypes originating from support samples to direct the classification of query points. However, the inherent fragility of point-level matching and the prevalent intra-class diversity pose great challenges to this cross-instance matching paradigm, leading to erroneous background activations or incomplete target excavation. In this work, we propose a simple yet effective framework in the spirit of Decoupled Localization and Expansion (DLE). The proposed DLE, including a structural localization module (SLM) and a self-expansion module (SEM), enjoys several merits. First, structural information is injected into the matching process through the agent-level correlation in SLM, and the confident target region can thus be precisely located. Second, more reliable intra-object similarity is harnessed in SEM to derive the complete target, and the conservative expansion strategy is introduced to reasonably constrain the expansion. Extensive experiments on two challenging benchmarks under different settings demonstrate that DLE outperforms previous state-of-the-art approaches by large margins."
Think before Placement: Common Sense Enhanced Transformer for Object Placement
Authors: Yaxuan Qin; Jiayu Xu; Ruiping Wang; Xilin Chen
Venue: ECCV 2024 (main)

Abstract: "Object placement is a task to insert a foreground object into a background scene at a suitable position and size. Existing methods mainly focus on extracting better visual features, while neglecting common sense about the objects and background. It leads to semantically unrealistic object positions. In this paper, we introduce Think Before Placement, a novel framework that effectively combines the implicit and explicit knowledge to generate placements that are both visually coherent and contextually appropriate. Specifically, we first adopt a large multi-modal model to generate a descriptive caption that identifies an appropriate position in the background for placing foreground object(Think ), then output proper position and size of the object (Place). The caption serves as an explicit semantic guidance for the subsequent placement of objects. Using this framework, we implement our model named CSENet, which outperforms baseline methods on the OPA dataset in extensive experiments. Further, we establish the OPAZ dataset to evaluate the zero-shot transfer capabilities of CSENet, where it also shows impressive performance across different foreground objects and scenes."
Oulu Remote-photoplethysmography Physical Domain Attacks Database (ORPDAD)
Authors: Marko Savic; Guoying Zhao
Venue: ECCV 2024 (main)

Abstract: "Remote photoplethysmography (rPPG) is an emerging technology that can detect the pulse rate remotely from face videos. However, it is easily influenced by the recording environment, as robustness to noise is still an open problem. This vulnerability can therefore be exploited to inject fake signals or impair predictions physically. In this study we propose the first dataset containing a wide set of physical domain attack scenarios divided in three categories (illumination, movement, concealment) that directly target the main weaknesses of rPPG. We propose the Oulu rPPG Physical Domain Attacks Database (ORPDAD) as a benchmark for evaluation of robustness to physical attacks. We perform extensive experiments on conventional hand-crafted and deep learning (end-to-end, non-end-to-end, CNN, transformer, self-supervised) methods and study their susceptibility to the attacks. We conclude by discussing the most critical vulnerabilities discovered and stress the importance of designing more secure solutions. Our code and instructions for requesting the dataset (with trained models) are available at: https://github.com/marukosan93/ORPDAD/."
Leveraging Imperfect Restoration for Data Availability Attack
Authors: YI HUANG; Jeremy Styborski; Mingzhi Lyu; Fan Wang; Wai-Kin Adams Kong
Venue: ECCV 2024 (main)

Abstract: "The abundance of online data is at risk of unauthorized usage in training deep learning models. To counter this, various Data Availability Attacks (DAAs) have been devised to make data unlearnable for such models by subtly perturbing the training data. However, existing attacks often excel against either Supervised Learning (SL) or Self-Supervised Learning (SSL) scenarios. Among these, a model-free approach that generates a Convolution-based Unlearnable Dataset (CUDA) stands out as the most robust DAA across both SSL and SL. Nonetheless, CUDA’s effectiveness against SSL is underwhelming and it faces a severe trade-off between image quality and its poisoning effect. In this paper, we conduct a theoretical analysis of CUDA, uncovering the sub-optimal gradients it introduces and elucidating the strategy it employs to induce class-wise bias for data poisoning. Building on this, we propose a novel poisoning method named Imperfect Restoration Poisoning (IRP), aiming to preserve high image quality while achieving strong poisoning effects. Through extensive comparisons of IRP with eight baselines across SL and SSL, coupled with evaluations alongside five representative defense methods, we showcase the superiority of IRP. Code: https://github.com/ lyumingzhi/IRP"