GenAI4ELab/papercli-papers huggingface dataset:
Exploring Phrase-Level Grounding with Text-to-Image Diffusion Model
Authors: Danni Yang; Ruohan Dong; Jiayi Ji; Yiwei Ma; Haowei Wang; Xiaoshuai Sun; Rongrong Ji
Venue: ECCV 2024 (main)

Abstract: "Recently, diffusion models have increasingly demonstrated their capabilities in vision understanding. By leveraging prompt-based learning to construct sentences, these models have shown proficiency in classification and visual grounding tasks. However, existing approaches primarily showcase their ability to perform sentence-level localization, leaving the potential for leveraging contextual information for phrase-level understanding largely unexplored. In this paper, we utilize Panoptic Narrative Grounding (PNG) as a proxy task to investigate this capability further. PNG aims to segment object instances mentioned by multiple noun phrases within a given narrative text. Specifically, we introduce the DiffPNG framework, a straightforward yet effective approach that fully capitalizes on the diffusion’s architecture for segmentation by decomposing the process into a sequence of localization, segmentation, and refinement steps. The framework initially identifies anchor points using cross-attention mechanisms and subsequently performs segmentation with self-attention to achieve zero-shot PNG. Moreover, we introduce a refinement module based on SAM to enhance the quality of the segmentation masks. Our extensive experiments on the PNG dataset demonstrate that DiffPNG achieves strong performance in the zero-shot PNG task setting, conclusively proving the diffusion model’s capability for context-aware, phrase-level understanding. Source code is available at https://github.com/nini0919/DiffPNG."
FocusDiffuser: Perceiving Local Disparities for Camouflaged Object Detection
Authors: Jianwei Zhao; Xin Li; Fan Yang; Qiang Zhai; Ao Luo; Zhicheng Jiao; Hong Cheng
Venue: ECCV 2024 (main)

Abstract: "Detecting objects seamlessly blended into their surroundings represents a complex task for both human cognitive capabilities and advanced artificial intelligence algorithms. Currently, the majority of methodologies for detecting camouflaged objects mainly focus on utilizing discriminative models with various unique designs. However, it has been observed that generative models, such as Stable Diffusion, possess stronger capabilities for understanding various objects in complex environments; Yet their potential for the cognition and detection of camouflaged objects has not been extensively explored. In this study, we present a novel denoising diffusion model, namely FocusDiffuser, to investigate how generative models can enhance the detection and interpretation of camouflaged objects. We believe that the secret to spotting camouflaged objects lies in catching the subtle nuances in details. Consequently, our FocusDiffuser innovatively integrates specialized enhancements, notably the Boundary-Driven LookUp (BDLU) module and Cyclic Positioning (CP) module, to elevate standard diffusion models, significantly boosting the detail-oriented analytical capabilities. Our experiments demonstrate that FocusDiffuser, from a generative perspective, effectively addresses the challenge of camouflaged object detection, surpassing leading models on benchmarks like CAMO, COD10K and NC4K. Code and pre-trained models are available at https://github.com/JWZhao-uestc/FocusDiffuser."
Improving image synthesis with diffusion-negative sampling
Authors: Alakh Desai; Nuno Vasconcelos
Venue: ECCV 2024 (main)

Abstract: "For image generation with diffusion models (DMs), a negative prompt n can be used to complement the text prompt p, helping define properties not desired in the synthesized image. While this improves prompt adherence and image quality, finding good negative prompts is challenging. We argue that this is due to a semantic gap between humans and DMs, which makes good negative prompts for DMs appear unintuitive to humans. To bridge this gap, we propose a new diffusion-negative prompting () strategy. is based on a new procedure to sample images that are least compliant with p under the distribution of the DM, denoted as diffusion-negative sampling (). Given p, one such image is sampled, which is then translated into natural language by the user or a captioning model, to produce the negative prompt n∗ . The pair (p, n∗ ) is finally used to prompt the DM. is straightforward to implement and requires no training. Experiments and human evaluations show that performs well both quantitatively and qualitatively and can be easily combined with several DM variants."
AvatarPose: Avatar-guided 3D Pose Estimation of Close Human Interaction from Sparse Multi-view Videos
Authors: Feichi Lu; Zijian Dong; Jie Song; Otmar Hilliges
Venue: ECCV 2024 (main)

Abstract: "Despite progress in human motion capture, existing multi-view methods often face challenges in estimating the 3D pose and shape of multiple closely interacting people. This difficulty arises from reliance on accurate 2D joint estimations, which are hard to obtain due to occlusions and body contact when people are in close interaction. To address this, we propose a novel method leveraging the personalized implicit neural avatar of each individual as a prior, which significantly improves the robustness and precision of this challenging pose estimation task. Concretely, the avatars are efficiently reconstructed via layered volume rendering from sparse multi-view videos. The reconstructed avatar prior allows for the direct optimization of 3D poses based on color and silhouette rendering loss, bypassing the issues associated with noisy 2D detections. To handle interpenetration, we propose a collision loss on the overlapping shape regions of avatars to add penetration constraints. Moreover, both 3D poses and avatars are optimized in an alternating manner. Our experimental results demonstrate state-of-the-art performance on several public datasets."
FedVAD: Enhancing Federated Video Anomaly Detection with GPT-Driven Semantic Distillation
Authors: Fan Qi; Ruijie Pan; Huaiwen Zhang; Changsheng Xu
Venue: ECCV 2024 (main)

Abstract: "The imperative for smart surveillance systems to robustly detect anomalies poses a unique challenge given the sensitivity of visual data and privacy concerns. We propose a novel Federated Learning framework for Video Anomaly Detection that operates under the constraints of data heterogeneity and privacy preservation. We utilize Federated Visual Consistency Clustering to group clients on the server side. Further innovation is realized with an Adaptive Semantic-Enhanced Distillation strategy that infuses public video knowledge into our framework. During this process, Large Language Models are utilized for semantic generation and calibration of public videos. These video-text pairs are then used to fine-tune a multimodal network, which serves as a teacher in updating the global model. This approach not only refines video representations but also increases sensitivity to anomalous events. Our extensive evaluations showcase FedVAD’s proficiency in boosting unsupervised and weakly supervised anomaly detection, rivaling centralized training paradigms while preserving privacy. The code will be made available publicly at https://github.com/Eurekaer/FedVAD."
SignGen: End-to-End Sign Language Video Generation with Latent Diffusion
Authors: Fan Qi; Yu Duan; Changsheng Xu; Huaiwen Zhang
Venue: ECCV 2024 (main)

Abstract: "The seamless transformation of textual input into natural and expressive sign language holds profound societal significance. Sign language is not solely about hand gestures. It encompasses vital facial expressions and mouth movements essential for nuanced communication. Achieving both semantic precision and emotional resonance in text-to-sign language translation is of paramount importance. Our work pioneers direct end-to-end translation of text into sign language videos, encompassing a realistic representation of the entire body and facial expressions. We go beyond traditional diffusion models by tailoring the multi-modal conditions for sign language videos. Additionally, our modified motion-aware sign generation framework enhances alignment between text and visual cues in sign language, further improving the quality of the generated sign language videos. Extensive experiments show that our approach significantly outperforms the state-of-the-art approaches in terms of semantic consistency, naturalness, and expressiveness, presenting benchmark quantitative results on the RWTH-2014, RWTH-2014-T, WLASL, CSL-Daily, and AUTSL. Our code is available at https://github.com/mingtiannihao/SignGen."
"Idling Neurons, Appropriately Lenient Workload During Fine-tuning Leads to Better Generalization"
Authors: Hongjing Niu; Hanting Li; Bin Li; Feng Zhao
Venue: ECCV 2024 (main)

Abstract: "Pre-training on large-scale datasets has become a fundamental method for training deep neural networks. Pre-training provides a better set of parameters than random initialization, which reduces the training cost of deep neural networks on the target task. In addition, pre-training also provides a large number of feature representations, which may help improve generalization capabilities. However, this potential advantage has not received enough attention and has been buried by rough fine-tuning. Based on some exploratory experiments, this paper rethinks the fine-tuning process and gives a new perspective on understanding fine-tuning. Moreover, this paper proposes some plug-and-play fine-tuning strategies as alternatives for simple fine-tuning. These fine-tuning strategies all preserve pre-trained features better by creating idling of some neurons, leading to better generalization."
Diffusion Prior-Based Amortized Variational Inference for Noisy Inverse Problems
Authors: Sojin Lee; Dogyun Park; Inho Kong; Hyunwoo J. Kim
Venue: ECCV 2024 (main)

Abstract: "Recent studies on inverse problems have proposed posterior samplers that leverage the pre-trained diffusion models as powerful priors. These attempts have paved the way for using diffusion models in a wide range of inverse problems. However, the existing methods entail computationally demanding iterative sampling procedures and optimize a separate solution for each measurement, which leads to limited scalability and lack of generalization capability across unseen samples. To address these limitations, we propose a novel approach, Diffusion prior-based Amortized Variational Inference (DAVI) that solves inverse problems with a diffusion prior from an amortized variational inference perspective. Specifically, instead of separate measurement-wise optimization, our amortized inference learns a function that directly maps measurements to the implicit posterior distributions of corresponding clean data, enabling a single-step posterior sampling even for unseen measurements. Extensive experiments on image restoration tasks, , Gaussian deblur, 4× super-resolution, and box inpainting with two benchmark datasets, demonstrate our approach’s superior performance over strong baselines. Code is available at https://github.com/mlvlab/DAVI."
The Gaussian Discriminant Variational Autoencoder (GdVAE): A Self-Explainable Model with Counterfactual Explanations
Authors: Anselm Haselhoff; Kevin Trelenberg; Fabian Küppers; Jonas Schneider
Venue: ECCV 2024 (main)

Abstract: "Visual counterfactual explanation (CF) methods modify image concepts, , shape, to change a prediction to a predefined outcome while closely resembling the original query image. Unlike self-explainable models (SEMs) and heatmap techniques, they grant users the ability to examine hypothetical ”what-if” scenarios. Previous CF methods either entail post-hoc training, limiting the balance between transparency and CF quality, or demand optimization during inference. To bridge the gap between transparent SEMs and CF methods, we introduce the GdVAE, a self-explainable model based on a conditional variational autoencoder (CVAE), featuring a Gaussian discriminant analysis (GDA) classifier and integrated CF explanations. Full transparency is achieved through a generative classifier that leverages class-specific prototypes for the downstream task and a closed-form solution for CFs in the latent space. The consistency of CFs is improved by regularizing the latent space with the explainer function. Extensive comparisons with existing approaches affirm the effectiveness of our method in producing high-quality CF explanations while preserving transparency. Code and models are public."
Accelerating Image Generation with Sub-path Linear Approximation Model
Authors: Chen Xu; Tianhui Song; Weixin Feng; Xubin Li; Tiezheng Ge; Bo Zheng; Limin Wang
Venue: ECCV 2024 (main)

Abstract: "Diffusion models have significantly advanced the state of the art in image, audio, and video generation tasks. However, their applications in practical scenarios are hindered by slow inference speed. Drawing inspiration from the consistency models, we propose the Sub-Path Linear Approximation Model (SPLAM), which can accelerate diffusion models while maintaining high-quality image generation. SPLAM treats the PF-ODE trajectory as a series of PF-ODE sub-paths divided by sampled points, and harnesses sub-path linear (SL) ODEs to form a progressive and continuous error estimation along each individual PF-ODE sub-path. The optimization on such SL-ODEs allows SPLAM to construct denoising mapping with smaller cumulative approximated error. An efficient distillation method is also developed to facilitate the incorporation of pre-trained diffusion models, such as latent diffusion models. The extensive experimental results demonstrate SPLAM achieves remarkable training efficiency, requiring only 6 A100 GPU days to produce a high-quality generative model capable of 2 to 4-step generation. Comprehensive evaluations on LAION, MS COCO 2014, and MS COCO 2017 datasets also illustrate that SPLAM surpasses the existing acceleration methods in few-step generation tasks, achieving state-of-the-art performance both on FID and the quality of the generated images."
Safe-CLIP: Removing NSFW Concepts from Vision-and-Language Models
Authors: Samuele Poppi; Tobia Poppi; Federico Cocchi; Marcella Cornia; Lorenzo Baraldi; Rita Cucchiara
Venue: ECCV 2024 (main)

Abstract: "Large-scale vision-and-language models, such as CLIP, are typically trained on web-scale data, which can introduce inappropriate content and lead to the development of unsafe and biased behavior. This, in turn, hampers their applicability in sensitive and trustworthy contexts and could raise significant concerns in their adoption. Our research introduces a novel approach to enhancing the safety of vision-and-language models by diminishing their sensitivity to NSFW (not safe for work) inputs. In particular, our methodology seeks to sever “toxic” linguistic and visual concepts, unlearning the linkage between unsafe linguistic or visual items and unsafe regions of the embedding space. We show how this can be done by fine-tuning a CLIP model on synthetic data obtained from a large language model trained to convert between safe and unsafe sentences, and a text-to-image generator. We conduct extensive experiments on the resulting embedding space for cross-modal retrieval, text-to-image, and image-to-text generation, where we show that our model can be remarkably employed with pre-trained generative models. Our source code and trained models are available at: https://github.com/aimagelab/safe-clip. ∗ Equal contribution."
TetraDiffusion: Tetrahedral Diffusion Models for 3D Shape Generation
Authors: Nikolai Kalischek; Torben Peters; Jan Dirk Wegner; Konrad Schindler
Venue: ECCV 2024 (main)

Abstract: "Probabilistic denoising diffusion models (DDMs) have set a new standard for 2D image generation. Extending DDMs for 3D content creation is an active field of research. Here, we propose TetraDiffusion, a diffusion model that operates on a tetrahedral partitioning of 3D space to enable efficient, high-resolution 3D shape generation. Our model introduces operators for convolution and transpose convolution that act directly on the tetrahedral partition, and seamlessly includes additional attributes like color. Our design generates mesh geometry much more efficiently: Compared to existing mesh diffusion techniques, TetraDiffusion is up to 200× faster. At the same time, it reduces memory consumption and can operate at substantially higher resolution than existing mesh generators. Using only standard consumer hardware, it sets a new standard in terms of spatial detail and outperforms other mesh generators across a range of quality metrics. For additional results and code see our project page tetradiffusion.github. io."
Camera Calibration using a Collimator System
Authors: Shunkun Liang; Banglei Guan; Zhenbao Yu; Pengju Sun; Yang Shang
Venue: ECCV 2024 (main)

Abstract: "Camera calibration is a crucial step in photogrammetry and 3D vision applications. In practical scenarios with a long working distance to cover a wide area, target-based calibration methods become complicated and inflexible due to site limitations. This paper introduces a novel camera calibration method using a collimator system, which can provide a reliable and controllable calibration environment for cameras with varying working distances. Based on the optical geometry of the collimator system, we prove that the relative motion between the target and camera conforms to the spherical motion model, reducing the original 6DOF relative motion to 3DOF pure rotation motion. Furthermore, a closed-form solver for multiple views and a minimal solver for two views are proposed for camera calibration. The performance of our method is evaluated in both synthetic and real-world experiments, which verify the feasibility of calibration using the collimator system and demonstrate that our method is superior to the state-of-the-art methods. Demo code is available at https : //github.com/LiangSK98/CollimatorCalibration"
Label-free Neural Semantic Image Synthesis
Authors: Jiayi Wang; Kevin A Laube; Yumeng Li; Jan Hendrik Metzen; Shin-I Cheng; Julio Borges; Anna Khoreva
Venue: ECCV 2024 (main)

Abstract: "Recent work has shown great progress in integrating spatial conditioning to control large, pre-trained text-to-image diffusion models. Despite these advances, existing methods describe the spatial image content using hand-crafted conditioning inputs, which are either semantically ambiguous (e.g., edges) or require expensive manual annotations (e.g., semantic segmentation). To address these limitations, we propose a new label-free way of conditioning diffusion models to enable fine-grained spatial control. We introduce the concept of neural semantic image synthesis, which uses neural layouts extracted from pre-trained foundation models as conditioning. These layouts provide rich descriptions of the desired image, containing both semantics and detailed geometry of the scene. We experimentally show that images synthesized via neural semantic image synthesis achieve similar or superior pixel-level alignment of semantic classes compared to those created using expensive semantic label maps. At the same time, they capture better semantics, instance separation, and object orientation than other label-free conditioning options, such as edges or depth. Moreover, we show that images generated by neural layout conditioning can effectively augment real data for training various perception tasks."
Exploring Reliable Matching with Phase Enhancement for Night-time Semantic Segmentation
Authors: Yuwen Pan; Rui Sun; Naisong Luo; Tianzhu Zhang; Yongdong Zhang
Venue: ECCV 2024 (main)

Abstract: "Semantic segmentation of night-time images holds significant importance in computer vision, particularly for applications like night environment perception in autonomous driving systems. However, existing methods tend to parse night-time images from a day-time perspective, leaving the inherent challenges in low-light conditions (such as compromised texture and deceiving matching errors) unexplored. To address these issues, we propose a novel end-to-end optimized approach, named NightFormer, tailored for night-time semantic segmentation, avoiding the conventional practice of forcibly fitting night-time images into day-time distributions. Specifically, we design a pixel-level texture enhancement module to acquire texture-aware features hierarchically with phase enhancement and amplified attention, and an object-level reliable matching module to realize accurate association matching via reliable attention in low-light environments. Extensive experimental results on various challenging benchmarks including NightCity, BDD and Cityscapes demonstrate that our proposed method performs favorably against state-of-the-art night-time semantic segmentation methods."
Multiscale Sliced Wasserstein Distances as Perceptual Color Difference Measures
Authors: Jiaqi He; Zhihua Wang; Leon Wang; Tsein-I Liu; Yuming Fang; Qilin Sun; Kede Ma
Venue: ECCV 2024 (main)

Abstract: "Contemporary color difference (CD) measures for photographic images typically operate by comparing co-located pixels, patches in a “perceptually uniform” color space, or features in a learned latent space. Consequently, these measures inadequately capture the human color perception of misaligned image pairs, which are prevalent in digital photography (, the same scene captured by different smartphones). In this paper, we describe a perceptual CD measure based on the multiscale sliced Wasserstein distance, which facilitates efficient comparisons between non-local patches of similar color and structure. This aligns with the modern understanding of color perception, where color and structure are inextricably interdependent as a unitary process of perceptual organization. Meanwhile, our method is easy to implement and training-free. Experimental results indicate that our CD measure performs favorably in assessing CDs in photographic images, and consistently surpasses competing models in the presence of image misalignment. Additionally, we empirically verify that our measure functions as a metric in the mathematical sense, and show its promise as a loss function for image and video color transfer tasks. The code is available at https://github.com/real-hjq/MS-SWD."
DiscoMatch: Fast Discrete Optimisation for Geometrically Consistent 3D Shape Matching
Authors: Paul Roetzer; Ahmed Abbas; Dongliang Cao; Florian Bernard; Paul Swoboda
Venue: ECCV 2024 (main)

Abstract: "In this work we propose to combine the advantages of learning-based and combinatorial formalisms for 3D shape matching. While learning-based methods lead to state-of-the-art matching performance, they do not ensure geometric consistency, so that obtained matchings are locally non-smooth. On the contrary, axiomatic, optimisation-based methods allow to take geometric consistency into account by explicitly constraining the space of valid matchings. However, existing axiomatic formalisms do not scale to practically relevant problem sizes, and require user input for the initialisation of non-convex optimisation problems. We work towards closing this gap by proposing a novel combinatorial solver that combines a unique set of favourable properties: our approach (i) is initialisation free, (ii) is massively parallelisable and powered by a quasi-Newton method, (iii) provides optimality gaps, and (iv) delivers improved matching quality with decreased runtime and globally optimal results for many instances. all_papers.txt decode_tex_noligatures.sh decode_tex_noligatures.sh~ decode_tex.sh decode_tex.sh~ ECCV_abstracts.csv ECCV_abstracts_good.csv ECCV.csv ECCV.csv~ ECCV_new.csv generate_list.sh generate_list.sh~ generate_overview.sh gen.sh gen.sh~ HOWTO HOWTO~ pdflist pdflist.copied RCS snippet.html These authors contributed equally to this work."