GenAI4ELab/papercli-papers huggingface dataset:
DrivingDiffusion: Layout-Guided Multi-View Driving Scenarios Video Generation with Latent Diffusion Model
Authors: Li Xiaofan; Zhang Yifu; Ye Xiaoqing
Venue: ECCV 2024 (main)

Abstract: "With the surge in autonomous driving technologies, the reliance on comprehensive and high-definition bird’s-eye-view (BEV) representations has become paramount. This burgeoning need underscores the demand for extensive multi-view video datasets, meticulously annotated to facilitate advanced research and development. Nonetheless, the acquisition of such datasets is impeded by prohibitive costs associated with data collection and annotation. There are two challenges when synthesizing multi-view videos given a 3D layout: Generating multi-view videos involves handling both view and temporal dimensions. How to generate videos while ensuring cross-view consistency and cross-frame consistency? 2) How to ensure the precision of layout control and the quality of the generated instances? Addressing this critical bottleneck, we introduce a novel spatial-temporal consistent diffusion framework, , engineered to synthesize realistic multi-view videos governed by 3D spatial layouts. adeptly navigates the dual challenges of maintaining cross-view and cross-frame coherence, along with meeting the exacting standards of layout fidelity and visual quality. The framework operates through a tripartite methodology: initiating with the generation of multi-view single-frame images, followed by the synthesis of single-view videos across multiple cameras, and culminating with a post-processing phase. We corroborate the efficacy of through rigorous quantitative and qualitative evaluations, demonstrating its potential to significantly enhance autonomous driving tasks without incurring additional costs. https://drivingdiffusion.github.io"
Evaluating the Adversarial Robustness of Semantic Segmentation: Trying Harder Pays Off
Authors: Levente Halmosi; Bálint Mohos; Márk Jelasity
Venue: ECCV 2024 (main)

Abstract: "Machine learning models are vulnerable to tiny adversarial input perturbations optimized to cause a very large output error. To measure this vulnerability, we need reliable methods that can find such adversarial perturbations. For image classification models, evaluation methodologies have emerged that have stood the test of time. However, we argue that in the area of semantic segmentation, a good approximation of the sensitivity to adversarial perturbations requires significantly more effort than what is currently considered satisfactory. To support this claim, we re-evaluate a number of well-known robust segmentation models in an extensive empirical study. We propose new attacks and combine them with the strongest attacks available in the literature. We also analyze the sensitivity of the models in fine detail. The results indicate that most of the state-of-the-art models have a dramatically larger sensitivity to adversarial perturbations than previously reported. We also demonstrate a size-bias: small objects are often more easily attacked, even if the large objects are robust, a phenomenon not revealed by current evaluation metrics. Our results also demonstrate that a diverse set of strong attacks is necessary, because different models are often vulnerable to different attacks. Our implementation is available at https://github.com/szegedai/Robust-Segmentation-Evaluation."
SkyScenes: A Synthetic Dataset for Aerial Scene Understanding
Authors: Sahil S Khose; Anisha Pal; Aayushi Agarwal; . Deepanshi; Judy Hoffman; Prithvijit Chattopadhyay
Venue: ECCV 2024 (main)

Abstract: "Real-world aerial scene understanding is limited by a lack of datasets that contain densely annotated images curated under a diverse set of conditions. Due to inherent challenges in obtaining such images in controlled real-world settings, we present , a synthetic dataset of densely annotated aerial images captured from Unmanned Aerial Vehicle (UAV) perspectives. We carefully curate images from to comprehensively capture diversity across layouts (urban and rural maps), weather conditions, times of day, pitch angles and altitudes with corresponding semantic, instance and depth annotations. Through our experiments using , we show that (1) models trained on generalize well to different real-world scenarios, (2) augmenting training on real images with data can improve real-world performance, (3) controlled variations in can offer insights into how models respond to changes in viewpoint conditions (height and pitch), weather and time of day, and (4) incorporating additional sensor modalities (depth) can improve aerial scene understanding. Our dataset and associated generation code are publicly available at: https://hoffman-group.github.io/SkyScenes/ Aerial scene understanding, Synthetic-to-Real generalization, Segmentation, Domain Generalization, Synthetic Data"
Large-Scale Multi-Hypotheses Cell Tracking Using Ultrametric Contours Maps
Authors: Jordão Bragantini; Merlin Lange; Loïc A Royer
Venue: ECCV 2024 (main)

Abstract: "In this work, we describe a method for large-scale 3D cell-tracking through a segmentation selection approach. The proposed method is effective at tracking cells across large microscopy datasets on two fronts: (i) It can solve problems containing millions of segmentation instances in terabyte-scale 3D+t datasets; (ii) It achieves competitive results with or without deep learning, bypassing the requirement of 3D annotated data, that is scarce in the fluorescence microscopy field. The proposed method computes cell tracks and segments using a hierarchy of segmentation hypotheses and selects disjoint segments by maximizing the overlap between adjacent frames. We show that this method is the first to achieve state-of-the-art in both nuclei- and membrane-based cell tracking by evaluating it on the 2D epithelial cell benchmark and 3D images from the cell tracking challenge. Furthermore, it has a faster integer linear programming formulation, and the framework is flexible, supporting segmentations from individual off-the-shelf cell segmentation models or their combination as an ensemble. The code is available as supplementary material."
GSD: View-Guided Gaussian Splatting Diffusion for 3D Reconstruction
Authors: Yuxuan Mu; Xinxin Zuo; Chuan Guo; Yilin Wang; Juwei Lu; Xiaofei Wu; Songcen Xu; Peng Dai; Youliang Yan; Li Cheng
Venue: ECCV 2024 (main)

Abstract: "We present GSD, a diffusion model approach based on Gaussian Splatting (GS) representation for 3D object reconstruction from a single view. Prior works suffer from inconsistent 3D geometry or mediocre rendering quality due to improper representations. We take a step towards resolving these shortcomings by utilizing the recent state-of-the-art 3D explicit representation, Gaussian Splatting, and an unconditional diffusion model. This model learns to generate 3D objects represented by sets of GS ellipsoids. With these strong generative 3D priors, though learning unconditionally, the diffusion model is ready for view-guided reconstruction without further model fine-tuning. This is achieved by propagating fine-grained 2D features through the efficient yet flexible splatting function and the guided denoising sampling process. In addition, a 2D diffusion model is further employed to enhance rendering fidelity, and improve reconstructed GS quality by polishing and re-using the rendered images. The final reconstructed objects explicitly come with high-quality 3D structure and texture, and can be efficiently rendered in arbitrary views. Experiments on the challenging real-world CO3D dataset demonstrate the superiority of our approach. Project page: https://yxmu.foo/GSD/"
AdaDiff: Accelerating Diffusion Models through Step-Wise Adaptive Computation
Authors: Shengkun Tang; Yaqing Wang; Caiwen Ding; Yi Liang; Yao Li; Dongkuan Xu
Venue: ECCV 2024 (main)

Abstract: "Diffusion models achieve great success in generating diverse and high-fidelity images, yet their widespread application, especially in real-time scenarios, is hampered by their inherently slow generation speed. The slow generation stems from the necessity of multi-step network inference. While some certain predictions benefit from the full computation of the model in each sampling iteration, not every iteration requires the same amount of computation, potentially leading to inefficient computation. Unlike typical adaptive computation challenges that deal with single-step generation problems, diffusion processes with a multi-step generation need to dynamically adjust their computational resource allocation based on the ongoing assessment of each step’s importance to the final image output, presenting a unique set of challenges. In this work, we propose , an adaptive framework that dynamically allocates computation resources in each sampling step to improve the generation efficiency of diffusion models. To assess the effects of changes in computational effort on image quality, we present a timestep-aware uncertainty estimation module (UEM). Integrated at each intermediate layer, the UEM evaluates the predictive uncertainty. This uncertainty measurement serves as an indicator for determining whether to terminate the inference process. Additionally, we introduce an uncertainty-aware layer-wise loss aimed at bridging the performance gap between full models and their adaptive counterparts. Comprehensive experiments including class-conditional, unconditional, and text-guided image generation across multiple datasets demonstrate superior performance and efficiency of relative to current early exiting techniques in diffusion models. Notably, we observe enhanced performance on FID, with an acceleration ratio reduction of around 45%. Another exciting observation is that adaptive computation can synergize with other efficiency-enhancing methods such as reducing sampling steps to accelerate inference."
PFedEdit: Personalized Federated Learning via Automated Model Editing
Authors: Haolin Yuan; William Paul; John Aucott; Philippe Burlina; Yinzhi Cao
Venue: ECCV 2024 (main)

Abstract: "Federated learning (FL) allows clients to train a deep learning model collaboratively while maintaining their private data locally. One challenging problem facing FL is that the model utility drops significantly once the data distribution gets heterogeneous, or non-i.i.d, among clients. A promising solution is to personalize models for each client, e.g., keeping some layers locally without aggregation, which is thus called personalized FL. However, previous personalized FL often suffer from suboptimal utility because their choice of layer personalization is based on empirical knowledge and fixed for different datasets and distributions. In this work, we design , the first federated learning framework that leverages automated model editing to optimize the choice of personalization layers and improve model utility under a variety of data distributions including non-i.i.d. The high-level idea of is to assess the effectiveness of every global model layer in improving model utility on local data distribution once edited, and then to apply edits on the top-k most effective layers. Our evaluation shows that outperforms six state-of-the-art approaches on three benchmark datasets by 6% on the model’s performance on average, with the largest accuracy improvement being 26.6%. is open-source and available at this repository: https://github. com/Haolin-Yuan/PFedEdit"
De-Confusing Pseudo-Labels in Source-Free Domain Adaptation
Authors: Idit Diamant; Amir Rosenfeld; Idan Achituve; Jacob Goldberger; Arnon Netzer
Venue: ECCV 2024 (main)

Abstract: "Source-free domain adaptation aims to adapt a source-trained model to an unlabeled target domain without access to the source data. It has attracted growing attention in recent years, where existing approaches focus on self-training that usually includes pseudo-labeling techniques. In this paper, we introduce a novel noise-learning approach tailored to address noise distribution in domain adaptation settings and learn to de-confuse the pseudo-labels. More specifically, we learn a noise transition matrix of the pseudo-labels to capture the label corruption of each class and learn the underlying true label distribution. Estimating the noise transition matrix enables a better true class-posterior estimation, resulting in better prediction accuracy. We demonstrate the effectiveness of our approach when combined with several source-free domain adaptation methods: SHOT, SHOT++, and AaD. We obtain state-of-the-art results on three domain adaptation datasets: VisDA, DomainNet, and OfficeHome."
GenerateCT: Text-Conditional Generation of 3D Chest CT Volumes
Authors: Ibrahim Ethem Hamamci; Sezgin Er; Anjany Sekuboyina; Enis Simsar; Alperen Tezcan; Ayse Gulnihan Simsek; Sevval Nil Esirgun; Furkan Almas; Irem Dogan; Muhammed Furkan Dasdelen; Chinmay Prabhakar; Hadrien Reynaud; Sarthak Pati; Christian Bluethgen; Mehmet Kemal Ozdemir; Bjoern Menze
Venue: ECCV 2024 (main)

Abstract: "Text-conditional medical image generation is vital for radiology, augmenting small datasets, preserving data privacy, and enabling patient-specific modeling. However, its applications in 3D medical imaging, such as CT and MRI, which are crucial for critical care, remain unexplored. In this paper, we introduce GenerateCT, the first approach to generating 3D medical imaging conditioned on free-form medical text prompts. GenerateCT incorporates a text encoder and three key components: a novel causal vision transformer for encoding 3D CT volumes, a text-image transformer for aligning CT and text tokens, and a text-conditional super-resolution diffusion model. Without directly comparable methods in 3D medical imaging, we benchmarked GenerateCT against cutting-edge methods, demonstrating its superiority across all key metrics. Importantly, we explored GenerateCT’s clinical applications by evaluating its utility in a multi-abnormality classification task. First, we established a baseline by training a multi-abnormality classifier on our real dataset. To further assess the model’s generalization to external datasets and its performance with unseen prompts in a zero-shot scenario, we employed an external dataset to train the classifier, setting an additional benchmark. We conducted two experiments in which we doubled the training datasets by synthesizing an equal number of volumes for each set using GenerateCT. The first experiment demonstrated an 11% improvement in the AP score when training the classifier jointly on real and generated volumes. The second experiment showed a 7% improvement when training on both real and generated volumes based on unseen prompts. Moreover, GenerateCT enables the scaling of synthetic training datasets to arbitrary sizes. As an example, we generated 100,000 3D CT volumes, fivefold the number in our real dataset, and trained the classifier exclusively on these synthetic volumes. Impressively, this classifier surpassed the performance of the one trained on all available real data by a margin of 8%. Lastly, domain experts evaluated the generated volumes, confirming a high degree of alignment with the text prompts. Access our code, model weights, training data, and generated data at https://github.com/ibrahimethemhamamci/GenerateCT."
EraseDraw : Learning to Insert Objects by Erasing Them from Images
Authors: Alper Canberk; Maksym Bondarenko; Ege Ozguroglu; Ruoshi Liu; Carl Vondrick
Venue: ECCV 2024 (main)

Abstract: "Creative processes such as painting often involve creating different components of an image one by one. Can we build a computational model to perform this task? Prior works often fail by making global changes to the image, inserting objects in unrealistic spatial locations, and generating inaccurate lighting details. We observe that while state-of-the-art models perform poorly on object insertion, they can remove objects and erase the background in natural images very well. Inverting the direction of object removal, we obtain high-quality data for learning to insert objects that are spatially, physically, and optically consistent with the surroundings. With this scalable automatic data generation pipeline, we can create a dataset for learning object insertion, which is used to train our proposed text-conditioned diffusion model. Qualitative and quantitative experiments have shown that our model achieves state-of-the-art results in object insertion, particularly for in-the-wild images. We show compelling results on diverse insertion prompts and images across various domains. In addition, we automate iterative insertion by combining our insertion model with beam search guided by CLIP."
SuperFedNAS: Cost-Efficient Federated Neural Architecture Search for On-Device Inference
Authors: Alind Khare; Animesh Agrawal; Aditya Annavajjala; Payman Behnam; Myungjin Lee; Hugo M Latapie; Alexey Tumanov
Venue: ECCV 2024 (main)

Abstract: "Neural Architecture Search (NAS) for Federated Learning (FL) is an emerging field. It automates the design and training of Deep Neural Networks (DNNs) when data cannot be centralized due to privacy, communication costs, or regulatory restrictions. Recent federated NAS methods not only reduce manual effort but also help achieve higher accuracy than traditional FL methods like FedAvg. Despite the success, existing federated NAS methods still fall short in satisfying diverse deployment targets common in on-device inference including hardware, latency budgets, or variable battery levels. Most federated NAS methods search for only a limited range of neuro-architectural patterns, repeat them in a DNN, thereby restricting achievable performance. Moreover, these methods incur prohibitive training costs to satisfy deployment targets. They perform the training and search of DNN architectures repeatedly for each case. addresses these challenges by decoupling the training and search in federated NAS. co-trains a large number of diverse DNN architectures contained inside one supernet in the FL setting. Post-training, clients perform NAS locally to find specialized DNNs by extracting different parts of the trained supernet with no additional training. takes O(1) (instead of O(N )) cost to find specialized DNN architectures in FL for any N deployment targets. As part of , we introduce —a novel FL training algorithm that performs multi-objective federated optimization of DNN architectures (≈ 5 ∗ 108 ) under different client data distributions. achieves upto 37.7% higher accuracy or upto 8.13x reduction in MACs than existing federated NAS methods. Code is released at https://github.com/gatech-sysml/superfednas."
Towards Reliable Evaluation and Fast Training of Robust Semantic Segmentation Models
Authors: Francesco Croce; Naman D. Singh; Matthias Hein
Venue: ECCV 2024 (main)

Abstract: "Adversarial robustness has been studied extensively in image classification, especially for the ℓ∞ -threat model, but significantly less so for related tasks such as object detection and semantic segmentation, where attacks turn out to be a much harder optimization problem than for image classification. We propose several problem-specific novel attacks minimizing different metrics in accuracy and mIoU. The ensemble of our attacks, , shows that existing attacks severely overestimate the robustness of semantic segmentation models. Surprisingly, existing attempts of adversarial training for semantic segmentation models turn out to be weak or even completely non-robust. We investigate why previous adaptations of adversarial training to semantic segmentation failed and show how recently proposed robust backbones can be used to obtain adversarially robust semantic segmentation models with up to six times less training time for and the more challenging . The associated code and robust models are available at https://github.com/nmndeep/robust-segmentation."
Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training
Authors: David Wan; Jaemin Cho; Elias Stengel-Eskin; Mohit Bansal
Venue: ECCV 2024 (main)

Abstract: "Highlighting particularly relevant regions of an image can improve the performance of vision-language models (VLMs) on various vision-language (VL) tasks by guiding the model to attend more closely to these regions of interest. For example, VLMs can be given a “visual prompt”, where visual markers such as bounding boxes delineate key image regions. However, current VLMs that can incorporate visual guidance are either proprietary and expensive or require costly training on curated data with visual prompts. We introduce Contrastive Region Guidance (CRG), a training-free guidance method that enables open-source VLMs to respond to visual prompts. CRG contrasts model outputs produced with and without visual prompts, factoring out biases revealed by the model when answering without the information required to produce a correct answer. CRG achieves substantial improvements in a wide variety of VL tasks: When region annotations are provided, CRG increases absolute accuracy by up to 11.1% on ViP-Bench, a collection of six diverse region-based tasks such as recognition, math, and object relationship reasoning. We also show CRG’s applicability to spatial reasoning, with 10% improvement on What’sUp, as well as to compositional generalization – improving accuracy by 11.5% and 7.5% on two challenging splits from SugarCrepe – and to image-text alignment for generated images, where we improve by 8.4 AUROC and 6.8 F1 points on SeeTRUE. CRG also allows us to re-rank proposed regions in referring expression comprehension and phrase grounding benchmarks like RefCOCO/+/g and Flickr30K Entities, with an average gain of 3.2% in accuracy. Our analysis explores alternative masking strategies for CRG, empirically validating CRG’s design choices.1 1 Project page: https://contrastive-region-guidance.github.io/."
Keypoint Promptable Re-Identification
Authors: Vladimir Somers; Alexandre Alahi; Christophe De Vleeschouwer
Venue: ECCV 2024 (main)

Abstract: "Occluded Person Re-Identification (ReID) is a metric learning task that involves matching occluded individuals based on their appearance. While many studies have tackled occlusions caused by objects, multi-person occlusions remain less explored. In this work, we identify and address a critical challenge overlooked by previous occluded methods: the Multi-Person Ambiguity (MPA) arising when multiple individuals are visible in the same bounding box, making it impossible to determine the intended ReID target among the candidates. Inspired by recent work on prompting in vision, we introduce Keypoint Promptable ReID (KPR), a novel formulation of the ReID problem that explicitly complements the input bounding box with a set of semantic keypoints indicating the intended target. Since promptable re-identification is an unexplored paradigm, existing datasets lack the pixel-level annotations necessary for prompting. To bridge this gap and foster further research on this topic, we introduce , a novel dataset with keypoints labels, that features strong inter-person occlusions. Furthermore, we release custom keypoint labels for four popular ReID benchmarks. Experiments on person retrieval, but also on pose tracking, demonstrate that our method systematically surpasses previous state-of-the-art approaches on various occluded scenarios. Our code, dataset and annotations are available at https://github. com/VlSomers/keypoint_promptable_reidentification."