GenAI4ELab/papercli-papers huggingface dataset:
Dissolving Is Amplifying: Towards Fine-Grained Anomaly Detection
Authors: Jian Shi; Pengyi Zhang; Ni Zhang; Hakim Ghazzai; Peter Wonka
Venue: ECCV 2024 (main)

Abstract: "Medical imaging often contains critical fine-grained features, such as tumors or hemorrhages, which are crucial for diagnosis yet potentially too subtle for detection with conventional methods. In this paper, we introduce DIA, dissolving is amplifying. DIA is a fine-grained anomaly detection framework for medical images. First, we introduce dissolving transformations. We employ diffusion with a generative diffusion model as a dedicated feature-aware denoiser. Applying diffusion to medical images in a certain manner can remove or diminish fine-grained discriminative features. Second, we introduce an amplifying framework based on contrastive learning to learn a semantically meaningful representation of medical images in a self-supervised manner, with a focus on fine-grained features. The amplifying framework contrasts additional pairs of images with and without dissolving transformations applied and thereby emphasizes the dissolved fine-grained features. DIA significantly improves the medical anomaly detection performance with around 18.40% AUC boost against the baseline method and achieves an overall SOTA against other benchmark methods. Our code is available at https://github.com/shijianjian/DIA.git."
StyleCity: Large-Scale 3D Urban Scenes Stylization
Authors: Yingshu Chen; Huajian Huang; Tuan-Anh Vu; Ka Chun Shum; Sai-Kit Yeung
Venue: ECCV 2024 (main)

Abstract: "Creating large-scale virtual urban scenes with variant styles is inherently challenging. To facilitate prototypes of virtual production and bypass the need for complex materials and lighting setups, we introduce the first vision-and-text-driven texture stylization system for large-scale urban scenes, StyleCity. Taking an image and text as references, StyleCity stylizes a 3D textured mesh of a large-scale urban scene in a semantics-aware fashion and generates a harmonic omnidirectional sky background. To achieve that, we propose to stylize a neural texture field by transferring 2D vision-and-text priors to 3D globally and locally. During 3D stylization, we progressively scale the planned training views of the input 3D scene at different levels in order to preserve high-quality scene content. We then optimize the scene style globally by adapting the scale of the style image with the scale of the training views. Moreover, we enhance local semantics consistency by the semantics-aware style loss which is crucial for photo-realistic stylization. Besides texture stylization, we further adopt a generative diffusion model to synthesize a style-consistent omnidirectional sky image, which offers a more immersive atmosphere and assists the semantic stylization process. The stylized neural texture field can be baked into an arbitrary-resolution texture, enabling seamless integration into conventional rendering pipelines and significantly easing the virtual production prototyping process. Extensive experiments demonstrate our stylized scenes’ superiority in qualitative and quantitative performance and user preferences. Project page: https://chenyingshu.github.io/stylecity3d."
ViG-Bias: Visually Grounded Bias Discovery and Mitigation
Authors: Badr-Eddine Marani; Mohamed Hanini; Nihitha Malayarukil; Stergios Christodoulidis; Maria Vakalopoulou; Enzo Ferrante
Venue: ECCV 2024 (main)

Abstract: "The proliferation of machine learning models in critical decision-making processes has underscored the need for bias discovery and mitigation strategies. Identifying the reasons behind a biased system is not straightforward, since in many occasions they are associated with hidden spurious correlations which are not easy to spot. Standard approaches rely on bias audits performed by analyzing model performance in predefined subgroups of data samples, usually characterized by common attributes like gender or ethnicity when it comes to people, or other specific attributes defining semantically coherent groups of images. However, it is not always possible to know a priori the specific attributes defining the failure modes of visual recognition systems. Recent approaches propose to discover these groups by leveraging large vision language models, which enable the extraction of cross-modal embeddings and the generation of textual descriptions to characterize the subgroups where a certain model is underperforming. In this work, we argue that incorporating visual explanations (e.g. heatmaps generated via GradCAM or other approaches) can boost the performance of such bias discovery and mitigation frameworks. To this end, we introduce Visually Grounded Bias Discovery and Mitigation (ViG-Bias), a simple yet effective technique which can be integrated to a variety of existing frameworks to improve both discovery and mitigation performance. Our comprehensive evaluation shows that incorporating visual explanations enhances existing techniques like DOMINO, FACTS and Bias-to-Text, across several challenging datasets, including CelebA, Waterbirds, and NICO++."
DiffBIR: Toward Blind Image Restoration with Generative Diffusion Prior
Authors: Xinqi Lin; Jingwen He; Ziyan Chen; Zhaoyang Lyu; Bo Dai; Fanghua Yu; Yu Qiao; Wanli Ouyang; Chao Dong
Venue: ECCV 2024 (main)

Abstract: "We present DiffBIR, a general restoration pipeline that could handle different blind image restoration tasks in a unified framework. DiffBIR decouples blind image restoration problem into two stages: 1) degradation removal: removing image-independent content; 2) information regeneration: generating the lost image content. Each stage is developed independently but they work seamlessly in a cascaded manner. In the first stage, we use restoration modules to remove degradations and obtain high-fidelity restored results. For the second stage, we propose IRControlNet that leverages the generative ability of latent diffusion models to generate realistic details. Specifically, IRControlNet is trained based on specially produced condition images without distracting noisy content for stable generation performance. Moreover, we design a region-adaptive restoration guidance that can modify the denoising process during inference without model re-training, allowing users to balance quality and fidelity through a tunable guidance scale. Extensive experiments have demonstrated DiffBIR’s superiority over state-of-the-art approaches for blind image super-resolution, blind face restoration and blind image denoising tasks on both synthetic and real-world datasets. The code is available at https://github.com/XPixelGroup/DiffBIR."
Assessing Sample Quality via the Latent Space of Generative Models
Authors: Jingyi Xu; Hieu Le; Dimitris Samaras
Venue: ECCV 2024 (main)

Abstract: "Advances in generative models increase the need for sample quality assessment. To do so, previous methods rely on a pre-trained feature extractor to embed the generated samples and real samples into a common space for comparison. However, different feature extractors might lead to inconsistent assessment outcomes. Moreover, these methods are not applicable for domains where a robust, universal feature extractor does not yet exist, such as medical images or 3D assets. In this paper, we propose to directly examine the latent space of the trained generative model to infer generated sample quality. This is feasible because the quality a generated sample directly relates to the amount of training data resembling it, and we can infer this information by examining the density of the latent space. Accordingly, we use a latent density score function to quantify sample quality. We show that the proposed score correlates highly with the sample quality for various generative models including VAEs, GANs and Latent Diffusion Models. Compared with previous quality assessment methods, our method has the following advantages: 1) pre-generation quality estimation with reduced computational cost, 2) generalizability to various domains and modalities, and 3) applicability to latent-based image editing and generation methods. Extensive experiments demonstrate that our proposed methods can benefit downstream tasks such as few-shot image classification and latent face image editing. Code is available at https://github.com/cvlab-stonybrook/LS-sample-quality."
Relightable Neural Actor with Intrinsic Decomposition and Pose Control
Authors: Diogo Carbonera Luvizon; Vladislav Golyanik; Adam Kortylewski; Marc Habermann; Christian Theobalt
Venue: ECCV 2024 (main)

Abstract: "Creating a controllable and relightable digital avatar from multi-view video with fixed illumination is a very challenging problem since humans are highly articulated, creating pose-dependent appearance effects, and skin as well as clothing require space-varying BRDF modeling. Existing works on creating animatible avatars either do not focus on relighting at all, require controlled illumination setups, or try to recover a relightable avatar from very low cost setups, i.e. a single RGB video, at the cost of severely limited result quality, e.g. shadows not even being modeled. To address this, we propose Relightable Neural Actor, a new video-based method for learning a pose-driven neural human model that can be relighted, allows appearance editing, and models pose-dependent effects such as wrinkles and self-shadows. Importantly, for training, our method solely requires a multi-view recording of the human under a known, but static lighting condition. To tackle this challenging problem, we leverage an implicit geometry representation of the actor with a drivable density field that models pose-dependent deformations and derive a dynamic mapping between 3D and UV spaces, where normal, visibility, and materials are effectively encoded. To evaluate our approach in real-world scenarios, we collect a new dataset with four identities recorded under different light conditions, indoors and outdoors, providing the first benchmark of its kind for human relighting, and demonstrating state-of-the-art relighting results for novel human poses."
Sur^2f: A Hybrid Representation for High-Quality and Efficient Surface Reconstruction from Multi-view Images
Authors: Zhangjin Huang; Zhihao Liang; Kui Jia
Venue: ECCV 2024 (main)

Abstract: "Multi-view surface reconstruction is an ill-posed, inverse problem in 3D vision research. It involves modeling the geometry and appearance with appropriate surface representations. Most of the existing methods rely either on explicit meshes, using surface rendering of meshes for reconstruction, or on implicit field functions, using volume rendering of the fields for reconstruction. The two types of representations in fact have their respective merits. In this work, we propose a new hybrid representation, termed Sur2 f, aiming to better benefit from both representations in a complementary manner. Technically, we learn two parallel streams of an implicit signed distance field and an explicit surrogate surface (Sur2 f) mesh, and unify volume rendering of the implicit signed distance function (SDF) and surface rendering of the surrogate mesh with a shared, neural shader; the unified shading promotes their convergence to the same, underlying surface. We synchronize learning of the surrogate mesh by driving its deformation with functions induced from the implicit SDF. In addition, the synchronized surrogate mesh enables surface-guided volume sampling, which greatly improves the sampling efficiency per ray in volume rendering. We conduct thorough experiments showing that Sur2 f outperforms existing reconstruction methods and surface representations, including hybrid ones, in terms of both recovery quality and recovery efficiency."
HO-Gaussian: Hybrid Optimization of 3D Gaussian Splatting for Urban Scenes
Authors: Zhuopeng Li; Yilin Zhang; Chenming Wu; Jianke Zhu; Liangjun Zhang
Venue: ECCV 2024 (main)

Abstract: "The rapid growth of 3D Gaussian Splatting (3DGS) has revolutionized neural rendering, enabling real-time production of high-quality renderings. However, the previous 3DGS-based methods have limitations in urban scenes due to reliance on initial Structure-from-Motion (SfM) points and difficulties in rendering distant, sky and low-texture areas. To overcome these challenges, we propose a hybrid optimization method named HO-Gaussian, which combines a grid-based volume with the 3DGS pipeline. HO-Gaussian eliminates the dependency on SfM point initialization, allowing for rendering of urban scenes, and incorporates the Point densification to enhance rendering quality in problematic regions during training. Furthermore, we introduce Gaussian Direction Encoding as an alternative for spherical harmonics in the rendering pipeline, which enables view-dependent color representation. To account for multi-camera systems, we introduce neural warping to enhance object consistency across different cameras. Experimental results on widely used autonomous driving datasets demonstrate that HO-Gaussian achieves photo-realistic rendering in real-time on multi-camera urban datasets."
Pseudo-keypoint RKHS Learning for Self-supervised 6DoF Pose Estimation
Authors: Yangzheng Wu; Michael Alan Greenspan
Venue: ECCV 2024 (main)

Abstract: "We address the simulation-to-real domain gap in six degree-of-freedom pose estimation (6DoF PE), and propose a novel self-supervised keypoint voting-based 6DoF PE framework, effectively narrowing this gap using a learnable kernel in RKHS. We formulate this domain gap as a distance in high-dimensional feature space, distinct from previous iterative matching methods. We propose an adapter network, which is pre-trained on purely synthetic data with synthetic ground truth poses, and which evolves the network parameters from this source synthetic domain to the target real domain. Importantly, the real data training only uses pseudo-poses estimated by pseudo-keypoints, and thereby requires no real ground truth data annotations. Our proposed method is called RKHSPose, and achieves state-of-the-art performance among self-supervised methods on three commonly used 6DoF PE datasets including LINEMOD (+4.2%), Occlusion LINEMOD (+2%), and YCB-Video (+3%). It also compares favorably to fully supervised methods on all six applicable BOP core datasets, achieving within −11.3% to +0.2% of the top fully supervised results."
Consistent 3D Line Mapping
Authors: Xulong Bai; Hainan Cui; Shuhan Shen
Venue: ECCV 2024 (main)

Abstract: "We address the problem of reconstructing 3D line segments along with line tracks from multiple views with known camera poses. The basic pipeline is first generating 3D line segment proposals for each 2D line segment, then selecting the best proposals, merging them to produce 3D line segments and line tracks, and finally performing non-linear optimization. Our key contributions are focused on exploring and alleviating the inconsistency problems in classical approaches. In the best proposal selection, we analyze the inherent inconsistency problem of support relationships from 2D to 3D determined during proposal evaluation using multiple views and propose an iterative algorithm to handle it. In line track building, we impose 2D collinearity constraints to enhance the consistency of the elements in each line track. In optimization, we introduce coplanarity constraints and jointly optimize points, lines, planes, and vanishing points, enhancing the consistency of the structure of the line map. Experimental results demonstrate that our emphasis on consistency enables our line maps to achieve state-of-the-art completeness and accuracy, while also generating longer and more robust line tracks. Code is available at https://github. com/3dv-casia/clmap."
Distributed Active Client Selection With Noisy Clients Using Model Association Scores
Authors: Kwang In Kim
Venue: ECCV 2024 (main)

Abstract: "Active client selection (ACS) strategically identifies clients for model updates during each training round of federated learning. In scenarios with limited communication resources, ACS emerges as a superior alternative to random client selection, significantly improving the convergence rate. However, existing ACS methods struggle with clients providing noisy updates, those from noisy labels. To address this challenge, we present a new ACS algorithm for scenarios with unknown noisy clients. Our algorithm constructs a client sampling distribution based on the global association among model updates, which quantifies the ability of a client’s model update to align with those from other clients. By leveraging these associations, we efficiently identify and mitigate the impact of clients with substantial noise that could disrupt training. This approach is simple, computationally efficient, and requires no hyperparameter tuning. Experiments on six benchmark datasets demonstrate that conventional ACS methods fail to outperform random selection. In contrast, our approach significantly enhances convergence speed while using the same communication resources."
PixOOD: Pixel-Level Out-of-Distribution Detection
Authors: Tomas Vojir; Jan Sochman; Jiri Matas
Venue: ECCV 2024 (main)

Abstract: "We propose a pixel-level out-of-distribution detection algorithm, called , which does not require training on samples of anomalous data and is not designed for a specific application which avoids traditional training biases. In order to model the complex intra-class variability of the in-distribution data at the pixel level, we propose an online data condensation algorithm which is more robust than standard K-means and is easily trainable through SGD. We evaluate on a wide range of problems. It achieved state-of-the-art results on four out of seven datasets, while being competitive on the rest. The source code is available at https://github.com/vojirt/PixOOD."
GarmentCodeData: A Dataset of 3D Made-to-Measure Garments With Sewing Patterns
Authors: Maria Korosteleva; Timur Levent Kesdogan; Fabian Kemper; Stephan Wenninger; Jasmin Koller; Yuhan Zhang; Mario Botsch; Olga Sorkine-Hornung
Venue: ECCV 2024 (main)

Abstract: "Recent research interest in learning-based processing of garments, from virtual fitting to generation and reconstruction, stumbles on a scarcity of high-quality public data in the domain. We contribute to resolving this need by presenting the first large-scale synthetic dataset of 3D made-to-measure garments with sewing patterns, as well as its generation pipeline. GarmentCodeData contains 115,000 data points that cover a variety of designs in many common garment categories: tops, shirts, dresses, jumpsuits, skirts, pants, etc., fitted to a variety of body shapes sampled from a custom statistical body model based on CAESAR [?], as well as a standard reference body shape, applying three different textile materials. To enable the creation of datasets of such complexity, we introduce a set of algorithms for automatically taking tailor’s measures on sampled body shapes, sampling strategies for sewing pattern design, and propose an automatic, open-source 3D garment draping pipeline based on a fast XPBD simulator [?], while contributing several solutions for collision resolution and drape correctness to enable scalability."
Towards a Density Preserving Objective Function for Learning on Point Sets
Authors: Haritha Jayasinghe; Ioannis Brilakis
Venue: ECCV 2024 (main)

Abstract: "Accurate measurement of the discrepancy between point sets is crucial for point cloud learning tasks. Chamfer distance (CD) is favoured over more effective loss metrics such as Earth Mover’s Distance (EMD) for this purpose due to its computational efficiency. Previous investigations into loss function improvements exclusively focus on 3D losses as static metrics, and ignore their dynamic behaviour during training. We show that directly modifying the correspondence criteria can prevent clustering of points during training, leading to more uniform point distributions. We propose UniformCD, a novel 3D distance metric that prioritises matching the relative local densities of point neighbourhoods when assigning correspondences. The proposed loss improves performance on various tasks such as cloud completion, parametric model optimisation, as well as downstream task performance in self-supervised learning, achieving SOTA EMD results among point set objective functions. We show that our method exploits local density information to converge towards globally optimum density distributions, narrowing the disparity between CD and EMD. Source code is available on Github."
AnatoMask: Enhancing Medical Image Segmentation with Reconstruction-guided Self-masking
Authors: Yuheng Li; Tianyu Luan; Yizhou Wu; Shaoyan Pan; Yenho Chen; Xiaofeng Yang
Venue: ECCV 2024 (main)

Abstract: "Due to the scarcity of labeled data, self-supervised learning (SSL) has gained much attention in 3D medical image segmentation, by extracting semantic representations from unlabeled data. Among SSL strategies, Masked image modeling (MIM) has shown effectiveness by reconstructing randomly masked images to learn detailed representations. However, conventional MIM methods require extensive training data to achieve good performance, which still poses a challenge for medical imaging. Since random masking uniformly samples all regions within medical images, it may overlook crucial anatomical regions and thus degrade the pretraining efficiency. We propose AnatoMask, a novel MIM method that leverages reconstruction loss to dynamically identify and mask out anatomically significant regions to improve pretraining efficacy. AnatoMask takes a self-distillation approach, where the model learns both how to find more significant regions to mask and how to reconstruct these masked regions. To avoid suboptimal learning, Anatomask adjusts the pretraining difficulty progressively using a masking dynamics function. We have evaluated our method on 4 public datasets with multiple imaging modalities (CT, MRI, and PET). AnatoMask demonstrates superior performance and scalability compared to existing SSL methods. The code is available at https://github.com/ricklisz/AnatoMask."
VF-NeRF: Viewshed Fields for Rigid NeRF Registration
Authors: Leo Segre; Shai Avidan
Venue: ECCV 2024 (main)

Abstract: "3D scene registration is a fundamental problem in computer vision that seeks the best 6-DoF alignment between two scenes. This problem was extensively investigated in the case of point clouds and meshes, but there has been relatively limited work regarding Neural Radiance Fields (NeRF). In this paper, we consider the problem of rigid registration between two NeRFs when the position of the original cameras is not given. Our key novelty is the introduction of Viewshed Fields (VF), an implicit function that determines, for each 3D point, how likely it is to be viewed by the original cameras. We demonstrate how VF can help in the various stages of NeRF registration, with an extensive evaluation showing that VF-NeRF achieves SOTA results on various datasets with different capturing approaches such as LLFF and Objaverese. Our code will be made publicly available."
Task-Driven Uncertainty Quantification in Inverse Problems via Conformal Prediction
Authors: Jeffrey Wen; Rizwan Ahmad; Phillip Schniter
Venue: ECCV 2024 (main)

Abstract: "In imaging inverse problems, one seeks to recover an image from missing/corrupted measurements. Because such problems are ill-posed, there is great motivation to quantify the uncertainty induced by the measurement-and-recovery process. Motivated by applications where the recovered image is used for a downstream task, such as soft-output classification, we propose a task-centered approach to uncertainty quantification. In particular, we use conformal prediction to construct an interval that is guaranteed to contain the task output from the true image up to a user-specified probability, and we use the width of that interval to quantify the uncertainty contributed by measurement-and-recovery. For posterior-sampling-based image recovery, we construct locally adaptive prediction intervals. Furthermore, we propose to collect measurements over multiple rounds, stopping as soon as the task uncertainty falls below an acceptable level. We demonstrate our methodology on accelerated magnetic resonance imaging (MRI): https://github.com/jwen307/TaskUQ."