GenAI4ELab/papercli-papers huggingface dataset:
Revisit Anything: Visual Place Recognition via Image Segment Retrieval
Authors: Kartik Garg; Sai Shubodh; Shishir N Y Kolathaya; Madhava Krishna; Sourav Garg
Venue: ECCV 2024 (main)

Abstract: "Accurately recognizing a revisited place is crucial for embodied agents to localize and navigate. This requires visual representations to be distinct, despite strong variations in camera viewpoint and scene appearance. Existing visual place recognition pipelines encode the whole image and search for matches. This poses a fundamental challenge in matching two images of the same place captured from different camera viewpoints: the similarity of what overlaps can be dominated by the dissimilarity of what does not overlap. We address this by encoding and searching for image segments instead of the whole images. We propose to use open-set image segmentation to decompose an image into ‘meaningful’ entities (i.e., things and stuff). This enables us to create a novel image representation as a collection of multiple overlapping subgraphs connecting a segment with its neighboring segments, dubbed SuperSegment. Furthermore, to efficiently encode these SuperSegments into compact vector representations, we propose a novel factorized representation of feature aggregation. We show that retrieving these partial representations leads to significantly higher recognition recall than the typical whole image based retrieval. Our segments-based approach, dubbed , sets a new state-of-the-art in place recognition on a diverse selection of benchmark datasets, while being applicable to both generic and task-specialized image encoders. Finally, we demonstrate the potential of our method to “revisit anything” by evaluating our method on an object instance retrieval task, which bridges the two disparate areas of research: visual place recognition and object-goal navigation, through their common aim of recognizing goal objects specific to a place. Source code: https://github.com/AnyLoc/Revisit-Anything."
EcoMatcher: Efficient Clustering Oriented Matcher for Detector-free Image Matching
Authors: Peiqi Chen; Lei Yu; Yi Wan; Yongjun Zhang; Jian Wang; Liheng Zhong; Jingdong Chen; Ming Yang
Venue: ECCV 2024 (main)

Abstract: "Detector-free local feature matching methods have demonstrated significant performance improvements since leveraging the power of Transformer architecture. The global receptive field allows for simultaneous interaction among all elements, proving particularly beneficial in regions with low texture or repetitive patterns. However, Transformer-based methods are confronted by how to achieve a balance between computational cost and expressive efficacy when dealing with numerous features. In this work, we revisit existing detector-free methods and propose EcoMatcher, a universal matcher based on implicit clustering termed Context Clusters. By introducing coarse-grained features as clustering centers, similar features are allocated to the same center, forming distinct clustering patterns. Features within the same cluster are then dispatched with identical messages from their center but at varying scales depending on the similarity metrics. This process defines a novel feature extraction paradigm for both self-understanding and cross-interaction of image pairs, facilitating fusing multi-level features and reducing overall complexity. EcoMatcher proves to be a competitive detector-free method in terms of memory consumption and runtime speed, while also achieves strong performance on mainstream benchmarks."
DGD: Dynamic 3D Gaussians Distillation
Authors: Isaac Labe; Noam Issachar; Itai Lang; Sagie Benaim
Venue: ECCV 2024 (main)

Abstract: "We tackle the task of learning dynamic 3D semantic radiance fields given a single monocular video as input. Our learned semantic radiance field captures per-point semantics as well as color and geometric properties for a dynamic 3D scene, enabling the generation of novel views and their corresponding semantics. This enables the segmentation and tracking of a diverse set of 3D semantic entities, specified using a simple and intuitive interface that includes a user click or a text prompt. To this end, we present , a unified 3D representation for both the appearance and semantics of a dynamic 3D scene, building upon the recently proposed dynamic 3D Gaussians representation. Our representation is optimized over time with both color and semantic information. Key to our method is the joint optimization of the appearance and semantic attributes, which jointly affect the geometric properties of the scene. We evaluate our approach in its ability to enable dense semantic 3D object tracking and demonstrate high-quality results that are fast to render, for a diverse set of scenes. Our project webpage is available on https://isaaclabe.github.io/DGD-Website/"
Semantic Diversity-aware Prototype-based Learning for Unbiased Scene Graph Generation
Authors: Jaehyeong Jeon; Kibum Kim; Kanghoon Yoon; Chanyoung Park
Venue: ECCV 2024 (main)

Abstract: "The scene graph generation (SGG) task involves detecting objects within an image and predicting predicates that represent the relationships between the objects. However, in SGG benchmark datasets, each subject-object pair is annotated with a single predicate even though a single predicate may exhibit diverse semantics (i.e., semantic diversity), existing SGG models are trained to predict the one and only predicate for each pair. This in turn results in the SGG models to overlook the semantic diversity that may exist in a predicate, thus leading to biased predictions. In this paper, we propose a novel model-agnostic Semantic Diversity-aware Prototype-based Learning () framework that enables unbiased predictions based on the understanding of the semantic diversity of predicates. Specifically, DPL learns the regions in the semantic space covered by each predicate to distinguish among the various different semantics that a single predicate can represent. Extensive experiments demonstrate that our proposed model-agnostic DPL framework brings significant performance improvement on existing SGG models, and also effectively understands the semantic diversity of predicates. The code is available at https://github.com/JeonJaeHyeong/ DPL.git."
DiffuMatting: Synthesizing Arbitrary Objects with Matting-level Annotation
Authors: Xiaobin Hu; Xu Peng; Donghao Luo; Xiaozhong Ji; Jinlong Peng; ZhengKai Jiang; Jiangning Zhang; Taisong Jin; Chengjie Wang; Rongrong Ji
Venue: ECCV 2024 (main)

Abstract: "Due to the difficulty and labor-consuming nature of getting highly accurate or matting annotations, there only exists a limited amount of highly accurate labels available to the public. To tackle this challenge, we propose a DiffuMatting which inherits the strong Everything generation ability of diffusion and endows the power of ‘matting anything’. Our DiffuMatting can 1). act as an anything matting factory with high accurate annotations 2). be well-compatible with community LoRAs or various conditional control approaches to achieve the community-friendly art design and controllable generation. Specifically, inspired by green-screen-matting, we aim to teach the diffusion model to paint on a fixed green screen canvas. To this end, a large-scale green-screen dataset (Green100K) is collected as a training dataset for DiffuMatting. Secondly, a green background control loss is proposed to keep the drawing board as a pure green color to distinguish the foreground and background. To ensure the synthesized object has more edge details, a detailed-enhancement of transition boundary loss is proposed as a guideline to generate objects with more complicated edge structures. Aiming to simultaneously generate the object and its matting annotation, we build a matting head to make a green-color removal in the latent space of the VAE decoder. Our DiffuMatting shows several potential applications (e.g., matting-data generator, community-friendly art design and controllable generation). As a matting-data generator, DiffuMatting synthesizes general object and portrait matting sets, effectively reducing the relative MSE error by 15.4% in General Object Matting. The dataset is released in our project page at https:// diffumatting.github.io."
Self-Guided Generation of Minority Samples Using Diffusion Models
Authors: Soobin Um; Jong Chul Ye
Venue: ECCV 2024 (main)

Abstract: "We present a novel approach for generating minority samples that live on low-density regions of a data manifold. Our framework is built upon diffusion models, leveraging the principle of guided sampling that incorporates an arbitrary energy-based guidance during inference time. The key defining feature of our sampler lies in its self-contained nature, , implementable solely with a pretrained model. This distinguishes our sampler from existing techniques that require expensive additional components (like external classifiers) for minority generation. Specifically, we first estimate the likelihood of features within an intermediate latent sample by evaluating a reconstruction loss w.r.t. its posterior mean. The generation then proceeds with the minimization of the estimated likelihood, thereby encouraging the emergence of minority features in the latent samples of subsequent timesteps. To further improve the performance of our sampler, we provide several time-scheduling techniques that properly manage the influence of guidance over inference steps. Experiments on benchmark real datasets demonstrate that our approach can greatly improve the capability of creating realistic low-likelihood minority instances over the existing techniques without the reliance on costly additional elements. Code is available at https://github. com/soobin-um/sg-minority."
DEVIAS: Learning Disentangled Video Representations of Action and Scene
Authors: Kyungho Bae; Youngrae Kim; Geo Ahn; Jinwoo Choi
Venue: ECCV 2024 (main)

Abstract: "Video recognition models often learn scene-biased action representation due to the spurious correlation between actions and scenes in the training data. Such models show poor performance when the test data consists of videos with unseen action-scene combinations. Although Scene-debiased action recognition models might address the issue, they often overlook valuable scene information in the data. To address this challenge, we propose to learn DisEntangled VIdeo representations of Action and Scene (DEVIAS), for more holistic video understanding. We propose an encoder-decoder architecture to learn disentangled action and scene representations with a single model. The architecture consists of a disentangling encoder (DE), an action mask decoder (AMD), and a prediction head. The key to achieving the disentanglement is employing both DE and AMD during training time. The DE uses the slot attention mechanism to learn disentangled action and scene representations. For further disentanglement, an AMD learns to predict action masks, given an action slot. With the resulting disentangled representations, we can achieve robust performance across diverse scenarios, including both seen and unseen action-scene combinations. We rigorously validate the proposed method on the UCF-101, Kinetics-400, and HVU datasets for the seen, and the SCUBA, HAT, and HVU datasets for unseen action-scene combination scenarios. Furthermore, DEVIAS provides flexibility to adjust the emphasis on action or scene information depending on dataset characteristics for downstream tasks. DEVIAS shows favorable performance in various downstream tasks: Diving48, Something-Something-V2, UCF-101, and ActivityNet. The code is available at https://github.com/KHU-VLL/DEVIAS."
AD3: Introducing a score for Anomaly Detection Dataset Difficulty assessment using VIADUCT dataset
Authors: Jan D Lehr; Jan H Philipps; Alik Sargsyan; Martin Pape; Jörg Krüger
Venue: ECCV 2024 (main)

Abstract: "The field of visual Industrial Anomaly Detection (IAD) has brought forth many new semi-supervised learning methods in recent years. At the same time, there have been few new datasets for benchmarking the methods. The most popular dataset is MVTec-AD dataset, because of its diversity of categories and availability of industrial objects. But many methods already achieve AUROC scores of more than 99 % on the MVTec-AD dataset. The defects of the categories that the dataset provides appear to be easily detectable. Furthermore, there is no existing approach to statistically describe the defects that need to be found in IAD datasets. This paper presents a new dataset for visual industrial anomaly detection and a novel approach for Anomaly Detection Dataset Difficulty assessment with the AD3 score. The new dataset named VIADUCT contains 49 categories and 10,986 high resolution images from eleven different sectors. Through the support of several manufacturing companies, numerous real inspection problems are presented through the dataset. It contains a large number of different defects with detailed pixel-wise annotations. The VIADUCT dataset is compared with other state of the art datasets to underline its added value. Therefore, we provide an overview for each dataset regarding the number of categories, images, defect categories and defects. In addition to these obvious comparisons the defects of the datasets are described with the AD3 score. This novel score is used to analyze the size of the defects and the similarity between the defect and its corresponding object. Using seven selected methods from industrial anomaly detection, a benchmark is performed on the new dataset, showing that there is still potential for improvement. It is shown that the VIADUCT dataset is the largest dataset in the field of image-based industrial anomaly detection. In addition to its very small defects which are hard to recognize, the dataset also offers the greatest variance of possible defects and the most defect classes. Describing the datasets with AD3 score it can be found that VIADUCT dataset have the most inconspicuous defects. With the AD3 score we are able to create a-priori knowledge for every single defect in IAD datasets. The AD3 score correlates with the results of the IAD method benchmark, showing that it can be used to estimate defect detection difficulty. In the future, new objects can be assessed to see whether defects can be recognized using IAD methods before an energy-intensive benchmark is performed. The simple calculation of the AD3 score generates valuable a-priori knowledge and can save resources."
RoomTex: Texturing Compositional Indoor Scenes via Iterative Inpainting
Authors: Qi WANG; Ruijie Lu; Xudong XU; Jingbo Wang; Michael Yu Wang; Bo Dai; Gang Zeng; Dan Xu
Venue: ECCV 2024 (main)

Abstract: "The advancement of diffusion models has pushed the boundary of text-to-3D object generation. While it is straightforward to composite objects into a scene with reasonable geometry, it is nontrivial to texture such a scene perfectly due to style inconsistency and occlusions between objects. To tackle these problems, we propose a coarse-to-fine 3D scene texturing framework, referred to as RoomTex, to generate high-fidelity and style-consistent textures for untextured compositional scene meshes. In the coarse stage, RoomTex first unwraps the scene mesh to a panoramic depth map and leverages ControlNet to generate a room panorama, which is regarded as the coarse reference to ensure the global texture consistency. In the fine stage, based on the panoramic image and perspective depth maps, RoomTex will refine and texture every single object in the room iteratively along a series of selected camera views, until this object is completely painted. Moreover, we propose to maintain superior alignment between RGB and depth spaces via subtle edge detection methods. Extensive experiments show our method is capable of generating high-quality and diverse room textures, and more importantly, supporting interactive fine-grained texture control and flexible scene editing thanks to our inpainting-based framework and compositional mesh input. Our project page is available at https://qwang666.github.io/RoomTex/."
Class-Agnostic Object Counting with Text-to-Image Diffusion Model
Authors: Xiaofei Hui; Qian Wu; Hossein Rahmani; Jun Liu
Venue: ECCV 2024 (main)

Abstract: "Class-agnostic object counting aims to count objects of arbitrary classes with limited information (, a few exemplars or the class names) provided. It requires the model to effectively acquire the characteristics of the target objects and accurately perform counting, which can be challenging. In this work, inspired by that text-to-image diffusion models hold rich knowledge and comprehensive understanding of real-world objects, we propose to leverage the pre-trained text-to-image diffusion model to facilitate class-agnostic object counting. Specifically, we propose a novel framework named CountDiff with careful designs, leveraging the pre-trained diffusion model’s comprehensive understanding of image contents to perform class-agnostic object counting. The experiments show the effectiveness of CountDiff on both few-shot setting with exemplars provided and zero-shot setting with class names provided."
Mask2Map: Vectorized HD Map Construction Using Bird's Eye View Segmentation Masks
Authors: Sehwan Choi; Jun Won Choi; Jungho Kim; Hongjae Shin
Venue: ECCV 2024 (main)

Abstract: "In this paper, we introduce Mask2Map, a novel end-to-end online HD map construction method designed for autonomous driving applications. Our approach focuses on predicting the class and ordered point set of map instances within a scene, represented in the bird’s eye view (BEV). Mask2Map consists of two primary components: the Instance-Level Mask Prediction Network (IMPNet) and the Mask-Driven Map Prediction Network (MMPNet). IMPNet generates Mask-Aware Queries and BEV Segmentation Masks to capture comprehensive semantic information globally. Subsequently, MMPNet enhances these query features using local contextual information through two submodules: the Positional Query Generator (PQG) and the Geometric Feature Extractor (GFE). PQG extracts instance-level positional queries by embedding BEV positional information into Mask-Aware Queries, while GFE utilizes BEV Segmentation Masks to generate point-level geometric features. However, we observed limited performance in Mask2Map due to inter-network inconsistency stemming from different predictions to Ground Truth (GT) matching between IMPNet and MMPNet. To tackle this challenge, we propose the Inter-network Denoising Training method, which guides the model to denoise the output affected by both noisy GT queries and perturbed GT Segmentation Masks. Our evaluation conducted on nuScenes and Argoverse2 benchmarks demonstrates that Mask2Map achieves remarkable performance improvements over previous state-of-the-art methods, with gains of 10.1% mAP and 4.1% mAP , respectively. Our code can be found at https://github.com/SehwanChoi0307/Mask"
SUP-NeRF: A Streamlined Unification of Pose Estimation and NeRF for Monocular 3D Object Reconstruction
Authors: Yuliang Guo; Abhinav Kumar; Cheng Zhao; Ruoyu Wang; Xinyu Huang; Liu Ren
Venue: ECCV 2024 (main)

Abstract: "Monocular 3D reconstruction for categorical objects heavily relies on accurately perceiving each object’s pose. While gradient-based optimization in a NeRF framework updates the initial pose, this paper highlights that scale-depth ambiguity in monocular object reconstruction causes failures when the initial pose deviates moderately from the true pose. Consequently, existing methods often depend on a third-party 3D object to provide an initial object pose, leading to increased complexity and generalization issues. To address these challenges, we present SUP-NeRF, a Streamlined Unification of object Pose estimation and NeRF-based object reconstruction. SUP-NeRF decouples the object’s dimension estimation and pose refinement to resolve the scale-depth ambiguity, and introduces a camera-invariant projected-box representation that generalizes cross different domains. While using a dedicated pose estimator that smoothly integrates into an object-centric NeRF, SUP-NeRF is free from external 3D detectors. SUP-NeRF achieves state-of-the-art results in both reconstruction and pose estimation tasks on the nuScenes dataset. Furthermore, SUP-NeRF exhibits exceptional cross-dataset generalization on the KITTI and Waymo datasets, surpassing prior methods with up to 50% reduction in rotation and translation error."
Forbes: Face Obfuscation Rendering via Backpropagation Refinement Scheme
Authors: Jintae Kim; Seungwon Yang; Seong-Gyun Jeong; Chang-Su Kim
Venue: ECCV 2024 (main)

Abstract: "A novel algorithm for face obfuscation, called Forbes, which aims to obfuscate facial appearance recognizable by humans but preserve the identity and attributes decipherable by machines, is proposed in this paper. Forbes first applies multiple obfuscating transformations with random parameters to an image to remove the identity information distinguishable by humans. Then, it optimizes the parameters to make the transformed image decipherable by machines based on the backpropagation refinement scheme. Finally, it renders an obfuscated image by applying the transformations with the optimized parameters. Experimental results on various datasets demonstrate that Forbes achieves both human indecipherability and machine decipherability excellently. The source codes are available at https://github.com/mcljtkim/Forbes."
Pyramid Diffusion for Fine 3D Large Scene Generation
Authors: Yuheng Liu; Xinke Li; Xueting Li; Lu Qi; Chongshou Li; Ming-Hsuan Yang
Venue: ECCV 2024 (main)

Abstract: "Diffusion models have shown remarkable results in generating 2D images and small-scale 3D objects. However, their application to the synthesis of large-scale 3D scenes has been rarely explored. This is mainly due to the inherent complexity and bulky size of 3D scenery data, particularly outdoor scenes, and the limited availability of comprehensive real-world datasets, which makes training a stable scene diffusion model challenging. In this work, we explore how to effectively generate large-scale 3D scenes using the coarse-to-fine paradigm. We introduce a framework, the Pyramid Discrete Diffusion model (PDD), which employs scale-varied diffusion models to progressively generate high-quality outdoor scenes. Experimental results of PDD demonstrate our successful exploration in generating 3D scenes both unconditionally and conditionally. We further showcase the data compatibility of the PDD model, due to its multi-scale architecture: a PDD model trained on one dataset can be easily fine-tuned with another dataset. Code is available at https://github.com/yuhengliu02/pyramid-discrete-diffusio"
ShoeModel: Learning to Wear on the User-specified Shoes via Diffusion Model
Authors: Wenyu Li; Binghui Chen; Yifeng Geng; Xuansong Xie; Wangmeng Zuo
Venue: ECCV 2024 (main)

Abstract: "With the development of the large-scale diffusion model, Artificial Intelligence Generated Content (AIGC) techniques are popular recently. However, how to truly make it serve our daily lives remains an open question. To this end, in this paper, we focus on employing AIGC techniques in one filed of E-commerce marketing, , generating hyper-realistic advertising images for displaying user-specified shoes by human. Specifically, we propose a shoe-wearing system, called ShoeModel, to generate plausible images of human legs interacting with the given shoes. It consists of three modules: (1) shoe wearable-area detection module (WD), (2) leg-pose synthesis module (LpS) and the final (3) shoe-wearing image generation module (SW). Them three are performed in ordered stages. Compared to baselines, our ShoeModel is shown to generalize better to different type of shoes and has ability of keeping the ID-consistency of the given shoes, as well as automatically producing reasonable interactions with human. Extensive experiments show the effectiveness of our proposed shoe-wearing system. Figure ?? shows the input and output examples of our ShoeModel."